Data/stats capture for analytics

Aidan · 3 November 2022 01:00

Most of the information exposed through the Grafana analytics platform is obtained by running NCI tools daily on a daily basis, using the Jenkins instance running on the same VM as the Grafana tools.

The Jenkins job status is viewable here

https://accessdev.nci.org.au/jenkins/blue/organizations/jenkins/CLEX%20Admin%2Fdump_stats/activity

The scripts themselves are in this repository:

The tools use a dedicated conda environment to isolate the stat dumping from changes in the main conda/analysis3 environment.:

github.com

coecms/dumpstats/blob/master/run.sh#L9


      
          #!/bin/bash
          
          
eval "$(conda shell.bash hook)"
          
          
# Ensure correct modules loaded
          module purge
          module use /g/data3/hh5/public/modules
          module load conda
          conda activate stats
          
          
# Get latest version of config.yaml
          git pull
          
          
# Run doit with the default dodo.py script. Use a bunch
          # of threads to speed it up. Select just the dump_SU and
          # dump_storage tasks
          doit -n 4 -P thread dump_SU dump_lquota dump_storage
          
          
# Split the upload into a separate call using a single thread

The environment is located here on gadi:

/g/data3/hh5/public/apps/miniconda3/envs/cms/stats

As it isn’t designed to be updated frequently, updates are done by loading the environment and manually updating installed packages.

Aidan · 3 November 2022 01:04

@dale.roberts This is a start of documenting the stats dumping aspect of the analytics. I’ve changed this to a wiki post to emphasise it should be edited and updated.

Topic		Replies	Views
Stats capture should run as a service user analytics	6	191	23 October 2022
Grafana database cleanup analytics grafana	15	500	22 November 2022
About the analytics category analytics analytics , grafana , nci	0	122	20 October 2022
ACCESS-NRI Postgres DB for capturing provenance, usage metrics and COSIMA Cookbook index analytics analytics , cosima , provenance	9	209	17 November 2022
ACCESS-NRI Intake catalog released today. Here is what you need to know ACCESS-NRI Releases python , release , catalogue , intake	1	121	5 December 2023

Data/stats capture for analytics

Related Topics