Dashboards for managing NCI resources

Technical Infrastructure
1

Hello all.

I’m interested in developing simple tools to help monitor NCI resources across the various projects associated with 21st Century Weather.

Before I build some simple bash and python scripts, does the community have any existing solutions?

I note this thread has been locked:

That thread suggested some possibilities.

  • Rui at NCI has set up a ‘voila’ thingie on ARE that can do jupyter-based dashboards (do these still exist?)
  • Resurrecting the old CLEX dashboard
    GitHub - coecms/dashboard
  • Replicating an tools used by NRI to monitor their resources.

Suggestions are welcome :slight_smile:

1 Like
2

@rui.yang might be able to comment.

NCI systems folks might set up a regular dump of NCI resource commands if you ask nicely (help@nci.org.au). At least they have done so in the past. Otherwise you need a user who is a member of all the groups you want to capture and set up a scheduled job to do it. This is what we’re doing at ACCESS-NRI (from GitHub).

ACCESS-NRI is using a grafana cloud account (free tier) for the grafana dashboarding and a postgresql DB back-end on a NECTAR VM. Happy to share what we’ve done, but note we’re going to refactor the part that updates the DB to use the Django server/apps @tmcadam and @CharlesTurner have developed.

3

If you have access to your scheme’s info in mancini you can get csv files for historical compute and storage use for all of the scheme’s projects in the scheme dashboard. It’s a bit of a hassle to set up authentication to download automatically though - if you’re interested I can provide a script.

We have some python scripts that collect this as well as nci_account and nci_files_report output that dump the output into a /g/data space that’s mounted on a web server, then have a basic dashboard using plotly to present the data.

We may at some stage look at using PowerBI to set up a dashboard instead, since we have access to that through sharepoint.

4

Thanks for the replies.

Grafana cloud accounts with postgresql DB back-end on a NECTAR VM moving to Django server/apps is beyond my skills and time constraints at this stage.

But .csv files plotted using python and plotly would be in scope.

I do have memberships to all the relevant projects.

How does one obtain scheme info via Mancini?

Otherwise I can follow the NRI path and just set up a cron to capture output of nci accounting command-line tasks.

1 Like
5

You need to be listed as a scheme manager to see the scheme level info - whoever’s responsible in your org for distributing NCI compute resources

6

Ok I’m now delegated CI for all CoE projects - giving me access to project accounting data via

https://my.nci.org.au/mancini/project/<project-id>/accounting

Do I now follow up with NCI to get the .csv data? Is that the underlying data the plotly image on the Mancini page is using?

7

It’s a different interface https://my.nci.org.au/mancini/scheme/{scheme}/compute/csv, https://my.nci.org.au/mancini/scheme/{scheme}/storage/csv, but probably the same underlying data.

You should be able to get the data source that plotly is displaying by looking at your browser developer tools. You can authenticate with mancini in a python session with

from contextlib import contextmanager
from bs4 import BeautifulSoup as BS
import requests

@contextmanager
def mancini_session():
    with requests.Session() as s:
        s.headers['Origin'] = 'https://my.nci.org.au'

        r = s.get("https://my.nci.org.au/mancini/login")
        r.raise_for_status()

        # Collect xsrf tokens required to log in
        soup = BS(r.text, 'html.parser')
        login_form = soup.find(id='login-form')
        form = {}
        for i in login_form.find_all('input'):
            if i.get('type', None) == 'submit':
                continue
            form[i['name']] = i.get('value', None)

        # Do the login
        form['username'] = os.environ['SCHEME_USER']
        form['password'] = os.environ['SCHEME_PASS']
        headers = {'Referer': 'https://my.nci.org.au/mancini/login'}

        r = s.post('https://my.nci.org.au/mancini/login', data=form, headers=headers)
        r.raise_for_status()

        yield s

with mancini_session() as s:
    # Replace with whatever URL you have access to
    r = s.get(f'https://my.nci.org.au/mancini/scheme/{scheme}/compute/csv')
1 Like
8

If you’re already a member of all required projects however it’s probably simplest to just use nci_account’s info and not worry about mancini:

import requests
import pymunge

token = pymunge.encode().decode('utf-8')

url = f"http://gadi-pbs-01.gadi.nci.org.au:8811/v0/nciaccount/project/{project}"
headers = {'Authorization': f"MUNGE {token}"}

r = requests.get(url, headers=headers)
1 Like