[Question]: Is it possible to deploy NVSentinel on-prem on an HPC cluster with a GPU partition managed by SLURM.

[ovalerio]

Prerequisites

• I searched existing issues and docs

Question

Dear NVSentinel Team,

I learned in the GTC News, about NVSentinel being a new tool for self-remediation of an organization GPU resources. I am interested in the hardware/node health detection components of the software.

Our system is not a kubernetes/cloud, but the more traditional on-prem HPC setup. We are already using the DCGM Exporter and found it useful to log and gather intelligence on the system status and utilization. This motivates my question:

Would it be possible to operate NVSentinel without all the bells and whistles? More like a traditional alarming system?

I would argue that would be a very valuable tool for our local sys admin team. :)

Thanks!

Category

Installation/Deployment

Context

No response

[lalitadithya]

Hey, yes we should be able to deploy just the health monitor and the platform connector to get information to a database a stop there. Can you please share more info on what monitoring/database systems you use so that I can see if I can provide some deployment options for you?

[ovalerio]

Thanks @lalitadithya,

For the hardware monitoring we use Prometheus and set up alarms that trigger when some threshold is hit. Stuff like temperature getting too high or humidity. This kind of monitoring is looking onto the hardware itself but sometimes double down to help us understand if there is load in the network or the users are using the storage in a parallel and GPFS friendly fashion.

For a few software pipelines that are run frequently we have created custom logging that post all the logging events to logstash and those will land in elasticsearch.

Therefore I would say prometheus/grafana/mysql and logstash/elasticsearch/kibana are the two monitoring options we are more familiar with.

My pipeline dream would be to have these sources overlapping and be able to tell if there are performance bottlenecks that our users are hitting unknowingly. The view inside our GPU nodes is a bit opaque. For the CPU nodes we can use reportseff and that directly gives an overview of the user jobs CPU and memory efficiency. I understand there is a jobstats module from Princeton, that will enable us to have that level of detail, but we haven't been able to set it up yet.

[lalitadithya]

Ahh, so we do have a way to push events into either MongoDB or Postgres database right now. You mentioned that you are already using elasticsearch/kibana, does this mean you have a mongodb database we can push events to?

If yes, what we can do would be do a add systemd service that runs the health monitors and platform connectors as docker containers (or just processes) that can push to mongodb. Would that be helpful for your usecase?

I'm thinking how we can prometheus without causing a cardinality issue.

[ovalerio]

HI @lalitadithya,

Thanks again for taking the time to explore this idea together. This is what I thought about your questions:

I would choose processes rather than containers. Most HPC clusters don't support running Docker containers and our team is not very familiar with them. We sometimes use apptainer and we can try with that one if needed, but I have the impression using a container will only add memory footprint without adding any value.

Regarding the choice between Postgres or MongoDB. I incline myself for Postgres. It is a much better fit with our current stack. It is SQL-friendly and Grafana talks to it natively without extra plugins. Having NVSentinel events landing to Postgres tables will make it easy to query and match.

The cardinality is an important concern. I would avoid bloating Prometheus. I would prefer NVSentinel writing structured event records to Postgres and then use Grafana to query both Prometheus time-series and the event history from Postgres.

Since we are already running the DCGM Exporter, I would like to confirm with you if NVSentinel's health monitor can share the existing DCGM process or it needs to run its own? I would prefer to avoid running two DCGM processes on the same node if possible.

[lalitadithya]

Ahh, I see. We do support postgres so it should be possible to use that as a backend.

If we'd like to make use of processes instead of containers, then you will likely have to build the binaries for now since the CI doesn't publish binaries it only does container images. You'd only need the platform connector, gpu-health-monitor and syslog-health-monitor. Once you have the binaries in places you will need to configure the platform connector with the postgres URL to start pushing to the database. You can find the migration scripts here. If you can share some info about your postgres setup such as what type of authentication you are using, version etc, I can try and generate the required config file and env vars for you.

Yes, it is possible to share the same instance of DCGM between the exporter and NVSentinel. That is typically how we run it on Kubernetes as well. DCGM exposes an API that can be queried by multiple clients.