Add option to not expose specifc metrics (e.g. userdata)?

[buti1021]

Hey everyone,
Thanks for the great work in this project!
We have a potential requirement on which I'd like to get your opinion on.
As the data in our prometheus is stored for quite a long time and prometheus isn't too great at specifying different retention times for different metrics, the data exposed via the slurm_user_* metrics are collected and stored in our prometheus just as the data that does not include any personal information. We don't want to store personal data for that long.

We have at least two options here:

1. Adjust the scraping to not scrape the slurm_user_* metrics. This would probably be quite easy to configure via prometheus. However the exporter would then still expose the data (which is not a big problem, but uneccessary in our case)
2. Change which metrics are exposed by the prometheus_slurm_exporter. As far as I can tell, this is currently not possible? This would probably be the better solution. However, I am not sure if this would be useful for others and thus worth the work or not and how much work that would require (I'm not experienced with go and haven't done a deep dive into the source code) I'd probably say adding an option like --exclude-metrics which takes a comma separated list would be a good way to go, but happy for other opinions as well.

Thanks!

[abhinavDhulipala]

Hi! This is a great question! Thanks for contributing. Tentatively, I'm thinking a simple regex filter will suffice. This will take a bit of work though, as we will essentially need to a writer object that will filter based on the regex. I.e wherever we have the following:

ch <- prometheus.MustNewConstMetric(jc.featureJobMemAlloc, prometheus.GaugeValue, metric.allocMem, feature)

We replace it with:

config.EmitOrIgnore(ch, jc.featureJobMemAlloc, prometheus.GaugeValue, metric.allocMem, feature)

At least, off the top of my head, that's the best I could do. Maybe, the prometheus client library has some built in functionality for this. Not sure, I'll take a look around and see what other exporters do. I'm open to contributions if you're willing, otherwise, I'd say we can get to this sometime late next week.