Add prometheus

[RMcVelia]

Context

Add prometheus monitoring for non-prod clusters

Currently scrapes below metrics

• prometheus service
  /metrics from the prometheus svc endpoint
• cadvisor
  /api/v1/nodes/${node}/proxy/metrics/cadvisor
• nodes
  /api/v1/nodes/${node}/proxy/metrics
• api server
  /metrics from the prometheus api endpoint
  had to disable tls verify for it to work.

May need to think about if we want all of these metrics.
Can check node api metrics via kubectl get --raw /api/v1/nodes/${node}/proxy/metrics or /proxy/metrics/cadvisor

Alert rules, scrape configs, etc will be improved in later PR's.

Changes proposed in this pull request

Add required terraform resources
Add initial doc

Guidance to review

make development terraform-plan ENVIRONMENT=cluster6
scripts/pfwd.sh
http://localhost:8080/

Before merging

Remove "--storage.tsdb.retention.time=12h" from the prometheus deployment

Checklist

• I have performed a self-review of my code, including formatting and typos
• I have cleaned the commit history
• I have added the Devops label
• I have attached the pull request to the trello card

[temitope777]

This is quite interesting @RMcVelia
good job