Full-Stack Monitoring with Prometheus & Grafana

This sets up a Prometheus + Grafana stack that scrapes real targets, alerts on symptoms users actually feel rather than on raw resource metrics like CPU% by default, and provisions the dashboard as code.

Scraping targets with prometheus.yml

Prometheus pulls metrics rather than receiving them, so the core of the config is just: where to look, and how often.

global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
scrape_configs:
  - job_name: "checkout-api"
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ["payments"]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: checkout-api
        action: keep
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
 
  - job_name: "node-exporter"
    kubernetes_sd_configs:
      - role: endpointslice
        namespaces:
          names: ["monitoring"]
    relabel_configs:
      # one target per node-exporter pod, i.e. one per node
      - source_labels: [__meta_kubernetes_service_name]
        regex: node-exporter
        action: keep
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node
 
rule_files:
  - "alert.rules.yml"
 
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

kubernetes_sd_configs does service discovery so you're not hand-maintaining a target list every time a pod reschedules, and relabel_configs filters that discovery down to the one app you actually asked for, skip it and Prometheus scrapes every pod in the namespace. The keep action with a regex match on a pod label is the pattern I reach for first.

The node-exporter job uses discovery for a different reason. A static node-exporter:9100 target goes through the Service, which load-balances each scrape to whichever node-exporter pod it picks, so you'd get a different node's metrics every 15 seconds under one series. The endpointslice role expands the Service into its backing pods instead, giving one target per DaemonSet pod, labelled with the node it runs on. Prometheus needs RBAC to list and watch pods, services and EndpointSlices for this. If you run the Prometheus Operator (kube-prometheus-stack), you'd express both jobs as a ServiceMonitor or PodMonitor rather than hand-writing scrape_configs, and the chart ships the node-exporter one for you.

Picking what to alert on: RED and USE

The fastest way to alert fatigue is alerting on whatever the exporter happens to expose. Two frameworks fix that:

• RED (for request-driven services): Rate, Errors, Duration. If a service handles requests, these three cover what a user actually experiences.
• USE (for resources): Utilization, Saturation, Errors. Use this for the infrastructure underneath: nodes, disks, queues.

CPU utilization is a USE metric about the node. "Requests are timing out" is a RED metric about the user's experience. Alert on the second, and use the first as debugging context once you're already paged.

A real alerting rule: error rate with for:

Here's the alert I ship for an HTTP service, built around RED's error rate:

groups:
  - name: checkout-api.rules
    rules:
      - alert: CheckoutAPIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout-api", status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout-api"}[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "Checkout API error rate above 2% for 10 minutes"
          description: "{{ $value | humanizePercentage }} of requests to checkout-api are returning 5xx over the last 5 minutes."
          runbook_url: "https://runbooks.internal/checkout-api-errors"

The for: 10m is the single most under-used field in Prometheus alerting. Without it, the alert fires the instant the expression crosses 2%, even on a single 30-second blip during a deploy, then immediately resolves. Requiring the condition to hold for 10 straight minutes filters that out while still catching anything real and sustained. I tune the duration per alert: 2-3 minutes for something genuinely urgent, 10-15 minutes for anything that self-heals from normal retry/backoff behavior.

Routing and grouping in Alertmanager

Prometheus decides whether to alert; Alertmanager decides who hears about it and how often. Route by the labels your rules already set:

route:
  receiver: "default-slack"
  group_by: ["alertname", "team"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
        - team="payments"
      receiver: "payments-oncall-pagerduty"
      continue: false
 
receivers:
  - name: "default-slack"
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook-url
        channel: "#alerts-general"
  - name: "payments-oncall-pagerduty"
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-payments

Alertmanager doesn't template its own config file, so secrets have to arrive as files. api_url_file and routing_key_file read the Slack webhook URL and the PagerDuty integration key at load time, and neither ends up in the repo. Put both in one Kubernetes Secret and mount it read-only at /etc/alertmanager/secrets in the Alertmanager pod. Each key becomes a file of the same name:

kubectl -n monitoring create secret generic alertmanager-secrets \
  --from-literal=slack-webhook-url='https://hooks.slack.com/services/XXX/YYY/ZZZ' \
  --from-literal=pagerduty-payments='<pagerduty-integration-key>'

In a GitOps setup I'd have External Secrets (or a sealed secret) create that Secret from the cloud secret manager rather than running this by hand. With the Prometheus Operator, list it under spec.secrets on the Alertmanager resource instead. The operator mounts it at /etc/alertmanager/secrets/alertmanager-secrets/, so adjust the paths to match.

group_by is what stops a single bad deploy from paging you fifteen times for fifteen pods of the same rollout. group_wait gives related alerts a short window to arrive together before the first notification fires; repeat_interval controls how often an unresolved alert re-notifies, which should be long enough that on-call isn't repaged every few minutes.

Provisioning the Grafana dashboard as code

The dashboard that matters is the one that ships with the service, not the one someone builds by hand in the UI. Provisioning does that:

Grafana reads datasources and dashboard providers from separate directories under /etc/grafana/provisioning/, so you need two files. The datasource:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
 
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

And the dashboard provider, which tells Grafana where to find the dashboard JSON:

# /etc/grafana/provisioning/dashboards/checkout-api.yaml
apiVersion: 1
 
providers:
  - name: "checkout-api"
    orgId: 1
    folder: "Payments"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/checkout-api

Pinning the datasource uid lets the dashboard JSON reference it by a stable ID across environments. I set the folder explicitly with folder. The alternative, foldersFromFilesStructure: true, derives folders from the directory layout and requires folder and folderUid to be unset, so you have to pick one.

Pair that with a dashboard JSON file in /var/lib/grafana/dashboards/checkout-api/ (generated once from the UI, then checked into the repo), and the dashboard, its datasource, and alert rules all deploy together through the same pipeline as the service. Without that step, the dashboard drifts as soon as someone edits a panel by hand, and eventually nobody trusts it.

How the pieces fit together

Closing

Scrape config, one alert rule, a routing tree, one provisioned dashboard. What separates a team that trusts its alerts from one that mutes the channel is whether every alert answers "is a user affected right now," backed by a for: window long enough to ignore the noise.