Prometheus Monitoring Stack – Full Flow Diagram

Scenario 01

Run Prometheus inside Docker

:9090

Step 1

Check Docker Installation

docker --version

✓ Docker version 29.2.1

Step 2

Check Running Containers

docker ps

→ No containers running yet

Step 3

Create Monitoring Directory

rm -rf monitoring mkdir monitoring cd monitoring

→ Clean project folder created

Step 4

Confirm Directory Path

pwd

✓ /Users/manjunathkv/monitoring

Step 5

Create prometheus.yml

touch prometheus.yml nano prometheus.yml

→ Edit configuration file

Step 6

Remove Old Container

docker rm -f prometheus

→ Safety step; no container existed

Step 7

Run Prometheus Container

docker run -d \ --name prometheus \ -p 9090:9090 \ -v $(pwd)/prometheus.yml:\ /etc/prometheus/prometheus.yml \ prom/prometheus

✓ Container ID: 2d28fea6363d…

Step 8

Verify Container Running

docker ps

✓ prom/prometheus · Up · :9090

Step 9

Verify Prometheus UI

http://localhost:9090/targets

✓ State: UP · Target healthy

prometheus.yml config

global: scrape_interval: 15s scrape_configs: - job_name: "prometheus" static_configs: - targets: ["host.docker.internal:9090"]

Stack State After S01:

Prometheus :9090

Scenario 02

Add Node Exporter · CPU, RAM, Disk

:9100

Step 1

Pull Node Exporter Image

docker pull prom/node-exporter

→ Downloads from Docker Hub

Step 2

Run Node Exporter Container

docker run -d \ --name node-exporter \ -p 9100:9100 \ prom/node-exporter

✓ Background container started

Step 3

Verify Container

docker ps

✓ node-exporter · Up · :9100

Step 4

Check Metrics

http://localhost:9100/metrics

✓ node_cpu_seconds_total, node_memory, node_disk…

Step 5

Update prometheus.yml

nano ~/prometheus.yml

→ Add node_exporter job

Step 6

Restart Prometheus

docker restart prometheus

→ Reload new config

Step 7

Verify Two Targets UP

http://localhost:9090/targets

✓ prometheus · UP
✓ node_exporter · UP

Steps 8–10

Test Metrics Queries

node_cpu_seconds_total node_memory_MemAvailable_bytes node_disk_read_bytes_total

✓ CPU, RAM & Disk metrics visible

Updated prometheus.yml

global: scrape_interval: 15s scrape_configs: - job_name: "prometheus" static_configs: - targets: ["host.docker.internal:9090"] - job_name: "node_exporter" ← NEW static_configs: - targets: ["host.docker.internal:9100"]

Stack State After S02:

Prometheus :9090

Node Exporter :9100

Scenario 03

Add Alertmanager

:9093

Step 1

Create alertmanager.yml

cd ~/monitoring touch alertmanager.yml nano alertmanager.yml

→ Basic receiver config

Step 2

Pull Alertmanager Image

docker pull prom/alertmanager

→ Official image downloaded

Step 3

Run Alertmanager

docker run -d \ --name alertmanager \ -p 9093:9093 \ -v $(pwd)/alertmanager.yml:\ /etc/alertmanager/alertmanager.yml \ prom/alertmanager

✓ Container started on :9093

Step 4

Verify Container

docker ps

✓ alertmanager · Up · :9093

Step 5

Open Alertmanager UI

http://localhost:9093

✓ Alertmanager dashboard visible

Step 6

Connect Prometheus → Alertmanager

nano ~/prometheus.yml

→ Add alerting: section

Step 7

Restart Prometheus

docker restart prometheus

✓ Alertmanager now connected

Step 8

Verify in Prometheus UI

http://localhost:9090 Status → Runtime & Build Info

✓ Alertmanager: connected

Updated prometheus.yml

global: scrape_interval: 15s alerting: ← NEW alertmanagers: - static_configs: - targets: - "host.docker.internal:9093" scrape_configs: - job_name: "prometheus" static_configs: - targets: ["host.docker.internal:9090"] - job_name: "node_exporter" static_configs: - targets: ["host.docker.internal:9100"]

Stack State After S03:

Prometheus :9090

Node Exporter :9100

Alertmanager :9093

Scenario 04

Create CPU Alert Rule

HighCPUUsage > 80%

Step 1

Create alert.rules.yml

cd ~/monitoring touch alert.rules.yml nano alert.rules.yml

→ Define HighCPUUsage rule

Step 2

Add rule_files to prometheus.yml

nano ~/prometheus.yml

→ Add rule_files: section

Step 3

Remount Prometheus with Rules

docker rm -f prometheus docker run -d \ --name prometheus \ -p 9090:9090 \ -v ~/prometheus.yml:... \ -v ~/monitoring/alert.rules.yml:... \ prom/prometheus

✓ Both volumes mounted

Step 4

Verify Rules Loaded

http://localhost:9090 → Alerts

✓ HighCPUUsage → Inactive

Step 5

Test Alert Query

100 - (avg by(instance) (rate(node_cpu_seconds_total {mode="idle"}[1m])) * 100)

→ Current CPU % shown

Step 6

Watch Alert Trigger

http://localhost:9093

→ Alert appears when CPU >80% for 1min

alert.rules.yml

groups: - name: cpu-alerts rules: - alert: HighCPUUsage expr: (100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)) > 80 for: 1m labels: severity: warning annotations: summary: "High CPU usage detected" description: "CPU usage is above 80% for more than 1 minute."

Stack State After S04:

Prometheus :9090

Node Exporter :9100

Alertmanager :9093

Alert Rule: HighCPUUsage

Scenario 05

Email Alerts via Gmail SMTP

SMTP :587

Step 1

Edit alertmanager.yml

cd ~/monitoring nano alertmanager.yml

→ Add Gmail SMTP config

Step 2

Generate Gmail App Password

Google Account → Security → 2FA → App Passwords → Mail → Other

✓ 16-digit app password generated

Step 3

Restart Alertmanager

docker restart alertmanager

✓ New SMTP config loaded

Step 4

Verify Container

docker ps

✓ alertmanager → :9093 → running

Step 5

Check Prometheus → Alertmanager

http://localhost:9090/status → Alertmanagers

✓ host.docker.internal:9093 → UP

Step 6

Force CPU Load (Test)

yes > /dev/null & # wait ~1 min killall yes

→ Triggers alert → sends email

alertmanager.yml (SMTP)

global: smtp_smarthost: 'smtp.gmail.com:587' smtp_from: 'your@gmail.com' smtp_auth_username: 'your@gmail.com' smtp_auth_password: 'xxxx xxxx xxxx xxxx' ← App Password smtp_require_tls: true route: receiver: email-alert receivers: - name: email-alert email_configs: - to: 'recipient@gmail.com' send_resolved: true

Stack State After S05:

Prometheus :9090

Node Exporter :9100

Alertmanager :9093

Alert Rule Active

Email via Gmail SMTP

Scenario 06

Add Grafana Dashboard

:3000

Step 1

Pull Grafana Image

docker pull grafana/grafana

→ Official Grafana image downloaded

Step 2

Run Grafana Container

docker run -d \ --name grafana \ -p 3000:3000 \ grafana/grafana

✓ Running on port 3000

Step 3

Verify 4 Containers Running

docker ps

✓ prometheus, node-exporter,
alertmanager, grafana

Step 4

Login to Grafana

http://localhost:3000 user: admin / pass: admin

→ Change password on first login

Step 5

Add Prometheus Data Source

Connections → Data Sources → Add → Prometheus URL: http://host.docker.internal:9090 → Save & Test

✓ Data source is working

Step 6

Import Node Exporter Dashboard

Dashboards → Import Dashboard ID: 1860 Select: Prometheus → Import

✓ Full dashboard loaded

Step 7

View Live Metrics

CPU Usage · Memory Usage Disk IO · Disk Space Network Traffic · System Load

✓ Graphs updating every few seconds

Final Stack:

Prometheus :9090

Node Exporter :9100

Alertmanager :9093

Alert Rule Active

Email Alerts Live

Grafana :3000