}

Prometheus + Grafana: Linux Server Monitoring from Scratch (2026)

Last updated: 10 min read

Prometheus + Grafana: Linux Server Monitoring from Scratch (2026)

This tutorial walks you through building a complete Linux monitoring stack using Prometheus 3.10.0, Node Exporter 1.10.2, and Grafana 12.4.2, tested on Rocky Linux 10.1 in March 2026. By the end you will have real-time dashboards, PromQL queries for CPU/memory/disk, and Alertmanager sending notifications to email and Slack.

Architecture Overview

The stack is composed of four components that work together:

  • Node Exporter (port 9100) — runs on every Linux host and exposes thousands of OS-level metrics in Prometheus exposition format.
  • Prometheus (port 9090) — a pull-based time-series database. Every 15 seconds it scrapes /metrics from the configured targets (Node Exporter endpoints) and stores the data locally.
  • Grafana (port 3000) — a visualization layer. It queries Prometheus via its HTTP API and renders dashboards with charts, gauges, and tables.
  • Alertmanager (port 9093) — receives firing alerts from Prometheus and routes them to email, Slack, PagerDuty, or any other receiver.

The data flow is: Node Exporter → Prometheus (scrapes) → Grafana (queries) and Prometheus → Alertmanager (pushes alerts).

Step 1: Install Node Exporter

Node Exporter is a single static binary with no dependencies.

# Download and extract
curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.10.2/node_exporter-1.10.2.linux-amd64.tar.gz
tar xzf node_exporter-1.10.2.linux-amd64.tar.gz
sudo mv node_exporter-1.10.2.linux-amd64/node_exporter /usr/local/bin/

# Create a dedicated system user
sudo useradd --no-create-home --shell /bin/false node_exporter

Create the systemd unit file at /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Verify it is running and exposing metrics:

curl -s http://localhost:9100/metrics | head -20

You should see lines like node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67. If you do, Node Exporter is working correctly.

Step 2: Install Prometheus

curl -LO https://github.com/prometheus/prometheus/releases/download/v3.10.0/prometheus-3.10.0.linux-amd64.tar.gz
tar xzf prometheus-3.10.0.linux-amd64.tar.gz
sudo mv prometheus-3.10.0.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus-3.10.0.linux-amd64/{consoles,console_libraries} /etc/prometheus/

sudo useradd --no-create-home --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

prometheus.yml

Create /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s        # How often to scrape targets
  evaluation_interval: 15s    # How often to evaluate alerting rules

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]
    # To monitor additional hosts, add them here:
    # - targets: ["192.168.1.10:9100", "192.168.1.11:9100"]

The global.scrape_interval of 15 seconds is a good default. Lower values give finer resolution at the cost of more storage; higher values reduce cardinality. Create the systemd unit at /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus Monitoring
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Open http://localhost:9090 in a browser and navigate to Status → Targets to confirm both prometheus and node jobs show UP.

Step 3: Install Grafana

Grafana provides official YUM and APT repositories.

Rocky Linux / RHEL / Fedora:

cat <<'EOF' | sudo tee /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana

Ubuntu / Debian:

sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://apt.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y grafana

Start and enable Grafana:

sudo systemctl enable --now grafana-server

Access Grafana at http://localhost:3000. The default credentials are admin / admin. You will be prompted to change the password on first login.

Step 4: Add Prometheus as a Data Source

  1. In Grafana, click Connections → Data Sources → Add data source.
  2. Select Prometheus.
  3. Set the URL to http://localhost:9090.
  4. Leave authentication empty (we will add security later).
  5. Click Save & test. You should see "Successfully queried the Prometheus API."

Step 5: Import the Node Exporter Full Dashboard

Dashboard ID 1860 (Node Exporter Full) is the most-imported dashboard on Grafana Labs, with over 30 million downloads. It provides pre-built panels for CPU, memory, disk, network, and more.

  1. Click Dashboards → Import.
  2. Enter 1860 in the "Import via grafana.com" field and click Load.
  3. Select your Prometheus data source from the dropdown.
  4. Click Import.

You now have a production-quality dashboard with no manual panel configuration required.

Step 6: PromQL Basics

PromQL (Prometheus Query Language) is used both inside Grafana panels and in alerting rules. Here are the essential metrics exposed by Node Exporter:

# Total CPU seconds broken down by mode (idle, user, system, iowait)
node_cpu_seconds_total

# Available memory in bytes
node_memory_MemAvailable_bytes

# Time spent doing I/O on a disk (in seconds)
node_disk_io_time_seconds_total

# Filesystem size and available space
node_filesystem_size_bytes
node_filesystem_avail_bytes

The rate() function calculates the per-second average rate of a counter over a time window:

# CPU usage percentage (all cores, all modes except idle)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

# Disk usage percentage for the root filesystem
(1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100

# Disk I/O utilization
rate(node_disk_io_time_seconds_total[5m])

Step 7: Create a Custom Dashboard

In Grafana, click Dashboards → New → New Dashboard → Add visualization.

CPU Usage panel:

  • Query: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
  • Visualization: Time series
  • Unit: Percent (0-100)
  • Threshold: 80% (orange), 90% (red)

Memory Usage panel:

  • Query: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
  • Visualization: Gauge
  • Unit: Percent (0-100)

Disk Space panel:

  • Query: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100
  • Visualization: Bar gauge
  • Unit: Percent (0-100)

Save the dashboard and set an auto-refresh interval of 30 seconds via the dropdown in the top-right corner.

Step 8: Alerting Rules in Prometheus

Create the directory and a rules file:

sudo mkdir -p /etc/prometheus/rules
sudo nano /etc/prometheus/rules/node_alerts.yml

groups:
  - name: node_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% for more than 5 minutes."

      - alert: DiskSpaceLow
        expr: (1 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})) * 100 > 90
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Disk space critical on {{ $labels.instance }}"
          description: "Root filesystem is {{ $value | printf \"%.1f\" }}% full."

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."

Reload Prometheus to pick up the new rules:

sudo systemctl reload prometheus
# Or send a SIGHUP if reload is not configured:
# curl -X POST http://localhost:9090/-/reload

Navigate to http://localhost:9090/alerts to see the rules and their current state (Inactive, Pending, or Firing).

Step 9: Alertmanager Setup

Download and install Alertmanager:

curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz
sudo mv alertmanager-0.27.0.linux-amd64/{alertmanager,amtool} /usr/local/bin/
sudo mkdir -p /etc/alertmanager

Create /etc/alertmanager/alertmanager.yml with both email and Slack receivers:

global:
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "[email protected]"
  smtp_auth_username: "[email protected]"
  smtp_auth_password: "YOUR_SMTP_PASSWORD"

route:
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "team-notifications"
  routes:
    - match:
        severity: critical
      receiver: "pagerduty-critical"

receivers:
  - name: "team-notifications"
    email_configs:
      - to: "[email protected]"
        send_resolved: true
    slack_configs:
      - api_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK"
        channel: "#alerts"
        send_resolved: true
        title: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "YOUR_PAGERDUTY_KEY"
        send_resolved: true

Create the systemd unit at /etc/systemd/system/alertmanager.service:

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo mkdir -p /var/lib/alertmanager
sudo chown prometheus:prometheus /var/lib/alertmanager
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

Docker Compose Alternative

If you want to run the entire stack without installing anything on the host, use this docker-compose.yml:

version: "3.9"

services:
  prometheus:
    image: prom/prometheus:v3.10.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
    ports:
      - "9090:9090"

  node_exporter:
    image: prom/node-exporter:v1.10.2
    pid: host
    network_mode: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"

  grafana:
    image: grafana/grafana:12.4.2
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

volumes:
  prometheus_data:
  grafana_data:

Start the entire stack with:

docker compose up -d

Security: Nginx Reverse Proxy with Basic Auth

Never expose Grafana or Prometheus directly to the internet. Put Nginx in front with TLS and basic authentication.

# Install nginx and apache2-utils (for htpasswd)
sudo dnf install -y nginx httpd-tools   # Rocky Linux
# sudo apt install -y nginx apache2-utils  # Ubuntu

# Create a password file
sudo htpasswd -c /etc/nginx/.htpasswd admin

Create /etc/nginx/conf.d/monitoring.conf:

server {
    listen 443 ssl;
    server_name monitoring.example.com;

    ssl_certificate     /etc/letsencrypt/live/monitoring.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/monitoring.example.com/privkey.pem;

    # Grafana
    location / {
        auth_basic           "Monitoring";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass           http://127.0.0.1:3000;
        proxy_set_header     Host $host;
        proxy_set_header     X-Real-IP $remote_addr;
    }

    # Prometheus (restrict to internal network only)
    location /prometheus/ {
        allow 10.0.0.0/8;
        deny  all;
        auth_basic           "Prometheus";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass           http://127.0.0.1:9090/;
    }
}

server {
    listen 80;
    server_name monitoring.example.com;
    return 301 https://$host$request_uri;
}

Apply firewall rules so only Nginx is reachable from outside:

# Rocky Linux (firewalld)
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --reload

# Block direct access to Prometheus, Grafana, and Alertmanager from external IPs
sudo firewall-cmd --permanent --add-rich-rule='rule port port="9090" protocol="tcp" source not address="127.0.0.1" drop'
sudo firewall-cmd --permanent --add-rich-rule='rule port port="3000" protocol="tcp" source not address="127.0.0.1" drop'
sudo firewall-cmd --permanent --add-rich-rule='rule port port="9093" protocol="tcp" source not address="127.0.0.1" drop'
sudo firewall-cmd --reload

Summary

You now have a production-grade monitoring setup:

ComponentVersionPortPurpose
Node Exporter1.10.29100OS metrics collection
Prometheus3.10.09090Metrics storage and alerting
Grafana12.4.23000Visualization and dashboards
Alertmanager0.27.09093Alert routing and notification

Key next steps:

  • Add more hosts by appending their host:9100 addresses to the node job targets in prometheus.yml.
  • Explore the Grafana dashboard marketplace for application-specific dashboards (MySQL, PostgreSQL, NGINX, Docker).
  • Set up Grafana alerting as an alternative to Prometheus/Alertmanager for simpler deployments.
  • Use promtool check config /etc/prometheus/prometheus.yml and promtool check rules /etc/prometheus/rules/*.yml to validate configuration before reloading.

Related Articles

Leonardo Lazzaro

Software engineer and technical writer. 10+ years experience in DevOps, Python, and Linux systems.

More articles by Leonardo Lazzaro