Creating a simple alert in Prometheus

Create a new file /etc/prometheus/rules.yaml with following content:

groups:
- name: node
  rules:
  - alert: LowDiskSpace
    expr: 100 * node_filesystem_free_bytes{job="nodes"} / node_filesystem_size_bytes{job="nodes"} < 10
    labels:
      severity: warning
      environment: prod
    annotations:
      summary: "node {{.Labels.instance}} has low disk space"

You can see that defining alert is the same as defining a rule. Next, update the /etc/prometheus/prometheus.yml:

...
rule_files:
  - "/etc/prometheus/rules.yaml"
...

Restart Prometheus:

systemctl restart prometheus

Finally, navigate to Alerts in the Prometheus UI