Monitoring

Stormkit ships an optional monitoring stack: Prometheus, Grafana, and the standard exporters for the host, PostgreSQL and Redis. It answers the questions an operator asks about the box:

• Is the machine running out of CPU, memory or disk?
• Is PostgreSQL near its connection limit, and is Stormkit waiting on its own connection pool?
• Is Redis close to maxmemory, and is it evicting keys?

This is instance health, not per-app analytics. Traffic, response times and deployment history for individual applications already live in the Stormkit UI under Access Logs and Analytics; these dashboards deliberately have no app dimension.

Nothing here runs unless you ask for it. The configuration files sit on disk after installation, and a Docker Compose profile decides whether the containers start.

Enabling it

Two commands from the directory holding your docker-compose.yaml:

sed -i 's/^PROMETHEUS_METRICS=.*/PROMETHEUS_METRICS=true/' .env
docker compose --profile monitoring up -d

The second command does double duty. It starts the monitoring containers, and it recreates hosting and workerserver so they pick up the new environment variable. There is no separate restart step.

install.sh generates a GRAFANA_ADMIN_PASSWORD for you. If you are enabling this on an instance installed before monitoring existed, set one yourself — the profile refuses to start without it rather than falling back to admin/admin:

echo "GRAFANA_ADMIN_PASSWORD='$(openssl rand -base64 18 | tr -dc 'a-zA-Z0-9' | head -c 24)'" >> .env

Reaching Grafana

Grafana is published on loopback only, so it is not reachable from the internet. Open an SSH tunnel:

ssh -L 3000:127.0.0.1:3000 you@your-server

Then visit http://localhost:3000 and sign in as admin with the password from your .env. The dashboards are already provisioned under the Stormkit folder — there is nothing to import.

GRAFANA_ADMIN_PASSWORD is read only on Grafana's first start. Once the admin user exists in the grafana volume, editing that variable and restarting has no effect — the old password keeps working. This matters if you are rotating after a suspected leak: change it in Grafana itself, or run

docker compose exec grafana grafana-cli admin reset-admin-password <new-password>

If you would rather expose Grafana properly, put your own TLS termination and authentication in front of it. Do not simply change the published address to 0.0.0.0.

The dashboards

Stormkit — Host. Disk free and used per filesystem, CPU by mode, memory, load average against core count, and disk I/O. Container layer mounts (tmpfs, overlay) are filtered out so the disk figures describe real disks. It also charts resident memory and goroutine counts for the two Stormkit processes, where steady monotonic growth is the signature of a leak.

Stormkit — Dependencies. PostgreSQL connections against max_connections, database size, deadlocks, cache hit ratio and transaction rates; Redis memory against maxmemory, evictions, hit ratio and client counts. It also has a row for Stormkit's own connection pool, described below.

The first panel on the Host dashboard is Scrape targets up, expected to read 5. If it reads 3, the two stormkit-* targets are down and PROMETHEUS_METRICS is almost certainly still false.

Stormkit's own metrics

Almost everything on these dashboards comes from the standard exporters. The one thing Stormkit exports itself is its database connection pool:

Metric	Meaning
stormkit_db_connections{state}	Connections in the pool, in_use or idle
stormkit_db_connections_max	The pool's configured maximum
stormkit_db_wait_total	Times a caller had to wait for a free connection
stormkit_db_wait_seconds_total	Total time spent waiting
stormkit_db_closed_total{reason}	Connections closed, by the limit that closed them

stormkit_db_wait_total is the one worth watching. postgres_exporter reports what the server sees; it cannot tell you that Stormkit is queuing for a slot in its own pool. A sustained wait rate means the pool is undersized, and it can happen while PostgreSQL itself looks completely idle.

Stormkit also exports HTTP response time (stormkit_lb_response_time_ms) and the usual Go runtime metrics.

Using your own Prometheus

You do not need the bundled stack. Set PROMETHEUS_METRICS=true and scrape hosting:2112 and workerserver:2112 from wherever your Prometheus lives.

PROMETHEUS_PORT changes the port. It is used exactly as given and never silently moves, so your scrape configuration cannot end up pointing at nothing. If the port is taken or the value is not a valid port number, metrics are disabled and an error is logged — the service itself keeps serving traffic, so check the logs if a target is unexpectedly down. Changing this in the bundled stack also means updating the targets in monitoring/prometheus.yml, which addresses hosting:2112 and workerserver:2112 directly.

The dashboard JSON under monitoring/grafana/dashboards/ imports into any Grafana, as long as your Prometheus datasource has the uid stormkit-prometheus or you remap it during import.

Turning it off

docker compose --profile monitoring down

Add -v to discard the retained metrics data as well. Setting PROMETHEUS_METRICS=false and recreating the services stops Stormkit exposing metrics at all.

Security notes

• The metrics endpoints on port 2112 are unauthenticated. They are never published to the host — only reachable from inside the Compose network.
• Prometheus is not published to the host either. Its API is unauthenticated and allows arbitrary queries, so only Grafana can reach it.
• Grafana binds 127.0.0.1, disables anonymous access and disables sign-up.
• node_exporter mounts the host filesystem read-only. This is safe because no deployed application code runs in that container. Do not replicate that mount into hosting or workerserver, where deployments execute as child processes.
• postgres_exporter currently reuses the Stormkit database credentials. If you want to tighten that, create a dedicated monitoring role with pg_monitor and point DATA_SOURCE_NAME at it.

Installing the files on an older instance

Instances installed before monitoring existed will not have the configuration on disk. From the directory holding your docker-compose.yaml:

BASE=https://raw.githubusercontent.com/stormkit-io/stormkit-io/main/deploy/monitoring

mkdir -p monitoring/grafana/provisioning/datasources \
         monitoring/grafana/provisioning/dashboards \
         monitoring/grafana/dashboards

curl -sfo monitoring/prometheus.yml "$BASE/prometheus.yml"
curl -sfo monitoring/grafana/provisioning/datasources/prometheus.yml "$BASE/grafana/provisioning/datasources/prometheus.yml"
curl -sfo monitoring/grafana/provisioning/dashboards/dashboards.yml "$BASE/grafana/provisioning/dashboards/dashboards.yml"
curl -sfo monitoring/grafana/dashboards/stormkit-host.json "$BASE/grafana/dashboards/stormkit-host.json"
curl -sfo monitoring/grafana/dashboards/stormkit-dependencies.json "$BASE/grafana/dashboards/stormkit-dependencies.json"

You will also need the monitoring services in your docker-compose.yaml. The simplest route is to re-download it, since the file is not meant to be edited by hand:

curl -so docker-compose.yaml https://raw.githubusercontent.com/stormkit-io/stormkit-io/main/deploy/docker-compose.yaml