Start with the question, not the monitoring stack
“The machine feels slow” is not a metric. Before choosing a dashboard, collector, or database, turn the complaint into a question that a specific signal can answer. That keeps an investigation small enough to understand and makes retention, labels, and alerts consequences of the question instead of defaults copied from a tool.
This is a source-backed architecture explainer. Bottom, Beszel, Prometheus, and InfluxDB were not installed or run. No overhead, scale, support, retention, reliability, or “lighter than” result is claimed, and no private machine inventory or production telemetry was used.
Four observation layers
- Now — interactive host inspection. Which process or resource is changing while I watch?
- Host history — host agent and history. Did CPU, memory, disk, network, temperature, or a container change before the symptom?
- Service behavior — scraped dimensional metrics. Which request, queue, error, or dependency signal changed across instances?
- Longer-lived events — purpose-chosen time-series storage. Do I need a general time-series record queried beyond one monitoring server’s local history?
The layers can complement each other, but more layers are not automatically better. If the question is “which process is consuming memory right now?”, a terminal view can be more direct than building a long-retention metrics system. If the question is “did errors rise only on one service instance last night?”, a process screen is not enough.
What the examples actually establish
| Example | Narrow documented role | Do not infer |
|---|---|---|
| Bottom | Its canonical repository describes a cross-platform terminal process/system monitor with CPU, memory, network, disk, temperature, and process views. | That every signal exists on every platform, that it records durable history, or that it has negligible overhead. |
| Beszel | Official documentation describes a hub with agents that collect system and container metrics and retain history. | That it fits a particular network, security model, fleet size, or operating system without current verification. |
| Prometheus | Official docs define timestamped time series identified by metric names and labels; its server includes a local TSDB and remote-storage interfaces. | That high-cardinality labels are harmless, local storage is a cluster, or any exporter answers the reader’s actual question. |
| InfluxDB 3 Core | Current official docs describe line-protocol writes and SQL/InfluxQL queries for time-series data. | Behavior from InfluxDB 1.x or 2.x, Flux support in Core, unlimited query behavior, or that a second database is necessary. |
Write an observation plan
Download the blank question-to-signal worksheet. For one symptom, write one primary question and complete these fields:
- Decision: what will change if the signal crosses the expected boundary?
- Signal: name the measurement and unit, not merely the dashboard panel.
- Scope: process, host, container, service instance, request path, or another explicit subject.
- Dimensions: keep aggregation labels bounded; use separate unique identifiers only when correlation genuinely requires them.
- Cadence and window: choose how often to observe and how far back the question needs.
- Expected comparison: declare a baseline, previous window, known event, or other reference before looking for a story.
- Retention and access: name who needs the data, for how long, and how it will be deleted or exported.
- Action: state the human response; an alert without an action is merely an interruption.
If the question cannot name a decision, defer collection. If the proposed label space can grow with users, requests, paths, or arbitrary values, reconsider the data model before ingestion. If a shorter-lived local observation answers the question, do not manufacture a permanent telemetry obligation.
A worked example without measurements
Symptom: an invented build becomes intermittently slow. A bounded plan might begin with “is one local process saturating CPU or memory during the slow interval?” If the symptom spans time, the next question might be “does host resource pressure consistently precede it?” Only after the behavior is tied to a service would request/error metrics and dimensions become relevant. The sequence is hypothetical; no build, host, or metric was observed.
Primary sources:
- Bottom canonical repository
- Beszel official documentation
- Prometheus data model
- Prometheus storage
- InfluxDB 3 Core reference
- InfluxDB 3 Core query documentation
The four-layer question map, worksheet, example, and conclusions are original. Product pages support only the narrow roles stated above.