Handling data

Know what kind of data you're handling — before you pick any method, name each signal by its shape, its purpose, and its direction. What a signal is decides how you move it, how fast, and where it's stored. Mixing kinds — and polling everything at the fastest rate — is what creates load a controller can't sustain.

Every point you touch is some combination of the three:

By shape — is it an event or a stream?

• Event — something happened at a moment: a button press, a state change, a fault, a batch complete. Discrete and irregular, and each one matters on its own. You handle it when it fires — you don't poll for it.
• Stream — a continuous series of readings sampled on a clock: temperature, flow, level, vibration. Regular and high-volume, where the latest value usually matters more than any single earlier one. You read it at a cadence and often only keep the trend.

By purpose — what is the value for?

• Telemetry — readings you observe and trend. Continuous and latency-tolerant: a second late only moves a timestamp.
• Control — a value that drives a decision or an actuator. Small and time-critical: a second late changes an outcome.
• Config — a setting that shapes how the app runs. Written rarely, read often.

By direction — read or write?

Reading a value out of a device and writing one back into it are not the same cost or risk. Writes touch the process; treat them with more care and a tighter path than the reads you take for observation.

Put it together — for each point, ask: event or stream? Does a delay change a decision or only a timestamp? Read or write? Those three answers set its rate, its path, and its store — and point you at the methods in the next tabs. The common mistake is treating every point as one population and polling all of it at the fastest rate; usually only a handful genuinely need the fast path, and the rest just starve the ones that do.

Telemetry one way, control another — give each kind its own route so the fast path never inherits the slow one's load.

Select when — a flow carries both telemetry and control (from Classify). This split is the foundational method; the pacing and storage choices all follow from it.

Telemetry and control have different timing needs. Share one route and one protocol, and the fast, time-critical path inherits the load of the slow, high-volume one — a burst of telemetry can delay a control decision.

How — buffer telemetry in the source and pull it at its real cadence, timestamped at acquisition, then batch it into history. Keep the control path to the minimum tag set on a dedicated, faster route. Reads you take for observation and writes that touch the process never share a lane.

The payoff — the fast path stays lean no matter how much telemetry flows, and each kind can be paced and stored on its own terms.

Pace a fast source into a slow sink — add the brake that a fast-in / slow-out path doesn't have on its own.

Select when — a fast or bursty source (a stream, or high-rate events) feeds a slower sink. This is the #1 event-driven failure, so reach for it whenever input can outrun output.

A fast source (an MQTT topic, a tight poll) feeding a slow sink (a DB write, a remote API) has no natural brake. Messages pile up in memory faster than they drain, and the runtime eventually runs out of heap and dies.

How — a delay node in rate-limit mode; batching (join into chunks), which also cuts per-write overhead; or dropping stale readings when only the latest matters. Watch heap and the node's queue — if it only grows, you have backpressure, not a spike.

The tell — a backend holding a live connection (an MQTT subscription) can't be pooled away; pace at the source or offload the heavy work.

One place for state — keep a logical object in context instead of threading it through wires.

Select when — you're passing the same object through many nodes just to move it, or state must survive a restart. Messages are verbs; context is nouns.

Context is shared memory with a defined scope. It holds a logical object in one place instead of threading it through fifteen nodes just to carry it — wire gymnastics to avoid storing a value is the real anti-pattern.

How — store the object once under a namespaced key at the narrowest scope that works. Node scope is private to that one node, so anything shared between nodes starts at flow scope (then global only if it must cross tabs). Each node reads the one key it needs, and a persistent store holds anything that must survive a restart.

Watch out — one writer per key, serialize concurrent updates, and keep enough on the wire to stay debuggable.

Static in env vars, runtime in persisted context — decide which kind each setting is, then store it accordingly.

Select when — the flow has settings. The question for each one: does it change per environment at deploy, or while running, by a user?

Static config changes per environment and is set at deploy — broker host, DB connection. It's resolved at deploy time and read-only to the running flow. Runtime config changes while running, by a user, with no redeploy.

How — keep static config in env vars or a config node; put user-editable config in persisted context (or a config file), edited through the UI via an intent message and read by the flow at execution time.

The rule — if a value would ever be changed through a button or form, it is not an env var.