Separate telemetry, state, and events
Telemetry is a measurement such as temperature or battery voltage. State is the current interpreted condition such as online, high temperature, or maintenance. An event records that a transition occurred at a particular time. Keeping these concepts separate makes the system easier to query and prevents every measurement from becoming an alert.
Define topic and payload contracts before connecting many devices. Include protocol version, device identity, timestamp or sequence, units, and status flags. Validate payloads at the ingestion boundary. Preserve raw messages for diagnostics, but normalize data before applying common rules.
Use retained messages deliberately
A retained state message can help a new dashboard immediately show the last known value, but it is not proof that the device is currently online. Include measurement time and calculate staleness. Avoid retaining transient event notifications that should only be processed once.
Use a last-will message where it supports the design, but understand its limits. A clean disconnect may publish an explicit offline state, while a network failure relies on broker keepalive behavior. Application-level last-seen logic is still needed to account for devices that report infrequently.
Define offline status from expected behavior
A device reporting every minute and a battery sensor reporting every six hours need different thresholds. Store the expected interval per device profile and add a grace factor. Scheduled maintenance and intentionally inactive devices should not create the same alarm as an unexpected loss.
Generate an offline event only when the state changes from healthy to late or offline. Clear it when a valid message resumes. Record last good timestamp, expected interval, transition time, and recovery time. Repeating the same email every polling cycle creates alert fatigue without adding information.
Use persistence and hysteresis for measurement alarms
A temperature or level crossing a threshold for one noisy sample may not justify an alarm. Require persistence, a moving window, or repeated samples. Use a different clear threshold so the state does not chatter around one boundary. Document the rule in engineering units.
Where severity matters, define warning and critical states with independent transitions. Do not derive severity only in the display; the state engine should produce a consistent result for notifications, reports, and APIs. Preserve the measurement that triggered the transition.
Distinguish acknowledge from clear
Acknowledgement means a person saw or accepted responsibility for the event. Clear means the measured condition returned to normal or an authorized action closed it. These are different timestamps. An acknowledged high-temperature alarm may remain active until the temperature falls.
Keep assignee, acknowledgement note, clear reason, and user identity. This creates an audit trail and supports shift handover. A dashboard that removes an event as soon as someone clicks it loses operational history.
Queue notifications outside the rule engine
The rule engine should save the event and place a notification job in a queue. Email, SMS, WhatsApp, or push delivery can then retry without blocking telemetry processing. Store provider response and final status. Deduplicate jobs using the event identifier.
Apply escalation rules carefully: notify the operator first, then a supervisor if an active critical event remains unacknowledged. Rate-limit repeated notifications. Provide quiet hours only where safety and operations permit, and make suppressed notifications visible in the event record.
Test failure and replay behavior
Test broker restart, duplicate delivery, out-of-order timestamps, retained stale data, malformed payloads, network partition, delayed recovery, and notification-provider failure. MQTT quality of service helps delivery but does not remove the need for idempotent processing.
Use event IDs and device sequence numbers to avoid duplicate state transitions. Document broker configuration, access control, TLS, topic permissions, payload schema, rule versions, and retention. A dependable alert platform is a state-management system, not just a collection of threshold scripts.