Yeazdani monogramYeazdaniListen

System design · AWS

From device to insight: an IoT telemetry pipeline that does not melt S3 or your bill

12 min read

Field devices are chatty, clocks are wrong, and “just dump it in S3” becomes a 400-million-object lake. This is the pipeline I use when telemetry has to be cheap to ingest and still queryable.

IoT CoreIoT RulesKinesis Data StreamsFirehoseS3GlueAthena

My undergraduate work was an IoT waste-collection system. The research question was routing. The engineering question, years later, is still the same: a thousand devices will happily send you a JSON blob every ten seconds, and none of them agree on time, schema, or what “full” means. The pipeline has to absorb that without coupling ingest to analytics.

Ingest is a contract with the device

Devices publish MQTT to AWS IoT Core. Topics are structured: `factory/{siteId}/bin/{deviceId}/telemetry`. An IoT Topic Rule fans out. I do not run business logic in the rule SQL. The rule’s job is routing and a light envelope: device id from the topic, a broker timestamp, and the raw payload. Schema enforcement happens after the firehose, not before — a firmware bug must not drop the only evidence you have.

Two-speed pipeline
IoT Core (MQTT)
  → Rule: telemetry
      → Kinesis Data Streams     (hot path: alarms, live map)
      → Firehose → S3 (raw)      (cold path: lake)
  → Rule: shadows
      → Device shadow / DynamoDB (current state only)

Hot path consumers: Lambda / ECS workers
Cold path: Glue crawler or Iceberg table → Athena / QuickSight

Hot path versus lake

Alarms (“bin overflowing”, “truck off-route”) cannot wait for a crawler. Those events go through Kinesis with a partition key of deviceId so a single noisy device cannot reorder another device’s stream. Consumers keep a short window — typically 5–15 minutes of state in DynamoDB or Redis — and emit derived events. The lake is append-only Parquet, partitioned by date and site, never by device. Device-level partitions create millions of tiny files; Athena will charge you for the privilege of listing them.

Time is a lie

  • Store broker_received_at from IoT Core and device_observed_at from the payload. Reports always join on broker time. Device clocks drift, especially on cheap GSM modules.
  • Reject or quarantine payloads whose observed time is more than 24 hours in the future. Do not silently write them into next year’s partition.
  • Deduplicate on (deviceId, device_observed_at, sequence) with a DynamoDB TTL table on the hot path. The lake stays raw; duplicates are a query concern.

What this buys you

Operations get a live map from the hot path. Finance gets a cost that scales with compacted bytes, not MQTT messages. Data science gets a stable Athena table instead of a folder of yesterday’s JSON. And when a firmware release corrupts a field, you still have the raw objects — which is the whole point of not validating too early.