Manufacturing IoT Data Pipelines on AWS

Modern manufacturing equipment is instrumented to a degree that would have seemed extraordinary a decade ago. A CNC machining centre generates vibration, temperature, spindle load, and coolant pressure readings at 100Hz. A packaging line PLC publishes hundreds of tag values every second. A conveyor system sends motor current, speed, and belt tension data continuously. The challenge for manufacturers is not collecting this data — it arrives whether you want it or not. The challenge is building the infrastructure to store it efficiently, query it meaningfully, and act on it before a bearing failure costs $200,000 in unplanned downtime.

This post describes the AWS architecture Infra IT Consulting uses for manufacturing IoT data pipelines: IoT Core for device connectivity, Kinesis Data Streams for real-time ingestion, Timestream for time-series storage and query, SageMaker for predictive maintenance models, and the OPC-UA integration pattern that connects plant-floor systems to the cloud pipeline.

OPC-UA Integration: Bridging the Plant Floor and the Cloud

OPC Unified Architecture (OPC-UA) is the dominant standard for industrial data exchange. PLCs from Siemens, Rockwell Automation, Schneider Electric, and Beckhoff all publish data via OPC-UA servers, and any serious manufacturing IoT architecture must speak it natively rather than requiring proprietary adapters for each equipment vendor.

The bridge between OPC-UA and AWS is typically an edge gateway — an industrial PC or ruggedised server sitting in the control cabinet network. AWS IoT Greengrass runs on this gateway and hosts a custom component that acts as an OPC-UA client: it connects to OPC-UA servers on the plant network (which is air-gapped or firewall-separated from the corporate IT network), reads configured tag subscriptions, and publishes the values to AWS IoT Core via MQTT over TLS.

The OPC-UA subscription model is important here. Rather than polling tags at a fixed interval (which wastes bandwidth and storage on unchanged values), OPC-UA subscriptions publish a value only when it changes beyond a configured deadband. For a temperature sensor with a deadband of 0.5°C, a reading is published only when temperature changes by more than half a degree. This reduces message volume by 60–80% on typical plant equipment compared to fixed-interval polling, directly reducing Kinesis and storage costs.

AWS IoT Core receives these MQTT messages and routes them via IoT Rules to downstream services. A rule matching the topic plant/{site}/{machine}/telemetry routes to Kinesis Data Streams for real-time processing and to Amazon Timestream via a direct IoT Rules integration for time-series persistence.

Kinesis Data Streams for Real-Time Processing

The real-time processing layer sits on Kinesis Data Streams. Its role is to evaluate incoming sensor data against operational rules, detect anomalies, and trigger alerts before problems become failures.

A Lambda consumer on the Kinesis stream evaluates rules defined in DynamoDB: if the spindle motor current on machine CNC-047 exceeds 95% of rated current for more than 30 seconds, send an SNS notification to the maintenance supervisor. If coolant temperature exceeds 42°C on any machine in building 3, send an alert and write an incident record to the maintenance management system via API. These rule-based alerts handle the straightforward cases — explicit threshold violations that the process engineers already know are dangerous.

For more subtle anomaly detection, a Kinesis Data Analytics (Apache Flink) application maintains rolling statistical baselines per machine and flags deviations beyond two standard deviations from the rolling 7-day mean. This catches gradual degradation patterns — a bearing whose vibration RMS is slowly increasing week over week — that do not trigger explicit threshold rules until the failure is imminent.

All records on the Kinesis stream also fan out to Kinesis Data Firehose, which buffers and delivers them to S3 in Parquet format. The S3 data serves as the historical training dataset for SageMaker models and as the analytical source for equipment utilisation and OEE (Overall Equipment Effectiveness) reporting.

Amazon Timestream for Time-Series Storage

IoT sensor data is fundamentally different from transactional or event data in its query patterns. The most common queries are: “What was the vibration on machine X between 14:00 and 16:00 yesterday?” and “Plot coolant temperature over the last 72 hours.” These are time-bounded range scans over a single series, not aggregations across millions of rows. A general-purpose analytical database like Redshift handles these queries, but inefficiently — it reads far more data than necessary and charges you for it.

Amazon Timestream is purpose-built for this pattern. Its storage architecture separates recent data (memory store, sub-millisecond query latency) from historical data (magnetic store, cost-efficient for long retention). For manufacturing IoT, we configure the memory store to retain 7 days of data (covering the typical troubleshooting window after an incident) and the magnetic store to retain 13 months (covering annual performance comparisons for OEE benchmarking).

Timestream’s query language is SQL with time-series extensions: bin() for time-bucketing, interpolate_linear() for filling gaps caused by connectivity interruptions, and MEASURE_VALUE for accessing the sensor reading value. A query for 1-minute average spindle load on a specific machine over the last 24 hours is a single, readable SQL statement that returns in under 100 milliseconds against a dataset with billions of records.

Timestream’s native integration with Amazon QuickSight allows operations teams to build equipment dashboards directly against Timestream data without an intermediate data warehouse layer. A maintenance engineer can open a dashboard showing vibration trends for all machines in their area with sub-second refresh, using data that is at most a few seconds old.

Predictive Maintenance with SageMaker

The highest-value use case for manufacturing IoT data is predictive maintenance: using historical sensor data and failure records to predict which equipment is likely to fail before it does, allowing maintenance to be scheduled proactively rather than reactively.

The training dataset for a predictive maintenance model combines sensor history from Timestream (exported to S3 via Timestream’s scheduled query export feature) with failure event records from the CMMS (Computerised Maintenance Management System). Each training example is a time window of sensor readings labelled with whether a failure occurred within N hours after the window end.

Amazon SageMaker processes this dataset using a combination of feature engineering (statistical features of the time-series window: mean, standard deviation, maximum, rate of change) and model training. For manufacturing datasets with tens of millions of training windows, SageMaker’s managed training infrastructure distributes the job across multiple GPU instances, reducing training time from hours to minutes.

The deployed model runs as a SageMaker endpoint. The Kinesis Data Analytics Flink application extracts a feature vector from each machine’s real-time sensor window every 15 minutes and invokes the SageMaker endpoint for inference. Machines with a predicted failure probability above the configured threshold are flagged in the maintenance management system with a recommended inspection date.

A mid-size Ontario auto parts manufacturer implementing this pattern reduced unplanned downtime by 34% in the first year, with a payback period of under 8 months on the platform build investment.

For related MLOps patterns, see MLOps Meets Data Engineering.

Operational Analytics and OEE Reporting

Beyond maintenance, the IoT data platform enables operational analytics that manufacturing operations teams have historically had to approximate from manual data collection. OEE — the product of Availability, Performance, and Quality — is the standard manufacturing efficiency metric, but calculating it accurately requires integrating production count data, downtime records, and cycle time data that live in different systems.

In the AWS architecture, an AWS Glue ETL job runs nightly to join Kinesis-delivered production counter data (units produced per shift per machine) from S3, planned production schedules from the MES (Manufacturing Execution System) via API, and downtime records from the CMMS. The result is an OEE fact table in Redshift, queryable by machine, line, shift, product, and date. A QuickSight dashboard makes this available to plant managers without requiring them to understand the underlying data model.

See our post on operational analytics on AWS for the broader patterns that underpin this reporting layer.

Conclusion

Manufacturing IoT data pipelines require a careful integration of OPC-UA protocol handling at the edge, real-time stream processing for alerting, time-series optimised storage for sensor history, and machine learning for predictive maintenance — none of which can be adequately handled by a single AWS service or a generic data platform template.

Infra IT Consulting has worked with manufacturers in automotive, food processing, and discrete manufacturing to design and implement IoT data pipelines that deliver real operational outcomes. If your plant is generating sensor data that is not yet working for you, contact us to discuss a platform design tailored to your equipment portfolio and maintenance objectives.