Building Real-Time Dashboards with Kinesis and QuickSight

Batch reporting used to be acceptable. You ran your ETL overnight, refreshed the dashboard in the morning, and stakeholders made decisions with yesterday’s data. In most industries, that is no longer sufficient. Operations teams need to see order volumes by the minute. Security teams need to detect anomalies as they happen. Marketing teams want to watch campaign performance update in real time during a product launch. The demand for near-real-time dashboards has moved from “nice to have” to a core expectation.

AWS offers a coherent stack for building real-time analytics pipelines: Amazon Kinesis for streaming ingestion and processing, Amazon S3 for durable storage, and Amazon QuickSight for visualisation. This post walks through the architecture, the implementation, and the practical considerations that determine whether your real-time dashboard actually behaves in real time.

Understanding the End-to-End Latency Budget

Before building, it helps to be clear about what “real-time” means in the context of dashboards. True real-time (sub-second updates) requires a different architecture than near-real-time (1-5 minute latency). Most business dashboards fall into the near-real-time category, where QuickSight’s SPICE refresh mechanism is adequate. For sub-second requirements, you need a different approach — typically a WebSocket-driven frontend backed by Amazon ElastiCache or DynamoDB Streams.

This post focuses on the near-real-time pattern (1-5 minute end-to-end latency), which covers the majority of operational analytics use cases.

The latency budget across the pipeline:

Stage	Typical Latency
Event → Kinesis Data Streams	< 1 second
Kinesis Data Streams → Firehose buffering	60-300 seconds (configurable)
Firehose → S3	< 1 second after buffer flush
S3 → Athena query / Redshift COPY	10-30 seconds
QuickSight SPICE refresh	Configurable, minimum 15 min on Standard

For dashboards needing updates faster than 15 minutes, you use QuickSight in direct query mode against Athena or Redshift, skipping SPICE entirely.

Architecture: The Streaming Pipeline

The reference architecture for a real-time dashboard pipeline on AWS:

[Event Producers]           (web apps, IoT devices, microservices)
       │
       ▼
[Kinesis Data Streams]      (durable, ordered, replayable event buffer)
       │
       ├──► [Lambda / KDA]  (real-time enrichment, filtering, aggregation)
       │
       ▼
[Kinesis Data Firehose]     (buffering, format conversion, S3 delivery)
       │
       ▼
[Amazon S3]                 (raw events in Parquet, partitioned by time)
       │
       ▼
[AWS Glue Data Catalog]     (schema discovery, partition registration)
       │
       ▼
[Amazon Athena]             (serverless SQL over S3 events)
       │
       ▼
[Amazon QuickSight]         (direct query dashboard, auto-refresh)

Setting Up Kinesis Data Streams

Start with a Kinesis Data Stream sized for your event throughput. Each shard provides 1 MB/s write capacity and 2 MB/s read capacity. For most operational dashboards, a 2-4 shard stream is sufficient to begin.

import boto3
import json
import time

kinesis = boto3.client('kinesis', region_name='ca-central-1')

def publish_event(stream_name: str, event: dict, partition_key: str):
    """Publish a single event to Kinesis Data Streams."""
    kinesis.put_record(
        StreamName=stream_name,
        Data=json.dumps(event).encode('utf-8'),
        PartitionKey=partition_key
    )

# Example: publish an order event
publish_event(
    stream_name='prod-order-events',
    event={
        'event_type': 'order_placed',
        'order_id': 'ord-20240309-00142',
        'customer_id': 'cust-8821',
        'amount_cad': 149.99,
        'region': 'ON',
        'timestamp': int(time.time() * 1000)
    },
    partition_key='cust-8821'
)

Using customer_id as the partition key ensures all events for a given customer land on the same shard, preserving order for per-customer aggregations downstream.

Configuring Kinesis Data Firehose

Kinesis Data Firehose reads from the Data Stream and delivers data to S3 in configurable batches. The key settings are the buffer interval (60-900 seconds) and buffer size (1-128 MB). Smaller buffer intervals mean lower latency to S3 but more small files, which hurt Athena query performance. A 60-second buffer with dynamic partitioning is a good starting point:

{
  "DeliveryStreamName": "order-events-to-s3",
  "DeliveryStreamType": "KinesisStreamAsSource",
  "KinesisStreamSourceConfiguration": {
    "KinesisStreamARN": "arn:aws:kinesis:ca-central-1:123456789:stream/prod-order-events",
    "RoleARN": "arn:aws:iam::123456789:role/firehose-delivery-role"
  },
  "ExtendedS3DestinationConfiguration": {
    "BucketARN": "arn:aws:s3:::analytics-events-bucket",
    "Prefix": "order-events/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/",
    "ErrorOutputPrefix": "errors/order-events/",
    "BufferingHints": {
      "SizeInMBs": 64,
      "IntervalInSeconds": 60
    },
    "DataFormatConversionConfiguration": {
      "InputFormatConfiguration": {
        "Deserializer": { "OpenXJsonSerDe": {} }
      },
      "OutputFormatConfiguration": {
        "Serializer": { "ParquetSerDe": {} }
      }
    }
  }
}

The Prefix pattern creates a Hive-compatible partition structure in S3 that Athena can query efficiently. Converting from JSON to Parquet at the Firehose stage reduces storage costs by 60-80% and dramatically improves Athena scan performance.

Registering Partitions with AWS Glue

For Athena to query new partitions as they arrive, the Glue Data Catalog must know about them. Two approaches:

Option 1: MSCK REPAIR TABLE — runs a full partition discovery scan. Works but is slow and expensive for large tables. Not suitable for real-time use.

Option 2: AWS Glue Crawlers on a schedule — run every 5-15 minutes to discover new partitions. Simple to configure, adds 5-15 minutes to end-to-end latency.

Option 3: Lambda-triggered partition registration — an S3 event triggers a Lambda function that calls batch_create_partition on the Glue catalog. This reduces partition registration latency to under 10 seconds:

import boto3

glue = boto3.client('glue', region_name='ca-central-1')

def register_partition(database: str, table: str, partition_values: list, s3_location: str):
    glue.batch_create_partition(
        DatabaseName=database,
        TableName=table,
        PartitionInputList=[{
            'Values': partition_values,
            'StorageDescriptor': {
                'Location': s3_location,
                'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
                'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',
                'SerdeInfo': {
                    'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
                }
            }
        }]
    )

For real-time dashboards, Option 3 is the right choice. It adds minimal complexity and keeps end-to-end latency tight.

Building the QuickSight Dashboard in Direct Query Mode

With Athena as the data source, QuickSight can query the latest data on every dashboard load without SPICE. Configure the Athena data source in QuickSight and create datasets that use direct SQL:

SELECT
    DATE_TRUNC('minute', FROM_UNIXTIME(timestamp / 1000)) AS event_minute,
    region,
    COUNT(*) AS order_count,
    SUM(amount_cad) AS revenue_cad,
    AVG(amount_cad) AS avg_order_value
FROM "analytics_db"."order_events"
WHERE year = '2024'
  AND month = '03'
  AND day = '09'
  AND hour >= LPAD(CAST(HOUR(NOW() AT TIME ZONE 'America/Toronto') - 2 AS VARCHAR), 2, '0')
GROUP BY 1, 2
ORDER BY 1 DESC

The date filter using partition columns is critical — without it, Athena scans the entire table on every dashboard refresh. Always filter on the partition columns explicitly.

In QuickSight, set the dataset to Direct Query mode and configure auto-refresh. QuickSight will re-execute the Athena query on each page load, delivering data that is at most 60-90 seconds old (the Firehose buffer interval plus Athena query time).

Cost Optimisation

Real-time pipelines can incur unexpected costs if not tuned:

• Kinesis shards — each shard costs approximately $0.015/hour. Right-size your stream and use Enhanced Fan-Out only when you have multiple consumers that need independent read throughput.
• Athena scans — every QuickSight page load triggers an Athena query. Use Parquet format, partition pruning, and columnar projection to minimise data scanned. Consider a materialized view in Athena or a pre-aggregation Lambda to reduce per-query cost.
• QuickSight sessions — direct query mode triggers Athena queries on every session. For high-traffic dashboards with many concurrent users, SPICE with a 15-minute refresh schedule may be cheaper.

Conclusion

A well-architected Kinesis + QuickSight pipeline delivers near-real-time operational dashboards with end-to-end latency under two minutes — sufficient for the vast majority of business use cases. The key design decisions are Firehose buffer sizing, partition registration strategy, and the choice between direct query and SPICE, all of which are driven by your latency requirements and cost constraints.

Infra IT Consulting designs and implements real-time data pipelines for clients across Canada, the UK, and Africa. Whether you are starting from scratch or improving an existing streaming architecture, we can help. Contact us to discuss your requirements.

Related reading:

• Building a Data Lake on S3
• Amazon QuickSight Guide: Features, Pricing & Best Practices
• Athena SQL Best Practices