This repository was archived by the owner on Feb 26, 2026. It is now read-only.
Open
Conversation
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
felixgateru
suggested changes
Jul 30, 2025
Contributor
felixgateru
left a comment
felixgateru
left a comment
There was a problem hiding this comment.
Could you also please add screenshots of how reports look like in the ui?
felixgateru
suggested changes
Jul 30, 2025
felixgateru
suggested changes
Jul 30, 2025
blog/reports/reports.md
Outdated
|
|
||
| If you've been working with IoT systems for a while, you know that collecting sensor data is just the beginning. The real challenge is turning that flood of telemetry into actionable insights that actually help your team make decisions. That's where Magistrala's reporting system comes in. | ||
|
|
||
| After spending months working with this feature, I wanted to share what I've learned about building robust IoT reports. Magistrala's reporting service has become one of our go-to tools for everything from daily operational summaries to executive dashboards. |
Contributor
There was a problem hiding this comment.
Does the service offer dashboards?
|
|
||
| After spending months working with this feature, I wanted to share what I've learned about building robust IoT reports. Magistrala's reporting service has become one of our go-to tools for everything from daily operational summaries to executive dashboards. | ||
|
|
||
| <!-- truncate --> |
blog/reports/reports.md
Outdated
|
|
||
| ## System Architecture | ||
|
|
||
| The reporting system isn't just bolted on as an afterthought - it's built into Magistrala's core architecture. Here's what you're working with: |
Contributor
There was a problem hiding this comment.
Could you use formal language instead?
| Your IoT Devices → Message Brokers → Readers Service → Reports Service → Reports | ||
| ``` | ||
|
|
||
| What's nice is that the reporting service doesn't reinvent the wheel. It hooks into the existing readers infrastructure, so you're querying the same data store your other services use. |
Contributor
There was a problem hiding this comment.
I am assuming a reader may not know what the readers service in magistrala is, you can give briefcontext into that
blog/reports/reports.md
Outdated
| ```json | ||
| { | ||
| "channel_id": "{{CHANNELID}}", | ||
| "client_ids": ["{{THINGID}}"], |
Contributor
There was a problem hiding this comment.
Suggested change
| "client_ids": ["{{THINGID}}"], | |
| "client_ids": ["{{CLIENTID}}"], |
wambui-pixel
suggested changes
Jul 30, 2025
blog/reports/reports.md
Outdated
|
|
||
| When you create custom HTML templates, Magistrala validates them to ensure they'll work correctly with the PDF generation system. Your template must include: | ||
|
|
||
| **Required Template Fields:** |
Contributor
There was a problem hiding this comment.
Suggested change
| **Required Template Fields:** | |
| **Required Template Fields:** | |
| - `{{$.Title}}` - Report title | |
| - `{{$.GeneratedDate}}` and `{{$.GeneratedTime}}` - Generation timestamps | |
| - `{{.Metric.Name}}`, `{{.Metric.ClientID}}`, `{{.Metric.ChannelID}}` - Metric information | |
| - `{{len .Messages}}` - Record count | |
| - `{{range .Messages}}...{{end}}` - Data iteration block | |
| - `{{formatTime .Time}}`, `{{formatValue .}}`, `{{.Unit}}`, `{{.Protocol}}`, `{{.Subtopic}}` - Message fields | |
| **Required HTML Structure:** | |
| - Valid HTML5 document with `<!DOCTYPE html>` | |
| - Complete `<html>`, `<head>`, `<body>`, and `<style>` tags | |
| - Proper table structure with `<table>`, `<thead>`, `<tbody>`, `<th>`, `<td>` | |
| - Expected table headers: Time, Value, Unit, Protocol, Subtopic | |
| **Required CSS Classes:** | |
| - `.page` - Main page container | |
| - `.header` - Report header section | |
| - `.content-area` - Main content area | |
| - `.metrics-section` - Metrics information display | |
| - `.data-table` - Data table styling | |
| - `.footer` - Report footer |
JeffMboya
suggested changes
Aug 3, 2025
blog/reports/reports.md
Outdated
| slug: iot-data-reports | ||
| title: "Building IoT Data Reports with Magistrala: A Developer's Deep Dive" | ||
| authors: steve | ||
| description: "A practical guide to building robust IoT reports with Magistrala's reporting system - from real-time data aggregation to automated PDF generation and email delivery." |
Contributor
There was a problem hiding this comment.
Suggested change
| description: "A practical guide to building robust IoT reports with Magistrala's reporting system - from real-time data aggregation to automated PDF generation and email delivery." | |
| description: "A practical guide to generating IoT reports using Magistrala" |
blog/reports/reports.md
Outdated
|
|
||
| ## Introduction | ||
|
|
||
| If you've been working with IoT systems for a while, you know that collecting sensor data is just the beginning. The real challenge is turning that flood of telemetry into actionable insights that actually help your team make decisions. That's where Magistrala's reporting system comes in. |
Contributor
There was a problem hiding this comment.
Suggested change
| If you've been working with IoT systems for a while, you know that collecting sensor data is just the beginning. The real challenge is turning that flood of telemetry into actionable insights that actually help your team make decisions. That's where Magistrala's reporting system comes in. | |
| If you’ve worked with IoT systems for any length of time, you know that collecting sensor data is only the beginning. The real work starts when you try to make sense of all the incoming data and turn it into information your team can use to monitor systems, track performance, or respond to issues. That’s where Magistrala’s reporting system comes in. |
blog/reports/reports.md
Outdated
|
|
||
| If you've been working with IoT systems for a while, you know that collecting sensor data is just the beginning. The real challenge is turning that flood of telemetry into actionable insights that actually help your team make decisions. That's where Magistrala's reporting system comes in. | ||
|
|
||
| After spending months working with this feature, I wanted to share what I've learned about building robust IoT reports. Magistrala's reporting service has become one of our go-to tools for everything from daily operational summaries to executive dashboards. |
Contributor
There was a problem hiding this comment.
Could you provide concrete examples of what you have been able to do or can do with Magistrala reports? Just saying `After spending months working with this feature' sounds generic
JeffMboya
suggested changes
Aug 3, 2025
blog/reports/reports.md
Outdated
|
|
||
| ## Introduction | ||
|
|
||
| If you've been working with IoT systems for a while, you know that collecting sensor data is just the beginning. The real challenge is turning that flood of telemetry into actionable insights that actually help your team make decisions. That's where Magistrala's reporting system comes in. |
Contributor
There was a problem hiding this comment.
Also, could we have the introduction be more interesting and intriguing? soemthing like
Introduction
Every IoT deployment eventually hits the same problem: it's not the lack of data - it’s the lack of answers. The devices are connected. The brokers are stable. The telemetry flows in, line after line, topic after topic, timestamp after timestamp. And then someone asks a simple question:
“Can I get a daily summary of temperature anomalies for Warehouse 3?”
“Can we send a weekly uptime report to the client?”
“Can we prove we stayed within power limits last month?”
That’s when the real work begins.
Over the past 3 months, I’ve been responsible for turning raw data streams into reports that our teams actually use - across ops, compliance, and even finance. What started as a one-off export script has turned into a fully automated, multi-tenant, multi-format reporting system powered by Magistrala.
We’ve now got scheduled daily reports that summarize temperature data from thousands of MQTT-connected sensors. We’ve got uptime reports pushed to Google Drive. And we’ve got branded PDFs sent to client inboxes without anyone needing to log into a dashboard. And to make it even better, they aren’t just dashboards or exports: they are structured, repeatable, timestamped reports with full auditability - and they’re built into the platform.This post walks through how that system works—how it’s architected, how reports are modeled, how templates are validated and rendered, and how we’ve used it in production to cut hours of manual work each week.
blog/reports/reports.md
Outdated
|
|
||
| <!-- truncate --> | ||
|
|
||
| ## System Architecture |
Contributor
There was a problem hiding this comment.
It'll be good to understand the significance of reporting before jumping into the architecture.Something like
Why Reporting Matters in IoT
In an IoT system, data is constant - but understanding is not.
Data comes in the form of sensor readings, status updates, error logs, event messages. It's granular, fast, and often ephemeral. But what most stakeholders want is not the data itself - it’s the summary of what happened over a defined period: the daily operational status, the weekly uptime statistics, the monthly energy use, the quarterly SLA compliance.
That’s what reporting delivers.
And it’s especially critical in IoT for three reasons:
Time makes context
Unlike traditional systems, IoT data is driven by time. A temperature reading means little on its own - but averaged hourly across 1,000 sensors, it reveals patterns. Spikes may indicate mechanical faults, and drops might signal failed sensors. Reporting aggregates time into meaning.
Human consumption is asynchronous
Dashboards are great for real-time ops. But most teams - finance, compliance, client managers -don’t log into dashboards. They rely on emailed summaries, PDF snapshots, and scheduled CSV exports to get the story. Reporting bridges this operational gap.
Accountability requires auditability
IoT platforms often sit between infrastructure and operations. When something breaks, the ability to look back and say “here’s exactly what we saw on Tuesday at 2pm” matters. Reports provide that frozen-in-time view that dashboards can’t.
For these reasons, we’ve come to treat reporting as a first-class function. It’s not a visualization tool. It’s a data product- automated, schedulable, auditable, and deliverable. In many platforms, reporting is treated as a feature added on after everything else. But that doesn’t scale. Because reporting, at its core, isn’t about charts - it’s about historical state.
And historical state has rules:
It must be queryable across time, device, and location.
It must be filterable down to the fields users care about - units, regions, thresholds.
It must be structured into something that can be rendered, exported, or sent without human intervention.
And most importantly - it must be auditable and repeatable.
That’s why Magistrala’s reporting system is tightly integrated with the core architecture. It doesn’t duplicate the message store. It doesn’t fork your telemetry pipelines. Instead, it acts as a query layer, a renderer, and a delivery engine on top of the data you already ingest and store.
And it does so through a well-defined set of primitives: reports, templates, schedules, formats, and metrics.
blog/reports/reports.md
Outdated
|
|
||
| <!-- truncate --> | ||
|
|
||
| ## System Architecture |
Contributor
There was a problem hiding this comment.
Suggested change
| ## System Architecture | |
| ## System Architecture: How Magistrala Reporting Works Under the Hood |
Make it specific to Magistrala
blog/reports/reports.md
Outdated
Comment on lines
+104
to
+113
| ### The Step-by-Step Process | ||
|
|
||
| Here's what happens when you generate a report: | ||
|
|
||
| 1. **Configuration Phase**: You create a report config with metrics, time ranges, and output preferences | ||
| 2. **Data Query**: Reports service calls the Readers service with your parameters | ||
| 3. **Data Retrieval**: Readers service queries the message store (PostgreSQL/TimescaleDB) | ||
| 4. **Processing**: Raw sensor data gets aggregated according to your config (hourly averages, daily sums, etc.) | ||
| 5. **Template Rendering**: If you want PDF/CSV, the template engine processes your data through HTML templates, then sends it to Gotenberg for PDF conversion | ||
| 6. **Delivery**: Depending on your action (view/download/email), the report gets returned or sent |
Contributor
There was a problem hiding this comment.
Suggested change
| ### The Step-by-Step Process | |
| Here's what happens when you generate a report: | |
| 1. **Configuration Phase**: You create a report config with metrics, time ranges, and output preferences | |
| 2. **Data Query**: Reports service calls the Readers service with your parameters | |
| 3. **Data Retrieval**: Readers service queries the message store (PostgreSQL/TimescaleDB) | |
| 4. **Processing**: Raw sensor data gets aggregated according to your config (hourly averages, daily sums, etc.) | |
| 5. **Template Rendering**: If you want PDF/CSV, the template engine processes your data through HTML templates, then sends it to Gotenberg for PDF conversion | |
| 6. **Delivery**: Depending on your action (view/download/email), the report gets returned or sent | |
| ## Following a Report: The Full Lifecycle | |
| To really understand how Magistrala’s reporting system works, it helps to follow a single report from start to finish. Not the code or the API docs - just the actual flow: from when someone creates a report config to when a PDF lands in their inbox. | |
| Here’s how that journey works. | |
| --- | |
| ### Step 1: Someone creates a report | |
| Every report starts with a config. | |
| This can be set up through the API or UI, and it defines what the report should do: | |
| * What time range to cover (e.g. last 24 hours) | |
| * What metric to pull (e.g. temperature, power usage) | |
| * Which devices or channels to include | |
| * How to aggregate the data (e.g. hourly averages) | |
| * What format to use (PDF, CSV, JSON) | |
| * Whether to send it over email, push it somewhere else, or just download it | |
| Once created, this config gets saved in Postgres. It’s not a one-time report—it’s a recipe the system can keep reusing. If you add a schedule to it (say, every day at 6am), the scheduler picks it up automatically. | |
| --- | |
| ### Step 2: The scheduler picks it up | |
| The scheduler runs in the background every minute. It checks the database to see if any reports are due to run based on their schedule. | |
| If it finds one, it locks that report (so it doesn’t get triggered twice), figures out the time range to use (like "now" minus 24 hours), and then hands the job off to the report service to actually generate it. | |
| --- | |
| ### Step 3: The data is pulled | |
| Next, the report service talks to the readers service over gRPC to fetch the actual sensor data. This is where it gets the readings—say, temperature from a set of devices over the last day. | |
| The readers service builds a SQL query under the hood (usually against TimescaleDB), using the filters and metric fields from the report config. If the report includes aggregation (like hourly averages), that happens here too. | |
| The result is a clean set of data: timestamps, values, units, and other metadata—ready to be turned into something people can read. | |
| --- | |
| ### Step 4: The report is rendered | |
| Once the data is in, the report service decides how to format it. | |
| * If it’s a CSV, it just writes out the rows. | |
| * If it’s a PDF, it needs to go through the template engine. | |
| The template engine takes your HTML layout (custom or default), drops in the data, and then passes the result to Gotenberg—an external service that turns HTML into a polished PDF. The result comes back as a finished file, ready to be sent. | |
| --- | |
| ### Step 5: It gets delivered | |
| Now the report is ready to go. | |
| Depending on how it was configured, it might: | |
| * Be sent as an email attachment | |
| * Be uploaded to object storage | |
| * Be posted to a webhook | |
| * Or just returned to the API caller for immediate download | |
| If email is involved, the system takes care of SMTP delivery, including things like retries if the mail server is slow or the address bounces. | |
blog/reports/reports.md
Outdated
|
|
||
| ### Understanding Metrics: The Core Filtering Mechanism | ||
|
|
||
| In the Magistrala reports system, **metrics** are the primary way to filter and retrieve data from the TimescaleDB database. Each metric in your report configuration corresponds to specific filtering parameters that the TimescaleDB readers use to query sensor data. |
Contributor
There was a problem hiding this comment.
Suggested change
| In the Magistrala reports system, **metrics** are the primary way to filter and retrieve data from the TimescaleDB database. Each metric in your report configuration corresponds to specific filtering parameters that the TimescaleDB readers use to query sensor data. | |
| When you're setting up a report in Magistrala, the most important thing you define - besides the time range - is the list of metrics. These tell the system what data to pull from TimescaleDB. But if you've never dug into how they work under the hood, it’s easy to miss just how much control they give you - and how things can quietly break if you don’t model them carefully. |
Signed-off-by: nyagamunene <stevenyaga2014@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request title should be
MG-XXX - descriptionorNOISSUE - descriptionwhere XXX is ID of issue that this PR relate to.Please review the CONTRIBUTING.md file for detailed contributing guidelines.
What does this do?
Which issue(s) does this PR fix/relate to?
Resolves # https://github.com/absmach/amdm/issues/290
List any changes that modify/break current functionality
Have you included tests for your changes?
Did you document any new/modified functionality?
Notes