---
# Re-architecting Data Pipelines
**URL:** https://www.sigmoid.com/case-studies/rearchitecting-data-pipelines/
Date: 2021-11-29
Author: Sigmoid
Post Type: page
Summary: 10X Improvement in Dashboard Query Performance By Re-architecting Data Pipelines Streamlining ETL pipelines to improve query performance and ensuring better forecasting and...Read More...
Featured Image: https://www.sigmoid.com/wp-content/uploads/2023/02/rearchitecting-data-pipelines-cs-thumbnail-opt.jpg
---
# 10X Improvement in Dashboard Query Performance By Re-architecting Data Pipelines
Streamlining ETL pipelines to improve query performance and ensuring better forecasting and predictive maintenance of machines.
[fluentform id="65"]
[lc_get_post post_type="lc_block" slug="case-studies-privacy-policy-statement"]
Streamlining ETL pipelines to improve query performance and ensuring better forecasting and predictive maintenance of machines
### Business Challenges
The customer is a leading manufacturer of industrial tools, household hardware, and security systems with thousands of machines installed across plants that automate processes, including welding and bolting. These machines produced IoT data such as current utilized, voltage fluctuations from the sensors installed, and collected on a centralized IoT server. The customer monitored this data on a central dashboard built in-house, however, the dashboard was slow and not scalable, resulting in delays in generating insights and predicting machine failures.
The customer wanted to enable:
- Central monitoring and diagnosis of machines across manufacturing factories
- A solution to better forecast the probability of downtime and malfunctioning of these machines in a cost-efficient way. It would save them millions of dollars in preventing product recall due to defective manufacturing
- Near real-time updates of dashboards
### Sigmoid Solution
We addressed the customer’s needs by building an ETL data pipeline and developed Spark-based data lake and high-speed dashboard to enable faster query processing time. Leveraging open-source technologies, we integrated and scaled the prediction and forecasting layers into the new solution while automating the deployment.
A centralized IoT MQTT Broker collected IoT data such as telemetry and faults from sensors across factories. The raw data was enriched, processed, and harmonized before writing it to the S3 data lake, which was used for monitoring and analysis. These pipelines employed machine learning scripts to predict and forecast events. We used NitroDb — Sigmoid’s proprietary platform and query accelerator — to create ML models for predictions.
[Download case study](#ip-cs-Form)
### Business Impact
We built an efficient and near real-time ETL data pipeline for the customer that resulted in 10X improvement in query processing time and increased scalability. We facilitated centralized monitoring of dashboards and reduced time to reflect data on dashboards from 3 days to less than an hour. The solution also ensured 2X reduction in time to market new monitoring features while keeping the cost of deployment minimal.
## Relevant Case Studies
#### [80% performance improvements in data pipelines](/case-studies/data-pipelines-performance-improvement/)
[Read case study](/case-studies/data-pipelines-performance-improvement/)
#### [Building robust data pipelines for a leading investment bank to make quality datasets ready for ML use cases](/case-studies/building-robust-data-pipelines/)
[Read case study](/case-studies/building-robust-data-pipelines/)
#### [24×7 monitoring and support using highly available, robust systems](/case-studies/improved-data-pipeline-availability/)
[Read case study](/case-studies/improved-data-pipeline-availability/)
#### [Data Analytics For CPG in the COVID Era](https://www.sigmoid.com/blogs/role-of-data-analytics-in-cpg-companies-in-the-covid-era/)
[Read blog](https://www.sigmoid.com/blogs/role-of-data-analytics-in-cpg-companies-in-the-covid-era/)
#### [Global COVID-19 Scenario: A Data Scientist’s Perspective (Part 1)](https://www.sigmoid.com/blogs/global-covid-19-scenario-a-data-scientists-perspective-part-1/)
[Read blog](https://www.sigmoid.com/blogs/global-covid-19-scenario-a-data-scientists-perspective-part-1/)
[lc_get_post post_type="lc_section" slug="find-out-footer-common-cta"]
---
## Navigation
- [Company](/about-sigmoid)
- [Newsroom](/newsroom)
- [Life at Sigmoid](/careers)
- [Takshashila](/takshashila)
- [Contact Us](/contact-us)
- [AI Strategy Blueprint your AI advantage](/enterprise-ai-strategy/)
- [Generative AI Drive innovation with Generative AI](/generative-ai/)
- [Responsible AI Build trust with ethical AI practices](/responsible-ai-in-enterprise/)
- [Agentic AI Reshape business with scalable agentic systems](/agentic-ai-solutions/)
- [AI Managed Services Ensure reliable AI performance](/ai-managed-services/)
- [Advanced Analytics Transform your business with data-driven insights](/advanced-data-analytics-solutions/)
- [Start Assessment](/agentic-ai-readiness-index/)
- [Data Strategy Strong data foundations for scalable AI](/data-analytics-strategy/)
- [Data Management Leverage data as a strategic asset](/ai-data-management-services/)
- [Data Ops Automate data for speed and quality](/data-devops/)
- [Data Engineering Deliver insights faster with scalable pipelines](/data-engineering/)
- [Cloud Transformation Modernize data to maximise efficiency](/cloud-migration/)
- [Download Whitepaper](/ebooks-whitepapers/building-data-products-in-a-data-mesh-to-drive-business-value/)
- [Data Modeling Structure data for better decisions](/data-modeling-services/)
- [Data Visualization Transform data into actionable stories](/data-visualization-service/)
- [BI Migration Enhance decision making with modern BI tools](/bi-migration/)
- [Data Observability Build trust with healthy, accurate data](/data-observability/)
- [Automated Insights Make smarter decisions with auto-generated insights](/automated-insights/)
- [Download Whitepaper](/ebooks-whitepapers/power-bi-hacks/)
- [CPG & Retail End-to-end analytics for planning, operations, and commercial excellence](/industries/cpg-analytics/)
- [Life Sciences Trusted intelligence across clinical, commercial, and operational workflows](/industries/life-sciences/)
- [Financial Services AI-powered analytics for risk, compliance and customer experience](/industries/banking-financial-analytics-services/)
- [Read case study](/case-studies/data-clean-room-enables-real-time-insights-to-improve-operational-efficiency/)
- [MediaIQ Advanced platform for in-flight marketing measurement](/accelerators/sigmoid-mediaiq-multi-touch-attribution-tool/)
- [CampaignIQ AI-driven platform for optimized campaign budget allocation](/accelerators/sigmoid-campaigniq/)
- [AssistBot GenAI email assistant that automates human-like responses](/accelerators/sigmoid-assistbot-for-ai-email-assistant/)
- [CreativeBot GenAI tool for personalized and brand-aligned creative design](/accelerators/sigmoid-creativebot/)
- [SocialBot GenAI platform to analyze digital conversations and trends](/accelerators/#marketing|socialbot)
- [DemandIQ Predict trends accurately and optimize inventory management](/accelerators/sigmoid-demandiq/)
- [NetworkIQ Track and optimize logistics operations in real-time to quickly address disruptions](/accelerators/sigmoid-networkiq/)
- [SupplyIQ End-to-end platform to optimize supply chain operations](/accelerators/sigmoid-supplyiq/)
- [ProcurementIQ Automated procurement operations for maximum savings, compliance and efficiency](/accelerators/sigmoid-procurementiq/)
- [RapidML Accelerated deployment for machine learning models](/accelerators/sigmoid-rapidml/)
- [DataGuard Comprehensive platform for proactive data quality management](/accelerators/data-quality-tool-sigmoid-dataguard/)
- [CloudPulse Cloud cost optimization platform with multi-cloud management](/accelerators/sigmoid-cloudpulse/)
- [RAPID GenAI foundation with built-in governance and cost clarity](/accelerators/sigmoid-rapid/)
- [AnalyticsBot GenAI based platform to streamline decision-making in analytics](/accelerators/sigmoid-analyticsbot/)
- [DataConnect Seamlessly ingest, integrate and harmonize data from diverse sources](/accelerators/sigmoid-dataconnect/)
- [Reconica AI-powered data harmonization and reconciliation engine](/accelerators/sigmoid-reconica/)
- [ConverseBot GenAI driven insights generation for automated insights from reports](/accelerators/#sales|conversebot)
- [iNRM Cross-lever revenue growth optimization platform](/accelerators/sigmoid-inrm/)
- [AssortmentIQ Optimize shelf layouts and assortment mix at scale with AI-based insights](/accelerators/sigmoid-assortmentiq/)
- [Read Whitepaper](/ebooks-whitepapers/building-agentic-ai-chatbots-for-business-process-transformation/)
- [Listen Podcast](/events/podcast/how-jack-in-the-box-is-redefining-personalization-and-supply-chain-with-ai/)
- [Blogs](/blogs/)
- [White Papers](/ebooks-whitepapers/)
- [Case Studies](/case-studies/)
- [Podcast](/events/podcast/#Podcasts)
- [Read Blog](/blogs/the-genai-adoption-triad-responsibility-ethics-and-explainability/)
- [ConverseBot](/accelerators/#sales|conversebot/)
---
## Footer Links
- [Talk to our AI experts](/contact-us/)
- [AI Strategy](/enterprise-ai-strategy/)
- [Agentic AI](/agentic-ai-solutions/)
- [Generative AI](/generative-ai/)
- [AI Managed Services](/ai-managed-services/)
- [Responsible AI](/responsible-ai-in-enterprise/)
- [Advanced Analytics](/advanced-data-analytics-solutions/)
- [Data Strategy](/data-analytics-strategy//)
- [Data Engineering](/data-engineering/)
- [Data Management](/ai-data-management-services/)
- [Cloud Transformation](/cloud-transformation/)
- [Data Ops](/data-devops/)
- [Data Visualization](/data-visualization-service/)
- [Automated Insights](/automated-insights/)
- [BI Migration](/bi-migration/)
- [Data Modeling](/data-modeling-services/)
- [Data Observability](/data-observability/)
- [CPG & Retail](/industries/cpg-analytics/)
- [Financial Services](/industries/banking-financial-analytics-services/)
- [Life Sciences](/industries/life-sciences/)
- [Case Studies](/case-studies/)
- [Thought Leadership](/ebooks-whitepapers/)
- [Blogs](/blogs/)
- [Company](/about-sigmoid/)
- [Newsroom](/newsroom/)
- [Accelerators](/accelerators/)
- [Careers](/careers/)
- [Privacy Policy |](/privacy-policy/)
- [Cookie Policy](/cookie-policy/)