---
# Automate data ingestion to enable near real-time access to insights
**URL:** https://www.sigmoid.com/blogs/automate-data-ingestion/
Date: 2021-05-20
Author: Sigmoid
Post Type: post
Summary: CPG Companies across the globe are looking to get more insights into the latest sales trends from their retailers, to make informed...Read More...
Categories: Data Management
Tags: AI/ML, Cloud Transformation
Featured Image: https://www.sigmoid.com/wp-content/uploads/2023/12/Automate-Data-Ingestion-to-Enable-Near-Real-time-Access-to-Insights-banner-opt.jpg
---
CPG Companies across the globe are looking to get more insights into the latest sales trends from their retailers, to make informed decisions across the supply chain and marketing. However, the lack of availability of granular data from retailers is a huge challenge to derive meaningful insights.
COVID-19 has further fueled the need to have insights from retailer data to track the tremendous shift in consumer buying behavior, especially the CPG brands. Forbes reports that US CPG sales increased by 10.3% last year. This is 5 times the normal rate of change year on year.
While the growth of [CPG](/industries/cpg-analytics/) companies was unprecedented, they faced challenges such as supply chain disruptions, unprecedented volatility in demand, reduction in supplies, and lack of clarity about future demand. Collecting and effectively analyzing different types of data from retailers including PoS, shopping cart, and other sales data mitigate these challenges by generating fast and granular insights necessary to make timely business decisions. Companies are also looking to create automated databases, [data ingestion pipelines](/etl-and-data-pipeline/) and automated data analysis from the data generated.
In this post, we provide detailed insight into how a leading FMCG firm was able to generate real-time insights leveraging the latest sales data from retailers and e-commerce websites to enable them with better reporting and [forecasting](/case-studies/cpg-demand-forecasting/). It leveraged the power of quality data and near real-time analytics to enable business users across the organization to make timely and informed business decisions on a daily basis.
## The challenge
The client is a leading American FMCG operating in multiple categories including household, and personal care products. Their products are stocked by all leading retailers both online and in brick-and-mortar stores.
As the company was trying to have real-time access to sales trends, they faced the following challenges:
- Lack of [customer-level data](/data-science-services/) from many retailers
- The older data pipelines were monolithic and teams across geographies were downloading the data manually before exporting it to BigQuery to observe the trends
- There were several missing data elements for events like holidays and weekends
- The data was only available to a chosen few members of the organization, making it difficult for the people on the ground to understand how their business was performing
- Most importantly, the report generation process was manual which made the whole process of generating insights slow. It typically took a week’s time to access particular data from the retailers
To address these challenges, the company was looking for ways to generate fast and granular insights, create data ingestion pipeline and automated reports from the retailer’s dashboard, for which the first step was to create a data lake.
There were challenges from retailers’ perspective too. For instance, some retailer dashboards did not provide key metrics such as ‘out of stock’ and ‘lost buy boxes’.
## Creating Data Lake For CPG Company
Sigmoid created a data foundation for the client by building a [Data Lake of the sales and forecast data](/case-studies/data-lake-creation/) from the retail vendors’ dashboards. We built a data lineage and processes to ensure data quality before pushing it to BigQuery, for further analysis.
Since the older pipelines were monolithic requiring manual interventions at various steps for currency conversion and aggregations, Sigmoid performed data harmonization between external sources and internal products and hierarchy. A binary search was implemented to identify data discrepancies rather than performing a full scan.
Additionally, our approach created several generic steps like drop rows/drop columns/remove characters, etc. which can be used in different sources with the minimum development effort. We even automated the download of reports from the vendor’s dashboard and monitored the correct flow of data from the dashboard to BigQuery.
The solution architecture consisted of the following steps:
- Granular ingestion, generic implementation
- Scripts for anomaly detection and fix
- Use internal FX data for USD conversion
- Automation on Google Cloud Platform
- Created specific processes to request and fulfill backfill of data
Owing to the new data foundation, the CPG company now has faster access to data due to the elimination of manual processes. Sigmoid was also able to accomplish tasks such as:
- Currency conversion before sending the data from Amazon dashboard to BigQuery using the Forex data
- Verify a year-old data and reinstate it for the mismatched dates
- Enabled data ingestion at all granularities (day, week, month, year)
- Designing and monitoring of the [ETL Process](/etl-and-data-pipeline/)
## Results
Using Sigmoid’s approach to real-time [data analytics](/blogs/top-data-analytics-trends-to-watch-out-in-2021/), the CPG company can currently download up to 300 different types of datasets every day spread across 14 countries and 22 companies. In a short span of 12 weeks, Sigmoid was able to automate data ingestion for one of America’s largest retailers which has the highest share of total sales in the country.
Sigmoid was able to provide the following benefits to the CPG company:
- Automation of reports across 10+ retailers
- Enabled the client with faster access to data, thereby reducing the time to get actionable insights
- Cut down the time to access data from 7 days to 2 hours, thereby reducing the time to generate actionable insights
- Enhance reporting across retailers and geographies
- Cost-saving of $1.5 million due to reduced dependence on third party reports
- Avoid stockout scenarios and Lost Buy Box scenario
- Accessibility to same quality data to over hundreds of employees which were earlier accessible to only the leadership teams
Download the case study document [here](/case-studies/data-lake-creation/).
## About the author
Srishti is Content Marketing Manager at Sigmoid with a background in tech journalism. She has extensively covered Data Science and AI space in the past and is passionate about technologies defining them.
[lc_the_tags]
## Featured blogs
[lc_get_posts post_type="post"
posts_per_page="4" orderby="date" output_view="lc_get_posts_mycustom_view" output_number_of_columns="4"
output_wrapper_class="row" output_article_class="shadow" output_hide_elements="Excerpt"
output_excerpt_length="0" output_excerpt_text="Read More" output_heading_tag="span"
output_featured_image_format="thumbnail" output_featured_image_class="card-img-left" ]
## Share
[addtoany]
Subscribe to get latest insights
## Talk to our experts
Get the best ROI with Sigmoid’s services in data engineering and AI
## Suggested readings
[View all](/blogs/)
#### [Why Is Data Engineering Critical To CPG Marketing Success](/blogs/data-engineering-cpg-marketing-success/)
[Read blog](/blogs/data-engineering-cpg-marketing-success/)
#### [Online Marketing Spend Optimization for a CPG Brand on a Leading E-commerce Platform](/blogs/campaign-optimization-cpg/)
[Read blog](/blogs/campaign-optimization-cpg/)
#### [Unlocking E-Commerce Growth for CPG with Data and Analytics](/blogs/unlocking-e-commerce-growth-for-cpg-with-data-and-analytics/)
[Read blog](/blogs/unlocking-e-commerce-growth-for-cpg-with-data-and-analytics/)
---
## Categories
- Data Management
---
## Navigation
- [Company](/about-sigmoid)
- [Newsroom](/newsroom)
- [Life at Sigmoid](/careers)
- [Takshashila](/takshashila)
- [Contact Us](/contact-us)
- [AI Strategy Blueprint your AI advantage](/enterprise-ai-strategy/)
- [Generative AI Drive innovation with Generative AI](/generative-ai/)
- [Responsible AI Build trust with ethical AI practices](/responsible-ai-in-enterprise/)
- [Agentic AI Reshape business with scalable agentic systems](/agentic-ai-solutions/)
- [AI Managed Services Ensure reliable AI performance](/ai-managed-services/)
- [Advanced Analytics Transform your business with data-driven insights](/advanced-data-analytics-solutions/)
- [Start Assessment](/agentic-ai-readiness-index/)
- [Data Strategy Strong data foundations for scalable AI](/data-analytics-strategy/)
- [Data Management Leverage data as a strategic asset](/ai-data-management-services/)
- [Data Ops Automate data for speed and quality](/data-devops/)
- [Data Engineering Deliver insights faster with scalable pipelines](/data-engineering/)
- [Cloud Transformation Modernize data to maximise efficiency](/cloud-migration/)
- [Download Whitepaper](/ebooks-whitepapers/building-data-products-in-a-data-mesh-to-drive-business-value/)
- [Data Modeling Structure data for better decisions](/data-modeling-services/)
- [Data Visualization Transform data into actionable stories](/data-visualization-service/)
- [BI Migration Enhance decision making with modern BI tools](/bi-migration/)
- [Data Observability Build trust with healthy, accurate data](/data-observability/)
- [Automated Insights Make smarter decisions with auto-generated insights](/automated-insights/)
- [Download Whitepaper](/ebooks-whitepapers/power-bi-hacks/)
- [CPG & Retail End-to-end analytics for planning, operations, and commercial excellence](/industries/cpg-analytics/)
- [Life Sciences Trusted intelligence across clinical, commercial, and operational workflows](/industries/life-sciences/)
- [Financial Services AI-powered analytics for risk, compliance and customer experience](/industries/banking-financial-analytics-services/)
- [Read case study](/case-studies/data-clean-room-enables-real-time-insights-to-improve-operational-efficiency/)
- [MediaIQ Advanced platform for in-flight marketing measurement](/accelerators/sigmoid-mediaiq-multi-touch-attribution-tool/)
- [CampaignIQ AI-driven platform for optimized campaign budget allocation](/accelerators/sigmoid-campaigniq/)
- [AssistBot GenAI email assistant that automates human-like responses](/accelerators/sigmoid-assistbot-for-ai-email-assistant/)
- [CreativeBot GenAI tool for personalized and brand-aligned creative design](/accelerators/sigmoid-creativebot/)
- [SocialBot GenAI platform to analyze digital conversations and trends](/accelerators/#marketing|socialbot)
- [DemandIQ Predict trends accurately and optimize inventory management](/accelerators/sigmoid-demandiq/)
- [NetworkIQ Track and optimize logistics operations in real-time to quickly address disruptions](/accelerators/sigmoid-networkiq/)
- [SupplyIQ End-to-end platform to optimize supply chain operations](/accelerators/sigmoid-supplyiq/)
- [ProcurementIQ Automated procurement operations for maximum savings, compliance and efficiency](/accelerators/sigmoid-procurementiq/)
- [RapidML Accelerated deployment for machine learning models](/accelerators/sigmoid-rapidml/)
- [DataGuard Comprehensive platform for proactive data quality management](/accelerators/data-quality-tool-sigmoid-dataguard/)
- [CloudPulse Cloud cost optimization platform with multi-cloud management](/accelerators/sigmoid-cloudpulse/)
- [RAPID GenAI foundation with built-in governance and cost clarity](/accelerators/sigmoid-rapid/)
- [AnalyticsBot GenAI based platform to streamline decision-making in analytics](/accelerators/sigmoid-analyticsbot/)
- [DataConnect Seamlessly ingest, integrate and harmonize data from diverse sources](/accelerators/sigmoid-dataconnect/)
- [Reconica AI-powered data harmonization and reconciliation engine](/accelerators/sigmoid-reconica/)
- [ConverseBot GenAI driven insights generation for automated insights from reports](/accelerators/#sales|conversebot)
- [iNRM Cross-lever revenue growth optimization platform](/accelerators/sigmoid-inrm/)
- [AssortmentIQ Optimize shelf layouts and assortment mix at scale with AI-based insights](/accelerators/sigmoid-assortmentiq/)
- [Read Whitepaper](/ebooks-whitepapers/building-agentic-ai-chatbots-for-business-process-transformation/)
- [Listen Podcast](/events/podcast/how-jack-in-the-box-is-redefining-personalization-and-supply-chain-with-ai/)
- [Blogs](/blogs/)
- [White Papers](/ebooks-whitepapers/)
- [Case Studies](/case-studies/)
- [Podcast](/events/podcast/#Podcasts)
- [Read Blog](/blogs/the-genai-adoption-triad-responsibility-ethics-and-explainability/)
- [ConverseBot](/accelerators/#sales|conversebot/)
## Tags
- AI/ML
- Cloud Transformation
---
## Footer Links
- [Talk to our AI experts](/contact-us/)
- [AI Strategy](/enterprise-ai-strategy/)
- [Agentic AI](/agentic-ai-solutions/)
- [Generative AI](/generative-ai/)
- [AI Managed Services](/ai-managed-services/)
- [Responsible AI](/responsible-ai-in-enterprise/)
- [Advanced Analytics](/advanced-data-analytics-solutions/)
- [Data Strategy](/data-analytics-strategy//)
- [Data Engineering](/data-engineering/)
- [Data Management](/ai-data-management-services/)
- [Cloud Transformation](/cloud-transformation/)
- [Data Ops](/data-devops/)
- [Data Visualization](/data-visualization-service/)
- [Automated Insights](/automated-insights/)
- [BI Migration](/bi-migration/)
- [Data Modeling](/data-modeling-services/)
- [Data Observability](/data-observability/)
- [CPG & Retail](/industries/cpg-analytics/)
- [Financial Services](/industries/banking-financial-analytics-services/)
- [Life Sciences](/industries/life-sciences/)
- [Case Studies](/case-studies/)
- [Thought Leadership](/ebooks-whitepapers/)
- [Blogs](/blogs/)
- [Company](/about-sigmoid/)
- [Newsroom](/newsroom/)
- [Accelerators](/accelerators/)
- [Careers](/careers/)
- [Privacy Policy |](/privacy-policy/)
- [Cookie Policy](/cookie-policy/)