The customer wanted to automate the entire process of data collection and visualize the data to analyze market trends, product search ratio, capture customer visits, page hits on sites, and more across regions.They collated and analyzed diverse datasets from multiple countries on a data storage system to track the performance of the website and search engines. However, they followed manual processes to load the datasets, which became difficult to manage with the constant increase in data. The customer wanted to migrate their datasets to Snowflake for a consistent data warehouse, optimize resources, and carry out cost-efficient queries based on data size.
Sigmoid designed automated data pipelines that ingested data to Snowflake from various external data sources through their existing system, APIs, and spreadsheets. We preprocessed data to create a single dataset to be fetched into Snowflake. Technologies like Airflow were used for pipeline scheduling and automation to eliminate manual efforts. In addition, Sigmoid built a dashboard in DOMO to compute different metrics and generate reports for the regional heads, marketing teams, sales teams, and media planners to see the sales and digital marketing data.