The customer used BigQuery to collate and analyze diverse datasets from multiple countries to track the performance of the website and search engines. However, they followed manual processes to load the datasets, which became difficult to manage with the constant increase in data. The customer wanted to migrate their datasets to Snowflake for a consistent data warehouse, optimize resources, and carry out cost-efficient queries based on data size. They wanted to automate the entire process and visualize the data to analyze market trends, product search ratio, capture customer visits, page hits on sites, and more across regions.
Sigmoid designed automated data pipelines that ingested data to Snowflake from various external data sources through BigQuery, APIs, and spreadsheets. We preprocessed data to create a single dataset to be fetched into Snowflake. Technologies like Airflow were used for pipeline scheduling and automation to eliminate manual efforts. In addition, Sigmoid built a dashboard in DOMO to compute different metrics and generate reports for the regional heads, marketing teams, sales teams, and media planners to see the sales and digital marketing data.