---
# Using Datawig, an AWS deep learning library for missing value imputation
**URL:** https://www.sigmoid.com/blogs/datawig-missing-value-imputation/
Date: 2021-12-02
Author: Sigmoid
Post Type: post
Summary: While training a Machine Learning model the quality of the model is directly proportional to the quality of data. However, in some...Read More...
Categories: Cloud Transformation
Tags: AI/ML, Data Management
Featured Image: https://www.sigmoid.com/wp-content/uploads/2023/10/Using-Datawig-an-AWS-Deep-Learning-Library-for-Missing-Value-Imputation-banner-opt.jpg
---
While training a Machine Learning model the quality of the model is directly proportional to the quality of data. However, in some cases, there are a lot of missing values in the dataset affecting the quality of prediction in the long run. Several methods can be used to fill the missing values and Datawig is one of the most efficient ones.
Datawig is a Deep Learning library developed by AWS Labs and is primarily used for “[Missing Value Imputation”](/case-studies/neural-network-architecture/). The library uses “mxnet” as a backend to train the model and generate the predictions.
In this blog, we will look into some of the important components of the library and how it can be used for imputing missing values in a dataset.
## Important Components of the Datawig Library
To understand how the library works let’s first go through some of the important components and understand what exactly they do.
- Column Encoders
- The column encoders convert the raw data of a column into an encoded numerical representation
- We have four Column encoders present in the Datawig library:
- Sequential Encoder For sequencing text data
- BowEncoder: Bag-of-words encoding for text data. (Hashing vectorizer or tfidf based on the algorithm used)
- Categorical Encoder: One hot encoding for categorical columns
- Numerical Encoder: For encoding numerical columns
- Column Featurizers
- Column featurizers are used to feed encoded data from “Column Encoders” into the imputer model’s computational graph for training and prediction
- We have four column featurizers present in the Datawig library:
- LSTMFeaturizer:To be used with Sequential Encoder and maps Sequence of input into vectors using LSTM
- BowFeaturizer: To be used with Bag-of-Words encoded columns
- EmbeddingFeaturizer: Maps encoded categorical columns into vector representation (word embeddings)
- NumericalFeaturizer: To be used with numerical encoded columns and extract features using fully connected layers
- Simple Imputer
- Using a simple imputer is the simplest way one can train a missing value imputation model. It only takes the below three parameters:
- Input_column: This represents the list of feature columns
- Output_column: This takes the name of the target column that one is training
- Output_path: The path where the trained model is supposed to be stored
- Say we have a dataset with 3 different columns a, b, c and based on a and b we want to fill the missing values of column c. Then, simple imputer will work as follows:
```
imputer = SimpleImputer(
input_columns=['a', 'b'],
output_column='c',
output_path = 'imputer_model'
)
#Fit an imputer model on the train data
imputer.fit(train_df=df_train)
predictions = imputer.predict(df_test)
```
- While using a simple imputer one doesn’t need to worry about encoding and featurizing different input columns as the library automatically detects the data type for each column and uses the encoders and featurizers accordingly
However, this results in less control over the training process. But in general, it yields good results
- After passing the above parameter, there can be two options::
- Imputer.fit: To train model
- Imputer.fit_hpo: To train and tune the model. It has a dictionary built in to choose the values from, also one can pass hyperparameters in the form of a custom dictionary to tune the model based on project requirements
- Imputer
- Imputer gives more control over the training process, which is one of the primary reasons for using Imputer over simple imputer
- Imputer takes four parameters as input:
- Data_featurizers: It’s a list of featurizers associated with different feature columns
- Label_encoders: It’s a list of encoded target columns
- Data_encoders: It’s a list of encoders associated with different feature columns
- Output_path: The path where the trained model is supposed to be stored
- Say we have a dataset with 3 different columns a, b, c, and based on a and b we want to fill the missing values of column c. Then, imputer will work as follows:
```
data_encoder_cols = [BowEncoder('a'), BowEncoder('b')]
label_encoder_cols = [CategoricalEncoder('c')]
data_featurizer_cols = [BowFeaturizer('a'), BowFeaturizer('b')]
imputer = Imputer(
data_featurizers=data_featurizer_cols,
label_encoders=label_encoder_cols,
data_encoders=data_encoder_cols,
output_path='imputer_model'
)
imputer.fit(train_df=df_train)
predictions = imputer.predict(df_test)
```
- After defining the imputer with the above parameters we can simply call the “.fit” function to begin the training
- Advantages of Imputer over SimpleImputer:
- More customization is possible for the training process
- Tuning the parameters while encoding the feature and target columns to get a balance between the training time and accuracy of the model
## How Datawig Helped in my Project?
- Overview of the Project
- We had a dataset with 50 different columns and we had to impute the missing values on 25 different columns out of those 50
- Out of the 25 columns 13 columns were numerical and 12 were categorical
- Part summary of the dataset
- Approach
- For each of the target columns (25 columns for which we are doing the imputation), we did the feature selection and then ran the Datawig using the Imputer Since we can run the Imputer over all the target columns, at one time on the loop it was pretty straightforward. After the base model results we went on to tune the model and below were the final results on the target columns.
- Numerical Columns:
The client-defined metric was RMSE/Standard deviation of
---
## Categories
- Cloud Transformation
---
## Navigation
- [Company](/about-sigmoid)
- [Newsroom](/newsroom)
- [Life at Sigmoid](/careers)
- [Takshashila](/takshashila)
- [Contact Us](/contact-us)
- [AI Strategy Blueprint your AI advantage](/enterprise-ai-strategy/)
- [Generative AI Drive innovation with Generative AI](/generative-ai/)
- [Responsible AI Build trust with ethical AI practices](/responsible-ai-in-enterprise/)
- [Agentic AI Reshape business with scalable agentic systems](/agentic-ai-solutions/)
- [AI Managed Services Ensure reliable AI performance](/ai-managed-services/)
- [Advanced Analytics Transform your business with data-driven insights](/advanced-data-analytics-solutions/)
- [Start Assessment](/agentic-ai-readiness-index/)
- [Data Strategy Strong data foundations for scalable AI](/data-analytics-strategy/)
- [Data Management Leverage data as a strategic asset](/ai-data-management-services/)
- [Data Ops Automate data for speed and quality](/data-devops/)
- [Data Engineering Deliver insights faster with scalable pipelines](/data-engineering/)
- [Cloud Transformation Modernize data to maximise efficiency](/cloud-migration/)
- [Download Whitepaper](/ebooks-whitepapers/building-data-products-in-a-data-mesh-to-drive-business-value/)
- [Data Modeling Structure data for better decisions](/data-modeling-services/)
- [Data Visualization Transform data into actionable stories](/data-visualization-service/)
- [BI Migration Enhance decision making with modern BI tools](/bi-migration/)
- [Data Observability Build trust with healthy, accurate data](/data-observability/)
- [Automated Insights Make smarter decisions with auto-generated insights](/automated-insights/)
- [Download Whitepaper](/ebooks-whitepapers/power-bi-hacks/)
- [CPG & Retail End-to-end analytics for planning, operations, and commercial excellence](/industries/cpg-analytics/)
- [Life Sciences Trusted intelligence across clinical, commercial, and operational workflows](/industries/life-sciences/)
- [Financial Services AI-powered analytics for risk, compliance and customer experience](/industries/banking-financial-analytics-services/)
- [Read case study](/case-studies/data-clean-room-enables-real-time-insights-to-improve-operational-efficiency/)
- [MediaIQ Advanced platform for in-flight marketing measurement](/accelerators/sigmoid-mediaiq-multi-touch-attribution-tool/)
- [CampaignIQ AI-driven platform for optimized campaign budget allocation](/accelerators/sigmoid-campaigniq/)
- [AssistBot GenAI email assistant that automates human-like responses](/accelerators/sigmoid-assistbot-for-ai-email-assistant/)
- [CreativeBot GenAI tool for personalized and brand-aligned creative design](/accelerators/sigmoid-creativebot/)
- [SocialBot GenAI platform to analyze digital conversations and trends](/accelerators/#marketing|socialbot)
- [DemandIQ Predict trends accurately and optimize inventory management](/accelerators/sigmoid-demandiq/)
- [NetworkIQ Track and optimize logistics operations in real-time to quickly address disruptions](/accelerators/sigmoid-networkiq/)
- [SupplyIQ End-to-end platform to optimize supply chain operations](/accelerators/sigmoid-supplyiq/)
- [ProcurementIQ Automated procurement operations for maximum savings, compliance and efficiency](/accelerators/sigmoid-procurementiq/)
- [RapidML Accelerated deployment for machine learning models](/accelerators/sigmoid-rapidml/)
- [DataGuard Comprehensive platform for proactive data quality management](/accelerators/data-quality-tool-sigmoid-dataguard/)
- [CloudPulse Cloud cost optimization platform with multi-cloud management](/accelerators/sigmoid-cloudpulse/)
- [RAPID GenAI foundation with built-in governance and cost clarity](/accelerators/sigmoid-rapid/)
- [AnalyticsBot GenAI based platform to streamline decision-making in analytics](/accelerators/sigmoid-analyticsbot/)
- [DataConnect Seamlessly ingest, integrate and harmonize data from diverse sources](/accelerators/sigmoid-dataconnect/)
- [Reconica AI-powered data harmonization and reconciliation engine](/accelerators/sigmoid-reconica/)
- [ConverseBot GenAI driven insights generation for automated insights from reports](/accelerators/#sales|conversebot)
- [iNRM Cross-lever revenue growth optimization platform](/accelerators/sigmoid-inrm/)
- [AssortmentIQ Optimize shelf layouts and assortment mix at scale with AI-based insights](/accelerators/sigmoid-assortmentiq/)
- [Read Whitepaper](/ebooks-whitepapers/building-agentic-ai-chatbots-for-business-process-transformation/)
- [Listen Podcast](/events/podcast/how-jack-in-the-box-is-redefining-personalization-and-supply-chain-with-ai/)
- [Blogs](/blogs/)
- [White Papers](/ebooks-whitepapers/)
- [Case Studies](/case-studies/)
- [Podcast](/events/podcast/#Podcasts)
- [Read Blog](/blogs/the-genai-adoption-triad-responsibility-ethics-and-explainability/)
- [ConverseBot](/accelerators/#sales|conversebot/)
## Tags
- AI/ML
- Data Management
---
## Footer Links
- [Talk to our AI experts](/contact-us/)
- [AI Strategy](/enterprise-ai-strategy/)
- [Agentic AI](/agentic-ai-solutions/)
- [Generative AI](/generative-ai/)
- [AI Managed Services](/ai-managed-services/)
- [Responsible AI](/responsible-ai-in-enterprise/)
- [Advanced Analytics](/advanced-data-analytics-solutions/)
- [Data Strategy](/data-analytics-strategy//)
- [Data Engineering](/data-engineering/)
- [Data Management](/ai-data-management-services/)
- [Cloud Transformation](/cloud-transformation/)
- [Data Ops](/data-devops/)
- [Data Visualization](/data-visualization-service/)
- [Automated Insights](/automated-insights/)
- [BI Migration](/bi-migration/)
- [Data Modeling](/data-modeling-services/)
- [Data Observability](/data-observability/)
- [CPG & Retail](/industries/cpg-analytics/)
- [Financial Services](/industries/banking-financial-analytics-services/)
- [Life Sciences](/industries/life-sciences/)
- [Case Studies](/case-studies/)
- [Thought Leadership](/ebooks-whitepapers/)
- [Blogs](/blogs/)
- [Company](/about-sigmoid/)
- [Newsroom](/newsroom/)
- [Accelerators](/accelerators/)
- [Careers](/careers/)
- [Privacy Policy |](/privacy-policy/)
- [Cookie Policy](/cookie-policy/)