1. Model selection
In this stage, data science teams start tweaking and building different models. After a model is selected, model weights are generated and assigned. Next, the idea is to mitigate modeling risk, which primarily deals with answering questions such as:
- Is the data sufficient enough to actually do the predictions at this point?
- Does it contain all the inputs needed to bring about the business change that is in question?
Choosing the right tech stack:
Technologies when selected diligently allows for interoperability across different modeling technologies, if the models are compatible across multiple stacks. Data scientists need to be given freedom to choose from a range of technology stacks or a range of modeling technologies so that they can conveniently explore. At the same time, there should be a check to refrain from technology that makes productionizing the model in consideration, complicated.
2. Model Testing and deployment
As models are being built, testing is a critical process as models are eventually integrated into operational environments where they work on new data everyday and still the output can not be widely inconsistent. The tests may be statistical in nature and may seem constrained, but it can not be side-lined.
After ensuring that the modeling risks have been eliminated, data is predictive in nature, and the right modeling techniques have been selected which can further be taken into production, the model is ready for next step – Deployment. Deployment is a very engineering-driven activity. Below are some of the key aspects that need attention to ensure a smooth process.
a. Codebase: The code base that has so far been written needs to be polished so that it can be battle tested and put into production.
b. Integration: Next the proper integration approach needs to be determined. Some questions that need to be answered are:
- Will there be an API endpoint that people are going to use for obtaining results from the model?
- Is it going to be a bulk process model that will be integrated with ETL tools?
- How to orchestrate the workflow? Will there be a cloud scheduler in Google? if Airflow is used, how will it be automated so that it is comfortable for teams to carry out new integrations into the workflow system?
- How to get detailed access and monitoring, to track SLA reliability.
c. Coding: What coding practices need to be employed?
d. Data Scientists:
- Are they going to be involved in discussions while selecting the development stack?
- Will they have full control over the system (Sigmoid recommends this) or will they just be able to check-in code and see production results?
- How to ensure deep involvement of data scientists from the start, and prevent siloed activities?
The choices that are made at this stage are about eliminating integration risk that involves Integrating different teams together and integration with operational systems. Teams now also witness to many results that start appearing at this stage, e.g. Lift in clicks in case of recommendation systems.
Going beyond deployment
Once the models are deployed, it is important to assess how they will be run and monitored in detail. If there is a sophisticated experimentation system, it is essential to measure the results of those experiments and update the business teams on the different models that are running along with the results they generate. And it’s very important at this stage to start involving more stakeholders to make sure that the results of the models and the ROI of the system is transparently reported for continued operations.