Model Deployment #

Key Challenges #

Common Deployment Cases #

1. New product
2. Automate/assist with manual task
3. Replace previous ML system

Deployment modes #

1. Shadow mode
   • ML system works in parallel with current solution (manual or previous automated system)
   • ML system’s decisions are not taken into account at this point
   • Monitoring system compares results from both systems to estimate accuracy and probably collect more training data
2. Canary deployment
   • ML system works in parallel with current solution
   • New system handles a small portion of traffic, e.g. 5%
   • If there’s no degradation, the portion is gradually increased
3. Blue/Green Deployment
   • ML system works in parallel with current solution
   • At some point a router in front of both systems switches all the traffic to the new system
   • In case of degradation, the rollback is easy

Degree of Automation #

flowchart LR HO[/Human Only/] SM[/Shadow Mode/] AA[/AI Assistance/] PA[/Partial Automation/] FA[/Full Automation/] HO---SM---AA---PA---FA

“Human in the loop” deployments:

• AI Assistance: ML System highlights interesting input, but the decision is still taken by human
• Partial Automation: the decisions are taken by the ML System, but if it is not sure, it forwards the request to human. The approach if very useful to collect more training data when the accuracy is not good enough.

Monitoring #

To build a monitoring dashboard:

• Brainstorm the things that could go wrong
• Define the metrics that would reflect these problems

It is OK to start with a big number of metrics and remove some of them as you understand which are not representative.

Examples of Metrics #