Introduction
https://www.educative.io/courses/grokking-the-machine-learning-interview
What to expect in a machine learning interview?
Problem-solving/ coding
This portion of the interview is fairly similar to other software engineering coding interviews where the interviewer gives a coding problem, such as perform an ‘In-order tree traversal’, and the candidate is expected to solve that in about half an hour. There is ample content available on how to best prepare for such questions.
Machine learning understanding
This area generally focuses on individual understanding of basic ML concepts such as supervised vs. unsupervised learning, reinforcement learning, classification vs. regression, deep learning, optimization functions, and the learning process of various ML algorithms. There are many courses and books that go over these fundamental concepts. They facilitate the learning of ML basics and help candidates prepare for the interview.
Machine learning system design discussion
This discussion focuses on the interviewee’s ability to solve an end-to-end machine learning problem and consists of open-ended questions.
Machine Learning system design interviews
In the ML system design interview portion, candidates are given open-ended machine learning problems and are expected to build an end-to-end machine learning system to solve that problem. It could be:
Build a recommendation system that shows relevant products to users.
Build a visual understanding system for a self-driving car.
Build a search-ranking system.
Approaching ML design questions
Understanding the problem
Ask questions for understanding the problem
Example 1: Design a search engine that displays the most relevent results in response
Is it a general search engine like google or bing, or a specilized search engine like Amazon's products search?
What kind of queries is expected to answer?
==> Build a generic search engine that returns relevant results for queries like "Richard Nixon", "Programming languages" etc.
Example 2: build a system to display a Twitter feed for a user
How the feed is currently displayed?
How it can be improved to provide a better experience for users
==> Given a list of tweets, train an ML model that predicts the probability of engagement of tweets and orders them based on that score.
Understand scale and latency requirements
Discuss the performance and capacity considerations of the system. This conversation will help you to clearly understand the scale of the system
Latency requirements
Example 1: a search engine problem:
Do we want to return the search result in 100 ms or 500 ms
Example 2: Twitter feed problem:
Do we want to return a list of relevant tweets in 300 ms or 400 ms
Scale of the data
Example 1: search engine problem:
How many requests per second d owe anticipate to handle
How many websites exist we want to enable through this search engine
If a query has 10 billion matching documents, how many of these would be ranked by our model
Example 2: Twitter feed problem:
How many tweets would we have to rank according to relevance for a user at a time
Defining metrics
Knowing our success criteria helps in understanding the problem and in selecting key architectural components. This is why it’s important to discuss metrics early in our design discussions.
Metrics for offline testing
AUC
Log loss
Precision/ recall/ F1 score
Metrics for online testing
It includes (1) component-wise and (2) end-to-end metrics.
Example 1: search engine problem:
Component-wise metric: NDCG for measuring the model performance online
end-to-end metric: user engagement and retention rates
Architecture discussion
You need to think about the components of the system and how the data will flow through those components.
Search Engine ML system
Query rewriting: for poorly worded or mispelled
Query understanding: helps identify the intent of a query
Document selection: selects relevant documents from the billions of documents
Ranker: ranks the result based on relevance
Results display: rerank the results based on diversity
Architecting for Scale: display relevant ads to users
Use funnel approach, where each stage will have fewer ads to process
Offline model building and evaluation
Traning data generation
Human labeled data
Through a user's interaction with a pre-existing system
Labeling the data based on users' actions
Feature engineering
User-based features
Item-based features
Statistic features of items
User-item features
Model training
If using the funnel approach, we need to have a simple model at the beginning and a complicate model in the end
Offline evaluation
Divide the data into training and validation sets
Online model execution and evaluation
Selecting proper online metrics for evaluating the performance of the model
Retention rate
Click-through rate
Convention rate
Monitoring model performance and retraining
What is model drift?
Model Drift refers to a model’s predictive performance degrading over time due to a change in the environment that violates the model’s assumptions.
How to identify model drift?
Monitor the accuracy of a deployed model is difficult
The predictions weren’t stored after they were generated
The predictions are stored but you can’t access the ground truth labels.
The predictions and labels are both available but can’t be joined together
Try to understand the mode drift in the offline model scenario
Method to infer the data drift:
Examine the feature distributions of training and live data
the range of possible values
histograms of values
whether the feature accepts NULLs and if so, the number of NULLs expect
Examine the correlations between features
monitoring correlation coefficients between features
training models with one or two features
training a set of models that each have one of the features removed
Examine the target distributions
Check if the target distribution changes significantly
How can you retrain your model
If we want to do the training every year and training does not take too long time, we may want to retrain the model with the new data
Train periodically, we can do batch retraining.
Train the model based on a new batch set of data
Automated model drift detection and training. Retrain with the new data
Online training method
Reference
Last updated
Was this helpful?