# Data Definition and Baseline

## 0. Learning Objective

* List the questions you need to answer in the process of data definition.
* Compare and contrast the types of data problems you need to solve for structured vs. unstructured and big vs. small data.
* Explain why label consistency is important and how you can improve it
* Explain why beating human level performance is not always indicative of success of an ML model.
* Make a case for improving human level performance rather than beating it.
* Identify how much training data you should gather given time and resource constraints.
* Describe the key steps in a data pipeline.
* Compare and contrast the proof of concept vs. production phases on an ML project.
* Explain the importance of keeping track of data provenance and lineage.

## 1. Define Data and Establish Baseline

### 1.1 Why is data definition hard?

The fundamental reason is label ambiguity

### 1.2 Major types of data problems

![](https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa0Xk3_5WyTdVrfbPk%2F-Mfa0v7o9Jc8DplwRBZ8%2FScreen%20Shot%202021-07-26%20at%2010.34.47%20PM.png?alt=media\&token=d41125dc-d32a-442c-9a3a-5093eaf5fa89)

* Unstructured vs structured data
  * Unstructured data
    * May or may not have huge collection of unlabeled examples x
    * Humans can label more data
    * Data augmentation more likely to be helpful
  * Structured data
    * May be more difficult to obtain more data
    * Human labeling may not be possible (with some exceptions)
* Small data (< 10K) vs. big data (>10K)
  * Small data
    * Clean labels are critical
    * Can manually look through dataset and fix labels
    * Can get all the labelers to talk to each other
  * Big data
    * Emphasis data process

### 1.3 Small data and label consistency

Why label consistency is important

![](https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa0Xk3_5WyTdVrfbPk%2F-Mfa28E6p1tsufmAd8g_%2FScreen%20Shot%202021-07-26%20at%2010.40.10%20PM.png?alt=media\&token=8ae58fe5-2ca5-4be6-825c-b01779f1999b)

### Big data problems can have small data challenges too

* Problems with large dataset but where there's a long tail of rare events in the input will have small data challenges too
  * Web search
  * Self-driving cars
  * Product recommendation systems

### 1.4 Improving label consistency

* Have multiple labelers label same example
* When there is disagreement, have MLE, subject matter expert, to discuss definition of y to reach agreement
* If labelers believe that x doesn't contain enough information, consider changing x
* Iterate until it is hard to significantly increase agreement
* Have a class/label to capture uncertainty
* Small data vs big data (unstructured data)
  * Small data
    * &#x20;Usually small number of labelers
    * Can ask labelers to discusss specific labels
  * Big data
    * Get consistent definition with small group&#x20;
    * Then send labeling instructions to labelers
    * Can consider having multiple labelers label every example and using voting to increase accuracy

### 1.5 Human Level Performance (HLP)

* Why measure HLP
  * Estimate Bayes error/irreducible error to help with error analysis and prioritization
* Other uses of HLP
  * In academia, establish and beat a respectable benchmark to support publication
  * Business or product owner asks for 99% accuracy. HLP helps establish a more reasonable target
  * "Prove" the ML system is superior to humans doing the job and thus the business or product owner should adopt it. -> **Rarely work**
    * Problem with beating HLP as a "proof" of ML superiority

### 1.6 Raising HLP

* When the label y comes from a human label, HLP << 100% may indicate ambiguous labeling instructions

* Improving label consistency will raise HLP

* This makes it harder for ML to beat HLP. But the more consistent labels will raise ML performance, which is ultimately likely to benefit the actual application performance

* HLP on structured data

  Structured data problems are less likely to involve human labelers ==> HLP is less frequently used

  * Some exceptions
    * User ID merging: Same person?
    * Based on network traffic, is the computer hacked?
    * Is the transaction fraudulent?
    * Spam account? Bot?
    * From GPS, what is the mode of transportation - on foot, bike, car, bus

## 2. Label and Organize Data

### 2.1 Obtaining data

* How long should you spend obtaining data?

  <img src="https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa2cw3b6n-lkD-zLye%2F-Mfa7ATrSzOTmt2WPBld%2FScreen%20Shot%202021-07-26%20at%2011.02.09%20PM.png?alt=media&#x26;token=b2c1f251-c8cb-4276-af2b-3242e5ec0bad" alt="" data-size="original">&#x20;

  * Get into this iteration loop as quickly as possible
  * Instead of asking: How long it would take to obtain m examples? =>&#x20;
    * Ask: How much data can we obtain in k days
  * Exceptions: If you have worked on the problem before and from experience you know you need m examples
* Inventory data: Brainstorm list of data sources

  <img src="https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa2cw3b6n-lkD-zLye%2F-Mfa7ovtDZXkVhP5Z_jJ%2FScreen%20Shot%202021-07-26%20at%2011.05.01%20PM.png?alt=media&#x26;token=983b6da2-01c1-46ab-ae1f-26bd783d13d0" alt="" data-size="original">&#x20;
* Labeling data
  * Options: In-house vs outsourced vs. crowdsourced
  * Having MLEs label data is expensive. But doing this for just a few days is usually fine
  * Don't increase data by more than 10x at a time

### 2.2 Data pipeline

![](https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa2cw3b6n-lkD-zLye%2F-Mfa8fIXggOVy8iyxoRX%2FScreen%20Shot%202021-07-26%20at%2011.08.42%20PM.png?alt=media\&token=edfaf1b0-8482-48e6-97e1-cf21ba39d3ab)

* POC (Proof-of-concept)
  * Goal is to decide if the application is workable and worth deploying
  * Focus on getting the prototype to work
  * It is OK if data pre-processing is manual. But take extensive notes/comments
* Production phase
  * After project utility is established, use more sophisticated tools to make sure the data pipeline is replicable
  * E.g. Tensorflow Transform, Apache Beam, Airflow

### 2.3 Meta-data, data provenance and lineage

Data pipeline example

![](https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-Mfa8j9MdQ_1iC436r1v%2F-Mfa9qWU-Q71stmrJEfE%2FScreen%20Shot%202021-07-26%20at%2011.13.53%20PM.png?alt=media\&token=a4f98d69-f7c7-4254-82c2-2e5deb172896)

* Meta-data
  * Examples:
    * Manufacturing visual inspection: time, factory, line #, camera settings, phone model, inspector ID
    * Speech recognition: Device type, Labeler ID, VAD model ID
  * Useful for:
    * Error analysis, Spotting unexpected effects
    * Keeping track of data provenance

## 3. Scoping

### 3.1 What is scoping

Scoping example: Ecommerce retailer looking to increase sales

* Better recommender system
* Better search
* Improve catalog data
* Inventory management
* Price optimization

Questions:

* What project should we work on?
* What are the metrics for success?
* What are the resources (data, time, people) needed?

### 3.2 Scoping process

1. Brainstorm business problems (not AI problems)
2. Brainstorm AI solutions
3. Assess the feasibility and value of potential solutions
4. Determine milestones
5. Budget for resources

**What are the top 3 things you wish were working better?**

* Increase conversion
* Reduce inventory
* Increase margin (profit per item)

**Separating problem identification from solution**

![](https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-MfaBOSJQKvdw7XubRxt%2F-MfaCMn68EcvBuGJRe84%2FScreen%20Shot%202021-07-26%20at%2011.24.42%20PM.png?alt=media\&token=25a67406-ea08-45e2-8c52-516cdacbd319)

### 3.3 Diligence on feasibility and value

* Feasibility: Is this project technically feasible?

  Use external benchmark (literature, other company, competitor)

  <img src="https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-MfaBOSJQKvdw7XubRxt%2F-MfaCuHmrrDo8RrTxqWQ%2FScreen%20Shot%202021-07-26%20at%2011.27.12%20PM.png?alt=media&#x26;token=992f8cb8-a892-41e2-8580-6ee1acc81285" alt="" data-size="original">&#x20;

  * Why use HLP to benchmark

    People are very good on unstructured data tasks

    * Criteria: Can a human, given the same data, perform the task?
  * Do we have features that are predictive?
* Value

  <img src="https://4115272635-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-Lxy_o_8Hh0RQ4QsSSWG%2F-MfaBOSJQKvdw7XubRxt%2F-MfaDf3ROhUXKeYgVZNV%2FScreen%20Shot%202021-07-26%20at%2011.30.33%20PM.png?alt=media&#x26;token=cd989386-dd27-4430-882e-198e3c8264a5" alt="" data-size="original">&#x20;

  * Ethical considerations
    * Is this project creating net positive social value?
    * Is this project reasonably fair and free from bias?
    * Have any ethical concern been openly aired and debated?

### 3.4 Milestones and resourcing

* Key specifications:
  * ML metrics (accuracy, precision, F1)
  * Software metrics (latency throughput, given compute resources)
  * Business metrics (revenue)
  * Resources needed (data, personnel, help from other teams)
  * Timeline
* If unsure, considering benchmark to other projects, or building a POC first
