Blueshift
  • Product  
    • PLATFORM
      • SmartHub CDP
      • Omnichannel Orchestration
      • Predictive Intelligence
      • Single Customer View
      • Audience Segmentation
      • 1:1 Personalization
    • SOLUTIONS
      • Email Automation
      • Mobile Marketing
      • Website Personalization
      • Audience Targeting
      • Contextual Chat
    • PLANS AND INTEGRATIONS
      • Integration Partners
      • Support Plans
      • Pricing
  • Customers
  • Resources  
    • Library
    • Blog
    • Videos
    • Documentation
    • Product Updates
    • Blueshift Academy
  • Company  
    • About Blueshift
    • Events
    • News & Awards
    • Careers
  • Contact Us
  • LOGIN
  • Search
  • Menu Menu
  • Product
    • PLATFORM
      • SmartHub CDP
      • Omnichannel Orchestration
      • Predictive Intelligence
      • Single Customer View
      • Audience Segmentation
      • 1:1 Personalization
    • SOLUTIONS
      • Email Automation
      • Mobile Marketing
      • Website Personalization
      • Contextual Chat
      • Retired – Audience Targeting
    • PLANS AND INTEGRATIONS
      • Integration Partners
      • Support Plans
      • Pricing
  • Customers
  • Resources
    • Library
    • Blog
    • Videos
    • Documentation
    • Product Updates
    • Blueshift Academy
  • Company
    • About Blueshift
    • Events
    • News & Awards
    • Careers
  • LOGIN
  • Contact Us

How do you know if your model is going to work? Part 4: Cross-validation techniques

Digital Marketing Practices & Insights
Data code

Concluding our guest post series!

Authors: John Mount (more articles) and Nina Zumel (more articles).

In this article we conclude our four part series on basic model testing.

When fitting and selecting models in a data science project, how do you know that your final model is good? And how sure are you that it’s better than the models that you rejected? In this concluding Part 4 of our four part mini-series “How do you know if your model is going to work?” we demonstrate cross-validation techniques.

Previously we worked on:

  • Part 1: The problem
  • Part 2: In-training set measures
  • Part 3: Out of sample procedures

Cross-validation techniques

Cross validation techniques attempt to improve statistical efficiency by repeatedly splitting data into train and test and re-performing model fit and model evaluation.

For example: the variation called k-fold cross-validation splits the original data into k roughly equal sized sets. To score each set we build a model on all data not in the set and then apply the model to our set. This means we build k different models (none which is our final model, which is traditionally trained on all of the data).

NewImage
Notional 3-fold cross validation (solid arrows are model construction/training, dashed arrows are model evaluation).

This is statistically efficient as each model is trained on a 1-1/k fraction of the data, so for k=20 we are using 95% of the data for training.

Another variation called “leave one out” (which is essentially Jackknife resampling) is very statistically efficient as each datum is scored on a unique model built using all other data. Though this is very computationally inefficient as you construct a very large number of models (except in special cases such as the PRESS statistic for linear regression).

Statisticians tend to prefer cross-validation techniques to test/train split as cross-validation techniques are more statistically efficient and can give sampling distribution style distributional estimates (instead of mere point estimates). However, remember cross validation techniques are measuring facts about the fitting procedure and not about the actual model in hand (so they are answering a different question than test/train split).

There is some attraction to actually scoring the model you are going to turn in (as is done with in-sample methods, and test/train split, but not with cross-validation). The way to remember this is: bosses are essentially frequentist (they want to know their team and procedure tends to produce good models) and employees are essentially Bayesian (they want to know the actual model they are turning in is likely good; see here for how it the nature of the question you are trying to answer controls if you are in a Bayesian or Frequentist situation).

Read more.

 

September 21, 2015/by Alan Brusky
Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on Pinterest
  • Share on LinkedIn
  • Share on Reddit
  • Share by Mail
https://blueshift.com/wp-content/uploads/How-do-you-know-if-your-model-Part-4-V2.jpg 640 1200 Alan Brusky https://blueshift.com/wp-content/uploads/blueshift-primary.svg Alan Brusky2015-09-21 22:00:422019-11-22 10:56:18How do you know if your model is going to work? Part 4: Cross-validation techniques

Recent Articles

  • Customer Data Platform experts David Raab and Shamir Duverseau join Blueshift in our exclusive webinar. Webinar Recap: What the CDP?! With the CDP Institute and Smart Panda LabsJanuary 15, 2021 - 5:10 am
  • new retail cx and marketing trends 8 Retail CX and Marketing Trends for the New YearJanuary 13, 2021 - 5:19 am
  • ai recommendations saw an 81% increase in marketing revenue Why You Need AI-Powered Recommendations in Your MarketingJanuary 7, 2021 - 6:07 am
  • hackathon 2020 winners Blueshift’s 2020 Hackathon RecapDecember 18, 2020 - 6:45 am
  • Creating Dynamic Templates for Mobile Channels is Easier than Ever with Blueshift’s Enhanced Creative StudioDecember 16, 2020 - 1:51 am
  • G2 Momentum Leader in Marketing Automation and CDP Grids Winter 2021 Blueshift Named a Momentum Leader for CDP and Marketing Automation Categories in G2’s Winter 2021 ReportDecember 15, 2020 - 6:25 am

Headquarters

433 California St, Suite 600,
San Francisco, CA 94104

Global Offices

Charlotte, NC
Pune, India
London, UK

hello@blueshift.com

Company

  • About Blueshift
  • Customers
  • News and Awards
  • Events
  • Careers
  • Contact Us

Platform

  • SmartHub CDP
  • Single Customer View
  • Audience Segmentation
  • Predictive Intelligence
  • 1:1 Personalization
  • Omnichannel Orchestration
  • Integration Partners

Solutions

  • Email Automation
  • Mobile Marketing
  • Website Personalization
  • Audience Targeting
  • Contextual Chat

Resources

  • Documentation
  • Developer Portal
  • Product Updates
  • Case Studies
  • Reports
  • RFP Guide
  • Blueshift Academy

© 2020 COPYRIGHT BLUESHIFT LABS, INC. PRIVACY POLICY   |   TERMS OF SERVICE   |   ANTI-SPAM POLICY

Scroll to top