When Should I Consider Model Rebuilds/Refreshes?

Should Infer Rebuild/Refresh the Model Daily?
 
We experimented with daily refreshes when Infer first launched because it required less data scientist supervision. However, we ultimately decided that it is better for our customers to invest the data science time in periodic refreshes instead.
 
Daily refreshing ends up being less helpful from a business sense- instead, we operate more like Google where we tend to make changes periodically instead of constantly changing.
 
  • Operational Challenges: Similar to Google's algorithm changing, if we had the scoring always shifting, operationalizing (especially reporting) would be much harder on our customers

  • Biases: There is a time bias that favors the most recent information (e.g,. what if you just imported a large list around a large conference - features around list buys would dominate your model)

  • Supervision: On any given refresh, there could be potentially big changes happening to your scoring without any active input from the business or data scientists

  • Danger: If your underlying customer isn't changing everyday - it's actually dangerous to refresh everyday as you'd be exponentially increase the chance of latching onto noise instead of significant signals on any given refresh.

 

Instead, we've invested into behind-the-scenes product solutions that ensure models remain accurate and robust. There's art as well as science in model building so that once a model has locked onto signal vs noise, it can generally handle a variety of things thrown at it. 

 
When Should I Refresh?

Here at Infer, we have a team of data scientists on hand monitoring the performance of your model. We'll also periodically check-in for proactive model rebuilds. 

However, there are also important reasons that you should notify your account manager regarding the need for a rebuild or refresh:

1. Schema changes: Infer takes into account many common fields when building your model. It’s important to note any changes to these fields that may affect the way we pull data.

Examples:

  • This could be as simple as changing the name of the ‘company’ field to ‘company_name’.
  • It could also be an issue if you were to change the fields in your ‘contact us’ form. We may have relied on certain fields in that form which would then affect the way your model is performing.

 The screenshot below is an example that includes many of the common fields we will pull from your automation system. Please notify us of any plans to change fields like this:

2. Business shift – If you are going through a business change it’s important to contact your account manager. The main reason is that you may not need an entirely new model but you may want to rebuild your existing model over new data. This is often a business decision rather than a data science decision and it’s helpful to talk through this. 

Essentially - has your underlying buyer changed? is what we're most concerned about.

Examples of business changes we should know about:

  • Go-To-Market Change: You have recently implemented new pricing, new products or anything that effectively changes who your underlying customer is (e.g., adding a new Free Trial option, or starting to bundle services)
  • Sales Model Change: You recently switched from an outbound sales model to an Inbound sales model
  • SFDC Object Change: You recently switched from working Leads to working Contacts
  • Lead Type Change: Your business rarely used a lead source such as List Buys and is now starting to spend more efforts against it.

 

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk