I recently came across an old issue of Wired Magazine which contained the article, Recipe for Disaster: The Formula That Killed Wall Street. It describes the rise and fall of a model that was used by Wall Street from 2001-2007 to price certain kinds of complex financial securities. The problem was that users of the model didn’t really understand its major economic assumptions and limitations, so they continued using the model even after the economy started behaving in ways that violated the model’s assumptions. And the results were disastrous.
In today’s business environment, where analytic tools such as response models, segmentation etc. are widely used to inform business decisions, it is critical for marketing executives to understand the limitations of the tools they are relying upon – and to know when these limitations are being violated. Otherwise, they risk incurring the same fate that befell these Wall Street investors.
Best Practices for Maintaining Analytic Tools
The best practices listed below will help executives who want to ensure that their analytic tools continue to be reliable sources of marketing information:
1.) Understand the assumptions and limitations inherent in your analytic tools
All analytic tools are built off a base population using data from a defined period of time. Know that population and know that period of time. Know what business goals the tools were built to support and ask the question, “How has your marketing environment changed since the model was originally built?” For example, have you seen any changes in: o the profile of your customer base?
o your mix of product offerings?
o your marketing channels and key messages?
o the competitive environment?
If there have been any significant changes in these areas, you need to work with your analytic team to think through the potential ramifications these changes might have on your model output and be comfortable that your model results are as relevant today as they were when the model was originally built.
2.) Keep close track of changes in model variables
Your analytic team or vendor will know what specific variables were used to build the original model as well as their range of values during the analysis time period. If certain values have changed significantly, you should consider having the model rebuilt or at least refreshed (re-estimating the model coefficients against a new population) to ensure its effectiveness. Standard reports that monitor the distribution of key variables in your models can be invaluable resources for tracking these changes.
3.) Commit to a systematic schedule of model maintenance
Committing to a systematic schedule of maintenance, including complete model rebuilds and simple model refreshes, as appropriate, can help head-off problems before they start and ensure that your marketing decision-making is supported by the current dynamics of your marketing environment.
Following these best practices can help ensure you’re making decisions using the best information available and not being deceived by outdated analytics. If you’d like to comment or have a private discussion, simply click the link below and for more information about iKnowtion, please visit our website at www.iknowtion.com. |
