Everyone Likes Validation … of their Marketing Models!

Validation of behavioral models is an interesting topic for marketing scientists. For the rest of our colleagues, it probably rates up there with the joke about a lawyer, an engineer, and an econometrician. (If you don’t know that joke, ask me sometime.)

Roughly, model validation is composed of two activities. First, the accuracy of the model is measured by comparing model predictions to known values. Second, the ability of the model to classify marketing prospects into more-desirable and less-desirable groups is assessed. This ability is known as rank-ordering.

On a recent project we used response models to optimize return on marketing investment (ROMI) for a client. The models were developed by a third-party. The models were new and untested, and it was my job to validate the performance of the vendor’s models and develop estimates of the response curves from the models.

The statistic the vendors used to gauge model rank-ordering was KS separation, which is another name for the Kolmogorov-Smirnov (K-S) statistic. “KS separation” is probably not a household term for most of you. It wasn’t for me in the context of model validation. (I’ve seen the K-S statistic used to assess whether the customers in two samples behave alike enough such that they can be treated as coming from the same audience.) My thought upon reviewing the vendor’s report was, “I wonder what other professionals are using to validate model rank-ordering?”

By now most you are thinking, “rank-ordering is clearly a ‘numbers guy’ topic. I’ve got better ways to spend my time online.” But wait. Rank-ordering should matter to all of us in marketing. Improved rank-ordering translates directly into better prospect selection and improved ROMI.

With this in mind, I went on a quest to identify the ways that rank-ordering can be assessed. I was familiar with the Gini Index, “lift indices,” Pearson Chi-square tests, ROC’s, and Brier scoring. After a quick search on Google (the reference librarian of bloggers), I also found references to KS separation, and the divergence index.

In light of the diversity in measures available, how should an analyst validate a new model? A 2004 forum on modeling hosted by the Wharton School concluded “rather than establishing some arbitrary statistical criteria [...], the central question for validation is whether the model is working as intended.” For the entire report, please visit http://fic.wharto...onf%20Summary.pdf In 2006, a researcher with the US Treasury concluded that “no single test is statistically powerful enough to be sufficient.” To view the complete presentation, please visit http://www.occ.tr...ession5_Hasan.pdf

For non-analysts, a takeaway from this discussion is to require your vendors to consider several measures of rank-ordering during model validation. As an analyst, I intend to take my own advice and consider multiple validation metrics in my modeling engagements.

Post a Comment	Private Discussion	Share/Email This

Posted by on 04/22/2008 1:47 PM

Comments (0)

Post a Comment

Comments are moderated and may be edited or removed at the discretion of the folks at iKnowtion.

Name:

Email Address:

URL (optional):

Remember personal info?

Comments: (You may use HTML tags for style)

Please verify the security code:


	RECENT POSTS

	Show Me the Lift - Lift Modeling Explained

	Big Data and the Cable Industry: Part 1

	Linking Customer Experience to Operational and Business Performance

	Why Marketing Optimization?

	Marketing Analytics - Combining Predictive Analytics with Common Business Sense

	Keys to Marketing Success with Zettabytes of Unstructured Data

	Text Analytics: The Next Big Wave of Customer Intelligence

	A Framework for Influencing Customer Experience

	CRM Retargeting Improves Marketing ROI

	Big Data's 5 Missing Ingredients

	Early Customer Engagement Drives Future Revenue


	ARCHIVES

	May 2013
	April 2013
	February 2013
	January 2013
	2012
	2011
	2010
	2009
	2008
	2007


	ABOUT iKNOWTION

	iKnowtion is a marketing and analytic consultancy that helps Fortune 1000 companies optimize the impact of marketing expenditures from demand spending to customer focused initiatives. iKnowtion's industry knowledge and expertise, combined with a comprehensive and proven analytic approach provides Global leaders in automotive, financial services, retail and consumer goods businesses with the marketing insights and analytical engines required to dramatically improve business results. Founded in 1999, the company is headquartered in Burlington, Massachusetts. To learn more about us, please visit www.iknowtion.com.


	ABOUT Wiseguys

	With so much change going on in the field of marketing, we felt that it was high time we try to stimulate a dialog that focuses on the significant transition taking place within the marketing sciences area of many companies. Most experienced marketers agree that new tools and approaches are needed to help allocate and measure marketing resources more effectively. If you're trying to tackle these kinds of issues in your company, join us at Wise Guys - we'd like to hear from you.


	CONTRIBUTORS

	Rafael Bradley

	Bill Duffy

	Susanne Hensel-Goodwin

	Ken Howes

	Don Peppers and Martha Rogers, Ph.D.

	Mike McGuirk

	Geoff Miller

	Michael Moscaritolo

	Marcy Riordan

	Don Ryan

	Niren Sirohi, Ph.D.


	iKnowtion, 25 Burlington Mall Road, Suite 409, Burlington, MA 01803 Tel 781-494-9989 Fax 781-494-9766 E-mail inquiry@iknowtion.com

	Copyright 2010, iKnowtion LLC. All rights reserved. Terms of Use \| Sitemap.