A/B Testing, Feature Flipping and going too far - Pivotal Labs

"You risk building a broken product when you try to subcompose the UX." says @gsiener at

A/B testing is probably not worth your time. When you start hooking metrics up to your product, the feedback is addictive. All of a sudden you’ve got lots of actionable data and you’re tacking validation goals onto feature stories. This is great, but I implore you to not take it too far. You’ve probably read stories proclaiming how effective A/B testing is for Twitter , 37 Signals and even the Obama campaign . There’s no shortage of third-party services that boast one-click setup via javascript snippet and claim to deliver a double digit boost to your bottom line. I talked in my last post about the concept of opportunity cost and it’s with this lens that I view excessive testing and experimentation. If you are still in growth mode, you’re still figuring out what A is. There’s too much at stake (and too few developer cycles) to distract yourself with subtle experiments that...

Full article: http://pivotallabs.com/ab-testing-feature-flipping-and-go...