- Business
- CBS Interactive
- Relationship
- In-houseExperimentation Lead
- Industry
- Streaming media
An experimentation program, built from scratch.
A testing practice the team could run on its own, and a conversion forecast accurate enough for finance to plan with.
- 3–8%MAPE on the conversion forecast
- 0 → 1A/B testing program, team trained
This was in-house work Shawn led as Experimentation Lead at CBS Interactive.
The challenge
The streaming platform was making subscription and content decisions without a repeatable testing practice or a forecast finance could plan around.
What was done
- 01
Built the A/B testing program, trained analysts and product managers to run it, and went looking for optimization opportunities instead of waiting for requests. The team started running tests faster.
- 02
Built a conversion-rate forecast model accurate to within 3–8% MAPE, used for financial forecasting and planning.
- 03
Identified the signals that predict whether a free trial turns into a subscription. Marketing used them to improve acquisition and retention.
- 04
Improved content recommendations on the app homepage through behavioral analysis, which increased engagement and content discovery.
The result
A testing practice the team could keep running, and a forecast accurate enough for finance to plan with.
When finance builds its plan on your forecast, the forecast has to be right.