When you change a product, how do you know whether the change caused an improvement? An experiment compares people who got the change with people who did not, so you can see the difference fairly. Some product analyst roles, especially at larger fintechs, ask for experiment design and analysis. This lesson is optional, but the thinking behind it will make you a sharper product person even if you never run a formal test.
The basic idea
In a simple experiment, often called an A/B test:
- Group A (the control) sees the current version.
- Group B (the variant) sees the change.
- People are assigned to groups randomly, so the groups are similar.
- You compare one chosen metric between the groups.
Random assignment matters. If you give the new version only to users in Accra, any difference might be about Accra, not your change.
Decide before you look
Before starting, write down:
- The hypothesis. "Suggesting a smaller starting amount will increase the share of savers still depositing after four weeks."
- The metric you will compare.
- The size of difference that would matter to the business.