Today, at work, I had to investigate some of the discrepancies behind the results of an A/B test. I was stuck on the problem for several hours, looking at it from different angles and bouncing ideas off the data scientist who was challenging the results of the experiment.
After a brief walk, I decided to address this investigation in a more structured and scientific way. I was thinking about clarifying the problem statement as precisely as possible, and listing down all the possible causes that could lead to the observed discrepancies. So I decided to write a document, with the idea of sharing it later with the data scientist and the rest of the team, to get some fresh eyes on that problem.
After describing the context, and right before addressing the possible causes, I decided to write a section that contained some information about how the A/B test was setup, and how the platform that runs it works.
The moment I was done writing these down, I realised the solution was lying in front of me. The solution to the problem that was bugging me for an entire day, had literally just emerged after 10 min of writing something down.
This is a tip for solving any problem: write down an investigation document. Writing down allows structuring, and organising ideas. Additionally a document that is meant to be shared, requires from its author a certain level of rigour and conciseness. All of these are a very important for problem solving.
Finally, as a nice side effect, a shared document is an invitation for collaboration.