Why data scientists and analysts should keep a journal

If you’re in analytics, you should be keep a journal. Or call it a log or running commentary. This is something you keep in addition to your burn list.

A journal helps you to:

• Remember what you explored

• Exploit more reliably

• Recall inspiration

Some of the best people in analytics are great explorers. They experiment. They examine diversity. The analyze variation. They examine options in the context of systems. They’re great inductive thinkers because they had to learn the hard way to become inductive thinkers. Exploration, on its own, is a very high value activity, but also carries the biggest risk. It’s also the riskiest and the hardest to explain to those who do not explore. Exploration is an attempt to nail it.

There’s also exploitation. Some of the best people in analytics are great exploiters. Exploitation generates reliability. They select methods. They understand tradeoffs. They make choices. They write code to standardize outputs. They drive unity in a message and in a communication style. They seek consistency. They build reliable programs. Exploitation is an attempt to scale it. Exploitation is the destruction of original thought for the sake of heuristic. And it’s awesome. (Though, that terminology is nasty and worthy of review.)

Both activities are essential.

Keeping a log helps you keep track of what you explored and why. It enables you to keep track of what is now to be scaled, and how to scale it. It allows you to make better estimates. In a very real way, it makes you a much better exploiter.

It’s particularly important, if you’re ever to become better at making decisions, to understand what you knew at the time of making that decision. There’s a crucial difference between what you remember thinking at the time, with the benefit of hindsight, and what you objectively knew you knew. Since exploration is happening in tandem with exploitation on most days, you, yourself, have a scrambled perception of what is what. In effect, you’re changing the outcome by measuring it.

(Moreover, a particularly bright mind in comparative analysis once told me: “you might be thinking about something, but you’re not really thinking unless you’re writing it out, then you’re really thinking.”)

The final benefit is the capturing of inspiration. Amazing ideas happen at any time. Indeed, most of my thoughts are just empty desert. Empty, granular, and boring. Do purposely prospect for oases. Linger. Capture the unusual thoughts you’re having. Even if they’re not immediately salient, you’ll be glad that you did. And the future you will be surprised with what you thought at the time.

In sum – keep an analytics journal. It’s a good habit.

***

I’m Christopher Berry.
I tweet about analytics @cjpberry
I write at christopherberry.ca