Author: Christopher Berry

It’s awesome to watch Pinterest grow. A post at High Scalability reveals just how much they’ve grown. TL;DR: 80 million objects stored in S3 with 410 terabytes of user data, 10x what they had in August. EC2 instances have grown by 3x. 150 EC2 instances in the web tier 90 instances for in-memory caching, which removes database load And a few notes about technology that caused a smile: Written in Python and Django  Hadoop-based Elastic Map Reduce is used for data analysis and costs only a few hundred dollars a month One of the fastest growing sites in history. Sites AWS for making it possible to handle 18 million visitors in March, a 50% increase from the previous month, with[…]

It’s the Victoria Day long weekend in most parts of Canada, and, whereas our colleagues in Moncton and Halifax will be working, many analytics practitioners in Toronto and Vancouver will be playing German board games and drinking beer. One of the most popular of the German board games is “The Settlers of Catan“. If you know the game, skip ahead to the next bold title. If you don’t know the game, read on for a painless summary. Here’s the TL;DR Summary (Too Long; Didn’t Read): The objective of the game is to win 10 victory points before anybody else does. (Keeps everybody in) You earn two victory points for every city you build, 2 for having the longest road, 2[…]

Consider the following two, distilled, points of view: Statement 1: “Big Data Analytics is going to change the way we do business. Sure, a lot of it will be routine “I’m okay!” status updates from sensors, but making sense of the key parts of it, like “help me, I’m failing”, will be extremely useful. Companies that were previously exempt from competing on analytics will be disrupted by new entrants who will compete better, either by being more effective or being more efficient. Big Data Analytics is already having a disruptive impact in marketing, where it never used to before, and is gaining huge traction in medicine. There is reason to believe that Big Data Analytics will cause better decision making[…]

Is the assumption that better data causes better decisions credible? It depends. If evidence to the contrary of an individuals aspiration comes to light, and that individual refuses to update their expectations or aspiration, then even the most pristine, accurate, precise and real time data will fail to change their mind. If evidence to the contrary of an individuals aspiration comes to light, and that individual updates their expectations or aspirations accordingly, then it will be effective at changing their mind. The key element that decides the effectiveness of data is the human. Great data can cause great managers to make better decisions. Great data doesn’t cure ignorance. Maybe the broader commentary on the value of Big Data has more[…]

Where does the assumption that better data causes better decisions come from? I don’t know for sure. But I can point to two possible sources: The enlightenment and the scientific revolution Robert McNamara and the Whiz Kid movement The entire scientific method is predicated on data. A hypothesis is either accepted as truth or rejected as false based on the data. There is no other arbiter. Faith or strength of opinion has nothing to do with it. The data decides. The assumption that greater knowledge causes greater outcomes flows from that fact. That, if you’re honest about being wrong, that everybody benefits. (Disturbingly, that trend may be reversing in The West, as negative findings have been disappearing from most disciplines.)[…]

Yesterday, I summarized two statements and asked four questions about Big Data Analytics. Today, we knock off the first question. Did anything go wrong in Business Intelligence generally and Web Analytics specifically? The extreme skeptics towards Big Data Analytics argue that Business Intelligence failed. Others point to cases of success. Who’s right? Today, we see: Major decisions about pricing, like those recently at Netflix, continue to be made without any analytical support (or, the analytics were completely ignored). Few people, to this day, really understand the numbers they’re looking at. (Few can explain the accurate definition of ‘time spent on site’, for instance). Hundreds of thousands of firms continue to compete just fine without any analytics at all, without any[…]

Consider the following two, distilled, points of view: Statement 1: “Big Data Analytics is going to change the way we do business. Sure, a lot of it will be routine “I’m okay!” status updates from sensors, but making sense of the key parts of it, like “help me, I’m failing”, will be extremely useful. Companies that were previously exempt from competing on analytics will be disrupted by new entrants who will compete better, either by being more effective or being more efficient. Big Data Analytics is already having a disruptive impact in marketing, where it never used to before, and is gaining huge traction in medicine. There is reason to believe that Big Data Analytics will cause better decision making[…]

John Rauser is a data scientist at Amazon. He put forward a perfectly good definition of Big Data at a conference last week. He said that big data is defined as: “Any amount of data that’s too big to be handled by one computer.” That’s a great definition. I like that. Because if it can’t be handled by one computer, it belongs to a special class of problems that are caused when data is distributed and fragmented. That’s great John, thank you. – Bring on the Anti-Hypers It’s great to use terms like ‘bandwagon‘ and ‘data fetish‘. If you cut through the negativity and really strip it down, you’ll see there’s a concern that storage of data is not the[…]

I believe in markets. Last week’s post on ArtScience Groupe’s sparked a great debate about the use of markets in operations. For the most part, firms compete on the free market. Yet, most firms do not take advantage of the market mechanism in their operations. We certainly have methods to measure the efficiency of markets.  We have decades of experience in managing markets.  Everybody understands the risks that free markets pose. Why isn’t the mechanism used more often in operations and operations research? *** I’m Christopher Berry.I tweet about analytics @cjpberryI write at christopherberry.ca