First, a thread of thought. Second, a brief exhortation. Summary: Chris Broadfoot showed some pretty amazing visualizations he had created using some open data. Mark Hahnel showed Figshare, which aims to help academics make their data open and available. He’s a big part of the open data movement. Flip Kromer, CTO of Infochimps, built the core technology that made sharing that data set possible, earlier on in the day. (And he kicked my ass in a German boardgame). Why I’m optimistic: I see in Chris’ work was the opportunity for the public and decision makers to make very well informed decisions about transportation policy. Relevant. I see in Mark’s work was the opportunity for others to, with greater ease, replicate[…]
Month: March 2012
eMetrics San Francisco is this week, and #measure can expect the usual volume of hashtags and quotes. For those of us at home or in the office, the flow can be pretty annoying. That torrent causes a fairly warped view of what’s really going on. eMetrics is far more than the witty one liners delivered in a BIG way in REAL TIME. There’s a lot of substantive material. A few questions to ask yourself: What is the definition of Big Data? What is the definition of Real Time? Can either help me win? Analysts aren’t alone in feeling like there’s too much data coming at them. Is more really better? More data might not be the right answer 80% of[…]
You may have recently clicked a link leading to this paper by Robert Ghrist on Barcodes. You may have also read a previous post about MINE. And finally, this month I talked about histograms and proceeded to subject you to their importance of seeing the data, again and again and again. TL;DR: Seeing the data helps analysts understand the data. Showing the data alone isn’t explaining the data. The first question, in response to seeing a line on a chart, is “why”? Sure, if the line is going up, I caused that. If it’s going down, that’s the weather’s fault. Fine. Those are great, convenient reasoning, guesses. It’s much harder to assert that a relationship between two things really exists.[…]
There are really big problems in education, health, and energy that could benefit from advanced machine learning techniques made possible by the suite of so-called big data technologies. Why is it possible to solve them now? Why aren’t they solved yet? It’s because technologies for distributed storage and processing, pioneered and open-sourced by companies like Google, are available. Distributed computing systems, like but not restricted to ‘the cloud’, have brought down the cost of such operations. Finally, enough people have spent enough time trying, failing, and succeeding, to be able to use those technologies successfully. In other words, there are very good physical technologies and social technologies that are now in place. Ontario has very huge, centralized, repositories of very[…]