24 Nov 2010 - permalink
The last decade has been about data. Computers and the internet has made available unfathomable amounts of data on just about anything (albeit a lot of it of questionable quality). More than 10 years ago, Google changed how we search through a very small part of it by understanding the importance of statistical analysis. They called it PageRank.
Companies like Google, Facebook and LinkedIn build their business around collecting data about their users and selling it to their customers. But it’s not the data itself, it’s the relations, the correlations hidden within the terabytes. It is visible every time they “magically” suggest other people we might know or when Amazon displays a different book, equally interesting. iTunes Genius or Netflix works similarly.
Like the beauty of fractals, which the late Benoit Mandelbrot helped describe, and which we cannot comprehend without the billions of calculations possible on computers, we need help making sense of the seemingly disconnected bits of information that is everywhere.
Hal Varian, chief economist at Google put it like this:
I keep saying that the sexy job in the next 10 years will be statisticians. And I’m not kidding.
Rockstar statistician Hans Rosling has made the important point that it is not data we lack. It is understanding it. His remarkable appearances on TED made it clear that our beliefs of the world can be very far from the truth (YouTube list). He founded Gapminder to visualize complex or large datasets in order to help us understand – and brilliantly so.
Two years ago Wired published a story called The End of Theory argues how data is changing the scientific method. We no longer depend on the idea for a theory. Rather, computers will be programmed to look for patterns in the sea of data and help us find correlations. Statistics is increasingly becoming the foundation of knowledge. From data visualizations in newspapers to research in medicine and physics, data science is coming.
Google recently released Google Refine, based on software by Freebase, that makes it easy to work with messy data, clean it up and convert to a veriety of formats. Good data is the basis for useful statistics and Refine makes it a lot easier than juggling regular expressions.
There are now several sources online (IBM’s Many Eyes, Gapminder and Freebase to name a few), each with huge datasets on diverse subjects, ready for analysis. Around the world, governments are opening their databases to the public, allowing anyone to find new relations or make it accessible to a greater audience.
Understanding the data we have will help us understand the world and statistics, data parsing and communication is how to do it. To end with a quote from Hans Rosling:
The seemingly impossible is possible. We can have a good world.
23 Nov 2010 - permalink
Jeffery Kluger for Time:
One by one, the berms we’ve built between ourselves and the beasts are being washed away. Humans are the only animals that use tools, we used to say. But what about the birds and apes that we now know do as well? Humans are the only ones who are empathic and generous, then. But what about the monkeys that practice charity and the elephants that mourn their dead? Humans are the only ones who experience joy and a knowledge of the future. But what about the U.K. study just last month showing that pigs raised in comfortable environments exhibit optimism, moving expectantly toward a new sound instead of retreating warily from it? And as for humans as the only beasts with language? Kanzi himself could tell you that’s not true.
It’s long overdue that we start viewing animals in a manner consistent with what we know. The biggest difference between them and us may well be the arrogance with which we treat them.
18 Nov 2010 - permalink
Warren Buffett:
When the crisis struck, I felt you would understand the role you had to play. But you’ve never been known for speed, and in a meltdown minutes matter. I worried whether the barrage of shattering surprises would disorient you. You would have to improvise solutions on the run, stretch legal boundaries and avoid slowdowns, like Congressional hearings and studies. You would also need to get turf-conscious departments to work together in mounting your counterattack. The challenge was huge, and many people thought you were not up to it.
Well, Uncle Sam, you delivered. People will second-guess your specific decisions; you can always count on that. But just as there is a fog of war, there is a fog of panic — and, overall, your actions were remarkably effective.
17 Nov 2010 - permalink
Yesterday Facebook announced Messages, a mix between email, chat and SMS. The attempt at redefining communication is a somewhat similar thought that brought Google Buzz (which didn’t get the adoption they hoped for).
Ray Sun explains what is wrong with this approach:
[They] designed this feature to abstract away texts vs. email vs. chat. But 99% of real humans don’t care about this, and won’t experience his “relief” because this was never a problem in the first place. Teens always text each other, because they always have their cell phones & this is the way to reach them. And to reach your grandma, you use email. Simple. No mixing up grandma with your girlfriend.
With +500 million users, Facebook had every opportunity to grab a huge piece of email, but they seem to have overcomplicated things.
With growing privacy concerns, giving Facebook even greater access to your communication seems like a really bad idea.
10 Nov 2010 - permalink
Matt Mullenweg on shipping and shipping early:
Usage is like oxygen for ideas. You can never fully anticipate how an audience is going to react to something you’ve created until it’s out there. That means every moment you’re working on something without it being in the public it’s actually dying, deprived of the oxygen of the real world. It’s even worse because development doesn’t happen in a vacuum — if you have a halfway decent idea, you can be sure that there are two or three teams somewhere in the world that independently came up with it and are working on the same thing, or something you haven’t even imagined that disrupts the market you’re working in.
Saying “enough” can be very difficult