Data Deluge? Upgrade Your Data Filters? No, Upgrade You Data Analysis!

Clay Shirky says welcome the Data Deluge, but improve your Data Filters

Vodpod videos no longer available.


Clay Shirky thinks the problem is filters. That has been the pat answer for as long as there has been data. However, that is not the answer. The answer to accept all the data and analyze it through better visualization. It is better to change the granularity of data and see “the big picture” to see the trends, make a hypothesis, make predictions. We have to learn induction instead of continually resorting to deduction.

There is no bad data. There is a lot of bad analysis.

They Failed, But They Were Not Afraid

gwb

When we think of physicists, the majority of us think of them as geniuses who understand the universe.  But the truth is, they do not.  Physicists, make models to explain and predict phenomena.  They are called “hypotheses”.  If a hypothesis stands up after many experiments it is called a “theory”, but it just a model.

Einstein was not a success.  He had some victories along the way, but he never achieved his goal.  Neither have the quantum physicists over the last eighty years.  They have corroborated many hypothesis that led to their ideas being called theories, but their theories are not proven.  That is because they do not predict all of the phenomena that are observed.  There are exceptions that cannot be explained.  From the first living thing that reacted to reality organisms have been trying to model their universe.  And every model is incomplete up to the physicists we have today.

Religion has become the word for the denial that a theory is incomplete.  Every religion on earth is guilty of it.  Scientists are guilty of it as well.  I am right every time I say that a model is wrong.  Goedel would agree with me.  The only constant appears to be that no theory is constant.  In fact, science requires us to speak in absolutes to leave room for falsifiability.  We have to leave no room for exceptions in order to discover the exception and change our worldview.

That is what a singularity is all about.  We refine our theory until it is cornered and then it realizes its transition into a new reality.  If we succeed in cornering out theory there is a climax.  If we fail in cornering our theory, there is an anti-climax.

Hiroshima was the singularity of the Second World War.  Nagasaki was the follow through to the exhaustion of Imperial Japan.

But Einstein’s life as a whole ended with an anti-climax as have the lives of every physicist who followed him.

Death is the ultimate anti-climax.  But do not be afraid.  No success nor failure is the complete measure of a person.  Accepting, changing strategy and trying again fearlessly is.