Data Flood

Now that we have so much data, we need tools to manage and understand it.  This reminds me of my eternal quest for a better to-do list that can manage all my tasks. There is just too much data for traditional tools to manage.

In a speech given just a few weeks before he was lost at sea off the California coast in January 2007, Jim Gray, a database software pioneer and a Microsoft researcher, sketched out an argument that computing was fundamentally transforming the practice of science.

Dr. Gray called the shift a “fourth paradigm.” The first three paradigms were experimental, theoretical and, more recently, computational science. He explained this paradigm as an evolving era in which an “exaflood” of observational data was threatening to overwhelm scientists. The only way to cope with it, he argued, was a new generation of scientific computing tools to manage, visualize and analyze the data flood.

…

“We have access to too much data now to understand what’s going on,” Dr. Horvitz said….

…

In his chapter, “I Have Seen the Paradigm Shift, and It Is Us,” John Wilbanks, the director of Science Commons, a nonprofit organization promoting the sharing of scientific information, argues for a more nuanced view of data explosion.

“Data is not sweeping away the old reality,” he writes. “Data is simply placing a set of burdens on the methods and the social habits we use to deal with and communicate our empiricism and our theory.”

From A Deluge of Data Shapes a New Era in Computing , Book: The Fourth Paradigm: Data-Intensive Scientific Discovery