Tuesday, October 22, 2013

Big Data Needs Big Theory

The world of Big Data is here. Sophisticated statistical techniques are available to work with it. But what we also need are nice theories which allow for subtleties that might be revealed through such statistical precision. In physics, one is never doing a regression, as such. One has a rich theoretical structure, and maybe millions of more events or measurements, and the problem is a matter of decision how to make "cuts" on the data, choosing the relevant events, grounded in a theoretical physical understanding of why you are doing that. You may eventually fit a data set, but in general that is at the end of enormous amounts of theorizing and data analysis. You have a bump or a whatever, and you have a good idea of its shape and all the kinds of confounding physics below it.  If all you had was a regression, it won't be physics.