It should be no surprise that there is growing interest in data analytics and a rise of “post-relational” analytic data stores. Traditional databases work well to load data in, but they are becoming problematic to get answers out when faced with growing scale and complexity. There is a growing problem with Big Data, defined by Adam Jacobs in a recent ACMqueue article, “Pathologies of Big Data.” Big Data is “data whose size forces us to look beyond the tried-and-true methods that are prevalent at that time.” Given our growing data tsunami, many people are looking beyond the tried-and-true methods of relational databases. Billions of rows can easily be stored, but Jacobs describes the pathologies of query scalability, even when asking for basic counts!
You can store data all day, but the value of data is in its query: in its exploitation. I remember first hearing this word, “exploitation”, from the leader of a UK intelligence agency on one of my trips to London. Whether for government or commercial interests, the time to analyze and act is what matters. The word struck me and has remained with me every since. Exploitation time is what matters. Furthermore, the number of queries is increasing faster than the growth of the data tsunami itself. Query time is what matters because response time in most critical, and critical times drive the massive load of many simultaneous requests.
