Big Data Reports Q1 2013

It’s been a long time since 2010 when I first read the economist magazine special report for Managing Information
 That was the first time I got really interested in the “new field” of Big Data and until now I have tried to watched all the news and new directions very closely. This is why I believed at the time that Big Data seemed to be something more than a new buzz. Nowadays, only for the first quarter of 2013, I’ve read four interested reports related with Big Data. Some of them are more technical and some are more focused on business strategy.
 Starting with McKinsley Quartely which was published on March 2013, it provided some useful conclusions about how a company can make a start with big data.We can summarize McKinsley’s advice in three simple steps i) Collection and integration extraordinary volumes of new data ii) Apply advanced analytics models for optimise operation and  protect outcomes and iii) Creating tools to translate the output of models into tangible business actions. I totally agree with the third step. More or less, it is easy to collect and build a big data set  and apply data mining or machine learning algorithms on it. These actions are useless if you can not focus on translating these results to every day business practice. Data scientists should keep in mind to provide easy methods or tools for their customers to “consume” the data and understand new patterns in order to improve their company’s efficiency. 

In Gather 2013 BI Magic Quadrant report, you can find the analysis of 22 vendors active in BI domain. I focus on two companies which, according to Gather, are the winners in ”ability to execute” and “completeness of vision” categories. Microsoft wins in category “ability to execute”. The company has three tools PowerPivot,  SQL Server Analysis Services and Power View.

A lot of people are familiar with MS Excel and Microsoft used the Excel’s popularity with the Power Pivot. It is rather easy for anyone to analyse a Big Data set using Power Pivot. Microsoft called this “self service business intelligence”. Another important aspect for the company is the Hadoop Distribution by Microsoft which is available on Windows Azure Platform and it is called HDInsight. IBM leads in category “completeness of vision”. IBM is one of the first companies which offered BI solutions to their costumers and already holds a large range of products in its portfolio. It is worth mentioning the InfoSphere Streams and the new products like Cognos Insight and Cognos Express. 

I recommended you to read this report in order to have a clear view of BI ecosystem of tools and solutions today. There you can find companies like QlinkView which provide new capabilities that enable users to build interactive visualizations for discovering new insights.

The third report is the Big Data Vendor Revenue and Market Forecast 2012-2017  from Wikibon. You can find a table of more than 60 vendors Wikibon tracked and/or modelled during the 2012 for the Big Data revenue report. The total Big Data revenue for the 2012 is $11,448M which was divided in three categories i) 40% of the total revenue refers to Big Data Hardware Revenue ii) 21% is for Big Data Software Revenue and iii) 39% is for Big Data Services Revenue. The Big Data market forecast yearly revenue for the 2017 is $47.8B. Wikibon states that these increases were because of a number of certain factors.

  • “An increased awareness of the benefits of Big Data as it is applied to industries beyond the Web, most notably financial services, pharmaceuticals, and retail;
  • The maturation of Big Data software such as Hadoop, NoSQL data stores, in-memory analytic engines, and massively parallel processing analytic databases;
  • Increasingly sophisticated professional services and practices that assist enterprises in practically applying Big Data hardware and software to business use cases;
  • Increased investment in Big Data infrastructure by massive Web properties – most notable Google, Facebook, and Amazon – and government agencies for intelligence and counter-terrorism purposes.”  
Source:http://wikibon.org

Trying to summarize the results of the three reports above, it is clear that the Big Data market share is rapidly growing. It is important that there are now tools available to help users like business analysts or managers with  pre-packaged content and applications to  ”consume” the data, find insights and move on taking data driven decisions for their organisations. Also, if you intend to design a big data service for a niche market and want to be successful, you must keep in mind that the end user somehow must be able to to translate the output of models into tangible business actions. 

Quirrel and MongoDB

Months ago we started a big data project. We decided that a 10gens MongoDB is going to be our primary database.

The next step is to search and evaluate different tools and technologies in order to analyse the data set. Precog is one platform  that I really feel the need to examine more closely to evaluate whether it fits our current needs for data analysis or not. One of Precog’s products is the Labcoat, a data analysis tool that uses Quirrel, a declarative query language designed to perform analytics and statistics on large or heterogeneous datasets.  Well I have some previous experience in using R for data analysis, but right now I am not able to confirm the statement that the Quirrel is the “R for Big data”. My first impressions though are very  good.  You can easily read and understand others code and there are already a number of available standard functions to use. It includes native support for json and was built to natively work with the JSON query language. That is why Quirrel is able to do analytics on json datasets easily because it is set-oriented, which means that all data are represented as sets of JSON values. 
The reasons above should make happy the  MongoDB users and maybe in the near future Quirrel will become the first choice for advanced analytics. 
At the moment  I am trying to master the quirrel and analyze our dataset in order to discover some interesting insights. 
If you want to give a try on Quirrel you can download precog for MongoDB  . It is free. It takes only few minutes to install and setup the tool. When it starts running on your browser, it automatically locates your MongoDB instance along with the available databases and collections. And then, you are ready to run some Quirrel queries on your MongoDB data.
Unfortunately, as it usually happens with every new technology, available resources and tutorials are few. The following list might help you to start your journey in Quirrel.

A New Beginning

I’ve come a long way with much work and also the opportunity to work on some very interesting projects. Through this blog I will have the chance to share some of my thoughts and some interesting conclusions. Stay tuned because I am ready to share my experiences while I try to keep a certain schedule for new posts.
My interest is still in big data which proved to be not just another buzz word in the tech ecosystem but also a whole new field in computing that can enhance our daily lives. For the enterprises,  it is a significant competitive advantage. Big Data quickly became a new “mantra” for the new startup wave which led to the birth of several startups that nowadays have the ability to focus on niche markets, especially in education and healthcare.
The possibilities that exist through cloud computing help in establishing and developing new concepts with the minimum costs. A few years ago that was not possible and any similar attempt required big financial investments in infrastructure e.g. data center.
Current environment looks better than ever with several examples of companies which have managed to be funded with enough money by venture capitals and it seems that this desire for new ideas and applications of big data will easily find its way. Several reports published in March 2013 confirm all of the above. More on these reports in my next post.

Computer science remains one of the fastest growing fields, with the U.S. Bureau of Labor Statistics forecasting almost 20 percent increases in computing-related jobs by 2020. While myriad efforts at the national, state and local levels have contributed to four years of sustained growth in undergraduate computer science programs, accelerated growth and diversification remains critical to cultivating the next generation of technology industry leaders.

“Computing is the world’s newest great science. Yet, even though enrollments in U.S. computer science programs are on a four-year rise, it’s still not enough to satisfy the workforce demands of a technology-driven global economy.
U.S. Bolsters National Push to Expand Computing Education (via courtenaybird)

(via emergentfutures)

Source: cc.gatech.edu