Scientists use big data analysis in order to search for patterns that are hidden or buried that are essential in influencing a predicted outcome. These "patterns" to watch out for apparently also require human intuition as well.
Now, researchers from MIT (Massachusetts Institute of Technology) are now carrying out plans to eliminate the need for human intuition when it comes to big data analysis and instead, rely on digital computing solely, to filter through all the data for predictive patterns.
This big challenge in big data analysis will be performed by MIT researchers with the help of the Computer Science and Artificial Intelligence Laboratory (CSAIL) using the prototype software system known as "Data Science Machine".
To test the machine's ability, scientists made this as an entry to three science competitions where the prototype already won 615 times over 906 human teams. During the first two competitions, the Data Science Machine's predictions were 94 percent and 96 percent as accurate compared to the other successful submissions.
On the third competition, the prototype's prediction was 87 percent accurate. The human teams went through months of analysis to get the patterns right but the Data Science Machine's predictions were just as accurate and the results were revealed in just two to 12 hours.
According to Max Canter whose master thesis became the basis for the Data Science Machine, this machine is considered to be a natural complement to human intelligence due to the fact that there's much data to be analyzed, providing a first step to any process and solution.
As the name implies, big data analysis involves a massive and multitasking network. Much of its system is based on automatic algorithms however, humans are still crucial to search for features that will reveal these secret patterns. This human intuition is important since it allows for a visualization of the predicted result where data can support the desired outcome.
According to Kalyan Veeramachaneni who is a CSAIL research scientist from the CSAIL's Anyscale Learning for All movement, this machine learning strategy can be utilized to provide solutions for big data analysis like determining how a wind powered farm can generate a certain amount of energy.
These new findings will be presented at the IEEE International Conference on Data Science and Advanced Analytics on October 19 to 21 in France.