I have seen companies spend large sums of money and invest huge amounts of human capital building and operating big data solutions with very little ROI to show for it. Over the last year or two, IoT has become a frequent board room topic. Business executives have suddenly embraced IoT and are building strategies that promise to transform their companies by leveraging the power of real-time information and machine learning. IoT initiatives are appearing with robust business plans, complete with funding. IoT has become the killer app for big data.
To understand why IoT and big data go hand in hand, we must understand the underlying technologies. . There are four major categories of technology used to implement IoT. At the lowest level is called fog or edge computing. The “fog” is comprised of many types of connected devices and sensors that communicate data in real time. These devices are enabled by a variety of communication technologies and protocols.
In the fog, devices and objects with sensors can determine when a status or condition of an object changes and then make intelligent decisions based on that information. For example, a company managing a fleet of delivery trucks can detect when truck parts are performing suboptimally and schedule these trucks for preventative maintenance long before the vehicle breaks down. These types of real-time decisions greatly reduce maintenance costs and improve the overall delivery performance of the fleet. I like to refer to the data in the fog as small data.
This small data is then transmitted to a company’s datacenter, often run and managed as a virtual datacenter on an Infrastructure as a Service (IaaS) cloud vendor, such as Amazon (AWS) or Google GOOGL +1.14% (GCP). Once the data is brought into the datacenter, it goes through a process of data ingestion, where it’s scrubbed and stored in databases built for handling enormous volumes. This is the sweet spot for big data.