When Clouds First Formed

    Tech was moving fast in 2008. While Apple was rolling out its first iPhones, I was already on the trail of a related next big thing — cloud computing.

A handful of young companies were building very big data centers. Some aimed to sell spare time on them; others used them to offer free services.

Amazon had started renting its servers in 2002. It would take years before the service was easy to use, popular and profitable enough to attract rivals.

Google was still thought of mainly as the company that won the Internet search wars, beating out rivals like AltaVista and Yahoo. Before most of us knew what was going on, it launched a menu of free consumer services -- email in 2004, digital maps in 2005 and video through its acquisition of startup YouTube in 2006. 

Initially, the free services didn’t seem all that interesting. We knew they were a way of amassing data, and we knew data was the new oil, the fuel of cloud computing.

With enough data, the big data centers behind cloud computing could make internet search more accurate and faster. And they could analyze trends – like who was buying what -- for their own benefit or to sell to their customers.

On their racks and racks of networked PC server boards, the cloud computing companies ran an emerging class of big-data algorithms, complex mathematical formulas that were closely guarded corporate secrets. Google’s MapReduce was one of the weapons that helped it win the internet search engine wars.

Something that powerful doesn’t stay secret long. Folks at Yahoo, struggling to survive, decided in June 2009 to give away their big data algorithm for free. They called it Hadoop, after a toy stuffed elephant that belonged to a lead developer’s child. The software let Yahoo store and process huge amounts of data across more than 25,000 of its servers, a decent-sized data center in those days.

At the June 2009 event where Hadoop was released to an audience of more than 700 developers, an IBM executive explained its significance.

“Hadoop is setting the context for a new set of apps [the average business] couldn't get access to or were too expensive to write but now have a bright future,” said Rod Smith, a vice president for emerging Internet technologies at IBM.

“We think tools like this will be helpful in collecting and extracting content and letting users run operations on it over and over again,” Smith told me. “But so far this is still cookie dough—it's not fully baked yet,” he said.

Nevertheless, the software was already in use by companies including Facebook, Hulu and News Corp.

Less than six months later, I wrote an article about a Silicon Valley venture capital firm that launched a $100 million Big Data Fund, centered on Hadoop. At a sold-out conference, the VCs said software like Hadoop was gaining traction and applications built on it would emerge quickly.

Some predicted in the next century most companies would ditch their in-house computers and tech staff, just as, in the last century, they stopped paying for systems and specialists to generate their own electricity.

Today, software like Hadoop still has a role to play, but it has taken a back seat to the most powerful big-data algorithm ever discovered – deep learning, aka AI.

Next: Chasing the big-data server