Categories: Tech & Society

Common Crawl – Free Database Of The Entire Web, Competition For Google

We all know Google! Google started off as nothing but a website indexer, built a more efficient algorithm for ranking Web pages and finally built its success on crawling the Web—using software that visits every page in order to build up a vast index of online content.

A nonprofit called Common Crawl is now using its own Web crawler and making a giant copy of the Web that it makes accessible to anyone. Common Crawl supplies a database of over five billion Web pages that can be accessed and analyzed by anyone, in the hope that it will inspire new research or online services.

“The Web represents, as far as I know, the largest accumulation of knowledge, and there’s so much you can build on top,” says entrepreneur Gilad Elbaz, who founded Common Crawl. “But simply doing the huge amount of work that’s necessary to get at all that information is a large blocker; few organizations … have had the resources to do that.”

Elbaz says he noticed around five years ago that researchers with new ideas about how to use Web data felt compelled to take jobs at Google because it was the only place they could test those ideas. He says Common Crawl’s data will make it easier for novel ideas to gain traction, both in the world of startups and in academic research.

Common Crawl has so far indexed more than five billion pages, adding up to 81 terabytes of data, made available through Amazon’s cloud computing service. For about $25 a programmer could set up an account with Amazon and get to work crunching Common Crawl data, says Lisa Green, Common Crawl’s director.

More details at MIT Technology Review.

Prateek Panda

Prateek is the Founder of TheTechPanda. He's passionate about technology startups and entrepreneurship and enjoys speaking to new founders every day. Prateek has also been consistently regarded as one of the top marketing experts in the region.

Recent Posts

Once more with feeling: An ambitious bid to build a digital Disneyland for fans

This entrepreneur, who entered the startup ecosystem when he was only thirteen years old, wants…

6 days ago

From UPI to Web3: How India’s fintech wave is reshaping the future of payments

India’s fintech sector is accelerating at breakneck speed, with digital payments becoming the new default…

2 weeks ago

AI Launches: SaaS, content management, ecommerce, cybersecurity, logistics, entertainment, & investment

The Tech Panda takes a look at recent launches in the superfast field of Artificial…

2 weeks ago

M&A: The art of the deal

The Tech Panda takes a look at recent mergers and acquisitions within various tech ecosystems…

2 weeks ago

AI companion: From comfort to concern, are we flirting with “Addictive Intelligence”?

Strange are the ways in which Artificial Intelligence (AI) has been offering us humans companionship.…

2 weeks ago