Now Google is embarking on, perhaps, its most ambitious indexing venture yet: indexing countless billions of lines of code as part of the new Google Code Search.
Google Code Search on Google Labs gives users a place to search for publicly accessible source code. It looks like a regular Google Search page but instead of searching Web pages, it's going to search billions of lines of code.
"The two ways that source code lives on the Internet is in archives, things like Zip files, gzip, etc. And then in software-control repositories like SourceForge.net,
Google's code hosting, and other places," Google product manager Tom Stocky told
internetnews.com.
"We'll be crawling all of that."
Google isn't just going to index the Zip archive files. They're actually going to open up the files and index all the individual files within in.
In the case of software-control repositories like CVS and SVN, Google will go into the public access and index the individual files within them.
Google's regular Googlebot crawler is being used to find and identify the Zip files. In the case of software-control repositories, Stocky noted that it's a different kind of crawler that has to access the CVS or SVN server and speak in a different protocol to then get the information back.
The total task is staggering.
Stocky was unable to provide a figure, but he did note the Google Code Search has billions of lines of code.
"We're not getting more specific than that, but it is a significant number," Stocky noted.
Google Code Search will offer users a number of different ways to find the code they are looking for. Users can perform search queries based on software license, programming language and by file name.
"We also support regular expressions, so instead of searching for keywords you can search for patterns of words," Stocky explained. "For people that know how to use regular expressions well you can get really specific search and search over some really obscure stuff."
Google is also launching an API for Code Search as part of the launch. The API will utilize Google's GDATA API format.
At launch there will be no Google AdSense ads on the results pages and the Code Search results are not integrated into the main Google index.
One of the possible uses of Google Code Search is for developers to do searches for their own code and see where people are using it. It may also help to combat plagiarism and software license use infractions.
"If you own code and someone else is posting illegally, there is a process where we can remove it from the index," Stocky noted.
Most of the code indexed by Google Code Search is open source-licensed. Stocky noted that Google doesn't believe that much, if any, is proprietary since it's all posted in public places.
"In the case of CVS and SVN there is a password capability so we believe if someone didn't want it to be seen by the outside world they would either have a password or not post it to a public place," Stocky commented.
Google Code Search is available
here or via the advanced search option from Google.com
Google Launches Code Search 
Linear Mode
