Public Terabyte Dataset Project
Public Terabyte Dataset Project
This page has more details on the Public Terabyte Dataset project, which was recently announced at the ACM data mining unconference.
- The data comes from a crawl of 50-200M pages from the
100K1 million top (by US-based traffic) domains. - The crawl is done by a custom Bixo workflow created by Scale Unlimited, built on top of Cascading/Hadoop and running in EC2 using Amazon’s Elastic MapReduce service.
- We’ll be trying hard to avoid spam/adult content, though getting totally clean results is of course impossible.
- We honor the robots nofollow and various “no archive” HTML meta tags, and we’ll comply promptly with requests by web site hosters to remove any of their content from the datasets. Note that the rel=”nofollow” attribute on links themselves does not mean that the link shouldn’t be followed, but rather that the link shouldn’t be used when calculating a “PageRank” score.
- The resulting data will be stored as compressed
warcAvro files in S3. - Hosting for the dataset is being provided by Amazon.
- Access to the data is free, assuming you’re running code in EC2.
- The code used to run the crawl, as well as code to access the crawl data, will be available at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263.
There’s a form where you can request information and provide input on the crawl.
13 Responses
Post a comment
How did you get the top 100K websites by traffic?
Is the datasets all pages from those websites?
How did you get Amazon to host the data?
What do you view as the value of this dataset?
Hi Dan,
1. Top sites by traffic come from Alexa web services API.
2. The dataset focuses on these sites, but expands to include others.
3. Amazon is helping because they are interested in useful public datasets and examples of effectively using EMR.
4. The dataset has highest value for people working on text processing algorithms, and for doing performance baselining/optimizations.
– Ken
Alexa and quantcast both have a free “top 1 million” list that is updated daily.
Hi Joel,
Excellent input, thanks! The Alexa web service API offers a bit more information (e.g. is it “adult”), but it feels out of date (e.g. no walmart.com data). By merging these together, I’ll have a much better see list.
– Ken
premiumdrops.com is also offering copies of .com/.net/.org zones if you want to constitue a wide seed list.
@elhoim – I hadn’t found the domain lists on premiumdrops.com …good stuff.
Sounds like you’ve been poking around this space a bit in the past
– Ken
What is the status of the crawl? When will a sample be released?
This dataset will make a lot of people very happy.
Hi Joseph,
I’d love to release the dataset too
We’d run into a cost issue using AWS’s SimpleDB for many links (e.g. 1 billion), so had to revert back to storing the crawl state (aka CrawlDB) in Hadoop SequenceFiles. That, plus wanting to use spot instances for better pricing meant re-working some of the (apparently abandoned) hadoop-ec2 scripts that come with Hadoop.
I think we should have a dataset (maybe not the full terabyte, but big) ready in a month, since we’ve dealt with the above two issues.
Regards,
– Ken
Ken,
Great! I am most excited to see the dataset when it is available. Let me know if you would like me to write followup posts to help you announce it.
I would also be interested in seeing a writeup of the technical challenges you encountered, and how you resolved them. But I’m most eager to play with the actual data!