Comments for Big Data Solutions | Scale Unlimited

Comment on Public Datasets by kkrugler

kkrugler — Fri, 13 May 2011 20:33:48 +0000

Hi Arjun,

For valid domains, I would suggest using the Alexa and Quantcast top 1M domain lists.

I’ve added you to the list of people to update when our dataset is ready.

Regards,

– Ken

Comment on Public Datasets by Arjun

Arjun — Fri, 13 May 2011 06:07:18 +0000

Hi,

I am pursuing Research on Web Mining, actually for research i need Date Set of available Web Domains (website addresses updated till 2011/10).

Please reply to my mail id that where could i get it?

Thanking you,

Regards,,,
Arjun.

Comment on Announcing the Public Terabyte Dataset project by kkrugler

kkrugler — Thu, 17 Mar 2011 00:12:03 +0000

Hi Ian,

Unfortunately the crawl is still tied to a pending release of some new functionality by Amazon, so we’re still in a (very long) holding pattern. But there’s some light at the end of the tunnel…

– Ken

Comment on Announcing the Public Terabyte Dataset project by Ian Upright

Ian Upright — Fri, 11 Mar 2011 07:27:12 +0000

Any news on the status of this crawl? Any update on when might it be available? Thanks..

Comment on First Sample of Public Terabyte Dataset by kkrugler

kkrugler — Mon, 18 Oct 2010 23:17:37 +0000

Hi anonymous,

My mistake, the “previous comments” reference was for this page: http://bixolabs.com/datasets/public-terabyte-dataset-project/

As far as restrictions on commercial use – that’s going to be up to Amazon’s lawyers, but in general your use of this publicly available web data is subject to the same conditions as if you crawled it yourself. Which means respecting copyrights, removal of data upon notice by the owner, etc. all apply.

Regards,

– Ken

Comment on First Sample of Public Terabyte Dataset by Anonymous

Anonymous — Mon, 18 Oct 2010 22:32:06 +0000

Thanks. But, I cant find your comment where you explain the reason behind the delay. Anyways, I am desperately waiting for this data. Will there be any restrictions on using this data for commercial purpose when it is released?

Comment on First Sample of Public Terabyte Dataset by kkrugler

kkrugler — Mon, 18 Oct 2010 16:54:56 +0000

Hi anonymous,

The actual dataset hasn’t been released yet – see previous comments as to why, and my best guess re timing.

Regards,

– Ken

Comment on First Sample of Public Terabyte Dataset by anonymous

anonymous — Mon, 18 Oct 2010 03:15:12 +0000

Can someone tell me how to access actual dataset and not the sample file?

Comment on Public Terabyte Dataset Project by Joseph Turian

Joseph Turian — Sat, 09 Oct 2010 15:59:07 +0000

Ken,

Great! I am most excited to see the dataset when it is available. Let me know if you would like me to write followup posts to help you announce it.

I would also be interested in seeing a writeup of the technical challenges you encountered, and how you resolved them. But I’m most eager to play with the actual data!

Comment on Public Terabyte Dataset Project by kkrugler

kkrugler — Wed, 15 Sep 2010 19:54:18 +0000

Hi Joseph,

I’d love to release the dataset too

We’d run into a cost issue using AWS’s SimpleDB for many links (e.g. 1 billion), so had to revert back to storing the crawl state (aka CrawlDB) in Hadoop SequenceFiles. That, plus wanting to use spot instances for better pricing meant re-working some of the (apparently abandoned) hadoop-ec2 scripts that come with Hadoop.

I think we should have a dataset (maybe not the full terabyte, but big) ready in a month, since we’ve dealt with the above two issues.

Regards,

– Ken