80legs Subscription Plans and Free Web-Crawling

We have just updated 80legs with some exciting new changes.  Starting today, the 80legs service will be divided into 3 tiers: Basic, Plus and Premium.  Since the time we've launched, we've noticed that our customer base can be classified into 3 major groups - light, medium and heavy users.  Each of these plans is targeted to each group and designed to fulfill their specific needs.

Here are details on each plan:

Basic Plan:

  • Free to use
  • Normal crawling speed (up to 1 request/second/domain)
  • Access to 80legs Web Portal
  • 1 job running at a time
  • Up to 100K crawled pages per job
  • Low priority in 80legs job queue
  • No recurring jobs allowed

Plus Plan:

  • $99/month + crawling fees
  • Fast crawling speed (up to 5 requests/second/domain)
  • Access to 80legs Web Portal and API
  • Up to 3 jobs running at a time
  • Up to 1M crawled pages per job
  • Normal priority in 80legs job queue
  • Recurring jobs allowed

Premium Plan:

  • $299/month + crawling fees
  • Ultra-fast crawling speed (up to 10 requests/second/domain)
  • Access to 80legs Web Portal and API
  • Up to 5 jobs running at a time
  • Up to 10M crawled pages per job
  • Preferred priority in 80legs job queue
  • Recurring jobs allowed

Existing users can sign up for a plan by going to the new Subscription section in the 80legs Web Portal, where there are complete details and instructions on signing up for a plan.

We're really excited about these changes.  Of course, the Basic Plan now enables completely free web-crawling, which until today has been completely unheard of.  The Plus and Premium Plans give heavier users the ability to set up and run more intensive crawls.

If any of our users have questions about the changes, please contact us or submit a tickets.  We're always happy to hear from you!

Comments (24)

Dec 20, 2009
API-user said...
"We’re really excited about these changes."
Users, that want to try out your API and then decide, if they want to pay 99$/month, are of cause excited too! Now they do not have this question at all, because they simply can't access your API without paying 99$! Great changes!
Dec 21, 2009
API. said...
It basically is 99 Dollar more expensive for API users and there is a limit on crawls per job now. Why punish your most important users, and reward the clients you don't make anything from with a Free plan? Without wanting to be offensive, honestly this is a much less attractive offering than before.

Wasn't the goal of 80legs to make crawling accessible? With a 99 Dollar subscription for using the API, this is not the case anymore.

Dec 21, 2009
Shion Deysarkar said...
The Plus and Premium plans include more benefits than just API access, such as increased crawling rates, ability to return more data, etc.

That said, we have been getting some feedback on the API issue, and are looking into making a limited version of the API available to Basic users.

Dec 22, 2009
80Legs, 50k Computers and a Web Crawler | Tech Startups said...
[...] If you’re interested, the free Basic specs are below. Plus and Premium are listed on their blog. [...]
Dec 23, 2009
80Legs Inc. offers free crawling service as part of the Subscription Plans « said...
[...] Read more at 80Legs Inc official blog. [...]
Dec 23, 2009
john said...
I am trying to contact you (80legs) through your contact page, as I have a question about the terms of use agreement. But in order to leave a comment, I have to sign in, and in order to sign in I have to agree to the terms of use agreement. Kind of a chicken and egg thing.

Can you email me via the address supplied with this comment?
Thanks,
John

Jan 01, 2010
venkat said...
How do you arrive at crawling fee?
Jan 03, 2010
Shion Deysarkar said...
If you're asking how we came up with the numbers for the prices, it's mostly based on our analysis of demand for the crawling and the value we've placed on the tools the plans offer.

If you're asking about what the price for the crawling fees are, they are $2.00 per million pages crawled and $0.03 per CPU-hour used.

Jan 19, 2010
James Simmons said...
Oh wow. I just did a 10 million page crawl for like what, $25? Now if I want to do that exact same thing again I'll have to pay $300/mo ON TOP of the crawling fees? This is terrible, you've sank my battleship. I learned Java to use you guys for Pete's sake. Do you know what that's like to a Python developer?

Madness.

Jan 24, 2010
CondesendingButRight said...
@James Simmons Jesus you're whiney. If learning Java was hard you cant be that good as a developer. I had to learn both Java and Python - it was not hard.
Jan 25, 2010
maroc said...
Existing users can sign up for a plan by going to the new Subscription section in the 80legs Web Portal, where there are complete details and instructions on signing up for a plan.
Feb 10, 2010
James Simmons said...
@CondesendingButRight: I find it hard to believe that anyone not working for 80legs would be so upset over my comment to anonymously insult me. Or are you really that pissed that I despise Java? Hard to believe, then again, you're a Java developer.

Comments on this blog are moderated and do not appear until an 80legs employee approves them. Perhaps they need to learn discretion, or perhaps they think future customers will see this blog post and say, "wow I really want to use their service."

PS: I'm posting this comment with full details visible, and I'm certainly not afraid to do so.

Feb 10, 2010
Shion Deysarkar said...
1. Comments are automatically posted but we can delete them if we want.

2. We generally believe in letting people express what they want.

@James: We realize that our new pricing is not suited to everyone, but we're sticking with it as it's helped our operations and business in a number of ways.

Feb 11, 2010
James Simmons said...
Thanks for the quick response Shion. I can certainly understand that, and I look forward to continue using your service.
Feb 25, 2010
NotHappy said...
Is there a plan to stop your spider from attacking my web site with hundreds of sessions per day from hundreds of different IP address?

You're the small web site owner's worst nightmare.. you're the plague!

I have 57 sessions on my site now, 10 from real users and 47 from your spider... thats not cool!

Feb 25, 2010
Shion said...
We provide several options for a webmaster to stop their site from crawling their site: http://80legs.pbworks.com/FAQ#Webmasters
Mar 09, 2010
Mindy Franklin said...
Please forward to the President or VP of Marketing:

I am writing to introduce myself and SS | PR.

We have a deep Rolodex of editorial contacts with trades and national magazines in the mainstream media. My firm has been in business for over 25 years and has strong editorial relationships. Our agency can be instrumental in generating features, industry trend stories, etc.

If you would like to talk or schedule a meeting please let me know.

Regards,
Mindy

Apr 03, 2010
GWIZIT said...
Reply Too:NotHappy

Yes it's the plague!

Shion says they provide options to stop their spider: http://80legs.pbworks.com/FAQ#Webmasters

Yet - I have done exactly what above link says, and it has encouraged even more spider activity.

Getting attacked by 008/0.83; http://www.80legs.com/spider.html from all angles - What a JOKE!

Apr 03, 2010
Shion Deysarkar said...
Gwizit,

It takes some time for the crawl requests to go down due to the distributed nature of our system. We have internally blocked crawl requests as well.

- Shion

Apr 03, 2010
algdfa said...
Is there a plan to stop your spider from attacking my web site with hundreds of sessions per day from hundreds of different IP address?

You’re the small web site owner’s worst nightmare.. you’re the plague!

I have 57 sessions on my site now, 10 from real users and 47 from your spider… thats not cool!

Apr 06, 2010
GWIZIT said...
Appreciate the reply & placing internal block crawl requests.

I probably wouldn't mind having more spider activity if my midget server could handle it.

O-Well ... Beats paying for hosting & alot more challenging this way.

Thanks Again

Apr 12, 2010
edithann said...
I don't want you anywhere near my website. We have you blocked in our robots.txt file but you choose to ignore that. Get off my website and stay off. www.northcountrygazette.org

Host: 96.246.238.92
/robots.txt
Http Code: 200 Date: Apr 12 09:15:47 Http Version: HTTP/1.1 Size in Bytes: 598
Referer: -
Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620





/2008/08/14/security_blackout/
Http Code: 200 Date: Apr 12 09:15:47 Http Version: HTTP/1.1 Size in Bytes: 73463
Referer: -
Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620

Apr 12, 2010
Shion Deysarkar said...
There was a problem with our block for your site; it should be fixed now.
Apr 22, 2010
Marsha Friedman said...
Interesting back and forth. Think we will skip this one.

Leave a comment...