Testing out some improvements to our crawling back-end
Some of our users may have noticed we recently lowered the limit on # of pages crawled per job to just 10,000. This is a temporary measure while we test out some major improvements to the crawling back-end. If the tests go well, the limit will be pushed back up to 1 million (probably 10 million, actually) by the end of this week or early next week. We think this will be the last major upgrade to our back-end. It should allow 80legs to more-easily scale into the billions of pages crawled per job.
So please bear with us as we continue to work on the service - thanks so much!!
