Get data from massive web crawling system

Amazon owns Alexa, and today Alexa is announcing a web service that allows developers to make special requests for content during Alexa\’s web crawls. The Wall Street Journal article gives the example of asking the crawler to help you identify new music files or images that have recently been posted to the web.

I\’m interested in using the Alexa service to identify and track the number of links coming in to sites I am promoting (or competing with). The service is free for 10,000 requests a month and then there is a charge of $0.15 for each additional 1,000 requests, according to

Last month I met the evangelist from Amazon\’s six other web services at a BYU event, including the amazing Mechanical Turk.

It\’s becoming more difficult to keep track of all the web services and APIs coming out from the major internet companies.

Who knows the best blog or web site that tracks all of these?

6 thoughts on “Get data from massive web crawling system

  1. […] Other Thoughts If Everyone Had A Foundry – Seth Godin Get data from massive web crawling system – Paul Allen Alexa Allows Personal Search Development Alexa’s Crawler is Yours – Marc Eisenstadt Alexa’s search platform – Rajesh Jain […]

  2. Alexa Web Search Platform – It’s a Problem

    There’s just one problem with the new Alexa Web Search Platform, which was recently introduced. Just one. Alexa charges money for it. It’s cheap. $1. In fact, that’s really cheap. The only reason it could be a problem is because of a…

Leave a Reply


More Posts Logo Design

Some of you know that my partners and I are launching a new company that aims to help a billion or more people soar with

Cold Showers Can Benefit Your Health

I hated what I read in the Harvard Business Review last Friday: Cold showers have health benefits. I’ve always ignored people who advocate taking cold

People Like Me Need People Like You

“People Like Me Need People Like You” Someday I’ll write a book about I’ll highlight the people who built Ancestry from the ground up–unsung

The Team Coaching Zone Podcast

To listen to this podcast, click HERE. Announcer: Welcome to the Team Coaching Zone podcast. Join your host Dr. Krister Lowe and today’s leading organizational

My Favorite Artist

Artists, poets, and prophets can help us make sense of the craziness that we experience in the world. My favorite artist is Regina Spektor. When

The Coach You Need

When you need a helping hand in life or at work you might decide to hire a coach. There are many amazing executive, career, leadership,

Beware of Taking Advice

Next Friday I get to deliver a keynote at Phoenix’s biggest startup event. But beware of taking advice from me–or anyone else. Startup events are

An Entrepreneurial Superpower

One of my entrepreneurial superpowers is asking questions at scale. Before I started Ancestry I went to libraries and conferences and asked questions. I learned

Paul hopes to bring the strengths message to all 50 states and to 50 countries by 2025.
Contact him so he can help you spark a strengths movement in your area, company, or industry.