Neil's News

Site Spider

17 May 2011

Site Spider is a Chrome extension that recursively crawls a website and builds a table of all resources and their status codes. What's unique about Site Spider is that it runs client-side within your browser, meaning it runs with the same permissions as you have. No more trying to make wget play nicely with a site's authentication system.

A particularly useful feature is the ability to restrict the spider's crawl based on a regular expression. It can also be configured to crawl one link beyond the regular expression, thus allowing one to check all external links on a website for link rot.

Install Site Spider at the Chrome Web Store, or browse its source code at Google Code.

This extension took far too long to push out the door. A combination of Chrome bugs and lawyers made the development process interesting indeed. Glad it's finally out.

If you are in the Bay Area and are tempted to steal one of Google's bicycles, don't. They are extremely distinctive and Google Security is ... effective.

Also, this weekend is the Bay Area Maker Faire. See you there!

< Previous | Next >