How to use Screaming Frog SEO Trips & Tricks

If you dont know what Screaming Frog is, (then where have you been!?) Its a tool that lets you crawl an entire site on demand. But thats not all its good forI was recently working on a clients site and I found myself turning to the tool more and more for tasks other than the standard crawl feature. I thought it might be useful to put together a list of other ways to use Screaming Frog.

1  Verifying Google Webmaster Tools Crawl Errors

I recently wrote a post on how to do a site audit using Webmaster Tools. Im a big fan of its features, but I locate that the GWT doesnt update the crawl errors frequently enough. This means there are often things like 404s reported when theyve actually already been fixed. Ive been using Screaming Frog to solve this problem. Below is my new process for 404s and other common errors.

Crawl the entire site and put all 404s into a spreadsheet.

Download all 404s from Google Webmaster tools and put them into the similar spreadsheet.

Remove duplicates.

Copy all URLs into a text document and save it as 404.txt

Using the Screaming Frog list mode, upload the 404.txt document and begin the crawl

Screaming Frog List Mode

You can then export every the remaining 404s and fix them.

2  Checking If a Site Migration was Successful

A site I was working on recently changed their URL structure. For a couple of reasons, some of the URLs were not redirecting correctly. The list in Screaming Frog mode came in useful for checking which URLs were not redirecting correctly.

I got the client to send me a list of every the old URLs and followed the same process as above to locate out which of the URLs were returning a 404. I then downloaded every the URLs with the problem and passed them to the developer to fix. This made identifying the problem really easy.

3  Finding Infinite Links

Sometimes websites that use relative URLs can make never ending chains of links. Again, this recently happened on a clients site. They were using relative URLs everywhere except some pages on the blog.

This meant that sometimes when they linked to a page upon the blog, it was being appended to the existing URL. For example:

http://www.example.com linking to www.example.com/page1

Was creating http://www.example.com/www.example.com/page1

This was causing infinite lists of URLs. This means that search engines could be wasting their time crawling pages that technically dont exist. Because this wasnt the case upon every page, I had to identify where upon the site the issue was. When looking at some of the examples, the cause was using links that didnt include the http. To find where this was happening I used the custom feature.

Screaming Frog Custom Feature

This is below the configuration tab. I asked it to include only pages that included the html :

<href=www.

This then returned every of the pages that were linking to other pages in this way.

4  Checking a List of Links

During processes of outreach, you often end up creating a large list of pages you are expecting links from. Going through each one to check that the link actually exists upon the page can be a boring job. To speed up the process, the Screaming Frog list mode lets you check a stream of URLs very speedily. There is already a post upon the Screaming Frog blog upon how to do this: Auditing Backlinks Using Screaming Frog.

5  Creating a Sitemap

Screaming Frog makes creating an XML sitemap really easy, but its important that you set up the crawl correctly before you start. If you dont limit the spider to crawl only those pages that you desire in your sitemap, you can cease up with a bunch or URLs that shouldnt be in there. An example of this is with WordPress, which I discovered when I crawled my site.

A common problem with WordPress is that it creates pages like http://www.craigbradford.co.uk/about-craig/?replytocom=12 when people depart comments.

I dont want these pages indexed and definitely not in my sitemap, so I can use the exclude tool (which is below the configuration menu), to ensure every with this style of URL tail is excluded.

Screaming Frog Exclude Feature

Once you have set up the configuration, let Screaming Frog compete a full crawl of the site. Once complete you have the option to export the sitemap. Under the main navigation go to Export then prefer sitemap. You can then upload it to your site and submit it through webmaster tools.

6  Check Sitemap for Errors

Duane Forrester from Bing recently said that Bing permit a 1% level of dirt in a sitemap. Dirt could be anything from 404 errors to redirects.

Screaming Frog can be used to save your sitemap tidy and healthy. If you have the XML file like shown above, you can simply change to list mode and upload the XML file. Screaming frog can then crawl every of the URLs and say you if there are any errors such as 404s or pages that are redirecting.

7  Using Screaming Frog for Linkbuilding

When doing outreach, I often find it easier if I first contact a link target through something like Twitter.

Taking this one step further, an simple and innocent way to profit upon someones radar would be to crawl their site for them, find a blog post that returns a 404 and tell them about it.

Assuming you dont use an SEO profile, this is a good way to be kind and draw attention to the fact that you read their blog. Now, when you actually do contact them for outreach, its not out of nowhere and theyll at least recognise your name and face.

8  IP and User Agent Redirection

Two features that dont gain as much attention are the proxy option and the ability to change user agent. Taking them in order, the proxy feature can be useful for clients that are using IP based redirects. To look whats going upon you can use buy a set of international proxies or else attempt some of the free ones at Hide My Ass. You can them compare the results. To use this, pick Configure then Proxy.

Screaming Frog Proxy Setting

Tick the bin for User Proxy Server and enter your proxy details. When you crawl the site now, it will be using the international address instead of yours. If you are going to do this, I would recommend paying for private proxies as the free ones can be quite short-tempered.

Changing user agent can be useful for checking if websites are treating search engine crawlers such as Googlebot differently to users. It can also pick up if robots.txt is explicitly blocking definite content from individual search engines. To use this feature just select Configuration and then User agent, it doesnt receive much easier than that.