How to speed up the processing of redirects during a website migration?

Unfortunately, building a redirect plan with Swan (or without it, we are not savages) is not the end of the ordeal that a website migration project is. I regularly receive messages from webmasters and SEOs telling me that the old URLs are still indexed. It is for them (and therefore now for you) that we are writing this tutorial today, to explain how to speed up this process and add it to your SEO checklist for your website redesign, whether it is simply a change of URLs, of site structure or just of domain name.

DISCLAIMER 1: Applying this tutorial does not mean you can remove the redirects afterwards. As we showed in our article on how long to keep redirects, you must leave them in place, even after several years.

DISCLAIMER 2: No, Google will not take this tutorial into account immediately, but you will quickly see that it helps speed up the processing of the migration plan.

To explain why old URLs do not disappear from Google's index, we need to go back to how Google initially discovers URLs.
Google goes through the pages in its index and, for each one, adds every URL targeted by an outgoing link to a queue for crawling and possible indexing.

This is where we hit the paradox of a migration done too well: there are no longer any pages pointing to the old URLs (apart from external links that have not been updated), because you will have taken care to update all your internal linking.

You have understood: we will have to encourage Google to go back to these old URLs. From there, we have two technical options:

• Create a sitemap.xml to submit in the Search Console
• Create internal links to encourage the crawl.

We will set up both solutions, the first mainly for monitoring and the second to encourage Google to crawl the URLs.

For the first one, the process is fairly simple: you will need the list of URLs to make "disappear" in an Excel file. Normally the list of source URLs from your redirect plan should be enough.

Settings to generate a sitemap of 301s only

1. Switch Screaming Frog to list mode
2. Paste the list of URLs to force a crawl
3. Then click Sitemap to export a sitemap containing only the 3xx codes (assuming these are the only ones you want to monitor)
4. Upload the sitemap to your site's FTP
5. Submit the URL of this newly created sitemap in the "Sitemaps" report of the Search Console.
   

"Page indexing" report in the Search Console (formerly "Coverage")

The idea here is to follow the gradual decrease in the number of indexed URLs.

Here we will set up an HTML page listing links to all the URLs. We will simply create a list of href links with the URL itself as the anchor. This has no impact; it is a purely arbitrary choice for simplicity, as any other choice would make it impractical to automate a large number of anchors for the URLs (and the goal here is not semantic SEO).

We will use a simple concatenation in Excel.

You simply need to add all the generated links to the content of a page, without adding anything else. Ideally, you should then link to this page from your site's footer. (Here again the anchor of the footer link will have no impact; I recommend something explicit simply so as not to confuse users who land on this page, which has no real UX value.)

Placing this link in the footer is meant to ensure that Google discovers this page every time it crawls.

Depending on how often your site is crawled (which depends in particular on its popularity, and therefore on the number of referring domains), you should gradually encourage Google to discover all the 301s and therefore purge the old URLs from its index.

In a previous article on how long to keep redirects, we said that, to keep things simple, it was wise never to remove redirects that are in place. On the other hand, you can remove the setup we have just described once the redirects you care about have been processed. You can even, if you feel like it, keep it up to date with the URLs still in place.

If you have any questions, comments or suggestions, feel free to share them on LinkedIn along with this article; we will be happy to answer!