Error 404 Guide for SEO & Usability (FAQs & More)

Originally published in 2011. Updated April 2022.

404 Error Pages play a very important role in both SEO and website usability. While often perceived as the bubonic plague of SEO 👀, 404s serve a simple purpose that should be strategically embraced. 

Over the years, we've dealt with “404 page not found errors” exhaustively. And in doing so, we've been asked a lot of questions. Below we've compiled FAQs that we get from clients about 404 Not Found pages, including things that commonly go wrong (so you know how to fix them), plus advice on how to set up and monitor 404 errors properly.

This guide is extensive, so use the links below in the table of contents to find the questions that are most relevant to you.

A 404 error (HTTP 404), also called a “header response code” or “http status code”, or simply "crawl errors", is the computer equivalent of saying “Not Found” or “Page Not Found.”

Here’s the “tech speak” definition:

“The server has not found anything matching the Request-URI. No indication is given of whether the condition is temporary or permanent. The 410 (Gone) status code SHOULD be used if the server knows, through some internally configurable mechanism, that an old resource is permanently unavailable and has no forwarding address. This status code is commonly used when the server does not wish to reveal exactly why the request has been refused, or when no other response is applicable.”
- via w3.org

In other (less technical) words, the 404 response code essentially tells search engines - and users - that the resource (or requested URL) being referenced does not exist or literally can’t be found.

It’s a robot’s version of a shrug and blank look.

There are many other response error types, and all URLs return response codes of some sort. A correctly functioning page, for example, should return a “200” status code, which means “Found.” The other major error type is called "server errors", represented by HTTP status codes 500-599. Different error types help web admins diagnose the source of the errors so they can fix them appropriately.

404 errors are generally the most common error type. They are also often handled incorrectly by well-meaning people - hence the purpose of writing this article! 404 HTTP errors are sometimes called "Client Errors", where the "client" typically refers to the user's web browser (Google Chrome, IE, Firefox, etc.)

A soft 404 is essentially a page that doesn’t return a 404 response code, but Google(bot) nevertheless believes the page is experiencing an error. Soft 404 errors occur when the server sends a 200 status for the requested page (implying the page exists) but Google thinks that the page should return a 404 error.

• You can find out if Google believes any of the pages of your site are soft 404s via the Coverage Report, one of the many tools in GSC to find and fix errors, located in the "Excluded" section of Google Search Console. More on that below.
• Sometimes these pages are good, functional pages and Google made a mistake.
• More often, however, they are pages that are blank / have no content - but they aren’t issuing a proper 404 status code (like they should be.)
• See: How to Fix Soft 404s

Essentially, a soft 404 does not issue a 404 header status code; it (perhaps incorrectly) issues a 200 (Okay) status code instead. However, Google thinks that the page should return a 404, thereby issuing the page a soft 404.

A proper 404 page, on the other hand, does correctly issue a 404 (Page Not Found) HTTP status code. So while 404 errors simply indicate pages that do not exist, soft 404 errors are slightly more ambiguous, indicating a page that should not or does not exist but still does in fact exists according to the CMS.

A 404 error tells search engines, website visitors, and web admins when page URLs are broken, or never existed in the first place. From a technical perspective, 404s are useful communication tools that help us take action.

When we can see where these error codes occur, we can fix the problem for future website visitors (generally via a 301 redirect to the new location of that content), thereby preserving the power of the page that once existed (... or the incorrect link to a page that never did. That’s what we call “broken backlink building".)

If you don’t issue a 404 response code, you won’t inherently know that the page - and therefore the user - is experiencing an error.

So YES. A 404 page is useful. You can’t fix what you don’t know about!

Errors 400 to 499 are all various types of Client Errors. Some other common 4xx errors include:

• 400 (Bad Request) - the server can't process the request due to a perceived client error, such as inaccurate request syntax or invalid request message framing.
• 401 (Unauthorized) - the request cannot be fulfilled because it lacks valid authentication credentials.
• 403 (Forbidden) - this one commonly occurs when you crawl a site too quickly and the platform/server (e.g. Shopify!) will deny you access to reduce their web server load.
• 410 (Gone) - whatever this resource was, it's now gone - and that's permanent & intentional.
• Find a full list of all the 4{xx} status errors.

There are several instances that can cause 404 errors to occur. Some of the most common reasons how and when pages experience 404s include: 

• When pages have been moved (e.g. a missing page, perhaps due to it getting moved to a different section of your website) or removed (e.g. a non-existent or deleted page was forgotten and never redirected)
• When a web admin, CMS user, or software engineer mistypes a URL on a page or "page template" or has a copy-and-paste mistake (e.g. the wrong URL was linked to)
• When broken links or accidentally truncated links occur on web pages, social media posts, or in an email message
• Real soft 404s occur when a page issues a 200 (OK) status when it should have issued some other error because something went wrong. Most commonly the website should have issued a 404 error, but didn’t. Usually, this is an indication that 404s aren’t functioning properly. Sometimes it works in some sections of your site, but not others.

Understand the relationship between 404 errors and SEO, including common mistakes and the repercussions of inadequate 404 setups.

Yes, they do say this! But unfortunately, Google sometimes says things that aren’t – strictly speaking – accurate. (See: Marketers Say Most of Google’s Public Statements Are False or Misleading.) Here are some clarifying points behind why you shouldn’t allow pages to keep 404ing in most cases: 

• Actively linking to 404s is just bad - for everyone. Lost users, and therefore lost revenue opportunities.
• Having too many 404s, or other site errors, can contribute to an overall high error rate on your site, which can cause Google to distrust your site over time. Plus, you really don’t want Googlebot spending time crawling your 404s instead of your functional, high-quality content - right?
• ‍Error URLs can have SEO equity - that you can't/don't receive if you don't fix them.
• When there is a business case for not redirecting some URLs - consider issuing a 410 “Gone” message instead. Use cases for not redirecting things: your website site got hacked and the URLs in question were spammy / malware issues! Or your purchased your domain from a 3rd party, and there used to be pages for a different business - and you actively don’t want them associated with your new business.) Googlebot tends to respect 410s much more quickly than 404s.‍

The issue with 404s, as we see it, is that Googlebot appears to treat them as "temporary". If there are active links to 404 pages - either on your site or elsewhere on the web - Google can and will keep checking them to see when they will get fixed. They will continue to do this for months - and sometimes years - after the URLs stop working (even when you fix these), if you don’t resolve them.

You can validate this yourself by viewing your log files, and seeing the quantity/frequency Googlebot hits 404ing pages.

If/when you need to "vet" what gets redirected (eg if there are business or budget or tech reasons NOT to do so), here's our recommended process.

There is some dissent online debating the question of whether 404 errors are bad for SEO.  Here’s a breakdown of why 404s can be disruptive for both SEO and user experience. 

In theory, at least, 404 errors are "bad" because they represent errors on your website (or on the web, but relating to your website). But mistakes happen, and they are easily forgiven on a small scale.

That said, a user that comes across a 404 page is less likely to return to that site later. (We’ll cover how to make this less likely, below.)

Too many 404 header response errors - or 403s, 500s, or really any other type of 4{xx} or 5{xx} error - across a site can create an overall high error rate vs. success rate. This results in trust issues: if Google (or Bing, etc.) sees too many errors vs. functional pages, they won’t want to send users to your site. 

Why would they? That user will get lost and mad, and have a bad experience. It would mean that Google failed at its job - helping the user quickly & easily find the answer to their question.

If your website is actively linking to a bunch of non-functional pages, that means you are passing "page rank" (e.g. SEO equity) … to nothing. Picture a scenario wherein your website is like a purse with holes in it, with money dripping out. No bueno!

That said - 404 errors can and will happen no matter what. A properly functioning 404 page will notify the web admin that the error occurred, and where it occurred. Then we can go about fixing it via a 301 (permanent) redirect. The user, and search engines, never need to come across this particular issue again.

Key Point: Some teams see 404s as “bad” and so avoid this inherent "badness" by simply choosing not to issue a 404 status code on a 404 page. The trouble is - the errors real users are experiencing can and will occur no matter what. Let’s not “shoot the messenger!” Mistakes happen, and error codes help us identify those! Issuing a correct 404 status code means that you can more easily find issues in order to fix them.

So… to state the obvious… don’t forget to fix them! Too many outstanding, unfixed 404s can be actively bad, or can be big missed opportunities. Plus it’s just a bad user experience.

Mistakes and improper setups with 404 pages are not uncommon. Here are some of the most prevalent mishaps involving 404 setups.

• Redirecting to a 404 page. This hurts everyone. Users are lost, search engines think everything is honky-dory when it's not, and since you don't know when it's happening - you can't fix it.
• Automatically redirecting to the page you assume search engines and users want. A risky solution, that can easily go wrong (typically because they are sent to irrelevant content - like your homepage.) It's best not to assume. Find the issue and fix it.
• Serving a 404 message on the page visually, but not delivering the corresponding 404 http response code. This hurts everyone (for the same reason as the "redirect to a 404 page" item above.)
• Serving a different status code to different user agents. For example, are you sending Bingbot a 404 (not found), and users in a browser a 200 (okay) message? Sometimes this occurs across different devices, e.g. there's a 404 on mobile (and for Googlebot-Mobile) but not on desktop browsers.
• Not finding & resolving critical errors - on a regular basis. Engineering teams are busy - and they don't want more work, so it's not uncommon for them to refuse to do to. Especially when they don't fully understand the value of the work to the business. ‍
• SPAs (Single Page Applications) can't create a proper 404 "out of the box". Here’s the correct way to fix that.

When 404 errors are neglected or setup mistakes go overlooked, the following repercussions can unfold:

• If there are too many 404 errors occurring on the website, the 404 URL itself can start ranking (an issue that occurs when you combine a redirect to a 404 page with a URL that doesn't issue a proper 404 http header response).
• Pages on the website that should be benefiting from the link are not, lowering the overall search engine ranking potential. This results in less traffic to the website overall and you are left to figure out what to do next to recover your traffic.
• Users see a 404 page - instead of the page they should be seeing - often causing them to leave.

Unfortunately, these users are not likely to return:

• 404 pages have a high bounce rate - i.e., the percentage of visitors who visit this page first (often via a search engine) and immediately leave the website.
• 404 pages have a high exit rate - i.e., the percentage of visitors that find this page from clicking a link somewhere on your website, whereupon they immediately leave.

Learn the techniques and processes used to identify 404 errors and what action you can take to fix them.

There are several good ways to pinpoint 404 errors on your site. We’d honestly recommend doing them all, since some tools can find issues that other tools don’t see.

• Crawl your website. There are a host of really great crawlers, including Screaming Frog for small sites, and DeepCrawl for very large/enterprise sites (both often work in all situations, it’s often a personal preference or budget consideration. Sitebulb is another fantastic tool.) Each will tell you what status code errors are linked on your site.
• Check the Google Search Console (GSC) Coverage Report, in the Excluded section:

• Check Google Analytics (GA) Search for page titles that contain “not found” or “404” - this is a nice list that tells you how many times a particular page has been hit, and therefore can better help you prioritize which errors need fixing first.

• Check Your Log Files. This is our favorite way to check for 404 errors (and other issues too!), since you can see exactly who - aka which bot - is hitting what errors, on what specific pages, how frequently, and your site's total "error rate". Unfortunately, not everyone can get access to log files, depending on your platform and host. Talk to your IT and/or development team to see if this is an option for you. You’ll need a log analyzing tool to do this; our favorite is from Screaming Frog.

If you are having issues finding the 4{xx} error on the page it’s linked from (e.g. from internal links), here are some things you can test. If you can’t find the URL using any of these methods, it’s likely that the issue used to be present but has been fixed.

1) Periodically run a new crawl of the site to find and fix errors old and new.

• First, check if the link is from an inlink or a redirect. Screaming Frog’s 404 inlink report will tell you which is which; other crawlers like DeepCrawl and Sitebulb can give you similar data.
• If the answer is “AHREF”, then it's findable via a direct hyperlink and you can proceed to the steps below.
• If the answer is “HTTP Redirect”, then the source of the link is another linked URL, which then redirects to your broken page. Check your redirect file for the original URL (e.g. in Apache, or an Htaccess file) or your website's admin center/CMS (for example, in a WordPress redirect plugin.) Update the redirect location so it’s no longer pointing to a broken page.

2) View Source on the URL in question and Control+Find (C+F) for the broken URLs’ path.

3) View Rendered Source (3rd party browser plugins like this View Rendered Source plugin for Chrome) will show you source code AFTER the browser has rendered it (e.g after JavaScript and CSS have run.) Again, C+F to find the broken URL path.

4) Use Google Analytics and search for "404" or "Page Not Found" in the Title. Once you narrow down into the offending URL, click into the Navigation panel to find the previous pages in that path

5) Run the URL through the Mobile-Friendly Tool from GSC. Once it’s run, click the HTML tab, and Copy + Paste the contents of the results into a Word doc, text editor, etc. Again, C+F to find the broken URL path.

Pro Tip: Using the Mobile-Friendly Tool gives you actual Googlebot-rendered code - an EXCELLENT resource for QAing issues like this.

It’s not uncommon for 404 errors to happen on URLs... that never existed on your website. This is generally due to either:

• Issues with Googlebot crawling JS, HTML or CSS incorrectly. Find these errors via GSC > in the Excluded report. Fortunately, as Google has improved its ability to crawl JS effectively, so this happens less and less commonly.
• External links to your site that have broken paths (e.g. another website is actively linking to a non-functional page on your website.) You can discover these with 3rd party tools. My favorite is the Broken Backlink report in Ahrefs (a paid SEO tool.) You can export this data in a CVS file, then crawl each URL in List Mode to see what’s actively 404ing today.

Pro Tips: 

1) Sometimes you'll see 404 errors in GSC without a "source", and which you can't find any links to - internally or externally. Depending on the volume, you might ignore these, or just redirect them anyway.

2) Web caching can be a blocker to the QA process - it's possible that a particular URL is not currently 404ing, but was - or that it wasn't, but it is now. So remember to clear your website's cache, and your browser's cache too, if you experience any weirdness.

3) Beware creating "redirect hops" (a redirect from page A to page B and then to page C, for example) or "redirect loops" (e.g. a redirect from page D to page E and then back to page F, so the web visitor can't access any functional URL!) Most good crawlers can help you identify these in action.

4) Keep in mind that not every external link is worth redirecting. If the broken link is from a spammy or really low-value website, you might be better off ignoring it. You can determine this answer for yourself, again via paid 3rd party tools OR via a manual visual inspection. Examples of ways to determine quality quickly include: Ahref’s DR (domain rating) and UR (URL rating metrics,) Moz’s DA (Domain Authority), or Spam scores. Learn more about vetting errors to determine what you fix.

There are a variety of ways to fix 404 errors, depending on your site’s setup/platform and your software development team’s capabilities/priorities. But the simplest way to fix a 404 error is to implement a 301 redirect.

• You might be able to implement this for yourself in your website’s admin center. If so, this is the easiest way to do so! Just pick out the best new URL this page should be redirected to (ideally something highly relevant.)
• Sometimes you’ll need to go through your software or IT team. It’s not uncommon for them to push back on doing this work - full stop, or due to how many there are to implement (sometimes there are a lot). They don’t always see the value, and they are spread thin working on other business priorities. It’s generally something we do recommend fighting for, within limits. See more on this below.

Can you prevent 404 errors from occurring? Nope. They are just a reality of the world, and ignoring them won’t help. Consider 404s "technical debt" that you need to find/vet/resolve periodically.

Want to learn more about redirects? Semrush can help!

Part 1: Vet your GSC “Soft 404” list to ensure that any issues found are actually issues to fix... because they aren’t always.

To do so, export the Soft 404 error URLs from GSC, and inspect them 1) manually (visually - do they look like they have content?), 2) via a crawler for status code checker (for the actual status code they issue), and potentially 3) via a JavaScript audit.

One fascinating use case about Soft 404s was shared by the amazing Paige Ford: "We discovered multiple Soft 404s on valid Help Center articles, and theorized that Google wouldn’t index them because the help articles were explaining error codes - where those messages could be misinterpreted as 404 messages. We kept error codes (that weren’t 404 or 500) but removed the error message, and Google ended up indexing  it."

• For items that were errors and you actively want those URLs indexed - and you believe they are okay now - submit them for indexing via GSC.
• For items that aren’t a priority to index, you may be able to ignore them. It really depends on how many there are - don't let there be so many you don't see actual, real issues!

Part 2: Share your vetted list of issues with your software development team or fix them directly. In most cases it’s one of the following:

• It’s not a valid page that search engines should be accessing or indexing anyway. In this case, you can either block the page/page path via the Robots.txt file, or via the Remove URL tool in GSC if you need to get it deindexed.
• It’s a page that should have issued a 404 error, but didn’t. Your software team will need to dig in to understand why that’s happening and resolve it. Once it’s correctly issuing a 404 again, you can proceed to part 3.
• There's a JavaScript crawling or rendering issue resulting in visible content that users can see - but search engine bots can't. To troubleshoot if this is happening, you can use the Inspect URL tool in GSC, or use rendering tools like View Rendered Source, your browser's Inspect Element tooltip, or the HTML that's rendered from the Mobile-Friendly Tool.

Part 3: Follow the steps outlined to fix regular 404s, above.

There are two potential error types for when you see 404s ranking: 

1) The 404 URL template is ranking in search (the URL likely has "404" or "page-not-found" in the page slug.)

In this scenario, chances are, you made the classic mistake of Redirecting broken pages to the 404 template URL (see: What are the most common 404 mistakes?) Most often that URL issues a 200 status code.

To fix it:

• Ensure that the URL issues a 404 status error (instead of a 200, or any other header status code.)
• Stop redirecting to a 404 template; this causes all the page equity of all redirected pages to consolidate on the template itself - which is why it's ranking.
• Those steps alone may solve the problem, but if you want the process to move faster, consider using the Remove URL tool in GSC to deindex it more quickly.

2) Any other URL - that is properly serving a 404 header status code - remains indexed and ranking.

This typically occurs when the page (or pages) used to be valid, are indexed, but now - shouldn't be. Site migrations are common cause - changing URLs en masse.

Usually it's just a matter of Google not getting around to crawling the old URL - if they don't do this, they won't "see" the 404 error happening consistently over time, and therefore know to flush the URL(s) out of their index. Submitting the URL's to GSC can move that process along more quickly; here's how to do that for several URLs at once with a Sitemap XML.

In both cases, it may help to learn more about how to deindex pages and site asset.

JavaScript and Single Applications Pages (SPAs) present unique challenges in handling 404s and other status codes.

With SPAs, users get the same HTML regardless of the URL they're visiting. They download the site’s JavaScript application code, and the SPA renders the appropriate content. 

While the experience feels a lot like browsing multiple pages, you're only viewing one JavaScript application and technically just one “page” (per the actual number of index.html pages being served.)

So, 404 error pages in SPAs don’t work the same. Unlike static pages, Google has no way of telling which "pages" in SPAs are valid, and which or not. And this can create problems for SEO.

To cover for these JS 404 gaps, we break down several potential solutions and Apache/Nginx code examples in a related post found here. Two of those techniques include: 

• Inclusion solution: Of the simplest options for smaller sites is the Inclusion list solution, which involves creating an 'include' sitemap and adding a block of config code that explicitly returns a 200 for those listed URLs. ‍
• Exclusion solution: Like the Inclusion list technique but the inverse, you can create a sitemap of URLs that you know people are accessing but are not valid on the site. When pages are frequently added and removed, the exclusion solution can work best.

Though we generally recommend fixing 404s and other errors via 301 redirects, there are times when fixing them all is not possible, or not feasible, often due to technical constraints, competing business priorities, or internal politics. (Sometimes it's just not a battle worth fighting!)

Here's how we approach this "error vetting & prioritization " process:

• Identify error pages that website visitors experience. Use Google Analytics and/or your log files to identify these. Prioritize them by frequency, and potentially by historically-driven revenue (e.g. redirect pages that made you a lot of money in the past!)
• Identify error pages that search engine bots experience. Use your log files to identify these pages. Prioritize them by frequency and longevity (in other words - is Googlebot still hitting a page months and years after it was retired? If so, consider redirecting it. Googlebot is checking it repeatedly for a reason!)
• Identify error pages that have live/functional backlinks to them. URLs with more/better backlinks are more important to redirect, so you can benefit from the power of these existing links; this is called "broken backlink building."

You might also consider using GSC's Prioritization Insights in the error report - allegedly, these are in rank priority order.

If the URLs you see in GSC don't meet any of the above qualifications, you may be safe ignoring them (and letting them keep 404ing.) Alternatively, consider utilizing a 410 (Gone) status code instead of a 404. Google tends to respect 410 errors more quickly (e.g. by deindexing them and stopping crawling them.)

Here’s our step-by-step process to confirm if your 404 page is working as expected:

1) Find a 404 URL & audit it manually.

• This can be as simple as sticking a random string of letters after the domain / homepage and clicking enter, e.g. at https://thegray.company/i-made-this-up
• DO NOTE, however, that larger, more complex sites might have functionality differences in different site sections - in other words, the 404 page might work in some places on your site, but not in others.
• If you have a small site, don’t worry about this. If you have a large site, at minimum, try these QA steps within each major site section (e.g. https://thegray.company/services/i-made-this-up AND https://thegray.company/clients/i-made-this-up.) Monitor GSC more carefully for 404 errors and soft 404 errors, just in case.

2) Confirm that the page is issuing the correct 404 response. We recommend httpstatus.io. Copy + paste the fake URL into the field and click the Check Status button (you can check multiple URLs at one time, and also test via different user agents):

• Confirm that it’s issuing a 404 code. (Rinse and repeat as needed for 404s in other site sections)
• It should not: redirect (301 or 302) to a 404, OR issue a 200 (ok) status.

3) Outside of this, the primary items to pay attention to are around usability: if/when the user hits the page, do you successfully help them find their way again? How can you do that better? Read more about this and best practices for 404 pages. (You can also learn more about QA for SEO.)

No, definitely not. Don't index error pages! That's like asking for a less ideal user experience!

(This question was contributed by Sarah McDowell, who's needed to answer this question for her clients. Sarah co-hosts WTSEO's Community Podcast - be sure to check it out!)

Advance your craft in handling 404s with next-level thinking. Explore the best practices and the best examples on how to use 404 pages effectively.

The best 404 pages should offer more than just on-brand messaging and clear next steps. They should also abide by certain UX and functionality standards, as instructed below.

• Step 1: Remain at the URL that was called, (for example: https://www.thegray.company/i-made-this-up/) - e.g. DON’T redirect this to some other page like /404.html. The error should load on the URL with the error, so we can see the error - where it occurred - and therefore fix it properly.
• Step 2: A 404 "http status error" should be issued from the server. Use a tool like httpstatus.io to double-check that it’s working.
• Step 3: A 404 message - visible to the website visitor, on the page - should clearly explain what happened, and include resources, links, and ideally the ability to search your site to find what they were looking for. More on how to make this UX friendly below.

There are several things you can do to improve your website's 404 page - and the likelihood that website visitors will stick around and check out the rest of your site. Below are some of the most important 404 page best practices to keep in mind.

• Explain, in plain English (or whatever language your site targets), exactly what happened. This is for the user's benefit…. because that matters!
• Include a link to your contact page so users can attempt to solve the issue with your help.
• Include a simplified HTML sitemap (embedded in the body of the page) so the user can find their own way. Track what's being searched most often and make those resources easier to find.
• Consider adding a search bar to the 404 page template - so the user can look for the resource if they don't see listed already.
• Make sure the 404 template’s title includes the text “404” or “Page Not Found” - this way you can figure out which pages your site's visitors are hitting, and the frequency this occurs at, in Google Analytics (or your analytics tracking tool of choice.)

Have fun with it! Extend your brand's personality & connect with your audience. Examples of great implementations of this can be found here.

Based on some of the 404 page best practices mentioned above, we’ve compiled several of our favorite examples. In addition to using logic and including helpful ways to get users back on track, keep your 404 pages on-brand with added personality and creativity.

‍

Contact us if you need help fixing or improving the 404 page functionality on your website - we’d love to help!