Crawl errors in Search Console: 404, soft 404 and server errors, one by one

In Search Console, the Pages (indexing) report lists what Google tried to read and could not, or read and decided not to keep. Not everything there is a problem. The job is to separate what is normal from what needs action, and to treat each type its own way.

What each error means

What the report says What it means What to do
Not found (404) The page does not exist at that address. If it was deleted on purpose and nobody looks for it, leave it. If it has links or visits, create a 301 redirect to the closest match. See 301 redirects.
Soft 404 The page answers “all fine” (200) but looks empty or says “not found”. Either put real content on the page, or make it return 404 or redirect. Common with empty internal searches and categories with no posts.
Server error (5xx) The site failed when Google tried to read it. If it is a one-off, ignore it. If it repeats, the server or a plugin is failing: see errors 500, 403, 404 and 508 and talk to us.
Redirect error A loop, a chain that is too long, or a destination that fails. Follow the redirect with curl -I and leave a single hop to the final destination.
Blocked by robots.txt A robots.txt file forbids the robot. If you did not mean it, correct the rule. See robots.txt and sitemap.xml.
Access denied (401 or 403) The server refused the robot. It may be a firewall or security rule. See checking that Google can see your site.

The method, in order

1 Open the Pages report and look first at the “not indexed” reasons with the most addresses. Do not start with the smallest.
2 Click a reason and look at a few addresses. Open them in the browser, in a private window, and confirm what Google saw. URL Inspection shows the current status.
3 Ask: should it be on Google? A thank-you page, an internal search address or a duplicate should not. Those “errors” are normal.
4 Fix those that should. Make them answer 200, link to them from other pages, or redirect the old address to the new.
5 Request validation. After fixing a group, use the button to validate the fix. Google visits those addresses again, and the check can take a while. It notifies you when done.
A 404 is not a penalty. Every site has pages that stopped existing, and Google knows it. What hurts is having 404s on pages with links or lots of demand, and broken internal links. Do not redirect everything to the home page just to make the report look clean.
If 5xx errors repeat, the crawl stats report (in Settings) shows the time and frequency. Compare with the site’s logs in cPanel. And if a site protection is blocking the real robot, fix that rule rather than switching it off. See visitor statistics.

Does Search Console show server errors on your pages? Send us the address, the error type and the date.

Open a support ticket

SEE ALSO

Checking that Google can see your site

Website errors 500, 403, 404 and 508 explained

301 redirects

Why your site does not show on Google

RECOMMENDED PRODUCT

Web hosting with cPanel

Domain and SSL included, daily backups and the panel you already know. from 5.940,00 Kz/mo (3-year plan, with coupon)

See plans
  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

The best SEO plugins for WordPress and how to choose one An SEO plugin gathers, in a single panel inside WordPress, the technical settings that help search engines understand yo...
Being found on Google: Business Profile, Search Console and the basics Having a website is not the same as being found. They are different things, and the second does not happen by itself - y...
E-mail marketing: doing it properly without burning your domain E-mail is still the channel that returns most per unit of money spent - and also the one where it is easiest to ruin eve...
SMS marketing: Sender ID, the rules here, and how to write one that works SMS has an advantage no other channel has here: it reaches everybody, without depending on internet access, an installed...
What Yoast SEO is, and what it actually does Yoast SEO is a WordPress plugin that gathers, into a single panel, the technical settings that help search engines under...
The canonical tag: duplicate pages, www or not, and which address Google keeps The same content can open at more than one address: with and without www, on http and https, with a slash at the end or...