Updates to Google HTTP status codes, Googlebot, and job posting help documents

Google has changed some of its documentation to help Google search over the past few days. Updated documents include HTTP status codes, Googlebot, and job posting help documentation.

HTTP status codes

The HTTP status code document has added a whole large section for 404 errors which was not in old version. Here is a new section:

soft errors 404

A soft 404 error is when the URL that returns a page tells the user that the page does not exist and also a 200 (success) status code. In some cases, it may be a page without main content or a blank page. Such pages can be created for various reasons by the web server or content management system of your site or the user’s browser. For example:

• Missing included file on server side.
• Disconnected from the database.
• Empty internal search results page.
• Unloaded or otherwise missing JavaScript file.

A bad user experience is to return a status code of 200 (success) and then display or suggest an error message or some error on the page. Users may think the page is a working live page, but then some sort of error appears. Such sites are excluded from Search.

When Google’s content-based algorithms detect that a page is actually an error page, the Search Console will display a soft 404 error on the site Index Coverage Report.

Correct soft errors 404

Depending on the state of the page and the desired result, you can solve soft 404 errors in several ways: Try to determine which solution would be best for your users.

The page and content are no longer available

If you removed the page and there is no alternate page with similar content on your site, return a 404 (not found) or 410 (missing) response code (status) for the page. These status codes indicate to search engines that the page does not exist and that the content should not be indexed.

If you have access to server configuration files, you can make these error pages useful to users by customizing them. A good 404 custom page helps people find the information they are looking for, and also offers other useful content that encourages people to further research your site. Here are some tips for designing a useful 404 custom page:

• Make it clear to your visitors that the page they are looking for cannot be found. Use language that is friendly and inviting.
• Make sure your 404 page looks and feels the same (including navigation) as the rest of your site.
• Consider adding links to your most popular articles or posts, as well as links to your site’s homepage.
• Think about how to allow users to report a broken connection.

Custom 404 pages are created exclusively for users. Because these pages are useless from a search engine perspective, make sure the server returns a 404 HTTP status code to prevent the page from being indexed.

The page or content is now somewhere else

If your page has moved or has a clear replacement on your site, return a 301 (permanent conversion) to redirect the user. This will not interfere with their browsing experience and is also a great way to let search engines know your new site location. Use URL Review Tool to verify that your URL is actually returning the correct code.

The site and content still exist

If a good page was marked with a soft 404 error, it probably didn’t load properly for Googlebot, was missing critical resources, or displayed a visible error message while rendering. Use URL Review Tool to view the rendered content and the returned HTTP code. If the rendered page is empty, almost empty, or the content has an error message, your page may rely on many unloading resources (images, scripts, and other non-textual elements), which can be interpreted as soft 404. Reasons to resources cannot be loaded, they include blocked resources (blocked by robots.txt), too many resources per page, various server errors, slow loading or very large resources. Hint tip on this from Kenichi Suzuki onwards Twitter.

Googlebot

On Googlebot, how many bytes of textual content, such as HTML, Googlebot will specifically investigate here. Here are the new lines of text:

Googlebot can search the first 15 MB of content in an HTML or supported text-based file. After the first 15 MB of the file, Googlebot stops crawling and only considers the first 15 MB of content for indexing.

Job postings

On jobsGoogle has specified that you must include the addressCountry property when using the jobLocation property.

These are the changes observed in the last few days in Google’s help documentation.

Forum discussion on Twitter.