Google explains why Google does no longer index and does no longer index every URL

John Mueller of Google has written a totally certain and sincere explanation of why Google (and third-celebration seo equipment) don’t look for and index each URL or hyperlink on the internet. He defined that crawling isn’t goal, it’s miles high priced, it may be inefficient, the net is changing lots, there may be spam and rubbish and all this needs to be taken under consideration.

John wrote this precise solution on Reddit the solution to why “Why don’t search engine optimization gear display all the oneway links?” but he replied to it in phrases of Google search. He said:

there’s no goal manner to look the net nicely.

it’s far theoretically not possible to search the whole lot, as the wide variety of actual URLs is absolutely endless. For the reason that no one can manage to pay for to shop an endless quantity of URLs in a database, all internet spiders assume, simplify, and bet what is realistically well worth crawling.

or even then, for realistic purposes, you may’t move slowly all of this all of the time, the internet doesn’t have sufficient connectivity and bandwidth for that, and it expenses a whole lot of cash to access many sites on a ordinary foundation (for the spider and for the web site proprietor).

within the beyond, a few pages exchange speedy, others haven’t modified in 10 years – so spiders attempt to keep effort through focusing greater on pages they count on to exchange than on those they assume not to exchange. Will alternate.

We then contact at the element in which spiders try and determine out which pages are actually beneficial. The net is full of garbage that nobody cares approximately, pages which have been useless because of spam. These pages may also nevertheless change often, they may have affordable URLs, however they may be handiest for the repository and any search engine that cares about its users will ignore them. Every so often it’s now not simply apparent garbage. Increasingly, websites are technically high-quality, but they just don’t obtain “coloration” from a first-class standpoint to need to be searched for greater.

consequently, all spiders (along with seo gear) run on a very simplified set of URLs, they need to figure out how often to look, which URLs to search for greater regularly, and which parts of the net to ignore. There aren’t any set rules for any of this, so each tool will must decide on its very own. This is why engines like google have exclusive listed content, why seo gear listing extraordinary hyperlinks, why all of the measurements constructed on those are so different.

I concept it’d be true to emphasise this because it is useful for seo to read and understand this.

forum dialogue on Reddit.