Pay Attention to What Google Will Index for SEO Needs

Much of the intended goal of SEO revolves around preparing a website's content and technical underpinnings for the automated process Google routinely performs in which search crawlers discover pages and index them so as to determine how high they should appear in relevant SERPs. Some pages can be decided by the owner of the website to be better off not being indexed for some reason or another, but a crawler passing over a website to update its indexing ordinarily follows every available internal link that has been set up among the website's hierarchy of pages.

Since some pages would benefit readers too much to be orphaned on account of SEO concerns, a webmaster instead leaves instructions within a file named robots.txt that is located within the top level of the site's domain. Up until this point, there had been two methods of using this file to instruct potential crawlers to disavow specified pages as potential results in SERPs. The first uses the term "disallow" to effectively block crawlers from working with specific URLs at all and keep the contents of those pages irrelevant to the ranking scores Google internally awards to the website otherwise. The second uses the term "noindex" as a milder alternative that lets Google parse and award "latent" rankings to pages while preventing them from appearing on the SERPs regardless.

Recently, Google has announced that its crawlers will no longer react to "noindex" syntax strings in the robots.txt file. This is largely because the same syntactic functionality can be achieved by inserting similar syntax directly into a web page's head tag. More relevantly, since there was little readily available documentation to explain the nuances between the "noindex" and "disallow" functions, many websites' owners erroneously used the former with the expectation that it carried the functionality of the latter. Pages that the owners had marked only with "noindex" would end up getting their potentially messy and harmful contents considered by Google's crawlers at the expense of the rest of the website's authority rankings even though the offending pages were not actually being displayed on search results pages. For more information click here https://www.reddit.com/r/SEO/comments/cduoqr/googletostopsupportingnoindexdirectivein/.