Indexing is binary and it is checkable in about thirty seconds, which makes it the cheapest question in SEO to answer and the one most often skipped.
How do you check it properly?
Use the URL Inspection tool in Google Search Console, not a site: search. The site: operator gives you an estimate that is frequently wrong in both directions. URL Inspection gives you the actual status of the actual URL, plus the reason when the answer is no.
There are three checks, in increasing order of usefulness:
- A site: search Type site:yourdomain.com into Google. Useful for a rough count and for spotting pages you did not know were indexed. The number is an estimate and should never be quoted as a figure.
- URL Inspection Paste one URL into the Search Console inspection box. This is the authoritative answer for that page, and it names the reason.
- The Page Indexing report The whole site, grouped by reason, with the URL lists. This is where you find out that four hundred pages share one problem.
What do the indexing statuses actually mean?
Most of them fall into two families: you told Google not to index the page, or Google decided not to. The first family is a bug you can fix in minutes. The second is a judgement about the page, and fixing it means changing the page.
| Status | What it means | What to do |
|---|---|---|
| Excluded by noindex tag | The page tells Google not to index it. | Remove the tag. Check your CMS defaults and any staging configuration. |
| Blocked by robots.txt | The crawler was not allowed to fetch it. | Fix the disallow rule. Note that robots.txt does not remove a page from results, it only stops the crawl. |
| Alternate page with proper canonical tag | Working as intended. Another URL is the indexed version. | Nothing, unless the canonical points somewhere you did not expect. |
| Duplicate, Google chose different canonical | You nominated one URL and Google picked another. | Make the pages meaningfully different, or consolidate them into one. |
| Discovered, currently not indexed | Google knows the URL exists and has not spent a crawl on it. | Link to it from a page that is already indexed, and check whether the page is worth the crawl at all. |
| Crawled, currently not indexed | Google fetched it and declined to store it. | A quality judgement. Thin, duplicated or templated pages live here. |
| Soft 404 | The page returns 200 but looks empty to Google. | Either return a real 404, or put content on it. |
What if the page is fine and still not indexed?
Then the problem is usually demand rather than mechanics. Google indexes a subset of what it finds, and the subset is chosen on whether the page looks likely to be useful to somebody. A page nothing links to, that duplicates a page you already have, is a page it is rational to skip.
In practice the fixes that work, in the order we try them:
- Link to it from somewhere that matters An internal link from a page that is already indexed and already gets traffic. Buried pages get treated as buried.
- Make it not a near duplicate If it is the tenth variation on one template with a city name swapped, the pattern is the problem, not the page.
- Put the content in the HTML If the body only appears after JavaScript runs, rendering is queued and not guaranteed. Server render the parts that matter.
- Request indexing, once Useful for a single new or changed page. It is not a fix for a pattern, and hammering it does nothing.
- Prune the pages you do not need Cutting three hundred thin pages frequently gets the remaining fifty indexed, because crawl attention stops being spread across nothing.
Does a sitemap make any difference?
A sitemap helps Google find URLs. It does not persuade Google to index them. If your pages are discovered but not indexed, adding them to a sitemap changes nothing, because discovery was never the problem.
Have one anyway, keep it accurate, and do not list URLs in it that you have marked noindex or canonicalised elsewhere. A sitemap full of contradictions is a sitemap that gets trusted less.