Most advice about indexing assumes you want everything indexed: submit the sitemap, check the coverage report, chase the pages that didn't make it. That is the right instinct for a new site with fifteen pages, and the wrong one for a site with four hundred, most of which a plugin generated.
The index is not a scoreboard. A page in it is a page that can be shown to someone as an answer, and pages that can't be a good answer earn nothing by being there.
The takeaway up front: a page belongs in the index if it could be the only page a stranger ever sees on your site and still do its job. Everything else falls into "keep it out" or into an awkward middle you decide case by case — and the tool you use to keep it out matters, because the common ones do genuinely different things.
The test that sorts most of it
Imagine someone arrives on the page directly from a search result. They have not seen your homepage, they do not know your brand, and they will not click anything else.
Can the page serve them?
If yes — it answers a question, describes a service, sells something, tells them who you are — it should be indexed. If no, because the page is a fragment, a duplicate, a step in a process they haven't started, or a list of links with no content of its own, it should not be.
That question resolves most pages on a normal site. It also explains the awkward ones, which are awkward precisely because the answer is "sometimes."
Pages that almost always belong in
- Home, about, and contact. Contact especially — it is often the page that ranks for your business name plus a city.
- Service and product pages. One page per thing you actually sell. These are what the rest of your visibility work exists to support.
- Blog posts and guides with substance. Anything that answers a question in full.
- Location pages, when each one is genuinely different — a real address, real staff, real service details. Near-identical pages with a swapped town name are covered in the middle section below.
- The pages your listings point at. If a directory entry or citation sends people to a URL, that URL should be indexable, or the listing is aimed at nothing.
Pages that almost always stay out
- Thank-you and confirmation pages. Meaningless without the step before them, and an indexed one distorts conversion tracking when people land on it directly.
- Internal search result pages. Every query creates a URL, so left open they generate an unbounded set of thin pages assembled from other pages.
- Cart, checkout, and account pages. No stranger can use them.
- Filtered and faceted URLs. Colour, size, price, sort order — one product listing can spawn hundreds of near-identical variants.
- Staging and development copies. They leak into the index more often than anyone expects, and compete with the real thing.
- Print-style duplicates, and any alternate rendering of a page that already exists.
- Paginated pages beyond the first, usually — page 7 of a blog archive is a list of titles with nothing of its own to say.
- Author archives on a single-author site, a second copy of the blog index under another name.
The awkward middle, and how to decide
These are the pages where "it depends" is the honest answer, so here is what it depends on.
| Page type | Index it when | Keep it out when |
|---|---|---|
| Category archives | The category is a topic someone would search for, and the page has an intro and a curated set | It is an automatic list of every post, duplicating the blog index |
| Tag archives | Tags are few, deliberate, and function as topic hubs | Tags were applied loosely and each holds one or two posts |
| Location pages | Each has distinct address, service, staff, or pricing details | They are one template with the place name swapped |
| PDFs and downloads | The document is the useful thing and stands alone | It duplicates a page you already have, or exists only after a form |
| Thin blog posts | You will expand it into something that answers the question | It exists to hit a publishing quota |
| Gated content pages | The landing page describes the offer usefully | The page is only reachable and only meaningful post-signup |
A useful tie-breaker: would you be happy for this to be someone's first impression of the site? If you would rather they landed somewhere else, that is your answer.
The second tie-breaker is maintenance. Every indexable page is one you have implicitly promised to keep accurate, and a hundred location pages you will never update again are a hundred small liabilities.
What "keeping it out" actually costs you
Index bloat is often described as if a search engine punishes you for extra pages. That is not quite what happens, and the real costs are more specific:
- Crawl attention gets spread thin. A crawler has a finite appetite for your site. Time spent on filter URLs is time not spent noticing your new service page.
- You compete with yourself. When three of your own pages could answer the same query, the engine picks one, and it may not pick the one you would have.
- Your reporting gets noisier. It is harder to see what is working when hundreds of URLs sit in the report doing nothing.
None of these is a catastrophe, which is why this is a decision rather than an emergency. It matters most on larger sites, shops, and any site where a plugin generates URLs automatically.
Four ways to keep a page out — and why they are not interchangeable
This is the part most people get wrong, and one of the combinations actively backfires.
1. A noindex meta tag (or header). Tells the engine: crawl this, but don't index it. This is the right default for a page that must remain reachable to visitors but shouldn't be a search result — thank-you pages, internal search, gated flows.
2. robots.txt disallow. Tells the crawler not to fetch the URL at all. Useful for large families of generated URLs where you want to save crawl attention entirely, such as faceted parameters. Its limitation matters: a blocked URL can still be indexed as a bare link with no content, if other pages link to it.
3. A canonical tag. Says: this page is a variant of that one; consolidate them. Right for genuine duplicates and near-duplicates — a print version, a URL with tracking parameters, the same product under two paths. It is a strong hint rather than a command.
4. Simply leaving it out of the sitemap. Says nothing at all. A sitemap is a list of pages you recommend; omission is not exclusion, and an omitted page that is linked will still be found. This is the weakest of the four and works only alongside one of the others.
The trap worth remembering: don't block a page in robots.txt and put noindex on it. The crawler obeys the block, never fetches the page, and therefore never sees the noindex — so the page can linger in the index precisely because you tried twice to remove it. Pick one. If a page is already indexed and you want it gone, use noindex and let it be crawled until it drops out; block it in robots.txt only after that.
For the mechanics of building and submitting the list of pages you do want, see our XML sitemap guide, and for how discovery, crawling and indexing fit together, start with how to get your website indexed.
A review you can run in half an hour
- List what is indexed. A
site:yourdomain.comsearch gives a rough picture; the Page Indexing report in Search Console gives the accurate one. If you haven't set that up, our Search Console walkthrough covers it. - Sort the surprises. Look for URL patterns you did not intend: parameters, search results, staging subdomains, attachment pages.
- Apply the landing-page test to each pattern, not to each URL. Patterns are how these problems appear and how they get fixed.
- Choose one tool per decision —
noindex, canonical, orrobots.txt— using the distinctions above. - Clean the sitemap so it lists only pages you want indexed, and nothing that returns a redirect, an error, or a
noindex. - Re-check in a few weeks. Removal is not instant; pages drop out as they are recrawled.
Do the opposite check while you are there: pages you want indexed that aren't. If they show as crawled but not indexed, that is a different problem with a different fix, covered in crawled — currently not indexed.
FAQ
Does having pages excluded from the index hurt my site?
No. A deliberate exclusion is a normal, expected signal — every large site has far more URLs than indexed pages. What hurts is the accidental version: a noindex left on a page you want to rank, which is worth checking first whenever an important page won't appear.
Should I index tag and category archives? Only if they are genuinely useful destinations — a real topic, a curated set, some introductory text. Automatically generated archives that simply relist your posts rarely earn a place, and they compete with the posts themselves.
How many pages should a small business site have indexed? There is no target number, and chasing one is how sites end up with filler. A useful rule of thumb rather than a rule: if you cannot say what query each indexed page is meant to answer, you probably have more indexed than you need.
Will removing pages from the index reduce my traffic? Only if those pages were bringing traffic — so check first. Open your search performance report, confirm the pages you plan to exclude receive effectively no clicks or impressions, and exclude them in batches rather than all at once so you can see the effect.
Decide once, then keep the list honest
Indexing is a judgment call about which pages deserve to represent you, and it is worth making deliberately rather than letting a plugin make it by default. Run the landing-page test across your page types, sort the awkward middle on usefulness and maintenance, pick one tool per exclusion, and make sure your sitemap agrees with the decisions you made.
Then make sure the pages that survived that filter are actually being found. Get your site submitted and listed where it counts with AddTopWebsite.
