SEO for large and programmatic sites
Sites with tens of thousands of pages, built from databases, feeds, or templates, live or die on how search engines crawl and index them. The work is managing collections of pages, not tuning a handful.
The usual story
A programmatic site launches, indexing climbs fast, and then a large share of pages drops out. Search Console fills with "Crawled, currently not indexed" and "Discovered, currently not indexed." The cause is rarely one bug. It's usually some mix of near-duplicate pages, URL variations generated by tools and filters, weak internal linking to deep pages, thin templates, and, on expired or repurposed domains, history the site inherited.
What I work on
- Segmented XML sitemaps. Separate sitemaps for each page type or collection, so Search Console reports indexing coverage per segment and problems show up where they live.
- Baselines and comparisons. Indexing measured per segment over time, so changes are judged against data rather than impressions.
- Parameter and tool URLs. Calculators, filters, sort orders, and search tools that spawn endless URL variations, handled with robots rules, canonicals, and link changes.
- Index bloat. Deciding which page types deserve indexing at all, and which should be merged, improved, or kept out.
- Template quality. Making sure each generated page carries enough unique, useful information to earn its place, rather than a template with a name swapped in.
- Internal linking at scale. Hub pages, related-item links, and breadcrumbs that give deep pages a path from the home page.
- Crawl efficiency. Server response times, crawl stats, redirect chains, and soft 404s that waste crawl attention.
- Reporting lag. Search Console reports trail reality by days or weeks. I check live behavior and server evidence before concluding a fix failed or worked.
Working with the development team
Large sites are built by developers, so the fixes go through them. I work in the repository and the team's chat tools, write changes as specs with acceptance criteria, and verify them with crawls and Search Console data. See developer specs and QA.
Common questions
Why did my programmatic pages drop out of the index?
Google indexes generously at first and then reassesses. Pages that look alike, add little beyond a template, or sit far from internal links are the first to drop. A diagnostic finds which segments are affected and why.
Does an expired domain help or hurt?
It can do either. Old links can help a new site get noticed, but a domain's history and a sudden change of topic can also weigh on how Google treats it. It's one more factor to separate from the site's own problems.
Updated October 5, 2026