Programmatic site: managing crawling and indexing at scale

A new site publishing prices for more than 20,000 trading cards, built programmatically on an expired domain, indexed well at first. Then large numbers of pages dropped out of Google's index.

My role

Technical SEO for crawl and index management, hired hourly, working with the owner over Teams and with access to the development repository on GitHub.

How I approach it

A site like this has to be managed as collections of pages, not a handful of landing pages. The questions are which page types deserve indexing, how tools and filters generate URL variations, and how search engines discover deep pages.

• Segmented sitemaps for each content type, so Search Console reports indexing coverage per segment.
• A baseline of indexing by segment, so every change is compared against data.
• Robots rules for parameterized tools whose URL variations add nothing to the index and draw crawl attention away from the pages that matter.
• Reporting lag accounted for. Search Console reports trail reality, so I check live behavior before concluding a change worked or failed.
• The domain's history treated as one factor among several, kept separate from the site's own problems.

Working with developers

The site is development-led, so fixes go through the repository. Access to the code means I can see how URLs are generated and specify changes precisely.

Every recommendation goes to the owner in writing, with the evidence behind it.

Status

Ongoing. No recovery figures are claimed here; indexing at this scale moves over weeks and months. See SEO for large and programmatic sites.

Updated October 5, 2026