> A trading card price site of 20,000+ pages on an expired domain lost indexing after a strong start. How Chris Abraham manages crawling at scale.
>
> Source: https://gerriscorp.com/case-studies/programmatic-indexing/ · Updated 2026-10-05 · By Chris Abraham, Gerris Corp

# Programmatic site: managing crawling and indexing at scale

A new site publishing prices for more than 20,000 trading cards, built programmatically on an expired domain, indexed well at first. Then large numbers of pages dropped out of Google's index.

## My role

Technical SEO for crawl and index management, hired hourly, working with the owner over Teams and with access to the development repository on GitHub.

## How I approach it

A site like this has to be managed as collections of pages, not a handful of landing pages. The questions are which page types deserve indexing, how tools and filters generate URL variations, and how search engines discover deep pages.
- **Segmented sitemaps** for each content type, so Search Console reports indexing coverage per segment.
- **A baseline** of indexing by segment, so every change is compared against data.
- **Robots rules for parameterized tools** whose URL variations add nothing to the index and draw crawl attention away from the pages that matter.
- **Reporting lag accounted for.** Search Console reports trail reality, so I check live behavior before concluding a change worked or failed.
- **The domain's history** treated as one factor among several, kept separate from the site's own problems.

## Working with developers

The site is development-led, so fixes go through the repository. Access to the code means I can see how URLs are generated and specify changes precisely.

Every recommendation goes to the owner in writing, with the evidence behind it.

## Status

Ongoing. No recovery figures are claimed here; indexing at this scale moves over weeks and months. See [SEO for large and programmatic sites](https://gerriscorp.com/services/large-sites/).

Written by [Chris Abraham](https://gerriscorp.com/about/), founder of Gerris Corp in Arlington, Virginia, from his own client engagements, with every technical claim checked against the documentation listed. Websites since 1994, digital marketing since 2002. [Editorial standards](https://gerriscorp.com/standards/) · [Upwork profile](https://www.upwork.com/freelancers/chrisjabraham)

Updated October 5, 2026