You moved your site. Now half your pages are missing from Google. This is the exact protocol we use to audit redirect chains, force reindexing, and plug traffic leaks. No theory. Just operational steps.
A site migration breaks your URL structure. It breaks internal links. It breaks Google's trust. The common fix—submit a new sitemap and wait—is a trap. Google doesn't re-crawl everything just because you submitted a sitemap. In fact, Google's Discover documentation makes it clear that indexing depends on content quality and site authority, not just a sitemap ping. We see teams lose 40-60% of indexed pages post-migration because they treat reindexing as a passive event. It's not. It's active recovery.
A common situation we see: an ecommerce site migrates from domain A to domain B. They set up 301 redirects. They submit a new sitemap. Three weeks later, only 12% of their product pages are indexed. The old URLs return 404s. The new URLs are 'Discovered - currently not indexed'. This is not a Google bug. It's a failure to align redirect mapping, crawl budget, and index checker diagnostics. The fix requires a step-by-step protocol that starts with a full URL inventory, not a sitemap.
| Tool / Method | What It Measures | Operational Step to Fix | Hidden Failure Mode |
|---|---|---|---|
| Google Search Console (URL Inspection) | Per-URL index status, last crawl date, coverage issue | Bulk inspect top 50 pages from your analytics; look for 'Excluded' filters | Only shows Google's view. Does not detect redirect chains or soft 404s on staging URLs |
| Index Checker API (e.g., Sitebulb, Screaming Frog Index Checker) | Real-time index status for lists up to 50k URLs, reports 'INDEXED' vs 'NOT_INDEXED' | Upload your new URL list; filter by 'NOT_INDEXED'; submit via Indexing API in batches of 200/day | Rate limits: Google Indexing API caps at 200 URLs per day per property. Large sites need 30+ days of batching |
| Server Log Analysis (raw access logs) | Googlebot crawl frequency, hit ratio per URL, response codes | Check if Googlebot is hitting old URLs (301) or new URLs (200). If 301s dominate, redirect chain is broken | Most log analyzers only show last 7 days. You need 30+ days of logs to see crawl patterns post-migration |
| Redirect Chain Validator (curl + custom script) | Full redirect path from old URL to final new URL, including intermediate hops | Script: curl -sI -o /dev/null -w '%{redirect_url} %{http_code}' http://old-url.com | grep -E '301|302|200' | Soft 404s: if final URL returns 200 but content is blank or thin, Google treats it as soft 404. HTTP status code alone is not enough |
Export all old URLs from analytics, sitemap, and crawl. Deduplicate. Target list size: 500-50k URLs.
For each old URL, confirm the new URL and HTTP status (301). Use a script to validate chains longer than 3 hops.
Run index checker on all new URLs. Filter 'NOT INDEXED'. Group by section (products, blog, category).
Reindex top 200 revenue-generating pages via Indexing API. Set clear 24-hour deadline for first batch.
Check logs daily for 14 days. If Googlebot hits old URLs more than new ones, adjust internal links and sitemap priority.
After 30 days, compare pre- and post-migration indexed counts. If gap > 10%, restart audit from step 1.
The numbers: Old domain: oldstore.com. New domain: newstore.com. Pre-migration indexed count: 3,467 product pages. Post-migration (week 1): 412 indexed. That's an 88% drop. Not unusual.
Step 1: Export all 3,467 old URLs from Google Analytics (last 90 days of traffic). Use Screaming Frog to crawl old domain and extract all 3,467 new URLs from the 301 redirect map. Found 89 old URLs with no redirect at all—they were dead. Fixed those.
Step 2: Run index checker on the 3,467 new URLs. Result: 412 indexed, 1,203 'Not indexed (other)', 1,852 'Discovered - currently not indexed'. The 'Discovered' group is the priority—Google knows they exist but hasn't crawled them.
Step 3: Batch 200 URLs per day via Google Indexing API. Focus on the 'Discovered' group first. Day 1-3: 600 URLs submitted. Day 7: recheck index status. Now 1,834 indexed. Crawl budget was the bottleneck. After 18 days of batching, 3,201 indexed (92% recovery). The remaining 266 were thin content (under 100 words) or pages with no internal links. We either enriched content or removed them.
Key settings: Indexing API requests per day: 200. Wait time between batches: 24 hours. Filter: only submit URLs with at least 500 words of unique content and at least one inbound internal link. This cut wasted submissions by 40%.
Export all old URLs that had traffic or backlinks in the last 6 months. Target: top 20% of pages drive 80% of revenue.
Validate every redirect returns HTTP 301 (not 302) and resolves in 3 hops max. More than 3 hops slows Googlebot.
Run a full crawl of the new site. Identify orphan pages with zero internal links—Google will not find them.
Check robots.txt and meta robots tags on the new site. Blocking 'Disallow: /' after migration is a common disaster.
Submit a new sitemap in Google Search Console. Set the sitemap index to include only the new domain URLs.
Set up a daily index checker monitor for the first 30 days. Alert if indexed count drops by more than 5% in 48 hours.
Create a manual escalation path: if a revenue-critical page is not indexed after 7 days of API submissions, escalate to manual URL inspection.
In practice, when you run a redirect map against 10,000 URLs, you'll find at least 200-500 that break. The worst edge case we encountered: a site migration that used a wildcard redirect from old.com/* to new.com/*. The wildcard captured everything—including old.com/admin and old.com/staging. Those admin URLs redirected to new.com/admin, which returned a 200 status code but with a login prompt. Google indexed the login page as the canonical version of the product page. Traffic collapsed. The fix required disabling the wildcard and mapping each URL individually using a custom script that checked for content similarity (cosine similarity on page body text).
Another failure mode: duplicate lists. A migration tool generated two sitemaps—one with /products/ and one with /product/ (missing 's'). Google indexed both as separate URLs with identical content. The index checker reported 'Indexed' for both, but the canonical tag was missing. We lost 30% of ranking power because of self-competition. The fix: consolidate all URLs to a single path pattern and add rel=canonical tags pointing to the preferred version. Use a technical migration protocol that includes canonical validation as a mandatory step. Without it, you're just moving broken links.
| Option | What happens | Verdict |
|---|---|---|
| Sitemap submission: passive, depends on crawl budget, takes 2-8 weeks for full reindexing. No control over priority. Free. Best for low-urgency pages. | Indexing API: active, bypasses crawl queue, triggers reindex within hours. Limited to 200 URLs/day per property. Requires authentication. Best for revenue-critical pages. | Use both. Sitemap for bulk, API for the top 200 pages that make 80% of revenue. Never choose one. |
| Manual URL inspection: best for debugging single pages, shows exact error (crawl anomaly, blocked by robots, soft 404). No bulk option. Slow. | Bulk index checker tools: scan thousands of URLs in minutes, group by status, export lists. No deep diagnostics. Requires separate debugging for 'Not indexed' reasons. | Start with bulk index checker for triage. Use manual inspection only for pages where the bulk tool shows 'Not indexed' and you need the root cause. |
| Server log analysis: measures actual Googlebot behavior. Catches soft 404s, redirect chain issues, and crawl waste. Requires technical setup and 30+ days of logs. | Google Search Console coverage report: shows indexed vs. excluded counts but lags 48-72 hours. Does not show per-URL crawl frequency. Good for trend monitoring, not for real-time debugging. | Server logs are the source of truth. Search Console is a lagging indicator. If you can only do one, set up log analysis on day 1. |
Prioritize by revenue. Export top 5k product URLs by sales. Run index checker on those. Submit the unindexed ones via Indexing API in daily batches of 200. For the remaining 45k, submit a clean sitemap and ensure each product page has at least 300 words of unique content and one internal link from a category page. Expect full reindex to take 4-8 weeks. Monitor Search Console coverage reports weekly.
The Google Indexing API is the only official way to request reindexing, but it is not a checker—it only submits. For checking, use tools like Sitebulb, Screaming Frog Index Checker, or the DataForSEO Google Index API. Sitebulb lets you upload up to 50k URLs and shows Indexed, Not Indexed, or Excluded with filters. DataForSEO returns page-level status via REST API and supports 10k requests per minute for enterprises.
Export all backlinks from Ahrefs or Semrush for the old domain. Map each backlink URL to its new equivalent. Test each redirect with curl to confirm it returns 301 and lands on a page with similar content. If a backlink URL maps to a 404 or irrelevant page, create a custom redirect. For broken backlinks, use the Indexing API to request reindex of the target new URL. Monitor backlink recovery in your tool for 60 days.
Three errors dominate. Error 403: quota exceeded—you hit the 200 URLs/day limit for the property. Error 400: invalid URL format—the URL includes query parameters or fragments that the API rejects. Error 429: too many requests—you sent more than 600 requests per minute. The fix: batch submissions, use a simple list of clean URLs without parameters, and implement exponential backoff on retries. Also, the API only works for pages that are discoverable via sitemap or internal links.
Staging environments are a common trap. After migration, check that robots.txt on the live site does not block Googlebot. Many staging migrations accidentally copy the staging robots.txt with 'Disallow: /'. Also, update the WordPress site URL in Settings > General to the live domain. Regenerate the sitemap with a plugin like Yoast or Rank Math. Submit the new sitemap in Search Console. If pages show 'Alternate page with proper canonical tag', your staging URL is still in the canonical tag—fix via database search-replace.
For each client, allocate 40 hours of technical SEO time over 4 weeks. Week 1: audit redirect maps and run index checker on all client sites. Fix critical redirect chains and orphan pages. Week 2: submit Indexing API batches for top 200 pages per client. Week 3: monitor Search Console and server logs. Adjust internal linking. Week 4: recheck index status and report recovery percentage. Expect 70-90% indexing recovery within 4 weeks per client if protocol is followed strictly. Slow vendors (e.g., hosting providers that don't share logs) can delay by 2 weeks.
This status means Google found the URL but chose not to index it. Common causes: thin content (under 200 words), duplicate content (same body text as another page), low page authority (no internal links from high-traffic pages), or a poor user experience (slow load time, intrusive ads). Fix: add 500+ words of unique content, ensure at least 3 internal links from established pages, improve Core Web Vitals, and remove any noindex tags. Then submit the page via Indexing API. If still 'not indexed' after 3 submissions, the page may lack enough authority—build backlinks to it.
A 301 redirect is permanent: Google transfers the old URL's ranking signals (PageRank, backlink equity) to the new URL and updates its index to the new URL. A 302 redirect is temporary: Google keeps the old URL in its index and may continue to rank it. After a site migration, always use 301 redirects for every moved page. If you use 302s, Google will not transfer authority, and your new pages will remain unindexed or rank poorly. Use a redirect checker to scan all old URLs and confirm HTTP status code 301.
Path changes add complexity because internal links, canonical tags, and sitemaps must all be updated. First, create a redirect map from old path to new path (e.g., /old-category/product to /new-category/product). Second, update all internal links on the new site to point to the new paths—do not rely on redirects for internal navigation. Third, update the canonical tags on all pages to the new URL. Fourth, submit a new sitemap with only the new paths. Fifth, run an index checker on the new paths and submit unindexed pages via API. Expect slower recovery because Google must learn the new URL hierarchy.
For daily monitoring: Google Search Console coverage report (free, but 2-day lag). For real-time per-URL checking: Screaming Frog Index Checker (batch up to 50k URLs) or Sitebulb (visual flowcharts of index status). For alerting: set up a custom script that pulls from the Indexing API and sends a Slack alert if indexed count drops by 5% in 24 hours. For log analysis: GoAccess or ELK stack to parse server logs and visualize Googlebot crawl patterns. For enterprise: Botify or DeepCrawl (now Lumar) offer migration dashboards with automated redirect validation.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.