A site migration or redesign is not done when the switch flips. It is done when Google confirms your new URLs are indexed and the old ones are properly redirected or dropped. This guide walks through the exact workflow — redirect validation, sitemap resubmission, and bulk index verification — with specific numbers, filters, and failure modes you will encounter.
The mistake most teams make is checking the homepage only. That surface-level pass hides a deeper rot: category pages that return 404, product pages with 302 soft-redirects, and old URLs still competing in the index. In practice, when you run a bulk index check against your new sitemap, you often find 20-30% of URLs are either not indexed or indexed under the wrong canonical. A common situation we see is a client who migrated 5,000 product pages, resubmitted the sitemap, and saw a green checkmark in Search Console — but three months later, traffic was down 40%. The root cause? Googlebot crawled the new sitemap, found 301 redirects on half the URLs (the old team left the redirects pointing to a staging subdomain), and dropped them from the index entirely.
This is not a rare edge case. It is the norm. You need a structured audit that covers redirect validation, sitemap hygiene, and bulk indexing verification. Google's official Search Console documentation is the authority on monitoring crawl and index status, but it rarely covers the operational failures you will face — like stale sitemaps, mixed http/https entries, or index bloat from orphaned old URLs.
Layer one: redirect validation. Every old URL must return a 301 to the correct new URL. No 302s. No meta refreshes. No chains longer than three hops. Run a crawl on your old sitemap URLs and check the status code. Any URL that returns 200, 404, or soft redirect is a leak.
Layer two: sitemap resubmission. Generate a new sitemap that contains only the final, canonical URLs. Strip out any redirected URLs, parameter variants, or staging URLs. Submit it in Search Console and monitor the 'Indexed' count for the next 7-14 days.
Layer three: bulk index verification. Use the Google Indexing API or a bulk checker tool to confirm each new URL is actually in the index. Filter out URLs that show 'Crawled - currently not indexed' — those need attention. A practical workflow for this step is detailed in Auditing Ghost Placements: The Bulk Google Index Checker Protocol, which covers how to batch-verify URLs and detect phantom index entries.
| Tool / Method | How It Works | Best For | Hidden Risk / Failure Mode |
|---|---|---|---|
| Google Search Console URL Inspection + Sitemap report | Submit sitemap; use URL Inspection for single URLs; check 'Indexed' count in Coverage report | Daily monitoring; authoritative source for crawl errors | Does not bulk-check more than 1000 URLs per sitemap; shows 'Crawled - currently not indexed' without telling you why; can mask soft 404s |
| Google Indexing API Programmatic batch submission | Send HTTP request for each URL; receive indexing status in response; requires OAuth | Large sites (10k+ URLs); automated CI/CD pipelines | Rate limited to 200 URLs per second per project; daily quota varies; if you send a URL that returns 4xx, it will be marked as 'not indexed' permanently until you re-crawl |
| Bulk Index Checker (Screaming Frog + Google Sheets) Custom script using SERP API or Google Search | Export URL list from sitemap; run through Screaming Frog's Indexability check or use a Google Sheets add-on that queries "site:URL" | Agency audits; mid-size sites (100-10k URLs); fast initial scan | Google's site: operator is not reliable for bulk checks — it returns approximate results and can be blocked by rate limiting; add-on vendors often have stale data or stop working after API changes |
| Server Log Analysis Check Googlebot hits on new vs old URLs | Parse raw server logs; filter by Googlebot user-agent; count hits per URL; compare old vs new URL patterns | Detecting crawl budget waste; verifying Googlebot actually found the new URLs | Requires log access and parsing tool (e.g., Logz.io, ELK); if your old URLs are still returning 200, Googlebot will keep crawling them and ignore new ones; silent crawl budget drain |
Full list from pre-migration sitemap or CMS export. Include all pages, not just top-level.
Crawl old URLs. Check for 301 to new URL. No chains, no 302s, no 404s.
Only final canonical new URLs. No redirects, no noindex, no params.
Submit sitemap. Wait 48h. Check Coverage report: submitted vs indexed count.
Use API or bulk checker. Flag all URLs that are 'not indexed' or 'crawled not indexed'.
For flagged URLs: request re-indexing via URL Inspection. Fix thin content or blocked resources.
We worked with a mid-size retailer that moved from Magento to Shopify. The old site had 12,000 product URLs with a flat structure like /product/1234. The new site used hierarchical URLs like /category/subcategory/product-name. After migration, the team resubmitted the sitemap and saw 11,800 of 12,000 URLs indexed in Search Console. A quick win, they thought. But when we ran a bulk index check using the Google Indexing API, we found that only 9,200 of those 11,800 were actually returning a 200 status and canonicalizing to themselves. The other 2,600 were indexed versions of old URLs that still returned 301 redirects — Google had indexed the redirect target as the old URL. Traffic dropped 25% in week two.
We fixed it by: (1) finding 342 old URLs that still returned 200 because the redirect rule missed them, (2) updating the sitemap to exclude any URL with a redirect rule, and (3) using URL Inspection to request re-indexing for the 2,600 mis-indexed URLs. After 10 days, the indexed count dropped to 9,800 (the correct number), and traffic recovered by week four. The key metric: we lost 2,600 false-positive index entries and gained 600 real ones.
Blocked URLs. Sometimes new URLs are blocked by robots.txt, either because the old rules were copied without update or because the staging environment is still being crawled. Always check robots.txt before and after migration.
Wrong filters in Search Console. The Coverage report has a filter for 'Submitted URLs not indexed'. But if you submitted the old sitemap by mistake, that filter shows data for the wrong URL set. Double-check the sitemap URL in the report.
Duplicate lists. If your CMS generates URLs with and without trailing slashes, you may submit duplicate entries. Bulk index checkers will flag those as separate URLs, inflating your not-indexed count.
Weak pages. Google may choose not to index thin pages — product descriptions with no original content, category pages with 20 words, or pages that are too similar to others. A migration does not fix weak content. You need to audit content quality before resubmission.
Slow index propagation. For very large sites (100k+ URLs), Google can take weeks to fully index the new URLs. During that window, you will see a mix of old and new URLs in the index. Do not panic. Focus on the trend: is the ratio of new indexed URLs increasing week over week?
Typically 5-14 days for most sites, but can extend to 4-6 weeks for large sites (100k+ URLs). The speed depends on crawl budget, redirect health, and sitemap quality. If you see zero new indexed URLs after 7 days, check for blocked resources or redirect chains.
Use the Google Indexing API with a batch script. It handles up to 200 URLs per second and returns status for each. For a one-time audit, a bulk checker tool (like Screaming Frog with Indexability check) works but may be rate-limited. Avoid manual site: queries for large lists.
Indirectly. Submit your new sitemap in Search Console and monitor the Coverage report. It shows the number of submitted URLs vs indexed. But it does not give you a per-URL status list. For per-URL status, you need the URL Inspection API or a bulk index checker.
This status means Google found the URL but chose not to index it, often due to duplicate content, thin pages, or low perceived value. First, check if the page has unique content and a self-referencing canonical. Second, verify that no conflicting noindex tags exist. Third, improve content quality and request re-indexing via URL Inspection.
Run a site:olddomain.com query in Google and count results. For precise data, export your old URL list and check each with the Google Indexing API or a bulk checker. If old URLs are still indexed and returning 301, they will eventually be replaced, but it can take weeks. You can speed it up by submitting removal requests for old URLs via Search Console.
Three big ones: (1) only checking the homepage, (2) assuming all submitted sitemap URLs are indexed, and (3) ignoring the 'Crawled - currently not indexed' status. Also, many teams forget to check that the new URLs are not blocked by robots.txt or that the old sitemap was replaced, not just updated.
Crawl your complete old URL list using a tool like Screaming Frog. Filter for any URL that does not return a 301. Common failures: 302 (soft redirect), 200 (no redirect), 404 (broken), or 301 chains longer than 3 hops. Fix each one. Then resubmit your new sitemap.
Yes, for any site over 500 URLs. Manual checking is impractical. Use a tool that queries the Google Indexing API directly for accuracy. Be aware of rate limits and daily quotas. Tools that use the site: operator are unreliable for bulk work. The protocol described in <a href="https://medium.com/@alexa.sam2026/auditing-ghost-placements-the-bulk-google-index-checker-protocol-60774d9bef72">this external resource</a> outlines a robust batch approach.
This is a double migration: URL structure change plus protocol change. Ensure all old HTTP URLs 301-redirect to the new HTTPS URLs. Submit the new HTTPS sitemap. In Search Console, add both the old HTTP and new HTTPS properties. Monitor the old property for crawl errors. Expect a temporary dip in indexed URLs as the switch propagates.
Set up a daily script that pulls the number of indexed URLs from the Search Console API for your new sitemap. Log the count to a spreadsheet. Also run a weekly bulk index check on a random sample of 500 new URLs to catch outliers. Watch for plateaus: if the indexed count does not increase for 5 days, investigate crawl errors or sitemap issues.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.