--- title: "Google Indexing Field Guide: How Google Picks What to Index" url: https://encited.com/blog/google-indexing-field-guide description: "How Google discovers, crawls, renders and indexes pages, how crawl budget and canonicals work, what each site type needs, and every Pages report status." lang: en --- Image: Google Indexing Field Guide: How Google Decides What to Index (https://encited.com/blog/thumbnails/how-to-get-pages-indexed-on-google.webp) # Google Indexing Field Guide: How Google Decides What to Index Jul 3, 2026 July 3, 2026 · Updated Sep 25, 2026 September 25, 2026 · by Aki from Encited How Google discovers, crawls, renders and indexes pages, how crawl budget and canonicals work, what each site type needs, and every Pages report status. Google indexes a page in four stages: it discovers the URL, crawls it, renders any JavaScript, and then decides whether the page earns a place in the index. Crawl budget sets how fast pages move through the first stages, and Google's choice of canonical and its judgment of quality settle the last one. Each status in the Search Console Pages report tells you which stage stopped a page. You've submitted the sitemap, requested indexing and waited, and some pages still aren't on Google. This field guide explains how Google makes those decisions, how crawling differs from indexing, what each type of site needs, and what every Pages report status means. For the practical steps alone, 10 ways to get Google to index your website (https://encited.com/blog/google-index) is the short version. ## Key takeaways - Crawling is Googlebot fetching a page, indexing is Google deciding to keep it, and a page can stall at discovery, crawling, rendering or indexing. - Crawl budget combines how many requests your server handles with how much Google wants your content, and most small sites hit the second limit. - Google treats your canonical tag as a hint and picks its own canonical when links, sitemaps and redirects disagree. - New domains usually wait two to eight months for normal indexing, and three to five real links plus strong first pages shorten it. - Brochure sites stall on missing links, empty JavaScript pages or near-identical location pages. - Directories and programmatic sites should release linked clusters of about 25 pages and give every page unique data. - Blogs index fastest when each post is linked from an older one and the publishing pace matches the crawl budget. - Online stores need unique product copy, strict control of filter URLs and a plan for discontinued products. - JavaScript sites need server-side rendering, static generation or pre-rendering, because Google renders late and most AI crawlers never do. - Many Pages report statuses, such as Page with redirect, Alternate page with proper canonical tag and an intentional Excluded by noindex tag, mean the site is working as intended. - Important pages stuck for weeks are the ones that need action. ## How Google indexing works Getting a page into Google's index is a pipeline with four stages. A page can stall at any of them, and Search Console's status tells you which one. ### Discovery Google learns that the URL exists. It finds URLs through links on pages it already knows, through XML sitemaps, and through direct requests in Search Console. Discovery on its own gets a page nowhere. Google knows about billions of URLs it never crawls, and a page stuck at this stage shows up as "Discovered - currently not indexed". ### Crawling Googlebot fetches the page. When that happens depends on crawl budget, which the next section covers. Google crawls with Googlebot Smartphone for almost every site, because it indexes the mobile version of your pages. Whatever a phone visitor doesn't get, Google usually doesn't get either. ### Rendering If the page builds its content with JavaScript, there's an extra stage. Googlebot reads the raw HTML first. Pages that need JavaScript go into a separate rendering queue, where a headless Chrome runs the scripts and captures the final page. That queue can take hours, days or longer, it's rate-limited, and one script error or failed API call can leave the render empty. Many single-page apps and sites made with AI builders stall right here, with nothing in the first HTML for Google to judge. ### Indexing Google evaluates what it crawled and rendered, then decides whether the page earns a place. Duplicates get grouped under one canonical URL. Everything else is weighed on whether it adds anything the index lacks. A page that fails here shows up as "Crawled - currently not indexed" or as one of the duplicate statuses. Google's John Mueller has said many times that it's normal for part of every site to stay out. ## How crawl budget works Crawl budget is the number of pages Googlebot is willing and able to fetch from your site in a given period. Google describes it as two things combined. Crawl capacity is how many requests your server handles without slowing down or failing. Crawl demand is how much Google wants your content, based on how popular the site is, what previous crawls found, how often you publish something new, and how much the site is trusted on its topic. Google's own documentation says crawl budget is mainly a concern for sites with more than a million pages, or more than 10,000 pages that change daily. In practice, much smaller sites run into crawl demand limits, because demand is a quality judgment, and a new or thin site hasn't earned much of it. Crawl budget gets spent on every URL Googlebot requests. Parameter combinations from filters and sorting, internal search results, redirect chains, soft 404s and duplicate paths all take crawls. On a site where those make up most of the requests, the pages you care about wait. Crawl stats in Search Console, under Settings, shows how many requests Googlebot makes each day, the average response time, and the share of requests by response code and file type. ## How Google picks a canonical When several URLs show the same or very similar content, Google indexes one and treats the rest as duplicates. The one it keeps is the canonical. Google weighs your `rel="canonical"` tag, redirects, internal links, the URLs in your sitemap, and whether the page is served over https, then makes its own choice. Your canonical tag is a strong hint that Google can overrule. It overrules it when the other signals point elsewhere, for example when your internal links and sitemap use the www version and your canonical tag uses the bare domain, or when two pages are so alike that Google thinks another one represents the content better. URL Inspection shows both the canonical you declared and the one Google selected. When they differ, align every signal on the version you want, or make the pages different enough to stand alone. ## Why new sites index slowly The pattern on new domains repeats. The site launches, the sitemap goes in, the homepage gets indexed within a week or two, and everything else stalls. Sometimes the homepage itself takes weeks. The cause is trust. A new domain has no history, no links and no visitors, so Google has little to go on when deciding whether forty new pages are a real business or one of the thousands of generated sites launched that day. AI tools made publishing nearly free, the volume of new pages has grown accordingly, and Google responded by crawling new domains cautiously and indexing them reluctantly. SEO people have called this the sandbox for twenty years. Google says no literal sandbox filter exists, but the behavior is easy to see: new domains index and rank slowly, and most people report a window of two to eight months depending on the niche. Two things shorten it: external links and strong first pages. The biggest single factor is a handful of real external links. Three to five links from sites Google already crawls give Googlebot a path to your domain and show that the site exists outside your own sitemap. Your Google Business Profile, industry directories and associations, a partner that lists its clients, a launch post in a community your buyers use and a founder profile all count. Link packages and mass-produced blog networks hurt, because they're exactly what Google's filters look for. The first pages Google reads shape its opinion of the whole domain, so publish your strongest pages first. A new site with a useful homepage, clear service pages and two or three substantial articles gets indexed faster than one with fifty thin pages. Clean technical signals matter as well: one https version of the site, no leftover staging noindex, a robots.txt that doesn't block anything important, clean 200 responses and a fast server. ## How to get a brochure site indexed A brochure site is the classic small business site: a homepage, an about page, services, contact, and maybe a few location pages. Five to twenty pages in total. Built as plain HTML, these are the easiest sites on the web to index. When one stalls, the cause is usually one of three things. The first is a new domain with no links, and everything in the section above applies. For a local business, the Google Business Profile link plus two or three local citations, such as the chamber of commerce, an industry association or a local listings site, is usually enough to get Google crawling. The second is a site built with a JavaScript framework or an AI website builder that sends Google an empty page. Run your homepage through the free Googlebot Simulator (https://encited.com/free-tools/crawler-simulator/googlebot-simulator) to see the text a crawler receives. If it's missing, the site needs to send its content as HTML, which the JavaScript section below covers. The third is pages that say almost the same thing. Ten location pages that differ only by the city name tend to get grouped as duplicates or skipped as thin. Give each page something real about that place or service: who you serve there, the work you've done, prices, photos or local details. With clean HTML, a link path from the homepage to every page, a submitted sitemap, a manual request for each page and a few external links, a small site usually gets fully indexed within a few weeks. ## How to get a directory or programmatic site indexed Directories, listing sites and programmatic SEO projects ask Google to index hundreds or thousands of similar pages from a domain that usually hasn't earned that much trust. That's where indexing strategy matters most. The common failure is launching 3,000 pages in the sitemap on day one, getting 50 indexed, and watching the other 2,950 sit in "Discovered - currently not indexed" for months. Google looked at the URL pattern, sampled a few pages, decided the pattern wasn't worth its crawls, and deprioritized the whole set. That judgment is slow to reverse. Release in semantic batches. Group pages into complete clusters, such as one city, one category or one vertical, and release one cluster at a time. Start with around 25 pages. If roughly 80% get indexed within a few weeks, release the next cluster and grow the batch size while the indexing rate holds. If the first batch indexes poorly, fix the template before publishing more, because every unindexed page from it lowers Google's opinion of the pattern. A complete, interlinked cluster, like every plumber in one city linked from a city hub, reads as a finished section of a site. Twenty-five random pages spread across the taxonomy read as samples from a page generator. Internal links do most of the work on a directory. They're how Google finds listings, how it judges their importance and how crawl priority flows through the site. Every listing needs a link from at least one hub page, and hubs need links from the homepage or the main navigation. Pages reachable only through the sitemap are the first to be ignored. Breadcrumbs, related-listing modules and HTML hub pages that list a whole cluster are worth more here than on any other type of site. Links inside the content, with descriptive anchor text such as "emergency plumbers in Austin", carry more weight than navigation links and tell Google what the target is about. Every page needs a reason to exist. The question Google is asking about each programmatic page is whether it holds information that isn't already on another page of this site or a hundred others. Unique data fields, reviews, real descriptions, photos and structured data answer it. A listing with a name, an address and boilerplate ends up in "Crawled - currently not indexed". Weak pages also weigh on the domain as a whole, which is why pruning them often improves indexing for the pages that remain. Control the URL space as well. Faceted filters, sort orders and internal search can generate near-endless URL variations. Block the combinations you don't want crawled in robots.txt, canonicalize variants to the clean version, and keep the sitemap to canonical URLs that return a 200 status. During the first months, avoid the moves that make a site look like spam: thousands of near-identical pages at once, a huge sitemap, automated indexing requests and bought links. Scale up after your indexing rate shows that Google trusts the pattern. ## How to get a blog indexed Fresh articles on an established domain are exactly what Googlebot is built to pick up, so blogs have the most direct path. The two variables are how much the domain is already trusted and how often you publish. On a blog that already has indexed content and traffic, new posts usually get indexed within hours to a few days. Publishing on a steady schedule trains Googlebot to come back often, because crawl frequency follows publishing frequency. Request indexing for each new post in URL Inspection anyway; it takes thirty seconds and often cuts the wait to hours. Link every new post from at least one older, indexed post, and show recent posts on your homepage so Googlebot finds them on its next visit. Match your publishing pace to the crawl budget you've earned. A young blog publishing ten AI-assisted posts a day will see most of them pile up unindexed, and a large backlog of unindexed pages counts against the domain. If your indexing rate is dropping, slow down and consolidate. For most new blogs, two or three strong posts a week is the realistic ceiling. A brand-new blog is a new site first, so the new-domain section applies: build the first external links, publish your strongest articles first, and expect the first posts to take weeks. Keep older posts current. Google recrawls pages it considers alive, and updating important older posts and linking to them from new ones keeps them crawled. Before expanding an old post, check that it won't start competing with another of your posts for the same searches. Two posts on one topic split the signals and usually both rank worse than one consolidated page would. ## How to get an e-commerce site indexed Online stores combine most of the hard parts: many pages, heavy templates, faceted navigation and often a JavaScript storefront. The directory advice above mostly applies, with four points specific to stores. Product pages need unique content. Manufacturer descriptions copied across your store and fifty competitors are the most common cause of duplicate and "Crawled - currently not indexed" statuses on product URLs. Rewrite the descriptions for your important products and add your own photos, specifications and reviews. Faceted navigation needs strict control. Color, size and sort parameters can multiply the number of crawlable URLs a hundredfold. Pick the few facet pages with real search demand that deserve indexing, canonicalize or block the rest, and keep them out of the sitemap. Discontinued and out-of-stock products need a deliberate decision. A discontinued product with a replacement should 301 to it. A product that's gone should return a real 404 or 410. An "unavailable" message served with a 200 status is a soft 404, and Google keeps recrawling it. The storefront has to render. Headless storefronts and heavily scripted themes often load product names, prices, descriptions and links to related products with JavaScript. Check a product page with the Googlebot Simulator (https://encited.com/free-tools/crawler-simulator/googlebot-simulator) and confirm those appear in the HTML a crawler receives. ## How to get a JavaScript site indexed A site can have good content, a clean sitemap and correct canonicals and still have almost nothing indexed months after launch, because Google can't see the content. On sites built with React, Vue, Angular or an AI website builder, the first HTML Googlebot receives is often a nearly empty file with a script tag. While those pages wait for rendering, Google judges the site by the empty versions it can read. Empty pages earn little crawl demand, fewer pages get crawled as a result, and fewer reach the rendering queue. New domains with no trust to spend feel this hardest. Google is the most patient crawler here. Bing renders far less JavaScript, social platforms almost never do, and most AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, read the raw HTML and move on (https://encited.com/blog/how-to-get-pages-indexed-by-openai-chatgpt). A site that renders in the browser gets indexed slowly by Google and stays unreadable to most of the other systems that decide whether people find it. There are three ways to send the content as HTML. A framework that generates or server-renders pages, such as Next.js, Astro, Nuxt or SvelteKit, is the cleanest choice for a new project. Adding server-side rendering to an existing app works but is a large engineering project. Pre-rendering serves crawlers a fully rendered, cached copy of each page while visitors get the normal app, and Google accepts it as long as the copy matches what users see. The SSR vs SSG vs pre-rendering guide (https://encited.com/blog/ssr-vs-pre-rendering) compares what a crawler receives from each. Encited pre-renders your pages for every search and AI crawler after a DNS change that takes about five minutes, with no code changes. Pre-rendered pages on Encited get indexed roughly 5x faster than the same sites served as JavaScript shells, and the crawl logs (https://encited.com/docs/crawl-logs/read-your-crawl-log) show every crawler visit and what it received. Image: Recent crawler visits log showing Googlebot hits per page with timestamps and cache-hit status (https://encited.com/marketing/product/crawl-logs-2802x1706.webp) Click to zoom Is Google reading an empty page on your site? Encited pre-renders your JavaScript pages for every search and AI crawler. DNS setup in about 5 minutes, plans from $19/mo, 14-day free trial. Start a free trial: https://encited.com/ Image: Encited's pre-rendered pages list showing rendered HTML snapshots served to crawlers (https://encited.com/marketing/product/pre-render-pages-2906x1964.webp) ## Every Search Console indexing status explained The Pages report, under Indexing in Search Console, lists every reason Google gives for leaving a URL out. Each entry below says what the status means and whether it needs a fix. ### Server error (5xx) Your server returned a 500-level error when Googlebot requested the page. This always needs fixing. Check hosting capacity, firewall and bot-protection rules that may block Googlebot, and whether the errors cluster at certain times of day. Persistent server errors also lower crawl capacity for the whole site. ### Redirect error Google couldn't follow the redirect: a loop, a chain that's too long, or a redirect to a dead or malformed URL. Trace the chain with the Googlebot Simulator (https://encited.com/free-tools/crawler-simulator/googlebot-simulator) or curl and make every redirect a single hop to a page that returns 200. ### Blocked by robots.txt Googlebot was told not to fetch the page. If that's intended, leave it. If you want the page indexed, remove the `Disallow` rule. Robots.txt blocks crawling, and it doesn't reliably prevent indexing, because a blocked URL can still be indexed from external links and shown without a description. Google also has to crawl a page to see a noindex tag on it, so blocking a page in robots.txt and adding noindex at the same time leaves the noindex unread. ### Blocked due to unauthorized request (401) or access forbidden (403) The page requires a login, or the server refused Googlebot. That's normal for account areas and staging environments. On public pages it usually means a CDN, firewall or bot-protection layer is challenging Googlebot. If the URLs shouldn't be public, find out how Google found them and remove the links or sitemap entries that expose them. ### Not found (404) The page doesn't exist. That's fine for content you removed, and Google crawls these less over time. If pages that should exist appear here, fix the links pointing to them or restore the pages, and 301 removed pages that have a replacement. ### Soft 404 The page tells visitors there's nothing here but returns a 200 status, or it's so thin that Google treats it as empty. It's common on empty category and tag pages and on out-of-stock products. Return a real 404 or 410, redirect to a relevant page, or add real content. Soft 404s keep getting recrawled and use up crawl budget. ### Blocked due to other 4xx issue The server returned a client error other than the ones above, such as 400 or 429. Check the actual status code the URL returns and fix it. A 429 means rate limiting is hitting Googlebot. ### Excluded by noindex tag Google found a noindex directive and followed it. That's correct when it's intended. The common accident is a staging noindex that shipped to production, or an SEO plugin setting applied to too many pages. Review this list regularly, because wanted pages appearing here is one of the most frequent self-inflicted indexing problems. ### Page with redirect The URL redirects elsewhere, so this URL won't be indexed and the destination might be. It's almost always healthy: http to https, trailing-slash normalization, old URLs pointing to new ones. Only look closer if a page you expect to be a final destination is redirecting. ### Alternate page with proper canonical tag The URL names another URL as its canonical, and Google agrees. These are usually parameter variants, tracking URLs or mobile alternates. It's working as intended and needs no action. ### Duplicate without user-selected canonical Google found duplicates of this page, you didn't declare a canonical, and Google picked one itself. Add canonical tags so the version you want is the version that gets indexed. URL Inspection shows which URL Google chose. ### Duplicate, Google chose different canonical than user You declared a canonical and Google overruled it, because it thinks another URL represents the content better. The pages are usually too similar, or your internal links and sitemap point to a different version than your canonical tags do. Make the pages distinct, or align every signal on one version. ### Crawled - currently not indexed Google fetched the page and decided to leave it out. The usual causes are an empty rendered page, a canonical pointing elsewhere, duplicate content, thin content, weak internal links or a site-wide quality judgment. The guide to fixing Crawled - currently not indexed (https://encited.com/blog/crawled-currently-not-indexed) walks through each cause. ### Discovered - currently not indexed Google knows the URL and hasn't crawled it. The cause is low crawl demand, crawl waste on other URLs, weak internal links or a slow server. The guide to fixing Discovered - currently not indexed (https://encited.com/blog/discovered-currently-not-indexed) covers each one. ### Blocked by page removal tool Someone filed a temporary removal request for this URL in Search Console. Removals last about six months. If the page shouldn't be hidden, open the Removals report and cancel the request. ### Indexed, though blocked by robots.txt The page was indexed from external links even though Googlebot can't crawl it, so it appears in results with no useful description. To keep it out, remove the robots.txt block and add a noindex tag, since Google has to crawl the page to see the noindex. To have it indexed properly, remove the block. ## How to read the Pages report Almost every healthy site has plenty of URLs in the "not indexed" part of the report, and much of it is Google correctly handling redirects, canonicals and intentional noindex tags. Read it as a description of your site. The goal is that every page you care about is indexed and everything else is excluded for a reason you chose. On content sites, having 5 to 15% of known URLs excluded is unremarkable. Important pages stuck for weeks are what deserve attention. When you're tracking hundreds of pages, Index Rush (https://encited.com/get-indexed-faster) pulls the live status of every sitemap URL from Search Console, shows the issues on each unindexed page, and recommends which pages to request first. ## FAQ ### Do backlinks affect indexing or only ranking? Both. Links are a path for discovery and a signal for crawl priority. A new domain with no external links and the same domain with five real ones often behave very differently: months of "Discovered - currently not indexed" in the first case, normal indexing in the second. ### My page is indexed but doesn't rank. Is that an indexing problem? No. Indexing puts the page in Google's library, and ranking decides whether it gets recommended. When URL Inspection says "URL is on Google", the remaining work is relevance, content quality and links. ### What is mobile-first indexing? Google crawls and indexes the mobile version of your pages with Googlebot Smartphone. Content, links and structured data that only appear on the desktop version may not be indexed. Make sure the mobile page carries the same content as the desktop page. ### Should I use the Indexing API or paid indexing tools? Google's Indexing API is documented for job postings and livestream event pages. Tools that push ordinary pages through it sometimes get them crawled, but it does nothing for the trust problem underneath, and pages indexed that way without merit tend to fall back out. The same effort spent on real links and internal linking lasts. ### Can Google index a site that renders in the browser? Eventually and partially. Google renders JavaScript, but the delays and failures in the rendering queue make it an unreliable foundation, and Bing, social platforms and most AI crawlers render far less. Sending the content as HTML through static generation, server-side rendering or pre-rendering fixes it for every crawler. ## Get discovered anywhere search happens Readable, citable, outranking pages. Book a demo: https://encited.com/book-a-demo Get started: https://encited.com/?authModal=true Image: Avatar (https://replymaven.com/api/uploads/Su6Q0nK63iZW0ZFJBRNWpmRuppVDLuab/98df5262-f992-45fa-a750-00a6c919baf9.png) How can we help? Get instant answers to your questions or leave a message for an engineer will reach out Ask AI about Encited See our docs (https://lovablehtml.com/docs) Contact support Leave a message We'll get back to you soon Ask AI about Encited Product questions, troubleshooting and Thinking Image: Preview Drop an image to attach Powered by ReplyMaven (https://replymaven.com/) ## Structured data ```json [ { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": " https://encited.com/" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": " https://encited.com/blog" }, { "@type": "ListItem", "position": 3, "name": "Google Indexing Field Guide: How Google Decides What to Index", "item": " https://encited.com/blog/google-indexing-field-guide" } ] }, { "@context": "https://schema.org", "@type": "Article", "headline": "Google Indexing Field Guide: How Google Decides What to Index", "description": "How Google discovers, crawls, renders and indexes pages, how crawl budget and canonicals work, what each site type needs, and every Pages report status.", "image": " https://encited.com/blog/thumbnails/how-to-get-pages-indexed-on-google.webp", "datePublished": "2026-07-03", "dateModified": "2026-09-25", "author": { "@type": "Person", "name": "Aki from Encited" }, "publisher": { "@type": "Organization", "name": "Encited.com", "alternateName": "LovableHTML.com", "logo": { "@type": "ImageObject", "url": "https://encited.com/brand/logo.svg" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": " https://encited.com/blog/google-indexing-field-guide" } }, { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "Do backlinks affect indexing or only ranking?", "acceptedAnswer": { "@type": "Answer", "text": "Both. Links are a path for discovery and a signal for crawl priority. A new domain with no external links and the same domain with five real ones often behave very differently: months of Discovered - currently not indexed in the first case, normal indexing in the second." } }, { "@type": "Question", "name": "My page is indexed but doesn't rank. Is that an indexing problem?", "acceptedAnswer": { "@type": "Answer", "text": "No. Indexing puts the page in Google's library, and ranking decides whether it gets recommended. When URL Inspection says URL is on Google, the remaining work is relevance, content quality and links." } }, { "@type": "Question", "name": "What is mobile-first indexing?", "acceptedAnswer": { "@type": "Answer", "text": "Google crawls and indexes the mobile version of your pages with Googlebot Smartphone. Content, links and structured data that only appear on the desktop version may not be indexed. Make sure the mobile page carries the same content as the desktop page." } }, { "@type": "Question", "name": "Should I use the Indexing API or paid indexing tools?", "acceptedAnswer": { "@type": "Answer", "text": "Google's Indexing API is documented for job postings and livestream event pages. Tools that push ordinary pages through it sometimes get them crawled, but it does nothing for the trust problem underneath, and pages indexed that way without merit tend to fall back out. The same effort spent on real links and internal linking lasts." } }, { "@type": "Question", "name": "Can Google index a site that renders in the browser?", "acceptedAnswer": { "@type": "Answer", "text": "Eventually and partially. Google renders JavaScript, but the delays and failures in the rendering queue make it an unreliable foundation, and Bing, social platforms and most AI crawlers render far less. Sending the content as HTML through static generation, server-side rendering or pre-rendering fixes it for every crawler." } } ] } ] ```