Appearance

Language

Build a personal website from 0

Why Is My Website Not Showing Up on Google? A 2026 Indexing and Launch Checklist

Site not showing up on Google? Use site: and Search Console's URL Inspection to see why it isn't indexed, fix it by report status, plus a launch checklist.

Kelvin Wang · · 17 min read

Key points

  1. A new site missing from Google is usually just new: Google says it can take a week or so before a new page or site is crawled and indexed, and there's no way to force it.
  2. Search Console's URL Inspection tells you why. "URL is unknown to Google" and "Discovered – currently not indexed" mostly mean wait; noindex, a robots.txt block or a 403 are yours to fix.
  3. The usual self-inflicted causes: a staging noindex left on, robots.txt blocking the whole site, an AI-built single-page app that serves empty HTML, and canonicals pointing at the wrong URL.
  4. Indexed means eligible to appear, not ranked. Google's AI Overviews need no extra files or markup either, and llms.txt is optional.

A site that doesn’t show up on Google is in one of two cases: Google hasn’t found the page yet, or it found it and didn’t index it. Search Console tells you which case you’re in, so start there and leave the titles and copy for later.

My site, kelvinwang.info, is an Astro site in Chinese and English, and on 16 November 2026 its sitemap listed 238 URLs. The troubleshooting examples are states it has actually been in, and the launch checklist names the files it uses.

Your site is live but not on Google? Run these three checks first

  1. Search Google for site:yourdomain. If any page shows up, the site is in the index and the problem is with individual pages. Google notes that site: doesn’t necessarily return every indexed URL 1, so use it to check presence, not to count.
  2. Work out how long the site has been live. Google says it can take a week or so before a new page or site starts being crawled and indexed 2, and a few weeks for Google to notice a new site or changes to one 1. The most common reason a site isn’t indexed is that it’s simply too new 3.
  3. Run the page through URL Inspection in Search Console. It reports “URL is on Google” or “URL is not on Google”, and the latter comes with a reason.

How should you set up Search Console so the sitemap actually counts?

A property that doesn’t match your URLs rejects the sitemap

Search Console has two property types 4:

  • Domain property (example.com): covers every subdomain and both http and https, verified with a DNS TXT record. For adding DNS records, see domains and DNS.
  • URL-prefix property (https://www.example.com/): covers only URLs under that exact prefix.

Sitemaps belong to the property you submit them in: a sitemap in one property isn’t visible in another, and you need owner permission on the property to submit one 5. The classic mistake is a host mismatch: the sitemap file lives at https://www.example.com/ but lists https://example.com/… URLs, or mixes http and https. The Sitemaps report flags this as URL not allowed or a path mismatch; the URLs in a sitemap have to start with exactly the same host as the sitemap’s own location 5. A domain property avoids most of this.

The three Sitemaps report statuses

StatusMeaningWhat to do
SuccessThe sitemap was loaded and processed; its URLs are queued for crawlingNothing
Has errorsSome entries failed; the URLs that parsed are still queuedOpen it for the error details, fix them and resubmit
Couldn't fetchGoogle couldn't retrieve the fileCommon causes are a robots.txt block, a 404 or a temporary server problem; fix it and resubmit
  • “Discovered pages” is the number of URLs parsed from the sitemap, not the number indexed. Google states that a URL found in a sitemap isn’t guaranteed to be crawled or indexed 5.
  • To see how many sitemap URLs are indexed, filter the Page indexing report by that sitemap.
  • “Last read” is the last time Google successfully fetched the sitemap.

URL Inspection and “Request indexing”

URL Inspection shows the version in Google’s index by default, not the page as it is now. Test live URL fetches the current page on the spot, and the two can disagree 6.

  • Request indexing has a daily limit. It typically takes a day or so and can take a week or two, and it doesn’t guarantee indexing 6.
  • You can’t request indexing for a page the live test says can’t be indexed. Fix the cause first.
  • For many pages, submit a sitemap instead of requesting them one by one 6.
  • Passing the live test only means Google can reach the page, not that it will index it 6.

An example from my site: one post was last crawled on 3 November, I changed it on 11 November, and URL Inspection still showed the old version. Google indexes the copy it last fetched; changing the site doesn’t update the index by itself.

What does each Page indexing report status mean?

The Pages entry in the Search Console sidebar is the Page indexing report: it lists the URLs that aren’t indexed and why. Google says not to expect every URL to be indexed; most sites have a few unindexed pages, but all the important ones should be indexed 2.

StatusMeaningWhat to do
URL is unknown to GoogleGoogle has never seen the URL (shown in URL Inspection)Make sure it's in the sitemap and linked from your own pages; you can request indexing
Discovered – currently not indexedGoogle knows the URL but hasn't crawled it, usually because crawling was expected to overload the site and was rescheduled, so there's no last crawl dateMostly wait; make sure important pages have internal links
Crawled – currently not indexedCrawled but not indexed this time; Google calls it a common message for many sitesInspect only the important pages
URL marked 'noindex'Google found a noindex directive when it tried to index the pageRemove noindex if you want it indexed
Blocked by robots.txtrobots.txt doesn't let Google crawl the pageLift the block if you want it indexed; to keep a page out of the index, use noindex rather than robots.txt
Duplicate, Google chose different canonical than userYou declared a canonical, Google thinks another URL is betterCompare both canonicals in URL Inspection and check whether the pages really are duplicates
Alternate page with proper canonical tagThe page correctly points to an indexed canonicalNothing
Page with redirectA redirecting URL isn't indexed itselfNormal; check that the target is indexed
Redirect errorChain too long, a loop, a URL too long, or a bad or empty URL in the chainRedirect straight to the final URL
Soft 404The page returns 200 but looks like an error page or is nearly emptyReturn a real 404 for pages that don't exist; for a single-page app, server-render or prerender so the HTML has content
Not found (404)The URL returns 404Fix only the ones you link to or list in the sitemap; for moved pages, 301 to the new URL
Blocked due to access forbidden (403)The server refused GooglebotCheck your firewall, bot protection and password protection
Server error (5xx)The server failed when the page was requestedCheck server logs; URLs that keep failing are dropped from the index

Meanings and actions are summarized from Google’s help for the Page indexing report and URL Inspection 2 6; labels may differ slightly in your interface. Once fixed, use Validate fix in the report; Google says validation usually finishes within about two weeks, sometimes longer 2.

Three states my own site has been in

  • “URL is unknown to Google” for weeks. Some of the exchange pages I launched on 26 October 2026 sat in the sitemap for weeks while URL Inspection still said Google didn’t know them. A sitemap only tells Google the URLs exist; Google schedules when to fetch them.
  • Googlebot hitting old URLs and getting 404s. Some post URLs changed when I moved the posts from a database to files, and one URL had been published with a typo; Googlebot kept coming back to all of them. On 16 November 2026 I gave them 301s to the current posts; Google treats a 301 as a strong signal that the redirect target should be processed 7, and readers following old links no longer land on an error page.
  • A last crawl date from before the update. That’s the post from the previous section: the site changed, the index didn’t.

What are the most common self-inflicted causes?

If you’ve waited a long time, or the report shows one of the statuses you have to fix yourself, it’s usually one of these:

  • A staging noindex shipped to production. A <meta name="robots" content="noindex"> tag or X-Robots-Tag header added during development and never removed. Removing noindex with JavaScript after load isn’t reliable either: when Google sees noindex it may skip rendering and never run that script 8.
  • robots.txt blocking the whole site. Disallow: / blocks everything, and plenty of site builders add it while a site is “in development”.
  • Content that exists only after JavaScript runs. AI site builders often produce a single-page app (SPA) whose server HTML is nearly empty; the content appears only once the browser runs JavaScript. Google queues pages for rendering, but it only sees what’s in the rendered HTML, and it recommends server-side rendering or pre-rendering because not all bots can run JavaScript 8. An error page that returns 200 also gets flagged as a soft 404 7.
  • Wrong canonicals. If every page’s <link rel="canonical"> points at the home page or at a staging domain, Google treats the pages as duplicates. Canonicals, redirects and sitemaps are all signals; Google makes the final call on which URL is canonical 9.
  • The preview domain got indexed. If your host’s default preview URL has no noindex and something links to it, Google may index that instead of your domain. Add noindex to the preview URL, or 301 it to your real domain.
  • Nothing links to the page. Google mainly finds new pages through links on pages it has already crawled 10. A page that’s only in the sitemap, with no internal links, is missing that path.
  • A firewall or bot protection blocking Googlebot. If the report shows a 403 or 5xx, check this. Google says 4xx errors such as 403 and 404 cause indexed URLs to be dropped, and so do persistent 5xx and 429 responses; don’t use 401 or 403 to limit crawl rate 7. To allow Googlebot, confirm requests with a reverse DNS lookup that resolves to googlebot.com, or match against Google’s published IP ranges; google.com and googleusercontent.com belong to Google’s other crawlers and user-triggered fetchers, not Googlebot 11. Anyone can set a User-Agent, so checking that alone isn’t enough.

What can’t you force Google to do, and when should you just wait?

  • Indexing and ranking are different things. “URL is on Google” means the page is eligible to appear in results, not that it will 6, and Google doesn’t guarantee that any site gets indexed at all 10.
  • There’s no way to force an update 3. Request indexing has a daily limit, and a sitemap is a hint. Requesting the same URL again won’t get it crawled any faster 21.
  • Changes can take anywhere from a few hours to several months to show, and Google suggests waiting a few weeks before judging whether a change worked 10.
  • Wait when: the live test passes, there’s no noindex or block, the status is “URL is unknown to Google” or “Discovered – currently not indexed”, the sitemap is submitted and the page has internal links. You’ve done what you can.
  • Don’t wait when: the status is noindex, a robots.txt block, 403, 5xx or a redirect error, or a site: search turns up your preview domain. None of these fix themselves.

Not live yet? Eight things to set up on launch day

ItemWhat it doesFile on my site
robots.txtLets crawlers in and points to the sitemappublic/robots.txt
SitemapLists every public URLsrc/pages/sitemap.xml.ts
Search ConsoleVerifies the site; you submit the sitemap therePUBLIC_GSC_VERIFICATION variable → meta tag
Titles and descriptionsA unique title, description and canonical per pagesrc/layouts/PublicLayout.astro
Share imagesAn og:image for every postsrc/integrations/og-images.ts
Structured dataJSON-LD: Person, WebSite, BlogPosting, BreadcrumbListsrc/lib/seo.ts, src/pages/[lang]/blog/[...slug].astro
hreflangChinese and English versions point at each otherhreflangLinks in src/i18n/config.ts
llms.txtA site map for AI tools (optional)src/pages/llms.txt.ts

Step 1: let crawlers in

How robots.txt and noindex shut Google out is covered under self-inflicted causes. Here’s the robots.txt at my site’s root:

User-agent: *
Allow: /
Disallow: /go/

Sitemap: https://kelvinwang.info/sitemap.xml
  • Everything is allowed except /go/, the referral-link redirects, which have no content of their own and also send a noindex header.
  • The last line tells crawlers where the sitemap is. Google’s docs list this as one way to submit a sitemap 12.
  • There are no separate rules for AI crawlers, so Googlebot, OAI-SearchBot and the rest are all allowed.

My staging environment lives on a separate subdomain, and the Worker adds a noindex header to its responses so test pages never get indexed.

Step 2: generate a sitemap

A sitemap is an XML file listing the URLs you want indexed. Google’s rules 12:

  • Up to 50,000 URLs or 50 MB (uncompressed) per file. Larger sites split it and tie the parts together with a sitemap index.
  • Use full absolute URLs, including https://.
  • Google uses <lastmod> only if it’s consistently accurate. Changes to the main content, structured data or links count; bumping a copyright year doesn’t.
  • Google ignores <priority> and <changefreq>, so skip them.
  • Submitting a sitemap is a hint. Google doesn’t promise to crawl from it.

Astro has an official @astrojs/sitemap package, and that’s all a first site needs. My site has its own src/pages/sitemap.xml.ts for two reasons:

  1. Each URL carries its language versions. Every <url> lists both the Chinese and English versions with xhtml:link, one of the three hreflang methods Google supports 13.
  2. lastmod only records real changes. Posts use updatedAt from their frontmatter. The exchange pages used to use a data-sync timestamp, but the sync runs every four hours whether or not the page changes, so on 5 November 2026 I removed it; those pages carry no lastmod.

The simplified core:

export const prerender = true;

export const GET: APIRoute = async () => {
  const entry = (loc, alts, lastmod) =>
    `<url><loc>${loc}</loc>${lastmod ? `<lastmod>${lastmod.toISOString()}</lastmod>` : ''}`
    + hreflangLinks(alts).map((a) => `<xhtml:link rel="alternate" hreflang="${a.hreflang}" href="${a.href}"/>`).join('')
    + '</url>';
  const all = await getPublishedPosts();
  const posts = all.map((p) => entry(pageUrl(postPath(p)), alternatesOf(p, all), p.data.updatedAt));
  return new Response(
    `<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">${posts.join('')}</urlset>`,
    { headers: { 'content-type': 'application/xml; charset=utf-8' } },
  );
};

prerender = true makes it a static file generated at build time. Drafts (draft: true in frontmatter) never appear in it.

Step 3: verify the site in Google Search Console and submit the sitemap

Search Console is Google’s free tool for seeing which pages are indexed, which searches show your site and what’s going wrong.

Pick a property type. The two types are compared earlier; for a first site, use a domain property, which covers www and non-www in one.

A domain property can only be verified with a DNS record; the HTML tag is for URL-prefix properties. My site uses the HTML tag. The code lives in the build variable PUBLIC_GSC_VERIFICATION, and PublicLayout.astro prints the tag only when it’s set:

{gscVerification && <meta name="google-site-verification" content={gscVerification} />}

Keeping the code out of the source means re-verifying or switching accounts is just a variable change.

Submit the sitemap. Once verified, open Sitemaps in the left menu, enter sitemap.xml and submit. The report shows when Google last read it and any errors 12.

Request indexing. You can request indexing for an individual new post in URL Inspection; the limits are covered above.

Step 4: titles and descriptions for every page

<title>: Google asks for descriptive, concise titles that differ from page to page, without repeated keywords; a brand name can go at the start or end behind a separator. Google sets no length limit, but truncates title links in results to fit the device width 14.

Meta description: Google builds snippets mainly from the page content, and uses the meta description when it describes the page more accurately. Write a unique description per page that summarizes it in real sentences, not a keyword list 15.

What my site does:

  • A post’s <title> is its seoTitle plus my name: title if there’s no seoTitle, then ” — ” and my name.
  • The description falls back from seoDescription to description, and only then to the start of the body. So every new post must have a description.
  • Every page has a <link rel="canonical">, and URLs always end in a slash (handled by pageUrl()), so one page never lives at two addresses.
  • Each post gets a 1200×630 share image generated after the build, used as its og:image, so links shared on social media get a preview.

My own rule for titles: use the words searchers actually type, name the brand, add the year (2026) where it fits, and put the question the post answers into the title.

Step 5: structured data

Structured data uses schema.org vocabulary to tell machines, in a form they can parse, that a page is an article, who wrote it and when. Google uses it to understand pages, and some types can appear as rich results.

Articles use Article, NewsArticle or BlogPosting. Google has no required properties and recommends headline, image, datePublished, dateModified and author (with name and url), with dates in ISO 8601 including a time zone 16.

The JSON-LD on my post pages (src/pages/[lang]/blog/[...slug].astro, fields excerpted):

const jsonLd = {
  '@context': 'https://schema.org',
  '@type': 'BlogPosting',
  headline: d.seoTitle ?? d.title,
  description: summary,
  datePublished: d.publishedAt.toISOString(),
  dateModified: d.updatedAt.toISOString(),
  author: personRef(site.person.name),
  image,
  url: canonical,
  inLanguage: bcp47(lang),
};

The other pages’ structured data, plus the shared personRef() and jsonLd() helpers, live in src/lib/seo.ts:

  • Home page: one Person and one WebSite. The Person has a fixed @id (https://kelvinwang.info/#person), and every post’s author points at that same @id and the about page URL, so Google can tie all the posts to one person.
  • About page: ProfilePage.
  • Post lists: CollectionPage containing an ItemList of posts.
  • Every public page type: BreadcrumbList, generated by the same component that draws the visible breadcrumbs.
  • Translations: the original points at its translation with workTranslation, and the translation points back with translationOfWork.

One safety detail: JSON-LD sits inside <script type="application/ld+json">, and a title containing a string like </script> would close the tag early. So jsonLd() escapes < as \u003c before printing.

Check the result with Google’s Rich Results Test 16.

Step 6: hreflang for a multilingual site

hreflang tells Google that a page has versions in other languages, so readers in Taiwan get the Chinese version and readers in the US get the English one. Google’s rules 13:

  • Use one of three methods; they work the same: <link rel="alternate" hreflang> in the <head>, an HTTP header, or xhtml:link in the sitemap.
  • Every version lists itself and all the others, and the links must go both ways. If A points to B and B doesn’t point back, the annotations are ignored.
  • Use fully qualified URLs with https://.
  • x-default is the fallback for readers whose language isn’t listed, and suits language pickers or home pages that redirect.

The <head> of a Chinese post on my site:

<link rel="alternate" hreflang="zh-Hant" href="https://kelvinwang.info/zh-hant/blog/xxx/" />
<link rel="alternate" hreflang="zh-TW" href="https://kelvinwang.info/zh-hant/blog/xxx/" />
<link rel="alternate" hreflang="zh-HK" href="https://kelvinwang.info/zh-hant/blog/xxx/" />
<link rel="alternate" hreflang="en" href="https://kelvinwang.info/en/blog/xxx/" />
<link rel="alternate" hreflang="x-default" href="https://kelvinwang.info/en/blog/xxx/" />

The English version carries the identical set. The same annotations go into both the <head> and the sitemap, both produced by hreflangLinks(), so the two can’t drift apart.

URLs use a language prefix, /zh-hant/… and /en/…, so every language version has its own URL, which is what Google recommends 17.

One place where my site goes against Google’s advice: my home page / redirects to /zh-hant/ or /en/ based on a cookie and the browser language. Google advises against automatically redirecting between language versions, because Googlebot mostly crawls from the US without an Accept-Language header and may miss the other versions 17. My setup relies on / not being a language version itself: neither language URL is ever redirected, every page has visible language links, and both versions are listed in the sitemap and hreflang.

Readers publishing in Chinese have extra work: Traditional and Simplified need separate URLs (zh-Hant and zh-Hans in hreflang), and Chinese web fonts run to megabytes.

Step 7: AI search and llms.txt

Google AI Overviews and AI Mode

Google’s position is plain 18:

  • A page that’s indexed and eligible to show with a snippet in Search is eligible for AI Overviews and AI Mode.
  • There are no additional technical requirements: no new machine-readable files, AI text files or markup, and no special schema.org structured data.
  • What it does recommend is ordinary SEO: allow crawling, link internally, keep important content in text, make structured data match what’s visible on the page, and verify the site in Search Console.

So if you’ve done steps 1 to 6, you’ve already prepared for AI search.

From OpenAI’s documentation 19:

  • OAI-SearchBot decides whether your site can appear in ChatGPT search results. To appear, allow it in robots.txt.
  • GPTBot collects training data and can be blocked on its own without affecting search.
  • ChatGPT-User only visits when someone asks ChatGPT to read a URL; it isn’t an automatic crawler.
  • Changes to robots.txt take about 24 hours to apply.

To appear in search without contributing to training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

My site doesn’t separate them; everything is allowed.

llms.txt

llms.txt is a proposal Jeremy Howard published in September 2024: a Markdown file at your site root that lets AI tools quickly see what the site contains. Only an H1 with the site’s name is required; a blockquote summary and H2 sections of links are optional 20. The proposal says its authors expected it to be used mainly at inference time, when an AI is answering a question, rather than for training 20.

So it’s optional; Google has already said AI search needs no AI text files 18. It’s also cheap to add. My site has one, generated at build time by src/pages/llms.txt.ts:

const lines = [
  `# ${en.person.name}`, '',
  `> ${en.person.description}`, '',
  en.person.about, '',
  ...sections,   // one section per language: home, post list, about, RSS, lab pages, then every post's title, URL and summary
];

The post list comes straight from the post data, so new posts never require editing the file.

What to watch each week after launch

  • The Search Console Pages report: any new “not indexed” reasons.
  • The Performance report: which queries are starting to get impressions, and which pages have a low click-through rate (their title or description may need work).
  • When you change titles, change one post at a time and note what changed and when. Compare a few weeks later so you know which change worked.
Go deeper What Breaks After You Launch an AI-Built Site: Failed Deploys, Leaked Keys, Database Permissions The other things worth checking regularly once you're live.

Sources

  1. 1. Get your website on Google Search · Google Search Central · checked 2026-11-16
  2. 2. Page indexing report · Search Console Help · checked 2026-11-16
  3. 3. Crawling and indexing FAQ · Google Search Central · checked 2026-11-16
  4. 4. Add a website property to Search Console · Search Console Help
  5. 5. Manage your sitemaps using the Sitemaps report · Search Console Help · checked 2026-11-16
  6. 6. URL Inspection tool · Search Console Help · checked 2026-11-16
  7. 7. How HTTP status codes, and network and DNS errors affect Google Search · Google Search Central · checked 2026-11-16
  8. 8. Understand the JavaScript SEO basics · Google Search Central · checked 2026-11-16
  9. 9. How to specify a canonical URL with rel="canonical" and other methods · Google Search Central · checked 2026-11-16
  10. 10. SEO Starter Guide · Google Search Central · checked 2026-11-16
  11. 11. Verifying Googlebot and other Google crawlers · Google Search Central · checked 2026-11-16
  12. 12. Build and submit a sitemap · Google Search Central
  13. 13. Tell Google about localized versions of your page · Google Search Central
  14. 14. Influencing your title links · Google Search Central
  15. 15. Control your snippets in search results · Google Search Central
  16. 16. Article structured data · Google Search Central
  17. 17. Managing multi-regional and multilingual sites · Google Search Central
  18. 18. AI features and your website · Google Search Central · checked 2026-11-07
  19. 19. Overview of OpenAI Crawlers · OpenAI · checked 2026-11-07
  20. 20. The /llms.txt file · llmstxt.org · checked 2026-11-07
  21. 21. Ask Google to recrawl your URLs · Google Search Central · checked 2026-11-16

About the author

Kelvin Wang

Electrical engineer building tools with AI agents, sharing hands-on notes on engineering and investing.

More about me