Dentolize · Clinic-Web Search Deindexing Walkthrough
On this pageBusiness viewTechnical view

Robots Meta Tag & robots.txt

Business view

Search engines like Google crawl the public web and index what they find so it can show up in search results. clinic-web is not public content — it's a login-gated dashboard where clinic staff manage patients, appointments, invoices, and insurance claims. There's no scenario where a clinic-web page belongs in someone's Google search results, and having it indexed anyway is at best noise (irrelevant results, wasted crawl budget) and at worst a minor information-exposure concern — a search engine that indexes page titles or cached fragments of a login-gated app can leak more than intended, even though the actual data behind the login stays protected.

This PR fixes that by telling every well-behaved crawler, in the standard way crawlers expect: "you may look, but don't list this, don't follow links out of it, and don't keep a cached copy." That instruction lives directly in the page itself, so it applies uniformly no matter which route someone lands on.

Nothing changes for staff using the app. Login, navigation, and every feature behave identically — this is purely metadata for automated crawlers.

Technical view

The two levers, and why both are needed

There are two independent ways to tell a crawler "don't index this":

  1. A robots meta tag in the page's <head>, e.g.

<meta name="robots" content="noindex, nofollow, noarchive">. This is read after a crawler successfully fetches the page.

  1. A robots.txt Disallow rule, which tells a crawler not to fetch

a URL at all.

These interact: if robots.txt disallows a URL, a compliant crawler never fetches it — which means it never sees the noindex meta tag either. For years clinic-web's robots.txt had no Disallow, so crawlers were free to fetch and index everything (packages/clinic-web/public/robots.txt prior to this PR had no Allow/Disallow lines at all under User-agent: *, which crawlers treat as unrestricted).

This PR adds the meta tag as the actual indexing instruction, and adds an explicit Allow: / to robots.txt so that instruction can be seen and honored — blocking the whole app via robots.txt instead would have hidden the noindex tag from crawlers and, per Google's documented behavior, could still let the bare URL appear in search results with no description.

index.html — the meta tag

packages/clinic-web/public/index.html:6-7:

<!-- Every route of this app is served from this one file. Nothing here is meant for search results, so none of it is indexed. -->
<meta content="noindex, nofollow, noarchive" name="robots" />

clinic-web is a client-rendered single-page app — every route (/, /auth/login, a specific patient's chart, etc.) is served the same static index.html by the web server, then React Router takes over client-side. Because there is exactly one HTML document backing every route, this one tag applies to the entire app; there is no per-route way to opt back in even if a future page wanted to be indexable — that would require serving a different index.html for that route.

The tag sits right after the existing <link href="https://my.dentolize.com" rel="canonical" /> at packages/clinic-web/public/index.html:8, which predates this PR. That canonical tag and the new noindex tag are not in conflict — canonical tags tell a crawler which URL is the authoritative version of a page it has decided to index; noindex tells it not to index the page at all. A crawler that respects noindex won't reach the point of caring about canonicalization for this app.

Directive breakdown:

DirectiveEffect
noindexDon't include this page in search results.
nofollowDon't crawl links found on this page to discover more URLs.
noarchiveDon't keep/show a cached copy of this page.

robots.txt — the crawl policy

packages/clinic-web/public/robots.txt (full file, post-PR):

# https://www.robotstxt.org/robotstxt.html
User-agent: *

# The app itself stays fetchable on purpose: the noindex in index.html only works if a crawler is allowed to read the
# page. Blocking it here instead would hide that tag and let the bare URL be listed anyway.
Allow: /

# Documents carry no markup, so a noindex cannot travel with them and refusing the fetch is the only lever left. None
# of these are meant for search results either.
Disallow: /*.pdf$
Disallow: /*.xlsx$
Disallow: /*.jpg$
  • Allow: / (line 6) — applies to all crawlers (User-agent: *, line 2) and

explicitly permits fetching every path, which is what lets the noindex meta tag above actually get read and honored.

  • Disallow: /*.pdf$, /*.xlsx$, /*.jpg$ (lines 10-12) — block crawling of

three file extensions directly, using the wildcard/end-anchor syntax (* = any characters, $ = end of URL) supported by Google's and most major crawlers' robots.txt parser. These exist because generated/uploaded documents (exported invoices, spreadsheets, patient-related photos) are served as raw files with no HTML <head> — there is no document to put a noindex tag in, so refusing the fetch via robots.txt is the only available lever for that content type.

Both files are served as static assets from packages/clinic-web/public/ (Create React App-style public directory — note %PUBLIC_URL% templating elsewhere in index.html), so no server-side or build-time logic is involved; what's in the file is exactly what's served in production.

Verified behavior in the sandbox

Fetching /robots.txt and evaluating document.querySelector('meta[name="robots"]') on both the (unauthenticated) login route and the (authenticated) dashboard route in the PR's sandbox deploy confirmed:

  • robots.txt serves the new Allow/Disallow rules shown above.
  • The noindex meta tag is present identically on both routes, confirming

the single-index.html behavior described above. See Walkthrough for the screenshots.

What this PR does not change

  • No change to packages/server, GraphQL resolvers, or any API behavior.
  • No change to authentication, permissions, or any data access.
  • No change to packages/clinic-mobile. Mobile apps aren't crawled by web

search engines and don't serve robots.txt/HTML meta tags in the same way.

  • This repository has no patient-web or standalone clinic-web-canary app

(the clinic-web-canary package here is a component library consumed by clinic-web, not a separately served app), so there was nothing else in scope for this PR to touch.