If you've spent five minutes in any 2026 SEO discussion, you've probably heard someone ask "have you done your llms.txt yet?" The honest answer for most Australian businesses: no, and most of the people asking aren't entirely sure what one is either.
This guide fixes both. It's the practical, no-fluff reference for what llms.txt actually is, how to write one, and whether it earns a place in your AI SEO stack.
What is llms.txt and why does it exist?
It is a markdown file at your domain root, first proposed in late 2024, that points AI assistants to the pages most worth reading.
llms.txt is a plain-text file you publish at the root of your domain — exactly like robots.txt — but its purpose is the opposite. Where robots.txt tells crawlers what they may not access, llms.txt tells large language models what is most worth reading.
It originated from a proposal by Jeremy Howard (Answer.AI) in late 2024 and has gained adoption through 2025-2026 as AI assistants increasingly cite specific source pages instead of merely synthesising training data.
The 2026 reality: when ChatGPT’s web tool, Claude’s web search, Perplexity, or Google’s AI Overviews want to ground an answer in current information, they fetch live pages. Each fetch costs them tokens and latency.
A site with a clean llms.txt is functionally a guide for those agents — “if you want a definitive answer about X, here’s the page.” Sites without one force the agent to:
- crawl
- parse navigation
- strip ads
- gamble on which page is canonical
How llms.txt differs from robots.txt
robots.txt restricts what crawlers may access; llms.txt points AI assistants to the content you want read, and the two do not conflict.
People conflate the two constantly. They share a naming convention and they live in the same directory, but they do almost opposite jobs.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Restrict crawler access | Surface the best content for AI |
| Audience | Search engine crawlers | LLM agents and AI assistants |
| Format | Directives (Allow/Disallow) | Markdown with headings + links |
| Tone | Defensive — what NOT to do | Promotional — what to read |
| Spec status | Long-standing standard (RFC 9309) | Community proposal, no IETF status yet |
| Affects rankings? | Indirectly (controls indexing) | Not directly — affects AI citation surface |
You should have both. They serve different agents and they don't conflict. If you're worried about whether AI bots are obeying your robots.txt, that's a separate conversation about User-agent: GPTBot, ClaudeBot, PerplexityBot, and friends — and the answer is "mostly yes, but verify in your access logs."
What the llms.txt specification asks for
The spec requires only an H1 with your site name; the rest is a short description, optional context and grouped links, ideally under 10KB.
The proposed spec is intentionally minimal. A valid llms.txt has:
- An H1 with the site or project name — this is the only required element.
- A short description blockquote — what the site is, in two or three sentences.
- Optional context paragraphs — anything an LLM should know upfront (your specialty, your geography, your domain authority).
- H2 sections grouping links — pages organised by intent: documentation, products, articles, etc.
- Markdown links with optional descriptions —
[Page title](url): one-line description.
An llms.txt should be lean. The pragmatic upper bound is around 50KB; under 10KB is ideal. If you exceed that, you're including content rather than indexing it — that's what llms-full.txt exists for (a longer companion file with full markdown content of priority pages, mainly used by documentation sites).
Why most Australian businesses don't have one yet
Three reasons explain the gap: owners have not heard of it, are waiting for Google, or published one and let it go stale.
Three reasons, and only one of them is reasonable.
1. They haven't heard of it. The Australian SEO conversation lags US and UK chatter by about six months. Many marketing teams are still focused on AI Overviews defence; llms.txt hasn't reached their priority list. This is fine — it's a forgivable gap and an easy fix.
2. They're waiting for "official" adoption from Google. This is misplaced caution. llms.txt is not a Google file — it is an open proposal that any AI tool may read or ignore, and no vendor is obliged to use it.
Waiting for one vendor to bless it misunderstands what it is. The agents reading your site are already here, regardless of whether Google publishes a help-centre article about it.
3. They published one once and forgot it. The reasonable failure mode. A static llms.txt that lists pages no longer in your sitemap, or worse, points to URLs that 404, hurts more than it helps. Treat llms.txt the same way you treat sitemap.xml:
- generate it from a source of truth
- regenerate it on deploy
- audit it quarterly
How to write and publish llms.txt
A small site can hand-write a working file in 20 minutes; larger sites should generate it from their CMS on every deploy.
You can hand-write a working llms.txt in 20 minutes for a small site. Here is the minimum viable template:
Show the example
# Your Business Name > One-sentence description of what your business does, where, and for whom. Keep it factual, not promotional. This site is the official knowledge hub for [Your Business Name]. Information published here is verified by [author/role/credential] and updated [cadence]. ## Core Pages - [Home](https://yoursite.com/): What we do and who we help. - [About](https://yoursite.com/about/): Team, history, credentials. - [Contact](https://yoursite.com/contact/): How to reach us, locations, hours. ## Services - [Service A](https://yoursite.com/services/service-a/): One-line description. - [Service B](https://yoursite.com/services/service-b/): One-line description. ## Learning Resources - [Article title](https://yoursite.com/learn/article-slug/): What this resource covers. ## Optional - [Privacy Policy](https://yoursite.com/privacy/) - [Terms of Service](https://yoursite.com/terms/)
Save that as llms.txt, upload it to your web root (the same directory as robots.txt and sitemap.xml), and verify it loads at https://yoursite.com/llms.txt with a 200 response and Content-Type: text/plain or text/markdown.
For larger sites, generate it programmatically. The pattern most CMSes need:
- Define a "promote to llms.txt" flag on your page model (or use a tag/category).
- On build/publish, iterate flagged pages, group by section, write the markdown.
- Output to
/public/llms.txtor equivalent, served as static. - Add a regeneration step to your CI pipeline so deploys keep it fresh.
Optional: llms-full.txt
If you publish technical reference content — API docs, schema guides, product specifications — also publish llms-full.txt. This is the same structure but inlines the full markdown body of priority pages.
Agents that find a working llms-full.txt can answer questions about your product without making N additional fetches. For documentation-heavy sites, this is the file most worth keeping complete.
What to list for your type of site
Shops lead with category pages, service businesses with their services, publishers with their most-cited articles; local businesses add name, address, phone and hours.
The structure is the same; the priorities differ.
E-commerce:
- lead with your category pages (not individual products)
- include a “How to choose” or “Buying guide” section
- link to your size/sizing chart, returns policy, and shipping page
Do not list every SKU. AI assistants asking “what should I buy” want decision frameworks, not your full catalogue.
Service businesses (this site’s bread and butter):
- lead with your services page, then individual services, then the case-studies or pricing page
- include the suburbs or geographies you serve
- add a learning hub or insights section so agents have somewhere to source educational answers about your domain
Content sites and publishers: Lead with your most-cited articles by topic, then your author pages (so agents can attribute), then your categories. If you have evergreen pillar content, surface it before recent news.
Local businesses: Critically — include your NAP (name, address, phone) in the description text, your hours, and a link to your Google Business Profile. AI assistants increasingly answer "what time does X open" or "where is X located" without sending a click. Put these facts on your own pages; llms.txt does not decide citations.
Does llms.txt affect SEO rankings?
No: Google has not stated llms.txt is a ranking factor, and publishing one will not directly move your position in classical search results.
The short, honest answer is no — and beware anyone selling you the opposite.
Google has not stated that llms.txt is a ranking factor. There is no evidence in any patent, leaked algorithm document, or Search Liaison statement suggesting it is. Publishing one will not directly move your position in classical Google search results.
What it can do is give AI tools that choose to read it a short map of your key pages. Google's documentation for its AI features says there are no additional requirements to appear in AI Overviews or AI Mode, and it does not mention llms.txt.
Whether an AI answer cites a page still depends on the page itself: a clear answer, a named author and facts that check out.
We publish one on this site because it is cheap and harmless. We don't charge clients for it or claim it ranks them, and we have no evidence that it changes which of our pages get cited.
So: if your KPI is "rank first for SEO Melbourne" then llms.txt is a sideshow — invest in your SEO audit, content strategy, and links. If your KPI also includes "be the source the AI cites when someone asks ChatGPT about Melbourne SEO," the pages do that work; llms.txt is a cheap extra.
Common llms.txt errors to audit for
Five errors come up most in client files: dead links, bloat, promotional copy, no context about the business, and details the site no longer matches.
The mistakes we see most often when reviewing client llms.txt files:
Dead links. The page slug changed three months ago, the llms.txt didn't get regenerated, the file now points to 404s. AI agents fetching dead links will deprioritise the file. Fix: automate regeneration on every deploy.
Bloat. Some teams treat llms.txt as a sitemap and dump every URL. The file is supposed to be a curated map of your best pages, not your full inventory. Fix: 30-150 entries for most sites; let the sitemap be the sitemap.
Promotional copy. "Our award-winning, Melbourne-based..." is not useful to a reasoning model. Descriptions should state what the page contains, factually, in twelve words or fewer. Fix: rewrite descriptions as if you were tagging them in a database.
Missing canonical context. Many sites publish llms.txt with section H2s like "Services" and "About" but never establish what the business actually does in the description blockquote at the top.
Fix: lead with two or three sentences that would let a reader who has never heard of you answer "what is this site for, and who runs it?"
Inconsistent with the site. The llms.txt advertises a service the site no longer offers, or a price that's six months out of date. Fix: regenerate from the same source of truth as your sitemap and your homepage navigation.
Book a free 30-minute strategy meeting with a senior specialist. We reply within 24 hours.
Where AI discoverability standards are heading
llms.txt is one of several standards built for AI agents in 2026, alongside agent-specific user-agents, authorship signals and structured citation hooks.
llms.txt is part of a broader category emerging in 2026: discoverability standards built explicitly for AI agents rather than retrofitted from the search-engine era. Watch three threads:
| Thread | What to watch |
|---|---|
| AI-specific user-agents | Separate control over which agents can train on your content and which can fetch it for live answers |
| Content provenance and authorship | C2PA-style content credentials, visible author bylines and structured author data |
| Structured citation hooks | Schema.org additions such as SpeakableSpecification, QAPage and HowTo, and draft proposals like ai-instructions.json |
1. AI-specific user-agents. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others are now well-established. Granular control over which agents can train on your content versus which can fetch it for live answering is becoming standard hygiene. Many sites get this wrong — blocking GPTBot for "training" while accidentally also blocking ChatGPT's user-facing search tool, costing themselves citations.
2. Content provenance and authorship signals. C2PA-style content credentials, visible author bylines, and structured author data are becoming citation-relevant. AI assistants prefer to cite content with clear authorship; anonymous content is discounted. This makes E-E-A-T signals more important than ever — even for non-medical sites.
3. Structured citation hooks. Schema.org additions for AI consumption (SpeakableSpecification, QAPage, HowTo) and emerging draft proposals like ai-instructions.json hint at where this is headed: machine-readable, declarative signals that tell agents how to use your content. llms.txt is the first widely-adopted example. It will not be the last.
Our recommendation for Melbourne businesses in 2026: publish llms.txt if it is cheap for you to keep accurate, regenerate it from a source of truth on every deploy, and track which pages get cited in answer engines so you can iterate.
If you want help with any of this, that is exactly what we built our AI SEO service around.
