llms.txt for your website.
Your website already holds the context an AI system needs. The work is choosing which public pages belong in a small Markdown file, explaining why each one matters, and publishing that file where agents can request it.
Discover the real site
A website can have more than one sitemap. Nested indexes and common sitemap paths help reveal the public surface.
Add useful AI context
Titles and URLs are not enough. A short description helps a reader decide which page to open next.
Keep the result current
Websites change. A scheduled update can detect additions and removals so the public file does not go stale.
What belongs in the file
Turn a large website into a useful starting point.
An llms.txt file should not repeat every paragraph from every page. It should give enough business context to orient the reader, then link to the pages that contain the details.
Business overview
What you do, who you serve, and what makes the offering distinct.
Products and services
The pages that explain your main offers and the problems they solve.
Docs and support
Getting started guides, technical references, FAQs, and help content.
Current information
Pricing, policies, announcements, and other pages that need regular review.
Include
- The homepage or product overview
- Pricing, plans, or service menus
- Getting-started docs and API reference
- Support, shipping, returns, or policy pages people already quote
- A small set of cornerstone articles, not the full blog archive
Exclude
- Account, checkout, and other authenticated URLs
- Faceted search, sort orders, and paginated lists
- Tag archives and thin marketing variants
- Legal pages that are only needed after a sale, unless agents already ask for them
- Any URL you would not want an assistant to treat as canonical
Website as the source
Discovery should follow how the site is actually published.
Start from robots.txt and sitemap.xml, including nested sitemap indexes. Follow public links only when the sitemap is incomplete. Summaries should come from the page itself, not from a generic blurb about the industry.
01
Discover
Sitemaps, nested indexes, and public page paths
02
Understand
Titles, descriptions, content, and business context
03
Organize
Useful headings, links, and page summaries
04
Publish
A consistent public /llms.txt URL
Site size
Small sites, large sites, and documentation subsites.
Small marketing site
Write the file by hand. Ten honest links beat a generated dump of thirty near-duplicates.
Large product site
Generate from the sitemap, then cut. Keep root llms.txt short. Do not publish a preview that only covers the first few URLs.
Docs at /docs
Publish /docs/llms.txt for the documentation tree. The official format says a file covers URLs under its path, and the most specific file wins.
A sensible workflow
Use your website as the source of truth.
The generator can handle finding pages and writing consistent summaries. You still control which website you add, when you unlock the full result, and where the public file is used.
See the step-by-step processEnter a website in the format you already use.
Let sitemap discovery find public pages, including nested sitemaps.
Review a free preview before unlocking the complete site.
Publish only the complete file at /llms.txt, then re-check it.
Ready when your website is
Give every important page a better introduction.
Start with one public website. If you manage several sites, keep each one as its own project in the dashboard.
Common questions
Questions about llms.txt for websites.
Does every website need an llms.txt file?
Need is the wrong word. If agents already ask about your product, docs, or policies, a curated file saves them from scraping noisy HTML. A five-page brochure site can skip it. A docs-heavy product usually should not.
Will llms.txt make ChatGPT or Google cite my website?
No reliable citation or ranking promise comes with the file. It is on-demand orientation for agents. Search still uses your pages, sitemap, and robots.txt.
Should a large website put everything in one file?
No. Keep the root file small. Documentation can live at /docs/llms.txt, which covers URLs under /docs/. Agents should pick the most specific file that applies.
What website pages should I exclude?
Login walls, checkout, faceted search, tag archives, print views, and thin duplicates. Also skip any URL you would not want quoted in an answer.
Can I generate llms.txt from an existing website?
Yes. Discovery should start from sitemaps and public HTML, then you review which URLs belong. A generator preview is a sample, not a complete publishable file.
Keep exploring
More useful llms.txt resources.
Move from the format to implementation, examples, and a live check without losing the thread.