Markdown mirrors
Every page on this site has a .md twin at the same path, so an AI agent can read the content instead of parsing the layout.
- Year
- 2026
- Role
- Everything
- Stack
- Node.js
- Status
- Active
Append .md to any URL on this site and you get the same page as clean Markdown. The footer link on each page points at it.
The problem it solves#
An increasing share of the traffic that matters is not a browser. It is a model fetching the page to answer a question. That client pays the full cost of my layout — nav, footer, breadcrumbs, structured data — to extract the one thing it wants, which is the text.
The mirror is the same content with the chrome removed: front matter with the canonical URL and dates, then the body. It is generated from the same source in the same build, so it cannot drift out of sync with the HTML.
The one decision worth explaining#
The obvious way to keep mirrors out of search results is Disallow: /*.md$ in robots.txt. That is wrong, and I wrote it that way first before catching it.
Disallow blocks fetching. It would block the exact agents the files exist for, while doing nothing that a noindex does not do better. Search engines would still not index them, and the AI clients would get a 403-shaped answer from a file built specifically for them.
The correct tool is an X-Robots-Tag: noindex, nofollow header on /*.md, which is what the Cloudflare Pages _headers file does. The files stay fetchable and stay out of the index. Two separate questions — can you read this and should this appear in results — and one directive for each.
Also there#
/llms.txt at the root: a single plain-text index of every page with its description, so an agent can see the whole site without crawling it.