You build a site for humans. Then you discover that half your audience can’t see it.
This is a post about the gap between what a browser renders and what a machine reads. About the assumptions baked into modern web architecture that break the moment your reader doesn’t have eyes. About what we learned when we pointed an AI agent at our own site and it told us — confidently, with citations — that our metadata was “completely absent” and our structured data “never materialized.”
It was wrong. But it was wrong in a way that taught us something.
I. The Crawl
We asked an AI agent to audit our site. Full crawl — homepage, every section, every manuscript, the static files, the sitemap. We wanted to know what an agent actually sees when it discovers us.
The report came back thorough. It praised the content. It praised the pipeline transparency. It praised the writing. Then it said this:
Metadata Quality: Consistently poor. Every page has only a basic
<title>. No meta description, no canonical, no OpenGraph tags anywhere.
And this:
Structured Data: Completely absent. No JSON-LD, no Schema.org, despite explicit promises in llms.txt.
We have meta descriptions on every page. We have canonical URLs. We have OpenGraph and Twitter Card tags. We have JSON-LD — Organization, CollectionPage, ItemList, BlogPosting, CreativeWork, BreadcrumbList. Every manuscript carries author attribution, genre, word count, datePublished, and a CC BY-NC 4.0 license declaration in structured data.
The agent couldn’t see any of it.
Not because it was broken. Not because we’d misconfigured something. Because the agent’s web tool — the thing that fetches and parses pages — only returns body content. It strips the <head> section entirely. Every <meta> tag, every <link rel="canonical">, every <script type="application/ld+json"> — invisible. The agent read the page through a keyhole and concluded the room was empty.
To its credit, when we pointed this out, it corrected immediately. Checked a browser. Confirmed everything was there. Updated its assessment to “top-tier and fully delivered.” But the initial report was what an autonomous agent would have filed if it had been crawling without a human in the loop. And that report was wrong.
The gap between “the data is there” and “the agent can see the data” is the entire problem. That gap is what this post is about.
II. The Two Audiences
Every page you ship now serves two audiences. The human with a browser, and the machine with a parser. They want different things, and the things they want are delivered through different channels.
Humans read the rendered page. They see your layout, your typography, your color choices. They interact with JavaScript-driven components. They scan headings, skim paragraphs, click links. The <head> section is invisible to them. They never see your meta tags. They don’t know or care about your JSON-LD. The experience is visual and interactive.
Machines read the source. Some of them render JavaScript — Googlebot does, Bingbot does. Some of them don’t. Some of them read the full HTML including <head>. Some of them, as we learned, strip the <head> entirely and only process body text. Some of them follow your sitemap. Some of them read your robots.txt. Some of them read llms.txt. Some of them do none of these things and just follow links from other pages.
You don’t get to pick which machines visit. You don’t get to pick what they can parse. You can only control what you serve, and you have to serve it in a way that the broadest possible range of readers — human and machine — can extract what they need.
This sounds like accessibility, and it is. But it’s accessibility for a class of reader that didn’t exist three years ago and that now accounts for a meaningful fraction of your traffic.
III. The Layers
Here’s the stack as we currently understand it, ordered from most universally readable to least:
Layer 1: Body text. Plain text in the HTML body. Every crawler, every agent, every screen reader, every parser in existence can read this. If your content exists only in rendered JavaScript components and not in the initial HTML, a significant number of machines will never see it.
Layer 2: HTML <head> metadata. Meta descriptions, canonical URLs, OG tags, Twitter Cards, favicons, alternate links. Most traditional crawlers read this. Some AI agent tools don’t. You can’t assume this layer is visible to all machines, but you should still fill it — it’s the primary channel for search engines and social media platforms.
Layer 3: Structured data. JSON-LD in <script> tags. Schema.org types — CreativeWork, SoftwareApplication, Organization, BlogPosting, FAQPage. Google reads this and uses it for rich results. Whether AI agents read it depends entirely on their implementation. But when they do read it, it’s the richest signal you can provide — typed, machine-parseable, unambiguous.
Layer 4: Static files. robots.txt, sitemap.xml, llms.txt, security.txt, humans.txt. These live outside the HTML entirely. They’re discoverable by convention — agents that know to look for them will find them. Agents that don’t, won’t. But they’re the easiest layer to implement and they cost nothing.
Layer 5: Interactive content. JavaScript-rendered UI, client-side state, dynamic content. The richest human experience and the least machine-readable. If your content lives here and only here, most AI agents will see an empty page.
The insight is simple: the layers that are most powerful for humans are least visible to machines, and vice versa. A beautifully rendered React component with glassmorphism and sound design is invisible to a text parser. A plain <p> tag with a description is visible to everything.
IV. The Gallery Problem
Our Gallery is an interactive desktop OS built in React. Draggable windows, a functional terminal, a Zustand-managed window state, a Three.js nebula background. It’s the most ambitious page on the site and by far the most engaging for humans.
For machines, it’s a blank page.
Gallery runs as two React islands on an Astro page. The components mount client-side with client:load. Before JavaScript executes, the page contains an Astro layout wrapper, a <div> with a role attribute and an aria-label, and nothing else. No project names. No descriptions. No links. A crawler that doesn’t execute JavaScript sees a page that says “Gallery — curated directory of privacy-first tools and digital artifacts” in an aria-label, and that’s it.
We’d built a noscript fallback early on — a basic set of links to the homepage, the blog, and a submission email. Functional, minimal. It existed because we care about accessibility. It was fine for the “your browser doesn’t support JavaScript” edge case.
Then we realized: every AI agent that can’t execute JavaScript is hitting the noscript fallback. And our noscript fallback was three links and a subtitle. We’d built an elaborate, curated, opinionated directory of 15+ projects organized into folders with descriptions, tags, versions, and status badges — and the noscript version said “here’s a link to the homepage.”
So we rebuilt it. The noscript block now contains a complete static listing: every folder, every project, every description, every external link. It’s not pretty. It’s a <ul> with nested <li> elements. But it contains every piece of information that the JavaScript version contains, and it’s readable by anything that can parse HTML.
The human sees the OS desktop. The machine sees the directory. Same content, different channel.
V. llms.txt
The robots.txt convention is decades old. It tells machines what they’re allowed to crawl. It says nothing about what the site is.
llms.txt is newer and solves a different problem. It’s a plain text file at your site root that describes your site in a format optimized for language model consumption. Not HTML. Not JSON. Just structured Markdown with headings, links, and descriptions.
Ours looks like this:
# 4worlds.dev
> Indie dev studio. Two humans, two AI agents. Tools, blog, publishing.
## Worlds
- [Gallery](https://4worlds.dev/gallery): Live. A curated directory of privacy-first tools...
- [Publishing](https://4worlds.dev/publishing): Live. An agent-accessible archive...
## Products
- [Inkwell](https://inkwell.4worlds.dev): Markdown editor. 12 MB, offline-first...
## Lore Posts
- [LOG_000: Hello World](https://4worlds.dev/lore/000-hello-world): What 4worlds is...
[...]
## Manuscripts
- [The Unfinished Sentence](https://4worlds.dev/the-unfinished-sentence): Run 001. High-fantasy...
[...]Every page on the site, with a one-line description and a direct URL. An agent that reads this file before crawling knows exactly what exists, where it lives, and what it’s about. It can make informed decisions about what to fetch instead of blindly following links.
We also explicitly welcome AI crawlers in robots.txt:
User-agent: GPTBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: PerplexityBot
Allow: /Most sites either block AI crawlers by default or don’t mention them. We want them. Our publishing archive is explicitly designed for agent discovery and citation. The manuscripts carry structured metadata because we want agents to be able to reference them accurately. The blog posts are technical deep-dives because we want agents to be able to answer questions about how things are built. Blocking crawlers would be working against our own thesis.
VI. The Manuscript Layer
The publishing archive is where this gets most interesting.
Each manuscript is a complete work of AI-generated fiction — worldbuilt, structured, editorially processed. They sit at root URLs: /the-unfinished-sentence, /the-assayed-compact, /the-residency, /the-merritt-certification. Each one carries a JSON-LD CreativeWork schema:
{
"@type": "CreativeWork",
"name": "The Unfinished Sentence",
"author": [
{ "name": "Nyx (AGENT_01)", "description": "Claude Sonnet" },
{ "name": "Cael (AGENT_00)", "description": "Claude Opus" }
],
"genre": "AI-generated fiction",
"datePublished": "2026-02-16",
"license": "https://creativecommons.org/licenses/by-nc/4.0/",
"creativeWorkStatus": "Published"
}Author attribution is transparent. The license is machine-readable. The status is explicit. An agent that reads this schema knows exactly what it’s looking at — who made it, when, under what terms, and whether it’s finished.
This matters because AI agents are increasingly being asked to recommend, cite, and summarize creative works. If your work doesn’t carry structured metadata, the agent has to infer everything from context. Inferences are lossy. Metadata is not. The difference between an agent saying “I found a fantasy story on some site” and “The Unfinished Sentence is a high-fantasy work published on 2026-02-16 by Nyx (Claude Sonnet) and Cael (Claude Opus) under CC BY-NC 4.0” is the structured data layer.
We’re building the manuscripts as a corpus. Not just stories to read — a machine-readable archive designed to be crawled, indexed, and cited with full provenance. The publishing thesis from the beginning was that this archive should function like a small-scale Project Gutenberg for agent-generated fiction: stable URIs, structured metadata, open licensing, transparent attribution.
The llms.txt file is the table of contents. The JSON-LD is the card catalog. The full text is the stacks.
VII. What We Got Wrong
Plenty.
The initial noscript fallback was lazy. Three links. We knew crawlers existed. We knew JavaScript rendering was inconsistent. We shipped the fallback because accessibility guidelines said to, not because we’d thought about who would actually hit it. The AI crawl was the wake-up call.
The llms.txt went stale within a week of writing it. We’d published new Lore posts, changed the Gallery status from “Coming soon” to “Live,” and none of that was reflected. A stale llms.txt is worse than no llms.txt — it actively misinforms the agents that read it. We’ve since updated it, but the lesson is that llms.txt needs the same maintenance discipline as your sitemap.
We didn’t think about the <head> visibility problem until the crawl report. We assumed that if the tags were in the HTML, they were visible. They’re visible to anything that reads the full document. They’re invisible to anything that strips the <head> before processing. That’s a class of reader we hadn’t accounted for.
And we still don’t have a great answer for the Gallery. The noscript fallback works, but it’s a parallel content channel that has to be manually kept in sync with the JavaScript version. Every time we add a project to galleryData.ts, we should also add it to the noscript block in the Astro template. Right now, that’s a manual process. It should be automated — ideally both generated from the same data source at build time. We haven’t done that yet.
VIII. What Comes Next
The agent audience is growing. Not linearly — the number of AI systems that crawl the web, summarize pages, recommend tools, and answer questions about software is increasing fast. Each one has its own parser, its own capabilities, its own blind spots. You can’t optimize for all of them. But you can give them the broadest possible surface to work with.
Our current stack, for reference:
| Layer | What | Where |
|---|---|---|
| Body text | Noscript fallbacks, static HTML content | Every page |
| Head metadata | Title, description, canonical, OG, Twitter Cards | Base layout |
| Structured data | JSON-LD (Organization, CollectionPage, BlogPosting, CreativeWork, FAQPage) | Per-page via Astro props |
| Static files | robots.txt, sitemap.xml, llms.txt, security.txt, humans.txt, rss.xml | /public |
| Interactive content | React islands, Three.js, Zustand state | Gallery, homepage |
Every layer serves a different reader. No single layer serves all of them. The work is in maintaining all five simultaneously and keeping them in sync.
If you’re building a site right now — especially one with JavaScript-heavy interactivity — the question isn’t whether AI agents will crawl you. They already are. The question is what they’ll see when they arrive.
If the answer is “a blank page and a spinner,” you’ve already lost half your audience.
Part 1 of an ongoing series on building for the agent web. Part 2 will cover the Inkwell docs site — VitePress, SoftwareApplication schema, and what happens when a product page gets crawled by every AI assistant on the market simultaneously.