Why publish content for machines, not just humans?

Every AI answer is assembled from retrieved material, and retrieval is cheapest when the material is already structured. A product catalog served as clean JSON with prices, availability and specs is trivially ingestible; the same data scattered across styled HTML pages must be scraped, parsed and guessed at. As agentic usage grows — assistants completing tasks, comparing options, checking availability — the sites that offer machine-readable doors get used by machines. This is the content-side complement to crawler control (robots.txt for AI crawlers): instead of only deciding what bots may take, you decide what you hand them.

Which machine-readable channels exist?

  • JSON-LD on pages. The baseline: schema embedded in HTML. See the structured data deep dive.
  • RSS/Atom feeds. A structured stream of new content — see RSS and AI discovery.
  • JSON feeds and data endpoints. Stable URLs returning current catalogs, pricing, documentation indexes or datasets.
  • Public APIs with OpenAPI documentation. For platforms and marketplaces, a documented API is how agents transact and verify facts programmatically.
  • llms.txt. A markdown index at the site root telling LLM systems what your site offers and where the canonical resources live — see llms.txt.

What belongs in a machine-readable layer?

Expose the facts AI answers ask for most: product names, SKUs, prices, currencies, availability; company facts (name, founding, locations, contact); documentation indexes; changelogs; pricing tables; store locations and hours for local businesses. Every field you serve as structured data is a field the grounding stage can verify exactly, instead of reconciling three differently-worded prose claims. Keep the machine layer consistent with the human pages — divergence between your JSON price and your HTML price is precisely the kind of contradiction grounding penalizes; see grounding fact consistency.

Implementation checklist

  • Start with stable, versioned JSON endpoints for your highest-value fact sets (catalog, pricing, docs index, company facts).
  • Document endpoints with OpenAPI and keep the spec at a well-known URL.
  • Publish and maintain llms.txt pointing to canonical resources, feeds and API docs.
  • Serve correct content types (application/json), permissive CORS for read-only public data, and cache headers that let fetchers be polite.
  • Authenticate anything sensitive; machine-readable does not mean exposing private data — robots.txt is not access control.
  • Monitor which agents and bots actually consume the endpoints in AI crawler analytics.

Common pitfalls

The first is drift: the API says one price, the page says another, because the JSON feed updates hourly and the CMS page updates weekly. Pick one source of truth and generate both surfaces from it. Second, unstable URLs and schemas — every breaking change severs integrations and erodes the reliability that makes agents prefer you. Third, ignoring retrieval-augmented systems entirely: RAG-based agents fetch and quote documents, so well-structured markdown documentation pages are themselves a machine-readable asset — see retrieval-augmented generation. Fourth, over-engineering: a marketplace does not need a public API on day one; JSON-LD plus a clean feed covers most GEO value.

What should stay prose-only?

Not everything belongs in feeds and APIs. Opinion, narrative case writing and long-form analysis are consumed by retrieval systems as text passages, and over-structuring them adds nothing. Reserve the machine-readable layer for facts that agents verify or act on: identity, prices, availability, specs, dates, locations, documentation structure. A useful design test: if a field would embarrass you when wrong in an AI answer — an outdated price, a discontinued product, a wrong phone number — it belongs in a maintained structured channel with one source of truth. If it is inherently interpretive, keep it in well-written prose built for passage retrieval, per answer block writing. The two layers compound: structured facts survive grounding, structured prose wins retrieval.

Where to go next

Anchor your entity data with Organization schema, expose freshness via feeds, and understand consuming systems in RAG.

Frequently asked questions

Do AI engines use public APIs today?
Increasingly, yes — agentic assistants and shopping features consume structured endpoints where available, and documented APIs make your facts the easiest to verify. Even when unused directly, an API forces a single source of truth that keeps pages and markup consistent.
Is llms.txt a standard?
It is a widely discussed emerging convention, not a formal standard. Publishing it is cheap and harmless, and it gives LLM-oriented tools a curated map of your canonical resources.
What is the minimum viable machine-readable layer?
JSON-LD on every page, a valid RSS or JSON feed for new content, and accurate, consistent facts in both. Add APIs when you have data that agents need programmatically, like live availability.