Why Most Published Data Is Invisible to AI
Businesses publish data constantly ... pricing tables, product specs, research findings, directory listings ... but most of it lives inside HTML markup with no structured layer behind it. A human reading the page can tell it is a dataset. An AI system crawling the page cannot, unless the data is exposed in a format it can parse without guessing: a discrete file, a defined schema, and a stable location. Without those three things, the same numbers that a person can read in a table are effectively invisible to a retrieval system.
Pick a Format Built for Machines, Not Just Browsers
The underlying data should exist as a structured file, typically JSON, independent of the HTML page that displays it. JSON is preferred over CSV for anything with nested relationships (a product with multiple specs, a listing with multiple categories) because it preserves structure that a flat CSV loses. The file should be reachable at a predictable, stable URL, not generated only client-side by JavaScript. If a dataset only exists as rendered HTML with no underlying file, it is content, not a dataset, as far as most AI retrieval systems are concerned.
Describe It With Dataset Schema
Schema.org defines a Dataset type specifically for this purpose, with properties like name, description, license, distribution (the actual file URL and encoding format), creator, and dateModified. Adding a Dataset JSON-LD block to the page that hosts or describes the data tells AI systems, unambiguously, that this is a structured dataset, what it contains, who published it, and where the machine-readable file lives. This is the same principle behind the TechArticle and FAQPage schema used elsewhere on this site: give the machine an explicit label instead of asking it to infer one from prose.
Publish a Catalog, Not Just Individual Files
A single dataset file is easy to find once you know it exists. The harder problem is discovery: how does an AI system learn that the dataset exists in the first place? The answer is a catalog endpoint, commonly something like /ai/catalog.json, that lists every published dataset on the site with its title, description, format, and URL. Referencing that catalog from llms.txt or an AI sitemap means a retrieval system only has to find one entry point to learn everything a site has published, rather than crawling the entire site hoping to stumble onto structured files.
License and Version It Explicitly
AI systems that retrieve and reuse data need to know what they are allowed to do with it. An explicit license, whether that is a standard license like CC-BY or a custom usage statement, removes ambiguity that would otherwise get resolved by an AI system simply guessing or declining to cite the source at all. Include an updated_at or dateModified value on every dataset and bump it whenever the underlying data changes, so retrieval systems can tell a stale cached copy from the current version instead of silently citing outdated numbers.