A growing share of the people looking for a contractor never see a search results page. They ask an assistant, the assistant answers, and the businesses inside that answer get the call. So when we built the 470-page site for America Premier, a residential general contractor in Richmond, Virginia, we treated the AI crawlers the way most sites treat Googlebot: as guests we actively want in the building.
This is not a speculative bet. The plumbing that feeds answer engines is public and documented, and wiring into it took a small fraction of the build. Here is exactly what shipped.
A robots file that says yes
Most robots files are written defensively, and plenty of sites now block AI crawlers outright. We went the other way. America Premier's robots file explicitly welcomes 18 named AI crawlers by user agent, so there is no ambiguity about whether the machines that answer questions are allowed to read the site. The list includes:
- GPTBot, OAI-SearchBot, and ChatGPT-User from OpenAI
- ClaudeBot, Claude-SearchBot, and Claude-User from Anthropic
- PerplexityBot, Google-Extended, Applebot, and Applebot-Extended
- Meta-ExternalAgent, DuckAssistBot, MistralAI-User, Amazonbot, and CCBot, among others
Blocking these bots protects nothing on a marketing site. Every page exists to be read, quoted, and passed along. If a model is going to recommend a remodeler in Richmond, we want it working from the client's actual pages, not from a stale third-party summary.
It is also a reversible decision. A robots file is one deploy away from changing its mind. But the default matters: for a business whose entire site is public marketing content, readable is the point.
llms.txt and the full-content companion
The site also serves an llms.txt file, following the llmstxt.org convention: a structured plain-text brief covering who the company is, what it does, where it works, and how to get an estimate. An assistant that fetches one file gets the whole picture without crawling hundreds of pages.
Alongside it sits a full-content companion, llms-full.txt, which serializes the entire catalog into a single fetch: all 40 services, all 417 sub-service pages, the FAQs, and the pricing ranges with their citations. Both files are generated from the same data sources that render the pages, so they can never say more than the pages do.
That last clause matters more than the format. A hand-maintained brief starts drifting the week after launch. A generated one stays correct for as long as the site does.
Telling the index you shipped
Deploys announce themselves too. The site carries an IndexNow pipeline that pulls the live sitemap and submits every URL to the IndexNow API in batches of up to 10,000 URLs, with the ownership key file served from the site itself. On the full submission, 466 URLs came back accepted.
IndexNow feeds Bing's index, and Bing feeds several of the assistants people actually use. Instead of waiting for a crawler to wander back on its own schedule, the site tells the index the moment something changes. For a site that publishes hundreds of service pages at once, that is how the long tail gets discovered at all.
Making the next step machine-readable
Being readable is half the job. The other half is making the action obvious. Every page on the site already routes human visitors to one booking flow, so we described that flow in structured data as well: a ReserveAction on the business's JSON-LD node and on the booking page itself.
The goal is that when someone asks an assistant how to get an estimate from America Premier, the machine can resolve the question to a URL instead of a shrug. The schema names the action and the target. It is the same one-conversion-path discipline we apply to people, extended to the machines.
Table stakes, not a trick
None of this is exotic. The robots entries are a few lines each. The llms files are generated from the same typed data as the pages. The IndexNow ping is one small script that runs after deploy. What it buys is a fair shot at being the source the answer gets built from.
We now ship llms.txt on virtually every site we build, and the deeper stack, the crawler allowlist, the full-content file, the IndexNow pings, is our default for lead-generation clients. If you are asking your own web team for the short version, it is this:
- A robots file that explicitly allows the AI crawlers you want quoting you
- An llms.txt brief generated from the same data as your pages, so it can never drift
- A sitemap submitted through IndexNow when you ship, not whenever a crawler returns
- Structured data that names your one conversion action, so the machine knows what to do next
Search used to mean ten blue links. Increasingly it means one answer. The businesses quoted inside that answer will be the ones that made themselves easy to read.
