DATAAI SEARCHROBOTS.TXTLLMS.TXTSTRUCTURED DATA

Can ChatGPT Read YC Startup Websites? We Checked 826

BY VLADISLAV GERASIMCHUK, FOUNDER OF ROASTWEB.COM AND AI PLATFORMS EXPERT••9 MIN READ
Can ChatGPT Read YC Startup Websites? We Checked 826

We checked the websites of 826 startups from Y Combinator's four 2026 batches to see what AI search tools can read on them. For each site we read robots.txt the way crawlers do, looked for an llms.txt file, read the structured data in the homepage, and loaded the homepage in a browser with JavaScript on to compare its text with the HTML the server sends. Almost none of these startups shut out AI search, but about one in 13 shows most of its homepage text only to visitors whose browser runs JavaScript.

In one line: Of 826 YC 2026 startup websites checked in November 2026, only 2 block an AI search crawler in robots.txt, 11% block AI training crawlers (mostly through Cloudflare's managed robots.txt), 44% have a real llms.txt file, and 7.7% of homepages have less than a quarter of their text in the HTML the server sends.

Key takeaways

  • ▸Blocking AI search is rare. Only 2 sites block OAI-SearchBot, Claude-SearchBot or PerplexityBot, the crawlers ChatGPT search, Claude and Perplexity use to find pages.
  • ▸92 sites (11.1%) block AI training crawlers such as GPTBot and ClaudeBot. 90 of them still allow every AI search crawler, and 74 of the 92 use Cloudflare's managed robots.txt.
  • ▸On 62 of 807 homepages (7.7%), most of the text is added by JavaScript after the page loads. Vercel's 2024 study found that none of the major AI crawlers ran JavaScript.
  • ▸360 sites (43.6%) have a real llms.txt file. Another 68 answer /llms.txt with their normal web page, which a quick check can mistake for the file.
  • ▸39% of homepages have no structured data at all, while 58% describe the company with Organization or Person markup.

How we checked the sites

The company list comes from Y Combinator's directory, through the open yc-oss API as published on November 6, 2026: 830 companies in the Winter, Spring, Summer and Fall 2026 batches. 828 list a website, and 826 of those websites answered.

For each site we:

  • ▸fetched robots.txt and read it by the rules of the robots.txt standard (RFC 9309), for each crawler on its own
  • ▸requested /llms.txt and counted it only when the answer was a plain-text file, not a web page
  • ▸downloaded the homepage the way a browser asks for it and read the text, headings, tags and structured data in that HTML
  • ▸loaded the homepage in headless Chrome with JavaScript on and counted the words on screen

The robots.txt, llms.txt and structured data checks are the ones behind RoastWeb's AI search score. The browser comparison was added for this study.

The homepage numbers, the browser comparison included, cover 807 sites, because 17 showed a bot-check page instead of their homepage and 2 returned errors. A site can also block crawlers at its firewall or CDN, which robots.txt doesn't show, and we didn't test that. The companies are young and their sites change often, so the numbers describe early November 2026.

Do YC startups block AI search crawlers?

Almost never. Each AI company runs a separate crawler for search, and its documentation says what blocking it does. OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity."

Search crawlerUsed forSites blocking it
OAI-SearchBotChatGPT search1 of 826
Claude-SearchBotClaude search1 of 826
PerplexityBotPerplexity2 of 826

One of those sites blocks every crawler, Googlebot included, with "User-agent: *" followed by "Disallow: /". The other blocks PerplexityBot by name, in its list of AI crawlers. On the other side, 159 sites (19%) name at least one AI search crawler in robots.txt, and all but one of them allow it.

Which crawlers are for training, and does blocking them matter?

Training crawlers collect pages that may be used to train AI models. Blocking them is a decision about training and doesn't take a site out of AI search. OpenAI says GPTBot crawls content "that may be used in training our generative AI foundation models." Anthropic says restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets." Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Training crawlerRun bySites blocking it
BytespiderByteDance90 (10.9%)
CCBotCommon Crawl79 (9.6%)
GPTBotOpenAI74 (9.0%)
Google-ExtendedGoogle73 (8.8%)
Applebot-ExtendedApple73 (8.8%)
ClaudeBotAnthropic72 (8.7%)

92 sites (11.1%) block at least one training crawler, and 90 of them still allow all three search crawlers.

74 of those 92 have a section in robots.txt that starts with "BEGIN Cloudflare Managed content". Cloudflare's documentation says it adds managed rules to robots.txt "only when Enable Bot Preference Sync is turned on" and at least one of the site's AI policies is set to block, and that its training setting also "enables edge blocking for AI training crawlers." If your site runs behind Cloudflare, open your robots.txt and make sure the managed section says what you want.

Can AI crawlers read the homepage without JavaScript?

Most can, but about one in 13 can't read much. On 62 of the 807 homepages we loaded in a browser (7.7%), less than a quarter of the words on screen were in the HTML the server sent. Once JavaScript ran, the median page among them showed 439 words, and 53 of them had fewer than 20 words in the HTML.

AI crawlers read that HTML. In Vercel's December 2024 analysis of crawler traffic, "none of the major AI crawlers currently render JavaScript," a list that included OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot and PerplexityBot. Google's Gemini was the exception, because it uses Googlebot's infrastructure to render pages. A crawler that doesn't run JavaScript sees these homepages as close to empty.

How the site was built made most of the difference:

Built withHomepagesText mostly from JavaScript
Next.js3796
React or Vue single-page app8949
Astro700
Framer320
Webflow250
Lovable142
Other or not detected1985

49 of the 89 client-side React or Vue apps (55%) sent HTML with almost no text, against 6 of 379 Next.js sites and none of the Astro, Framer or Webflow sites. A single-page app sends a nearly empty page and a script, and the script writes the text in the browser. A site that renders pages on the server, or builds them ahead of time, sends the text in the HTML.

How many startups have an llms.txt file?

360 of 826 sites (43.6%) have a real llms.txt: a plain-text file at /llms.txt, usually a markdown list of the site's main pages with a line about each. Another 68 sites answer /llms.txt with their normal web page and a success status, so a check that only looks at the status code counts a file that isn't there.

The file is a proposal from llmstxt.org, which suggests "adding a /llms.txt markdown file to websites to provide LLM-friendly content." Google says you "don't need to create new machine readable files, AI text files, or markup" to appear in its AI features, so an llms.txt helps the tools that read it and makes no difference to Google.

Do they use structured data?

Structured data on the homepageSites (of 807)
Any JSON-LD with a type494 (61.2%)
Organization or Person466 (57.7%)
sameAs links to other profiles381 (47.2%)
SoftwareApplication172 (21.3%)
FAQPage104 (12.9%)
None at all313 (38.8%)

Organization markup with sameAs links to the company's LinkedIn, X and Crunchbase pages tells search engines and AI tools which company a site belongs to. Every JSON-LD block we found parsed cleanly and named a type.

Titles, descriptions and headings

Most of the basics were in place. Every homepage had a title, and 93% had a meta description of at least 50 characters. 10% had no H1 heading and 7% had more than one. 76% set a canonical URL and 80% had an og:image for link previews.

Four homepages carry a noindex tag, which asks Google and Bing to leave the page out of their results. That can be on purpose for a company still in stealth, and it's the first thing to check when a site shows up nowhere.

What to check on your own site

  1. ▸Open your robots.txt. Make sure OAI-SearchBot, Claude-SearchBot and PerplexityBot aren't blocked, by name or by a "User-agent: *" group with "Disallow: /". Blocking GPTBot or ClaudeBot is a separate decision about training.
  2. ▸Read the Cloudflare section if your robots.txt has one, and check that its rules match what you want.
  3. ▸View the page source of your homepage. If your headline and product description aren't in it, crawlers that don't run JavaScript can't read them. Server-side rendering or static generation puts them back in the HTML.
  4. ▸Add Organization markup with your company name, logo and sameAs links to your profiles.
  5. ▸Add an llms.txt that lists your most useful pages, if people find you through AI tools.

Run a free RoastWeb scan to see these checks for your own homepage. For the details of each check, see our guide to technical SEO for AI agents, and for speed and security numbers from small business sites, what 430 website scans show.

Frequently Asked Questions

Does blocking GPTBot keep my site out of ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers. You can block GPTBot and still allow OAI-SearchBot.

How many websites have an llms.txt file?

In this check, 360 of 826 YC 2026 startup websites (43.6%) had a real llms.txt file in November 2026. Another 68 answered /llms.txt with their normal web page instead of a text file.

Do AI crawlers run JavaScript?

In Vercel's December 2024 analysis, none of the major AI crawlers rendered JavaScript, including OpenAI's crawlers, ClaudeBot and PerplexityBot. Google's Gemini was the exception because it uses Googlebot. Text that only appears after JavaScript runs may never reach the others.

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot finds pages for ChatGPT search results. GPTBot collects content that may be used to train OpenAI's models. Each follows its own rules in robots.txt, so a site can allow one and block the other.

Why does my robots.txt have a Cloudflare section?

Cloudflare can manage part of a site's robots.txt. When Bot Preference Sync is on and one of the site's AI policies is set to block, Cloudflare adds a section marked BEGIN Cloudflare Managed content with rules for AI crawlers. 74 of the 826 sites in this check had one.

How can I check my own site?

Run a free RoastWeb scan on your homepage. Its AI search score reads robots.txt for the AI search and training crawlers, checks how much of your text is in the HTML, looks for an llms.txt file and checks your structured data.

Sources

TEST YOUR
WEBSITE NOW

Get a free, brutally honest audit.

ROAST MY SITE →
←BACK TO ALL POSTS