# We Tested America's 100 Biggest Retailers Before Black Friday > We tested the websites of NRF's 100 biggest US retailers 16 days before Black Friday: phone speed, what AI can read, AI crawler rules and llms.txt. By Vladislav Gerasimchuk, Founder of RoastWeb.com and AI platforms expert. Published 2026-11-10, updated 2026-11-10. Original: https://roastweb.com/blog/100-biggest-retailers-black-friday-test Black Friday is November 27, and Adobe expects AI traffic to US retail sites, meaning shoppers clicking through from AI chats and browsers, to rise 159% on Thanksgiving compared with last year. So on November 10 and 11, we tested the websites of America's 100 biggest retailers, as ranked by the National Retail Federation. We measured how fast each page shows its main content on a phone, how much text a crawler that doesn't run JavaScript gets, which AI crawlers each robots.txt lets in and who publishes an llms.txt. Where a retailer's home page already linked a Black Friday page, we tested that page. **In one line:** Sixteen days before Black Friday, only 10 of America's 100 biggest retailers linked a Black Friday page from their home page. For one in four real Chrome visits, the main content took more than 4 seconds on 9 of the 92 pages Google has data for, 7 sites sent almost no text to crawlers that don't run JavaScript, only Amazon blocks OpenAI's search crawler, and 16 publish an llms.txt. **The quick version** - 10 of the 100 linked a Black Friday page from their home page on November 10 and 11: Walmart, Amazon, Home Depot, Best Buy, Sherwin-Williams, Ulta Beauty, Discount Tire, Kay Jewelers, IKEA and Overstock. - For real Chrome users, 9 of 92 pages took more than 4 seconds to show their main content for one visit in four, led by Hy-Vee at 6.8 seconds. In Google's mobile lab test, the median page took 14.4 seconds. - 7 sites sent under 100 words to a crawler that doesn't run JavaScript while a browser showed hundreds or thousands. Giant Eagle went from 0 words to 4,422. - Of the 73 robots.txt files we could read, only Amazon blocks OpenAI's search crawler, and only Amazon and Aldi block Perplexity's and Claude's. - 35 sites refused our plainly named test bot, and 16 publish an llms.txt. ## How we tested The list is the [NRF Top 100 Retailers 2026](https://nrf.com/research-insights/top-retailers/top-100-retailers/top-100-retailers-2026-list), which ranks companies by their 2025 US retail sales. For a company with several store brands, we tested its biggest brand's site, such as Food Lion for Royal Ahold Delhaize USA and T.J. Maxx for TJX. Every site we tested is listed in [our data file](/blog/100-biggest-retailers-black-friday-test.csv). For each retailer we opened the home page and looked for a link to a Black Friday page. We tested that page if there was one, and the home page if not. Then: - **Speed:** Google's PageSpeed Insights, mobile. It gives two numbers. The lab test loads the page on a simulated mid-range phone over a slow 4G connection. The real-user number comes from the Chrome UX Report: 75% of real Chrome visits over the last 28 days showed the main content within that time. - **Text:** we fetched each page without running JavaScript, as most AI crawlers do, with a test bot that says who it is (RoastWebStudy). Then we loaded it in a standard headless Chrome browser with JavaScript on and counted the words on screen both ways. - **AI crawlers:** we read each robots.txt by the rules of the standard, RFC 9309, for OpenAI's, Anthropic's, Perplexity's and other AI crawlers, on the exact page we tested. - **llms.txt:** we checked for a plain-text file at /llms.txt. We didn't try to get past any bot check. A page that refused our test counts as refused, and it may still load fine for shoppers. The tests ran from a home connection in California, where Discount Tire's site sends visitors to America's Tire, the name the company uses there, so that's the page we tested. ## Only 10 had a Black Friday page on the home page Sixteen days out, most of the biggest retailers hadn't put a Black Friday link on their home page yet, or didn't show one to our test. Ten had one: Walmart (its Black Friday Deals event, November 12 to 15), Amazon, Home Depot, Best Buy, Sherwin-Williams, Ulta Beauty, Discount Tire (as America's Tire), Kay Jewelers, IKEA (early Black Friday deals for IKEA Family members) and Overstock (early access). Academy Sports linked only single-category Black Friday pages, so we tested its home page. ## How fast do the biggest retailers load on a phone? For real shoppers, most of them are fine. In the Chrome UX Report, the median page showed its main content within 2.18 seconds for 75% of visits, and 57 of the 92 pages Google has data for met its "good" mark of 2.5 seconds or less. Google calls anything over 4 seconds poor, and 9 pages were over it: | Retailer | Main content, 75% of visits | |---|---| | Hy-Vee | 6.8 s | | Family Dollar | 6.5 s | | Bass Pro Shops | 5.4 s | | Kay Jewelers | 5.3 s | | Dollar Tree | 4.9 s | | Kroger | 4.8 s | | AutoZone | 4.8 s | | Health Mart | 4.6 s | | Aldi | 4.5 s | The fastest for real users were Nordstrom at 0.5 seconds, Circle K, IKEA and Home Depot at 1.0, and Abercrombie & Fitch and Amazon at 1.2. Google's lab test is harder on everyone. It loads each page on a simulated mid-range phone over a slow 4G connection with nothing saved from earlier visits, and the median page took 14.4 seconds to show its main content. 83 of the 86 pages it could load took over 4 seconds, 61 took over 10, and the median Lighthouse performance score was 31 out of 100. Fourteen pages wouldn't load in the lab test at all. The lab number shows what a shopper on an older phone and a slow connection gets. Taps are slow on some sites too. Google rates a page poor when it takes more than half a second (500 milliseconds) to respond to a tap for one visit in four. Nine pages were over that line: Kroger (1,495 ms), Sephora (996 ms), Bath & Body Works (712 ms), Discount Tire (670 ms), Michaels (561 ms), Family Dollar (558 ms), American Eagle (554 ms), Advance Auto Parts (553 ms) and 7-Eleven (502 ms). ## How much can an AI crawler read? Vercel's study of AI crawler traffic, published in December 2024, found that "none of the major AI crawlers currently render JavaScript," naming OpenAI's three crawlers, Anthropic's ClaudeBot and PerplexityBot. Google's Gemini is the exception, since it uses Googlebot. For most AI tools, a page is the text that arrives before any script runs. 59 sites sent their page to our test bot. On 7 of them, the page held almost no text without JavaScript, while the same page in a browser showed hundreds or thousands of words: | Retailer | Words without JS | Words with JS | |---|---|---| | Giant Eagle | 0 | 4,422 | | Hy-Vee | 84 | 2,608 | | Discount Tire | 28 | 1,745 | | Raley's | 0 | 971 | | Family Dollar | 0 | 896 | | Dollar Tree | 0 | 632 | | Health Mart | 15 | 415 | Of the other 52, we could compare both views on 34. Twenty-nine of them sent at least half their text without JavaScript, and 5 sent less than half: Costco, AT&T, Wegmans, Hobby Lobby and The Exchange. 34 of the 59 had prices in the page HTML, and 26 had no structured data at all, the schema.org code that spells out facts like prices for machines. ## Which retailers block AI crawlers? Very few, on the pages we tested. We could read 73 robots.txt files. Twenty sites refused our test bot even that file, and the Commissary site has none. Of the 73: - **Amazon** blocks OpenAI's search crawler (OAI-SearchBot) and ChatGPT-User, plus Perplexity's and Claude's search crawlers. - **Aldi** blocks every bot it doesn't name. It names and allows OpenAI's crawlers and blocks Perplexity's by name, so Claude's crawlers fall under the block. - **Six** block at least one AI training crawler, such as GPTBot or ClaudeBot: Amazon, Aldi, Food Lion (Common Crawl only), Stater Bros., WinCo Foods and Weis Markets. - **The other 71** let AI search and shopping crawlers read the page we tested. robots.txt is only half the picture. 35 of the 100 sites refused our test bot, which said plainly who it was, with an error or a bot check. AI crawlers announce themselves the same way, so a wall like that can stop them too, unless the retailer lets them through by name, and that part can't be seen from outside. ## Who publishes an llms.txt? 16 of the 100: Target, CVS, Ace Hardware, AT&T, Verizon, AutoZone, Wegmans, Discount Tire, PetSmart, Williams-Sonoma, Dell, Camping World, Dillard's, Kay Jewelers, Urban Outfitters and American Eagle. Each one is a real file that describes the store for AI tools. llms.txt is a proposed plain-text guide to a site for AI tools, first published at [llmstxt.org](https://llmstxt.org/) in September 2024. ## What a small store can take from this - **Test your Black Friday page on a phone now.** PageSpeed Insights shows the lab number and, if your store has enough visits, the real-user number too. - **Put product text and prices in the HTML.** If a page only fills in after scripts run, most AI crawlers get an empty page. - **Check which AI crawlers your robots.txt blocks.** Our guide to [whether ChatGPT can shop your store](/blog/can-chatgpt-shop-your-store) explains which crawler does what. - **Link your Black Friday page from your home page** a week or more ahead, so shoppers and crawlers find it before the day. A free RoastWeb roast of your store gives you an AI search score next to speed and SEO. It reads your robots.txt for the AI search and training crawlers, checks how much of your text is in the HTML, looks for an llms.txt file and checks your structured data. [Run a free roast](/) on your Black Friday page before the weekend. For more data, see [what we found on 826 startup websites](/blog/yc-startups-ai-search-readiness) and [what 430 website scans show](/blog/website-scan-data-2026). ## We'll test again We'll run the same test on Friday, November 20, and on the morning of Black Friday, November 27, and add the results to this page. The November 10 and 11 numbers stay as they are, so the changes are easy to see. ## The data All 100 rows are in a CSV file: the page we tested, both speed numbers, word counts with and without JavaScript, the robots.txt results for six AI crawlers and llms.txt. You're welcome to reuse it with a link to this page: [100-biggest-retailers-black-friday-test.csv](/blog/100-biggest-retailers-black-friday-test.csv). ## Frequently Asked Questions ### Which big US retailers have the slowest websites on phones? In Google's real-user data on November 10, 2026, the slowest pages among NRF's top 100 belonged to Hy-Vee (6.8 seconds to show the main content for 75% of visits), Family Dollar (6.5), Bass Pro Shops (5.4), Kay Jewelers (5.3) and Dollar Tree (4.9). Google rates anything over 4 seconds as poor. ### Do big retailers block ChatGPT and other AI shopping assistants? Rarely in robots.txt. Of the 73 robots.txt files we could read, only Amazon blocks OpenAI's search crawler, and only Amazon and Aldi block Perplexity's and Claude's. But 35 of the 100 sites refused a plainly named test bot, and a wall like that can stop AI crawlers too. ### Can AI crawlers read pages built with JavaScript? Most can't. Vercel's study found that none of the major AI crawlers run JavaScript, with Google's Gemini as the exception through Googlebot. In our test, 7 big retailers sent under 100 words to a crawler that doesn't run JavaScript, while a browser showed hundreds or thousands. ### What is a good page load time for Black Friday? Google's mark for a good Largest Contentful Paint, the moment the main content shows, is 2.5 seconds or less for 75% of visits, and over 4 seconds is poor. 57 of the 92 retailer pages with real-user data met the good mark on November 10. ### How can I test my own store before Black Friday? Run your Black Friday page through PageSpeed Insights for speed, view the page source to see whether your product text and prices are in the HTML, and open /robots.txt to see which crawlers you block. A free RoastWeb roast checks all of these in one report. ## Sources - National Retail Federation, "NRF Top 100 Retailers 2026 List," retrieved November 10, 2026: https://nrf.com/research-insights/top-retailers/top-100-retailers/top-100-retailers-2026-list - Adobe, "U.S. Holiday Shopping Season to Hit Record $275.1 Billion Online, Rising 6.7% YoY," September 28, 2026, retrieved November 10, 2026: https://news.adobe.com/news/2026/09/adobe-us-holiday-shopping-season-to-hit-record - Vercel, "The rise of the AI crawler," December 17, 2024, retrieved November 10, 2026: https://vercel.com/blog/the-rise-of-the-ai-crawler - Google, PageSpeed Insights API v5 and Chrome UX Report, data pulled November 10 and 11, 2026: https://developers.google.com/speed/docs/insights/v5/about - web.dev, "Largest Contentful Paint (LCP)," retrieved November 10, 2026: https://web.dev/articles/lcp - web.dev, "Interaction to Next Paint (INP)," retrieved November 10, 2026: https://web.dev/articles/inp - IETF, "RFC 9309: Robots Exclusion Protocol," September 2022: https://www.rfc-editor.org/rfc/rfc9309 - llmstxt.org, "The /llms.txt file," retrieved November 10, 2026: https://llmstxt.org/ - RoastWeb, the data for this post, November 10 and 11, 2026: https://roastweb.com/blog/100-biggest-retailers-black-friday-test.csv