Knowledge Base
Integrating robots.txt and llms.txt with a GPTWeb Sidekick Site
Great question! When you run GPTWeb as a Sidekick Site on a separate subdomain (like gpt.yourcompany.com) rather than your main domain, you need to handle robots.txt and llms.txt at both the main site and the Sidekick site, since search engines and AI crawlers treat each hostname independently. The goal is to make sure traditional SEO crawlers and AI agents both discover and correctly index your conversational content.
robots.txt and llms.txt Setup for Sidekick Site
| Location | File | Recommended Setup |
|---|---|---|
| Main domain (yourcompany.com) | robots.txt | Allow standard crawlers; add a Sitemap reference pointing to the Sidekick subdomain sitemap |
| Main domain (yourcompany.com) | llms.txt | List key product and company URLs, plus a link to the Sidekick site so LLM crawlers discover the conversational layer |
| Sidekick subdomain (gpt.yourcompany.com) | robots.txt | Allow GPTBot, ClaudeBot, PerplexityBot, Googlebot; reference the Sidekick sitemap |
| Sidekick subdomain (gpt.yourcompany.com) | llms.txt | Curated index of your top prompts and topics so AI agents know what conversations are available |
| DNS / Hosting | CNAME | Point gpt.yourcompany.com to your GPTWeb tenant URL |
| Cross-linking | HTML head | Add canonical and cross-domain links between main site and Sidekick |
GPTBot, Claude, Perplexity
AI Crawler Coverage
< 30 min
Setup Time
+2-3x
Discoverability Lift
A few practical tips: keep your llms.txt concise and curated (think top 20-50 high-value URLs and prompts, not your entire site), use the Sidekick subdomain's robots.txt to explicitly allow AI bots since some hosts default to blocking them, and add a small link from your main site footer to the Sidekick so both human visitors and crawlers find it. Want help mapping this out for your setup? [](gptweb://modal/demo). Also worth a look: Use Cases and Getting Started. GPTWeb is the future of engagement, websites and marketing automation combined, built for the AI era, built for now.