How to Fix robots.txt for ChatGPT and Gemini in Magento 2

Magento 2 stores are being excluded from AI search answers due to outdated robots.txt files written for 2019 Google standards. OpenAI explicitly states that sites opting out of OAI-SearchBot will not appear in ChatGPT search answers.

Why This Matters

Technical visibility in AI assistants depends on specific crawler access that standard wildcard rules often block. While many developers focus on schema and sitemaps, a single ‘Disallow: /’ directive in a staging-inherited robots.txt file serves as a hard gate that invalidates all other AEO signals. This creates technical debt where stores lose cumulative indexing history and real-time search placement because they lack explicit ‘Allow’ rules for the seven primary AI crawlers.

Key Insights

OpenAI’s OAI-SearchBot is the critical crawler for real-time ChatGPT search answers and product recommendations.
Magento’s default wildcard block ‘User-agent: *’ creates a baseline that AI bots inherit, often leading to silent exclusion.
Robots.txt uses first-match semantics; explicit ‘Allow’ rules must precede wildcard ‘Disallow’ rules to be effective.
The seven essential bots for AEO visibility include OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, and Google-Extended.
The angeo/module-aeo-audit tool can automate the verification of these seven bots to ensure pass/fail status.

Working Examples

Correct robots.txt configuration for Magento 2 to allow AI bots before wildcard rules.

User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /

Sitemap: https://yourstore.com/sitemap.xml
Sitemap: https://yourstore.com/llms.txt

User-agent: *
Disallow: /index.php/
Disallow: /*?

Nginx configuration to ensure robots.txt is routed through Magento instead of serving a static file.

location = /robots.txt {
    try_files $uri $uri/ /index.php$is_args$args;
}

Practical Applications

Use Case: Configure Magento admin under Stores → Configuration → General → Design to insert AI ‘Allow’ rules at the top of the file.
Pitfall: Leaving ‘Disallow: /’ from a staging environment in production, which prevents all AI indexing despite valid sitemaps.
Use Case: Update Nginx configurations to route robots.txt requests through the PHP entry point instead of serving static files from pub/robots.txt.
Pitfall: Placing the ‘User-agent: *’ block before specific AI bot rules, causing the bots to ignore their specific ‘Allow’ directives.

References:

On This Page

How to Fix robots.txt for ChatGPT and Gemini in Magento 2