Description
CrawlPod AI Visibility shows you what AI crawlers — ChatGPT, Claude, Perplexity, and others — actually do on your WordPress site, and gives you the basic controls and files they look for. Everything runs locally, on your own server.
AI crawler analytics
See which AI crawlers have visited, how often, and which pages, with a trend over time. The dashboard also includes a “never visited” list: pages no AI crawler has fetched yet, so you know what to check first.
Caching awareness
A page served from a full-page cache never runs PHP, so a crawler visit to that page can’t be logged. CrawlPod detects whether WP Rocket, LiteSpeed Cache, W3 Total Cache, or WP Super Cache is active and shows the specific setting that excludes AI crawler user agents from the cache, so those visits get logged.
AI crawler access control
Allow or block GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, Amzn-SearchBot, Amzn-User, and Meta-ExternalAgent individually. If a physical robots.txt file already exists on your server, CrawlPod detects it and shows the exact lines to add to it manually, since WordPress can’t apply its own settings on top of a physical file.
AI visibility score
A 0–100 score computed locally from checks covering crawler access, structured data, meta tags, heading hierarchy, content depth, sitemap presence, and llms.txt. The weighting behind the score is published on the Score screen.
llms.txt and llms-full.txt
Generated from your actual content and kept up to date automatically as you publish. Choose which post types and individual pages are included; anything your SEO plugin marks noindex is left out.
Works alongside your SEO plugin
Yoast SEO, Rank Math, All in One SEO, SEOPress, and Slim SEO are detected automatically. When one is active, CrawlPod defers to it for structured data instead of generating its own.
Learn more at https://crawlpod.com/wordpress.
External services
This plugin does not connect to any external service and makes no HTTP requests of any kind. Crawler detection, analytics, scoring, and llms.txt generation all run entirely against your own WordPress database and server filesystem.
The crawler settings screen shows a documentation link next to each AI crawler (for example, to openai.com or anthropic.com) pointing to that vendor’s own published crawler documentation, for your reference. These are plain links for you to open in your browser if you choose to — the plugin itself never requests them or any other remote URL.
Screenshots
Installation
- Upload the plugin to
/wp-content/plugins/, or install it through the Plugins screen in your WordPress admin. - Activate the plugin.
- Visit CrawlPod Settings. On first activation, CrawlPod scans your site (SEO plugin, caching plugin, WooCommerce, an existing robots.txt file) and shows what it found.
- Review crawler access, choose which content to include in llms.txt, and set how long to keep analytics data.
- Visit CrawlPod Analytics after AI crawlers have had a chance to visit, and CrawlPod Score for your AI visibility score.
FAQ
-
Does this plugin make any external requests?
-
No. See “External services” below.
-
Will the analytics be accurate if I use a caching plugin?
-
Only for requests that reach PHP — a fully cached page is served before WordPress runs, so that visit isn’t logged. CrawlPod detects WP Rocket, LiteSpeed Cache, W3 Total Cache, and WP Super Cache, and shows the specific setting in each one that excludes AI crawler user agents from the cache so their requests reach PHP and get logged.
-
Why is llms.txt weighted low in the score?
-
Google’s own guidance is that sites don’t need special files for AI-powered Search, and no major AI vendor has confirmed that llms.txt drives citations. It’s included because it costs little to generate, weighted accordingly.
-
Will this create duplicate structured data if I already use an SEO plugin?
-
No. CrawlPod detects Yoast SEO, Rank Math, All in One SEO, SEOPress, and Slim SEO, and defers to whichever is active rather than generating its own.
-
What happens if I have a physical robots.txt file on my server?
-
WordPress only generates its own robots.txt when no physical file exists at the site root. If one does, CrawlPod shows an admin notice with the exact lines to add to that file manually.
-
Does this work on multisite?
-
Yes — each site keeps its own settings, tables, and generated llms.txt.
-
Is a paid version required?
-
No. Every feature described above — AI crawler analytics, the never-visited list, cache-detection guidance, crawler access control, the AI visibility score, and llms.txt generation — is fully functional in the free plugin, with no feature-limited or time-limited restrictions.
Reviews
Contributors & Developers
“CrawlPod AI Visibility” is open source software. The following people have contributed to this plugin.
Contributors
Translate “CrawlPod AI Visibility” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
1.0.0
- Initial release: AI crawler analytics with caching awareness, AI crawler access control, llms.txt/llms-full.txt generation, AI visibility score, SEO plugin conflict handling.