Power your agent
The publisher's own URL. Not a PlugMCP endpoint.
Host a copy with our keys and logs.
https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-checkThe Claude button above opens Claude with the name and URL filled in. You still confirm it there.
claude mcp add --transport http io.github.kriton-dev-ai-crawler-access-check https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-checkRun in a terminal.
{ "mcpServers": { "io.github.kriton-dev-ai-crawler-access-check": { "command": "npx", "args": [ "-y", "mcp-remote", "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check" ] } }}claude_desktop_config.json
Remote URL or a command on your machine? How the two installs differ
Run on PlugMCP
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
LaunchTools
No tool list yet. Remote servers get one after the first health check.
FAQ
Check declared robots.txt permissions for six AI crawlers at URL paths
{ "mcpServers": { "io.github.kriton-dev-ai-crawler-access-check": { "url": "https://mcp.apify.com/?tools=hermeskd/ai-crawler-access-check", "headers": {} } }}Copy the http config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.
No required environment variable is listed on this record.
AI Crawler Checker — Robots.txt Policy For Six Bots is reached over HTTP.
Last checked 2026-11-09. Check declared robots.txt permissions for six AI crawlers at URL paths
From the publisher
README is not crawled yet. No repository on file.
Check declared robots.txt permissions for six AI crawlers at URL paths
© 2026 MCP Directory.