Web Crawler MCP Server (SEO Crawler) by Insightful Pipe
Comprehensive SEO auditing and web crawling through AI with MCP.
Part of the Insightful Pipe MCP Server Collection — The Web Crawler MCP server enables Claude, ChatGPT, Cursor, and other AI assistants to perform SEO audits, extract data, and analyze websites. Run 25+ specialized SEO checks, generate sitemaps, and extract brand information.
MCP Server URL
https://crawler.insightfulmcp.com/
What is Web Crawler MCP?
Web Crawler MCP is a remote Model Context Protocol server that provides AI assistants with comprehensive SEO auditing and web crawling capabilities. This integration allows you to:
- Run 25+ specialized SEO checkers on any URL
- Generate XML sitemaps and robots.txt files
- Extract brand design elements from websites
- Audit technical SEO, content quality, and security
- Analyze mobile responsiveness and page speed
Installation
Claude
- Copy the MCP Server URL:
https://crawler.insightfulmcp.com/ - Open Claude Connectors Settings
- Scroll to the bottom and click Add custom connector
- Paste the URL and click Add
- Click Connect and sign in to InsightfulPipe if asked. Claude then lists the connector as connected.
ChatGPT
Custom MCP servers are added through ChatGPT's Developer mode. Availability depends on your ChatGPT plan, and workspace admins may need to allow it.
- Turn on Developer mode in ChatGPT settings
- Create a new app for a remote MCP server and paste the URL:
https://crawler.insightfulmcp.com/ - Sign in to InsightfulPipe if asked. The connection finishes as soon as you are signed in.
See OpenAI's guide: Developer mode and MCP apps in ChatGPT
Claude Code
claude mcp add --transport http crawler https://crawler.insightfulmcp.com/
Cursor
Add the server to ~/.cursor/mcp.json (all projects) or .cursor/mcp.json (one project):
{
"mcpServers": {
"crawler": {
"url": "https://crawler.insightfulmcp.com/"
}
}
}
When Cursor shows Needs authentication, click Connect and sign in to InsightfulPipe if asked.
Available Actions
33 actions: 33 read, 0 write.
Read Actions (33)
Control What Your AI Can Do
You decide what AI agents can do with each connected account:
- Turn individual actions on or off for every connected account, so agents only see the actions you allow.
- Connect as Read-only or Read & Write. A read-only connection can only enable read actions.
- Destructive actions stay off by default. Actions such as deletes are disabled until an admin enables them.
- Team access per account. Restricted team members only use the accounts they are granted, with the read actions enabled on them.
Usage Examples
Full SEO Audit
"Run a comprehensive SEO audit on https://example.com"
Check Technical SEO
"Check the robots.txt and indexability of https://example.com"
Analyze Internal Links
"Map all the internal links on this page"
Generate Sitemap
"Generate an XML sitemap for https://example.com"
Extract Brand Elements
"Extract the brand colors, fonts, and logos from https://company.com"
Security Audit
"Check the SSL certificate and security headers for this site"
Why Web Crawler MCP?
For SEO Professionals
- Comprehensive audits - 25+ specialized checks
- Technical SEO - Robots, canonical, structured data
- Content analysis - Quality scoring and optimization
For Web Developers
- Debug issues - Find broken links, redirect chains
- Performance - Page speed and resource analysis
- Security - SSL and header validation
For Agencies
- Client audits - Quick site assessments
- Competitive analysis - Analyze competitor sites
- Reporting - Generate findings conversationally
Security & Privacy
- Secure requests - HTTPS support
Pricing
The Web Crawler MCP server is included in every InsightfulPipe plan, together with all other MCP servers and the CLI. Plans start at $29.99/month, and you can try it for 7 days. See insightfulpipe.com/pricing for current plans.
Ready-Made Skills and Prompts
- Claude skills for SEO and marketing — ready-made skills that run on your connected data
- Comprehensive SEO Site Audit
- Technical SEO Health Check
- Content Quality Assessment
Explore More MCP Servers by Insightful Pipe
Visit insightfulpipe.com/mcp-servers to discover our full collection of MCP servers for marketing and analytics.
SEO & Performance MCP Servers
- PageSpeed MCP - Performance analysis
- Google Search Console MCP - Search analytics
- Screenshot MCP - Visual capture
- Google My Business MCP - Local SEO
Analytics MCP Servers
- Google Analytics MCP - Website analytics
- Enrichment MCP - Data enrichment
Resources
Support
- Documentation: insightfulpipe.com/docs
- All MCP Servers: insightfulpipe.com/mcp-servers
- Email: [email protected]
Insightful Pipe — AI-powered marketing analytics through MCP servers. Explore all integrations →
License
MIT License - see LICENSE for details.