Contentgate

contentgate

Pre-deploy guards for content sites. Python CLI and an MCP server, sharing one test suite.

Documentation and background: adsandcode.com/en/open-source/contentgate

These checks exist because each of them was written the day after the thing it checks for went live and cost something.

negative tests
  keywords
    [ok] expected block got block  primary keyword appears once, threshold is three
    [ok] expected block got block  spaced variant must not satisfy the unspaced keyword
    ...
  160 passed, 0 failed

checks
  [ ok ] keywords   choosing-a-cms (1200/1200 volume covered)
  [FAIL] keywords   headless-vs-monolith (300/750 volume covered)
      missing keyword 'headless cms' (volume 450): found 2 time(s), needs 3
  [ ok ] i18n       fr: 19 document(s)
  [ ok ] lastmod    19 page(s), 0 genuinely changed

[FAIL] 1 check(s) failed. Deploy blocked.

The one design rule

A guard that has never been shown to fail is decoration.

Every check ships negative tests: inputs it is required to reject. The suite runs before any verdict is reported, and a failing suite exits 2 without saying anything about your content. A checker that cannot prove it still catches a known-bad input does not get to tell you your site is fine.

This is not theoretical. Guards written for this project have included: one that read its keyword out of the site-wide footer and therefore passed every page ever; one that ran after the step that regenerated the file it was checking; and one whose exit code was discarded by the shell pipeline it lived in, so it printed "deploy blocked" while the deploy proceeded. All three looked fine. All three were found by asking them to fail on purpose.

The checks

Three details that look fussy and are not:

Whitespace is never normalised in keyword matching. web hosting and webhosting are different strings with different search volumes, and in languages that do not put spaces between words the gap between the two forms can be an order of magnitude. Letting one satisfy the other is how a guard passes a page that targets neither. This was found twice, on two pages, after publication.

Boilerplate is excluded from every body check. A call-to-action block present on every page will otherwise satisfy keywords and internal_links everywhere, which means both checks pass always.

Ties are not broken by dictionary order. Every keyword at the highest volume is primary. The alternative, taking the first maximum, makes the verdict depend on the order of keys in a JSON file, which is stable right up until someone reorders the table.

Two more that changed in 0.2.0:

A commercial link has to point at your own site. A relative link counts. An absolute link counts only when its host is in site_hosts or a subdomain of one, compared on label boundaries and read the way a browser reads it; without site_hosts, absolute links never count. A link to https://competitor.example/pricing/ ends in /pricing/ and sells someone else's product. The money path is matched against the path, not the query string, so /blog/?next=/pricing/ is not a pricing link.

A disclaimer has to negate the claim itself. A negation anywhere in the sentence used to be enough, so 「保證獲利，絕不會虧損」 (guaranteed profit, never a loss) passed because it negates the loss. Now a claim is exempt only when a negator or prohibition stands shortly before it in the same clause (at most three words or two Han characters between, no comma): 不保證獲利, 不做保證獲利宣稱, "we do not guarantee profit", "no guaranteed returns", "never promise guaranteed income". Idioms that only look negative ("not only", "don't miss", "no risk", 不僅, 不斷, 沒有風險) do not count, the 不 inside 穩賺不賠 is part of the claim, every phrase in a governed list shares the exemption, and a phrase named as forbidden ("guaranteed profit claims are banned", 保證獲利等字眼一律不得使用) is not asserted. It is a net for copy review, not a legal opinion, and it errs towards flagging.

Page-level SEO/GEO audit

New in 0.2.0: checks on a built page that is meant to be found, both through search engines (SEO) and through AI assistants that read and cite pages (GEO). Run them on the HTML your server actually sends, not on the source.

Text a crawler sees without JavaScript. Google renders JavaScript before indexing. Many AI crawlers do not: a December 2024 measurement by Vercel found that the crawlers behind ChatGPT and Claude do fetch JavaScript files but do not execute them, and that none of the major AI crawlers it observed render JavaScript. An article written into the page by a script is, to them, a navigation bar and a footer. The check drops <script> and <style>, drops <template> elements and any element with the hidden attribute, children included (aria-hidden is left alone: it hides an element from assistive technology, not from the screen), reads only <main> when the page has one, drops <nav> and <footer>, removes <wbr> and zero-width spaces without leaving a gap (a word broken for line wrapping is still one word), decodes entities and counts what is left. The default of 1200 characters is a house rule, not a number any search engine publishes. Short pages such as author profiles need a lower min_chars or no visible_text check at all.

Structured data, parsed. Every application/ld+json block goes through a real JSON parser, so a trailing comma or a NaN is a failure rather than something a text search would count as present. Types are collected from each item, from @graph, and from author, publisher and creator, where Organization and Person usually sit. The default requirement is Article, BreadcrumbList and Organization. A required type is satisfied by its schema.org subtypes: Article by all eighteen of them (NewsArticle and BlogPosting, which Google's Article documentation names, plus TechArticle, ScholarlyArticle, Report, SocialMediaPosting, LiveBlogPosting, the *NewsArticle family and the rest), and Organization by its direct subtypes (Corporation, NGO, NewsMediaOrganization, EducationalOrganization, OnlineBusiness, LocalBusiness ...) and the common families under them (schools, medical organisations, restaurants and other food establishments, lodging, stores, financial services). The Organization list is a curated part of the schema.org tree, not all of it; the full table is SUBTYPES in src/contentgate/guards/structured_data.py, and a type it does not list has to be required by name. A page that declares ProfilePage is not required to be an Article, since Google's ProfilePage documentation lists an author page on a news site among its intended uses. Requiring a type does not make any search engine show anything; it checks that what you meant to declare is there and parses. dateModified has to be true to mean anything: a date that moves on every build is the same lie the lastmod guard exists to stop.

Outbound links, resolved the way a browser resolves them. Google asks for paid and affiliate links to carry rel="sponsored", user-generated links rel="ugc", and nofollow where neither fits; an ordinary editorial link needs no rel at all, and several values may be given space- or comma-separated. A program cannot tell a citation from a placement, so every external link without one of the three is reported unless its site is in cite_ok. Before deciding where a link goes, the href is read as a browser reads it: tabs and newlines removed, a backslash treated as a slash, any run of slashes after https: starting the host. Without that, https://partner.example\.gov.example/ looks to urllib like a host under gov.example, while every browser opens partner.example. Hosts are compared on label boundaries, so evilexample.org is not under example.org, and a page's own subdomains count as internal. When a tag repeats rel, the first one wins, as it does in the browser.

A reported citation is not a reason to add nofollow to it. Google does not ask for any rel on ordinary editorial links, and marking every citation nofollow to make the check pass would misdescribe them. List the sites you cite in cite_ok instead, matched with their subdomains, for example ["gov.uk", "who.int"]: a suffix only government bodies can register is a reasonable entry. A registry anyone can register under, such as co.uk, github.io or pages.dev, is refused with an error, because listing it would exempt every site below it. Paid or affiliate partners never belong in cite_ok: they need rel="sponsored".

Images. Google uses alt text, together with the image and the page around it, to understand what an image shows; alt="" is a deliberate statement that an image is decorative and passes. A missing width lets text jump when the image arrives. web.dev also accepts reserving the space with CSS aspect-ratio, which an attribute check cannot see; if that is how your site does it, leave images out of the checks.

Favicon. Google reads the icon from a <link> in the head of the home page, so the check runs only when the page URL's path is empty or /. Accepted rel values are the ones Google documents (icon, which shortcut icon contains, apple-touch-icon, apple-touch-icon-precomposed). Google Search "supports the following favicon file formats: BMP, GIF, ICO, PNG, JPEG, PPM, and TIFF", so an SVG or WebP icon on its own fails. The format is read from the file extension of the URL path, ignoring any query string or fragment, or, for a data: URI, from its MIME type (data:image/png;... passes, data:image/svg+xml,... does not). Whether the file exists, its size and whether it is square need the built files and are not checked here.

Stock phrases. An editorial style check, not a detector of machine-written text: it finds phrases you have decided read as filler, and proves nothing about who or what wrote the page. The default list is Traditional Chinese, because that is where it was built and tested (總的來說, 值得注意的是, 綜上所述 and nine more). It is one editor's taste, and it is meant to be replaced: connectives such as 因此， and 然而， will flag perfectly ordinary sentences. For any other language, supply your own list; the defaults will never match. Matching is literal and case-insensitive, on the text a crawler sees, so a phrase inside a script or in the navigation does not count.

Duplicate sentences. Sentences are split on 。！？ and on ., ! or ? followed by whitespace, so 4.99 does not end a sentence; sentences under ten characters are ignored. The share is measured in characters and in both directions, and the larger figure decides, because a short page copied whole into a long one is a small share of the long page and all of the short one. The default limit of 15% is a house rule.

Sources, checked when this release was written: qualifying outbound links (Google Search Central, updated 2025-12-10), favicons in search results (Google Search Central, updated 2026-08-28), ProfilePage structured data (Google Search Central, updated 2026-09-08), Article structured data (Google Search Central, updated 2026-09-08), image SEO (Google Search Central, updated 2026-03-02), optimizing layout shift (web.dev, updated 2025-02-07), and, as a third-party measurement, AI crawlers and JavaScript (Vercel, 2024-12-17).

Install

Python 3.8+, standard library only, no dependencies.

git clone https://github.com/adsandcode/contentgate
cd contentgate
python -m contentgate init          # writes contentgate.json
python -m contentgate check

Exit codes are the contract:

Wire it in front of your deploy and honour the exit code:

python -m contentgate check || exit 1
npx wrangler pages deploy site

The page audit also runs on a single built file, without a config:

python -m contentgate audit site/guide/index.html --url https://www.example.org/guide/
python -m contentgate audit site/index.html --url https://www.example.org/ \
    --cite-ok gov.uk --phrases phrases.txt --only visible_text --only favicon
python -m contentgate compare site/a/index.html site/b/index.html --max-share 0.15

--url is the address the page is served at: it decides which links are external and whether the page is the home page. --phrases is a UTF-8 file with one phrase per line and replaces the default list.

Configuration

contentgate.json, next to your project:

{
  "root": ".",
  "scope": ["article", "post"],
  "exclude_classes": ["cta"],
  "guards": {
    "keywords": {
      "enabled": true,
      "table": "content/keywords.json",
      "page_for_slug": "site/blog/{slug}/index.html",
      "primary_min_hits": 3
    },
    "internal_links": {
      "enabled": true,
      "pages": "site/blog/*/index.html",
      "money_paths": ["/products/", "/plans/"],
      "require_figure": true,
      "site_hosts": ["www.example.org"]
    },
    "translation": {
      "enabled": true,
      "source_dir": "content/blog",
      "locales": ["de", "fr", "ja"],
      "same_script_locales": [],
      "native": "content/blog/_native.json"
    },
    "lastmod": {
      "enabled": true,
      "pages": "site/**/index.html",
      "state": ".contentgate-lastmod.json",
      "strip_classes": [],
      "bulk_change_threshold": 0.3
    }
  }
}

scope is the element that holds the real content, as [tag, class]. exclude_classes is your boilerplate. Both matter more than they look.

same_script_locales is for targets that legitimately share the source script, such as Simplified Chinese from a Traditional Chinese source. Without it, the leftover-characters check fails every document in that locale forever.

bulk_change_threshold stops a run that moves more than that share of lastmod dates at once. Adding a block to every page changes every hash, which moves every date, which is the lie the check exists to prevent. It looks identical to a legitimate mass edit, so the run stops and asks rather than writing silently.

content/keywords.json maps a page to its terms and their real monthly volumes:

{
  "choosing-a-cms": { "headless cms": 450, "cms comparison": 200 }
}

Use numbers from a tool. A guard fed invented volumes enforces invented priorities.

The page audit and the duplicate check read the built site:

{
  "guards": {
    "page_audit": {
      "enabled": true,
      "site_dir": "site",
      "site_url": "https://www.example.org",
      "pages": "**/*.html",
      "skip": ["404.html"],
      "checks": ["visible_text", "structured_data", "outbound_links", "images", "favicon", "stock_phrases"],
      "min_chars": 1200,
      "require_types": ["Article", "BreadcrumbList", "Organization"],
      "require_date_modified": true,
      "cite_ok": ["gov.uk"],
      "stock_phrases": null
    },
    "duplicate_sentences": {
      "enabled": true,
      "site_dir": "site",
      "pages": "**/*.html",
      "skip": ["404.html"],
      "max_share": 0.15
    }
  }
}

Each file is judged under the URL it is served at: index.html is /, guide/index.html is /guide/, guide/faq.html is /guide/faq.html. Files named in skip are left out at every level. A glob that matches nothing fails rather than passing an empty site. stock_phrases: null keeps the default list; [] flags nothing. The duplicate check finds pairs through an index of shared sentences instead of comparing every page with every other.

MCP server

The same checks as tools an assistant can call while it is writing a page, rather than after the page is deployed.

npx mcp-server-contentgate

{
  "mcpServers": {
    "contentgate": { "command": "npx", "args": ["-y", "mcp-server-contentgate"] }
  }
}

Tools: check_keywords, check_translation, check_internal_links, check_claims, sitemap_lastmod, content_hash, audit_page (the six single-page checks above, each switchable) and compare_pages (duplicate sentences between two pages).

Every tool is pure: content in, verdict out. No filesystem access, no network access, nothing written. The server runs its negative tests at startup and refuses to start if they fail.

The same guards are importable as plain functions, without starting anything: import { checkClaims, auditPage } from "mcp-server-contentgate". The package's main is a module with no side effects; only the mcp-server-contentgate command starts the server.

Why there are two implementations

The Python CLI runs in a deploy pipeline. The MCP server runs inside an assistant while the content is being written. Neither replaces the other.

They share fixtures/cases.json, and both are required to produce identical verdicts on every case in it. If they disagree, one has drifted and CI fails.

python -m contentgate selftest     # 160 passed, 0 failed
cd mcp && npm run build && npm run selftest

Agreeing on page markup takes more than the same regular expressions. The Python side reads links, images and icons with the standard library's HTML parser and decodes entities with html.unescape; the TypeScript side carries a port of that parser's tokeniser and of html.unescape with the full entity table, so … is one character in both. The parser itself changed how it reads badly malformed markup in 2025, so the shared cases stick to markup every supported Python version reads the same way.

Contributing

A pull request that adds a check must add its negative tests to fixtures/cases.json and implement it on both sides. That is the whole bar.

If you found a way to make a guard pass something it should have caught, that is the most useful issue you can file.

Licence

MIT.

Maintainer

Built and maintained by Ads & Code.