SEO & GEO — making comments indexable

Comments are fresh, relevant, user-generated content — exactly what search engines reward. But like every embedded comment system, the GraphComment widget lives in an iframe: crawlers attribute its content to the iframe's domain, not to your page. Out of the box, your comments don't work for your ranking.

GraphComment offers three ways to fix that. The first two follow Google's documented dynamic rendering pattern: bots receive the same comments human visitors see, as plain crawlable HTML. Same content, different form — an accepted practice, not cloaking.

Know your crawlers: SEO vs GEO

Not all bots read pages the same way:

• Search engine crawlers (Googlebot, Bingbot…) render pages in a real headless browser — they execute JavaScript and index the final DOM.
• Generative AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Amazonbot…) fetch the raw HTML and do not execute JavaScript. If your comments only appear via script, these engines never see them.

Making your discussions visible to that second group is what's now called GEO — Generative Engine Optimization. It requires the comments to be in the HTML before any script runs, which only server-side rendering can do.

The three levels

Level 1 — WordPress plugin: SEO mode, AI crawlers included

If you run WordPress, the GraphComment plugin's SEO mode covers everything: detected crawlers are served your native WordPress copies of the comments — real server-rendered HTML, no JavaScript involved. This works because the plugin keeps a WordPress-side backup of every comment and decides server-side, on the user agent, before the page is sent.

That covers classic search engines and, since 5.1.0, the crawlers of generative engines — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, Amazonbot, Bytespider, Mistral and others (20 AI and training crawlers, including Common Crawl's CCBot). The plugin ships a dated copy of our 55-entry crawler reference list, as of September 2026: covering a newly appeared crawler requires a plugin update.

📘

Serving is not consenting

The plugin serves comments to AI crawlers; it does not decide what they may do with them. If you want to opt out of model training while staying visible in AI search, that policy belongs in your robots.txt (e.g. User-agent: GPTBot or Google-Extended directives) — not in the plugin.

Level 2 — SEO inline: one checkbox, zero integration

For any site that embeds the standard widget, whatever your platform. It is included in every plan and on by default on every new site — you can turn it off at any time. The checkbox lives in your back-office under Settings → Graphcomment → Site settings → SEO inline; if your site predates this feature, that's where you switch it on.

When a JavaScript-capable crawler renders one of your pages, the widget script detects it and — instead of creating the iframe — asks our server for the thread and injects the comments as plain HTML directly into your page's DOM, together with schema.org markup (NewsArticle + Comment JSON-LD). The rendered DOM that Google indexes then contains your comments as part of your page.

The honest details:

• JavaScript-capable crawlers only (Googlebot, Bingbot…). Generative AI crawlers never run the widget script — for them, see levels 1 and 3. For JavaScript-capable crawlers that identify themselves as bots, the final decision is made server-side: that list of recognized crawlers grows on our side, without any change on your site.
• The block mirrors what an anonymous visitor sees in the widget: approved, visible comments, pinned first, nothing added for the machines. It is served as plain text (formatting stripped), limited to the first 20 top-level comments with replies flattened beyond one level — a "See all comments" link points to the full thread.
• Human visitors are never affected — they keep the regular widget, and the bot detection adds no network round-trip for them.
• Only the standard comment-thread integration is served this way; AMP, overlay and counter/top-widget integrations keep their regular behavior.
• Responses carry Cache-Control: public, max-age=600 and Vary: Origin, User-Agent. If you put a CDN or a reverse proxy in front of your own pages, honour that Vary: the body depends on the user agent, and a shared cache that ignores it will serve the bot block to humans.
• Pages served to bots this way are not counted in your billed loads, and a fair-use rate limit protects your site from a runaway crawler. (This covers the pages served through this path — it is not a blanket statement about every bot request your site receives.)
• Responses are cacheable (Cache-Control: public, max-age=600).

Level 3 — Server-side, any stack: all plans, free included

Levels 1 and 2 leave one gap: generative AI crawlers outside WordPress. They don't execute JavaScript, so the widget never runs for them — and if you're not on WordPress, the plugin can't help either.

Level 3 closes it. If you control your rendering pipeline (SSR, static generation, edge middleware), you call our endpoint yourself, on behalf of the bot, and inject the HTML into your page. No plan gate, no quota to buy, nothing to ask us for — it's available on every plan, free included. There is no GraphComment code on this path: you make one HTTP call and paste the result.

Before you start: two prerequisites

1. Make sure SEO inline is ticked in your back-office. It's the same checkbox as level 2 (Settings → Graphcomment → Site settings → SEO inline), and it gates both paths. On by default on new sites; on an older site you may have to switch it on. Unticked, the endpoint returns 204 for everyone — server-side calls included. That's by design and it is silent: you get an empty response, not an error.
2. Get your SSO keys (Settings → Authentication → Unidirectional SSO). Server-side calls are signed with your sso_private_key — see Signing your call below. The private key never leaves your server; only the signature travels. You don't need to have set up SSO for this: if your site has no key pair yet, one is generated the first time you open that screen.

The endpoint

GET https://api.graphcomment.com/api/pub/seo-render/<shortname>/thread

And one header that matters more than all the parameters:

⚠️

Forward the bot's User-Agent verbatim

We decide server-side whether the caller is a crawler we serve, based on that header. If you send your HTTP client's own user agent (axios/1.6, GuzzleHttp/7, python-requests/2.31…), you get a 204 with an empty body and no explanation — the most common way this integration silently does nothing. Pass through the User-Agent of the request you received.

Signing your call

The signature proves the call comes from your server. It's the same mechanism as unidirectional SSO, so if you already have SSO you have the code: one string of 3 parts separated by single spaces.

key = <payload_base64> <signature_hex> <timestamp>

• timestamp — current unix time, in seconds, generated at call time.
• signature — hexadecimal HMAC-SHA1 of <payload_base64> + " " + <timestamp>, keyed with your sso_private_key.
• payload_base64 — a base64 string; we don't read its content on this endpoint. Standard or URL-safe base64 both work, and shorter is better — e30= (base64 of {}) is the recommended value.

A signature expires about 3 minutes after its timestamp. Sign a fresh one on every call — never cache it, never hardcode the timestamp.

// Node — the whole signing step
const crypto = require('crypto');

function signCall(ssoPrivateKey) {
  const payload = Buffer.from('{}').toString('base64');      // 'e30='
  const ts = Math.floor(Date.now() / 1000);
  const sig = crypto.createHmac('sha1', ssoPrivateKey)
    .update(`${payload} ${ts}`)
    .digest('hex');
  return `${payload} ${sig} ${ts}`;
}

An absent, malformed, expired or wrong signature is never an error: the call is simply treated as a regular one. You still get your comments — they just count against the per-IP limit.

What you get back

• 200 — a ready-to-inject <section class="graphcomment-seo" data-gc-seo="inline">…</section> block with the comments and a <script type="application/ld+json"> block (NewsArticle + Comment markup). Paste it as-is.
• 204, empty — nothing to render. Legitimate and common: we have no thread for that identity yet — which covers most never-commented pages, though a thread that exists with zero comments returns a 200 with an empty block — SEO inline unticked, the user agent isn't a crawler we serve, the page is a forum topic, or our side is temporarily unable to resolve the thread. Serve your page without the block — never show an error, never cache a 204 (we send Cache-Control: no-store on purpose).
• 400 — no usable thread identity in your query. Fix your parameters; this one is a bug on your side.
• 404 — unknown shortname.
• 429 — you hit the fair-use limit (see below). Same behavior as 204: serve the page without the block.

Fail open, always. A timeout, a 429, a 204, a network blip — in every case you render your page without the comments block. Your page must never depend on our response.

Rate limit and caching

Two limits apply, both over a one-minute window: 120 requests per minute per site, and 300 requests per minute per IP.

The per-IP one is the one a server-side integration meets first, because it funnels all of your bot traffic through a single egress address — that is exactly the limit a signed call lifts. The per-site limit of 120/min still applies to signed calls; it is usually plenty, but remember that pages without any discussion consume the quota just like the others (the limit is applied before we know whether a thread exists). Two habits keep you well under it:

• Cache the returned HTML for ~10 minutes, keyed by the thread — not by the request URL. Your signed key changes on every call, so caching by URL would never hit. Ten minutes matches our own Cache-Control: public, max-age=600.
• Don't call us on pages that have no discussion. If you know a page has no comments, skip the call entirely.

Express

const express = require('express');
const crypto = require('crypto');

const APP = express();
const GC_BASE = 'https://api.graphcomment.com/api/pub/seo-render';
const SHORTNAME = process.env.GC_SHORTNAME;
const SSO_PRIVATE_KEY = process.env.GC_SSO_PRIVATE_KEY;   // server-side only
const CACHE = new Map();                                   // thread id -> { html, expires }

function signCall() {
  const payload = Buffer.from('{}').toString('base64');
  const ts = Math.floor(Date.now() / 1000);
  const sig = crypto.createHmac('sha1', SSO_PRIVATE_KEY).update(`${payload} ${ts}`).digest('hex');
  return `${payload} ${sig} ${ts}`;
}

// Serve the block to CRAWLERS ONLY. This test must come BEFORE the cache lookup:
// the cache is keyed by thread, so a cached entry warmed by a bot would otherwise be
// returned to human visitors — who also get the widget iframe, and therefore see the
// comments twice. (The `data-gc-seo` marker does not save you here: the loader checks
// the user agent first, so it never applies that guard to a human.)
function isCrawler(ua) {
  return /bot|crawler|spider|-user|gptbot|claudebot|perplexitybot|applebot/i.test(ua);
}

async function fetchComments(threadUid, userAgent) {
  if (!isCrawler(userAgent)) return '';             // humans get the widget, nothing else

  const cached = CACHE.get(threadUid);
  if (cached && cached.expires > Date.now()) return cached.html;

  const url = new URL(`${GC_BASE}/${SHORTNAME}/thread`);
  url.searchParams.set('uid', threadUid);           // your thread identity
  url.searchParams.set('key', signCall());          // fresh on every call

  try {
    const res = await fetch(url, {
      headers: { 'User-Agent': userAgent },         // the BOT's UA, forwarded as-is
      signal: AbortSignal.timeout(2000),            // never let us slow your page down
    });
    if (res.status !== 200) return '';              // 204 / 429 / 4xx -> render without the block
    const html = await res.text();
    CACHE.set(threadUid, { html, expires: Date.now() + 10 * 60 * 1000 });
    return html;
  } catch (err) {
    return '';                                      // timeout or network error -> fail open
  }
}

APP.get('/articles/:slug', async (req, res) => {
  const article = await loadArticle(req.params.slug);
  // Skip the call entirely when you already know there's no discussion.
  const comments = article.commentCount
    ? await fetchComments(article.id, req.get('user-agent') || '')
    : '';
  res.send(renderPage(article, comments));
});

Cloudflare Worker

Runs at the edge, in front of an origin you don't want to touch: it injects the block into the HTML your origin already returns.

const GC_BASE = 'https://api.graphcomment.com/api/pub/seo-render';

async function signCall(ssoPrivateKey) {
  const payload = btoa('{}');
  const ts = Math.floor(Date.now() / 1000);
  const key = await crypto.subtle.importKey(
    'raw',
    new TextEncoder().encode(ssoPrivateKey),
    { name: 'HMAC', hash: 'SHA-1' },
    false,
    ['sign'],
  );
  const mac = await crypto.subtle.sign('HMAC', key, new TextEncoder().encode(`${payload} ${ts}`));
  const sig = [...new Uint8Array(mac)].map((b) => b.toString(16).padStart(2, '0')).join('');
  return `${payload} ${sig} ${ts}`;
}

export default {
  async fetch(request, env, ctx) {
    const page = await fetch(request);                       // your origin, untouched
    const userAgent = request.headers.get('user-agent') || '';
    const uid = threadUidFor(new URL(request.url));          // your mapping page -> thread
    if (!uid || !page.headers.get('content-type')?.includes('text/html')) return page;

    // CRAWLERS ONLY, and BEFORE the cache lookup — the cache is keyed by thread, so an
    // entry warmed by a bot would otherwise be injected into human pages, on top of the
    // widget iframe. The bug only shows once the cache is warm: never in staging.
    if (!/bot|crawler|spider|-user/i.test(userAgent)) return page;

    // Edge cache keyed by THREAD, not by the (always different) signed URL.
    const cacheKey = new Request(`https://gc-seo.internal/${uid}`, { method: 'GET' });
    const cache = caches.default;
    let block = await cache.match(cacheKey);

    if (!block) {
      const url = new URL(`${GC_BASE}/${env.GC_SHORTNAME}/thread`);
      url.searchParams.set('uid', uid);
      url.searchParams.set('key', await signCall(env.GC_SSO_PRIVATE_KEY));
      let res;
      try {
        res = await fetch(url, {
          headers: { 'User-Agent': userAgent },
          signal: AbortSignal.timeout(2000),
        });
      } catch (err) {
        return page;                                         // fail open
      }
      if (res.status !== 200) return page;                   // 204 / 429 / 4xx -> fail open
      block = new Response(await res.text(), {
        headers: { 'Cache-Control': 'public, max-age=600' },
      });
      ctx.waitUntil(cache.put(cacheKey, block.clone()));
    }

    const html = await block.text();
    return new HTMLRewriter()
      .on('#graphcomment', {
        element(el) { el.append(html, { html: true }); },
      })
      .transform(page);
  },
};

Next.js middleware / server component

In the App Router, fetch in the server component and render the block next to your widget container. Node's crypto is available in the Node.js runtime; on the Edge runtime, use the crypto.subtle version from the Worker example above.

// app/articles/[slug]/page.jsx  — server component
import crypto from 'node:crypto';
import { headers } from 'next/headers';
import { unstable_cache } from 'next/cache';

const GC_BASE = 'https://api.graphcomment.com/api/pub/seo-render';

function signCall() {
  const payload = Buffer.from('{}').toString('base64');
  const ts = Math.floor(Date.now() / 1000);
  const sig = crypto
    .createHmac('sha1', process.env.GC_SSO_PRIVATE_KEY)
    .update(`${payload} ${ts}`)
    .digest('hex');
  return `${payload} ${sig} ${ts}`;
}

// Cached per thread, and CRAWLERS ONLY. `unstable_cache` keys on the second argument —
// unlike `fetch`'s Data Cache, which keys on the URL. Since our URL carries a signature
// that changes on every call, a bare `fetch` cache never hits: one upstream call per
// request, and your per-site quota burns for nothing. (`next: { tags }` does not help —
// tags drive `revalidateTag`, they are not part of the cache key.)
const cachedBlock = (uid, userAgent) =>
  unstable_cache(
    async () => {
      const url = new URL(`${GC_BASE}/${process.env.GC_SHORTNAME}/thread`);
      url.searchParams.set('uid', uid);
      url.searchParams.set('key', signCall());
      const res = await fetch(url, {
        headers: { 'User-Agent': userAgent },
        cache: 'no-store',                                   // the outer cache is the real one
        signal: AbortSignal.timeout(2000),
      });
      return res.status === 200 ? await res.text() : '';
    },
    [`gc-seo:${uid}`],
    { revalidate: 600 },
  )();

async function getCommentsHtml(uid) {
  const userAgent = (await headers()).get('user-agent') || '';
  // CRAWLERS ONLY, and before the cache: a cached entry must never reach a human, who
  // also gets the widget iframe and would see the comments twice.
  if (!/bot|crawler|spider|-user/i.test(userAgent)) return '';

  try {
    return await cachedBlock(uid, userAgent);
  } catch (err) {
    return '';                                               // fail open
  }
}

export default async function ArticlePage({ params }) {
  const { slug } = await params;
  const article = await loadArticle(slug);
  const comments = article.commentCount ? await getCommentsHtml(article.id) : '';

  return (
    <>
      <Article data={article} />
      {comments ? <div dangerouslySetInnerHTML={{ __html: comments }} /> : null}
      <div id="graphcomment" />
    </>
  );
}

The HTML we return is sanitized server-side — comment content is served as plain text, with formatting stripped — so injecting it directly is safe.

⚠️

Mark your block and avoid double rendering

If the page also loads the widget script, keep the data-gc-seo attribute that comes with our block (or add your own — see Avoid double rendering). Without it, a bot that does run JavaScript would get your server-rendered comments and the widget's inline block.

Is your site letting AI crawlers in?

Everything above assumes the crawlers can actually reach your pages. Often they can't — and nothing on our side can tell you, because the block happens on your infrastructure, before the request ever gets to us. If your comments aren't showing up in AI answers, check these four things before anything else.

1. Your robots.txt

Generative crawlers honour robots.txt. A single Disallow line, or a directive inherited from a template you copied years ago, is enough to make all of this inert:

User-agent: GPTBot
Disallow: /

The user agents that matter here: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Google's Gemini training token), Meta-ExternalAgent, Amazonbot, Applebot-Extended, and CCBot (Common Crawl).

CCBot deserves a moment. Common Crawl isn't an answer engine — it won't send you a visitor or a citation. It builds the public corpus that a large share of open training datasets derive from. Blocking it costs you nothing today and removes your discussions from tomorrow's models. If your goal is to be known by AI, leave it open; if your goal is to opt out of training while staying visible in AI search, that's exactly the distinction Google-Extended and Applebot-Extended exist for.

2. Your WAF or CDN

This is the one that catches most people. Cloudflare has blocked AI crawlers by default since 2025 — the Block AI bots / AI Scrapers and Crawlers setting, under Security → Bots. It can be on without anyone having turned it on. Other CDNs and WAFs ship comparable rules. An open robots.txt in front of a WAF that returns 403 achieves nothing.

3. Your host

Shared hosting often carries its own bot rules, invisible from your control panel. And some hosts throttle or ban aggressively on user-agent patterns.

4. The proof: your access logs — not a spoofed request

Here's the test everyone reaches for first:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" -I https://your-site.example/some-article/

It's worth running, but it can lie to you in the most convincing way possible — a 403 or 429 here does not prove that real crawlers are blocked.

⚠️

A 403/429 on a spoofed user agent usually means anti-spoofing, not a block

Many hosts reject a request that claims to be a crawler but comes from an address that provably isn't one. We measured this on our own blog (OVH shared hosting, August 2026): a spoofed CCBot user agent from an office IP got 429 Your IP has been banned, while the real CCBot was getting 200s on the very same site the same month. The curl test said "blocked". The truth was the opposite.

So treat curl as a smoke test, and settle the question with one of these two, which can't be faked:

• Your access logs, filtered by publisher IP range. Look for the crawler's user agent and check the address it came from against the publisher's published ranges (OpenAI, Perplexity, Google's and Bing's verification tools). This matters more than it sounds: on our own logs, 99.5 % of the requests claiming to be CCBot in August 2026 were scanners spoofing the user agent. A raw grep on the user agent measures noise. What you want to know is: did a request from a real crawler's address get a 200?
• A Common Crawl capture. Search your domain in the Common Crawl index. If recent snapshots of your pages are there, CCBot is reaching you — no interpretation needed. It's a positive control, and it's free.

Summary

Avoid double rendering

If you already render comments into your HTML server-side and the page also loads the widget script, mark your server-rendered block with the data-gc-seo attribute (any value):

<section data-gc-seo="server">…server-rendered comments…</section>

When a bot triggers the SEO inline path, the widget script checks for an existing [data-gc-seo] element and steps aside — your server-rendered block is never duplicated.

How to check

• WordPress (level 1): curl -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://your-site.example/your-post/ — the response HTML contains the native WordPress comments instead of the widget container.

• SEO inline (level 2): in Chrome DevTools → Network conditions → set the user agent to Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and reload: the iframe is replaced by a <section data-gc-seo="inline"> block containing your comments. Google's Rich Results Test shows the same block in the HTML rendered by the real Googlebot.

• Server-side (level 3): call the endpoint the way your integration does, with a real crawler user agent and a fresh signature — you should get 200 and a body starting with <section class="graphcomment-seo". Then load the page itself with curl (no JavaScript) and confirm the block is in the HTML your server returns:

  Bash

  curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
    https://your-site.example/your-article/ | grep -c 'data-gc-seo'

  A 0 means your integration didn't inject anything. In order of likelihood: the User-Agent wasn't forwarded, the thread identity was missing or wrong (you'd get a 400), or SEO inline is unticked in your back-office (you'd get a 204 — see the prerequisites above).

• Search Console: URL inspection → View crawled page — the comments appear in the rendered HTML.

• Are the crawlers even getting in? If everything above checks out but you're still invisible in AI answers, the problem is upstream of us — see Is your site letting AI crawlers in?.