replypier/HELP_CENTER_INFRA.md

Help Center hosting — infrastructure runbook

The public help center is served by the backend itself (host-dispatched SSR — see backend/app/core/help_center_host.py). This runbook covers the deploy-time pieces that live outside the repo: DNS, TLS and the host nginx on the production VPS. frontend/nginx.conf and the docker-compose files need no changes.

As deployed on the ReplyPier VPS (2026-07-14): the base domain is help.replypier.site (orgs serve at {slug}.help.replypier.site). DNS is Route53, so the wildcard cert is issued and auto-renewed with the dns-route53 plugin. Placeholders (<VPS_IPV4>, <ZONE_ID>, …) stand in for the live values; the real ones live on the VPS and in ops notes, not in this public repo.

1. One-time platform setup

DNS (Route53)

Point a wildcard under the domain you own at the VPS. On the cloud box this is the replypier.site hosted zone (<ZONE_ID>):

• *.help.replypier.site. A <VPS_IPV4>
• *.help.replypier.site. AAAA <VPS_IPV6>

One wildcard record covers every org — no per-org DNS. Create them via the API:

aws route53 change-resource-record-sets --hosted-zone-id <ZONE_ID> \
  --change-batch file://wildcard.json   # UPSERT A + AAAA for *.help.<domain>

Backend env (backend.env on the VPS):

HELP_CENTER_BASE_DOMAIN=help.replypier.site
HELP_CENTER_TARGET_IPS=["<VPS_IPV4>"]   # JSON array — see gotcha below

Gotcha: HELP_CENTER_TARGET_IPS is a frozenset setting, so pydantic-settings JSON-parses the env value. A bare IP or comma list (<VPS_IPV4>) fails validation and crash-loops the backend at startup — it must be a JSON array: ["<VPS_IPV4>"]. (Powers the per-org custom-domain A-record verification in §2; unused by the wildcard site.)

Wildcard certificate (DNS-01 via Route53)

HTTP-01 cannot issue wildcards. A dedicated IAM user with the standard certbot Route53 policy (route53:ListHostedZones + GetChange on *, ChangeResourceRecordSets on the zone) has its creds at ~/.aws/credentials (mode 600) on the box. Issue once:

docker run --rm \
  -v ~/replypier/certbot/conf:/etc/letsencrypt \
  -v ~/.aws:/root/.aws:ro \
  certbot/dns-route53 certonly --dns-route53 \
  -d '*.help.replypier.site' -d 'help.replypier.site' \
  --cert-name help.replypier.site \
  --non-interactive --agree-tos -m <ops-email>

Auto-renewal: a daily cron runs the compose certbot service as a one-shot, then reloads nginx only if a cert actually changed:

docker compose -f docker-compose.hostinger.yml \
  run --rm --entrypoint certbot certbot \
  renew --quiet --deploy-hook "echo reload-needed > /var/www/certbot/.reload"
# ...then, if the flag file appeared: rm it && docker compose exec -T nginx nginx -s reload

It uses the certbot/dns-route53 image and mounts /home/replypier/.aws:/root/.aws:ro, so the wildcard renews unattended (the renewal conf remembers authenticator = dns-route53). Existing HTTP-01 webroot certs still renew — the dns-route53 image is a superset of certbot/certbot. renew covers every cert in the shared certbot/conf, not just the wildcard.

The certbot service must stay one-shot — never give it a long-running loop entrypoint. It previously used sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait ${!}; done;', which broke renewal in two non-obvious ways:

1. compose run inherits the entrypoint. The "one-off" cron run never exited, so --rm never fired (a container leaked every night) and the cron's chained && nginx -s reload never ran at all.
2. sh -c '<script>' <args> silently discards the args (they land in $0/$1…, which the script ignores). So it renewed without the --deploy-hook, every 12h — beating the daily cron to the renewal. The cron's own renew then found nothing due, so the hook never fired.

Net effect: certs renewed on disk while nginx kept serving the old ones, masked only by deploys happening to restart nginx. Left alone that expires a live cert. The service is therefore declared with no entrypoint: (the image's default is certbot) and profiles: ["tools"], so compose up -d never starts it while compose run still works (it auto-enables the profile).

Regression check — the command must exit and leave nothing behind:

timeout 60 docker compose -f docker-compose.hostinger.yml \
  run --rm --entrypoint certbot certbot renew --quiet   # ~3s, rc=0
docker ps -a --filter name=certbot   # must be empty

Host nginx server block

Lives at ~/replypier/nginx/conf.d/help-center.conf (mounted read-only into the replypier-nginx-1 container). proxy_set_header Host $host is what makes backend host-dispatch resolve the org. Upstream is the compose service name backend:8000 (nginx shares the replypier-network), not a published host port.

# {slug}.help.replypier.site
server {
    listen 443 ssl http2;
    server_name *.help.replypier.site help.replypier.site;
    ssl_certificate     /etc/letsencrypt/live/help.replypier.site/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/help.replypier.site/privkey.pem;
    include /etc/nginx/conf.d/ssl-params.conf;

    location / {
        proxy_pass http://backend:8000/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

server {
    listen 80;
    server_name *.help.replypier.site help.replypier.site;
    location /.well-known/acme-challenge/ { root /var/www/certbot; }
    location / { return 301 https://$host$request_uri; }
}

2. Per-customer custom domain (runbook step per domain)

Customers add two records (shown in their FAQ admin UI). On the cloud box the CNAME target cname.replypier.site is not provisioned, so verification falls back to an A record pointing at HELP_CENTER_TARGET_IPS — either form works:

• CNAME help.customer.com → cname.replypier.site (or) A help.customer.com → <VPS_IPV4>
• TXT _replypier.customer.com → rp-verify=<token>

After they click Verify domain (DNS checks pass, ssl_status becomes pending), issue the certificate on the VPS — HTTP-01 works because the CNAME already routes to us:

sudo certbot --nginx -d help.customer.com

Add (or let certbot's nginx installer create) a server block identical to the wildcard one but with server_name help.customer.com; and the new cert paths. Reload nginx. The backend's SSL probe (https://<domain>/healthz) flips ssl_status to active on the customer's next verify/status check.

Cloudflare-proxied ("orange cloud") CNAMEs hide the record from DNS lookups and terminate TLS at Cloudflare — customers must grey-cloud (DNS-only) the record for verification, or the CNAME check will fail.

Automation (later)

When manual certbot becomes a burden: a host cron that polls a token-guarded admin endpoint for verified-but-unprovisioned domains and runs certbot --nginx -d <domain> — or move edge TLS to Caddy with on-demand TLS (ask endpoint). Deferred because Caddy would have to own port 443 for the whole VPS.

3. Verification checklist after deploy

1. curl -sS -o /dev/null -w '%{http_code}\n' https://<slug>.help.replypier.site/ → 200 (and http:// → 301).
2. https://<slug>.help.replypier.site in a browser → published FAQs render, widget loads; an article at /a/<faq-slug> returns 200.
3. docker exec replypier-backend-1 printenv HELP_CENTER_BASE_DOMAIN → help.replypier.site (and the backend is healthy, not crash-looping on HELP_CENTER_TARGET_IPS).
4. API unaffected: https://app.replypier.site / https://api.replypier.site/api/v1/... still route normally.
5. POST /ask from the page returns an answer and appears in help_center_queries.
6. docker ps -a --filter name=certbot → empty. compose up -d must not start a certbot container; one that sticks around means the long-running loop entrypoint is back (see the renewal note in §1) and nginx will stop reloading onto renewed certs.

4. Enterprise (cloud) rollout prerequisite

The feature is gated by the help_center plan flag, which lives in the enterprise submodule (separate repo/PR — required before cloud rollout):

• add "help_center": True to PRO and ENTERPRISE in enterprise/models/plan.py::get_default_features (FREE/BASE: False);
• backfill existing plan rows' features JSON with the same values;
• extend the message-limit check to add the org's period help_center_queries count (HelpCenterQueryRepository.count_for_period) to the bot-message count when the org runs on the ReplyPier-hosted model.

Until that PR deploys, cloud orgs see the upgrade lock (plan_allowed: false); self-hosted/OSS installs are unrestricted immediately.

5. Tunables (backend env)

HELP_CENTER_RESERVED_SLUGS (labels never claimable as slugs — must mirror real infra records on the base domain), FAQ_MAX_PAGES_PER_SOURCE (300), FAQ_MAX_BATCH_CHARS (15000), FAQ_IMPORT_MAX_PAGE_CHARS (100000), FAQ_IMPORT_FETCH_TIMEOUT (30s) — the FAQ_* caps bound LLM spend per generation run.