172 lines
11 KiB
Markdown
172 lines
11 KiB
Markdown
# ChatterMate end-to-end tests
|
|
|
|
Playwright tests that drive the real dashboard, the real widget and the real backend.
|
|
|
|
Each run gets its own stack:
|
|
- a throwaway Postgres database
|
|
- the backend
|
|
- a production build of the dashboard
|
|
- a fake LLM
|
|
- a pretend customer site
|
|
|
|
All of it runs on `127.0.0.1` ports that never touch a dev setup on `localhost`.
|
|
|
|
Unit tests (`backend/tests`, `frontend` vitest) still cover logic and edge cases. These tests cover **journeys**: what a customer or an agent actually does.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
cd e2e
|
|
npm ci
|
|
npx playwright install chromium
|
|
npm test # starts the stack if needed, runs everything
|
|
npm run test:smoke # just @smoke
|
|
npx playwright test specs/handover --headed # one area, watch it
|
|
npm run test:ui # Playwright UI mode
|
|
npm run report # last HTML report (traces on failure)
|
|
```
|
|
|
|
**Prerequisites:**
|
|
- A local Postgres with pgvector, reachable as `postgresql://$USER@127.0.0.1:5432` (set `E2E_PG_URL` otherwise). The user must be able to create databases and the `vector` extension.
|
|
- `backend/venv` with the backend requirements (or set `E2E_PYTHON`).
|
|
- `frontend/node_modules` (or set `E2E_FRONTEND_NODE_MODULES`).
|
|
- [`mailpit`](https://mailpit.axllent.org) on the PATH (`brew install mailpit`). Every email the backend sends lands there.
|
|
|
|
**Faster reruns:** keep the stack up between runs.
|
|
|
|
```bash
|
|
./scripts/stack.sh up # start everything once
|
|
npx playwright test # reuses the running stack
|
|
./scripts/stack.sh reset # fresh database
|
|
./scripts/stack.sh logs backend
|
|
./scripts/stack.sh down
|
|
```
|
|
|
|
**Safety:**
|
|
- `stack.sh` only ever manages `chattermate_e2e` and its template `chattermate_e2e_tpl`. It refuses any database name that does not end in `_e2e`.
|
|
- The backend reads only `e2e/.env.e2e` (through `ENV_FILE`), never `backend/.env`.
|
|
- The OSS edition runs with `NO_ENTERPRISE=1`, and the dashboard is built without `src/modules/enterprise`, so it behaves like a community install even in a checkout that has the submodules.
|
|
|
|
## Editions
|
|
|
|
| | OSS (default) | Enterprise (`E2E_EDITION=enterprise`) |
|
|
|---|---|---|
|
|
| Needs | nothing extra | the enterprise submodules checked out |
|
|
| Backend | `NO_ENTERPRISE=1`, so community behaviour | enterprise module loaded, its migrations and seed applied |
|
|
| Setup project | the first-run form (`specs/setup/`) | real signup with the emailed code (private suite) |
|
|
| Specs | `specs/**` | the private suite's specs (`--project=enterprise`) **and** every OSS spec (`--project=chromium`) |
|
|
|
|
Both setups write the same org contract (`src/org-state.ts`) through `prepareOrg()`, so an OSS spec never needs to know which edition it runs on.
|
|
|
|
The private suite lives in the enterprise frontend module's `e2e/` folder. It imports only `@e2e`, and nothing from it belongs in this repo.
|
|
|
|
## How the code is organised
|
|
|
|
```
|
|
specs/<feature>/*.spec.ts what the product does — no selectors, no raw HTTP
|
|
src/pages/ one class per dashboard screen; locators live ONLY here…
|
|
src/components/ …or here, for UI shared across screens (chat-widget.ts)
|
|
src/api/ typed seeding calls, one module per backend area
|
|
src/fakes/llm.ts scripts what the AI says (see below)
|
|
src/fixtures/ wires it together: agent, widget, inbox, llm, adminApi, org…
|
|
src/index.ts the only import specs use: `import { test, expect } from '@e2e'`
|
|
specs/setup/*.setup.ts runs once per run: org, AI config, support team, signed-in states
|
|
```
|
|
|
|
| When this changes… | …edit only this |
|
|
|---|---|
|
|
| A label or the DOM of a screen | that screen's page object (or the component) |
|
|
| An API route or payload | the module in `src/api/` |
|
|
| What a default test agent looks like | `src/fixtures/index.ts` |
|
|
| The model reply format | `services/fake-llm` (and run `contract_check.py`) |
|
|
|
|
## Rules
|
|
|
|
1. **Locators:** `getByRole` / `getByLabel` first, then `getByText` / `getByPlaceholder`. CSS only inside `src/pages` or `src/components`, with a comment saying why (usually "no accessible name yet"). When you need one, prefer adding an `aria-label` to the app instead.
|
|
2. **Web-first assertions only:** `await expect(locator).toHaveText(…)`. Never use `waitForTimeout`, and never `expect(await x.isVisible())`. Lint enforces both.
|
|
3. **Seed through the API, assert through the UI.** Use the UI for setup only when the setup *is* the feature under test.
|
|
4. **Every test is independent.** Fixtures give each test its own agent and widget with unique names. Never rely on another test's data or on test order.
|
|
5. **Org-wide settings** (AI config, business hours, ticket settings) are changed only in `*.org.spec.ts`. Those run in a separate `--workers=1` invocation (`npm run test:org`).
|
|
6. **Page objects don't assert,** except to wait for their own page to load. Assertions live in specs.
|
|
7. **Wrap user-level actions in `step()`** (`src/support/step.ts`), so reports read like a story.
|
|
8. **Tags in the title:**
|
|
- `@smoke` runs on every PR.
|
|
- `@<feature>` is the folder name.
|
|
- `@slow` marks a slow test.
|
|
- `@quarantine` excludes a test from required runs; it must link an issue.
|
|
9. **Never import `@playwright/test` in a spec.** Import `@e2e` instead, so every spec gets the shared fixtures. This is also what keeps the enterprise suite on a single Playwright copy.
|
|
|
|
## Scripting the AI
|
|
|
|
Each test's agent carries a hidden marker. The fake LLM uses it to find that test's script, so parallel tests never see each other's replies.
|
|
|
|
```ts
|
|
await llm.script(agent, (s) => s
|
|
.when(/refund/i).handOver('Let me get a teammate.') // transfer_to_human
|
|
.when(/broken/i).createTicket('I have logged it.', 'Widget broken') // runs the create_ticket tool
|
|
.when(/@/).captureLead('Thanks!', { email, name }) // request_lead_capture + lead_*
|
|
.when(/thanks/i).endChat('Goodbye!') // end_chat (+ rating if the agent asks)
|
|
.when(/pricing/i).callTool('search_knowledge_base', { query: 'pricing' }).thenReply('Plans start at $10.')
|
|
.otherwise().reply('Happy to help!'))
|
|
|
|
const sent = await llm.requests(agent) // what the backend sent the model: system, lastUser, toolResults…
|
|
```
|
|
|
|
- **Reply format:** replies are structured `ChatResponse` JSON, the same as with a real model. Handover, end-chat and lead capture are fields of that JSON. Tickets are a real tool call (`create_ticket`); the `create_ticket` JSON field alone creates nothing. `reply(message, fields)` sets any other field.
|
|
- **Pace conversations like a person:** wait for the agent's reply before the visitor's next message. Replies to back-to-back messages are produced concurrently and can arrive in either order.
|
|
- **Failures on purpose:** `.when(/x/).fail(500)`, `.after(25_000).reply(…)` (past `AGENT_RUN_TIMEOUT`), `.malformed()`, and `.costing(promptTokens, completionTokens)` for metering.
|
|
- **Contract check:** `services/fake-llm/contract_check.py` runs agno itself against the fake. Run it after upgrading agno.
|
|
|
|
## Channels and other third parties
|
|
|
|
The backend's calls to Telegram, Meta, Slack, LINE, SMS providers, HubSpot, Pipedrive, Jira and Shopify go to `services/fake-externals`. The `*_API_BASE` (and `PIPEDRIVE_OAUTH_BASE`, `JIRA_AUTH_BASE`, `SHOPIFY_ADMIN_BASE`) overrides in `.env.e2e` point them there; production never sets these. Email replies go over SMTP into Mailpit.
|
|
|
|
The fake records every call and answers like the provider would. Use `FakeExternals` to assert on what the backend sent.
|
|
|
|
Each channel has a driver in `src/channels/` that plays the provider:
|
|
- `connect` gives every test its own account, routed to the test agent
|
|
- `chatWith(name).send(text)` delivers a signed webhook exactly as the provider would (the signing secrets match `.env.e2e`)
|
|
- `replies()` reads what the business sent back, from the fake or Mailpit; channels with buttons also have `buttons()` and `tap()`
|
|
|
|
A journey that should hold on every channel loops over `channels` (or `channelsWithButtons`) from `src/channels/index.ts`. Adding a channel is one driver file and one line in that registry.
|
|
|
|
- **Replies are attributed:** a driver only counts what the backend sent *as that account* (its token, number or path), so a reply leaking from another account fails.
|
|
- **Provider limits are real:** the fake rejects payloads past each provider's published limits (`services/fake-externals/limits.ts`), the way the provider would.
|
|
- **Faults:** `new FakeExternals(request).fail({ provider, action, contains, status, body, times })` makes matching calls fail; scope it with `contains` (e.g. the customer id) so parallel tests are untouched.
|
|
- **MCP:** `services/fake-mcp` is a real MCP server (streamable HTTP) with a `lookup_order` tool.
|
|
|
|
## Fixtures worth knowing
|
|
|
|
- `customerPage` / `widget`: a visitor's browser, never signed in to the dashboard.
|
|
- `makeAgent(settings, groupIds)`, `makeMember({ role, groupIds, seeAllAiChats })`: more agents, and teammates signed in on their own request context.
|
|
- `handoverTeam` / `inbox`: each handover test gets its own teammate and group.
|
|
- `diagnostics` (automatic): a failed test gets the backend log, model requests and third-party calls attached.
|
|
- `expectUnchanged(what, read)`: the "and nothing else happens" check (the AI stays quiet, a forged webhook gets no reply).
|
|
|
|
## More than the default run
|
|
|
|
| What | How | Where it runs |
|
|
|---|---|---|
|
|
| Production-like backend | `E2E_REDIS_URL=redis://127.0.0.1:6379/5 E2E_BACKEND_WORKERS=2 ./scripts/stack.sh up`; enables `@redis` tests | nightly |
|
|
| Widget on Firefox, WebKit, phones | `E2E_MORE_BROWSERS=1 npx playwright test --project=widget-firefox …` | nightly |
|
|
| Visual | `npx playwright test --project=visual`; baselines are Linux-only, made by the nightly job (or a manual run with *update snapshots*) | nightly |
|
|
| Accessibility | `specs/a11y`; new serious findings fail; after fixing some, refresh with `UPDATE_A11Y_BASELINE=1` | every run |
|
|
| Size and speed budgets | `specs/perf` | every run |
|
|
| Migrations | `./scripts/upgrade-check.sh [ref]`: the last release's schema upgraded to HEAD must equal a fresh install | nightly |
|
|
| API fuzzing | `./scripts/api-fuzz.sh` (Schemathesis): every read endpoint, no 500s | nightly |
|
|
|
|
## Adding a test
|
|
|
|
1. Find or create `specs/<feature>/`, and add a row to `FEATURES.md`.
|
|
2. Write the spec in product language, using the fixtures (`agent`, `widget`, `inbox`, `llm`, `loginPage`, …).
|
|
3. If a screen isn't covered yet, add a page object in `src/pages/`. If seeding needs a new endpoint, add it to the module in `src/api/`.
|
|
4. `npx playwright test specs/<feature> --repeat-each=10` must pass before you open a PR.
|
|
5. Run `npm run lint && npm run typecheck`.
|
|
|
|
## Troubleshooting
|
|
|
|
- **Port already in use:** something else is on 18000, 15173, 14010, 14020, 14030, 13000, 11025 or 18025. Run `./scripts/stack.sh down`, or stop the other process.
|
|
- **The launcher never becomes clickable:** the suite runs with `reducedMotion: 'reduce'` because the launcher floats forever otherwise. Keep that option on any new browser context.
|
|
- **The customer site must be a real server:** Chrome blocks a page faked with `page.route` from calling `127.0.0.1` (Local Network Access).
|
|
- **The widget and loader are built from source on every run.** `stack.sh` builds the widget bundle and `chattermate.min.js` from the current `frontend/src/webclient` into `e2e/.cache`, and points the backend at that build with `WIDGET_ASSETS_DIR`. The committed production bundles in `backend/assets` and `frontend/public/webclient` are never rebuilt or modified.
|