docs: 交付文档、维护指引与端到端回归
补齐 README、开发计划、维护交接与 E2E 用例,执行最终验收。
This commit is contained in:
1 parent
5be5ca3376
commit
308332eb94
216 files changed
+37038
No files matched your search
@@ -0,0 +1,60 @@
|
||||
import { test, expect, connectJira, disconnectJira, fakeJira, routeTicketsToJira, FakeExternals, IntegrationsPage, unique, AI_REPLY } from '@e2e'
|
||||
|
||||
// One Jira site per org, so this runs in the serial `org` project and disconnects afterwards.
|
||||
|
||||
interface JiraGrant { access_token: string }
|
||||
interface JiraSite { id: string; url: string }
|
||||
|
||||
test.describe('Jira', () => {
|
||||
test.afterEach(async ({ adminApi }) => { await disconnectJira(adminApi) })
|
||||
|
||||
test('an admin connects Jira from Settings and the card links to their site @integrations', async ({ page, request }) => {
|
||||
const since = new Date()
|
||||
const integrations = new IntegrationsPage(page)
|
||||
await integrations.goto()
|
||||
await integrations.connect('Jira')
|
||||
await expect(page.getByText('Jira connected successfully!')).toBeVisible()
|
||||
|
||||
// The code Atlassian's consent screen sent back was traded with the app's own
|
||||
// credentials, and the site is the one that token reaches.
|
||||
const externals = new FakeExternals(request)
|
||||
const [grant] = (await externals.callsSince(since)).filter((c) => c.provider === 'jira-auth' && c.action === 'token')
|
||||
const [resources] = (await externals.callsSince(since)).filter((c) => c.action === 'accessible-resources')
|
||||
expect(grant.body).toMatchObject({ grant_type: 'authorization_code', client_id: 'e2e-jira-client-id', redirect_uri: 'http://127.0.0.1:18000/api/v1/jira/callback' })
|
||||
expect(resources.headers.authorization).toBe(`Bearer ${(grant.response as JiraGrant).access_token}`)
|
||||
const [site] = resources.response as JiraSite[]
|
||||
await expect(integrations.card('Jira').getByRole('link', { name: /Visit Jira Site/ })).toHaveAttribute('href', site.url)
|
||||
})
|
||||
|
||||
test('the AI files a Jira issue from a chat, in the project the agent is set up for @integrations', async ({ agent, widget, llm, adminApi, request }) => {
|
||||
const code = unique('jira-code')
|
||||
await connectJira(adminApi, code)
|
||||
await routeTicketsToJira(adminApi, agent.id)
|
||||
|
||||
const summary = unique('Refund not received')
|
||||
await llm.script(agent, (s) => s
|
||||
.when(/refund/i).callTool('create_jira_ticket', { summary, description: 'Charged twice for one order; wants one payment back.', priority: 'High' })
|
||||
.thenReply('I have raised this with our billing team.')
|
||||
.otherwise().reply('How can I help?'))
|
||||
await widget.embed()
|
||||
await widget.open()
|
||||
await widget.send('My refund never arrived')
|
||||
await expect(widget.lastAgentMessage()).toHaveText('I have raised this with our billing team.', AI_REPLY)
|
||||
|
||||
// Jira received the issue in the agent's project and type, at the priority the
|
||||
// model chose, sent with this org's token.
|
||||
const externals = new FakeExternals(request)
|
||||
const [grant] = await externals.calls('jira-auth', 'token', { contains: code })
|
||||
const [created] = await externals.calls('jira', 'issue', { contains: summary })
|
||||
expect(created.method).toBe('POST')
|
||||
expect(created.headers.authorization).toBe(`Bearer ${(grant.response as JiraGrant).access_token}`)
|
||||
expect(created.body).toMatchObject({
|
||||
fields: { summary, project: { key: fakeJira.projectKey }, issuetype: { id: fakeJira.taskTypeId }, priority: { id: '2' } },
|
||||
})
|
||||
|
||||
// The model is told the issue key and its link on the connected site.
|
||||
const { key } = created.response as { key: string }
|
||||
const toolResults = (await llm.requests(agent)).flatMap((r) => r.toolResults).join('\n')
|
||||
expect(toolResults).toContain(`.atlassian.net/browse/${key}`)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,62 @@
|
||||
import { test, expect, installShopify, uninstallShopify, useShopForAgent, FakeExternals, seedOrg, unique, AI_REPLY } from '@e2e'
|
||||
|
||||
// Linking a shop sets the org's domain, so this runs in the serial `org` project and
|
||||
// uninstalls (restoring the domain) afterwards.
|
||||
|
||||
test.describe('Shopify', () => {
|
||||
let shop: string
|
||||
let shopId: string | undefined
|
||||
|
||||
test.beforeEach(async ({ agent, adminApi }) => {
|
||||
shop = `${unique('e2e-shop')}.myshopify.com`
|
||||
shopId = await installShopify(adminApi, shop)
|
||||
await useShopForAgent(adminApi, agent.id, shopId)
|
||||
})
|
||||
|
||||
test.afterEach(async ({ adminApi, org }) => {
|
||||
if (shopId) await uninstallShopify(adminApi, shopId, org.orgId, seedOrg.domain)
|
||||
})
|
||||
|
||||
test('a customer asks where their order is and the AI answers from the shop @integrations', async ({ agent, widget, llm, request }) => {
|
||||
const email = widget.customer.email
|
||||
await llm.script(agent, (s) => s
|
||||
.when(/1042/).callTool('get_order_status', { order_number: '#1042', customer_email: email })
|
||||
.thenReply('Order #1042 has shipped with UPS.')
|
||||
.otherwise().reply('Happy to check — what is your order number and email?'))
|
||||
await widget.embed()
|
||||
await widget.open()
|
||||
await widget.send('Where is my order?')
|
||||
await expect(widget.lastAgentMessage()).toHaveText('Happy to check — what is your order number and email?', AI_REPLY)
|
||||
await widget.send(`It is 1042, ${email}`)
|
||||
await expect(widget.lastAgentMessage()).toHaveText('Order #1042 has shipped with UPS.', AI_REPLY)
|
||||
|
||||
// The shop was searched for this customer's order with the token its install
|
||||
// was granted, then the order itself was read.
|
||||
const externals = new FakeExternals(request)
|
||||
const [install] = await externals.calls('shopify', 'access_token', { contains: shop })
|
||||
const [search, read] = await externals.calls('shopify', 'graphql.json', { contains: shop })
|
||||
expect(search.headers['x-shopify-access-token']).toBe((install.response as { access_token: string }).access_token)
|
||||
expect(search.body).toMatchObject({ variables: { query: `email:${email.toLowerCase()} AND name:1042` } })
|
||||
expect(read.body).toMatchObject({ query: expect.stringContaining('query GetOrder') })
|
||||
|
||||
// What the model was told: the shipment and its tracking number.
|
||||
const toolResults = (await llm.requests(agent)).flatMap((r) => r.toolResults).join('\n')
|
||||
expect(toolResults).toContain('FULFILLED')
|
||||
expect(toolResults).toMatch(/1ZE2E\d+/)
|
||||
})
|
||||
|
||||
test('an order lookup without the customer\'s email never reaches the shop @integrations', async ({ agent, widget, llm, request }) => {
|
||||
await llm.script(agent, (s) => s
|
||||
.otherwise().callTool('get_order_status', { order_number: '1042', customer_email: '' })
|
||||
.thenReply('Could you tell me the email you ordered with?'))
|
||||
await widget.embed()
|
||||
await widget.open()
|
||||
await widget.send('Where is order 1042?')
|
||||
await expect(widget.lastAgentMessage()).toHaveText('Could you tell me the email you ordered with?', AI_REPLY)
|
||||
|
||||
const externals = new FakeExternals(request)
|
||||
expect(await externals.calls('shopify', 'graphql.json', { contains: shop })).toHaveLength(0)
|
||||
const toolResults = (await llm.requests(agent)).flatMap((r) => r.toolResults).join('\n')
|
||||
expect(toolResults).toContain('requires_user_input')
|
||||
})
|
||||
})
|
||||
Reference in new issue
Block a user