Evidence lives under {Skill.PluginRoot()}/assets.
+ +); +``` + +Six `Skill.*` members emit canonical tokens: `Skill.Arguments`, `Skill.PluginData`, +`Skill.PluginRoot`, `Skill.ProjectRoot`, `Skill.SessionIdentity`, and `Skill.SkillRoot`. +`Skill.Resource` renders a Markdown link (`[path](path)`), not a token. Host syntax is +applied during lowering, never in the component. + +A hand-authored `SKILL.md` in the same directory always wins — an authored file beats a +generated one — and the shadowed component reports the informational `AB4735` nudge. A rendered +module that fails to load reports `AB3003`; one that does not default-export a component +function, or does not export a `frontmatter` record, reports `AB3004`. + +`defineSkill` types a Skill definition next to a rendered source: + +```ts twoslash +import { defineSkill } from 'agent-bundle'; + +export const skill = defineSkill({ + description: 'Reviews release evidence and issues an auditable readiness verdict.', + name: 'release-review', +}); +``` + +## Inspecting what shipped + +```sh +npx agent-bundle inspect --root . --skills +``` + +The skill focus shows each discovered Skill, its provenance (`conventional` or `config`), its +resources, and the per-target lowering decisions. In the developer Workbench, the Skills page +renders the emitted document for each host. + +Raw HTML, JSX/MDX, and Mermaid inside Skill Markdown are inert in the Workbench renderer. That +is a deliberate containment boundary, not a rendering gap. + +## Next + +- [Hooks](./hooks.mdx) — react to lifecycle events. +- [MCP servers and Apps](./mcp.mdx) — expose tools, resources, and prompts. diff --git a/website/docs/en/guide/development/_meta.json b/website/docs/en/guide/development/_meta.json new file mode 100644 index 000000000..019782edf --- /dev/null +++ b/website/docs/en/guide/development/_meta.json @@ -0,0 +1 @@ +["index", "workbench", "testing", "evaluations"] diff --git a/website/docs/en/guide/development/evaluations.mdx b/website/docs/en/guide/development/evaluations.mdx new file mode 100644 index 000000000..a2f9d1386 --- /dev/null +++ b/website/docs/en/guide/development/evaluations.mdx @@ -0,0 +1,155 @@ +--- +description: 'Run agent-bundle eval suites: pass/fail/inconclusive semantics, minimum evidence, smoke versus reliability, comparison alignment, and the authenticated native harnesses.' +--- + +# Evaluations + +Tests prove that a route renders and that a generated server honours its wire contract. Evals ask +a different question: given a prompt, does the agent actually reach your plugin, and is the +outcome the one you claimed? That answer is probabilistic, so the eval runner is built to report +honestly rather than to look green. + +## Suites are typed modules + +Eval suites are discovered by convention from `evals/**/*.eval.ts`, and each one default-exports +`defineEvalSuite`: + +```ts twoslash +import { defineEvalSuite, expectOutcome, expectSkillActivation } from 'agent-bundle/eval'; + +export default defineEvalSuite({ + cases: [{ + assertions: [ + expectSkillActivation({ skill: 'service-readiness' }), + expectOutcome({ script: './graders/status-result.ts' }), + ], + fixture: './fixtures/status', + hosts: { portable: { model: 'deterministic' } }, + id: 'status-is-healthy', + invocation: { mode: 'explicit', skill: 'service-readiness' }, + prompt: 'Use service-readiness to verify the checked-in compiler service fixture.', + trials: 1, + }], + name: 'mcp-app-status', +}); +``` + +A case names its `id`, `prompt`, `fixture`, `hosts` (with a pinned model per host), `invocation` +mode — `automatic`, `explicit`, or `none` — its `assertions`, and how many `trials` to run. Case +ids must be unique within a suite. Suite discovery, the runs directory, and the optional semantic +grader are configured under the top-level `evals` config block; see +[Configuration](../../reference/configuration.mdx). + +```sh +npx agent-bundle eval --root . --suite mcp-app-status --trials 3 +npx agent-bundle eval compareEvidence lives under {Skill.PluginRoot()}/assets.
+ +); +``` + +六个 `Skill.*` 成员会输出规范 token：`Skill.Arguments`、`Skill.PluginData`、`Skill.PluginRoot`、 +`Skill.ProjectRoot`、`Skill.SessionIdentity` 与 `Skill.SkillRoot`。`Skill.Resource` 渲染的是 +Markdown 链接（`[path](path)`），不是 token。宿主语法在降级阶段应用，绝不在组件里应用。 + +同目录下手写的 `SKILL.md` 总是胜出——手写文件胜过生成文件——被遮蔽的组件会报告信息级的 `AB4735` +提示。渲染模块加载失败报告 `AB3003`；未默认导出组件函数、或未导出 `frontmatter` 记录，则报告 +`AB3004`。 + +`defineSkill` 可以在渲染式源码旁给 Skill 定义加上类型： + +```ts twoslash +import { defineSkill } from 'agent-bundle'; + +export const skill = defineSkill({ + description: 'Reviews release evidence and issues an auditable readiness verdict.', + name: 'release-review', +}); +``` + +## 检查发布了什么 + +```sh +npx agent-bundle inspect --root . --skills +``` + +skill focus 会展示每个被发现的 Skill、它的 provenance（`conventional` 或 `config`）、它的资源，以及 +逐 target 的降级判断。在开发者 Workbench 中，Skills 页面会渲染每个宿主的输出文档。 + +Skill Markdown 中的原始 HTML、JSX/MDX 与 Mermaid 在 Workbench 渲染器中是惰性的。这是刻意的隔离边界， +不是渲染缺失。 + +## 下一步 + +- [钩子](./hooks.mdx) —— 对生命周期事件作出反应。 +- [MCP 服务器与 MCP App](./mcp.mdx) —— 暴露工具、资源与提示词。 diff --git a/website/docs/zh/guide/development/_meta.json b/website/docs/zh/guide/development/_meta.json new file mode 100644 index 000000000..019782edf --- /dev/null +++ b/website/docs/zh/guide/development/_meta.json @@ -0,0 +1 @@ +["index", "workbench", "testing", "evaluations"] diff --git a/website/docs/zh/guide/development/evaluations.mdx b/website/docs/zh/guide/development/evaluations.mdx new file mode 100644 index 000000000..d8e228f15 --- /dev/null +++ b/website/docs/zh/guide/development/evaluations.mdx @@ -0,0 +1,138 @@ +--- +description: '运行 agent-bundle eval 套件：pass/fail/inconclusive 语义、最低证据、smoke 与可靠性之别、对比对齐，以及需要认证的原生 harness。' +--- + +# 评测 + +测试证明路由能渲染、生成式服务器遵守线上契约。Eval 追问的是另一个问题：给定一段提示词，智能体真的 +抵达了你的插件吗？其结果是否就是你所声称的那个？这个答案是概率性的，因此 eval 运行器的设计目标是诚实 +报告，而不是看起来一片绿。 + +## 套件是带类型的模块 + +Eval 套件按约定从 `evals/**/*.eval.ts` 中发现，每个套件默认导出 `defineEvalSuite`： + +```ts twoslash +import { defineEvalSuite, expectOutcome, expectSkillActivation } from 'agent-bundle/eval'; + +export default defineEvalSuite({ + cases: [{ + assertions: [ + expectSkillActivation({ skill: 'service-readiness' }), + expectOutcome({ script: './graders/status-result.ts' }), + ], + fixture: './fixtures/status', + hosts: { portable: { model: 'deterministic' } }, + id: 'status-is-healthy', + invocation: { mode: 'explicit', skill: 'service-readiness' }, + prompt: 'Use service-readiness to verify the checked-in compiler service fixture.', + trials: 1, + }], + name: 'mcp-app-status', +}); +``` + +一个 case 声明它的 `id`、`prompt`、`fixture`、`hosts`（逐宿主固定模型）、`invocation` 模式—— +`automatic`、`explicit` 或 `none`——它的 `assertions`，以及要运行多少次 `trials`。同一套件内的 case id +必须唯一。套件发现、运行记录目录与可选的语义 grader 都配置在顶层 `evals` 配置块中；见 +[配置](../../reference/configuration.mdx)。 + +```sh +npx agent-bundle eval --root . --suite mcp-app-status --trials 3 +npx agent-bundle eval compare