Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions config.yaml.full
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ tool:
observability:
# [optional] for exporting tracing data to Volcengine CozeLoop and APMPlus platform
opentelemetry:
trace_content: true # true | false, collect agent/LLM/tool input and output content in traces
apmplus:
endpoint: http://apmplus-cn-beijing.volces.com:4317
api_key:
Expand Down
1 change: 1 addition & 0 deletions docs/content/docs/framework/configuration.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ Unified prefix: `OBSERVABILITY_`

| Subcategory | Variable | Description |
| :- | :- | :- |
| OpenTelemetry | `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT` | Whether to collect Agent, LLM, and tool input/output content, default `true`; set to `false` to keep non-content trace data such as spans, latency, model metadata, token usage, and tool names |
| APMPlus | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT` | APMPlus reporting URL |
| | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY` | APMPlus auth key |
| | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME` | APMPlus service name |
Expand Down
1 change: 1 addition & 0 deletions docs/content/docs/framework/configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ volcengine:

| 子类 | 环境变量名称 | 释义 |
| :- | :- | :- |
| OpenTelemetry | `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT` | 是否采集 Agent、LLM 与工具调用的输入输出内容,默认 `true`;设为 `false` 后仍保留 Span、耗时、模型、Token、工具名等非内容 Trace 信息 |
| APMPlus | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT` | APMPlus 上报地址 |
| | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY` | APMPlus 鉴权密钥 |
| | `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME` | APMPlus 服务名称 |
Expand Down
3 changes: 2 additions & 1 deletion docs/content/docs/framework/observability/overview.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ VeADK's observability follows the [OpenTelemetry](https://opentelemetry.io/docs/
- **Visualized traces**: The complete flow of each request, from intake to response, can be traced;
- **Structured data**: Records context, state, event types, latency, and more;
- **Cross-component tracing**: Covers the agent, tools, memory modules, knowledge bases, and external interfaces;
- **Multi-agent coordination**: Analyzes the invocation relationships and data flow between agents in multi-agent scenarios.
- **Multi-agent coordination**: Analyzes the invocation relationships and data flow between agents in multi-agent scenarios;
- **Content collection control**: Disable Agent, LLM, and tool input/output content collection with `observability.opentelemetry.trace_content` while keeping non-content trace data.

## Supported Platforms

Expand Down
3 changes: 2 additions & 1 deletion docs/content/docs/framework/observability/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ VeADK 的可观测能力遵循 [OpenTelemetry](https://opentelemetry.io/docs/spe
- **可视化链路**:每次请求从接入到响应的完整流程都可被追踪;
- **结构化数据**:记录上下文、状态、事件类型、耗时等信息;
- **跨组件追踪**:覆盖 Agent、工具、记忆模块、知识库以及外部接口;
- **多 Agent 协调**:在多 Agent 协作场景中分析各 Agent 的调用关系与数据流向。
- **多 Agent 协调**:在多 Agent 协作场景中分析各 Agent 的调用关系与数据流向;
- **内容采集控制**:可通过 `observability.opentelemetry.trace_content` 关闭 Agent、LLM 与工具输入输出内容采集,仅保留非内容 Trace 信息。

## 支持的平台

Expand Down
24 changes: 14 additions & 10 deletions docs/content/docs/framework/observability/span-attributes.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ VeADK's span attribute naming and value conventions follow the [OpenTelemetry](h
Because the OpenTelemetry community's field conventions for generative AI are still evolving, the meaning of some fields may change.
</Callout>

<Callout type="info" title="Input and Output Content Collection">
`observability.opentelemetry.trace_content` or the `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT` environment variable controls Agent, LLM, and tool input/output content collection. When set to `false`, fields marked as controlled by content collection in the tables below are not written to spans; non-content data such as trace structure, model parameters, token usage, and tool names is still retained.
</Callout>

### Common

| No. | Attribute Name | Meaning | Notes |
Expand Down Expand Up @@ -65,15 +69,15 @@ VeADK's span attribute naming and value conventions follow the [OpenTelemetry](h
| 10 | `gen_ai.is_streaming` | Whether the response is streaming | Returns `None` |
| 11 | `gen_ai.operation.name` | The operation name | Always returns `chat`, used to uniformly identify the operation type |
| 12 | `gen_ai.span.kind` | The span kind | Always returns `llm`, conforming to OpenTelemetry semantic conventions |
| 13 | `gen_ai.prompt` | Structured information of the request input content | Records role, content, function calls, images, and other inputs in message order |
| 14 | `gen_ai.completion` | Structured information of the model response content | Records the text, function calls, and other output generated by the model |
| 13 | `gen_ai.prompt` | Structured information of the request input content | Records role, content, function calls, images, and other inputs in message order; controlled by content collection |
| 14 | `gen_ai.completion` | Structured information of the model response content | Records the text, function calls, and other output generated by the model; controlled by content collection |
| 15 | `gen_ai.usage.input_tokens` | The number of input tokens | Extracted from `params.llm_response.usage_metadata.prompt_token_count` |
| 16 | `gen_ai.usage.output_tokens` | The number of output tokens | Extracted from `params.llm_response.usage_metadata.candidates_token_count` |
| 17 | `gen_ai.usage.total_tokens` | The total number of tokens | Extracted from `params.llm_response.usage_metadata.total_token_count` |
| 18 | `gen_ai.usage.cache_creation_input_tokens` | The number of tokens used to create the cache | Extracted from `params.llm_response.usage_metadata.cached_content_token_count` |
| 19 | `gen_ai.usage.cache_read_input_tokens` | The number of tokens used to read the cache | Extracted from `params.llm_response.usage_metadata.cached_content_token_count` |
| 20 | `gen_ai.messages` | The complete conversation message events | Includes the structured event sequence of system instructions, user messages, tool responses, and assistant replies |
| 21 | `gen_ai.choice` | The model choice event | Represents the candidate response generated by the model (including function calls or text content) |
| 20 | `gen_ai.messages` | The complete conversation message events | Includes the structured event sequence of system instructions, user messages, tool responses, and assistant replies; controlled by content collection |
| 21 | `gen_ai.choice` | The model choice event | Represents the candidate response generated by the model (including function calls or text content); controlled by content collection |
| 22 | `input.value` | The complete LLM request body | *(for debugging)* The serialized output request object |
| 23 | `output.value` | The complete LLM response body | *(for debugging)* The serialized output response object |

Expand All @@ -83,10 +87,10 @@ VeADK's span attribute naming and value conventions follow the [OpenTelemetry](h
| - | - | - | - |
| 1 | `gen_ai.operation.name` | The operation name | Always returns `execute_tool`, uniformly identifying tool-call operations |
| 2 | `gen_ai.tool.name` | The tool name | Obtained from `params.tool.name`; if absent, it is `<unknown_tool_name>`; used for the TLS platform |
| 3 | `gen_ai.tool.input` | The tool input content | JSON serialization includes `name`, `description`, `parameters`, used to record tool-call arguments; used for the TLS platform |
| 4 | `gen_ai.tool.output` | The tool output content | JSON serialization includes `id`, `name`, `response`, recording tool execution results; used for the TLS platform |
| 5 | `cozeloop.input` | The tool input | Same as `gen_ai.tool.input`; used for the CozeLoop platform |
| 6 | `cozeloop.output` | The tool output | Same as `gen_ai.tool.output`; used for the CozeLoop platform |
| 3 | `gen_ai.tool.input` | The tool input content | JSON serialization includes `name`, `description`, `parameters`, used to record tool-call arguments; used for the TLS platform; controlled by content collection |
| 4 | `gen_ai.tool.output` | The tool output content | JSON serialization includes `id`, `name`, `response`, recording tool execution results; used for the TLS platform; controlled by content collection |
| 5 | `cozeloop.input` | The tool input | Same as `gen_ai.tool.input`; used for the CozeLoop platform; controlled by content collection |
| 6 | `cozeloop.output` | The tool output | Same as `gen_ai.tool.output`; used for the CozeLoop platform; controlled by content collection |
| 7 | `gen_ai.span.kind` | The span kind | Always returns `tool`, following OpenTelemetry semantic conventions; used for the APMPlus platform |
| 8 | `gen_ai.input` | The tool input | Same as `gen_ai.tool.input`; used for the APMPlus platform |
| 9 | `gen_ai.output` | The tool output | Same as `gen_ai.tool.output`; used for the APMPlus platform |
| 8 | `gen_ai.input` | The tool input | Same as `gen_ai.tool.input`; used for the APMPlus platform; controlled by content collection |
| 9 | `gen_ai.output` | The tool output | Same as `gen_ai.tool.output`; used for the APMPlus platform; controlled by content collection |
24 changes: 14 additions & 10 deletions docs/content/docs/framework/observability/span-attributes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ VeADK 的 Span 属性命名和值规范遵循 [OpenTelemetry](https://openteleme
由于 OpenTelemetry 社区对生成式 AI 的字段规范还在发展完善中,因此部分字段含义可能会发生变化。
</Callout>

<Callout type="info" title="输入输出内容采集开关">
`observability.opentelemetry.trace_content` 或环境变量 `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT` 控制 Agent、LLM 与 Tool 输入输出内容采集。设为 `false` 后,下表中标记“受内容采集开关控制”的字段不会写入 Span;普通链路、模型参数、Token 用量、工具名等非内容信息仍会保留。
</Callout>

### 通用类

| 序号 | 埋点字段名 | 含义 | 注释 |
Expand Down Expand Up @@ -65,15 +69,15 @@ VeADK 的 Span 属性命名和值规范遵循 [OpenTelemetry](https://openteleme
| 10 | `gen_ai.is_streaming` | 是否为流式响应 | 返回 `None` |
| 11 | `gen_ai.operation.name` | 操作名称 | 固定返回 `chat`,用于统一标识操作类型 |
| 12 | `gen_ai.span.kind` | Span 类型 | 固定返回 `llm`,符合 OpenTelemetry 语义约定 |
| 13 | `gen_ai.prompt` | 请求输入内容结构化信息 | 按消息顺序记录角色、内容、函数调用、图片等输入 |
| 14 | `gen_ai.completion` | 模型响应内容结构化信息 | 记录模型生成的文本、函数调用等输出内容 |
| 13 | `gen_ai.prompt` | 请求输入内容结构化信息 | 按消息顺序记录角色、内容、函数调用、图片等输入;受内容采集开关控制 |
| 14 | `gen_ai.completion` | 模型响应内容结构化信息 | 记录模型生成的文本、函数调用等输出内容;受内容采集开关控制 |
| 15 | `gen_ai.usage.input_tokens` | 输入 token 数量 | 从 `params.llm_response.usage_metadata.prompt_token_count` 提取 |
| 16 | `gen_ai.usage.output_tokens` | 输出 token 数量 | 从 `params.llm_response.usage_metadata.candidates_token_count` 提取 |
| 17 | `gen_ai.usage.total_tokens` | 总 token 数量 | 从 `params.llm_response.usage_metadata.total_token_count` 提取 |
| 18 | `gen_ai.usage.cache_creation_input_tokens` | 缓存创建所用 token 数量 | 从 `params.llm_response.usage_metadata.cached_content_token_count` 提取 |
| 19 | `gen_ai.usage.cache_read_input_tokens` | 缓存读取所用 token 数量 | 从 `params.llm_response.usage_metadata.cached_content_token_count` 提取 |
| 20 | `gen_ai.messages` | 完整对话消息事件 | 包括系统指令、用户消息、工具响应和助手回复的结构化事件序列 |
| 21 | `gen_ai.choice` | 模型选择事件 | 表示模型生成的候选响应(含函数调用或文本内容) |
| 20 | `gen_ai.messages` | 完整对话消息事件 | 包括系统指令、用户消息、工具响应和助手回复的结构化事件序列;受内容采集开关控制 |
| 21 | `gen_ai.choice` | 模型选择事件 | 表示模型生成的候选响应(含函数调用或文本内容);受内容采集开关控制 |
| 22 | `input.value` | 完整 LLM 请求体 | *(供调试使用)* 序列化输出请求对象 |
| 23 | `output.value` | 完整 LLM 响应体 | *(供调试使用)* 序列化输出响应对象 |

Expand All @@ -83,10 +87,10 @@ VeADK 的 Span 属性命名和值规范遵循 [OpenTelemetry](https://openteleme
| - | - | - | - |
| 1 | `gen_ai.operation.name` | 操作名称 | 固定返回 `execute_tool`,统一标识工具调用操作 |
| 2 | `gen_ai.tool.name` | 工具名称 | 从 `params.tool.name` 获取;若无则为 `<unknown_tool_name>`,用于 TLS 平台 |
| 3 | `gen_ai.tool.input` | 工具输入内容 | JSON 序列化包含:`name`、`description`、`parameters`,用于记录工具调用参数,用于 TLS 平台 |
| 4 | `gen_ai.tool.output` | 工具输出内容 | JSON 序列化包含:`id`、`name`、`response`,记录工具执行结果,用于 TLS 平台 |
| 5 | `cozeloop.input` | 工具输入 | 同 `gen_ai.tool.input`,用于 CozeLoop 平台 |
| 6 | `cozeloop.output` | 工具输出 | 同 `gen_ai.tool.output`,用于 CozeLoop 平台 |
| 3 | `gen_ai.tool.input` | 工具输入内容 | JSON 序列化包含:`name`、`description`、`parameters`,用于记录工具调用参数,用于 TLS 平台;受内容采集开关控制 |
| 4 | `gen_ai.tool.output` | 工具输出内容 | JSON 序列化包含:`id`、`name`、`response`,记录工具执行结果,用于 TLS 平台;受内容采集开关控制 |
| 5 | `cozeloop.input` | 工具输入 | 同 `gen_ai.tool.input`,用于 CozeLoop 平台;受内容采集开关控制 |
| 6 | `cozeloop.output` | 工具输出 | 同 `gen_ai.tool.output`,用于 CozeLoop 平台;受内容采集开关控制 |
| 7 | `gen_ai.span.kind` | Span 类型 | 固定返回 `tool`,遵循 OpenTelemetry 语义约定,用于 APMPlus 平台 |
| 8 | `gen_ai.input` | 工具输入 | 同 `gen_ai.tool.input`,用于 APMPlus 平台 |
| 9 | `gen_ai.output` | 工具输出 | 同 `gen_ai.tool.output`,用于 APMPlus 平台 |
| 8 | `gen_ai.input` | 工具输入 | 同 `gen_ai.tool.input`,用于 APMPlus 平台;受内容采集开关控制 |
| 9 | `gen_ai.output` | 工具输出 | 同 `gen_ai.tool.output`,用于 APMPlus 平台;受内容采集开关控制 |
2 changes: 2 additions & 0 deletions docs/content/docs/framework/observability/tracing.en.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,14 @@ To enable full-chain observability, attach an `OpentelemetryTracer` with one or
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY`: The API key of the APM service
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT`: The endpoint of the APM service, e.g. `http://apmplus-cn-beijing.volces.com:4317`
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME`: The service name for APM, e.g. `python_coder_agent`
- `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT`: Whether to collect Agent, LLM, and tool input/output content, default `true`

Or define them in `config.yaml`:

```yaml title="config.yaml"
observability:
opentelemetry:
trace_content: true
apmplus:
endpoint: ...
api_key: ...
Expand Down
4 changes: 2 additions & 2 deletions docs/content/docs/framework/observability/tracing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,13 @@ Tracing(链路追踪)对智能体执行过程进行**全链路记录**,是

- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_API_KEY`:APM 服务的 API Key
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_ENDPOINT`:APM 服务的 Endpoint,例如 `http://apmplus-cn-beijing.volces.com:4317`
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME`:APM 的 Service Name,例如 `python_coder_agent`
- `OBSERVABILITY_OPENTELEMETRY_APMPLUS_SERVICE_NAME`:APM 的 Service Name,例如 `python_coder_agent - `OBSERVABILITY_OPENTELEMETRY_TRACE_CONTENT`:是否采集 Agent、LLM 与工具输入输出内容,默认 `true`

或在 `config.yaml` 中定义:

```yaml title="config.yaml"
observability:
opentelemetry:
opentelemetry trace_content: true
apmplus:
endpoint: ...
api_key: ...
Expand Down
Loading
Loading