diff --git a/CHANGELOG.md b/CHANGELOG.md index b6a818f4..69aea81e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,26 @@ This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **Catalog sync with upstream OpenAPI**: added `Weather Station` (deviceType `WeatherStation`, sensor with `atmosphericPressure` field), `Lock Vision` and `Lock Vision Pro` (video smart locks with the same lock/unlock/deadbolt safety semantics as Smart Lock Pro), the `Smart Lock Pro Wifi` Matter alias on the existing Smart Lock entry, and `uploadImage ` on AI Art Frame. The upload command parameter is documented as a single https URL pending upstream parameter-shape clarification. +- **LLM condition USD and token budgets**: per-rule and global `llm_budget` now accept `max_tokens_per_hour` (hourly window, aligned with `max_calls_per_hour`) and `max_cost_per_day_usd` (24h window). Costs are computed from per-model USD pricing in `src/llm/pricing.ts` and reported on `DecideResult.usage`. Audit entries for `llm-condition` now carry `llmUsage`, and `llm-budget-exceeded` records `budgetDimension` (`calls | tokens | cost`), `budgetLimit`, and `budgetObserved`. New lints: `condition-llm-tokens-budget-zero` (warns when token cap is 0) and `condition-llm-cost-without-known-model` (warns when a USD cap is set with `provider: auto` since the cost dimension silently skips models not in the pricing table). +- **Cross-event aggregation in conditions**: new `event_count` condition counts how many events fired for a given device inside a rolling time window. Schema: + + ```yaml + conditions: + - event_count: + device: front-door # deviceId or alias + event: motion.detected # optional canonical event filter + window: "5m" # duration: 100ms | 30s | 5m | 1h | 1d + min: 3 # required floor + max: 10 # optional ceiling + ``` + + Backed by the same per-device JSONL ring at `~/.switchbot/device-history/.jsonl` used by `events history`, with rotation honored. The LLM-condition `recent_events` hook (declared in v0.2 schema since Track ฮบ but unwired) now also pulls from this fetcher, populating `context.recent_events` with up to N most-recent matching events on the trigger device. Engine-level `LlmConditionEvaluator` is now wired into `RulesEngine` (previously only `simulate` had it). New lints: `condition-event-count-bad-window` and `condition-event-count-max-below-min`. +- **Local / non-tool-use LLM provider**: new `provider: local` for `llm` conditions points at any OpenAI-compatible chat completions endpoint (Ollama, llama.cpp server, vLLM, LM Studio). Defaults to `http://localhost:11434/v1`; override with `SWITCHBOT_LOCAL_LLM_URL`, `SWITCHBOT_LOCAL_LLM_MODEL`. Because most local servers don't support OpenAI-style tool use, `decide()` falls back to a structured-output prompt that asks for a `{"pass": bool, "reason": str}` JSON object and runs one repair retry if the first response is not parseable. Operators on tool-use-capable local endpoints can opt in via YAML `tool_use: true` or `SWITCHBOT_LOCAL_LLM_TOOL_USE=1`. New `LLMProvider.capabilities.toolUse` flag exposes this to the rest of the system. New `doctor` check `local-llm-reachable` probes the configured endpoint when (and only when) policy uses `provider: local`. +- **Daemon JSON-RPC IPC transport**: `switchbot rules run` now exposes a JSON-RPC 2.0 endpoint over a Unix domain socket on POSIX (`~/.switchbot/daemon.sock`, mode 0600) and a per-user named pipe on Windows (`\\.\pipe\switchbot-daemon-`). v1 methods: `daemon.status`, `daemon.ping`, `daemon.reload`. New client at `src/daemon/client.ts` exposes `IpcDaemonClient.call()` and `.ping()`. Wire protocol: newline-delimited JSON-RPC. Sets the foundation for future `mcp serve --via-daemon` proxying so MCP clients can avoid per-call CLI cold-start. New `doctor` check `daemon-ipc` reports IPC reachability and round-trip latency when the daemon is running. + ### Fixed - **Daemon start failed in bundled builds** (BUG-001): CLI entry path resolution navigated above the dist/ directory when running from the single-file bundle. Now correctly detects the bundled scenario. diff --git a/README.md b/README.md index 713886a5..81821230 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ Under the hood every surface shares the same catalog, cache, and HMAC client โ€” - ๐ŸŽจ **Dual output modes** โ€” colorized tables by default; `--json` passthrough for `jq` and scripting - ๐Ÿ” **Secure credentials** โ€” HMAC-SHA256 signed requests; config file written with `0600`; env-var override for CI - ๐Ÿ” **Dry-run mode** โ€” preview every mutating request before it hits the API -- ๐Ÿงช **Fully tested** โ€” 2391 Vitest tests, mocked axios, zero network in CI +- ๐Ÿงช **Fully tested** โ€” 2465 Vitest tests, mocked axios, zero network in CI - โšก **Shell completion** โ€” Bash / Zsh / Fish / PowerShell ## Requirements @@ -244,7 +244,8 @@ With a policy.yaml (v0.2) you can declare automations that the CLI executes for you. Supported triggers: **MQTT** (device events), **cron** (schedule-driven), and **webhook** (local HTTP POST). Supported conditions: `time_between` (quiet hours), `device_state` -(live API check with per-tick dedup), and `llm` (AI decision โ€” see +(live API check with per-tick dedup), `event_count` (rolling-window +counts over per-device history), and `llm` (AI decision โ€” see below). Every fire is recorded in `~/.switchbot/audit.log`. `rules run` is long-running; use `daemon start` / `daemon reload` for the managed background mode. @@ -272,14 +273,22 @@ then: conditions: - llm: prompt: "Is the temperature above normal comfort range?" - provider: auto # auto | openai | anthropic + provider: auto # auto | openai | anthropic | local cache_ttl: 5m budget: max_calls_per_hour: 20 + max_tokens_per_hour: 100000 # optional rolling 1h token cap + max_cost_per_day_usd: 1.00 # optional rolling 24h USD cap on_error: pass # fail | pass | skip ``` -Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. `rules lint` flags misconfigured LLM conditions. +Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` for the cloud providers. +For `provider: local`, point `SWITCHBOT_LOCAL_LLM_URL` at any +OpenAI-compatible `/v1/chat/completions` endpoint (Ollama, llama.cpp, +vLLM, LM Studio); `SWITCHBOT_LOCAL_LLM_MODEL` picks the model and +`SWITCHBOT_LOCAL_LLM_TOOL_USE=1` opts into native tool-use when the +endpoint supports it (otherwise a structured-output fallback is used). +`rules lint` flags misconfigured LLM conditions. **Decision trace** โ€” set `automation.audit.evaluate_trace: sampled` (or `full`) in `policy.yaml` to record every evaluation decision. @@ -633,7 +642,7 @@ switchbot doctor switchbot doctor --json ``` -Runs local checks (Node version, credentials, profiles, catalog, catalog-schema, catalog-coverage, cache, quota, clock, MQTT, policy, MCP, keychain, path, inventory, audit, daemon, health, notify-connectivity, release-notes) and exits 1 if any check fails. `warn` results exit 0. The MQTT check reports `ok` when REST credentials are configured (auto-provisioned on first use). The `notify-connectivity` check probes webhook URLs declared in `type: notify` actions. Use this to diagnose connectivity or config issues before running automation. +Runs local checks (Node version, credentials, profiles, catalog, catalog-schema, catalog-coverage, cache, quota, clock, MQTT, policy, MCP, keychain, path, inventory, audit, daemon, daemon-ipc, health, notify-connectivity, local-llm-reachable, release-notes) and exits 1 if any check fails. `warn` results exit 0. The MQTT check reports `ok` when REST credentials are configured (auto-provisioned on first use). The `notify-connectivity` check probes webhook URLs declared in `type: notify` actions. `daemon-ipc` round-trips the JSON-RPC socket when the daemon is running (silently skipped otherwise); `local-llm-reachable` only fires when policy uses `provider: local`. Use this to diagnose connectivity or config issues before running automation. `--json` output includes `maturityScore` (0โ€“100) and `maturityLabel` (`production-ready` / `mostly-ready` / `needs-work` / `not-ready`) to give an at-a-glance readiness rating: @@ -807,7 +816,7 @@ npm install npm run dev -- # Run from TypeScript sources via tsx npm run build # Compile to dist/ -npm test # Run the Vitest suite (2391 tests) +npm test # Run the Vitest suite (2465 tests) npm run test:watch # Watch mode npm run test:coverage # Coverage report (v8, HTML + text) ``` diff --git a/docs/design/roadmap.md b/docs/design/roadmap.md index 2c1f48d9..e5d54e94 100644 --- a/docs/design/roadmap.md +++ b/docs/design/roadmap.md @@ -1,6 +1,6 @@ # Roadmap โ€” Phase 1 through Phase 4 -> **Status as of 2026-05-06:** Phase 1 complete, Phase 2 complete, +> **Status as of 2026-05-15:** Phase 1 complete, Phase 2 complete, > Phase 3A complete (keychain + install library + built-in CLI install > command), Phase 3B tracked in the separate companion skill repo, > Phase 4 shipped at v0.2 (rules engine with MQTT + cron + @@ -10,7 +10,13 @@ > and `policy_diff`; v2.15.0 flips `policy new` default schema to v0.2 > and starts the v0.1 deprecation window. > Tracks ฮธ (notify actions) and ฮท (LLM-backed rule suggestion) -> shipped in v3.0. +> shipped in v3.0. Track ฮบ (AI decision loop โ€” `rules trace`, +> `rules trace-explain`, `llm` conditions, `rules simulate`) +> shipped in v3.6.x. Track ฮผ (catalog sync โ€” Weather Station, Lock +> Vision, Lock Vision Pro, Smart Lock Pro Wifi alias, AI Art Frame +> `uploadImage`) and Track ฮป (USD/token budget, cross-event +> aggregation, local LLM providers, JSON-RPC IPC) are queued for +> the next release. > Note: Track ฮณ is a runtime capability increment on the v0.2 rule > model, not a separate policy schema version. @@ -196,13 +202,58 @@ the skill's `manifest.json` `roadmap` block, which points back here. warning on provider failure. `rules_suggest` MCP tool gains a `llm` parameter. All LLM calls are written to the audit log as `kind: llm-suggest` with backend, model, and latency fields. +- **Track ฮบ โ€” AI decision loop *(shipped, v3.6.x)*.** + `rules trace` records every condition evaluation; `rules + trace-explain` renders why a tick fired or was blocked. + `conditions: [- llm: { prompt, provider, ... }]` lets a rule call + an LLM as a condition (per-condition + global call budget, + cache, on_error fail/pass/skip). `rules simulate` replays a rule + against `~/.switchbot/device-history` for offline what-if. + Audit gains `llm-condition`, `llm-cache-hit`, + `llm-budget-exceeded` records. + +## In-flight (next release) + +- **Track ฮผ โ€” catalog sync.** + Adds Weather Station (read-only sensor), Lock Vision and Lock Vision + Pro (video locks with `lock` / `unlock` / `deadbolt` for the Pro), + the `Smart Lock Pro Wifi` alias for Matter-enabled Lock Pro, and + `uploadImage` on AI Art Frame. Pure data + tests; no schema bump. +- **Track ฮป.1 โ€” USD/token budget for `llm` conditions.** + `DecideResult.usage = { tokensIn, tokensOut, costUsd? }`; per-rule + `budget.max_tokens_per_hour` / `max_cost_per_day_usd` and global + `automation.llm_budget.{max_tokens_per_hour, max_cost_per_day_usd}`. + Audit `llm-budget-exceeded` carries `dimension: "calls" | "tokens" | "cost"`. + Pricing table at `src/llm/pricing.ts` (override via the policy + `automation.llm_pricing_overrides` field). +- **Track ฮป.2 โ€” cross-event aggregation.** + Non-LLM `event_count: { device, event?, window, min, max? }` + condition counts firings inside a rolling time window. Same + `EventWindowFetcher` populates the LLM `recent_events` hook so + prompts get the last N events of the trigger device for free. + Backed by `~/.switchbot/device-history/.jsonl`. +- **Track ฮป.3 โ€” local / non-tool-use LLM providers.** + `LLMProvider.capabilities.toolUse` flag gates a structured-output + fallback (JSON instruction + lenient parser + one repair retry) + for endpoints that don't support tool use. New `provider: local` + in policy points at any OpenAI-compatible `/v1/chat/completions` + (Ollama, llama.cpp, vLLM, LM Studio) via + `SWITCHBOT_LOCAL_LLM_URL`. `doctor` adds `local-llm-reachable`. +- **Track ฮป.4 โ€” daemon JSON-RPC 2.0 IPC.** + `rules run` now exposes `daemon.status`, `daemon.ping`, + `daemon.reload` over a Unix domain socket + (`~/.switchbot/daemon.sock`, mode 0600) on POSIX or a per-user + named pipe (`\\.\pipe\switchbot-daemon-`) on Windows. v1 + surface; future `mcp serve --via-daemon` will proxy MCP tool calls + through the same transport. `doctor` adds `daemon-ipc`. ## Next execution queue (ordered) -1. **Daemon mode for repeated agent invocations.** - Add a local long-lived process with Unix socket / named pipe transport. - Exit when: repeated MCP + plan runs no longer pay fresh-process startup, - and `doctor` can verify daemon health. +1. **`mcp serve --via-daemon` proxy.** + Route MCP tool calls through the JSON-RPC IPC so repeated agent + invocations skip the cold-start cost. + Exit when: `mcp serve --via-daemon list_devices` round-trips and + `doctor` confirms daemon health. 2. **Standalone MCP package (`npx @switchbot/mcp-server`).** Split MCP serve entrypoint into a tiny publishable package while preserving tool contract parity with the main CLI. diff --git a/docs/policy-reference.md b/docs/policy-reference.md index 7a1b9ccf..69b52a55 100644 --- a/docs/policy-reference.md +++ b/docs/policy-reference.md @@ -241,27 +241,77 @@ too. |-----------------|---------------------------------------------------------------|--------| | `time_between` | `[HH:MM, HH:MM]` local-time window, `start > end` โ†’ overnight | active | | `device_state` | `{ device, field, op, value }` read device status inline | active | +| `event_count` | Count events in a rolling window over device history | active | | `all` | AND-join multiple sub-conditions | active | | `any` | OR-join multiple sub-conditions | active | | `not` | Negate a sub-condition | active | | `llm` | AI judgement โ€” prompt an LLM before firing (see below) | active | +**`event_count` condition fields:** + +```yaml +conditions: + - event_count: + device: hallway-motion # alias or deviceId (required) + event: motion.detected # MQTT event name; omit to count all + window: "5m" # rolling window: \d+[smh] (required) + min: 3 # fire only if count >= min (required) + max: 10 # optional upper bound (count <= max) +``` + +Reads `~/.switchbot/device-history/.jsonl` (the same ring +buffer `events mqtt-tail` writes). Lints flag a missing history file +(`condition-event-count-no-history` โ€” likely typo) and a `max < min` +inversion (`condition-event-count-max-below-min`). For the LLM-free +path: an "alarm if motion โ‰ฅ3 times in 5m" guard does not need a model. + **LLM condition fields:** ```yaml conditions: - llm: prompt: "Is the temperature above normal comfort range?" - provider: auto # auto | openai | anthropic + provider: auto # auto | openai | anthropic | local timeout_ms: 5000 # 500โ€“10000 (default 5000) cache_ttl: 5m # none | \d+[smh] (default 5m) - recent_events: 5 # 0โ€“20 (default 5) โ€” recent events included in prompt + recent_events: 5 # 0โ€“20 (default 5) โ€” last N events of the + # trigger device included verbatim in prompt budget: - max_calls_per_hour: 10 # per-condition limit (default 10) + max_calls_per_hour: 10 # per-condition limit (default 10) + max_tokens_per_hour: 100000 # optional rolling 1h token cap + max_cost_per_day_usd: 1.00 # optional rolling 24h USD cap on_error: fail # fail | pass | skip (default fail) ``` -Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`. `rules lint` flags misconfigured LLM conditions. Global LLM budget can be set via `automation.llm_budget.max_calls_per_hour` (default 60). +Set `OPENAI_API_KEY` or `ANTHROPIC_API_KEY` for the cloud providers. +For `provider: local`, point `SWITCHBOT_LOCAL_LLM_URL` at any +OpenAI-compatible `/v1/chat/completions` endpoint (Ollama defaults +to `http://localhost:11434/v1`, llama.cpp / vLLM / LM Studio also +work). Models without tool-use call into a structured-output +fallback (JSON instruction prompt + lenient parser + one repair +retry); set `SWITCHBOT_LOCAL_LLM_TOOL_USE=1` if your local model +does support tool use. + +`rules lint` flags misconfigured LLM conditions, including +`condition-llm-tokens-budget-zero` (token cap set to 0 โ€” never +allowed) and `condition-llm-cost-without-known-model` (cost cap on a +model not in the pricing table โ€” won't be enforced). + +The audit log records every LLM condition outcome as +`kind: llm-condition` with `usage: { tokensIn, tokensOut, costUsd? }` +and emits `kind: llm-budget-exceeded` with +`dimension: "calls" | "tokens" | "cost"` when a cap fires. + +Global LLM budget (applied across all LLM conditions, in addition to +each condition's per-rule budget): + +```yaml +automation: + llm_budget: + max_calls_per_hour: 60 # default 60 + max_tokens_per_hour: 1000000 # optional + max_cost_per_day_usd: 10.00 # optional +``` **Destructive verbs are refused upstream.** The v0.2 validator rejects `lock`, `unlock`, `deleteWebhook`, `deleteScene`, diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index f428fc39..ba8e10db 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -22,8 +22,8 @@ import { import { validateLoadedPolicy } from '../policy/validate.js'; import { selectCredentialStore } from '../credentials/keychain.js'; import { getActiveProfile } from '../lib/request-context.js'; -import { readDaemonState } from '../lib/daemon-state.js'; -import { isPidAlive } from '../rules/pid-file.js'; +import { readDaemonState, DAEMON_PID_FILE } from '../lib/daemon-state.js'; +import { isPidAlive, readPidFile } from '../rules/pid-file.js'; interface Check { name: string; @@ -979,6 +979,120 @@ function checkNotifyConnectivity(): Check { }; } +async function checkLocalLlmReachable(): Promise { + // Only run if a policy is configured AND it references provider: local + // (either at the global automation level or in any rule's llm condition). + // Otherwise this check is silently skipped โ€” operators without local LLM + // setup shouldn't see warnings about an endpoint they don't use. + const policyPath = resolvePolicyPath(); + let loaded: { data: unknown }; + try { + loaded = loadPolicyFile(policyPath); + } catch { + return { name: 'local-llm-reachable', status: 'ok', detail: { present: false, message: 'no policy or policy unreadable โ€” check skipped' } }; + } + const policy = loaded.data as { + automation?: { rules?: unknown }; + } | null; + const automation = policy?.automation; + const ruleArr = Array.isArray(automation?.rules) ? (automation.rules as Array>) : []; + const usesLocal = ruleArr.some((rule) => { + const conds = Array.isArray(rule.conditions) ? (rule.conditions as Array>) : []; + return conds.some((c) => { + const llm = c.llm as Record | undefined; + return llm && llm.provider === 'local'; + }); + }); + + if (!usesLocal) { + return { name: 'local-llm-reachable', status: 'ok', detail: { applicable: false, message: 'no policy reference to provider:local โ€” check skipped' } }; + } + + const baseUrl = (process.env.SWITCHBOT_LOCAL_LLM_URL ?? 'http://localhost:11434/v1').replace(/\/v1\/?$/, '').replace(/\/+$/, ''); + const start = Date.now(); + try { + const reachable = await probeLocalLlmEndpoint(baseUrl); + const latencyMs = Date.now() - start; + if (!reachable) { + return { + name: 'local-llm-reachable', + status: 'fail', + detail: { baseUrl, latencyMs, message: 'endpoint did not respond โ€” start your local LLM server (e.g. `ollama serve`)' }, + }; + } + return { name: 'local-llm-reachable', status: 'ok', detail: { baseUrl, latencyMs } }; + } catch (err) { + return { + name: 'local-llm-reachable', + status: 'fail', + detail: { baseUrl, message: err instanceof Error ? err.message : String(err) }, + }; + } +} + +async function probeLocalLlmEndpoint(baseUrl: string): Promise { + const httpMod = await import('node:http'); + const httpsMod = await import('node:https'); + return new Promise((resolve) => { + let url: URL; + try { url = new URL(baseUrl); } catch { resolve(false); return; } + const isHttps = url.protocol === 'https:'; + const lib = isHttps ? httpsMod.default : httpMod.default; + const req = lib.request( + { + hostname: url.hostname, + port: url.port || (isHttps ? 443 : 80), + path: url.pathname || '/', + method: 'GET', + timeout: 3_000, + }, + (res) => { + // Any HTTP response (even 404) means the server is reachable. + res.on('data', () => { /* drain */ }); + res.on('end', () => resolve(true)); + res.resume(); + }, + ); + req.on('error', () => resolve(false)); + req.on('timeout', () => { req.destroy(); resolve(false); }); + req.end(); + }); +} + +async function checkDaemonIpc(): Promise { + // Probes the JSON-RPC IPC socket exposed by `switchbot rules run`. Treats + // the absence of a daemon process as 'ok' (informational): the daemon being + // stopped is a valid configuration. A reachable daemon must answer + // daemon.status within a short timeout for the check to pass. + const daemonPid = readPidFile(DAEMON_PID_FILE); + if (!daemonPid || !isPidAlive(daemonPid)) { + return { name: 'daemon-ipc', status: 'ok', detail: { applicable: false, message: 'daemon not running โ€” check skipped' } }; + } + + try { + const { IpcDaemonClient } = await import('../daemon/client.js'); + const client = new IpcDaemonClient({ timeoutMs: 1_500, connectTimeoutMs: 500 }); + const start = Date.now(); + const result = await client.ping(); + const latencyMs = Date.now() - start; + return { + name: 'daemon-ipc', + status: 'ok', + detail: { + socketPath: client.getSocketPath(), + latencyMs, + ipcStatus: result.status, + }, + }; + } catch (err) { + return { + name: 'daemon-ipc', + status: 'fail', + detail: { message: err instanceof Error ? err.message : String(err) }, + }; + } +} + interface CheckDef { name: string; @@ -1015,6 +1129,8 @@ const CHECK_REGISTRY: CheckDef[] = [ { name: 'daemon', description: 'daemon state file + runtime status', run: () => checkDaemon() }, { name: 'health', description: 'health endpoint availability (daemon --healthz-port)', run: () => checkHealthEndpoint() }, { name: 'notify-connectivity', description: 'webhook URLs from notify actions in policy.yaml', run: () => checkNotifyConnectivity() }, + { name: 'local-llm-reachable', description: 'local LLM endpoint reachable (only when policy uses provider:local)', run: () => checkLocalLlmReachable() }, + { name: 'daemon-ipc', description: 'daemon JSON-RPC IPC socket reachable (only when daemon is running)', run: () => checkDaemonIpc() }, ]; interface FixResult { diff --git a/src/commands/rules.ts b/src/commands/rules.ts index d90a688d..b6622718 100644 --- a/src/commands/rules.ts +++ b/src/commands/rules.ts @@ -273,6 +273,8 @@ function registerRun(rules: Command): void { let stopping = false; const pidPaths = getDefaultPidFilePaths(); writePidFile(pidPaths.pidFile); + const ipcStartedAt = new Date(); + let ipcServerHandle: { socketPath: string; close: () => Promise } | null = null; const cleanup = () => { clearPidFile(pidPaths.pidFile); // Drop any stale reload sentinel too โ€” this process won't see it. @@ -282,6 +284,9 @@ function registerRun(rules: Command): void { if (stopping) return; stopping = true; try { + if (ipcServerHandle) { + try { await ipcServerHandle.close(); } catch { /* best-effort */ } + } await engine.stop(); await client.disconnect(); } finally { @@ -295,7 +300,7 @@ function registerRun(rules: Command): void { await client.connect(); await engine.start(); - const doReload = async (trigger: 'signal' | 'sentinel'): Promise => { + const doReload = async (trigger: 'signal' | 'sentinel' | 'ipc'): Promise => { try { const fresh = loadAutomation(pathArg); if (!fresh) return; @@ -338,6 +343,37 @@ function registerRun(rules: Command): void { }, 2000); reloadPoll.unref(); + // IPC: start a JSON-RPC server on the daemon socket so MCP and other + // long-running clients can avoid per-call CLI cold-start overhead. + // Starts after engine.start() so daemon.status reflects real state. + try { + const { startIpcServer } = await import('../daemon/server.js'); + ipcServerHandle = await startIpcServer({ + handlers: { + 'daemon.status': () => ({ + status: 'running', + pid: process.pid, + startedAt: ipcStartedAt.toISOString(), + rulesActive: engine.getStats().rulesActive, + globalDryRun: opts.dryRun === true, + }), + 'daemon.ping': () => ({ ok: true, t: new Date().toISOString() }), + 'daemon.reload': async () => { + await doReload('ipc' as const); + return { ok: true, rulesActive: engine.getStats().rulesActive }; + }, + }, + onClientError: () => { /* silenced โ€” clients dropping mid-call is normal */ }, + }); + if (!isJsonMode()) { + console.error(`IPC: listening on ${ipcServerHandle.socketPath}`); + } + } catch (err) { + if (!isJsonMode()) { + console.error(`IPC: failed to start (${err instanceof Error ? err.message : String(err)}) โ€” daemon will run without IPC`); + } + } + if (!isJsonMode()) { console.error( `Rules engine started โ€” ${engine.getStats().rulesActive} active rule(s), ${opts.dryRun ? 'global dry-run' : 'live'}.`, diff --git a/src/daemon/client.ts b/src/daemon/client.ts new file mode 100644 index 00000000..c2b14580 --- /dev/null +++ b/src/daemon/client.ts @@ -0,0 +1,125 @@ +import net from 'node:net'; +import { randomUUID } from 'node:crypto'; +import { getDaemonSocketPath } from './socket-path.js'; +import type { JsonRpcResponse } from './server.js'; + +export interface IpcClientOptions { + socketPath?: string; + /** Per-call timeout in milliseconds. Default: 5000ms. */ + timeoutMs?: number; + /** Connection establishment timeout in milliseconds. Default: 2000ms. */ + connectTimeoutMs?: number; +} + +export class IpcDaemonClientError extends Error { + constructor(message: string, public readonly code?: number, public readonly data?: unknown) { + super(message); + this.name = 'IpcDaemonClientError'; + } +} + +/** + * Single-shot JSON-RPC client. Each `call()` opens a socket, sends one + * request, awaits the matching response, and closes. This keeps the client + * stateless โ€” callers don't have to manage a connection lifecycle, and the + * daemon doesn't have to deal with hung connections. + * + * For workloads that issue many calls back-to-back (e.g. `mcp serve + * --via-daemon` proxying tool calls), connection pooling can be added later + * without changing this surface. + */ +export class IpcDaemonClient { + private readonly socketPath: string; + private readonly timeoutMs: number; + private readonly connectTimeoutMs: number; + + constructor(opts: IpcClientOptions = {}) { + this.socketPath = opts.socketPath ?? getDaemonSocketPath(); + this.timeoutMs = opts.timeoutMs ?? 5_000; + this.connectTimeoutMs = opts.connectTimeoutMs ?? 2_000; + } + + getSocketPath(): string { + return this.socketPath; + } + + /** + * Sends a JSON-RPC request and resolves with the `result` field. Throws + * IpcDaemonClientError on transport failure, parse failure, timeout, or + * server-side error. + */ + async call(method: string, params?: unknown): Promise { + const id = randomUUID(); + const request = JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n'; + + return new Promise((resolve, reject) => { + const socket = net.createConnection(this.socketPath); + let buffer = ''; + let settled = false; + + const finish = (err: IpcDaemonClientError | null, value?: T): void => { + if (settled) return; + settled = true; + clearTimeout(callTimer); + clearTimeout(connectTimer); + socket.destroy(); + if (err) reject(err); + else resolve(value as T); + }; + + const callTimer = setTimeout(() => { + finish(new IpcDaemonClientError(`IPC call to ${method} timed out after ${this.timeoutMs}ms`)); + }, this.timeoutMs); + + const connectTimer = setTimeout(() => { + finish(new IpcDaemonClientError(`IPC connect timed out after ${this.connectTimeoutMs}ms`)); + }, this.connectTimeoutMs); + + socket.setEncoding('utf-8'); + socket.on('connect', () => { + clearTimeout(connectTimer); + socket.write(request); + }); + socket.on('data', (chunk: string) => { + buffer += chunk; + const newlineIdx = buffer.indexOf('\n'); + if (newlineIdx === -1) return; + const line = buffer.slice(0, newlineIdx).trim(); + if (!line) return; + try { + const response = JSON.parse(line) as JsonRpcResponse; + if ('error' in response) { + finish(new IpcDaemonClientError( + response.error.message, + response.error.code, + response.error.data, + )); + return; + } + finish(null, response.result as T); + } catch (err) { + finish(new IpcDaemonClientError(`Malformed JSON-RPC response: ${err instanceof Error ? err.message : String(err)}`)); + } + }); + socket.on('error', (err: NodeJS.ErrnoException) => { + const code = err.code === 'ENOENT' || err.code === 'ECONNREFUSED' + ? `IPC daemon not listening at ${this.socketPath} (${err.code})` + : `IPC socket error: ${err.message}`; + finish(new IpcDaemonClientError(code)); + }); + socket.on('end', () => { + if (!settled) finish(new IpcDaemonClientError('IPC server closed connection before responding')); + }); + }); + } + + /** + * Quick reachability probe. Resolves with the latency in milliseconds when + * the daemon responds to `daemon.status`; rejects otherwise. + */ + async ping(): Promise<{ latencyMs: number; status: unknown }> { + const start = Date.now(); + const status = await this.call('daemon.status'); + return { latencyMs: Date.now() - start, status }; + } +} diff --git a/src/daemon/server.ts b/src/daemon/server.ts new file mode 100644 index 00000000..1876acc9 --- /dev/null +++ b/src/daemon/server.ts @@ -0,0 +1,175 @@ +import net from 'node:net'; +import fs from 'node:fs'; +import path from 'node:path'; +import { getDaemonSocketPath } from './socket-path.js'; + +/** + * Minimal JSON-RPC 2.0 implementation tailored for the SwitchBot daemon. + * + * Wire protocol: each line on the socket is exactly one JSON-RPC message + * (newline-delimited). This sidesteps Content-Length framing while still + * being trivial for clients in any language to speak. Requests time out at + * the client; the server has no per-request timer. + * + * Permissions: on POSIX we chmod the socket file to 0600 after binding. + * On Windows, named pipes default to a DACL granting only the creating user + * access, which is what we want. + */ + +export interface JsonRpcRequest { + jsonrpc: '2.0'; + id?: string | number | null; + method: string; + params?: unknown; +} + +export interface JsonRpcSuccess { + jsonrpc: '2.0'; + id: string | number | null; + result: unknown; +} + +export interface JsonRpcError { + jsonrpc: '2.0'; + id: string | number | null; + error: { code: number; message: string; data?: unknown }; +} + +export type JsonRpcResponse = JsonRpcSuccess | JsonRpcError; + +export type RpcHandler = (params: unknown) => Promise | unknown; + +export interface IpcServerOptions { + socketPath?: string; + handlers: Record; + onClientError?: (err: Error) => void; +} + +export interface IpcServer { + socketPath: string; + close: () => Promise; + /** Returns true while the underlying net.Server is listening. */ + isListening: () => boolean; +} + +const ERR_PARSE = -32700; +const ERR_INVALID_REQUEST = -32600; +const ERR_METHOD_NOT_FOUND = -32601; +const ERR_INTERNAL = -32603; + +/** + * Starts a JSON-RPC server on the daemon's IPC endpoint. Returns a handle + * that can be closed to release the socket file (POSIX) or named pipe + * (Windows). On POSIX, a stale socket file from a previous crash is + * removed before binding. + */ +export async function startIpcServer(opts: IpcServerOptions): Promise { + const socketPath = opts.socketPath ?? getDaemonSocketPath(); + if (process.platform !== 'win32') { + await ensureParentDir(socketPath); + await removeStaleSocket(socketPath); + } + + const server = net.createServer((socket) => { + let buffer = ''; + socket.setEncoding('utf-8'); + socket.on('data', (chunk: string) => { + buffer += chunk; + let newlineIdx = buffer.indexOf('\n'); + while (newlineIdx !== -1) { + const line = buffer.slice(0, newlineIdx).trim(); + buffer = buffer.slice(newlineIdx + 1); + newlineIdx = buffer.indexOf('\n'); + if (!line) continue; + void handleLine(line, socket, opts); + } + }); + socket.on('error', (err: Error) => opts.onClientError?.(err)); + }); + + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, () => { + server.removeListener('error', reject); + resolve(); + }); + }); + + if (process.platform !== 'win32') { + try { + fs.chmodSync(socketPath, 0o600); + } catch { /* best-effort */ } + } + + return { + socketPath, + isListening: () => server.listening, + close: () => new Promise((resolve) => { + server.close(() => { + if (process.platform !== 'win32') { + try { fs.unlinkSync(socketPath); } catch { /* best-effort */ } + } + resolve(); + }); + }), + }; +} + +async function handleLine(line: string, socket: net.Socket, opts: IpcServerOptions): Promise { + let req: JsonRpcRequest; + let id: string | number | null = null; + try { + const parsed = JSON.parse(line) as JsonRpcRequest; + req = parsed; + id = parsed.id ?? null; + } catch (err) { + send(socket, errorResponse(null, ERR_PARSE, 'Parse error', String(err))); + return; + } + + if (req.jsonrpc !== '2.0' || typeof req.method !== 'string') { + send(socket, errorResponse(id, ERR_INVALID_REQUEST, 'Invalid Request: missing jsonrpc:"2.0" or method')); + return; + } + + const handler = opts.handlers[req.method]; + if (!handler) { + send(socket, errorResponse(id, ERR_METHOD_NOT_FOUND, `Method not found: ${req.method}`)); + return; + } + + try { + const result = await handler(req.params); + if (req.id !== undefined) { + send(socket, { jsonrpc: '2.0', id, result }); + } + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + send(socket, errorResponse(id, ERR_INTERNAL, message)); + } +} + +function send(socket: net.Socket, msg: JsonRpcResponse): void { + if (!socket.writable) return; + socket.write(JSON.stringify(msg) + '\n'); +} + +function errorResponse(id: string | number | null, code: number, message: string, data?: unknown): JsonRpcError { + return { jsonrpc: '2.0', id, error: data === undefined ? { code, message } : { code, message, data } }; +} + +async function ensureParentDir(socketPath: string): Promise { + const parent = path.dirname(socketPath); + await fs.promises.mkdir(parent, { recursive: true }); +} + +async function removeStaleSocket(socketPath: string): Promise { + try { + await fs.promises.unlink(socketPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') { + // Permission error or directory: not safe to bind, surface it. + throw err; + } + } +} diff --git a/src/daemon/socket-path.ts b/src/daemon/socket-path.ts new file mode 100644 index 00000000..16b30657 --- /dev/null +++ b/src/daemon/socket-path.ts @@ -0,0 +1,61 @@ +import os from 'node:os'; +import fs from 'node:fs'; +import path from 'node:path'; +import { execSync } from 'node:child_process'; + +/** + * Returns the IPC endpoint path for the daemon. POSIX gets a Unix domain + * socket inside `~/.switchbot/`; Windows gets a per-user named pipe whose + * default ACL only grants the creating user access. + * + * The exact form is what `net.createServer(path)` accepts on each platform. + */ +export function getDaemonSocketPath(): string { + if (process.platform === 'win32') { + return `\\\\.\\pipe\\switchbot-daemon-${getCurrentUserKey()}`; + } + return path.join(os.homedir(), '.switchbot', 'daemon.sock'); +} + +/** + * Returns true when the supplied path exists and is in a state where the + * daemon could legitimately bind to it. On POSIX this means the socket file + * exists; on Windows we always return true because named pipe presence is + * not observable through the filesystem. + */ +export function isDaemonSocketAvailable(socketPath: string): boolean { + if (process.platform === 'win32') return true; + try { + return fs.existsSync(socketPath); + } catch { + return false; + } +} + +let cachedUserKey: string | null = null; + +function getCurrentUserKey(): string { + if (cachedUserKey) return cachedUserKey; + const fromEnv = process.env.USERNAME ?? process.env.USER; + if (fromEnv) { + cachedUserKey = sanitize(fromEnv); + return cachedUserKey; + } + try { + const userInfo = os.userInfo(); + cachedUserKey = sanitize(userInfo.username); + return cachedUserKey; + } catch { /* fall through */ } + try { + const out = execSync('whoami', { encoding: 'utf-8', timeout: 2_000 }).trim(); + cachedUserKey = sanitize(out.split(/[\\\/]/).pop() ?? 'unknown'); + return cachedUserKey; + } catch { + cachedUserKey = 'unknown'; + return cachedUserKey; + } +} + +function sanitize(name: string): string { + return name.replace(/[^A-Za-z0-9._-]/g, '_').slice(0, 64) || 'unknown'; +} diff --git a/src/devices/catalog.ts b/src/devices/catalog.ts index 8605c418..84450725 100644 --- a/src/devices/catalog.ts +++ b/src/devices/catalog.ts @@ -240,9 +240,9 @@ export const DEVICE_CATALOG: DeviceCatalogEntry[] = [ { type: 'Smart Lock', category: 'physical', - description: 'Bluetooth/Wi-Fi electronic deadbolt that locks and unlocks a door via cloud API.', + description: 'Bluetooth/Wi-Fi electronic deadbolt that locks and unlocks a door via cloud API. Pro models support the Matter protocol when configured via the SwitchBot app (deviceType "Smart Lock Pro Wifi").', role: 'security', - aliases: ['Lock', 'Smart Lock Pro', 'Lock Pro'], + aliases: ['Lock', 'Smart Lock Pro', 'Lock Pro', 'Smart Lock Pro Wifi'], commands: [ { command: 'lock', parameter: 'โ€”', description: 'Lock the door', idempotent: true }, { command: 'unlock', parameter: 'โ€”', description: 'Unlock the door', idempotent: true, safetyTier: 'destructive', safetyReason: 'Physically unlocks the door โ€” anyone nearby can open it.' }, @@ -275,6 +275,29 @@ export const DEVICE_CATALOG: DeviceCatalogEntry[] = [ ], statusFields: ['battery', 'version', 'lockState', 'doorState', 'calibrate'], }, + { + type: 'Lock Vision', + category: 'physical', + description: 'Video smart lock with built-in camera; supports lock and unlock control plus status reporting.', + role: 'security', + commands: [ + { command: 'lock', parameter: 'โ€”', description: 'Lock the door', idempotent: true }, + { command: 'unlock', parameter: 'โ€”', description: 'Unlock the door', idempotent: true, safetyTier: 'destructive', safetyReason: 'Physically unlocks the door โ€” anyone nearby can open it.' }, + ], + statusFields: ['lockState', 'doorState', 'battery', 'keyList', 'version'], + }, + { + type: 'Lock Vision Pro', + category: 'physical', + description: 'Pro-tier video smart lock with camera; supports lock, unlock, and deadbolt control.', + role: 'security', + commands: [ + { command: 'lock', parameter: 'โ€”', description: 'Lock the door', idempotent: true }, + { command: 'unlock', parameter: 'โ€”', description: 'Unlock the door', idempotent: true, safetyTier: 'destructive', safetyReason: 'Physically unlocks the door โ€” anyone nearby can open it.' }, + { command: 'deadbolt', parameter: 'โ€”', description: 'Engage deadbolt', idempotent: true }, + ], + statusFields: ['lockState', 'doorState', 'battery', 'keyList', 'version'], + }, { type: 'Plug', category: 'physical', @@ -582,11 +605,12 @@ export const DEVICE_CATALOG: DeviceCatalogEntry[] = [ { type: 'AI Art Frame', category: 'physical', - description: 'Digital art frame that can switch to the next or previous image.', + description: 'Digital art frame that can switch images and accept new artwork uploads.', role: 'other', commands: [ { command: 'next', parameter: 'โ€”', description: 'Switch to the next image', idempotent: false }, { command: 'previous', parameter: 'โ€”', description: 'Switch to the previous image', idempotent: false }, + { command: 'uploadImage', parameter: '', description: 'Upload a new image from an https:// URL to display on the frame', idempotent: true, exampleParams: ['https://example.com/art.jpg'] }, ], statusFields: ['version'], }, @@ -612,6 +636,16 @@ export const DEVICE_CATALOG: DeviceCatalogEntry[] = [ commands: [], statusFields: ['temperature', 'humidity', 'battery', 'version'], }, + { + type: 'WeatherStation', + category: 'physical', + description: 'Outdoor weather station reporting temperature, humidity, and atmospheric pressure; read-only.', + role: 'sensor', + readOnly: true, + aliases: ['Weather Station'], + commands: [], + statusFields: ['temperature', 'humidity', 'atmosphericPressure', 'battery', 'version'], + }, { type: 'Motion Sensor', category: 'physical', diff --git a/src/devices/history-window.ts b/src/devices/history-window.ts new file mode 100644 index 00000000..c3b93f7c --- /dev/null +++ b/src/devices/history-window.ts @@ -0,0 +1,90 @@ +import fs from 'node:fs'; +import readline from 'node:readline'; +import { jsonlFilesForDevice, parseDurationToMs, type HistoryRecord } from './history-query.js'; + +export interface EventWindowOptions { + /** Inclusive lower bound (ms epoch). */ + sinceMs: number; + /** Inclusive upper bound (ms epoch). */ + untilMs: number; + /** Optional predicate to filter parsed records. Records that don't match are dropped. */ + eventFilter?: (record: HistoryRecord) => boolean; + /** Cap the number of records returned (newest-first ordering preserved). */ + limit?: number; +} + +/** + * Query a device's history JSONL files for records in [sinceMs, untilMs]. + * + * Walks rotation files newest-first so consumers that only care about the + * most-recent N events don't have to read the full archive. Stops as soon + * as a file's mtime is older than `sinceMs` (the file can't contain any + * records that match) or `limit` records have been collected. + * + * Returned records are in file order within each file (oldest-first + * inside a file, newer files first). + */ +export async function queryEventWindow( + deviceId: string, + opts: EventWindowOptions, +): Promise { + const { sinceMs, untilMs } = opts; + if (!Number.isFinite(sinceMs) || !Number.isFinite(untilMs)) return []; + if (sinceMs > untilMs) return []; + const limit = Math.max(0, opts.limit ?? Number.POSITIVE_INFINITY); + if (limit === 0) return []; + + // jsonlFilesForDevice returns oldest-first; walk newest-first so we can + // bail early once we hit a file whose mtime predates the window. + const files = jsonlFilesForDevice(deviceId).slice().reverse(); + const out: HistoryRecord[] = []; + + for (const file of files) { + let mtimeMs: number; + try { + mtimeMs = fs.statSync(file).mtimeMs; + } catch { + continue; + } + // The newest record in the file is no later than mtime; if the file + // ends before our window starts, neither it nor any older file can + // contribute. + if (mtimeMs < sinceMs) break; + + const records = await readWindowFromFile(file, sinceMs, untilMs, opts.eventFilter); + out.push(...records); + if (out.length >= limit) { + return out.slice(0, limit); + } + } + + return out; +} + +async function readWindowFromFile( + file: string, + sinceMs: number, + untilMs: number, + eventFilter: ((record: HistoryRecord) => boolean) | undefined, +): Promise { + const stream = fs.createReadStream(file, { encoding: 'utf-8' }); + const rl = readline.createInterface({ input: stream, crlfDelay: Infinity }); + const out: HistoryRecord[] = []; + for await (const line of rl) { + if (!line) continue; + let rec: HistoryRecord; + try { + rec = JSON.parse(line) as HistoryRecord; + } catch { + continue; + } + const tMs = Date.parse(rec.t); + if (!Number.isFinite(tMs)) continue; + if (tMs < sinceMs || tMs > untilMs) continue; + if (eventFilter && !eventFilter(rec)) continue; + out.push(rec); + } + return out; +} + +export { parseDurationToMs }; diff --git a/src/llm/index.ts b/src/llm/index.ts index 8ace362a..be152474 100644 --- a/src/llm/index.ts +++ b/src/llm/index.ts @@ -1,5 +1,6 @@ import { OpenAIProvider } from './providers/openai.js'; import { AnthropicProvider } from './providers/anthropic.js'; +import { LocalProvider } from './providers/local.js'; import type { LLMProvider, LLMProviderOptions } from './provider.js'; export type LLMBackend = 'openai' | 'anthropic' | 'local' | 'auto'; @@ -10,7 +11,8 @@ export type { LLMProvider, LLMProviderOptions }; export const LLM_AUTO_THRESHOLD = 4; export function createLLMProvider(backend: Exclude, opts: LLMProviderOptions = {}): LLMProvider { - if (backend === 'openai' || backend === 'local') return new OpenAIProvider(opts); + if (backend === 'openai') return new OpenAIProvider(opts); if (backend === 'anthropic') return new AnthropicProvider(opts); + if (backend === 'local') return new LocalProvider(opts); throw new Error(`Unknown LLM backend: ${backend}`); } diff --git a/src/llm/pricing.ts b/src/llm/pricing.ts new file mode 100644 index 00000000..ce316a71 --- /dev/null +++ b/src/llm/pricing.ts @@ -0,0 +1,50 @@ +/** + * Per-model USD pricing for input and output tokens. + * + * Numbers are USD per 1M tokens, copied from each provider's public pricing + * page at the time the catalog entry was added. They drift; if a model is + * missing or stale, `calculateCostUsd()` returns undefined and the cost + * dimension is treated as "unknown" โ€” calls and tokens budgeting still works, + * dollar budgeting is skipped for that model. + */ +export interface ModelPricing { + inUsdPer1M: number; + outUsdPer1M: number; +} + +export const PRICING: Record = { + // OpenAI โ€” https://openai.com/api/pricing/ + 'gpt-4o-mini': { inUsdPer1M: 0.15, outUsdPer1M: 0.60 }, + 'gpt-4o': { inUsdPer1M: 2.50, outUsdPer1M: 10.00 }, + 'gpt-4-turbo': { inUsdPer1M: 10.00, outUsdPer1M: 30.00 }, + + // Anthropic โ€” https://www.anthropic.com/pricing + 'claude-haiku-4-5-20251001': { inUsdPer1M: 1.00, outUsdPer1M: 5.00 }, + 'claude-sonnet-4-6': { inUsdPer1M: 3.00, outUsdPer1M: 15.00 }, + 'claude-opus-4-7': { inUsdPer1M: 15.00, outUsdPer1M: 75.00 }, +}; + +export function getModelPricing(model: string): ModelPricing | undefined { + return PRICING[model]; +} + +/** + * Calculate USD cost for a single LLM call given input/output token counts. + * Returns undefined for unknown models โ€” caller treats this as "skip cost + * dimension for this call but keep counting calls and tokens". + */ +export function calculateCostUsd( + model: string, + tokensIn: number, + tokensOut: number, +): number | undefined { + const pricing = PRICING[model]; + if (!pricing) return undefined; + const inputCost = (tokensIn / 1_000_000) * pricing.inUsdPer1M; + const outputCost = (tokensOut / 1_000_000) * pricing.outUsdPer1M; + return inputCost + outputCost; +} + +export function isPricedModel(model: string): boolean { + return model in PRICING; +} diff --git a/src/llm/provider.ts b/src/llm/provider.ts index f33a923f..8ebec3e4 100644 --- a/src/llm/provider.ts +++ b/src/llm/provider.ts @@ -1,15 +1,33 @@ +export interface DecideUsage { + tokensIn: number; + tokensOut: number; + /** Computed by `calculateCostUsd(model, tokensIn, tokensOut)`; undefined if model is not in the pricing table. */ + costUsd?: number; +} + export interface DecideResult { pass: boolean; reason: string; + /** Token + cost accounting for the underlying API call. Absent on cached + * results and when the provider does not report usage. */ + usage?: DecideUsage; } export interface DecideOptions { timeoutMs?: number; } +export interface ProviderCapabilities { + /** Whether this provider supports OpenAI/Anthropic-style structured tool use. + * When false, `decide()` falls back to plain chat completions and parses a + * JSON object out of the model's free-form text. */ + toolUse: boolean; +} + export interface LLMProvider { readonly name: string; readonly model: string; + readonly capabilities: ProviderCapabilities; generateYaml(systemPrompt: string, userIntent: string): Promise; decide(prompt: string, opts?: DecideOptions): Promise; } @@ -19,4 +37,7 @@ export interface LLMProviderOptions { baseUrl?: string; timeoutMs?: number; maxTokens?: number; + /** Optional capability override. Useful when wrapping a tool-use-capable + * endpoint that has been deployed without tool support, or vice-versa. */ + toolUse?: boolean; } diff --git a/src/llm/providers/anthropic.ts b/src/llm/providers/anthropic.ts index 37da09d5..764697a9 100644 --- a/src/llm/providers/anthropic.ts +++ b/src/llm/providers/anthropic.ts @@ -1,9 +1,11 @@ import https from 'node:https'; -import type { LLMProvider, LLMProviderOptions, DecideResult, DecideOptions } from '../provider.js'; +import type { LLMProvider, LLMProviderOptions, DecideResult, DecideOptions, ProviderCapabilities } from '../provider.js'; +import { calculateCostUsd } from '../pricing.js'; export class AnthropicProvider implements LLMProvider { readonly name = 'anthropic'; readonly model: string; + readonly capabilities: ProviderCapabilities = { toolUse: true }; private readonly apiKey: string; private readonly timeoutMs: number; private readonly maxTokens: number; @@ -122,12 +124,18 @@ export class AnthropicProvider implements LLMProvider { const json = JSON.parse(responseBody) as { content: Array<{ type: string; name?: string; input?: { pass: boolean; reason: string } }>; + usage?: { input_tokens?: number; output_tokens?: number }; }; const toolUse = json.content?.find(c => c.type === 'tool_use' && c.name === 'decide'); if (!toolUse?.input || typeof toolUse.input.pass !== 'boolean') { throw new Error('Anthropic decide: malformed tool-use response'); } - return { pass: toolUse.input.pass, reason: String(toolUse.input.reason ?? '').slice(0, 200) }; + const tokensIn = json.usage?.input_tokens ?? 0; + const tokensOut = json.usage?.output_tokens ?? 0; + const usage = json.usage + ? { tokensIn, tokensOut, costUsd: calculateCostUsd(this.model, tokensIn, tokensOut) } + : undefined; + return { pass: toolUse.input.pass, reason: String(toolUse.input.reason ?? '').slice(0, 200), usage }; } } diff --git a/src/llm/providers/local.ts b/src/llm/providers/local.ts new file mode 100644 index 00000000..76d066d3 --- /dev/null +++ b/src/llm/providers/local.ts @@ -0,0 +1,198 @@ +import https from 'node:https'; +import http from 'node:http'; +import type { LLMProvider, LLMProviderOptions, DecideResult, DecideOptions, ProviderCapabilities } from '../provider.js'; +import { calculateCostUsd } from '../pricing.js'; +import { decideViaStructuredOutput } from './structured-output-fallback.js'; + +const DEFAULT_LOCAL_BASE_URL = 'http://localhost:11434/v1'; + +/** + * LocalProvider points at any OpenAI-compatible /v1/chat/completions endpoint + * โ€” Ollama, llama.cpp server, vLLM, LM Studio, etc. By default we assume the + * endpoint does NOT support structured tool use and route `decide()` through + * the JSON-instruction + repair fallback. Operators can flip this with + * `toolUse: true` in YAML or via the `SWITCHBOT_LOCAL_LLM_TOOL_USE=1` env. + */ +export class LocalProvider implements LLMProvider { + readonly name = 'local'; + readonly model: string; + readonly capabilities: ProviderCapabilities; + private readonly apiKey: string; + private readonly baseUrl: string; + private readonly timeoutMs: number; + private readonly maxTokens: number; + + constructor(opts: LLMProviderOptions = {}) { + this.apiKey = process.env.SWITCHBOT_LOCAL_LLM_API_KEY ?? process.env.LOCAL_LLM_API_KEY ?? ''; + this.model = opts.model ?? process.env.SWITCHBOT_LOCAL_LLM_MODEL ?? 'llama3.2'; + this.baseUrl = stripTrailingV1(opts.baseUrl ?? process.env.SWITCHBOT_LOCAL_LLM_URL ?? DEFAULT_LOCAL_BASE_URL); + this.timeoutMs = opts.timeoutMs ?? 60_000; + this.maxTokens = opts.maxTokens ?? 1024; + + const envToolUse = parseBoolEnv(process.env.SWITCHBOT_LOCAL_LLM_TOOL_USE); + const toolUse = opts.toolUse ?? envToolUse ?? false; + this.capabilities = { toolUse }; + } + + async generateYaml(systemPrompt: string, userIntent: string): Promise { + const body = JSON.stringify({ + model: this.model, + messages: [ + { role: 'system', content: systemPrompt }, + { role: 'user', content: userIntent }, + ], + max_tokens: this.maxTokens, + temperature: 0, + }); + + const url = new URL(`${this.baseUrl}/v1/chat/completions`); + const isHttps = url.protocol === 'https:'; + const responseBody = await new Promise((resolve, reject) => { + const req = (isHttps ? https : http).request( + { + hostname: url.hostname, + port: url.port || (isHttps ? 443 : 80), + path: url.pathname, + method: 'POST', + headers: { + ...(this.apiKey ? { 'Authorization': `Bearer ${this.apiKey}` } : {}), + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(body), + }, + timeout: this.timeoutMs, + }, + (res) => { + const chunks: Buffer[] = []; + res.on('data', (c: Buffer) => chunks.push(c)); + res.on('end', () => { + const text = Buffer.concat(chunks).toString('utf-8'); + if (res.statusCode !== undefined && res.statusCode >= 400) { + reject(new Error(`Local LLM API error ${res.statusCode}: ${text.slice(0, 200)}`)); + } else { + resolve(text); + } + }); + }, + ); + req.on('error', reject); + req.on('timeout', () => req.destroy(new Error('LLM request timeout'))); + req.write(body); + req.end(); + }); + + const json = JSON.parse(responseBody) as { choices: Array<{ message: { content: string } }> }; + const content = json.choices?.[0]?.message?.content; + if (!content) throw new Error('Local LLM returned empty content'); + return content.replace(/^```ya?ml\n?/i, '').replace(/\n?```\s*$/i, '').trim(); + } + + async decide(prompt: string, opts: DecideOptions = {}): Promise { + const timeoutMs = opts.timeoutMs ?? this.timeoutMs; + + if (!this.capabilities.toolUse) { + return decideViaStructuredOutput({ + prompt, + apiKey: this.apiKey, + baseUrl: this.baseUrl, + model: this.model, + timeoutMs, + maxTokens: 256, + computeCostUsd: calculateCostUsd, + }); + } + + // Tool-use path: same wire format as OpenAI provider. + const body = JSON.stringify({ + model: this.model, + max_tokens: 256, + tools: [{ + type: 'function', + function: { + name: 'decide', + description: 'Return a boolean pass/fail decision with a brief reason.', + parameters: { + type: 'object', + properties: { + pass: { type: 'boolean' }, + reason: { type: 'string' }, + }, + required: ['pass', 'reason'], + }, + }, + }], + tool_choice: { type: 'function', function: { name: 'decide' } }, + messages: [{ role: 'user', content: prompt }], + }); + + const url = new URL(`${this.baseUrl}/v1/chat/completions`); + const isHttps = url.protocol === 'https:'; + const responseBody = await new Promise((resolve, reject) => { + const req = (isHttps ? https : http).request( + { + hostname: url.hostname, + port: url.port || (isHttps ? 443 : 80), + path: url.pathname, + method: 'POST', + headers: { + ...(this.apiKey ? { 'Authorization': `Bearer ${this.apiKey}` } : {}), + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(body), + }, + timeout: timeoutMs, + }, + (res) => { + const chunks: Buffer[] = []; + res.on('data', (c: Buffer) => chunks.push(c)); + res.on('end', () => { + const text = Buffer.concat(chunks).toString('utf-8'); + if (res.statusCode !== undefined && res.statusCode >= 400) { + reject(new Error(`Local LLM API error ${res.statusCode}: ${text.slice(0, 200)}`)); + } else { + resolve(text); + } + }); + }, + ); + req.on('error', reject); + req.on('timeout', () => req.destroy(new Error('LLM request timeout'))); + req.write(body); + req.end(); + }); + + const json = JSON.parse(responseBody) as { + choices: Array<{ + message: { + tool_calls?: Array<{ function: { name: string; arguments: string } }>; + }; + }>; + usage?: { prompt_tokens?: number; completion_tokens?: number }; + }; + const toolCall = json.choices?.[0]?.message?.tool_calls?.find(tc => tc.function.name === 'decide'); + if (!toolCall) throw new Error('Local LLM decide: no tool call in response'); + const args = JSON.parse(toolCall.function.arguments) as { pass: boolean; reason: string }; + if (typeof args.pass !== 'boolean') throw new Error('Local LLM decide: malformed function-call response'); + const tokensIn = json.usage?.prompt_tokens ?? 0; + const tokensOut = json.usage?.completion_tokens ?? 0; + const usage = json.usage + ? { tokensIn, tokensOut, costUsd: calculateCostUsd(this.model, tokensIn, tokensOut) } + : undefined; + return { pass: args.pass, reason: String(args.reason ?? '').slice(0, 200), usage }; + } + + /** Exposed for doctor `local-llm-reachable` check. */ + getEndpoint(): string { + return this.baseUrl; + } +} + +function stripTrailingV1(url: string): string { + return url.replace(/\/v1\/?$/, '').replace(/\/+$/, ''); +} + +function parseBoolEnv(v: string | undefined): boolean | undefined { + if (!v) return undefined; + const lower = v.trim().toLowerCase(); + if (['1', 'true', 'yes', 'on'].includes(lower)) return true; + if (['0', 'false', 'no', 'off'].includes(lower)) return false; + return undefined; +} diff --git a/src/llm/providers/openai.ts b/src/llm/providers/openai.ts index e2d5a426..f7e04825 100644 --- a/src/llm/providers/openai.ts +++ b/src/llm/providers/openai.ts @@ -1,10 +1,12 @@ import https from 'node:https'; import http from 'node:http'; -import type { LLMProvider, LLMProviderOptions, DecideResult, DecideOptions } from '../provider.js'; +import type { LLMProvider, LLMProviderOptions, DecideResult, DecideOptions, ProviderCapabilities } from '../provider.js'; +import { calculateCostUsd } from '../pricing.js'; export class OpenAIProvider implements LLMProvider { readonly name = 'openai'; readonly model: string; + readonly capabilities: ProviderCapabilities = { toolUse: true }; private readonly apiKey: string; private readonly baseUrl: string; private readonly timeoutMs: number; @@ -137,11 +139,17 @@ export class OpenAIProvider implements LLMProvider { tool_calls?: Array<{ function: { name: string; arguments: string } }>; }; }>; + usage?: { prompt_tokens?: number; completion_tokens?: number }; }; const toolCall = json.choices?.[0]?.message?.tool_calls?.find(tc => tc.function.name === 'decide'); if (!toolCall) throw new Error('OpenAI decide: no tool call in response'); const args = JSON.parse(toolCall.function.arguments) as { pass: boolean; reason: string }; if (typeof args.pass !== 'boolean') throw new Error('OpenAI decide: malformed function-call response'); - return { pass: args.pass, reason: String(args.reason ?? '').slice(0, 200) }; + const tokensIn = json.usage?.prompt_tokens ?? 0; + const tokensOut = json.usage?.completion_tokens ?? 0; + const usage = json.usage + ? { tokensIn, tokensOut, costUsd: calculateCostUsd(this.model, tokensIn, tokensOut) } + : undefined; + return { pass: args.pass, reason: String(args.reason ?? '').slice(0, 200), usage }; } } diff --git a/src/llm/providers/structured-output-fallback.ts b/src/llm/providers/structured-output-fallback.ts new file mode 100644 index 00000000..2728ca93 --- /dev/null +++ b/src/llm/providers/structured-output-fallback.ts @@ -0,0 +1,213 @@ +import https from 'node:https'; +import http from 'node:http'; +import type { DecideResult, DecideUsage } from '../provider.js'; + +export interface StructuredCallOptions { + prompt: string; + apiKey: string; + baseUrl: string; + model: string; + timeoutMs: number; + maxTokens?: number; + /** Optional override for the costUsd hook. */ + computeCostUsd?: (model: string, tokensIn: number, tokensOut: number) => number | undefined; +} + +const SYSTEM_INSTRUCTION = [ + 'You are a yes/no decision endpoint for a smart-home rule engine.', + 'Read the user prompt and reply with ONLY a JSON object in this exact shape:', + '{"pass": , "reason": ""}', + 'No prose, no markdown fences, no extra fields. The JSON must be valid and parseable.', +].join(' '); + +const FEW_SHOT_EXAMPLE = [ + 'Example input: "Is the front door locked? Status: lockState=lock"', + 'Example output: {"pass": true, "reason": "lockState reports lock"}', +].join(' '); + +const REPAIR_INSTRUCTION = [ + 'Your previous response was not valid JSON.', + 'Reply with ONLY the JSON object: {"pass": , "reason": ""}.', + 'No prose, no markdown.', +].join(' '); + +/** + * Calls a chat-completions endpoint without tool use, asks the model to return + * a JSON object describing the decision, and parses it. Performs one repair + * round-trip if the first response is not parseable. + */ +export async function decideViaStructuredOutput(opts: StructuredCallOptions): Promise { + const messages: Array<{ role: 'system' | 'user' | 'assistant'; content: string }> = [ + { role: 'system', content: `${SYSTEM_INSTRUCTION} ${FEW_SHOT_EXAMPLE}` }, + { role: 'user', content: opts.prompt }, + ]; + + const first = await chatCompletion(opts, messages); + const parsed = tryParseDecision(first.text); + if (parsed) { + return finalize(parsed, first.usage, opts); + } + + // Repair round. + messages.push({ role: 'assistant', content: first.text }); + messages.push({ role: 'user', content: REPAIR_INSTRUCTION }); + const second = await chatCompletion(opts, messages); + const repaired = tryParseDecision(second.text); + if (repaired) { + const merged = mergeUsage(first.usage, second.usage); + return finalize(repaired, merged, opts); + } + + throw new Error(`Structured output fallback could not parse a JSON decision after repair retry. Last response: ${second.text.slice(0, 200)}`); +} + +interface CompletionResult { + text: string; + usage?: { input_tokens: number; output_tokens: number }; +} + +async function chatCompletion( + opts: StructuredCallOptions, + messages: Array<{ role: string; content: string }>, +): Promise { + const body = JSON.stringify({ + model: opts.model, + max_tokens: opts.maxTokens ?? 256, + temperature: 0, + messages, + }); + + const url = new URL(`${opts.baseUrl.replace(/\/+$/, '')}/v1/chat/completions`); + const isHttps = url.protocol === 'https:'; + const responseBody = await new Promise((resolve, reject) => { + const req = (isHttps ? https : http).request( + { + hostname: url.hostname, + port: url.port || (isHttps ? 443 : 80), + path: url.pathname, + method: 'POST', + headers: { + ...(opts.apiKey ? { 'Authorization': `Bearer ${opts.apiKey}` } : {}), + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(body), + }, + timeout: opts.timeoutMs, + }, + (res) => { + const chunks: Buffer[] = []; + res.on('data', (c: Buffer) => chunks.push(c)); + res.on('end', () => { + const text = Buffer.concat(chunks).toString('utf-8'); + if (res.statusCode !== undefined && res.statusCode >= 400) { + reject(new Error(`Local LLM API error ${res.statusCode}: ${text.slice(0, 200)}`)); + } else { + resolve(text); + } + }); + }, + ); + req.on('error', reject); + req.on('timeout', () => req.destroy(new Error('LLM request timeout'))); + req.write(body); + req.end(); + }); + + const json = JSON.parse(responseBody) as { + choices?: Array<{ message?: { content?: string } }>; + usage?: { prompt_tokens?: number; completion_tokens?: number }; + }; + const text = json.choices?.[0]?.message?.content ?? ''; + if (!text) throw new Error('Local LLM returned empty content'); + const usage = json.usage + ? { input_tokens: json.usage.prompt_tokens ?? 0, output_tokens: json.usage.completion_tokens ?? 0 } + : undefined; + return { text, usage }; +} + +/** + * Lenient parser: pulls JSON-shaped `{...}` blocks out of the response, strips + * ```json fences if present, and validates the shape. Tries every `{` in the + * text in case the first one is non-JSON prose like `{step 1}`. + */ +export function tryParseDecision(text: string): { pass: boolean; reason: string } | null { + const stripped = text + .replace(/^```(?:json)?\s*/i, '') + .replace(/```\s*$/i, '') + .trim(); + + const candidates: string[] = [stripped, ...extractAllJsonObjects(stripped)]; + for (const candidate of candidates) { + if (!candidate) continue; + try { + const obj = JSON.parse(candidate) as Record; + if (typeof obj.pass !== 'boolean') continue; + const reason = typeof obj.reason === 'string' ? obj.reason : ''; + return { pass: obj.pass, reason: reason.slice(0, 200) }; + } catch { /* try next */ } + } + return null; +} + +function extractAllJsonObjects(text: string): string[] { + const out: string[] = []; + let i = 0; + while (i < text.length) { + const start = text.indexOf('{', i); + if (start === -1) break; + const block = readBalancedBraces(text, start); + if (!block) { + i = start + 1; + continue; + } + out.push(block); + i = start + block.length; + } + return out; +} + +function readBalancedBraces(text: string, start: number): string | null { + let depth = 0; + let inString = false; + let escape = false; + for (let i = start; i < text.length; i++) { + const ch = text[i]; + if (escape) { escape = false; continue; } + if (ch === '\\' && inString) { escape = true; continue; } + if (ch === '"') { inString = !inString; continue; } + if (inString) continue; + if (ch === '{') depth++; + else if (ch === '}') { + depth--; + if (depth === 0) return text.slice(start, i + 1); + } + } + return null; +} + +function mergeUsage( + a?: { input_tokens: number; output_tokens: number }, + b?: { input_tokens: number; output_tokens: number }, +): { input_tokens: number; output_tokens: number } | undefined { + if (!a && !b) return undefined; + return { + input_tokens: (a?.input_tokens ?? 0) + (b?.input_tokens ?? 0), + output_tokens: (a?.output_tokens ?? 0) + (b?.output_tokens ?? 0), + }; +} + +function finalize( + decision: { pass: boolean; reason: string }, + rawUsage: { input_tokens: number; output_tokens: number } | undefined, + opts: StructuredCallOptions, +): DecideResult { + let usage: DecideUsage | undefined; + if (rawUsage) { + const tokensIn = rawUsage.input_tokens; + const tokensOut = rawUsage.output_tokens; + const costUsd = opts.computeCostUsd + ? opts.computeCostUsd(opts.model, tokensIn, tokensOut) + : undefined; + usage = { tokensIn, tokensOut, costUsd }; + } + return { pass: decision.pass, reason: decision.reason, usage }; +} diff --git a/src/policy/schema/v0.2.json b/src/policy/schema/v0.2.json index d307b8d6..8fb8b9d3 100644 --- a/src/policy/schema/v0.2.json +++ b/src/policy/schema/v0.2.json @@ -107,6 +107,16 @@ "minimum": 0, "default": 60, "description": "Maximum LLM condition calls per hour across all rules." + }, + "max_tokens_per_hour": { + "type": "integer", + "minimum": 0, + "description": "Maximum total LLM tokens (input + output) per hour across all rules. Omit to leave unbounded." + }, + "max_cost_per_day_usd": { + "type": "number", + "minimum": 0, + "description": "Maximum cumulative USD cost per day across all rules. Skipped for models not in the pricing table." } } } @@ -331,6 +341,26 @@ } } }, + { + "type": "object", + "additionalProperties": false, + "required": ["event_count"], + "properties": { + "event_count": { + "type": "object", + "additionalProperties": false, + "required": ["device", "window", "min"], + "description": "Cross-event aggregation: count events from a device within a rolling window and assert against [min, max].", + "properties": { + "device": { "type": "string", "description": "deviceId or alias" }, + "event": { "type": "string", "description": "Optional canonical event filter (e.g. motion.detected). Matched against the classified event of each history record." }, + "window": { "type": "string", "pattern": "^\\d+(ms|s|m|h|d)$", "description": "Rolling window measured back from now, e.g. \"5m\", \"1h\"." }, + "min": { "type": "integer", "minimum": 0, "description": "Inclusive minimum count required for the condition to match." }, + "max": { "type": "integer", "minimum": 0, "description": "Optional inclusive maximum. Omit for \"at least min\"." } + } + } + } + }, { "type": "object", "additionalProperties": false, @@ -350,7 +380,9 @@ "type": ["object", "null"], "additionalProperties": false, "properties": { - "max_calls_per_hour": { "type": "integer", "minimum": 0, "default": 10 } + "max_calls_per_hour": { "type": "integer", "minimum": 0, "default": 10 }, + "max_tokens_per_hour": { "type": "integer", "minimum": 0, "description": "Hourly token cap (input + output combined). Omit to leave unbounded." }, + "max_cost_per_day_usd": { "type": "number", "minimum": 0, "description": "Daily USD cap. Skipped for models not in the pricing table." } } }, "on_error": { "enum": ["fail", "pass", "skip"], "default": "fail" }, diff --git a/src/rules/engine.ts b/src/rules/engine.ts index 2047c8ba..4fb2c99f 100644 --- a/src/rules/engine.ts +++ b/src/rules/engine.ts @@ -45,11 +45,15 @@ import { isCommandAction, isNotifyAction, isLlmCondition, + isEventCountCondition, } from './types.js'; import { executeNotifyAction } from './notify.js'; import { Cron } from 'croner'; import { writeAudit, writeEvaluateTrace } from '../utils/audit.js'; -import { TraceBuilder, shouldWriteTrace, HIGH_FREQ_EVENTS, type EvaluateTraceMode } from './trace.js'; +import { TraceBuilder, shouldWriteTrace, HIGH_FREQ_EVENTS, type EvaluateTraceMode, ruleVersion } from './trace.js'; +import { LlmConditionEvaluator } from './llm-condition.js'; +import { queryEventWindow } from '../devices/history-window.js'; +import type { EventWindowFetcher } from './matcher.js'; export interface LintIssue { rule: string; @@ -312,6 +316,31 @@ export function lintRules(automation: AutomationBlock | null | undefined): LintR }); } + // condition-llm-tokens-budget-zero: budget.max_tokens_per_hour set to 0 makes condition always fail + if (llm.budget?.max_tokens_per_hour === 0) { + issues.push({ + rule: r.name, + severity: 'warning', + code: 'condition-llm-tokens-budget-zero', + message: 'llm condition budget.max_tokens_per_hour is 0 โ€” condition will always take the on_error path.', + }); + } + + // condition-llm-cost-without-known-model: cost cap set with provider:auto cannot be evaluated until runtime + // resolves the model. Warn so the operator knows the cost dimension may silently skip if the chosen model + // is not in the pricing table. + if (llm.budget?.max_cost_per_day_usd !== undefined && llm.budget.max_cost_per_day_usd > 0) { + const provider = llm.provider ?? 'auto'; + if (provider === 'auto') { + issues.push({ + rule: r.name, + severity: 'warning', + code: 'condition-llm-cost-without-known-model', + message: 'llm condition sets max_cost_per_day_usd but provider is "auto" โ€” cost dimension is skipped for any model not in the pricing table.', + }); + } + } + // condition-llm-on-error-pass: on_error "pass" silently passes conditions when LLM is unavailable if (llm.on_error === 'pass') { issues.push({ @@ -323,6 +352,32 @@ export function lintRules(automation: AutomationBlock | null | undefined): LintR } } + // event_count condition lints + for (const c of (r.conditions ?? [])) { + if (!isEventCountCondition(c)) continue; + const ec = c.event_count; + + // event_count window must parse as a duration shortcut. + if (!/^\d+(ms|s|m|h|d)$/.test(String(ec.window ?? ''))) { + issues.push({ + rule: r.name, + severity: 'error', + code: 'condition-event-count-bad-window', + message: `event_count.window "${ec.window}" must match \`(ms|s|m|h|d)\` โ€” e.g. "5m", "1h".`, + }); + } + + // min/max sanity + if (ec.max !== undefined && ec.max < ec.min) { + issues.push({ + rule: r.name, + severity: 'error', + code: 'condition-event-count-max-below-min', + message: `event_count.max (${ec.max}) must be โ‰ฅ min (${ec.min}).`, + }); + } + } + const enabled = r.enabled !== false; const hasError = issues.some((i) => i.severity === 'error'); const hasUnsupported = issues.some((i) => i.code === 'trigger-unsupported'); @@ -376,6 +431,12 @@ export interface RulesEngineOptions { * axios client. */ statusFetcher?: DeviceStatusFetcher; + /** + * Override the cross-event aggregation fetcher used by event_count + * conditions and the LLM `recent_events` hook. Test seam โ€” production + * callers leave it unset and the engine reads from the JSONL ring. + */ + eventWindowFetcher?: EventWindowFetcher; } export interface EngineFireEntry { @@ -418,6 +479,8 @@ export class RulesEngine { * keeps the semantics of `max_per` honest. */ private pendingChain: Promise = Promise.resolve(); + private readonly llmEvaluator = new LlmConditionEvaluator(); + private readonly eventWindowFetcher: EventWindowFetcher = defaultEventWindowFetcher; private stats: EngineStats = { started: false, rulesLoaded: 0, @@ -807,6 +870,11 @@ export class RulesEngine { aliases: this.aliases, fetchStatus, trace, + event, + llmEvaluator: this.llmEvaluator, + ruleVersion: ruleVersion(rule), + globalLlmMaxCallsPerHour: this.opts.automation?.llm_budget?.max_calls_per_hour, + eventWindowFetcher: this.opts.eventWindowFetcher ?? this.eventWindowFetcher, }); if (!cond.matched) { // If conditions are not met, the trigger is no longer "continuously stable" โ€” @@ -1004,3 +1072,30 @@ export class RulesEngine { } } } + +/** + * Default cross-event aggregation fetcher used by the engine. Reads the + * append-only JSONL ring at ~/.switchbot/device-history/.jsonl + * for records inside the requested window, classifies each payload via + * the matcher's classifier, and (optionally) filters by canonical event. + */ +export async function defaultEventWindowFetcher( + deviceId: string, + opts: { sinceMs: number; untilMs: number; limit?: number; eventName?: string }, +): Promise { + const records = await queryEventWindow(deviceId, { + sinceMs: opts.sinceMs, + untilMs: opts.untilMs, + limit: opts.limit, + eventFilter: opts.eventName + ? (rec) => classifyMqttPayload(rec.payload).event === opts.eventName + : undefined, + }); + return records.map((rec) => ({ + source: 'mqtt' as const, + event: classifyMqttPayload(rec.payload).event, + t: new Date(rec.t), + deviceId, + payload: rec.payload, + })); +} diff --git a/src/rules/llm-condition.ts b/src/rules/llm-condition.ts index 3f74aa72..58fe8e70 100644 --- a/src/rules/llm-condition.ts +++ b/src/rules/llm-condition.ts @@ -3,6 +3,7 @@ import { deepSortedJson } from './trace.js'; import { writeAudit } from '../utils/audit.js'; import type { LlmCondition } from './types.js'; import type { EngineEvent } from './types.js'; +import type { DecideUsage } from '../llm/provider.js'; export interface LlmConditionContext { event: EngineEvent; @@ -18,21 +19,55 @@ export interface LlmEvaluateResult { cacheHit: boolean; reason: string; promptDigest: string; + /** Token and cost figures from the underlying provider call. Absent on + * cache hits and on errors. */ + usage?: DecideUsage; }; } +/** Effective budget caps applied to an LLM condition evaluation. */ +export interface LlmBudgetCaps { + /** Per-rule + global merged: per-rule wins when set, otherwise global applies. */ + maxCallsPerHour?: number; + maxTokensPerHour?: number; + maxCostPerDayUsd?: number; +} + const HOUR_MS = 60 * 60 * 1000; +const DAY_MS = 24 * HOUR_MS; + +interface BudgetCounter { + /** Calls + tokens use this hourly window. */ + hourlyStart: number; + calls: number; + tokens: number; + /** Cost uses a separate daily window. */ + dailyStart: number; + costUsd: number; +} export class LlmConditionEvaluator { - private cache = new Map(); - private callCounts = new Map(); + private cache = new Map(); + private budgetCounters = new Map(); async evaluate( condition: LlmCondition['llm'], context: LlmConditionContext, ruleVersion: string, - globalMaxCallsPerHour?: number, + budgetCaps?: LlmBudgetCaps | number, ): Promise { + // Backward compatibility: callers in the existing engine pass a single + // number meaning "global max_calls_per_hour". New callers pass a full + // LlmBudgetCaps object. + const caps: LlmBudgetCaps = typeof budgetCaps === 'number' + ? { maxCallsPerHour: budgetCaps } + : { ...(budgetCaps ?? {}) }; + + // Per-rule budget overrides global on a dimension-by-dimension basis. + if (condition.budget?.max_calls_per_hour !== undefined) caps.maxCallsPerHour = condition.budget.max_calls_per_hour; + if (condition.budget?.max_tokens_per_hour !== undefined) caps.maxTokensPerHour = condition.budget.max_tokens_per_hour; + if (condition.budget?.max_cost_per_day_usd !== undefined) caps.maxCostPerDayUsd = condition.budget.max_cost_per_day_usd; + const cacheKey = buildCacheKey(ruleVersion, condition.prompt, context); const ttlMs = parseCacheTtl(condition.cache_ttl ?? '5m'); @@ -53,31 +88,17 @@ export class LlmConditionEvaluator { } } - const perRuleMax = condition.budget?.max_calls_per_hour; - const effectiveMax = perRuleMax ?? globalMaxCallsPerHour; - if (effectiveMax !== undefined && effectiveMax > 0) { - const budgetKey = `${ruleVersion}:${condition.prompt.slice(0, 32)}`; - const now = Date.now(); - const entry = this.callCounts.get(budgetKey) ?? { count: 0, windowStart: now }; - if (now - entry.windowStart >= HOUR_MS) { - entry.count = 0; - entry.windowStart = now; - } - if (entry.count >= effectiveMax) { - writeAudit({ - auditVersion: 2, - t: new Date().toISOString(), - kind: 'llm-budget-exceeded', - deviceId: context.event.deviceId ?? '', - command: 'llm-condition', - parameter: null, - commandType: 'command', - dryRun: false, - }); - return onErrorResult(condition.on_error ?? 'fail', 'Budget exceeded'); - } - entry.count++; - this.callCounts.set(budgetKey, entry); + const budgetKey = `${ruleVersion}:${condition.prompt.slice(0, 32)}`; + const counter = this.rollCounter(budgetKey); + + // Pre-call: check call-count budget. Tokens and cost can only be checked + // after the call returns (we don't know how many tokens a call will use + // until it has run), but if a previous call already pushed us past the + // limit we short-circuit now. + const callViolation = this.checkPreCallBudget(counter, caps); + if (callViolation) { + this.emitBudgetExceeded(callViolation, context, condition); + return onErrorResult(condition.on_error ?? 'fail', `Budget exceeded (${callViolation.dimension})`); } const backend = resolveProvider(condition.provider ?? 'auto'); @@ -92,8 +113,24 @@ export class LlmConditionEvaluator { const result = await provider.decide(prompt, { timeoutMs: condition.timeout_ms ?? 5_000 }); const latencyMs = Date.now() - start; + // Account for this call's usage AFTER the call completed; subsequent + // calls in the same window will see the updated counter and may hit + // the token/cost ceiling. + counter.calls += 1; + if (result.usage) { + counter.tokens += result.usage.tokensIn + result.usage.tokensOut; + if (result.usage.costUsd !== undefined) { + counter.costUsd += result.usage.costUsd; + } + } + if (ttlMs > 0) { - this.cache.set(cacheKey, { result: result.pass, reason: result.reason, expiresAt: Date.now() + ttlMs }); + this.cache.set(cacheKey, { + result: result.pass, + reason: result.reason, + expiresAt: Date.now() + ttlMs, + usage: result.usage, + }); } return { @@ -105,12 +142,69 @@ export class LlmConditionEvaluator { cacheHit: false, reason: String(result.reason ?? '').slice(0, 200), promptDigest: cacheKey.slice(0, 8), + usage: result.usage, }, }; } catch (err) { return onErrorResult(condition.on_error ?? 'fail', String(err)); } } + + private rollCounter(key: string): BudgetCounter { + const now = Date.now(); + const existing = this.budgetCounters.get(key); + if (!existing) { + const fresh: BudgetCounter = { hourlyStart: now, calls: 0, tokens: 0, dailyStart: now, costUsd: 0 }; + this.budgetCounters.set(key, fresh); + return fresh; + } + if (now - existing.hourlyStart >= HOUR_MS) { + existing.hourlyStart = now; + existing.calls = 0; + existing.tokens = 0; + } + if (now - existing.dailyStart >= DAY_MS) { + existing.dailyStart = now; + existing.costUsd = 0; + } + return existing; + } + + private checkPreCallBudget( + counter: BudgetCounter, + caps: LlmBudgetCaps, + ): { dimension: 'calls' | 'tokens' | 'cost'; limit: number; observed: number } | null { + if (caps.maxCallsPerHour !== undefined && caps.maxCallsPerHour >= 0 && counter.calls >= caps.maxCallsPerHour) { + return { dimension: 'calls', limit: caps.maxCallsPerHour, observed: counter.calls }; + } + if (caps.maxTokensPerHour !== undefined && caps.maxTokensPerHour >= 0 && counter.tokens >= caps.maxTokensPerHour) { + return { dimension: 'tokens', limit: caps.maxTokensPerHour, observed: counter.tokens }; + } + if (caps.maxCostPerDayUsd !== undefined && caps.maxCostPerDayUsd >= 0 && counter.costUsd >= caps.maxCostPerDayUsd) { + return { dimension: 'cost', limit: caps.maxCostPerDayUsd, observed: counter.costUsd }; + } + return null; + } + + private emitBudgetExceeded( + violation: { dimension: 'calls' | 'tokens' | 'cost'; limit: number; observed: number }, + context: LlmConditionContext, + _condition: LlmCondition['llm'], + ): void { + writeAudit({ + auditVersion: 2, + t: new Date().toISOString(), + kind: 'llm-budget-exceeded', + deviceId: context.event.deviceId ?? '', + command: 'llm-condition', + parameter: null, + commandType: 'command', + dryRun: false, + budgetDimension: violation.dimension, + budgetLimit: violation.limit, + budgetObserved: violation.observed, + }); + } } function buildCacheKey(ruleVersion: string, promptTemplate: string, context: LlmConditionContext): string { diff --git a/src/rules/matcher.ts b/src/rules/matcher.ts index d5b222fa..99acffa5 100644 --- a/src/rules/matcher.ts +++ b/src/rules/matcher.ts @@ -24,6 +24,7 @@ import { isAnyCondition, isNotCondition, isLlmCondition, + isEventCountCondition, } from './types.js'; import { isWithinTuple } from './quiet-hours.js'; import type { TraceBuilder } from './trace.js'; @@ -103,8 +104,25 @@ export interface EvaluateConditionsContext { llmEvaluator?: LlmConditionEvaluator; ruleVersion?: string; globalLlmMaxCallsPerHour?: number; + /** + * Optional fetcher for cross-event aggregation. Called by `event_count` + * conditions and used to populate `recent_events` for LLM conditions. + * The matcher stays pure; engine callers wire this to history-window. + */ + eventWindowFetcher?: EventWindowFetcher; } +/** + * Pluggable history-window fetcher used by event_count conditions and the + * LLM condition's `recent_events` hook. Returns historical events for a + * device in [sinceMs, untilMs] (inclusive). Implementations should clamp + * `limit` to bound IO. + */ +export type EventWindowFetcher = ( + deviceId: string, + opts: { sinceMs: number; untilMs: number; limit?: number; eventName?: string }, +) => Promise; + /** * Evaluate all conditions; AND-joined at the top level. Composite nodes * (all/any/not) are evaluated recursively. Unsupported conditions short- @@ -224,9 +242,10 @@ async function evaluateSingle( }; } try { + const recent = await fetchRecentEvents(c.llm.recent_events, ctx); const res = await ctx.llmEvaluator.evaluate( c.llm, - { event: ctx.event }, + { event: ctx.event, recentEvents: recent }, ctx.ruleVersion ?? 'unknown', ctx.globalLlmMaxCallsPerHour, ); @@ -239,6 +258,45 @@ async function evaluateSingle( } } + if (isEventCountCondition(c)) { + if (!ctx.eventWindowFetcher) { + return { + matched: false, + failures: [], + unsupported: [{ keyword: 'event_count', hint: 'event_count evaluation requires a history fetcher; this call site did not provide one.' }], + }; + } + const ec = c.event_count; + const resolved = resolveDeviceRef(ec.device, ctx.aliases); + if (!resolved) return fail(`event_count: could not resolve device "${ec.device}" to an id (no matching alias).`); + const windowMs = parseDurationOrNull(ec.window); + if (windowMs === null || windowMs <= 0) { + return fail(`event_count: invalid window "${ec.window}" โ€” expected e.g. "30s", "5m", "1h".`); + } + const untilMs = now.getTime(); + const sinceMs = untilMs - windowMs; + try { + const events = await ctx.eventWindowFetcher(resolved, { + sinceMs, + untilMs, + limit: Math.max(ec.min, ec.max ?? ec.min) + 1, + eventName: ec.event, + }); + const count = events.length; + const min = ec.min; + const max = ec.max; + const ceilingViolated = max !== undefined && count > max; + const floorViolated = count < min; + if (floorViolated || ceilingViolated) { + const range = max !== undefined ? `${min}โ€“${max}` : `โ‰ฅ ${min}`; + return fail(`event_count ${ec.device}${ec.event ? ` ${ec.event}` : ''} in ${ec.window}: ${count} (expected ${range})`); + } + return ok; + } catch (err) { + return fail(`event_count ${ec.device}: fetch failed โ€” ${err instanceof Error ? err.message : String(err)}`); + } + } + return { matched: false, failures: [], @@ -322,6 +380,7 @@ function conditionKind(c: Condition): string { if (isTimeBetween(c)) return 'time_between'; if (isDeviceState(c)) return 'device_state'; if (isLlmCondition(c)) return 'llm'; + if (isEventCountCondition(c)) return 'event_count'; return 'unknown'; } @@ -329,6 +388,7 @@ function conditionConfig(c: Condition): unknown { if (isTimeBetween(c)) return c.time_between; if (isDeviceState(c)) return { device: c.device, field: c.field, op: c.op, value: c.value }; if (isLlmCondition(c)) return { prompt: c.llm.prompt.slice(0, 80) }; + if (isEventCountCondition(c)) return c.event_count; return undefined; } @@ -339,3 +399,35 @@ function pushConditionTrace(trace: TraceBuilder, c: Condition, sub: ConditionEva passed: sub.unsupported.length > 0 ? false : sub.matched, }); } + +function parseDurationOrNull(spec: string): number | null { + const m = String(spec ?? '').trim().match(/^(\d+)(ms|s|m|h|d)$/i); + if (!m) return null; + const n = Number(m[1]); + const unit = m[2].toLowerCase(); + const factor = unit === 'ms' ? 1 : unit === 's' ? 1_000 : unit === 'm' ? 60_000 : unit === 'h' ? 3_600_000 : 86_400_000; + return n * factor; +} + +async function fetchRecentEvents( + count: number | undefined, + ctx: EvaluateConditionsContext, +): Promise { + if (!count || count <= 0) return undefined; + if (!ctx.eventWindowFetcher || !ctx.event?.deviceId) return undefined; + // Look back 24h by default โ€” recent_events is a "last N matching events" + // hook, not a window-anchored aggregator. The fetcher caps the read. + const untilMs = ctx.event.t.getTime(); + const sinceMs = untilMs - 24 * 60 * 60 * 1000; + try { + const events = await ctx.eventWindowFetcher(ctx.event.deviceId, { + sinceMs, + untilMs, + limit: count, + eventName: ctx.event.event, + }); + return events.slice(-count); + } catch { + return undefined; + } +} diff --git a/src/rules/simulate.ts b/src/rules/simulate.ts index d3e9dde2..83c566e1 100644 --- a/src/rules/simulate.ts +++ b/src/rules/simulate.ts @@ -7,6 +7,7 @@ import { ThrottleGate, parseMaxPerMs } from './throttle.js'; import { ruleVersion } from './trace.js'; import { filterTraceRecords } from './trace.js'; import { matchesMqttTrigger } from './matcher.js'; +import { defaultEventWindowFetcher } from './engine.js'; import type { Rule, EngineEvent } from './types.js'; import type { RuleEvaluateRecord } from './trace.js'; @@ -122,6 +123,7 @@ export async function simulateRule(opts: SimulateOptions): Promise { + it('returns a platform-appropriate path', () => { + const sp = getDaemonSocketPath(); + if (process.platform === 'win32') { + expect(sp).toMatch(/^\\\\\.\\pipe\\switchbot-daemon-/); + } else { + expect(sp.endsWith('daemon.sock')).toBe(true); + } + }); +}); + +describe('Daemon IPC: server + client', () => { + let server: { close: () => Promise; socketPath: string } | null = null; + + afterEach(async () => { + if (server) await server.close(); + server = null; + }); + + it('responds to a basic JSON-RPC call with the registered handler', async () => { + const socketPath = tempSocketPath('basic'); + server = await startIpcServer({ + socketPath, + handlers: { + 'echo': (params) => ({ echoed: params }), + }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + const result = await client.call<{ echoed: unknown }>('echo', { hello: 'world' }); + expect(result).toEqual({ echoed: { hello: 'world' } }); + }); + + it('returns an error response when the method is unknown', async () => { + const socketPath = tempSocketPath('unknown'); + server = await startIpcServer({ + socketPath, + handlers: { 'known': () => ({ ok: true }) }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + await expect(client.call('does-not-exist')).rejects.toThrow(/Method not found/); + }); + + it('propagates handler errors as JSON-RPC error responses', async () => { + const socketPath = tempSocketPath('handler-error'); + server = await startIpcServer({ + socketPath, + handlers: { + 'boom': () => { throw new Error('handler exploded'); }, + }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + try { + await client.call('boom'); + expect.fail('Expected boom to throw'); + } catch (err) { + expect(err).toBeInstanceOf(IpcDaemonClientError); + expect((err as IpcDaemonClientError).message).toContain('handler exploded'); + } + }); + + it('handles concurrent calls without crossing responses', async () => { + const socketPath = tempSocketPath('concurrent'); + server = await startIpcServer({ + socketPath, + handlers: { + 'add': async (params) => { + const p = params as { a: number; b: number }; + await new Promise((r) => setTimeout(r, 10)); + return { sum: p.a + p.b }; + }, + }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + const results = await Promise.all([ + client.call<{ sum: number }>('add', { a: 1, b: 2 }), + client.call<{ sum: number }>('add', { a: 10, b: 20 }), + client.call<{ sum: number }>('add', { a: 100, b: 200 }), + ]); + expect(results.map((r) => r.sum)).toEqual([3, 30, 300]); + }); + + it('async handler results round-trip', async () => { + const socketPath = tempSocketPath('async'); + server = await startIpcServer({ + socketPath, + handlers: { + 'slow-status': async () => { + await new Promise((r) => setTimeout(r, 5)); + return { status: 'running', rulesActive: 7 }; + }, + }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + const result = await client.call<{ status: string; rulesActive: number }>('slow-status'); + expect(result).toEqual({ status: 'running', rulesActive: 7 }); + }); + + it('throws a recognizable error when the daemon is not listening', async () => { + const bogusPath = tempSocketPath('bogus-not-listening'); + const client = new IpcDaemonClient({ socketPath: bogusPath, timeoutMs: 1_500, connectTimeoutMs: 1_000 }); + await expect(client.call('daemon.status')).rejects.toThrow(/IPC daemon not listening|IPC connect timed out|IPC socket error/); + }); + + it('rejects malformed (non-JSON) lines with a parse error response', async () => { + const socketPath = tempSocketPath('malformed'); + server = await startIpcServer({ + socketPath, + handlers: { 'noop': () => ({}) }, + }); + + // Manually open a socket and send malformed data โ€” we shouldn't crash. + const net = await import('node:net'); + await new Promise((resolve, reject) => { + const sock = net.createConnection(socketPath); + let buffer = ''; + sock.on('connect', () => { + sock.write('not-json\n'); + }); + sock.on('data', (chunk: Buffer) => { + buffer += chunk.toString('utf-8'); + if (buffer.includes('\n')) { + try { + const response = JSON.parse(buffer.trim()); + expect(response.error.code).toBe(-32700); + sock.end(); + resolve(); + } catch (err) { + reject(err); + } + } + }); + sock.on('error', reject); + }); + }); + + it('client.ping() resolves with latency when daemon.status is registered', async () => { + const socketPath = tempSocketPath('ping'); + server = await startIpcServer({ + socketPath, + handlers: { + 'daemon.status': () => ({ status: 'running', rulesActive: 2 }), + }, + }); + + const client = new IpcDaemonClient({ socketPath, timeoutMs: 2_000, connectTimeoutMs: 1_000 }); + const result = await client.ping(); + expect(typeof result.latencyMs).toBe('number'); + expect(result.latencyMs).toBeGreaterThanOrEqual(0); + expect(result.status).toEqual({ status: 'running', rulesActive: 2 }); + }); + + it('isListening() reports true after start, false after close', async () => { + const socketPath = tempSocketPath('listening'); + const handle = await startIpcServer({ socketPath, handlers: { 'noop': () => ({}) } }); + server = handle; + expect(handle.isListening()).toBe(true); + await handle.close(); + server = null; + expect(handle.isListening()).toBe(false); + }); +}); diff --git a/tests/devices/catalog.test.ts b/tests/devices/catalog.test.ts index 57c2c07f..5081479f 100644 --- a/tests/devices/catalog.test.ts +++ b/tests/devices/catalog.test.ts @@ -51,6 +51,8 @@ const OFFICIAL_API_DEVICE_TYPES = [ 'Keypad Vision Pro', 'Lock Lite', 'Lock Ultra', + 'Lock Vision', + 'Lock Vision Pro', 'Meter', 'MeterPlus', 'MeterPro', @@ -83,6 +85,7 @@ const OFFICIAL_API_DEVICE_TYPES = [ 'Roller Shade', 'Smart Lock', 'Smart Lock Pro', + 'Smart Lock Pro Wifi', 'Smart Lock Ultra', 'Smart Radiator Thermostat', 'Standing Circulator Fan', @@ -90,6 +93,7 @@ const OFFICIAL_API_DEVICE_TYPES = [ 'Strip Light 3', 'Video Doorbell', 'Water Detector', + 'WeatherStation', 'WoIOSensor', ] as const; @@ -161,6 +165,8 @@ const OFFICIAL_SUPPORTED_DEVICE_LIST_NAMES = [ 'Keypad Vision', 'Keypad Vision Pro', 'Lock Ultra', + 'Lock Vision', + 'Lock Vision Pro', 'Standing Circulator Fan', 'Pan/Tilt Cam Plus 2K', 'Pan/Tilt Cam Plus 3K', @@ -168,6 +174,7 @@ const OFFICIAL_SUPPORTED_DEVICE_LIST_NAMES = [ 'Candle Warmer Lamp', 'Home Climate Panel', 'Smart Radiator Thermostat', + 'Weather Station', 'AI Art Frame', ] as const; @@ -287,6 +294,34 @@ describe('devices/catalog', () => { expect(tierOf('Smart Lock Ultra', 'unlock')).toBe('destructive'); }); + it('Lock Vision and Lock Vision Pro unlock are safetyTier: destructive', () => { + expect(tierOf('Lock Vision', 'unlock')).toBe('destructive'); + expect(tierOf('Lock Vision Pro', 'unlock')).toBe('destructive'); + }); + + it('AI Art Frame uploadImage is idempotent and exposes a URL example', () => { + const cmd = commandOf('AI Art Frame', 'uploadImage'); + expect(cmd, 'AI Art Frame should expose an uploadImage command').toBeDefined(); + expect(cmd?.idempotent).toBe(true); + expect(cmd?.parameter).toBeTruthy(); + expect(cmd?.exampleParams?.[0]).toMatch(/^https:\/\//); + }); + + it('Smart Lock entry resolves the Matter alias "Smart Lock Pro Wifi"', () => { + const match = findCatalogEntry('Smart Lock Pro Wifi'); + expect(match).not.toBeNull(); + expect(Array.isArray(match)).toBe(false); + expect((match as { type: string } | null)?.type).toBe('Smart Lock'); + }); + + it('Weather Station resolves and reports atmosphericPressure', () => { + const match = findCatalogEntry('Weather Station'); + expect(match).not.toBeNull(); + const entry = match as { type: string; statusFields: string[] } | null; + expect(entry?.type).toBe('WeatherStation'); + expect(entry?.statusFields).toContain('atmosphericPressure'); + }); + it('Garage Door Opener turnOn and turnOff are safetyTier: destructive', () => { expect(tierOf('Garage Door Opener', 'turnOn')).toBe('destructive'); expect(tierOf('Garage Door Opener', 'turnOff')).toBe('destructive'); diff --git a/tests/devices/history-window.test.ts b/tests/devices/history-window.test.ts new file mode 100644 index 00000000..144f07b3 --- /dev/null +++ b/tests/devices/history-window.test.ts @@ -0,0 +1,143 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import fs from 'node:fs'; +import path from 'node:path'; +import os from 'node:os'; +import { queryEventWindow } from '../../src/devices/history-window.js'; + +// queryEventWindow uses jsonlFilesForDevice which reads from +// ~/.switchbot/device-history. We point HOME at a tmpdir to keep tests +// hermetic. +function mkTempHome(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sb-history-')); + return dir; +} + +function seedHistoryFile(homeDir: string, deviceId: string, lines: string[]): string { + const dir = path.join(homeDir, '.switchbot', 'device-history'); + fs.mkdirSync(dir, { recursive: true }); + const file = path.join(dir, `${deviceId}.jsonl`); + fs.writeFileSync(file, lines.join('\n') + (lines.length > 0 ? '\n' : '')); + return file; +} + +describe('queryEventWindow', () => { + let originalHome: string | undefined; + let originalUserprofile: string | undefined; + let homeDir: string; + + beforeEach(() => { + originalHome = process.env.HOME; + originalUserprofile = process.env.USERPROFILE; + homeDir = mkTempHome(); + process.env.HOME = homeDir; + process.env.USERPROFILE = homeDir; + }); + + afterEach(() => { + process.env.HOME = originalHome; + process.env.USERPROFILE = originalUserprofile; + try { + fs.rmSync(homeDir, { recursive: true, force: true }); + } catch { /* */ } + }); + + it('returns empty when device has no history file', async () => { + const records = await queryEventWindow('AA_BB_CC', { sinceMs: 0, untilMs: Date.now() }); + expect(records).toEqual([]); + }); + + it('returns records inside the [sinceMs, untilMs] window', async () => { + const now = Date.parse('2026-05-15T08:00:00.000Z'); + const lines = [ + JSON.stringify({ t: '2026-05-15T07:00:00.000Z', topic: 't', payload: { context: { detectionState: 'DETECTED' } } }), + JSON.stringify({ t: '2026-05-15T07:30:00.000Z', topic: 't', payload: { context: { detectionState: 'DETECTED' } } }), + JSON.stringify({ t: '2026-05-15T07:45:00.000Z', topic: 't', payload: { context: { detectionState: 'NOT_DETECTED' } } }), + JSON.stringify({ t: '2026-05-15T07:55:00.000Z', topic: 't', payload: { context: { detectionState: 'DETECTED' } } }), + ]; + seedHistoryFile(homeDir, 'AA_BB_CC', lines); + + const records = await queryEventWindow('AA_BB_CC', { + sinceMs: now - 10 * 60 * 1000, // 10m back from 08:00 = 07:50 + untilMs: now, + }); + expect(records).toHaveLength(1); + expect(records[0].t).toBe('2026-05-15T07:55:00.000Z'); + }); + + it('returns no records when sinceMs > untilMs', async () => { + seedHistoryFile(homeDir, 'AA_BB_CC', [ + JSON.stringify({ t: '2026-05-15T07:00:00.000Z', topic: 't', payload: {} }), + ]); + const records = await queryEventWindow('AA_BB_CC', { sinceMs: 1000, untilMs: 500 }); + expect(records).toEqual([]); + }); + + it('honors limit by stopping after `limit` records', async () => { + const now = Date.parse('2026-05-15T08:00:00.000Z'); + const lines = Array.from({ length: 10 }, (_, i) => { + const t = new Date(now - (10 - i) * 60_000).toISOString(); + return JSON.stringify({ t, topic: 'shadow', payload: { context: { detectionState: 'DETECTED' } } }); + }); + seedHistoryFile(homeDir, 'AA_BB_CC', lines); + + const records = await queryEventWindow('AA_BB_CC', { + sinceMs: now - 60 * 60_000, + untilMs: now, + limit: 3, + }); + expect(records).toHaveLength(3); + }); + + it('applies eventFilter to drop non-matching records', async () => { + const now = Date.parse('2026-05-15T08:00:00.000Z'); + const lines = [ + JSON.stringify({ t: '2026-05-15T07:50:00.000Z', topic: 't', payload: { context: { detectionState: 'DETECTED' } } }), + JSON.stringify({ t: '2026-05-15T07:55:00.000Z', topic: 't', payload: { context: { detectionState: 'NOT_DETECTED' } } }), + JSON.stringify({ t: '2026-05-15T07:58:00.000Z', topic: 't', payload: { context: { detectionState: 'DETECTED' } } }), + ]; + seedHistoryFile(homeDir, 'AA_BB_CC', lines); + + const records = await queryEventWindow('AA_BB_CC', { + sinceMs: now - 30 * 60_000, + untilMs: now, + eventFilter: (rec) => { + const ctx = (rec.payload as { context?: { detectionState?: string } })?.context; + return ctx?.detectionState === 'DETECTED'; + }, + }); + expect(records).toHaveLength(2); + expect(records.every((r) => { + const ctx = (r.payload as { context: { detectionState: string } }).context; + return ctx.detectionState === 'DETECTED'; + })).toBe(true); + }); + + it('skips malformed JSON lines silently', async () => { + const now = Date.parse('2026-05-15T08:00:00.000Z'); + const lines = [ + '{ not valid json', + JSON.stringify({ t: '2026-05-15T07:55:00.000Z', topic: 't', payload: {} }), + 'still bad', + JSON.stringify({ t: '2026-05-15T07:58:00.000Z', topic: 't', payload: {} }), + ]; + seedHistoryFile(homeDir, 'AA_BB_CC', lines); + + const records = await queryEventWindow('AA_BB_CC', { + sinceMs: now - 30 * 60_000, + untilMs: now, + }); + expect(records).toHaveLength(2); + }); + + it('returns 0 records when limit is 0', async () => { + seedHistoryFile(homeDir, 'AA_BB_CC', [ + JSON.stringify({ t: '2026-05-15T07:55:00.000Z', topic: 't', payload: {} }), + ]); + const records = await queryEventWindow('AA_BB_CC', { + sinceMs: 0, + untilMs: Date.now(), + limit: 0, + }); + expect(records).toEqual([]); + }); +}); diff --git a/tests/llm/pricing.test.ts b/tests/llm/pricing.test.ts new file mode 100644 index 00000000..37d6fd35 --- /dev/null +++ b/tests/llm/pricing.test.ts @@ -0,0 +1,72 @@ +import { describe, it, expect } from 'vitest'; +import { calculateCostUsd, getModelPricing, isPricedModel, PRICING } from '../../src/llm/pricing.js'; + +describe('llm/pricing', () => { + describe('PRICING table', () => { + it('has at least one OpenAI and one Anthropic entry', () => { + const keys = Object.keys(PRICING); + expect(keys).toContain('gpt-4o-mini'); + expect(keys).toContain('claude-haiku-4-5-20251001'); + }); + + it('every entry has positive USD-per-1M numbers', () => { + for (const [model, p] of Object.entries(PRICING)) { + expect(p.inUsdPer1M, `${model} input price`).toBeGreaterThan(0); + expect(p.outUsdPer1M, `${model} output price`).toBeGreaterThan(0); + } + }); + + it('output price is at least input price for every entry', () => { + // Output tokens are always priced โ‰ฅ input on real provider sheets; + // a row failing this is almost certainly a typo when adding a new model. + for (const [model, p] of Object.entries(PRICING)) { + expect(p.outUsdPer1M, `${model} output should be โ‰ฅ input`).toBeGreaterThanOrEqual(p.inUsdPer1M); + } + }); + }); + + describe('calculateCostUsd', () => { + it('returns the expected USD figure for gpt-4o-mini', () => { + // 1M input tokens ร— $0.15 = $0.15; 1M output tokens ร— $0.60 = $0.60; total $0.75 + const cost = calculateCostUsd('gpt-4o-mini', 1_000_000, 1_000_000); + expect(cost).toBeCloseTo(0.75, 6); + }); + + it('scales linearly with token count', () => { + const cost1k = calculateCostUsd('gpt-4o-mini', 1_000, 1_000)!; + const cost10k = calculateCostUsd('gpt-4o-mini', 10_000, 10_000)!; + expect(cost10k).toBeCloseTo(cost1k * 10, 8); + }); + + it('returns undefined for an unknown model', () => { + expect(calculateCostUsd('unknown-model', 100, 100)).toBeUndefined(); + }); + + it('returns 0 cost for a known model when token counts are zero', () => { + expect(calculateCostUsd('claude-haiku-4-5-20251001', 0, 0)).toBe(0); + }); + + it('handles claude-haiku pricing correctly', () => { + // 1k input ร— $1 / 1M = $0.001; 1k output ร— $5 / 1M = $0.005; total $0.006 + const cost = calculateCostUsd('claude-haiku-4-5-20251001', 1_000, 1_000); + expect(cost).toBeCloseTo(0.006, 6); + }); + }); + + describe('getModelPricing / isPricedModel', () => { + it('getModelPricing returns the entry for a known model', () => { + const p = getModelPricing('gpt-4o-mini'); + expect(p).toBeDefined(); + expect(p!.inUsdPer1M).toBe(0.15); + }); + + it('getModelPricing returns undefined for an unknown model', () => { + expect(getModelPricing('llama3.2')).toBeUndefined(); + }); + + it('isPricedModel reports true for a known model and false for an unknown one', () => { + expect(isPricedModel('claude-haiku-4-5-20251001')).toBe(true); + expect(isPricedModel('llama3.2')).toBe(false); + }); + }); +}); diff --git a/tests/llm/providers/local.test.ts b/tests/llm/providers/local.test.ts new file mode 100644 index 00000000..e76b6713 --- /dev/null +++ b/tests/llm/providers/local.test.ts @@ -0,0 +1,146 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import http from 'node:http'; +import { LocalProvider } from '../../../src/llm/providers/local.js'; + +interface StubServer { + url: string; + close: () => Promise; + requests: Array<{ path: string; body: unknown }>; +} + +async function startStubServer(handler: (req: { body: unknown }, res: http.ServerResponse) => void): Promise { + const requests: Array<{ path: string; body: unknown }> = []; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on('data', (c: Buffer) => chunks.push(c)); + req.on('end', () => { + const raw = Buffer.concat(chunks).toString('utf-8'); + let body: unknown = null; + try { body = raw ? JSON.parse(raw) : null; } catch { body = raw; } + requests.push({ path: req.url ?? '', body }); + handler({ body }, res); + }); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', () => resolve())); + const addr = server.address(); + if (typeof addr === 'string' || !addr) throw new Error('Failed to bind stub server'); + const url = `http://127.0.0.1:${addr.port}`; + return { + url, + requests, + close: () => new Promise((resolve) => server.close(() => resolve())), + }; +} + +describe('LocalProvider', () => { + let server: StubServer | null = null; + + afterEach(async () => { + if (server) await server.close(); + server = null; + delete process.env.SWITCHBOT_LOCAL_LLM_TOOL_USE; + delete process.env.SWITCHBOT_LOCAL_LLM_URL; + }); + + it('declares toolUse:false by default (matches local LLM reality)', () => { + const p = new LocalProvider({ baseUrl: 'http://localhost:11434' }); + expect(p.capabilities.toolUse).toBe(false); + expect(p.name).toBe('local'); + }); + + it('respects explicit toolUse:true override from constructor', () => { + const p = new LocalProvider({ baseUrl: 'http://localhost:11434', toolUse: true }); + expect(p.capabilities.toolUse).toBe(true); + }); + + it('respects SWITCHBOT_LOCAL_LLM_TOOL_USE=1 env', () => { + process.env.SWITCHBOT_LOCAL_LLM_TOOL_USE = '1'; + const p = new LocalProvider({ baseUrl: 'http://localhost:11434' }); + expect(p.capabilities.toolUse).toBe(true); + }); + + it('decide() (no tool use) parses JSON from chat completion response', async () => { + server = await startStubServer((_req, res) => { + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.end(JSON.stringify({ + choices: [{ message: { content: '{"pass": true, "reason": "all systems nominal"}' } }], + usage: { prompt_tokens: 30, completion_tokens: 10 }, + })); + }); + const p = new LocalProvider({ baseUrl: server.url, model: 'llama3.2' }); + const result = await p.decide('Is the door locked?'); + expect(result.pass).toBe(true); + expect(result.reason).toBe('all systems nominal'); + expect(result.usage?.tokensIn).toBe(30); + expect(result.usage?.tokensOut).toBe(10); + }); + + it('decide() retries with repair instruction when first response is not JSON', async () => { + let callCount = 0; + server = await startStubServer((_req, res) => { + callCount++; + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + const body = callCount === 1 + ? { choices: [{ message: { content: 'I think the answer is no' } }] } + : { choices: [{ message: { content: '{"pass": false, "reason": "no match"}' } }] }; + res.end(JSON.stringify(body)); + }); + const p = new LocalProvider({ baseUrl: server.url, model: 'llama3.2' }); + const result = await p.decide('Is condition X true?'); + expect(result.pass).toBe(false); + expect(result.reason).toBe('no match'); + expect(server.requests.length).toBe(2); + }); + + it('decide() (no tool use) sends NO tools field in the request body', async () => { + server = await startStubServer((_req, res) => { + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.end(JSON.stringify({ + choices: [{ message: { content: '{"pass": true, "reason": "ok"}' } }], + })); + }); + const p = new LocalProvider({ baseUrl: server.url }); + await p.decide('test'); + const body = server.requests[0].body as Record; + expect(body.tools).toBeUndefined(); + expect(body.tool_choice).toBeUndefined(); + }); + + it('decide() throws when both attempts fail to produce parseable JSON', async () => { + server = await startStubServer((_req, res) => { + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.end(JSON.stringify({ + choices: [{ message: { content: 'I really cannot give you JSON sorry' } }], + })); + }); + const p = new LocalProvider({ baseUrl: server.url }); + await expect(p.decide('test')).rejects.toThrow(/Structured output fallback could not parse/); + }); + + it('decide() (tool use enabled) sends tool_choice=decide and parses tool_calls', async () => { + server = await startStubServer((_req, res) => { + res.statusCode = 200; + res.setHeader('content-type', 'application/json'); + res.end(JSON.stringify({ + choices: [{ message: { tool_calls: [{ function: { name: 'decide', arguments: JSON.stringify({ pass: true, reason: 'tool path' }) } }] } }], + usage: { prompt_tokens: 5, completion_tokens: 2 }, + })); + }); + const p = new LocalProvider({ baseUrl: server.url, toolUse: true }); + const result = await p.decide('hello'); + expect(result.pass).toBe(true); + expect(result.reason).toBe('tool path'); + const body = server.requests[0].body as Record; + expect(body.tools).toBeDefined(); + expect(body.tool_choice).toBeDefined(); + }); + + it('strips trailing /v1 in baseUrl so we do not double-up the path', () => { + const p = new LocalProvider({ baseUrl: 'http://localhost:11434/v1' }); + expect(p.getEndpoint()).toBe('http://localhost:11434'); + }); +}); diff --git a/tests/llm/structured-output-fallback.test.ts b/tests/llm/structured-output-fallback.test.ts new file mode 100644 index 00000000..b5d5b5ab --- /dev/null +++ b/tests/llm/structured-output-fallback.test.ts @@ -0,0 +1,63 @@ +import { describe, it, expect } from 'vitest'; +import { tryParseDecision } from '../../src/llm/providers/structured-output-fallback.js'; + +describe('tryParseDecision', () => { + it('parses a clean JSON object', () => { + const r = tryParseDecision('{"pass": true, "reason": "looks good"}'); + expect(r).toEqual({ pass: true, reason: 'looks good' }); + }); + + it('parses JSON wrapped in ```json fences', () => { + const r = tryParseDecision('```json\n{"pass": false, "reason": "no match"}\n```'); + expect(r).toEqual({ pass: false, reason: 'no match' }); + }); + + it('parses JSON wrapped in plain ``` fences', () => { + const r = tryParseDecision('```\n{"pass": true, "reason": "ok"}\n```'); + expect(r).toEqual({ pass: true, reason: 'ok' }); + }); + + it('extracts a JSON object embedded in surrounding prose', () => { + const r = tryParseDecision('Sure! Here is my answer: {"pass": true, "reason": "yes"} Have a nice day.'); + expect(r).toEqual({ pass: true, reason: 'yes' }); + }); + + it('handles nested braces correctly when extracting from prose', () => { + const r = tryParseDecision('Reasoning {step 1} -> {"pass": false, "reason": "blocked"} done'); + expect(r).toEqual({ pass: false, reason: 'blocked' }); + }); + + it('truncates reason at 200 chars', () => { + const longReason = 'x'.repeat(500); + const r = tryParseDecision(`{"pass": true, "reason": "${longReason}"}`); + expect(r?.reason.length).toBe(200); + }); + + it('returns null when no JSON object is present', () => { + expect(tryParseDecision('I cannot answer that question')).toBeNull(); + }); + + it('returns null when JSON has no pass field', () => { + expect(tryParseDecision('{"reason": "no pass"}')).toBeNull(); + }); + + it('returns null when pass is not a boolean', () => { + expect(tryParseDecision('{"pass": "yes", "reason": "stringy"}')).toBeNull(); + }); + + it('handles missing reason gracefully', () => { + const r = tryParseDecision('{"pass": true}'); + expect(r).toEqual({ pass: true, reason: '' }); + }); + + it('handles strings containing braces correctly', () => { + const r = tryParseDecision('{"pass": true, "reason": "the value was {abc}"}'); + expect(r).toEqual({ pass: true, reason: 'the value was {abc}' }); + }); + + it('handles escaped quotes inside strings', () => { + const r = tryParseDecision('{"pass": false, "reason": "he said \\"no\\""}'); + expect(r?.pass).toBe(false); + expect(r?.reason).toBe('he said "no"'); + }); +}); diff --git a/tests/rules/event-count-condition.test.ts b/tests/rules/event-count-condition.test.ts new file mode 100644 index 00000000..95d1078a --- /dev/null +++ b/tests/rules/event-count-condition.test.ts @@ -0,0 +1,233 @@ +import { describe, it, expect } from 'vitest'; +import { evaluateConditions, type EventWindowFetcher } from '../../src/rules/matcher.js'; +import type { EngineEvent } from '../../src/rules/types.js'; + +function makeEvent(overrides: Partial = {}): EngineEvent { + return { + source: 'mqtt', + event: 'motion.detected', + t: new Date('2026-05-15T08:00:00.000Z'), + deviceId: 'AA:BB:CC', + ...overrides, + }; +} + +function fetcherWithEvents(events: EngineEvent[]): EventWindowFetcher { + return async (deviceId, opts) => { + return events.filter((e) => { + if (e.deviceId !== deviceId) return false; + const tMs = e.t.getTime(); + if (tMs < opts.sinceMs || tMs > opts.untilMs) return false; + if (opts.eventName && e.event !== opts.eventName) return false; + return true; + }); + }; +} + +function buildEvents(deviceId: string, eventName: string, timestamps: string[]): EngineEvent[] { + return timestamps.map((iso) => ({ + source: 'mqtt' as const, + event: eventName, + t: new Date(iso), + deviceId, + })); +} + +describe('event_count condition', () => { + it('matches when count is within [min, max]', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents( + buildEvents('AA:BB:CC', 'motion.detected', [ + '2026-05-15T07:56:00.000Z', + '2026-05-15T07:57:00.000Z', + '2026-05-15T07:58:00.000Z', + ]), + ); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', event: 'motion.detected', window: '5m', min: 3 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(true); + expect(result.failures).toEqual([]); + }); + + it('fails when count is below min', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents( + buildEvents('AA:BB:CC', 'motion.detected', [ + '2026-05-15T07:56:00.000Z', + '2026-05-15T07:57:00.000Z', + ]), + ); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', event: 'motion.detected', window: '5m', min: 3 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(false); + expect(result.failures[0]).toContain('event_count'); + expect(result.failures[0]).toContain('2'); + }); + + it('fails when count exceeds max', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents( + buildEvents('AA:BB:CC', 'motion.detected', [ + '2026-05-15T07:55:00.000Z', + '2026-05-15T07:56:00.000Z', + '2026-05-15T07:57:00.000Z', + '2026-05-15T07:58:00.000Z', + '2026-05-15T07:59:00.000Z', + ]), + ); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', event: 'motion.detected', window: '5m', min: 1, max: 3 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(false); + expect(result.failures[0]).toContain('event_count'); + }); + + it('only counts events inside the rolling window', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents( + buildEvents('AA:BB:CC', 'motion.detected', [ + '2026-05-15T07:00:00.000Z', // outside 5m window + '2026-05-15T07:30:00.000Z', // outside 5m window + '2026-05-15T07:58:00.000Z', // inside + ]), + ); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', event: 'motion.detected', window: '5m', min: 3 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(false); + }); + + it('omits the event filter when "event" is not specified', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const events: EngineEvent[] = [ + ...buildEvents('AA:BB:CC', 'motion.detected', ['2026-05-15T07:58:00.000Z']), + ...buildEvents('AA:BB:CC', 'contact.opened', ['2026-05-15T07:59:00.000Z']), + ]; + const fetcher = fetcherWithEvents(events); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', window: '5m', min: 2 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(true); + }); + + it('flags as unsupported when no eventWindowFetcher is provided', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', window: '5m', min: 1 } }], + now, + {}, + ); + + expect(result.matched).toBe(false); + expect(result.unsupported).toHaveLength(1); + expect(result.unsupported[0].keyword).toBe('event_count'); + }); + + it('rejects malformed window strings as fail', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents([]); + + const result = await evaluateConditions( + [{ event_count: { device: 'AA:BB:CC', window: 'oops', min: 1 } }], + now, + { eventWindowFetcher: fetcher }, + ); + + expect(result.matched).toBe(false); + expect(result.failures[0]).toContain('window'); + }); + + it('resolves alias to deviceId before fetching', async () => { + const now = new Date('2026-05-15T08:00:00.000Z'); + const fetcher = fetcherWithEvents( + buildEvents('AA:BB:CC', 'motion.detected', [ + '2026-05-15T07:58:00.000Z', + ]), + ); + + const result = await evaluateConditions( + [{ event_count: { device: 'front-door', event: 'motion.detected', window: '5m', min: 1 } }], + now, + { + aliases: { 'front-door': 'AA:BB:CC' }, + eventWindowFetcher: fetcher, + }, + ); + + expect(result.matched).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// event_count condition lints +// --------------------------------------------------------------------------- + +describe('event_count condition lints', () => { + it('condition-event-count-bad-window fires for malformed window', async () => { + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'mqtt', event: 'motion.detected' }, + conditions: [{ event_count: { device: 'AA:BB:CC', window: 'forever', min: 3 } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some((i) => i.code === 'condition-event-count-bad-window')).toBe(true); + }); + + it('condition-event-count-max-below-min fires when max < min', async () => { + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'mqtt', event: 'motion.detected' }, + conditions: [{ event_count: { device: 'AA:BB:CC', window: '5m', min: 5, max: 2 } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some((i) => i.code === 'condition-event-count-max-below-min')).toBe(true); + }); + + it('event_count with valid window and ranges does not lint', async () => { + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'mqtt', event: 'motion.detected' }, + conditions: [{ event_count: { device: 'AA:BB:CC', window: '5m', min: 3, max: 10 } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some((i) => i.code?.startsWith('condition-event-count'))).toBe(false); + }); +}); diff --git a/tests/rules/llm-condition.test.ts b/tests/rules/llm-condition.test.ts index 6e06650b..2def166c 100644 --- a/tests/rules/llm-condition.test.ts +++ b/tests/rules/llm-condition.test.ts @@ -31,6 +31,16 @@ function mockProvider(pass: boolean, reason = 'ok') { }; } +// Mock provider whose decide() reports usage (token + cost) info. +function mockProviderWithUsage(pass: boolean, reason: string, usage: { tokensIn: number; tokensOut: number; costUsd?: number }) { + return { + name: 'mock', + model: 'mock-model', + generateYaml: vi.fn().mockResolvedValue(''), + decide: vi.fn().mockResolvedValue({ pass, reason, usage }), + }; +} + // --------------------------------------------------------------------------- // Tests // --------------------------------------------------------------------------- @@ -212,6 +222,79 @@ describe('LlmConditionEvaluator', () => { expect(r4.traceFields.reason).toContain('Budget exceeded'); }); + it('token budget exceeded: third call returns on_error result with dimension "tokens"', async () => { + const evaluator = new LlmConditionEvaluator(); + const provider = mockProviderWithUsage(true, 'ok', { tokensIn: 60, tokensOut: 40 }); + + const condition = { prompt: 'Check?', cache_ttl: 'none', budget: { max_tokens_per_hour: 150 } }; + const ctx = makeCtx(); + + // First call: 100 tokens consumed (60+40) + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + // Second call: 200 tokens cumulative โ€” over the 150 cap, but only the THIRD pre-call check sees it + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + const r3 = await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + + expect(provider.decide).toHaveBeenCalledTimes(2); + expect(r3.pass).toBe(false); + expect(r3.traceFields.reason).toContain('tokens'); + }); + + it('cost budget exceeded: USD ceiling stops further calls', async () => { + const evaluator = new LlmConditionEvaluator(); + const provider = mockProviderWithUsage(true, 'ok', { tokensIn: 100, tokensOut: 100, costUsd: 0.50 }); + + const condition = { prompt: 'Check?', cache_ttl: 'none', budget: { max_cost_per_day_usd: 0.75 } }; + const ctx = makeCtx(); + + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + const r3 = await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + + expect(provider.decide).toHaveBeenCalledTimes(2); + expect(r3.pass).toBe(false); + expect(r3.traceFields.reason).toContain('cost'); + }); + + it('cost dimension is skipped when usage.costUsd is undefined (unknown model)', async () => { + const evaluator = new LlmConditionEvaluator(); + // costUsd absent โ€” unknown model, cost cap should be ignored + const provider = mockProviderWithUsage(true, 'ok', { tokensIn: 100, tokensOut: 100 }); + + const condition = { prompt: 'Check?', cache_ttl: 'none', budget: { max_cost_per_day_usd: 0.001 } }; + const ctx = makeCtx(); + + // All three calls should succeed because the cost dimension is unknown + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + const r3 = await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + + expect(provider.decide).toHaveBeenCalledTimes(3); + expect(r3.pass).toBe(true); + }); + + it('traceFields.usage carries provider-reported tokens and cost', async () => { + const evaluator = new LlmConditionEvaluator(); + const provider = mockProviderWithUsage(true, 'ok', { tokensIn: 50, tokensOut: 25, costUsd: 0.0001 }); + + const result = await evaluateWithProvider(evaluator, provider, { prompt: 'Check?' }, makeCtx(), 'v1'); + expect(result.traceFields.usage).toEqual({ tokensIn: 50, tokensOut: 25, costUsd: 0.0001 }); + }); + + it('cache hit does not carry usage in trace (no provider call was made)', async () => { + const evaluator = new LlmConditionEvaluator(); + const provider = mockProviderWithUsage(true, 'ok', { tokensIn: 10, tokensOut: 5, costUsd: 0.0001 }); + + const condition = { prompt: 'Check?', cache_ttl: '5m' }; + const ctx = makeCtx(); + + await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + const cached = await evaluateWithProvider(evaluator, provider, condition, ctx, 'v1'); + + expect(cached.traceFields.cacheHit).toBe(true); + expect(cached.traceFields.usage).toBeUndefined(); + }); + it('reason is truncated to 200 chars from provider', async () => { const evaluator = new LlmConditionEvaluator(); const longReason = 'x'.repeat(300); @@ -331,6 +414,86 @@ describe('LLM condition lint rules', () => { expect(issues.some(i => i.code === 'condition-llm-budget-zero')).toBe(true); }); + it('condition-llm-tokens-budget-zero fires when max_tokens_per_hour is 0', async () => { + process.env.ANTHROPIC_API_KEY = 'key'; + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'cron', schedule: '0 8 * * *' }, + conditions: [{ llm: { prompt: 'Check?', budget: { max_tokens_per_hour: 0 } } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some(i => i.code === 'condition-llm-tokens-budget-zero')).toBe(true); + }); + + it('condition-llm-tokens-budget-zero does not fire when max_tokens_per_hour is positive', async () => { + process.env.ANTHROPIC_API_KEY = 'key'; + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'cron', schedule: '0 8 * * *' }, + conditions: [{ llm: { prompt: 'Check?', budget: { max_tokens_per_hour: 1000 } } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some(i => i.code === 'condition-llm-tokens-budget-zero')).toBe(false); + }); + + it('condition-llm-cost-without-known-model fires when cost cap set with provider:auto', async () => { + process.env.ANTHROPIC_API_KEY = 'key'; + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'cron', schedule: '0 8 * * *' }, + conditions: [{ llm: { prompt: 'Check?', budget: { max_cost_per_day_usd: 1.00 } } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some(i => i.code === 'condition-llm-cost-without-known-model')).toBe(true); + }); + + it('condition-llm-cost-without-known-model does not fire when provider is explicit', async () => { + process.env.ANTHROPIC_API_KEY = 'key'; + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'cron', schedule: '0 8 * * *' }, + conditions: [{ llm: { prompt: 'Check?', provider: 'anthropic', budget: { max_cost_per_day_usd: 1.00 } } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some(i => i.code === 'condition-llm-cost-without-known-model')).toBe(false); + }); + + it('condition-llm-cost-without-known-model does not fire when cost cap is zero', async () => { + process.env.ANTHROPIC_API_KEY = 'key'; + const { lintRules } = await import('../../src/rules/engine.js'); + const result = lintRules({ + enabled: true, + rules: [{ + name: 'test', + when: { source: 'cron', schedule: '0 8 * * *' }, + conditions: [{ llm: { prompt: 'Check?', budget: { max_cost_per_day_usd: 0 } } }], + then: [{ command: 'turnOn', device: 'light' }], + }], + }); + const issues = result.rules[0].issues; + expect(issues.some(i => i.code === 'condition-llm-cost-without-known-model')).toBe(false); + }); + it('condition-llm-on-error-pass fires when on_error is "pass"', async () => { process.env.ANTHROPIC_API_KEY = 'key'; const { lintRules } = await import('../../src/rules/engine.js');