diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3185a95d..29961678 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -186,6 +186,7 @@ crawler/src/ - **`findDuplicates`**: 重複タイトル・説明の検出 - **`findMismatches`**: メタデータ不一致の検出（canonical, og:title, og:description） - **`getResourceReferrers`**: リソースを参照しているページの特定 +- **`getDirectoryTree`** / **`listDirectoryChildren`** / **`listDirectoryPages`**: URL パスをディレクトリツリーとして閲覧するための read-model 専用 API（`viewer_directory_nodes` / `viewer_directory_pages`）。GET 時に URL 文字列の split/prefix 集計を一切行わない（下記「設計注意」参照） - **`checkHeaders`**: セキュリティヘッダーチェック（CSP, X-Frame-Options, X-Content-Type-Options, HSTS） - **`classifyContentType`**: 生 MIME を 18 個の `ContentTypeCategory`（`html`/`pdf`/`csv`/`word`/`excel`/`powerpoint`/`image`/`css`/`javascript`/`json`/`xml`/`font`/`audio`/`video`/`archive`/`text`/`other`/`unknown`）に正規化する純関数。`getSummary` の `contentTypeDistribution` 集計と `listPages` の `contentTypeCategory` フィルタが同じカテゴリ境界を共有する単一の源泉。**カテゴリ統合**: `csv` は CSV + TSV、`word` は .doc + .docx、`excel` は .xls + .xlsx、`powerpoint` は .ppt + .pptx、`json` は JSON + YAML、`text` は .txt + .md を 1 カテゴリにまとめる（拡張子別の分散を避け、ユーザーが「文書ファイル」「データファイル」をまとめて評価できるようにする設計判断） - **`applyCategoryFilter`**: `ContentTypeCategory` を Knex query に適用する SQL マッチャ。後述の「Content-Type ルール表」から派生し、JS classifier の優先順位を SQL に正確に投影する @@ -238,7 +239,7 @@ SQL マッチャ側はこの優先順位を「目的カテゴリの positive 節 **構成（単一パッケージ、backend + frontend 同居）:** -- **backend（`src/` → `tsc` → `lib/`）**: Hono アプリ。`start-viewer.ts`（サーバ起動 + ブラウザオープン + SIGINT graceful shutdown）、`create-app.ts`（全ルート登録 + `serveStatic` + エラーハンドラ）、`archive-context.ts`（`ArchiveManager` で 1 アーカイブを常駐保持）、`routes/register-*-route.ts`（query 関数 1:1 の 12 ルート） +- **backend（`src/` → `tsc` → `lib/`）**: Hono アプリ。`start-viewer.ts`（サーバ起動 + ブラウザオープン + SIGINT graceful shutdown）、`create-app.ts`（全ルート登録 + `serveStatic` + エラーハンドラ）、`archive-context.ts`（`ArchiveManager` で 1 アーカイブを常駐保持）、`routes/register-*-route.ts`（query 関数 1:1 の 22 ルート） - **frontend（`web/` → `vite build` → `lib/public/`）**: React 19 SPA。`@tanstack/react-query` + `@tanstack/react-table` をベースに、**ページネーションモードを TopBar から切替**できる（`PagedTable` = MPA、`VirtualTable` = `@tanstack/react-virtual` 経由の仮想スクロール、`DataTable` がモードで dispatch）。BrowserRouter（History API、未マッチ GET は Hono が index.html を返す SPA フォールバック）ルーティング、`@nitpicker/query` の型を DTO として再利用。ダーク/ライトテーマ切替（`data-theme` + localStorage、`web/theme/`）、i18n（en/ja、`web/i18n/` の自前辞書 + Context）、列リサイズ（マウス + 矢印キー）、ローディングスケルトン + `aria-busy` + グローバル進捗バー、画像サムネイルプレビュー、Mismatches の赤緑文字差分（`web/utils/diff-text.ts`）を備える。アクセシビリティ（WCAG 2.1 AA）対応として、`PagedTable` / `VirtualTable` 両方への明示的 ARIA グリッドロール、スキップリンク（`web/components/skip-link.tsx`）、フォームコントロールのアクセシブルネーム、ライブリージョン、`prefers-reduced-motion`、AA コントラストを実装（README「アクセシビリティ」節 + 下記「設計注意」参照） **データフロー:** @@ -252,7 +253,7 @@ nitpicker viewer → SIGINT/SIGTERM: manager.closeAll() → server.close() → resolve（CLI が exit） ``` -**REST API（アーカイブは起動時固定なので archiveId 不要）:** `GET /api/summary`, `/api/pages`, `/api/pages/detail?url=`, `/api/pages/html?url=`, `/api/links?type=`, `/api/resources`, `/api/resources/referrers?resourceUrl=`, `/api/images`, `/api/violations`, `/api/duplicates`, `/api/mismatches`, `/api/headers`, `/api/graph`（内部ページのリンクグラフ、`getLinkGraph`）, `/api/page-links`（全ページの status/referrers/headers 一覧、google-sheets「Links」シート相当、`listPageLinks`）, `/api/info`（開いているアーカイブの絶対パス、フッター表示用）。クエリパラメータ → query options 変換は `query-params/to-number.ts` / `to-boolean.ts`、エラーは `sanitize-error-message.ts` で絶対パスを伏せて JSON 返却（mcp-server と同方針）。 +**REST API（アーカイブは起動時固定なので archiveId 不要）:** `GET /api/summary`, `/api/pages`, `/api/pages/detail?url=`, `/api/pages/html?url=`, `/api/links?type=`, `/api/resources`, `/api/resources/referrers?resourceUrl=`, `/api/images`, `/api/violations`, `/api/duplicates`, `/api/mismatches`, `/api/headers`, `/api/graph`（内部ページのリンクグラフ、`getLinkGraph`）, `/api/page-links`（全ページの status/referrers/headers 一覧、google-sheets「Links」シート相当、`listPageLinks`）, `/api/directory-tree`（全 root の初期 3 depth ツリー、`getDirectoryTree`）, `/api/directory-tree/children?nodeId=`（1 ノード直下の子ディレクトリ、`listDirectoryChildren`）, `/api/directory-tree/pages?nodeId=&cursor=&limit=`（1 ディレクトリ直下ページの cursor 一覧、`listDirectoryPages`）, `/api/info`（開いているアーカイブの絶対パス、フッター表示用）。クエリパラメータ → query options 変換は `query-params/to-number.ts` / `to-boolean.ts`、エラーは `sanitize-error-message.ts` で絶対パスを伏せて JSON 返却（mcp-server と同方針）。 **バイナリ:** なし（CLI の `viewer` サブコマンド経由で起動） @@ -326,6 +327,20 @@ nitpicker viewer > > **見つかった構造的な制約（今回は追加対応せず）**: `EXPLAIN QUERY PLAN` を見ると、全ての filter/sort 組み合わせで `USE TEMP B-TREE FOR ORDER BY` が付く。原因は default view の `content_category IN ('html', 'unknown')` — 2 値の `IN` は、たとえ最適な `vp_default` を `INDEXED BY` で強制しても（`SEARCH ... USING COVERING INDEX vp_default (is_external=? AND content_category=?)` は出るが `USE TEMP B-TREE FOR ORDER BY` は消えない）解消しない。SQLite が `IN` の各値ごとに別々の index range を辿るため、結合後の `url_sort_key` 順を保証できず明示的な merge sort が必要になる（単一値の `content_category = 'html'` に絞ると同じ `vp_default` で温存ソート自体が消えることを確認済み）。**追加 index では解決しない**種類の制約であり、40万行規模でも target 内に収まっているため本 PR では対応を見送る。将来 archive 規模がさらに増える、または default view の latency を追い込む必要が出た場合は、「default view 専用の合成カラム（`is_default_category` のような boolean）で `IN` を等価条件に落とす」または「`html` 用と `unknown` 用の 2 range scan を明示的に UNION ALL + merge する」が候補。**検索キーワード**: 「facet 空」「動的フィルタ 空」「Pages facet」「USE TEMP B-TREE FOR ORDER BY」「viewer_pages ベンチマーク」「/api/pages ベンチマーク」。 +> **設計注意（ディレクトリツリー read model、issue #107）:** `viewer_directory_nodes` / `viewer_directory_pages` は `viewer_pages` と同じ denormalized 生カラム方式（`url_refs`/`text_refs` 等の ref 化テーブルは issue #139 で別途未着手のため使わない）。`buildDirectoryTreeRows`（`viewer-read-model/build-directory-tree-rows.ts`）が `buildViewerReadModel` 内で `viewer_pages` と同じ `sourceRows`（二重 SELECT なし）から純粋関数としてツリー全体をメモリ上に構築し、`node_id` もこの時点で連番採番して bulk insert する（SQLite の `INTEGER PRIMARY KEY` は `rowid` 同様に明示値 insert を受け付ける）。 +> +> **root_key はホスト単位、ただし internal ページを 1 件も持たないホストは除外する**: crawl scope は `(hostname, port, path)` トリプル（`find-scope-entry.ts`）なので、同一ホストの scope 外パスは `isExternal=1` でも同じツリーに属するのが正しく、`internal_descendant_page_count` / `external_descendant_page_count` を分けて持つ理由もここにある。一方、外部リンク先のみのホスト（twitter.com 等）を無条件にツリー化すると大規模サイトでは無意味な 1 ページツリーが大量発生するため、internal ページを 1 件も持たないホストは `viewer_directory_nodes` に一切現れない（`viewer_pages`/Pages 一覧には引き続き表示される）。 +> +> **ディレクトリ/ページの境界は末尾スラッシュで判定する**: `pathname` が `/` で終わらない場合は最後のセグメントをページのファイル名、それ以外を親ディレクトリチェーンとする。`/` で終わる場合（またはルート）は全セグメントがディレクトリチェーンとなり、そのディレクトリ自身に index ページとして直接ページが付く。`/blog/2024/post-1` と `/blog/2024/` は同じ `/blog/2024/` ノードに着地する。 +> +> **`has_children` は `direct_child_dir_count > 0` のみで判定し、`direct_page_count` を含めない**: `buildDirectoryTreeRows` の構築ロジック上、生成されるノードは必ず直下ページか子ディレクトリのどちらかを最低 1 つ持つため、両者の合計で判定すると理論上絶対に `false` にならない（デッドコード化する）。ツリー UI の展開矢印は `listDirectoryChildren` で取得できる子ディレクトリの有無だけを気にすればよく（直下ページは別の `/api/directory-tree/pages` パネルで表示される）、この定義であればリーフディレクトリ（ページのみ）とサブディレクトリを持つディレクトリを正しく区別できる。 +> +> **`getDirectoryTree` は `rootKey` パラメータを取らない**: アーカイブ自身が root 情報を知っているため、呼び出し側に事前指定を要求するのは無駄という判断（複数ホストがあれば `{ roots: [...] }` で一括返却）。`listDirectoryChildren` / `listDirectoryPages` も `nodeId` が一意にツリーを特定するため `rootKey` 不要。 +> +> **read model 未構築時は空応答（legacy フォールバックは存在しない）**: `/api/pages` の `listViewerPages` とは異なり、この機能には旧テーブルへの fallback 経路がそもそも無いため、3 関数とも `hasViewerReadModel` ではなく `isViewerReadModelCurrent` を guard に使う。バージョン不問の `hasViewerReadModel` だけを見ると、v3 以前（`viewer_directory_nodes` が存在しないスキーマ）の read model を掴んだ際に `no such table` の 500 になる。スキーマ変更を伴うため `VIEWER_READ_MODEL_SCHEMA_VERSION` を 3→4 に bump し、旧バージョンの read model は自動再ビルド対象にした。 +> +> **`getDirectoryTree` の ORDER BY は `path_sort_key` 単独、`root_key` を含めない**: 全 root を 1 クエリで返す設計上、`root_key` の等価フィルタが存在しないため、`vdn_root_depth_path (root_key, depth, path_sort_key, node_id)` のような `root_key` 先頭 index は `depth <= 3` という range 条件との組み合わせで一切活用できず、`EXPLAIN QUERY PLAN` で実測すると `USE TEMP B-TREE FOR LAST TERM OF ORDER BY` が付く（PR #96 の `idx_pages_listfilter` column 順ミスと同型の教訓）。`path_sort_key` を先頭に置いた `vdn_path_depth (path_sort_key, depth, node_id)` に張り替え、`ORDER BY path_sort_key` のみに変更することで `SCAN ... USING INDEX vdn_path_depth`（sort 無し、`depth` は残差フィルタ）に収まることを確認済み。root_key を ORDER BY から外しても、grouping は JS 側で `Map` に振り分けるだけなので各 root 内の相対順序（`path_sort_key` 昇順）は保たれる。**検索キーワード**: 「directory-tree」「ディレクトリツリー」「has_children」「vdn_path_depth」「USE TEMP B-TREE」。 + ### @nitpicker/cli `@d-zero/roar` ベースの統合 CLI。7つのサブコマンドを提供。全 analyze プラグインを `dependencies` に含んでおり、`npx` 実行時に `@nitpicker/core` の動的 `import()` がプラグインモジュールを解決できるようにしている。 diff --git a/CLAUDE.md b/CLAUDE.md index bcf27f43..ebb1f2ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -79,6 +79,8 @@ packages/ > **Note (`/api/pages` の viewer_pages fast path)**: `/api/pages` は `urlPattern`/`directory` 未指定 かつ `viewer_pages` read model が最新の場合のみ `listViewerPages`（narrow indexed read model + keyset cursor）を使い、それ以外は従来の `listPages`（wide table + offset）にフォールバックする。read model のビルドタイミング（issue #112）は3経路: (1) crawl完了時（`CrawlerOrchestrator.write()` 直前、CLI層の `ensureViewerReadModelQuietly` 経由）に persistent read model を自動ビルド、(2) `nitpicker viewer-build [--force]` による明示ビルド（既存アーカイブの事前永続化用）、(3) `Archive.openCached` 経由の read-only open（viewer/MCP/query CLI 共通）で read model が無い/古い場合の on-open opportunistic build（タールキャッシュ dir にのみ書き込み、元の `.nitpicker` は不変。専用ロックで多重ビルドを防止、失敗時は warn して legacy fallback）。stub mode は常にビルド対象外。legacy 経路も offset を素の文字列 cursor として返すことで、フロントの `nextCursor`-only 継続契約（仮想スクロール）をどちらの経路でも満たす。詳細は ARCHITECTURE.md の `@nitpicker/viewer` 節「設計注意（`/api/pages` の viewer_pages fast path...）」を正とする。 +> **Note (ディレクトリツリー read model、issue #107)**: `viewer_directory_nodes` / `viewer_directory_pages` は `viewer_pages` を返す `sourceRows` を再利用し `buildDirectoryTreeRows` が純粋関数としてメモリ上に構築する。**root_key はホスト単位、ただし internal ページを 1 件も持たないホストは除外**（外部リンク先ドメインの無意味な 1 ページツリーを防ぐ）。**ディレクトリ/ページ境界は末尾スラッシュで判定**（`/blog/2024/post-1` と `/blog/2024/` は同じ `/blog/2024/` ノードに着地）。**`has_children` は `direct_child_dir_count > 0` のみ**（`direct_page_count` を含めると構築ロジック上絶対に `false` にならないため、UI の展開矢印が意味を持つよう子ディレクトリの有無だけを見る）。この機能に legacy フォールバックは存在しないため、3関数（`getDirectoryTree`/`listDirectoryChildren`/`listDirectoryPages`）とも `hasViewerReadModel` ではなく `isViewerReadModelCurrent` を guard に使う。詳細は ARCHITECTURE.md の `@nitpicker/viewer` 節「設計注意（ディレクトリツリー read model...）」を正とする。 + ## CLI コマンド ```sh diff --git a/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.spec.ts b/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.spec.ts new file mode 100644 index 00000000..7adab197 --- /dev/null +++ b/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.spec.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from 'vitest'; + +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from '../viewer-read-model/viewer-read-model-schema-version.js'; + +import { decodeDirectoryPagesCursor } from './decode-directory-pages-cursor.js'; +import { encodeDirectoryPagesCursor } from './encode-directory-pages-cursor.js'; + +const NODE_ID = 42; + +describe('decodeDirectoryPagesCursor', () => { + it('decodes a cursor minted for the expected nodeId', () => { + const cursor = encodeDirectoryPagesCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + nodeId: NODE_ID, + pageUrlSortKey: 'https://example.com/a', + pageId: 7, + }); + expect(decodeDirectoryPagesCursor(cursor, NODE_ID)).toEqual({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + nodeId: NODE_ID, + pageUrlSortKey: 'https://example.com/a', + pageId: 7, + }); + }); + + it('throws on an undecodable string', () => { + expect(() => decodeDirectoryPagesCursor('%%%not-base64%%%', NODE_ID)).toThrow( + /not decodable/, + ); + }); + + it('throws on a decodable but malformed payload', () => { + const cursor = Buffer.from(JSON.stringify({ foo: 'bar' }), 'utf8').toString( + 'base64url', + ); + expect(() => decodeDirectoryPagesCursor(cursor, NODE_ID)).toThrow(/malformed/); + }); + + it('throws on a cursor minted under a stale schema version', () => { + const cursor = encodeDirectoryPagesCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION - 1, + nodeId: NODE_ID, + pageUrlSortKey: 'https://example.com/a', + pageId: 7, + }); + expect(() => decodeDirectoryPagesCursor(cursor, NODE_ID)).toThrow(/[Ss]tale/); + }); + + it('throws on a cursor minted for a different nodeId', () => { + const cursor = encodeDirectoryPagesCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + nodeId: NODE_ID + 1, + pageUrlSortKey: 'https://example.com/a', + pageId: 7, + }); + expect(() => decodeDirectoryPagesCursor(cursor, NODE_ID)).toThrow(/does not match/); + }); +}); diff --git a/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.ts b/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.ts new file mode 100644 index 00000000..9c1292e5 --- /dev/null +++ b/packages/@nitpicker/query/src/directory-pages-cursor/decode-directory-pages-cursor.ts @@ -0,0 +1,47 @@ +import type { DirectoryPagesCursorPayload } from './types.js'; + +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from '../viewer-read-model/viewer-read-model-schema-version.js'; + +/** + * Decodes and validates an opaque `/api/directory-tree/pages` cursor against + * the current request's `nodeId`. Rejects cursors minted under a different + * schema version or a different `nodeId` — replaying a cursor against a + * different directory would silently seek to a nonsensical position. + * @param cursor - The opaque cursor string from the request. + * @param expectedNodeId - The current request's `nodeId`. + * @returns The decoded, validated payload. + * @throws {Error} If the cursor is malformed, stale, or was minted for a + * different `nodeId`. + */ +export function decodeDirectoryPagesCursor( + cursor: string, + expectedNodeId: number, +): DirectoryPagesCursorPayload { + let payload: DirectoryPagesCursorPayload; + try { + payload = JSON.parse(Buffer.from(cursor, 'base64url').toString('utf8')); + } catch { + throw new Error('Invalid /api/directory-tree/pages cursor: not decodable'); + } + if ( + typeof payload !== 'object' || + payload === null || + typeof payload.v !== 'number' || + typeof payload.nodeId !== 'number' || + typeof payload.pageUrlSortKey !== 'string' || + typeof payload.pageId !== 'number' + ) { + throw new Error('Invalid /api/directory-tree/pages cursor: malformed payload'); + } + if (payload.v !== VIEWER_READ_MODEL_SCHEMA_VERSION) { + throw new Error( + 'Stale /api/directory-tree/pages cursor: read-model schema has changed since it was issued', + ); + } + if (payload.nodeId !== expectedNodeId) { + throw new Error( + 'Invalid /api/directory-tree/pages cursor: does not match the current nodeId', + ); + } + return payload; +} diff --git a/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.spec.ts b/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.spec.ts new file mode 100644 index 00000000..18767f0c --- /dev/null +++ b/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.spec.ts @@ -0,0 +1,22 @@ +import { describe, expect, it } from 'vitest'; + +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from '../viewer-read-model/viewer-read-model-schema-version.js'; + +import { encodeDirectoryPagesCursor } from './encode-directory-pages-cursor.js'; + +describe('encodeDirectoryPagesCursor', () => { + it('encodes a payload as a base64url string, decodable back to the same JSON', () => { + const payload = { + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + nodeId: 42, + pageUrlSortKey: 'https://example.com/a', + pageId: 7, + }; + const cursor = encodeDirectoryPagesCursor(payload); + expect(cursor).not.toContain('+'); + expect(cursor).not.toContain('/'); + expect(JSON.parse(Buffer.from(cursor, 'base64url').toString('utf8'))).toEqual( + payload, + ); + }); +}); diff --git a/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.ts b/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.ts new file mode 100644 index 00000000..63291483 --- /dev/null +++ b/packages/@nitpicker/query/src/directory-pages-cursor/encode-directory-pages-cursor.ts @@ -0,0 +1,11 @@ +import type { DirectoryPagesCursorPayload } from './types.js'; + +/** + * Encodes a `/api/directory-tree/pages` cursor payload as an opaque, + * URL-safe string. + * @param payload - The cursor payload to encode. + * @returns The base64url-encoded cursor. + */ +export function encodeDirectoryPagesCursor(payload: DirectoryPagesCursorPayload): string { + return Buffer.from(JSON.stringify(payload), 'utf8').toString('base64url'); +} diff --git a/packages/@nitpicker/query/src/directory-pages-cursor/types.ts b/packages/@nitpicker/query/src/directory-pages-cursor/types.ts new file mode 100644 index 00000000..3d036485 --- /dev/null +++ b/packages/@nitpicker/query/src/directory-pages-cursor/types.ts @@ -0,0 +1,17 @@ +/** + * Decoded shape of an opaque `/api/directory-tree/pages` cursor. Scoped to + * the endpoint's single fixed sort (`page_url_sort_key` ascending, `page_id` + * tie-breaker) — unlike `viewer-pages-cursor`'s `ViewerPagesCursorPayload`, + * there is no `sortBy`/`sortOrder`/`filterKey` to validate, since this + * endpoint supports none of those axes. + */ +export interface DirectoryPagesCursorPayload { + /** The read-model schema version this cursor was minted under — see `VIEWER_READ_MODEL_SCHEMA_VERSION`. */ + v: number; + /** The `nodeId` this cursor was minted for. A cursor minted for one directory must never be replayed against another. */ + nodeId: number; + /** The boundary row's `page_url_sort_key`. */ + pageUrlSortKey: string; + /** The boundary row's `page_id`. */ + pageId: number; +} diff --git a/packages/@nitpicker/query/src/get-directory-tree.spec.ts b/packages/@nitpicker/query/src/get-directory-tree.spec.ts new file mode 100644 index 00000000..1aeedf3f --- /dev/null +++ b/packages/@nitpicker/query/src/get-directory-tree.spec.ts @@ -0,0 +1,159 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { getDirectoryTree } from './get-directory-tree.js'; +import { buildViewerReadModel } from './viewer-read-model/build-viewer-read-model.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +describe('getDirectoryTree', () => { + describe('no read model built', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_get_directory_tree_no_model__', + ); + const archiveFilePath = path.resolve(workingDir, 'no-model-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns an empty array rather than throwing', async () => { + await expect(getDirectoryTree(archive)).resolves.toEqual([]); + }); + }); + + describe('populated tree', () => { + const workingDir = path.resolve(__dirname, '__test_fixtures_get_directory_tree__'); + const archiveFilePath = path.resolve(workingDir, 'get-tree-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + for (const url of [ + 'https://example.com/', + 'https://example.com/blog/2024/post-1', + // 4 levels deep — must be EXCLUDED from getDirectoryTree's + // depth<=3 initial load (still exists in the read model itself, + // per build-directory-tree-rows.spec.ts / build-viewer-read-model.spec.ts). + 'https://example.com/a/b/c/d/page', + ]) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns exactly one root for the single qualifying host', async () => { + const roots = await getDirectoryTree(archive); + expect(roots).toHaveLength(1); + expect(roots[0]?.rootKey).toBe('example.com'); + }); + + it('only includes nodes with depth <= 3, ordered by path', async () => { + const roots = await getDirectoryTree(archive); + const [root] = roots; + expect(root?.nodes.every((n) => n.depth <= 3)).toBe(true); + expect(root?.nodes.some((n) => n.path === '/a/b/c/')).toBe(true); + expect(root?.nodes.some((n) => n.path === '/a/b/c/d/')).toBe(false); + const paths = root?.nodes.map((n) => n.path) ?? []; + expect(paths).toEqual(paths.toSorted()); + }); + + it('exposes childCount as directChildDirCount + directPageCount, so callers never need to derive it themselves', async () => { + const roots = await getDirectoryTree(archive); + const root = roots[0]?.nodes.find((n) => n.path === '/'); + // Hardcoded against the fixture, not recomputed from the same + // formula the production code uses — root has 2 direct child dirs + // (blog, a) and 1 direct page (the root URL itself). + expect(root).toMatchObject({ + directChildDirCount: 2, + directPageCount: 1, + childCount: 3, + hasChildren: true, + }); + }); + }); +}); diff --git a/packages/@nitpicker/query/src/get-directory-tree.ts b/packages/@nitpicker/query/src/get-directory-tree.ts new file mode 100644 index 00000000..6423ecfa --- /dev/null +++ b/packages/@nitpicker/query/src/get-directory-tree.ts @@ -0,0 +1,131 @@ +import type { DirectoryTreeNode, DirectoryTreeRoot } from './types.js'; +import type { ArchiveAccessor } from '@nitpicker/crawler'; + +import { isViewerReadModelCurrent } from './viewer-read-model/is-viewer-read-model-current.js'; + +/** Maximum depth included in the initial tree load — see `docs/viewer-sql-query-plan.md`'s `/api/directory-tree`. */ +const INITIAL_DEPTH = 3; + +/** Row shape read from `viewer_directory_nodes` by {@link getDirectoryTree}. */ +interface DirectoryNodeRow { + /** `viewer_directory_nodes.node_id`. */ + node_id: number; + /** `viewer_directory_nodes.parent_node_id`. */ + parent_node_id: number | null; + /** `viewer_directory_nodes.root_key`. */ + root_key: string; + /** `viewer_directory_nodes.name`. */ + name: string; + /** `viewer_directory_nodes.path`. */ + path: string; + /** `viewer_directory_nodes.depth`. */ + depth: number; + /** `viewer_directory_nodes.direct_child_dir_count`. */ + direct_child_dir_count: number; + /** `viewer_directory_nodes.direct_page_count`. */ + direct_page_count: number; + /** `direct_child_dir_count + direct_page_count`, computed in SQL. */ + child_count: number; + /** `viewer_directory_nodes.descendant_page_count`. */ + descendant_page_count: number; + /** `viewer_directory_nodes.internal_descendant_page_count`. */ + internal_descendant_page_count: number; + /** `viewer_directory_nodes.external_descendant_page_count`. */ + external_descendant_page_count: number; + /** `viewer_directory_nodes.has_children`. */ + has_children: number; +} + +/** + * Maps one {@link DirectoryNodeRow} to its public {@link DirectoryTreeNode} shape. + * @param row - The source row. + * @returns The mapped node. + */ +function toDirectoryTreeNode(row: DirectoryNodeRow): DirectoryTreeNode { + return { + nodeId: row.node_id, + parentNodeId: row.parent_node_id, + name: row.name, + path: row.path, + depth: row.depth, + directChildDirCount: row.direct_child_dir_count, + directPageCount: row.direct_page_count, + childCount: row.child_count, + descendantPageCount: row.descendant_page_count, + internalDescendantPageCount: row.internal_descendant_page_count, + externalDescendantPageCount: row.external_descendant_page_count, + hasChildren: row.has_children === 1, + }; +} + +/** + * Reads the initial directory tree for every root (host) present in the + * archive's `viewer_directory_nodes` read model — depth ≤ 3 per host, flat + * (parent links only, no server-side nested tree construction), ordered by + * `path_sort_key`. Implements `docs/viewer-sql-query-plan.md`'s + * `/api/directory-tree` contract: one SELECT total (grouped by `root_key` in + * JS afterward), no recursive CTE, no GET-time counting. + * + * Takes no `rootKey` parameter by design — the archive already knows which + * hosts it crawled, so requiring a caller to guess or discover one first + * would be redundant. Callers needing a single host's tree just filter the + * returned array. + * @param accessor - The archive accessor to query. + * @returns One entry per distinct `root_key` with at least one node, each + * carrying its flat depth ≤ 3 node list. Returns `[]` when the read model + * has not been built, or has been built under a stale schema version (no + * legacy fallback exists for this feature, unlike `/api/pages` — + * `isViewerReadModelCurrent` is checked, not just `hasViewerReadModel`, + * so a v3-or-earlier read model — built before `viewer_directory_nodes` + * existed — returns `[]` instead of throwing a "no such table" error). + * @example + * // Render every root's initial tree; expand deeper nodes on demand via + * // listDirectoryChildren(accessor, { nodeId }). + * const roots = await getDirectoryTree(accessor); + * const primary = roots.find((r) => r.rootKey === 'example.com'); + */ +export async function getDirectoryTree( + accessor: ArchiveAccessor, +): Promise { + if (!(await isViewerReadModelCurrent(accessor))) { + return []; + } + + const knex = accessor.getKnex(); + const rows: DirectoryNodeRow[] = await knex('viewer_directory_nodes') + .where('depth', '<=', INITIAL_DEPTH) + .select( + 'node_id', + 'parent_node_id', + 'root_key', + 'name', + 'path', + 'depth', + 'direct_child_dir_count', + 'direct_page_count', + knex.raw('direct_child_dir_count + direct_page_count as child_count'), + 'descendant_page_count', + 'internal_descendant_page_count', + 'external_descendant_page_count', + 'has_children', + ) + // Deliberately `path_sort_key` alone, NOT `['root_key', 'path_sort_key']` + // — see `vdn_path_depth`'s comment in create-viewer-read-model-tables.ts + // for why leading the sort with `root_key` would defeat the index. A + // single global ascending sort by `path_sort_key` still leaves each + // root_key's own subsequence of rows correctly ordered by path once + // grouped below — sorting by one key and then filtering to a subset + // preserves that subset's relative order. + .orderBy('path_sort_key'); + + const rootsByKey = new Map(); + for (const row of rows) { + let root = rootsByKey.get(row.root_key); + if (!root) { + root = { rootKey: row.root_key, nodes: [] }; + rootsByKey.set(row.root_key, root); + } + root.nodes.push(toDirectoryTreeNode(row)); + } + return [...rootsByKey.values()]; +} diff --git a/packages/@nitpicker/query/src/list-directory-children.spec.ts b/packages/@nitpicker/query/src/list-directory-children.spec.ts new file mode 100644 index 00000000..0115a214 --- /dev/null +++ b/packages/@nitpicker/query/src/list-directory-children.spec.ts @@ -0,0 +1,162 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { getDirectoryTree } from './get-directory-tree.js'; +import { listDirectoryChildren } from './list-directory-children.js'; +import { buildViewerReadModel } from './viewer-read-model/build-viewer-read-model.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +describe('listDirectoryChildren', () => { + describe('no read model built', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_directory_children_no_model__', + ); + const archiveFilePath = path.resolve(workingDir, 'no-model-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns an empty array rather than throwing', async () => { + await expect(listDirectoryChildren(archive, { nodeId: 1 })).resolves.toEqual([]); + }); + }); + + describe('populated tree', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_directory_children__', + ); + const archiveFilePath = path.resolve(workingDir, 'children-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + for (const url of [ + 'https://example.com/', + 'https://example.com/a/b/c/d/page', + 'https://example.com/blog/2024/post-1', + ]) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it("returns the depth-4 node beyond getDirectoryTree's initial depth<=3 cutoff", async () => { + const roots = await getDirectoryTree(archive); + const depth3 = roots[0]?.nodes.find((n) => n.path === '/a/b/c/'); + expect(depth3).toBeDefined(); + expect(roots[0]?.nodes.some((n) => n.path === '/a/b/c/d/')).toBe(false); + + const children = await listDirectoryChildren(archive, { nodeId: depth3!.nodeId }); + expect(children).toHaveLength(1); + expect(children[0]).toMatchObject({ path: '/a/b/c/d/', depth: 4 }); + }); + + it('orders children by name and returns [] for a leaf with no child directories', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const nodes = firstRoot!.nodes; + const blog = nodes.find((n) => n.path === '/blog/2024/')!; + expect(await listDirectoryChildren(archive, { nodeId: blog.nodeId })).toEqual([]); + + const root = nodes.find((n) => n.path === '/')!; + const rootChildren = await listDirectoryChildren(archive, { nodeId: root.nodeId }); + expect(rootChildren.map((c) => c.name)).toEqual(['a', 'blog']); + }); + + it('truncates to the given limit', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const root = firstRoot!.nodes.find((n) => n.path === '/')!; + const limited = await listDirectoryChildren(archive, { + nodeId: root.nodeId, + limit: 1, + }); + expect(limited).toHaveLength(1); + }); + }); +}); diff --git a/packages/@nitpicker/query/src/list-directory-children.ts b/packages/@nitpicker/query/src/list-directory-children.ts new file mode 100644 index 00000000..838f262e --- /dev/null +++ b/packages/@nitpicker/query/src/list-directory-children.ts @@ -0,0 +1,109 @@ +import type { DirectoryTreeNode, ListDirectoryChildrenOptions } from './types.js'; +import type { ArchiveAccessor } from '@nitpicker/crawler'; + +import { isViewerReadModelCurrent } from './viewer-read-model/is-viewer-read-model-current.js'; + +/** Default/defensive cap on returned rows — see {@link ListDirectoryChildrenOptions.limit}. */ +const DEFAULT_LIMIT = 1000; + +/** Row shape read from `viewer_directory_nodes` by {@link listDirectoryChildren}. */ +interface DirectoryNodeRow { + /** `viewer_directory_nodes.node_id`. */ + node_id: number; + /** `viewer_directory_nodes.parent_node_id`. */ + parent_node_id: number | null; + /** `viewer_directory_nodes.name`. */ + name: string; + /** `viewer_directory_nodes.path`. */ + path: string; + /** `viewer_directory_nodes.depth`. */ + depth: number; + /** `viewer_directory_nodes.direct_child_dir_count`. */ + direct_child_dir_count: number; + /** `viewer_directory_nodes.direct_page_count`. */ + direct_page_count: number; + /** `direct_child_dir_count + direct_page_count`, computed in SQL. */ + child_count: number; + /** `viewer_directory_nodes.descendant_page_count`. */ + descendant_page_count: number; + /** `viewer_directory_nodes.internal_descendant_page_count`. */ + internal_descendant_page_count: number; + /** `viewer_directory_nodes.external_descendant_page_count`. */ + external_descendant_page_count: number; + /** `viewer_directory_nodes.has_children`. */ + has_children: number; +} + +/** + * Maps one {@link DirectoryNodeRow} to its public {@link DirectoryTreeNode} shape. + * @param row - The source row. + * @returns The mapped node. + */ +function toDirectoryTreeNode(row: DirectoryNodeRow): DirectoryTreeNode { + return { + nodeId: row.node_id, + parentNodeId: row.parent_node_id, + name: row.name, + path: row.path, + depth: row.depth, + directChildDirCount: row.direct_child_dir_count, + directPageCount: row.direct_page_count, + childCount: row.child_count, + descendantPageCount: row.descendant_page_count, + internalDescendantPageCount: row.internal_descendant_page_count, + externalDescendantPageCount: row.external_descendant_page_count, + hasChildren: row.has_children === 1, + }; +} + +/** + * Reads the direct child directory nodes of one node — a single SELECT keyed + * by `parent_node_id`, no recursive CTE. Implements + * `docs/viewer-sql-query-plan.md`'s `/api/directory-tree/children` contract, + * used to expand directories beyond the initial depth ≤ 3 load returned by + * {@link getDirectoryTree}. + * + * No `rootKey` parameter is needed — a `nodeId` already uniquely identifies + * which host's tree it belongs to. + * @param accessor - The archive accessor to query. + * @param options - See {@link ListDirectoryChildrenOptions}. + * @returns The direct child directory nodes, ordered by name. Returns `[]` + * when the read model has not been built or is stale (see + * `getDirectoryTree`'s docs on why `isViewerReadModelCurrent`, not just + * `hasViewerReadModel`, guards this), or when `nodeId` has no children. + * @example + * // User clicked a collapsed node in the tree UI: + * if (node.hasChildren) { + * const children = await listDirectoryChildren(accessor, { nodeId: node.nodeId }); + * } + */ +export async function listDirectoryChildren( + accessor: ArchiveAccessor, + options: ListDirectoryChildrenOptions, +): Promise { + if (!(await isViewerReadModelCurrent(accessor))) { + return []; + } + + const knex = accessor.getKnex(); + const rows: DirectoryNodeRow[] = await knex('viewer_directory_nodes') + .where('parent_node_id', options.nodeId) + .select( + 'node_id', + 'parent_node_id', + 'name', + 'path', + 'depth', + 'direct_child_dir_count', + 'direct_page_count', + knex.raw('direct_child_dir_count + direct_page_count as child_count'), + 'descendant_page_count', + 'internal_descendant_page_count', + 'external_descendant_page_count', + 'has_children', + ) + .orderBy(['name_sort_key', 'node_id']) + .limit(options.limit ?? DEFAULT_LIMIT); + + return rows.map(toDirectoryTreeNode); +} diff --git a/packages/@nitpicker/query/src/list-directory-pages.spec.ts b/packages/@nitpicker/query/src/list-directory-pages.spec.ts new file mode 100644 index 00000000..ce8cc427 --- /dev/null +++ b/packages/@nitpicker/query/src/list-directory-pages.spec.ts @@ -0,0 +1,219 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { getDirectoryTree } from './get-directory-tree.js'; +import { listDirectoryPages } from './list-directory-pages.js'; +import { buildViewerReadModel } from './viewer-read-model/build-viewer-read-model.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +/** + * Exhausts `/api/directory-tree/pages`-equivalent cursor pagination for one + * `nodeId`, following only `nextCursor`, and returns every item collected — + * mirrors `register-pages-route.spec.ts`'s `paginateAllViaNextCursor` helper + * but against `listDirectoryPages` directly rather than over HTTP. + * @param archive - The archive to query. + * @param nodeId - The directory node to paginate. + * @param limit - Page size per call. + * @param maxPages - Safety cap on iterations, to fail fast instead of + * looping forever if `nextCursor` never becomes `null`. + * @returns Every item across every page, in order. + */ +async function paginateAllDirectoryPages( + archive: InstanceType, + nodeId: number, + limit: number, + maxPages = 10, +) { + const items: { url: string }[] = []; + let cursor: string | undefined; + for (let i = 0; i < maxPages; i++) { + const result = await listDirectoryPages(archive, { nodeId, cursor, limit }); + items.push(...result.items); + if (!result.nextCursor) { + return items; + } + cursor = result.nextCursor; + } + throw new Error(`paginateAllDirectoryPages: exceeded maxPages (${maxPages})`); +} + +describe('listDirectoryPages', () => { + describe('no read model built', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_list_directory_pages_no_model__', + ); + const archiveFilePath = path.resolve(workingDir, 'no-model-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns an empty, terminated result rather than throwing', async () => { + await expect(listDirectoryPages(archive, { nodeId: 1 })).resolves.toEqual({ + items: [], + nextCursor: null, + }); + }); + }); + + describe('populated directory', () => { + const workingDir = path.resolve(__dirname, '__test_fixtures_list_directory_pages__'); + const archiveFilePath = path.resolve(workingDir, 'pages-test.nitpicker'); + let archive: InstanceType; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + for (const url of [ + 'https://example.com/blog/post-a', + 'https://example.com/blog/post-b', + 'https://example.com/blog/post-c', + ]) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: { ...META, title: url }, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns an empty, terminated result for a node with zero direct pages (e.g. the root, which only has a /blog/ child dir)', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const root = firstRoot!.nodes.find((n) => n.path === '/')!; + await expect(listDirectoryPages(archive, { nodeId: root.nodeId })).resolves.toEqual( + { + items: [], + nextCursor: null, + }, + ); + }); + + it('returns all 3 direct pages in one call when limit >= total', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const blog = firstRoot!.nodes.find((n) => n.path === '/blog/')!; + const result = await listDirectoryPages(archive, { + nodeId: blog.nodeId, + limit: 100, + }); + expect(result.items.map((i) => i.url).toSorted()).toEqual( + [ + 'https://example.com/blog/post-a', + 'https://example.com/blog/post-b', + 'https://example.com/blog/post-c', + ].toSorted(), + ); + expect(result.nextCursor).toBeNull(); + }); + + it('paginates to exhaustion with limit=1, with no duplicates/gaps, ending in nextCursor: null', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const blog = firstRoot!.nodes.find((n) => n.path === '/blog/')!; + const items = await paginateAllDirectoryPages(archive, blog.nodeId, 1); + expect(items.map((i) => i.url).toSorted()).toEqual( + [ + 'https://example.com/blog/post-a', + 'https://example.com/blog/post-b', + 'https://example.com/blog/post-c', + ].toSorted(), + ); + }); + + it('rejects a cursor minted for a different nodeId', async () => { + const [firstRoot] = await getDirectoryTree(archive); + const nodes = firstRoot!.nodes; + const blog = nodes.find((n) => n.path === '/blog/')!; + const root = nodes.find((n) => n.path === '/')!; + const first = await listDirectoryPages(archive, { nodeId: blog.nodeId, limit: 1 }); + await expect( + listDirectoryPages(archive, { nodeId: root.nodeId, cursor: first.nextCursor! }), + ).rejects.toThrow(/does not match/); + }); + + it('rejects a malformed cursor', async () => { + await expect( + listDirectoryPages(archive, { nodeId: 1, cursor: '%%%not-base64%%%' }), + ).rejects.toThrow(/not decodable/); + }); + }); +}); diff --git a/packages/@nitpicker/query/src/list-directory-pages.ts b/packages/@nitpicker/query/src/list-directory-pages.ts new file mode 100644 index 00000000..5e36e3fa --- /dev/null +++ b/packages/@nitpicker/query/src/list-directory-pages.ts @@ -0,0 +1,139 @@ +import type { + CursorPaginatedDirectoryPageList, + DirectoryPageListItem, + ListDirectoryPagesOptions, +} from './types.js'; +import type { ArchiveAccessor } from '@nitpicker/crawler'; + +import { decodeDirectoryPagesCursor } from './directory-pages-cursor/decode-directory-pages-cursor.js'; +import { encodeDirectoryPagesCursor } from './directory-pages-cursor/encode-directory-pages-cursor.js'; +import { isViewerReadModelCurrent } from './viewer-read-model/is-viewer-read-model-current.js'; +import { VIEWER_READ_MODEL_SCHEMA_VERSION } from './viewer-read-model/viewer-read-model-schema-version.js'; + +/** Default page size — matches `listViewerPages`'s own default. */ +const DEFAULT_LIMIT = 100; + +/** One id-resolution row from `viewer_directory_pages`. */ +interface DirectoryPageIdRow { + /** `viewer_directory_pages.page_id`. */ + page_id: number; + /** `viewer_directory_pages.page_url_sort_key`. */ + page_url_sort_key: string; +} + +/** One display-join row from `viewer_pages`. */ +interface ViewerPageDisplayRow { + /** `viewer_pages.page_id`. */ + page_id: number; + /** `viewer_pages.url`. */ + url: string; + /** `viewer_pages.title`. */ + title: string | null; + /** `viewer_pages.status`. */ + status: number | null; + /** `viewer_pages.content_category`. */ + content_category: string; +} + +/** + * Joins an already ID-limited, already-ordered `page_id` window back to + * `viewer_pages` for display fields — limit-then-join, the same golden rule + * `joinViewerPageIdsToListItems` follows for `/api/pages`. + * @param accessor - The archive accessor to query. + * @param pageIds - The page IDs to fetch, already filtered/sorted/limited by + * the `viewer_directory_pages` id-resolution stage. + * @returns The corresponding {@link DirectoryPageListItem} rows, in `pageIds` order. + */ +async function joinDirectoryPageIdsToListItems( + accessor: ArchiveAccessor, + pageIds: number[], +): Promise { + if (pageIds.length === 0) { + return []; + } + const knex = accessor.getKnex(); + const rows: ViewerPageDisplayRow[] = await knex('viewer_pages') + .whereIn('page_id', pageIds) + .select('page_id', 'url', 'title', 'status', 'content_category'); + const rowsById = new Map(rows.map((row) => [row.page_id, row])); + return pageIds + .map((id) => rowsById.get(id)) + .filter((row): row is ViewerPageDisplayRow => row != null) + .map((row) => ({ + pageId: row.page_id, + url: row.url, + title: row.title, + status: row.status, + contentCategory: row.content_category, + })); +} + +/** + * Lists the pages attached directly to one directory node — NEVER its + * descendants further down the tree (`/api/directory-tree/pages` never + * returns subtree pages, per `docs/viewer-sql-query-plan.md`). Cursor-paginated + * on `viewer_directory_pages`'s one fixed sort (`page_url_sort_key` ascending, + * `page_id` tie-breaker) — forward-only, unlike `listViewerPages`'s + * bidirectional keyset, since this endpoint has no virtual-scroll-upward + * requirement. + * @param accessor - The archive accessor to query. + * @param options - See {@link ListDirectoryPagesOptions}. + * @returns Up to `limit` pages plus a `nextCursor` for continuation, or + * `null` once the last page has been reached. Returns an empty, terminated + * result when the read model has not been built or is stale (see + * `getDirectoryTree`'s docs on why `isViewerReadModelCurrent`, not just + * `hasViewerReadModel`, guards this). + * @throws {Error} If `options.cursor` is malformed, stale, or was minted for + * a different `nodeId`. + * @example + * // Virtual-scroll continuation — the caller only ever inspects nextCursor: + * const page1 = await listDirectoryPages(accessor, { nodeId, limit: 100 }); + * const page2 = page1.nextCursor + * ? await listDirectoryPages(accessor, { nodeId, limit: 100, cursor: page1.nextCursor }) + * : null; + */ +export async function listDirectoryPages( + accessor: ArchiveAccessor, + options: ListDirectoryPagesOptions, +): Promise { + if (!(await isViewerReadModelCurrent(accessor))) { + return { items: [], nextCursor: null }; + } + + const { nodeId } = options; + const limit = options.limit ?? DEFAULT_LIMIT; + const knex = accessor.getKnex(); + + const qb = knex('viewer_directory_pages').where('node_id', nodeId); + if (options.cursor) { + const decoded = decodeDirectoryPagesCursor(options.cursor, nodeId); + qb.whereRaw('(page_url_sort_key, page_id) > (?, ?)', [ + decoded.pageUrlSortKey, + decoded.pageId, + ]); + } + const fetched: DirectoryPageIdRow[] = await qb + .select('page_id', 'page_url_sort_key') + .orderBy(['page_url_sort_key', 'page_id']) + .limit(limit + 1); + + const hasMoreAfter = fetched.length > limit; + const window = fetched.slice(0, limit); + const items = await joinDirectoryPageIdsToListItems( + accessor, + window.map((row) => row.page_id), + ); + + const lastRow = window.at(-1); + const nextCursor = + hasMoreAfter && lastRow + ? encodeDirectoryPagesCursor({ + v: VIEWER_READ_MODEL_SCHEMA_VERSION, + nodeId, + pageUrlSortKey: lastRow.page_url_sort_key, + pageId: lastRow.page_id, + }) + : null; + + return { items, nextCursor }; +} diff --git a/packages/@nitpicker/query/src/query.ts b/packages/@nitpicker/query/src/query.ts index 2d992ca4..74b4adcd 100644 --- a/packages/@nitpicker/query/src/query.ts +++ b/packages/@nitpicker/query/src/query.ts @@ -21,6 +21,7 @@ export { dropViewerReadModel } from './viewer-read-model/drop-viewer-read-model. export { ensureViewerReadModel } from './viewer-read-model/ensure-viewer-read-model.js'; export { findDuplicates } from './find-duplicates.js'; export { findMismatches } from './find-mismatches.js'; +export { getDirectoryTree } from './get-directory-tree.js'; export { getErrorKinds } from './get-error-kinds.js'; export { getLinkGraph } from './get-link-graph.js'; export { getPageDetail } from './get-page-detail.js'; @@ -36,6 +37,8 @@ export { getViewerReadModelVersion } from './viewer-read-model/get-viewer-read-m export { getViolations } from './get-violations.js'; export { hasViewerReadModel } from './viewer-read-model/has-viewer-read-model.js'; export { isViewerReadModelCurrent } from './viewer-read-model/is-viewer-read-model-current.js'; +export { listDirectoryChildren } from './list-directory-children.js'; +export { listDirectoryPages } from './list-directory-pages.js'; export { listImages } from './list-images.js'; export { listInventoryRuns } from './list-inventory-runs.js'; export { listIsolatedClusters } from './list-isolated-clusters.js'; diff --git a/packages/@nitpicker/query/src/types.ts b/packages/@nitpicker/query/src/types.ts index 9eac2704..80456968 100644 --- a/packages/@nitpicker/query/src/types.ts +++ b/packages/@nitpicker/query/src/types.ts @@ -1533,3 +1533,103 @@ export interface EnsureViewerReadModelOpportunisticallyOptions extends BuildView */ onWarn?: (message: string) => void; } + +/** + * One flat node in a directory tree, as returned by {@link getDirectoryTree} + * and {@link listDirectoryChildren}. `parentNodeId` is the only structural + * link — callers reconstruct the nested UI tree client-side from this flat + * list, since neither endpoint recurses server-side. + */ +export interface DirectoryTreeNode { + /** This node's unique id — stable across `getDirectoryTree`/`listDirectoryChildren` calls. */ + nodeId: number; + /** The parent directory's `nodeId`, or `null` for a host's root node. */ + parentNodeId: number | null; + /** This node's own path segment (e.g. `"2024"` for `/blog/2024/`), or `''` for a host's root node. */ + name: string; + /** This node's full path from the root (e.g. `/blog/2024/`, or `/` for a root node). */ + path: string; + /** This node's depth — a host's root node is `0`, incrementing by 1 per path segment. */ + depth: number; + /** Count of immediate child directory nodes (not pages) under this node. */ + directChildDirCount: number; + /** Count of pages attached directly to this node. */ + directPageCount: number; + /** + * `directChildDirCount + directPageCount`. The two addends are + * precomputed at read-model build time; the sum itself is a trivial O(1) + * SQL addition over those two already-fetched columns at query time — + * never a `COUNT`/`GROUP BY` scan over `viewer_directory_nodes` or + * `viewer_directory_pages`. + */ + childCount: number; + /** Total pages in this node's entire subtree, including its own `directPageCount`. */ + descendantPageCount: number; + /** Subset of `descendantPageCount` that is internal (in-scope). */ + internalDescendantPageCount: number; + /** Subset of `descendantPageCount` that is external (out-of-scope). */ + externalDescendantPageCount: number; + /** + * `true` iff `directChildDirCount > 0` — whether this node has child + * directories to expand via `listDirectoryChildren`/ + * `/api/directory-tree/children`. Deliberately excludes `directPageCount` + * — direct pages surface via the separate `/api/directory-tree/pages` + * panel, not as additional expandable tree rows. + */ + hasChildren: boolean; +} + +/** One host's worth of initial (depth ≤ 3) directory-tree nodes — see {@link getDirectoryTree}. */ +export interface DirectoryTreeRoot { + /** The host (hostname:port) this tree belongs to. */ + rootKey: string; + /** This root's flat, depth ≤ 3 node list, ordered by path. */ + nodes: DirectoryTreeNode[]; +} + +/** Options for {@link listDirectoryChildren}. */ +export interface ListDirectoryChildrenOptions { + /** The parent node whose direct child directories to list. */ + nodeId: number; + /** Defensive cap on returned rows. Defaults to 1000 — not a pagination contract, a directory realistically never has more child directories than this. */ + limit?: number; +} + +/** One directory-tree page-list entry, joined against `viewer_pages` for display — see {@link listDirectoryPages}. */ +export interface DirectoryPageListItem { + /** The page's id. */ + pageId: number; + /** The page's absolute URL. */ + url: string; + /** The page's `` text, or `null` when absent. */ + title: string | null; + /** HTTP status code, or `null` for not-yet-classified/errored rows. */ + status: number | null; + /** The page's {@link ContentTypeCategory}. */ + contentCategory: string; +} + +/** Options for {@link listDirectoryPages}. */ +export interface ListDirectoryPagesOptions { + /** The directory node whose direct pages to list (never its descendants). */ + nodeId: number; + /** + * Opaque cursor from a previous {@link CursorPaginatedDirectoryPageList}'s + * `nextCursor`. Omit for the first page. + */ + cursor?: string; + /** Maximum number of results to return. Defaults to 100. */ + limit?: number; +} + +/** + * Cursor-paginated result of {@link listDirectoryPages}. Forward-only (no + * `prevCursor`) — unlike {@link CursorPaginatedPageList}, this endpoint has + * no virtual-scroll-upward requirement to support. + */ +export interface CursorPaginatedDirectoryPageList { + /** The page-list items for this page. */ + items: DirectoryPageListItem[]; + /** Opaque cursor to fetch the next page, or `null` when this is the last page. */ + nextCursor: string | null; +} diff --git a/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.spec.ts b/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.spec.ts new file mode 100644 index 00000000..d899cea0 --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.spec.ts @@ -0,0 +1,228 @@ +import type { DirectoryTreeSourceRow } from './types.js'; + +import { describe, expect, it } from 'vitest'; + +import { buildDirectoryTreeRows } from './build-directory-tree-rows.js'; + +/** + * Shorthand for a {@link DirectoryTreeSourceRow} fixture row — `isExternal` + * defaults to `0` (internal) when omitted. + * @param id - The row's `pages.id`. + * @param url - The row's URL. + * @param isExternal - Optional `isExternal` override. + * @returns The fixture row. + */ +function row( + id: number, + url: string, + isExternal: number | null = 0, +): DirectoryTreeSourceRow { + return { id, url, isExternal }; +} + +describe('buildDirectoryTreeRows', () => { + it('returns no nodes and no pages for an empty input', () => { + expect(buildDirectoryTreeRows([])).toEqual({ nodes: [], pages: [] }); + }); + + it('creates exactly one depth-0 root node for a single root-only page', () => { + const { nodes, pages } = buildDirectoryTreeRows([row(1, 'https://example.com/')]); + expect(nodes).toHaveLength(1); + expect(nodes[0]).toMatchObject({ + parent_node_id: null, + root_key: 'example.com', + depth: 0, + name: '', + path: '/', + direct_child_dir_count: 0, + direct_page_count: 1, + descendant_page_count: 1, + internal_descendant_page_count: 1, + external_descendant_page_count: 0, + // No child DIRECTORIES (only a direct page) — has_children reflects + // expandability via listDirectoryChildren, not "has anything at all". + has_children: 0, + }); + expect(pages).toEqual([ + { + node_id: nodes[0].node_id, + page_id: 1, + page_url_sort_key: 'https://example.com/', + }, + ]); + }); + + it('lands a no-trailing-slash page and a trailing-slash page on the same directory node', () => { + const { nodes, pages } = buildDirectoryTreeRows([ + row(1, 'https://example.com/blog/2024/post-1'), + row(2, 'https://example.com/blog/2024/'), + ]); + const leaf = nodes.find((n) => n.path === '/blog/2024/'); + expect(leaf).toMatchObject({ depth: 2, direct_page_count: 2 }); + expect(pages.filter((p) => p.node_id === leaf?.node_id)).toHaveLength(2); + }); + + it('creates intermediate directories with zero direct pages of their own', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/a/b/c/d/page'), + ]); + const byPath = new Map(nodes.map((n) => [n.path, n])); + expect(byPath.get('/')).toMatchObject({ + depth: 0, + direct_page_count: 0, + direct_child_dir_count: 1, + }); + expect(byPath.get('/a/')).toMatchObject({ + depth: 1, + direct_page_count: 0, + direct_child_dir_count: 1, + }); + expect(byPath.get('/a/b/')).toMatchObject({ + depth: 2, + direct_page_count: 0, + direct_child_dir_count: 1, + }); + expect(byPath.get('/a/b/c/')).toMatchObject({ + depth: 3, + direct_page_count: 0, + direct_child_dir_count: 1, + }); + expect(byPath.get('/a/b/c/d/')).toMatchObject({ + depth: 4, + direct_page_count: 1, + direct_child_dir_count: 0, + }); + }); + + it('propagates descendant counts bottom-up through every ancestor', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/blog/2024/post-1'), + row(2, 'https://example.com/blog/2024/post-2'), + row(3, 'https://example.com/blog/'), + ]); + const byPath = new Map(nodes.map((n) => [n.path, n])); + expect(byPath.get('/blog/2024/')).toMatchObject({ + direct_page_count: 2, + descendant_page_count: 2, + }); + expect(byPath.get('/blog/')).toMatchObject({ + direct_page_count: 1, + descendant_page_count: 3, + }); + expect(byPath.get('/')).toMatchObject({ + direct_page_count: 0, + descendant_page_count: 3, + }); + }); + + it('splits descendant counts into internal/external, summing to descendant_page_count', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/'), + row(2, 'https://example.com/legacy/old.html', 1), + ]); + const root = nodes.find((n) => n.path === '/')!; + expect(root).toMatchObject({ + internal_descendant_page_count: 1, + external_descendant_page_count: 1, + descendant_page_count: 2, + }); + const legacy = nodes.find((n) => n.path === '/legacy/')!; + expect(legacy).toMatchObject({ + internal_descendant_page_count: 0, + external_descendant_page_count: 1, + descendant_page_count: 1, + }); + }); + + it('includes a same-host, out-of-scope (external) page in its host tree once the host qualifies', () => { + const { nodes, pages } = buildDirectoryTreeRows([ + row(1, 'https://example.com/'), + row(2, 'https://example.com/legacy/old.html', 1), + ]); + expect(nodes.some((n) => n.root_key === 'example.com' && n.path === '/legacy/')).toBe( + true, + ); + expect(pages.some((p) => p.page_id === 2)).toBe(true); + }); + + it('excludes a host with zero internal pages entirely — no nodes, no pages', () => { + const { nodes, pages } = buildDirectoryTreeRows([ + row(1, 'https://twitter.com/someaccount', 1), + ]); + expect(nodes).toEqual([]); + expect(pages).toEqual([]); + }); + + it('treats a null isExternal as internal (legacy pre-backfill rows)', () => { + const { nodes } = buildDirectoryTreeRows([row(1, 'https://example.com/', null)]); + expect(nodes.some((n) => n.root_key === 'example.com')).toBe(true); + const root = nodes.find((n) => n.path === '/')!; + expect(root.internal_descendant_page_count).toBe(1); + }); + + it('skips a row with an unparseable URL without throwing', () => { + expect(() => buildDirectoryTreeRows([row(1, 'not a valid url')])).not.toThrow(); + const { nodes, pages } = buildDirectoryTreeRows([row(1, 'not a valid url')]); + expect(nodes).toEqual([]); + expect(pages).toEqual([]); + }); + + it('builds two independent, non-colliding trees for two qualifying hosts', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/'), + row(2, 'https://example.org/'), + ]); + expect(nodes).toHaveLength(2); + const nodeIds = nodes.map((n) => n.node_id); + expect(new Set(nodeIds).size).toBe(nodeIds.length); + expect(new Set(nodes.map((n) => n.root_key))).toEqual( + new Set(['example.com', 'example.org']), + ); + }); + + it('merges two different subpaths of the same host into one tree as siblings (multi-root crawl)', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/blog/index'), + row(2, 'https://example.com/news/index'), + ]); + const roots = nodes.filter((n) => n.parent_node_id === null); + expect(roots).toHaveLength(1); + const blog = nodes.find((n) => n.path === '/blog/')!; + const news = nodes.find((n) => n.path === '/news/')!; + expect(blog.parent_node_id).toBe(roots[0].node_id); + expect(news.parent_node_id).toBe(roots[0].node_id); + expect(roots[0]).toMatchObject({ direct_child_dir_count: 2 }); + }); + + it('sets has_children to 0 for a leaf directory that has direct pages but no child directories', () => { + const { nodes } = buildDirectoryTreeRows([row(1, 'https://example.com/a/b')]); + // No trailing slash: 'b' is a page filename, so '/a/' is the leaf + // directory here — it has 1 direct page and 0 child directories. + const a = nodes.find((n) => n.path === '/a/')!; + expect(a).toMatchObject({ + direct_child_dir_count: 0, + direct_page_count: 1, + has_children: 0, + }); + }); + + it('sets has_children to 1 for a directory that has a child directory, even with zero direct pages of its own', () => { + const { nodes } = buildDirectoryTreeRows([row(1, 'https://example.com/a/b/c')]); + const a = nodes.find((n) => n.path === '/a/')!; + expect(a).toMatchObject({ + direct_child_dir_count: 1, + direct_page_count: 0, + has_children: 1, + }); + }); + + it('ignores query strings and hashes when resolving the directory chain', () => { + const { nodes } = buildDirectoryTreeRows([ + row(1, 'https://example.com/blog/2024/post-1?utm_source=x#section'), + ]); + expect(nodes.some((n) => n.path === '/blog/2024/')).toBe(true); + expect(nodes.find((n) => n.path === '/blog/2024/')).toMatchObject({ + direct_page_count: 1, + }); + }); +}); diff --git a/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.ts b/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.ts new file mode 100644 index 00000000..2a5d16bc --- /dev/null +++ b/packages/@nitpicker/query/src/viewer-read-model/build-directory-tree-rows.ts @@ -0,0 +1,311 @@ +import type { + DirectoryNodeInsertRow, + DirectoryPageInsertRow, + DirectoryTreeBuildResult, + DirectoryTreeSourceRow, +} from './types.js'; + +/** + * Structure-relevant fields extracted from one {@link DirectoryTreeSourceRow} + * by {@link parsePageRow}. + */ +interface ParsedPageRow { + /** The page URL's `host` (hostname:port). */ + host: string; + /** The directory segment chain this page's directory node lives at, root-to-leaf order. */ + dirSegments: string[]; + /** Normalised `0`/`1` form of the source row's `isExternal`. */ + isExternal: 0 | 1; + /** Copied from the source row's `id`. */ + id: number; + /** Copied from the source row's `url`. */ + url: string; +} + +/** + * Parses one source row's URL into its host and directory-segment chain, + * applying the trailing-slash boundary rule: a `pathname` NOT ending in `/` + * has its last segment treated as a page filename (the directory is formed + * from the preceding segments); a `pathname` ending in `/` (or empty/`/`) + * has EVERY segment form the directory chain, with the page attaching to + * that directory itself as an index page. Query strings and hashes never + * affect this — only `pathname` is consulted. + * @param row - The source row to parse. + * @returns The parsed row, or `null` when `row.url` fails `new URL()` + * parsing — such rows are skipped from the directory tree entirely (they + * remain ordinary `viewer_pages` rows, just outside this feature, mirroring + * `build-viewer-read-model.ts`'s `derivePathSortKey` catch-fallback + * precedent for legacy/malformed URLs — except here there is no + * substitute value, since a tree node requires a real host). + */ +function parsePageRow(row: DirectoryTreeSourceRow): ParsedPageRow | null { + let parsedUrl: URL; + try { + parsedUrl = new URL(row.url); + } catch { + return null; + } + const segments = parsedUrl.pathname.split('/').filter(Boolean); + const dirSegments = + parsedUrl.pathname.endsWith('/') || segments.length === 0 + ? segments + : segments.slice(0, -1); + return { + host: parsedUrl.host, + dirSegments, + isExternal: row.isExternal ? 1 : 0, + id: row.id, + url: row.url, + }; +} + +/** + * Creates a fresh directory node row with every count column zeroed — + * counts are filled in later by the page-attachment loop and + * {@link propagateDescendantCounts}. + * @param params - The node's identity/position fields. + * @param params.nodeId - The sequential id to assign. + * @param params.parentNodeId - The parent's `node_id`, or `null` for a root. + * @param params.rootKey - The host this node's tree belongs to. + * @param params.depth - The node's depth (root is `0`). + * @param params.name - The node's own path segment (`''` for the root). + * @param params.path - The node's full path from the root (`'/'` for the root). + * @returns A new, empty {@link DirectoryNodeInsertRow}. + */ +function createEmptyNode(params: { + nodeId: number; + parentNodeId: number | null; + rootKey: string; + depth: number; + name: string; + path: string; +}): DirectoryNodeInsertRow { + return { + node_id: params.nodeId, + parent_node_id: params.parentNodeId, + root_key: params.rootKey, + depth: params.depth, + name: params.name, + path: params.path, + name_sort_key: params.name, + path_sort_key: params.path, + direct_child_dir_count: 0, + direct_page_count: 0, + descendant_page_count: 0, + internal_descendant_page_count: 0, + external_descendant_page_count: 0, + has_children: 0, + }; +} + +/** + * Mutable working state threaded through directory-node creation while + * walking each row's segment chain. + */ +interface DirectoryTreeBuilderState { + /** The next unused sequential `node_id` to assign. */ + nextNodeId: number; + /** Every node created so far, keyed by `` `${host}::${path}` `` for O(1) get-or-create. */ + nodesByKey: Map<string, DirectoryNodeInsertRow>; +} + +/** + * Builds the `nodesByKey` lookup key for one host/path pair. `::` is not a + * valid character in a URL host or an already-percent-encoded pathname + * segment, so it cannot collide across different `(host, path)` pairs. + * @param host - The host (root_key). + * @param path - The node's full path. + * @returns The composite lookup key. + */ +function toNodeKey(host: string, path: string): string { + return `${host}::${path}`; +} + +/** + * Returns the existing depth-0 root node for `host` (representing its `/`), + * creating it first if this is the first row seen for that host. + * @param state - The shared builder state, mutated in place. + * @param host - The host (root_key) whose root node to fetch or create. + * @returns The host's root node (`path: '/'`, `depth: 0`). + */ +function getOrCreateRootNode( + state: DirectoryTreeBuilderState, + host: string, +): DirectoryNodeInsertRow { + const key = toNodeKey(host, '/'); + const existing = state.nodesByKey.get(key); + if (existing) { + return existing; + } + const root = createEmptyNode({ + nodeId: state.nextNodeId++, + parentNodeId: null, + rootKey: host, + depth: 0, + name: '', + path: '/', + }); + state.nodesByKey.set(key, root); + return root; +} + +/** + * Walks `dirSegments` from `host`'s root, creating any missing intermediate + * directory node along the way (standard filesystem-tree semantics — an + * ancestor directory exists even when it carries zero direct pages of its + * own), and returns the final (deepest) directory node the chain resolves + * to. + * @param state - The shared builder state, mutated in place. + * @param host - The host (root_key) this path belongs to. + * @param dirSegments - The directory segment chain, root-to-leaf order. + * @returns The directory node at the end of `dirSegments` — the root node + * itself when `dirSegments` is empty. + */ +function getOrCreateDirectoryNode( + state: DirectoryTreeBuilderState, + host: string, + dirSegments: readonly string[], +): DirectoryNodeInsertRow { + let current = getOrCreateRootNode(state, host); + let path = '/'; + for (const segment of dirSegments) { + path = `${path}${segment}/`; + const key = toNodeKey(host, path); + let node = state.nodesByKey.get(key); + if (!node) { + node = createEmptyNode({ + nodeId: state.nextNodeId++, + parentNodeId: current.node_id, + rootKey: host, + depth: current.depth + 1, + name: segment, + path, + }); + state.nodesByKey.set(key, node); + current.direct_child_dir_count += 1; + } + current = node; + } + return current; +} + +/** + * Folds each node's own (direct-only, as set by the page-attachment loop) + * `internal_descendant_page_count`/`external_descendant_page_count` up into + * its parent — processing nodes from deepest to shallowest so that, by the + * time a node is folded into ITS parent, it already holds its full subtree + * total (every descendant at every depth is added exactly once, since a + * node's children are always exactly one depth below it and are therefore + * always processed in an earlier iteration of this same pass). Also + * finalises `descendant_page_count` (the sum of the two) and `has_children` + * on every node. Mutates every row in `nodes` in place. + * + * `has_children` is deliberately `direct_child_dir_count > 0` alone, NOT + * `direct_child_dir_count + direct_page_count > 0` — every node this builder + * creates ends up with at least one direct page or one child directory (a + * node only exists because some page's path walk passed through or ended at + * it), so a "has ANY child or page" definition would always be `true` and + * carry no information. The viewer tree UI's expand arrow only ever reveals + * child DIRECTORIES via `/api/directory-tree/children` (direct pages come + * from the separate `/api/directory-tree/pages` panel, not additional tree + * rows), so `has_children` answers exactly the question that UI needs: does + * this node have anything to expand into. + * @param nodes - Every node across every host's tree. On entry, + * `internal_descendant_page_count`/`external_descendant_page_count` must + * hold only each node's OWN direct page counts. + */ +function propagateDescendantCounts(nodes: readonly DirectoryNodeInsertRow[]): void { + const byId = new Map(nodes.map((node) => [node.node_id, node])); + const byDepthDescending = nodes.toSorted((a, b) => b.depth - a.depth); + for (const node of byDepthDescending) { + if (node.parent_node_id === null) { + continue; + } + const parent = byId.get(node.parent_node_id); + if (!parent) { + continue; + } + parent.internal_descendant_page_count += node.internal_descendant_page_count; + parent.external_descendant_page_count += node.external_descendant_page_count; + } + for (const node of nodes) { + node.descendant_page_count = + node.internal_descendant_page_count + node.external_descendant_page_count; + node.has_children = node.direct_child_dir_count > 0 ? 1 : 0; + } +} + +/** + * Pure (no DB access) transform: given every listable page row, builds the + * full directory-tree forest — one root per eligible host — plus the + * direct-page memberships, computing every count column in-memory so the + * viewer's GET endpoints never split URLs or count children/descendants at + * request time (`docs/viewer-sql-query-plan.md`'s Golden Rules). + * + * A host is included in the output ONLY if at least one of its rows has + * `isExternal` falsy (an "internal" page) — hosts that exist purely as + * external link targets (e.g. a social-media profile linked from the site) + * are excluded entirely, since a directory tree of a domain the crawl never + * actually visited has no value. Once a host qualifies, BOTH its internal + * and external rows are included in that host's tree: crawl scope is a + * `(hostname, port, path)` triple (see `@nitpicker/crawler`'s + * `find-scope-entry.ts`), so a same-host, out-of-scope subpath is + * legitimately `isExternal = 1` while still belonging to the host's own + * tree — this is exactly why nodes track `internal_descendant_page_count` + * and `external_descendant_page_count` separately. + * + * Node ids are assigned sequentially (1-based, in tree-build order) by this + * function itself, so `parent_node_id` links are embedded in the returned + * plain objects before any DB insert happens — `buildViewerReadModel` bulk + * inserts the result with these explicit `node_id` values, which SQLite's + * `INTEGER PRIMARY KEY` accepts exactly like caller-supplied `rowid`s. + * + * Memory profile: the entire forest is built in memory before returning + * (no streaming/chunked variant) — the same tradeoff `buildViewerReadModel` + * already accepts for `viewer_pages`'s insert-row array, consistent with + * this read model's documented "minutes, not milliseconds" build-time + * budget (`docs/viewer-db-redesign-plan.md`). + * @param rows - Every listable page row — the same `sourceRows` array + * `buildViewerReadModel` already loads to populate `viewer_pages`. + * @returns The directory nodes and direct-page-membership rows to bulk-insert + * into `viewer_directory_nodes`/`viewer_directory_pages`. + */ +export function buildDirectoryTreeRows( + rows: readonly DirectoryTreeSourceRow[], +): DirectoryTreeBuildResult { + const parsedRows: ParsedPageRow[] = []; + for (const row of rows) { + const parsed = parsePageRow(row); + if (parsed) { + parsedRows.push(parsed); + } + } + + const internalHosts = new Set<string>(); + for (const row of parsedRows) { + if (row.isExternal === 0) { + internalHosts.add(row.host); + } + } + + const state: DirectoryTreeBuilderState = { nextNodeId: 1, nodesByKey: new Map() }; + const pages: DirectoryPageInsertRow[] = []; + for (const row of parsedRows) { + if (!internalHosts.has(row.host)) { + continue; + } + const node = getOrCreateDirectoryNode(state, row.host, row.dirSegments); + node.direct_page_count += 1; + if (row.isExternal === 0) { + node.internal_descendant_page_count += 1; + } else { + node.external_descendant_page_count += 1; + } + pages.push({ node_id: node.node_id, page_id: row.id, page_url_sort_key: row.url }); + } + + const nodes = [...state.nodesByKey.values()]; + propagateDescendantCounts(nodes); + + return { nodes, pages }; +} diff --git a/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.spec.ts b/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.spec.ts index b208e7d0..a6d63c93 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.spec.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.spec.ts @@ -702,4 +702,357 @@ describe('buildViewerReadModel', () => { expect(meta).toMatchObject({ source_row_count: 0 }); }); }); + + describe('directory tree population', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_build_read_model_directory_tree__', + ); + const archiveFilePath = path.resolve(workingDir, 'directory-tree-test.nitpicker'); + let archive: InstanceType<typeof Archive>; + + beforeAll(async () => { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + archive = await Archive.create({ filePath: archiveFilePath, cwd: workingDir }); + await archive.setConfig(BASE_CONFIG); + + // Root index page — attaches directly to the depth-0 root node. + await archive.setPage({ + url: parseUrl('https://example.com/')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // No trailing slash: attaches to directory /blog/2024/. + await archive.setPage({ + url: parseUrl('https://example.com/blog/2024/post-1')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // Trailing slash: same directory node /blog/2024/ as an index page — + // proves the boundary rule's second branch lands on the SAME node. + await archive.setPage({ + url: parseUrl('https://example.com/blog/2024/')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // Bare directory index at /blog/ itself. + await archive.setPage({ + url: parseUrl('https://example.com/blog/')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // Same host, out-of-scope subpath (isExternal: true) — must still + // belong to example.com's tree, counted as external. + await archive.setPage({ + url: parseUrl('https://example.com/legacy/old.html')!, + redirectPaths: [], + isExternal: true, + isTarget: false, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // A different same-host subtree (simulates a second multi-root crawl + // origin sharing the same host, e.g. `crawl .../blog/ .../news/`) — + // must become a sibling of /blog/, not a separate tree. + await archive.setPage({ + url: parseUrl('https://example.com/news/announcement')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // 4 levels deep — proves the build itself does not cap depth (only + // the /api/directory-tree endpoint caps its initial read at depth<=3). + await archive.setPage({ + url: parseUrl('https://example.com/a/b/c/d/page')!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // A host with ZERO internal pages — must be excluded from the + // directory tree entirely (no nodes, no pages), even though it + // remains a normal viewer_pages row. + await archive.setPage({ + url: parseUrl('https://twitter.com/someaccount')!, + redirectPaths: [], + isExternal: true, + isTarget: false, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + + // An unparseable URL (raw insert, bypassing setPage, same technique + // as the "legacy rows" describe block above) — must be skipped from + // the directory tree (no host to group by) while remaining a normal + // viewer_pages row. + await archive.getKnex()('pages').insert({ + url: 'not a valid url', + scraped: 1, + isTarget: 1, + isExternal: 0, + }); + + await buildViewerReadModel(archive); + }); + + afterAll(async () => { + if (archive) { + await archive.releaseHandle(); + } + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + /** + * Fetches one `viewer_directory_nodes` row by its `path` within + * `example.com`'s tree, throwing if it's missing — assertion helper + * only, not exported. + * @param path - The node's full path (e.g. `/blog/2024/`). + * @returns The matching row. + */ + async function getNodeByPath(path: string) { + const row = await archive + .getKnex()('viewer_directory_nodes') + .where({ root_key: 'example.com', path }) + .first(); + if (!row) { + throw new Error(`no viewer_directory_nodes row for path ${path}`); + } + return row; + } + + it('creates a root node (depth 0, path "/") with 4 direct child directories and 1 direct page', async () => { + const root = await getNodeByPath('/'); + expect(root).toMatchObject({ + depth: 0, + parent_node_id: null, + direct_child_dir_count: 4, + direct_page_count: 1, + has_children: 1, + }); + }); + + it("sums the whole tree's internal/external pages onto the root's descendant counts", async () => { + const root = await getNodeByPath('/'); + expect(root).toMatchObject({ + internal_descendant_page_count: 6, + external_descendant_page_count: 1, + descendant_page_count: 7, + }); + }); + + it('lands both the no-trailing-slash page and the trailing-slash index page on the same /blog/2024/ node', async () => { + const node = await getNodeByPath('/blog/2024/'); + expect(node).toMatchObject({ + depth: 2, + direct_child_dir_count: 0, + direct_page_count: 2, + internal_descendant_page_count: 2, + external_descendant_page_count: 0, + descendant_page_count: 2, + // No child DIRECTORIES (only 2 direct pages) — nothing to expand + // via listDirectoryChildren. + has_children: 0, + }); + const pages = await archive + .getKnex()('viewer_directory_pages') + .where('node_id', node.node_id) + .select('page_url_sort_key'); + expect(pages.map((p) => p.page_url_sort_key).toSorted()).toEqual( + [ + 'https://example.com/blog/2024/', + 'https://example.com/blog/2024/post-1', + ].toSorted(), + ); + }); + + it('folds descendant counts up through /blog/ (own index page + the /blog/2024/ subtree)', async () => { + const node = await getNodeByPath('/blog/'); + expect(node).toMatchObject({ + depth: 1, + direct_child_dir_count: 1, + direct_page_count: 1, + internal_descendant_page_count: 3, + external_descendant_page_count: 0, + descendant_page_count: 3, + }); + }); + + it('counts a same-host, out-of-scope page as external on its own directory node', async () => { + const node = await getNodeByPath('/legacy/'); + expect(node).toMatchObject({ + depth: 1, + direct_page_count: 1, + internal_descendant_page_count: 0, + external_descendant_page_count: 1, + descendant_page_count: 1, + }); + }); + + it('creates a sibling /news/ node under the same root rather than a separate tree', async () => { + const news = await getNodeByPath('/news/'); + const root = await getNodeByPath('/'); + expect(news).toMatchObject({ depth: 1, parent_node_id: root.node_id }); + }); + + it('creates every intermediate directory down to depth 4 with zero direct pages except the leaf', async () => { + const a = await getNodeByPath('/a/'); + const ab = await getNodeByPath('/a/b/'); + const abc = await getNodeByPath('/a/b/c/'); + const abcd = await getNodeByPath('/a/b/c/d/'); + expect(a).toMatchObject({ depth: 1, direct_page_count: 0, has_children: 1 }); + expect(ab).toMatchObject({ depth: 2, direct_page_count: 0, has_children: 1 }); + expect(abc).toMatchObject({ depth: 3, direct_page_count: 0, has_children: 1 }); + // The leaf has a direct page but no child directories of its own. + expect(abcd).toMatchObject({ depth: 4, direct_page_count: 1, has_children: 0 }); + }); + + it('excludes a host with zero internal pages entirely (no nodes, no pages)', async () => { + const knex = archive.getKnex(); + const nodes = await knex('viewer_directory_nodes') + .where('root_key', 'twitter.com') + .select('*'); + expect(nodes).toHaveLength(0); + }); + + it('skips a row with an unparseable URL from the directory tree without throwing', async () => { + const knex = archive.getKnex(); + const pages = await knex('viewer_pages') + .where('url', 'not a valid url') + .select('*'); + expect(pages).toHaveLength(1); + + const total = await knex('viewer_directory_pages').count<{ count: string }[]>({ + count: '*', + }); + // 7 attached pages: root, blog x1, blog/2024 x2, legacy, news, a/b/c/d — + // the unparseable-URL row and the twitter.com row contribute none. + expect(Number(total[0]?.count)).toBe(7); + }); + + it('rebuilds the directory tree idempotently — a second build leaves the same node/page counts and root counts', async () => { + const knex = archive.getKnex(); + const nodesBefore = await knex('viewer_directory_nodes').count<{ count: string }[]>( + { + count: '*', + }, + ); + const pagesBefore = await knex('viewer_directory_pages').count<{ count: string }[]>( + { + count: '*', + }, + ); + const rootBefore = await getNodeByPath('/'); + + await buildViewerReadModel(archive); + + const nodesAfter = await knex('viewer_directory_nodes').count<{ count: string }[]>({ + count: '*', + }); + const pagesAfter = await knex('viewer_directory_pages').count<{ count: string }[]>({ + count: '*', + }); + const rootAfter = await getNodeByPath('/'); + + expect(nodesAfter[0]?.count).toBe(nodesBefore[0]?.count); + expect(pagesAfter[0]?.count).toBe(pagesBefore[0]?.count); + expect(rootAfter).toMatchObject({ + direct_child_dir_count: rootBefore.direct_child_dir_count, + direct_page_count: rootBefore.direct_page_count, + descendant_page_count: rootBefore.descendant_page_count, + }); + // twitter.com must still be excluded after a rebuild, not accumulate + // duplicate/orphaned rows from the dropped-and-recreated tables. + expect( + await knex('viewer_directory_nodes').where('root_key', 'twitter.com').select('*'), + ).toEqual([]); + }); + }); }); diff --git a/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.ts b/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.ts index b5c4ffae..0bf5b4ab 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/build-viewer-read-model.ts @@ -4,6 +4,7 @@ import type { ArchiveAccessor } from '@nitpicker/crawler'; import { classifyContentType } from '../classify-content-type.js'; import { excludeSkippedPages } from '../exclude-skipped-pages.js'; +import { buildDirectoryTreeRows } from './build-directory-tree-rows.js'; import { computePageFacetBuckets } from './compute-page-facet-buckets.js'; import { createViewerReadModelTables } from './create-viewer-read-model-tables.js'; import { dropViewerReadModelTables } from './drop-viewer-read-model-tables.js'; @@ -199,14 +200,17 @@ function toViewerPageInsertRow(row: PagesSourceRow): ViewerPageInsertRow { } /** - * Performs a full rebuild of the viewer read model: drops all 5 tables if + * Performs a full rebuild of the viewer read model: drops all 7 tables if * present, recreates them, populates `viewer_pages` from the current - * `pages` write-model table, seeds one smoke-test row into - * `viewer_query_profiles`, writes the `viewer_count_buckets` totals row plus - * one row per distinct Pages-list facet value (see - * `computePageFacetBuckets`), and writes the `viewer_read_model_meta` row — - * all inside one transaction, so a mid-build failure leaves the previous - * read model (or no read model) intact, never a partially-built one. + * `pages` write-model table, populates `viewer_directory_nodes`/ + * `viewer_directory_pages` from that same page set (see + * `buildDirectoryTreeRows` for the tree-building rules), seeds one + * smoke-test row into `viewer_query_profiles`, writes the + * `viewer_count_buckets` totals row plus one row per distinct Pages-list + * facet value (see `computePageFacetBuckets`), and writes the + * `viewer_read_model_meta` row — all inside one transaction, so a mid-build + * failure leaves the previous read model (or no read model) intact, never a + * partially-built one. * * `viewer_pages` includes every listable page regardless of content-type * category (`scraped = 1 AND redirectDestId IS NULL`, plus excluding @@ -297,6 +301,24 @@ export async function buildViewerReadModel( onProgress?.({ insertedRows, totalRows }); } + // Reuses `sourceRows` (already loaded above for `viewer_pages`) instead + // of issuing a second `pages` SELECT — see `buildDirectoryTreeRows`'s + // docs for the tree-building rules (host eligibility, trailing-slash + // directory/page boundary, count propagation). + const { nodes: directoryNodes, pages: directoryPages } = buildDirectoryTreeRows( + sourceRows.map((row) => ({ id: row.id, url: row.url, isExternal: row.isExternal })), + ); + for (let start = 0; start < directoryNodes.length; start += INSERT_CHUNK_SIZE) { + await trx('viewer_directory_nodes').insert( + directoryNodes.slice(start, start + INSERT_CHUNK_SIZE), + ); + } + for (let start = 0; start < directoryPages.length; start += INSERT_CHUNK_SIZE) { + await trx('viewer_directory_pages').insert( + directoryPages.slice(start, start + INSERT_CHUNK_SIZE), + ); + } + const total = insertRows.length; await trx('viewer_query_profiles').insert({ scope: 'pages', diff --git a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.spec.ts b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.spec.ts index e3638797..f34505c9 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.spec.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.spec.ts @@ -30,7 +30,7 @@ describe('createViewerReadModelTables', () => { rmSync(workingDir, { recursive: true, force: true }); }); - it('creates all 5 tables and the named viewer_pages indexes', async () => { + it('creates all 7 tables and the named viewer_pages indexes', async () => { const knex = archive.getKnex(); await knex.transaction((trx) => createViewerReadModelTables(trx)); @@ -40,6 +40,8 @@ describe('createViewerReadModelTables', () => { 'viewer_query_profiles', 'viewer_count_buckets', 'viewer_page_anchors', + 'viewer_directory_nodes', + 'viewer_directory_pages', ]) { expect(await knex.schema.hasTable(table)).toBe(true); } @@ -85,4 +87,39 @@ describe('createViewerReadModelTables', () => { .orderBy('profile_key'); expect(rows.map((r) => r.profile_key)).toEqual(['a', 'b']); }); + + it('viewer_directory_nodes enforces a unique (root_key, path) constraint', async () => { + const knex = archive.getKnex(); + const baseNode = { + parent_node_id: null, + root_key: 'example.com', + depth: 0, + name: '', + path: '/', + name_sort_key: '', + path_sort_key: '/', + direct_child_dir_count: 0, + direct_page_count: 0, + descendant_page_count: 0, + internal_descendant_page_count: 0, + external_descendant_page_count: 0, + has_children: 0, + }; + await knex('viewer_directory_nodes').insert({ node_id: 1, ...baseNode }); + await expect( + knex('viewer_directory_nodes').insert({ node_id: 2, ...baseNode }), + ).rejects.toThrow(); + }); + + it('viewer_directory_pages enforces a composite (node_id, page_id) key, not a single-column rowid', async () => { + const knex = archive.getKnex(); + await knex('viewer_directory_pages').insert([ + { node_id: 1, page_id: 1, page_url_sort_key: 'https://example.com/a' }, + { node_id: 1, page_id: 2, page_url_sort_key: 'https://example.com/b' }, + ]); + const rows = await knex('viewer_directory_pages') + .select('page_id') + .orderBy('page_id'); + expect(rows.map((r) => r.page_id)).toEqual([1, 2]); + }); }); diff --git a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts index da7bad8e..872e8056 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/create-viewer-read-model-tables.ts @@ -1,19 +1,19 @@ import type { Knex } from 'knex'; /** - * Creates all 5 viewer-read-model tables (and `viewer_pages`'s named + * Creates all 7 viewer-read-model tables (and `viewer_pages`'s named * indexes) against the given connection. Assumes none of the tables * currently exist — callers (`buildViewerReadModel`) are responsible for * dropping any prior version first, inside the same transaction, so this * function is not itself idempotent. * * Every statement runs via `raw()` rather than knex's chainable schema - * builder: 4 of the 5 tables need `WITHOUT ROWID` / a composite primary key - * / a `CHECK` constraint, none of which the chainable builder can express - * (the same reason `page_html_blobs` / `page_html_ref` drop to `raw()` in - * `@nitpicker/crawler`'s `init-schema.ts`). Using `raw()` for the 5th table - * (`viewer_pages`) too keeps this function a single uniform style instead - * of mixing two schema-definition APIs. + * builder: 5 of the 7 tables need `WITHOUT ROWID` / a composite primary key + * / a `CHECK` constraint / a table-level `UNIQUE` constraint, none of which + * the chainable builder can express (the same reason `page_html_blobs` / + * `page_html_ref` drop to `raw()` in `@nitpicker/crawler`'s + * `init-schema.ts`). Using `raw()` for every table keeps this function a + * single uniform style instead of mixing two schema-definition APIs. * @param trx - An open Knex transaction (a plain `Knex` instance also works, * e.g. in tests that don't need transactional rollback). */ @@ -108,4 +108,53 @@ export async function createViewerReadModelTables(trx: Knex): Promise<void> { primary key(scope, profile_key, page_size, page_index) ) WITHOUT ROWID `); + + await trx.raw(` + CREATE TABLE viewer_directory_nodes ( + node_id integer primary key, + parent_node_id integer, + root_key text not null, + depth integer not null, + name text not null, + path text not null, + name_sort_key text not null, + path_sort_key text not null, + direct_child_dir_count integer not null, + direct_page_count integer not null, + descendant_page_count integer not null, + internal_descendant_page_count integer not null, + external_descendant_page_count integer not null, + has_children integer not null, + unique(root_key, path) + ) + `); + // `getDirectoryTree` filters `depth <= 3` (a range, not an equality) and + // orders by `path_sort_key` alone (grouping into per-root_key buckets + // happens in JS afterward — see that function's docs). Leading the index + // with `path_sort_key` lets SQLite satisfy the `ORDER BY` via a plain + // ascending index scan with `depth` checked as a cheap residual filter, + // instead of falling back to `USE TEMP B-TREE FOR ORDER BY`. Leading with + // `root_key`/`depth` instead (as `docs/viewer-sql-query-plan.md`'s + // aspirational single-root-per-request index does) would not help here: + // a range condition on a non-leading column can't be used to avoid a sort + // on a column that comes after it — the same index-column-order pitfall + // `idx_pages_listfilter` hit (see ARCHITECTURE.md). + await trx.raw( + 'CREATE INDEX vdn_path_depth ON viewer_directory_nodes(path_sort_key, depth, node_id)', + ); + await trx.raw( + 'CREATE INDEX vdn_parent_name ON viewer_directory_nodes(parent_node_id, name_sort_key, node_id)', + ); + + await trx.raw(` + CREATE TABLE viewer_directory_pages ( + node_id integer not null, + page_id integer not null, + page_url_sort_key text not null, + primary key(node_id, page_id) + ) WITHOUT ROWID + `); + await trx.raw( + 'CREATE INDEX vdp_node_url ON viewer_directory_pages(node_id, page_url_sort_key, page_id)', + ); } diff --git a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.spec.ts b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.spec.ts index 6cfb926b..1ebffad6 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.spec.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.spec.ts @@ -37,7 +37,7 @@ describe('dropViewerReadModelTables', () => { ).resolves.toBeUndefined(); }); - it('drops all 5 tables after they were created', async () => { + it('drops all 7 tables after they were created', async () => { const knex = archive.getKnex(); await knex.transaction((trx) => createViewerReadModelTables(trx)); for (const table of [ @@ -46,6 +46,8 @@ describe('dropViewerReadModelTables', () => { 'viewer_query_profiles', 'viewer_count_buckets', 'viewer_page_anchors', + 'viewer_directory_nodes', + 'viewer_directory_pages', ]) { expect(await knex.schema.hasTable(table)).toBe(true); } @@ -57,6 +59,8 @@ describe('dropViewerReadModelTables', () => { 'viewer_query_profiles', 'viewer_count_buckets', 'viewer_page_anchors', + 'viewer_directory_nodes', + 'viewer_directory_pages', ]) { expect(await knex.schema.hasTable(table)).toBe(false); } diff --git a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.ts b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.ts index 62ae2219..e53d5608 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model-tables.ts @@ -1,16 +1,18 @@ import type { Knex } from 'knex'; /** - * Drops all 5 viewer-read-model tables if present, against the given + * Drops all 7 viewer-read-model tables if present, against the given * connection. Shared by `buildViewerReadModel` (which drops before * recreating, inside its own rebuild transaction) and - * `dropViewerReadModel` (which drops with no recreate), so the 5-table + * `dropViewerReadModel` (which drops with no recreate), so the 7-table * list only needs to be kept in sync with `createViewerReadModelTables` * in one place. * @param trx - An open Knex transaction (a plain `Knex` instance also * works, e.g. in tests). */ export async function dropViewerReadModelTables(trx: Knex): Promise<void> { + await trx.schema.dropTableIfExists('viewer_directory_pages'); + await trx.schema.dropTableIfExists('viewer_directory_nodes'); await trx.schema.dropTableIfExists('viewer_page_anchors'); await trx.schema.dropTableIfExists('viewer_count_buckets'); await trx.schema.dropTableIfExists('viewer_query_profiles'); diff --git a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model.ts b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model.ts index b3b34863..0aed25a5 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/drop-viewer-read-model.ts @@ -3,9 +3,9 @@ import type { ArchiveAccessor } from '@nitpicker/crawler'; import { dropViewerReadModelTables } from './drop-viewer-read-model-tables.js'; /** - * Drops all 5 viewer-read-model tables if present. Idempotent — calling + * Drops all 7 viewer-read-model tables if present. Idempotent — calling * this on an archive with no read model is a no-op, not an error. Only - * ever touches the 5 `viewer_*` tables; the write-model tables (`pages`, + * ever touches the 7 `viewer_*` tables; the write-model tables (`pages`, * `anchors`, etc.) are never referenced. * @param accessor - The archive accessor to drop from. Must be writable * (`accessor.readOnly === false`). diff --git a/packages/@nitpicker/query/src/viewer-read-model/types.ts b/packages/@nitpicker/query/src/viewer-read-model/types.ts index d94b9e98..2ded6694 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/types.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/types.ts @@ -59,3 +59,101 @@ export interface FacetBucketRow { /** Number of `viewer_pages` rows in this build carrying `value` for this key. */ count: number; } + +/** + * Minimal shape `buildDirectoryTreeRows` needs from each listable `pages` + * row. A structural subset of `build-viewer-read-model.ts`'s private + * `PagesSourceRow` — the same `sourceRows` array already loaded for + * `viewer_pages` is reused here, so no second `pages` SELECT is issued. + */ +export interface DirectoryTreeSourceRow { + /** `pages.id` — becomes `viewer_directory_pages.page_id`. */ + id: number; + /** The page's absolute URL. */ + url: string; + /** + * `1`/`0` when known, `null` on legacy rows written before this column + * was backfilled — normalised the same way `toViewerPageInsertRow` does + * (`null` counts as internal). + */ + isExternal: number | null; +} + +/** + * One row to insert into `viewer_directory_nodes` — a single directory (or + * the depth-0 root standing for a host's `/`) within one host's directory + * tree, produced by `buildDirectoryTreeRows`. + */ +export interface DirectoryNodeInsertRow { + /** + * Sequential integer id assigned by `buildDirectoryTreeRows` itself, in + * tree-build order, and inserted verbatim as `viewer_directory_nodes`'s + * `INTEGER PRIMARY KEY` — SQLite accepts an explicit primary-key value on + * insert exactly like a `rowid`, so `parent_node_id` links can reference + * sibling rows in this same insert batch without a round-trip. + */ + node_id: number; + /** The parent directory's `node_id`, or `null` for the depth-0 root node of a host's tree. */ + parent_node_id: number | null; + /** The page URL's `host` (hostname:port) that this tree belongs to. */ + root_key: string; + /** `0` for the root (path `/`), incrementing by 1 per path segment. */ + depth: number; + /** This directory's own segment name (e.g. `"2024"` for `/blog/2024/`), or `''` for the depth-0 root. */ + name: string; + /** This directory's full path from the root, always starting and ending with `/` (e.g. `/blog/2024/`, or `/` for the root). */ + path: string; + /** + * Sort key for `name` — currently `name` verbatim. Kept as a separate + * column (rather than sorting on `name` directly) for the same reason + * `viewer_pages.url_sort_key`/`title_sort_key` are separate from their + * base columns: it is the index target, and future normalisation (e.g. + * case-folding) must not require an index rename. + */ + name_sort_key: string; + /** Sort key for `path` — currently `path` verbatim, see {@link DirectoryNodeInsertRow.name_sort_key}. */ + path_sort_key: string; + /** Count of immediate child directory nodes (not pages) under this node. */ + direct_child_dir_count: number; + /** Count of pages attached directly to this node (its own boundary/index pages). */ + direct_page_count: number; + /** Total pages in this node's entire subtree, including its own `direct_page_count`. */ + descendant_page_count: number; + /** Subset of `descendant_page_count` where the page's `isExternal` is falsy. */ + internal_descendant_page_count: number; + /** Subset of `descendant_page_count` where the page's `isExternal` is truthy. */ + external_descendant_page_count: number; + /** + * `1` iff `direct_child_dir_count > 0`, `0` otherwise — whether this node + * has child DIRECTORIES to expand via `/api/directory-tree/children`. + * Deliberately excludes `direct_page_count`: direct pages are a separate + * `/api/directory-tree/pages` panel, not additional tree rows, and every + * node this builder creates already has at least one page or child + * directory, so including `direct_page_count` here would make this + * always `1` (see `propagateDescendantCounts`'s docs in + * `build-directory-tree-rows.ts`). + */ + has_children: number; +} + +/** + * One row to insert into `viewer_directory_pages` — one page attached + * directly to a directory node (never a descendant further down the tree), + * produced by `buildDirectoryTreeRows`. + */ +export interface DirectoryPageInsertRow { + /** The owning directory's `node_id` — a {@link DirectoryNodeInsertRow.node_id}. */ + node_id: number; + /** The write-model `pages.id` this row represents. */ + page_id: number; + /** Sort key for this page's URL within its directory — currently the page's full URL verbatim. */ + page_url_sort_key: string; +} + +/** Return shape of `buildDirectoryTreeRows`. */ +export interface DirectoryTreeBuildResult { + /** Every directory node across every eligible host's tree. */ + nodes: DirectoryNodeInsertRow[]; + /** Every direct page-to-node membership row. */ + pages: DirectoryPageInsertRow[]; +} diff --git a/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts b/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts index e646f973..988f7b18 100644 --- a/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts +++ b/packages/@nitpicker/query/src/viewer-read-model/viewer-read-model-schema-version.ts @@ -7,4 +7,4 @@ * `viewer_read_model_meta.schema_version` to decide whether a rebuild is * needed. */ -export const VIEWER_READ_MODEL_SCHEMA_VERSION = 3; +export const VIEWER_READ_MODEL_SCHEMA_VERSION = 4; diff --git a/packages/@nitpicker/viewer/src/create-app.ts b/packages/@nitpicker/viewer/src/create-app.ts index 5567751c..9662a321 100644 --- a/packages/@nitpicker/viewer/src/create-app.ts +++ b/packages/@nitpicker/viewer/src/create-app.ts @@ -4,6 +4,9 @@ import { serveStatic } from '@hono/node-server/serve-static'; import { Hono } from 'hono'; import { registerArchiveInfoRoute } from './routes/register-archive-info-route.js'; +import { registerDirectoryTreeChildrenRoute } from './routes/register-directory-tree-children-route.js'; +import { registerDirectoryTreePagesRoute } from './routes/register-directory-tree-pages-route.js'; +import { registerDirectoryTreeRoute } from './routes/register-directory-tree-route.js'; import { registerDuplicatesRoute } from './routes/register-duplicates-route.js'; import { registerErrorKindsRoute } from './routes/register-error-kinds-route.js'; import { registerGraphRoute } from './routes/register-graph-route.js'; @@ -47,6 +50,9 @@ export function createApp(options: CreateAppOptions): Hono { registerPagesRoute(app, context); registerPageDetailRoute(app, context); registerPageHtmlRoute(app, context); + registerDirectoryTreeRoute(app, context); + registerDirectoryTreeChildrenRoute(app, context); + registerDirectoryTreePagesRoute(app, context); registerLinksRoute(app, context); registerResourcesRoute(app, context); registerResourceReferrersRoute(app, context); diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.spec.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.spec.ts new file mode 100644 index 00000000..31e6c091 --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.spec.ts @@ -0,0 +1,156 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { ArchiveManager, buildViewerReadModel, getDirectoryTree } from '@nitpicker/query'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { createApp } from '../create-app.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +/** + * Builds a fixture archive with a 4-deep directory chain and returns an + * in-process Hono app wired to it — mirrors + * `register-directory-tree-route.spec.ts`'s `buildFixture`. + * @param workingDir - Unique scratch directory for this fixture. + * @returns The app, archive, and manager — callers must close both in `afterAll`. + */ +async function buildFixture(workingDir: string) { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + const archive = await Archive.create({ + filePath: path.resolve(workingDir, 'fixture.nitpicker'), + cwd: workingDir, + }); + await archive.setConfig(BASE_CONFIG); + for (const url of ['https://example.com/', 'https://example.com/a/b/c/d/page']) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '<html></html>', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + await buildViewerReadModel(archive); + + const manager = new ArchiveManager(); + const { archiveId, mode } = await manager.open(archive.tmpDir); + const app = createApp({ + context: { + manager, + archiveId, + filePath: archive.tmpDir, + mode, + crawlerLockHolder: null, + }, + publicDir: '/tmp/no-such-dir-register-directory-tree-children-route-spec', + }); + return { app, archive, manager, archiveId }; +} + +describe('registerDirectoryTreeChildrenRoute (integration)', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_register_directory_tree_children_route__', + ); + let fixture: Awaited<ReturnType<typeof buildFixture>>; + + beforeAll(async () => { + fixture = await buildFixture(workingDir); + }); + + afterAll(async () => { + await fixture.manager.closeAll(); + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns the depth-4 node beyond the initial depth<=3 tree, via its depth-3 parent', async () => { + const accessor = fixture.manager.get(fixture.archiveId); + const roots = await getDirectoryTree(accessor); + const depth3 = roots[0]?.nodes.find((n) => n.path === '/a/b/c/'); + expect(depth3).toBeDefined(); + + const res = await fixture.app.request( + `/api/directory-tree/children?nodeId=${depth3!.nodeId}`, + ); + expect(res.status).toBe(200); + const body = (await res.json()) as { nodes: { path: string; depth: number }[] }; + expect(body.nodes).toEqual([ + expect.objectContaining({ path: '/a/b/c/d/', depth: 4 }), + ]); + }); + + it('returns 400 when nodeId is missing', async () => { + const res = await fixture.app.request('/api/directory-tree/children'); + expect(res.status).toBe(400); + expect(await res.json()).toEqual({ + error: 'Missing required query parameter: nodeId', + }); + }); + + it('returns an empty array for a node with no child directories', async () => { + const accessor = fixture.manager.get(fixture.archiveId); + const [firstRoot] = await getDirectoryTree(accessor); + const root = firstRoot!.nodes.find((n) => n.path === '/')!; + const res = await fixture.app.request( + `/api/directory-tree/children?nodeId=${root.nodeId}`, + ); + // root's only child is `/a/`, which DOES exist — assert the shape + // instead of emptiness here; emptiness is covered by a leaf node. + const body = (await res.json()) as { nodes: unknown[] }; + expect(Array.isArray(body.nodes)).toBe(true); + }); +}); diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.ts new file mode 100644 index 00000000..8377e105 --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-children-route.ts @@ -0,0 +1,28 @@ +import type { ArchiveContext } from '../types.js'; +import type { Hono } from 'hono'; + +import { listDirectoryChildren } from '@nitpicker/query'; + +import { toNumber } from '../query-params/to-number.js'; + +/** + * Registers `GET /api/directory-tree/children?nodeId=<id>` — the direct + * child directory nodes of one node, for on-demand expansion of directories + * beyond the initial depth ≤ 3 load returned by `/api/directory-tree`. + * @param app - The Hono application. + * @param context - The opened archive context. + */ +export function registerDirectoryTreeChildrenRoute( + app: Hono, + context: ArchiveContext, +): void { + app.get('/api/directory-tree/children', async (c) => { + const nodeId = toNumber(c.req.query('nodeId')); + if (nodeId == null) { + return c.json({ error: 'Missing required query parameter: nodeId' }, 400); + } + const accessor = context.manager.get(context.archiveId); + const nodes = await listDirectoryChildren(accessor, { nodeId }); + return c.json({ nodes }); + }); +} diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.spec.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.spec.ts new file mode 100644 index 00000000..21a77b06 --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.spec.ts @@ -0,0 +1,184 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { ArchiveManager, buildViewerReadModel, getDirectoryTree } from '@nitpicker/query'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { createApp } from '../create-app.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +/** + * Builds a fixture archive with 3 pages under `/blog/` and returns an + * in-process Hono app wired to it — mirrors + * `register-directory-tree-route.spec.ts`'s `buildFixture`. + * @param workingDir - Unique scratch directory for this fixture. + * @returns The app, archive, manager, and `archiveId` — callers must close + * the manager in `afterAll`. + */ +async function buildFixture(workingDir: string) { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + const archive = await Archive.create({ + filePath: path.resolve(workingDir, 'fixture.nitpicker'), + cwd: workingDir, + }); + await archive.setConfig(BASE_CONFIG); + for (const url of [ + 'https://example.com/blog/post-a', + 'https://example.com/blog/post-b', + 'https://example.com/blog/post-c', + ]) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '<html></html>', + meta: { ...META, title: url }, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + await buildViewerReadModel(archive); + + const manager = new ArchiveManager(); + const { archiveId, mode } = await manager.open(archive.tmpDir); + const app = createApp({ + context: { + manager, + archiveId, + filePath: archive.tmpDir, + mode, + crawlerLockHolder: null, + }, + publicDir: '/tmp/no-such-dir-register-directory-tree-pages-route-spec', + }); + return { app, archive, manager, archiveId }; +} + +/** + * Drives `/api/directory-tree/pages?nodeId=...` to exhaustion via + * `nextCursor` alone, collecting every item's URL in request order — mirrors + * `register-pages-route.spec.ts`'s `paginateAllViaNextCursor`. + * @param app - The in-process Hono app. + * @param nodeId - The directory node to paginate. + * @param limit - The page size. + * @param maxPages - Safety cap so a broken "never terminates" regression + * fails the test instead of hanging. + * @returns The concatenated URLs across every page, in order. + */ +async function paginateAllViaNextCursor( + app: ReturnType<typeof createApp>, + nodeId: number, + limit: number, + maxPages = 10, +): Promise<string[]> { + const urls: string[] = []; + let cursor: string | null = null; + for (let page = 0; page < maxPages; page++) { + const cursorParam = cursor ? `&cursor=${encodeURIComponent(cursor)}` : ''; + const res = await app.request( + `/api/directory-tree/pages?nodeId=${nodeId}&limit=${limit}${cursorParam}`, + ); + const body = (await res.json()) as { + items: { url: string }[]; + nextCursor: string | null; + }; + urls.push(...body.items.map((i) => i.url)); + if (!body.nextCursor) { + return urls; + } + cursor = body.nextCursor; + } + throw new Error(`paginateAllViaNextCursor: did not terminate within ${maxPages} pages`); +} + +describe('registerDirectoryTreePagesRoute (integration)', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_register_directory_tree_pages_route__', + ); + let fixture: Awaited<ReturnType<typeof buildFixture>>; + + beforeAll(async () => { + fixture = await buildFixture(workingDir); + }); + + afterAll(async () => { + await fixture.manager.closeAll(); + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('paginates to completion via nextCursor with limit=1, no duplicates or gaps', async () => { + const accessor = fixture.manager.get(fixture.archiveId); + const [firstRoot] = await getDirectoryTree(accessor); + const blog = firstRoot!.nodes.find((n) => n.path === '/blog/')!; + + const urls = await paginateAllViaNextCursor(fixture.app, blog.nodeId, 1); + expect(urls.toSorted()).toEqual( + [ + 'https://example.com/blog/post-a', + 'https://example.com/blog/post-b', + 'https://example.com/blog/post-c', + ].toSorted(), + ); + }); + + it('returns 400 when nodeId is missing', async () => { + const res = await fixture.app.request('/api/directory-tree/pages'); + expect(res.status).toBe(400); + expect(await res.json()).toEqual({ + error: 'Missing required query parameter: nodeId', + }); + }); +}); diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.ts new file mode 100644 index 00000000..26a3ab8c --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-pages-route.ts @@ -0,0 +1,32 @@ +import type { ArchiveContext } from '../types.js'; +import type { Hono } from 'hono'; + +import { listDirectoryPages } from '@nitpicker/query'; + +import { toNumber } from '../query-params/to-number.js'; + +/** + * Registers `GET /api/directory-tree/pages?nodeId=<id>&cursor=&limit=` — + * cursor-paginated direct pages of one directory node (never its + * descendants — see `listDirectoryPages`'s docs). + * @param app - The Hono application. + * @param context - The opened archive context. + */ +export function registerDirectoryTreePagesRoute( + app: Hono, + context: ArchiveContext, +): void { + app.get('/api/directory-tree/pages', async (c) => { + const nodeId = toNumber(c.req.query('nodeId')); + if (nodeId == null) { + return c.json({ error: 'Missing required query parameter: nodeId' }, 400); + } + const accessor = context.manager.get(context.archiveId); + const result = await listDirectoryPages(accessor, { + nodeId, + cursor: c.req.query('cursor') || undefined, + limit: toNumber(c.req.query('limit')), + }); + return c.json(result); + }); +} diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.spec.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.spec.ts new file mode 100644 index 00000000..b12c1f0b --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.spec.ts @@ -0,0 +1,194 @@ +import path from 'node:path'; + +import { tryParseUrl as parseUrl } from '@d-zero/shared/parse-url'; +import { Archive } from '@nitpicker/crawler'; +import { ArchiveManager, buildViewerReadModel } from '@nitpicker/query'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; + +import { createApp } from '../create-app.js'; + +const __filename = new URL(import.meta.url).pathname; +const __dirname = path.dirname(__filename); + +const BASE_CONFIG = { + baseUrl: 'https://example.com', + name: 'test', + version: '0.10.0', + recursive: true, + interval: 0, + image: true, + fetchExternal: false, + parallels: 1, + roots: ['https://example.com'], + excludes: [], + excludeKeywords: [], + excludeUrls: [], + maxExcludedDepth: 0, + retry: 3, + fromList: false, + disableQueries: false, + userAgent: 'test', + ignoreRobots: false, +}; + +const META = { + lang: null, + title: null, + description: null, + keywords: null, + noindex: false, + nofollow: false, + noarchive: false, + canonical: null, + alternate: null, + 'og:type': null, + 'og:title': null, + 'og:site_name': null, + 'og:description': null, + 'og:url': null, + 'og:image': null, + 'twitter:card': null, +}; + +/** + * Builds a fixture archive with a small directory tree (root + 2 depths + + * one 4-deep chain) and returns an in-process Hono app wired to it via the + * same read-only-open path the real viewer uses — mirrors + * `register-pages-route.spec.ts`'s `buildFixture`. + * @param workingDir - Unique scratch directory for this fixture. + * @param withReadModel - Whether to build the viewer read model before + * opening read-only. Unlike `/api/pages`, directory-tree has no legacy + * fallback, so `false` exercises the "read model not built" empty-response + * path rather than an alternate query backend. + * @returns The app, archive, and manager — callers must close both in `afterAll`. + */ +async function buildFixture(workingDir: string, withReadModel: boolean) { + const { mkdirSync } = await import('node:fs'); + mkdirSync(workingDir, { recursive: true }); + const archive = await Archive.create({ + filePath: path.resolve(workingDir, 'fixture.nitpicker'), + cwd: workingDir, + }); + await archive.setConfig(BASE_CONFIG); + for (const url of [ + 'https://example.com/', + 'https://example.com/blog/2024/post-1', + 'https://example.com/a/b/c/d/page', + ]) { + await archive.setPage({ + url: parseUrl(url)!, + redirectPaths: [], + isExternal: false, + isTarget: true, + status: 200, + statusText: 'OK', + contentType: 'text/html', + contentLength: 100, + responseHeaders: {}, + html: '<html></html>', + meta: META, + anchorList: [], + imageList: [], + isSkipped: false, + }); + } + if (withReadModel) { + await buildViewerReadModel(archive); + } + + const manager = new ArchiveManager(); + const { archiveId, mode } = await manager.open(archive.tmpDir); + const app = createApp({ + context: { + manager, + archiveId, + filePath: archive.tmpDir, + mode, + crawlerLockHolder: null, + }, + publicDir: '/tmp/no-such-dir-register-directory-tree-route-spec', + }); + return { app, archive, manager }; +} + +describe('registerDirectoryTreeRoute (integration)', () => { + describe('read model built', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_register_directory_tree_route__', + ); + let fixture: Awaited<ReturnType<typeof buildFixture>>; + + beforeAll(async () => { + fixture = await buildFixture(workingDir, true); + }); + + afterAll(async () => { + await fixture.manager.closeAll(); + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns one root for example.com with only depth<=3 nodes', async () => { + const res = await fixture.app.request('/api/directory-tree'); + expect(res.status).toBe(200); + const body = (await res.json()) as { + roots: { rootKey: string; nodes: { path: string; depth: number }[] }[]; + }; + expect(body.roots).toHaveLength(1); + expect(body.roots[0]?.rootKey).toBe('example.com'); + expect(body.roots[0]?.nodes.every((n) => n.depth <= 3)).toBe(true); + expect(body.roots[0]?.nodes.some((n) => n.path === '/a/b/c/')).toBe(true); + expect(body.roots[0]?.nodes.some((n) => n.path === '/a/b/c/d/')).toBe(false); + }); + + it('exposes childCount (directChildDirCount + directPageCount) and precomputed descendantPageCount on the root node', async () => { + const res = await fixture.app.request('/api/directory-tree'); + const body = (await res.json()) as { + roots: { + nodes: { + path: string; + directChildDirCount: number; + directPageCount: number; + childCount: number; + descendantPageCount: number; + }[]; + }[]; + }; + const root = body.roots[0]?.nodes.find((n) => n.path === '/'); + // Hardcoded against the fixture, not recomputed from the same + // formula the production code uses — root has 2 direct child dirs + // (blog, a) and 1 direct page (the root URL itself). + expect(root).toMatchObject({ + directChildDirCount: 2, + directPageCount: 1, + childCount: 3, + descendantPageCount: 3, + }); + }); + }); + + describe('read model not built', () => { + const workingDir = path.resolve( + __dirname, + '__test_fixtures_register_directory_tree_route_no_model__', + ); + let fixture: Awaited<ReturnType<typeof buildFixture>>; + + beforeAll(async () => { + fixture = await buildFixture(workingDir, false); + }); + + afterAll(async () => { + await fixture.manager.closeAll(); + const { rmSync } = await import('node:fs'); + rmSync(workingDir, { recursive: true, force: true }); + }); + + it('returns an empty roots array rather than a 500', async () => { + const res = await fixture.app.request('/api/directory-tree'); + expect(res.status).toBe(200); + expect(await res.json()).toEqual({ roots: [] }); + }); + }); +}); diff --git a/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.ts b/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.ts new file mode 100644 index 00000000..a60b0fdd --- /dev/null +++ b/packages/@nitpicker/viewer/src/routes/register-directory-tree-route.ts @@ -0,0 +1,23 @@ +import type { ArchiveContext } from '../types.js'; +import type { Hono } from 'hono'; + +import { getDirectoryTree } from '@nitpicker/query'; + +/** + * Registers `GET /api/directory-tree` — the initial (depth ≤ 3) directory + * tree for every root (host) in the archive, as a flat parent-linked node + * list per root. The frontend builds the nested UI tree client-side from + * `parentNodeId` links; this endpoint never recurses server-side. + * + * Takes no query parameters — the archive already knows which hosts it + * crawled, so there is nothing for a caller to specify up front. + * @param app - The Hono application. + * @param context - The opened archive context. + */ +export function registerDirectoryTreeRoute(app: Hono, context: ArchiveContext): void { + app.get('/api/directory-tree', async (c) => { + const accessor = context.manager.get(context.archiveId); + const roots = await getDirectoryTree(accessor); + return c.json({ roots }); + }); +}