From 5951aca08607fb3376b7180323441b376e582b5d Mon Sep 17 00:00:00 2001 From: Momentry Studio Date: Fri, 24 Jul 2026 20:19:47 +0800 Subject: [PATCH] feat: frame-based positioning and mark system foundation Core Changes: - Fix SearchView to use start_frame/end_frame directly (no time*fps conversion) - Add hard_delete support to delete_trace API - VideoPlayer: Main timeline + Mark system foundation - Proxy: Add local routes for auth, media, identity-matches, cluster-results - Add .gitignore to exclude build artifacts and dependencies Design Documents: - Multi-track Mark system design (.opencode/plans/) - Video editing positioning standards research Files Modified: - src/views/SearchView.vue: Frame positioning, ensureMinDuration (240 frames) - src/views/PeopleView.vue: batchDeleteGroups with hard_delete - src/api/index.ts: delete_trace with hard_delete body - src/components/VideoPlayer.vue: Timeline + Mark UI - src-tauri/src/proxy.rs: New local routes - AGENTS.md: Update documentation --- .gitignore | 18 + .../plans/multi_track_mark_system_design.md | 978 ++++++ ..._editing_positioning_standards_research.md | 281 ++ AGENTS.md | 14 +- docs/FACE_CLUSTERING_AGENT_REQUEST.md | 68 + docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md | 358 ++ docs/SEARCH_SOURCE_TYPE_REQUEST.md | 117 + docs/VLM_AGENT_TOOL_REQUEST.md | 90 + docs/core-api-fix-video-range.md | 186 ++ docs/core-api-pose-appearance-endpoint.md | 290 ++ docs/core-api-usage.md | 434 +++ docs/issues/corrupted-video-seek-issue.md | 147 + .../core-api-face-groups-endpoint.md | 451 +++ docs/proposals/face-group-naming-issue.md | 285 ++ package-lock.json | 111 + package.json | 5 +- src-tauri/Cargo.lock | 109 +- src-tauri/Cargo.toml | 3 + src-tauri/src/bin/proxy.rs | 18 +- src-tauri/src/db.rs | 110 +- src-tauri/src/main.rs | 334 +- src-tauri/src/proxy.rs | 462 ++- src/App.vue | 187 +- src/api/config.ts | 7 +- src/api/index.ts | 370 ++- src/assets/momentry-base.css | 51 +- src/components/VideoPlayer.vue | 442 ++- src/main.ts | 3 + src/router/index.ts | 23 +- src/store.ts | 840 ++++- src/views/AdminView.vue | 411 ++- src/views/ClientView.vue | 992 ++++-- src/views/LibraryView.vue | 1468 ++++++-- src/views/LoginView.vue | 79 +- src/views/PeopleView.vue | 2959 +++++++++++++---- src/views/PersonDetailView.vue | 280 +- src/views/SearchView.vue | 778 ++++- vite.config.ts | 14 +- 38 files changed, 12107 insertions(+), 1666 deletions(-) create mode 100644 .gitignore create mode 100644 .opencode/plans/multi_track_mark_system_design.md create mode 100644 .opencode/plans/video_editing_positioning_standards_research.md create mode 100644 docs/FACE_CLUSTERING_AGENT_REQUEST.md create mode 100644 docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md create mode 100644 docs/SEARCH_SOURCE_TYPE_REQUEST.md create mode 100644 docs/VLM_AGENT_TOOL_REQUEST.md create mode 100644 docs/core-api-fix-video-range.md create mode 100644 docs/core-api-pose-appearance-endpoint.md create mode 100644 docs/core-api-usage.md create mode 100644 docs/issues/corrupted-video-seek-issue.md create mode 100644 docs/proposals/core-api-face-groups-endpoint.md create mode 100644 docs/proposals/face-group-naming-issue.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dbb22a7 --- /dev/null +++ b/.gitignore @@ -0,0 +1,18 @@ +# Dependencies +node_modules/ + +# Build outputs +dist/ +src-tauri/target/ + +# Secrets +.gitea_token + +# Backup files +*.bak +*.backup +*.tar.gz + +# Database journals +*.sqlite-journal +*.sqlite-wal \ No newline at end of file diff --git a/.opencode/plans/multi_track_mark_system_design.md b/.opencode/plans/multi_track_mark_system_design.md new file mode 100644 index 0000000..da4d71e --- /dev/null +++ b/.opencode/plans/multi_track_mark_system_design.md @@ -0,0 +1,978 @@ +# 多轨 Mark 系统设计文档 + +## 一、概述 + +### 1.1 目标 + +构建一个多轨道视频/音频编辑展示系统，支持： +- 多轨 Video/Audio 同步播放 +- 每个轨道独立的 Mark 系统 +- 基于 Frame 的精确定位和对齐 + +### 1.2 核心原则 + +| 原则 | 说明 | +|------|------| +| **Frame 是唯一标准** | 所有定位、长度、对齐都用 Frame（整数） | +| **Time 仅用于显示** | Frame / FPS 转换，不存储 | +| **整数运算** | 所有 Frame 操作都是整数，避免浮点精度问题 | +| **原点对齐** | Frame 0 = Time 0，所有轨道共享同一原点 | +| **轨道独立** | 每个轨道有独立的 Mark 系统 | +| **同步播放** | 全局播放头对齐所有轨道 | + +### 1.3 概念统一 + +**核心洞察**：Trace 和 Segment 只是不同 tag 的 Marks + +``` +传统概念 → 统一概念 +──────────────────────────────── +Trace → Marks with tag='face' +Segment → Marks with tag='segment' +ASR 分析 → Marks with tag='asr' +Noise 检测 → Marks with tag='noise' +用户标记 → Marks with tag='user' +``` + +**好处**： +- 统一数据模型 +- 简化代码实现 +- 一致的交互行为 +- 灵活扩展新类型 + +--- + +## 二、架构设计 + +### 2.1 层级架构 + +``` +┌─────────────────────────────────────────────────────────────┐ +│ 应用层 (Application) │ +│ - 剪接工具 │ +│ - 审核工具 │ +│ - 播放器 │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ 项目层 (Project) │ +│ - 多轨道管理 │ +│ - 全局播放状态 │ +│ - FPS / TotalFrames 元数据 │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ 轨道层 (Track) │ +│ - Video Track (视频轨道 + Marks) │ +│ - Audio Track (音频轨道 + Marks) │ +│ - Mark Track (纯 Mark 轨道) │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Mark 层 (Mark) │ +│ - startFrame / endFrame (整数) │ +│ - tag / note / metadata │ +│ - 属于特定 Track │ +└─────────────────────────────────────────────────────────────┘ +``` + +### 2.2 数据流 + +``` +┌──────────────┐ +│ 数据源 │ +│ - Face API │ +│ - ASR API │ +│ - Noise API │ +└──────┬───────┘ + │ 获取数据 + ▼ +┌──────────────┐ +│ 转换层 │ +│ Time→Frame │ (用 FPS) +│ 统一格式 │ +└──────┬───────┘ + │ Mark 数据 + ▼ +┌──────────────┐ +│ Project │ +│ 多轨道管理 │ +└──────┬───────┘ + │ Track + Marks + ▼ +┌──────────────┐ +│ 展示层 │ +│ - Timeline │ +│ - 播放器 │ +└──────────────┘ +``` + +--- + +## 三、数据结构 + +### 3.1 核心类型定义 + +```typescript +// ========== 基础类型 ========== + +type Frame = number // 整数，绝对定位标准 + +// ========== Mark ========== + +interface Mark { + id: string + trackId: string // 所属轨道 + + // 定位（Frame，整数） + startFrame: Frame + endFrame?: Frame + + // 分类与内容 + tag: string // 'face' | 'asr' | 'noise' | 'user' | ... + note?: string + metadata?: any // 原始数据（如 ASR text, noise dB） + + // 审核状态 + status?: 'pending' | 'confirmed' | 'resolved' | 'ignored' + severity?: 'low' | 'medium' | 'high' +} + +// ========== Clip ========== + +interface Clip { + id: string + + // 源文件范围 + sourceStartFrame: Frame + sourceEndFrame: Frame + + // 时间轴位置（全局 Frame） + timelineStartFrame: Frame + timelineEndFrame: Frame +} + +// ========== Track ========== + +interface Track { + id: string + type: 'video' | 'audio' | 'mark-only' + name: string + enabled: boolean // 是否启用 + + // 轨道内容 + file_uuid?: string + clips?: Clip[] + + // 独立的 Mark 系统 + marks: Mark[] +} + +// ========== Project ========== + +interface Project { + id: string + name: string + + // 全局元数据（所有轨道共享） + fps: number // 帧率 + totalFrames: Frame // 总帧数（最长轨道） + duration: number // 总时长（秒），仅用于显示 + + // 多轨道 + tracks: Track[] + + // 播放状态 + playhead: Frame // 当前播放位置（全局） + playing: boolean +} +``` + +### 3.2 Frame 操作规范 + +```typescript +// 所有 Frame 操作都是整数 + +// Frame → Time（仅显示用） +function frameToTime(frame: Frame, fps: number): number { + return frame / fps +} + +// Time → Frame（数据转换用） +function timeToFrame(time: number, fps: number): Frame { + return Math.round(time * fps) +} + +// 位置百分比（Timeline 显示） +function frameToPercent(frame: Frame, totalFrames: Frame): number { + return (frame / totalFrames) * 100 +} + +// 格式化显示 +function formatFrame(frame: Frame, fps: number): string { + const sec = frame / fps + const m = Math.floor(sec / 60) + const s = Math.floor(sec % 60) + return `${m}:${s.toString().padStart(2, '0')}` +} +``` + +--- + +## 四、轨道系统 + +### 4.1 轨道类型 + +| 类型 | 说明 | 内容 | Marks | +|------|------|------|-------| +| **Video Track** | 视频轨道 | video clips + file_uuid | face, noise, blurry, user... | +| **Audio Track** | 音频轨道 | audio clips + file_uuid | asr, silence, beat, noise... | +| **Mark Track** | 纯标记轨道 | 无媒体内容 | user, comment, todo, review... | + +### 4.2 轨道 UI 布局 + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Global Playhead: Frame 12345 (0:24.5) │ +│ ▼ │ +│ ──────────────────────────────────────────────────────── │ +│ │ +│ ☑ V1 (主视频) │ +│ ├─ ████████████████████████████████ │ +│ └─ marks: │ +│ ├─ [face] ████ F100-120 │ +│ ├─ [face] ███████████ F500-650 │ +│ └─ [noise] ████ F800-900 │ +│ │ +│ ☑ A1 (原声) │ +│ ├─ ████████████████████████████████ │ +│ └─ marks: │ +│ ├─ [asr] ████ F100-120 "你好" │ +│ └─ [noise] ████ F800-900 │ +│ │ +│ ☑ M1 (审核标记) │ +│ └─ marks: │ +│ ├─ [user] ▼ F500 "需要检查" │ +│ └─ [user] ▼ F1000 "待确认" │ +│ │ +└─────────────────────────────────────────────────────────────┘ +``` + +### 4.3 轨道操作 + +```typescript +interface TrackOperations { + // 播放控制 + play(): void + pause(): void + seek(frame: Frame): void + + // Mark 操作 + addMark(mark: Omit): Mark + updateMark(id: string, updates: Partial): void + deleteMark(id: string): void + + // 批量操作 + batchUpdateMarks(filter: (m: Mark) => boolean, updates: Partial): void + batchDeleteMarks(filter: (m: Mark) => boolean): void + + // 导航 + nextMark(tag?: string): Mark | null + prevMark(tag?: string): Mark | null +} +``` + +--- + +## 五、Mark 系统 + +### 5.1 Mark 类型与来源 + +| Tag | 来源 | 数据格式 | 转换 | +|-----|------|----------|------| +| **face** | Face Trace API | first_frame, last_frame | 无需转换 | +| **asr** | ASR API | start_time, end_time | Time → Frame | +| **noise** | 噪声检测 | start_time, end_time | Time → Frame | +| **silent** | 静音检测 | start_time, end_time | Time → Frame | +| **user** | 用户创建 | frame | 无需转换 | +| **comment** | 用户评论 | frame | 无需转换 | + +### 5.2 Mark 数据源转换 + +```typescript +// Face Trace → Mark +function traceToMark(trace: FaceTrace, trackId: string): Mark { + return { + id: generateId(), + trackId, + startFrame: trace.first_frame, + endFrame: trace.last_frame, + tag: 'face', + note: trace.identity_name, + metadata: trace + } +} + +// ASR → Mark +function asrToMark(asr: AsrChunk, trackId: string, fps: number): Mark { + return { + id: generateId(), + trackId, + startFrame: timeToFrame(asr.start_time, fps), + endFrame: timeToFrame(asr.end_time, fps), + tag: 'asr', + note: asr.text, + metadata: asr + } +} + +// Noise → Mark +function noiseToMark(noise: NoiseSegment, trackId: string, fps: number): Mark { + return { + id: generateId(), + trackId, + startFrame: timeToFrame(noise.start_time, fps), + endFrame: timeToFrame(noise.end_time, fps), + tag: 'noise', + note: `${noise.db}dB`, + metadata: noise, + severity: noise.db > -30 ? 'high' : noise.db > -40 ? 'medium' : 'low' + } +} +``` + +### 5.3 Mark 操作表 + +| Tag | 操作 | 参数 | +|-----|------|------| +| **noise** | 删除片段 | padding: 5 frames | +| **silent** | 静音处理 | fade: 10ms | +| **blurry** | 标记忽略 | - | +| **face** | 绑定身份 | identity_uuid | +| **asr** | 编辑文本 | new_text | +| **user** | 自定义 | - | + +--- + +## 六、Mark 点击交互行为 + +### 6.1 核心概念 + +点击 Mark 时，根据**当前播放状态**和**全局 PlayMode** 决定行为。 + +### 6.2 Sync Lock（同步锁） + +```typescript +// 全局同步锁 +const syncLock = ref(true) // 默认锁定 + +// 锁定 🔒：点击 Mark → 所有轨道跳转到同一 frame +// 解锁 🔓：点击 Mark → 仅当前轨道跳转 +``` + +**UI 控件**： +``` +[🔒 Sync ON] / [🔓 Sync OFF] +``` + +### 6.3 PlayMode（全局播放模式） + +```typescript +type PlayMode = 'normal' | 'continue' | 'loop' +const globalPlayMode = ref('normal') +``` + +| 模式 | 行为 | +|------|------| +| **Normal** | 播放到 Mark.endFrame 后停止 | +| **Continue** | 播放到 Mark.endFrame 后继续播放 | +| **Loop** | 循环播放 Mark.startFrame ~ endFrame | + +**UI 控件**： +``` +[Normal] [Continue] [Loop] +``` + +### 6.4 点击逻辑流程 + +``` +┌─────────────────────────────────────────────────────┐ +│ 点击 Mark 触发 │ +├─────────────────────────────────────────────────────┤ +│ │ +│ 1. Seek 到 Mark.startFrame │ +│ ├─ SyncLock ON → 所有轨道同步跳转 │ +│ └─ SyncLock OFF → 仅当前轨道跳转 │ +│ │ +│ 2. 检查当前播放状态 │ +│ ├─ Pause → 只跳转，不播放 │ +│ └─ Play → 跳转后根据 PlayMode 播放 │ +│ │ +│ 3. 根据 PlayMode 设置播放行为 │ +│ ├─ Normal → 设置 stopAtFrame │ +│ ├─ Continue → 清除 stopAtFrame │ +│ └─ Loop → 设置 loopRange │ +│ │ +└─────────────────────────────────────────────────────┘ +``` + +### 6.5 代码实现 + +```typescript +// 播放状态 +const playhead = ref(0) +const isPlaying = ref(false) +const stopAtFrame = ref(null) +const loopRange = ref<{ start: Frame; end: Frame } | null>(null) + +// 点击 Mark +function onMarkClick(mark: Mark) { + // 1. Seek + if (syncLock.value) { + seekAllTracks(mark.startFrame) // 所有轨道同步 + } else { + seekTrack(mark.trackId, mark.startFrame) // 仅当前轨道 + } + + // 2. Pause 状态：只跳转，不播放 + if (!isPlaying.value) return + + // 3. Play 状态：设置播放行为 + switch (globalPlayMode.value) { + case 'normal': + stopAtFrame.value = mark.endFrame || mark.startFrame + loopRange.value = null + break + + case 'continue': + stopAtFrame.value = null + loopRange.value = null + break + + case 'loop': + // 不退出 loop（点击其他 mark 不退出） + loopRange.value = { + start: mark.startFrame, + end: mark.endFrame || mark.startFrame + } + break + } + + play() +} + +// PlayMode 切换 +function setPlayMode(mode: PlayMode) { + globalPlayMode.value = mode + + // 切换模式时退出 loop + if (mode !== 'loop') { + loopRange.value = null + } +} + +// 播放控制器 +function tick() { + if (!isPlaying.value) return + + const currentFrame = playhead.value + + // 检查 Loop 模式 + if (loopRange.value && currentFrame >= loopRange.value.end) { + seekToFrame(loopRange.value.start) + requestAnimationFrame(() => tick()) + return + } + + // 检查 Normal 模式的停止点 + if (stopAtFrame.value !== null && currentFrame >= stopAtFrame.value) { + pause() + return + } + + // Continue 模式：正常播放 + playhead.value++ + requestAnimationFrame(() => tick()) +} +``` + +### 6.6 Loop 退出条件 + +| 操作 | 是否退出 Loop | +|------|--------------| +| 点击其他 Mark | ❌ 不退出 | +| 切换 PlayMode | ✅ 退出 | +| 手动暂停 | ✅ 退出 | +| 关闭播放器 | ✅ 退出 | + +### 6.7 UI 控件设计 + +``` +┌─────────────────────────────────────────────────────┐ +│ 控制栏 │ +├─────────────────────────────────────────────────────┤ +│ [🔒 Sync] | [Normal] [Continue] [Loop] │ +│ │ +│ 状态指示： │ +│ - Sync: ON/OFF │ +│ - Mode: Normal/Continue/Loop │ +│ - Loop Range: F100-200 (如果 looping) │ +│ - Stop At: F300 (如果 normal mode) │ +└─────────────────────────────────────────────────────┘ +``` + +--- + +## 七、同步播放 + +### 6.1 播放机制 + +```typescript +class PlaybackController { + private project: Project + private playhead: Frame = 0 + private playing: boolean = false + private startTime: number = 0 + private startFrame: Frame = 0 + + play() { + this.playing = true + this.startTime = performance.now() + this.startFrame = this.playhead + + this.tick() + } + + private tick() { + if (!this.playing) return + + const elapsed = (performance.now() - this.startTime) / 1000 + this.playhead = this.startFrame + Math.round(elapsed * this.project.fps) + + // 同步所有轨道 + this.syncAllTracks(this.playhead) + + requestAnimationFrame(() => this.tick()) + } + + private syncAllTracks(frame: Frame) { + this.project.tracks.forEach(track => { + if (!track.enabled) return + + // 更新媒体内容 + if (track.type === 'video') { + this.syncVideo(track, frame) + } else if (track.type === 'audio') { + this.syncAudio(track, frame) + } + + // 高亮 Marks + this.highlightMarks(track.marks, frame) + }) + } + + seek(frame: Frame) { + this.playhead = frame + this.startTime = performance.now() + this.startFrame = frame + this.syncAllTracks(frame) + } +} +``` + +### 6.2 对齐保证 + +```typescript +// 验证原点对齐 +function validateAlignment(project: Project): boolean { + // 所有轨道共享原点 + const globalPlayhead = project.playhead + + project.tracks.forEach(track => { + // 所有轨道的当前帧 = 全局播放头 + const trackFrame = globalPlayhead + + // Mark 高亮基于同一 frame + track.marks.forEach(mark => { + const isActive = mark.startFrame <= trackFrame && + (mark.endFrame || mark.startFrame) >= trackFrame + }) + }) + + return true +} +``` + +--- + +## 八、Timeline 设计 + +### 8.1 Timeline 对齐原则 + +``` +所有 Timeline 共享同一基准： + +┌─────────────────────────────────────────────────────────────┐ +│ 基准：Frame 0 ~ TotalFrames (65000) │ +├─────────────────────────────────────────────────────────────┤ +│ │ +│ Main Timeline │ +│ ██████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ │ +│ ▲ Playhead: F12345 │ +│ │ +│ V1 Timeline │ +│ ████████████████████████████████ │ +│ [face] ████ [noise] ████ │ +│ │ +│ A1 Timeline │ +│ ████████████████████████████████ │ +│ [asr] ████ [asr] ████ │ +│ │ +│ M1 Timeline │ +│ [user] ▼ [user] ▼ │ +│ │ +│ 所有 Timeline 长度一致，位置百分比计算相同 │ +└─────────────────────────────────────────────────────────────┘ +``` + +### 8.2 Timeline 计算 + +```typescript +// 所有 Timeline 用相同公式 +const totalFrames = project.totalFrames + +// 位置（百分比） +function getPosition(frame: Frame): number { + return (frame / totalFrames) * 100 +} + +// 宽度（百分比） +function getWidth(startFrame: Frame, endFrame: Frame): number { + return ((endFrame - startFrame) / totalFrames) * 100 +} + +// 像素位置（用于精确渲染） +function getPixelPosition(frame: Frame, width: number): number { + return Math.round((frame / totalFrames) * width) +} +``` + +--- + +## 九、API 设计 + +### 9.1 Project API + +```typescript +// 创建项目 +POST /api/v1/project +{ + name: string, + fps: number, + totalFrames: number, + tracks: TrackConfig[] +} + +// 获取项目 +GET /api/v1/project/:id + +// 更新项目 +PUT /api/v1/project/:id + +// 添加轨道 +POST /api/v1/project/:id/track +{ + type: 'video' | 'audio' | 'mark-only', + name: string, + file_uuid?: string +} +``` + +### 9.2 Track API + +```typescript +// 获取轨道 +GET /api/v1/track/:id + +// 更新轨道 +PUT /api/v1/track/:id +{ + enabled?: boolean, + name?: string +} + +// 删除轨道 +DELETE /api/v1/track/:id +``` + +### 9.3 Mark API + +```typescript +// 获取轨道的所有 Marks +GET /api/v1/track/:trackId/marks + +// 添加 Mark +POST /api/v1/track/:trackId/mark +{ + startFrame: Frame, + endFrame?: Frame, + tag: string, + note?: string +} + +// 更新 Mark +PUT /api/v1/mark/:id +{ + startFrame?: Frame, + endFrame?: Frame, + tag?: string, + note?: string, + status?: string +} + +// 删除 Mark +DELETE /api/v1/mark/:id + +// 批量操作 +POST /api/v1/track/:trackId/marks/batch +{ + action: 'update' | 'delete', + filter: { tag?: string, status?: string }, + updates?: Partial +} +``` + +--- + +## 十、数据持久化 + +### 10.1 SQLite 表结构 + +```sql +-- 项目表 +CREATE TABLE projects ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + fps INTEGER NOT NULL, + total_frames INTEGER NOT NULL, + created_at TEXT DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT DEFAULT CURRENT_TIMESTAMP +); + +-- 轨道表 +CREATE TABLE tracks ( + id TEXT PRIMARY KEY, + project_id TEXT NOT NULL, + type TEXT NOT NULL, -- 'video' | 'audio' | 'mark-only' + name TEXT NOT NULL, + file_uuid TEXT, + enabled INTEGER DEFAULT 1, + position INTEGER, -- 轨道顺序 + created_at TEXT DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (project_id) REFERENCES projects(id) +); + +-- Mark 表 +CREATE TABLE marks ( + id TEXT PRIMARY KEY, + track_id TEXT NOT NULL, + start_frame INTEGER NOT NULL, + end_frame INTEGER, + tag TEXT NOT NULL, + note TEXT, + status TEXT DEFAULT 'pending', + severity TEXT, + metadata TEXT, -- JSON + created_at TEXT DEFAULT CURRENT_TIMESTAMP, + updated_at TEXT DEFAULT CURRENT_TIMESTAMP, + FOREIGN KEY (track_id) REFERENCES tracks(id) +); + +-- 索引 +CREATE INDEX idx_marks_track ON marks(track_id); +CREATE INDEX idx_marks_frame ON marks(start_frame, end_frame); +CREATE INDEX idx_marks_tag ON marks(tag); +``` + +--- + +## 十一、UI 组件 + +### 11.1 组件层级 + +``` +App +└─ ProjectEditor + ├─ Toolbar + │ ├─ PlayButton + │ ├─ FrameInput + │ └─ ZoomControl + ├─ TimelinePanel + │ ├─ PlayheadIndicator + │ ├─ TrackList + │ │ ├─ TrackItem (Video) + │ │ ├─ TrackItem (Audio) + │ │ └─ TrackItem (Mark-only) + │ └─ TimeScale + └─ VideoPlayer (可选) + └─ MarkOverlay +``` + +### 11.2 核心组件 + +#### TimelinePanel + +```vue + + + +``` + +--- + +## 十二、实现步骤 + +### 12.1 Phase 1: 基础架构（P0） + +1. **数据结构定义** + - 定义 TypeScript 接口 + - 实现 Frame 操作函数 + +2. **SQLite 存储** + - 创建表结构 + - 实现 CRUD API + +3. **单轨道 Mark 系统** + - Track + Mark 基础功能 + - Timeline 展示 + +### 12.2 Phase 2: 多轨播放（P1） + +1. **多轨道管理** + - Project 层实现 + - Track 增删改 + +2. **同步播放** + - PlaybackController + - 全局 Playhead + +3. **Timeline 对齐** + - 统一计算公式 + - 视觉对齐 + +### 12.3 Phase 3: 数据集成（P1） + +1. **数据源转换** + - Face Trace → Mark + - ASR → Mark + - Noise → Mark + +2. **FPS 管理** + - 视频元数据获取 + - Time ↔ Frame 转换 + +### 12.4 Phase 4: 操作功能（P2） + +1. **Mark 操作** + - 批量更新 + - 批量删除 + +2. **剪辑功能** + - 基于 Mark 的剪辑 + - 片段操作 + +--- + +## 十三、测试计划 + +### 13.1 单元测试 + +- Frame 操作函数 +- Time ↔ Frame 转换 +- Mark 过滤/排序 +- Timeline 位置计算 + +### 13.2 集成测试 + +- 多轨同步播放 +- Playhead 对齐 +- Mark 高亮状态 +- 数据持久化 + +### 13.3 E2E 测试 + +- 创建项目 +- 添加轨道 +- 创建/编辑/删除 Mark +- 播放和导航 + +--- + +## 十四、附录 + +### 14.1 命名约定 + +| 概念 | 变量名 | 类型 | +|------|--------|------| +| 帧 | `frame` | `Frame` (number) | +| 起始帧 | `startFrame` | `Frame` | +| 结束帧 | `endFrame` | `Frame` | +| 总帧数 | `totalFrames` | `Frame` | +| 帧率 | `fps` | `number` | +| 播放头 | `playhead` | `Frame` | +| 时间（秒） | `time` / `sec` | `number` | + +### 14.2 配置参数 + +| 参数 | 默认值 | 说明 | +|------|--------|------| +| 默认 FPS | 30 | 无视频时使用 | +| 最小 Mark 宽度 | 1px | 渲染时保证可见 | +| Timeline 高度 | 24px | 每个轨道行高 | +| Playhead 更新频率 | 60fps | requestAnimationFrame | + +--- + +## 十五、版本历史 + +| 版本 | 日期 | 变更 | +|------|------|------| +| 1.0 | 2026-07-24 | 初始设计文档 | +| 1.1 | 2026-07-24 | 添加概念统一说明、Mark 点击交互行为章节 | \ No newline at end of file diff --git a/.opencode/plans/video_editing_positioning_standards_research.md b/.opencode/plans/video_editing_positioning_standards_research.md new file mode 100644 index 0000000..7431426 --- /dev/null +++ b/.opencode/plans/video_editing_positioning_standards_research.md @@ -0,0 +1,281 @@ +# 视频编辑定位标准研究报告 + +## 一、行业标准概览 + +### 1.1 SMPTE Timecode（行业标准） + +**来源**: Society of Motion Picture and Television Engineers (SMPTE) +**标准**: SMPTE 12M (2008年修订为 SMPTE 12M-1, 12M-2, 12M-3) + +**核心原则**: +- **Frame-based**: 以帧为单位，格式 `HH:MM:SS:FF`（小时:分钟:秒:帧） +- **整数运算**: 所有帧号都是整数 +- **原点对齐**: Frame 0 = Time 0 +- **FPS 依赖**: 帧率决定显示时间，但帧数不变 + +**支持的帧率**: +| FPS | 用途 | +|-----|------| +| 23.98 (24÷1.001) | 北美 HDTV | +| 24 | 电影、ATSC、2K/4K/6K | +| 25 | PAL/SECAM（欧洲、澳大利亚）| +| 29.97 (30÷1.001) | NTSC（北美、日本）| +| 30 | ATSC | + +**Drop-Frame vs Non-Drop-Frame**: +- **Drop-Frame (DF)**: 每 minute 跳过 frame 0,1（除第10分钟），用于补偿 29.97fps +- **Non-Drop-Frame (NDF)**: 连续帧号 +- 表示法: DF 用分号 `HH;MM;SS;FF`，NDF 用冒号 `HH:MM:SS:FF` + +--- + +### 1.2 EDL (Edit Decision List) + +**用途**: 剪辑决策列表，用于记录剪辑点 + +**简单 EDL 格式**: + +**Time-based**: +``` +[begin second] [end second] [action] +5.3 7.1 0 # cut from 5.3s to 7.1s +15 16.7 1 # mute from 15s to 16.7s +``` + +**Frame-based**: +``` +#[begin frame] #[end frame] [action] +#127 #170 0 # cut from frame 127 to 170 +#360 #400 1 # mute from frame 360 to 400 +``` + +**关键点**: +- 支持两种定位方式：time（浮点）和 frame（整数） +- Action 类型：0=cut, 1=mute, 2=scene marker, 3=skip +- **推荐使用 Frame-based** 以保证精度 + +--- + +### 1.3 AAF (Advanced Authoring Format) + +**来源**: Advanced Media Workflow Association (AMWA) +**标准化**: SMPTE + +**特点**: +- 专业级跨平台数据交换格式 +- 包含 Essence Data（音频、视频等）和 Metadata +- 支持复杂的对象关系描述 +- 追踪从源文件到最终产出的完整历史 +- 用于"进行中的作品"（works in progress） + +**与 MXF 的关系**: +- AAF 用于编辑中的项目 +- MXF 用于交换完成的媒体产品 +- MXF 是 AAF 数据模型的子集 + +--- + +### 1.4 OpenTimelineIO + +**来源**: Pixar 开发，现由 Academy Software Foundation 维护 +**状态**: 成熟框架，广泛应用于影视行业 + +**核心设计**: +- **OpenTime**: 无依赖的时间处理库 +- **数据模型**: Timeline → Track → Clip → Media Reference +- **帧与时间分离**: + - `RationalTime`: value + rate（精确的时间表示） + - `TimeRange`: start_time + duration + - 帧是整数，时间是 RationalTime + +**示例**: +```python +import opentimelineio as otio + +timeline = otio.adapters.read_from_file("project.aaf") +for clip in timeline.find_clips(): + print(clip.name, clip.duration()) + # duration() 返回 RationalTime(value, rate) +``` + +**支持的格式**: +- Final Cut Pro XML +- AAF +- CMX 3600 EDL +- 原生 `.otio`, `.otioz`, `.otiod` + +--- + +## 二、主流软件对比 + +### 2.1 Adobe Premiere Pro + +**定位方式**: +- 使用 SMPTE Timecode +- 支持 Drop-Frame 和 Non-Drop-Frame +- 内部用帧号定位，显示用时间码 +- 导出格式：EDL, XML, AAF + +### 2.2 Final Cut Pro + +**定位方式**: +- 使用 Apple 时间码格式 +- Frame-based 内部处理 +- 导出格式：FCP XML（专有格式） +- 时间以 frame + fps 存储 + +### 2.3 DaVinci Resolve + +**定位方式**: +- 支持多种时间码格式 +- 项目设置选择 FPS（全局） +- Timeline 使用 frame 定位 +- 支持 EDL, AAF, OTIO 导出 + +### 2.4 Avid Media Composer + +**定位方式**: +- 专业级 SMPTE Timecode 实现 +- 支持 Drop-Frame 计数 +- 轨道级时间码管理 +- AAF 原生支持 + +--- + +## 三、核心设计原则总结 + +### 3.1 行业共识 + +| 原则 | 说明 | +|------|------| +| **Frame 是唯一标准** | 所有定位、剪辑、同步都用帧号 | +| **整数运算** | 避免浮点精度问题 | +| **FPS 是元数据** | 不改变帧数，只改变显示时间 | +| **原点对齐** | Frame 0 = Time 0 | +| **Time 仅用于显示** | Frame ÷ FPS = Time | + +### 3.2 数据流架构 + +``` +┌─────────────────────────────────────────────┐ +│ Source Data (各种格式) │ +│ - Video files (frame-based metadata) │ +│ - Audio files (time-based or sample-based) │ +│ - Analysis results (various formats) │ +└─────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ 统一转换层 (标准化) │ +│ - Time → Frame (用 FPS) │ +│ - Sample → Frame (用 sample_rate) │ +│ - 所有数据统一为 Frame │ +└─────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ 内部数据模型 (Frame-based) │ +│ - Mark.startFrame (integer) │ +│ - Mark.endFrame (integer) │ +│ - Clip.timelineStartFrame (integer) │ +│ - Project.fps (metadata) │ +└─────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────┐ +│ Timeline 展示层 │ +│ - 位置：frame / totalFrames * 100% │ +│ - 显示：frame / fps → "MM:SS" │ +│ - 所有轨道对齐 │ +└─────────────────────────────────────────────┘ +``` + +--- + +## 四、对 Momentry Studio 的启示 + +### 4.1 遵循行业标准 + +**推荐做法**: +1. ✅ 采用 Frame 作为唯一定位标准 +2. ✅ 所有 Frame 操作使用整数 +3. ✅ FPS 作为项目元数据 +4. ✅ Time 仅用于显示转换 +5. ✅ 原点对齐：Frame 0 = Time 0 + +### 4.2 数据格式建议 + +```typescript +// Mark 定义（符合行业标准） +interface Mark { + id: string + startFrame: number // integer, SMPTE-style + endFrame?: number // integer + tag: string + note?: string +} + +// Clip 定义（类似 OTIO） +interface Clip { + sourceStartFrame: number // integer + sourceEndFrame: number // integer + timelineStartFrame: number // integer + timelineEndFrame: number // integer +} + +// Project 定义 +interface Project { + fps: number // 元数据 + totalFrames: number // integer + duration: number // 仅用于显示，计算得出 +} +``` + +### 4.3 导出格式支持 + +**优先级**: +1. **P0 - 内部格式**: 自定义 JSON/SQLite（Frame-based） +2. **P1 - EDL**: 简单 EDL（Frame-based） +3. **P2 - OTIO**: OpenTimelineIO 格式（行业标准） +4. **P3 - AAF/FCPXML**: 专业软件互操作 + +--- + +## 五、参考资料 + +### 5.1 标准文档 + +- [SMPTE ST 12-1:2008](https://ieeexplore.ieee.org/document/7289820/) - Time and Control Code +- [CMX 3600 EDL Specification](http://xmil.biz/EDL-X/CMX3600.pdf) +- [OpenTimelineIO Documentation](https://opentimelineio.readthedocs.io/) + +### 5.2 相关 Wikipedia 文章 + +- [SMPTE timecode](https://en.wikipedia.org/wiki/SMPTE_timecode) +- [Edit decision list](https://en.wikipedia.org/wiki/Edit_decision_list) +- [Advanced Authoring Format](https://en.wikipedia.org/wiki/Advanced_Authoring_Format) + +### 5.3 开源项目 + +- [OpenTimelineIO](https://github.com/AcademySoftwareFoundation/OpenTimelineIO) - Pixar 开源 +- [OpenTimelineIO Plugins](https://github.com/OpenTimelineIO) - 各种适配器 + +--- + +## 六、结论 + +视频编辑行业的定位标准明确且统一： + +1. **Frame 是唯一标准** - 所有专业软件和行业标准都基于帧号 +2. **整数运算** - 避免浮点精度问题 +3. **FPS 作为元数据** - 不改变帧数，只影响显示时间 +4. **OpenTimelineIO 是最佳参考** - 现代、开源、行业标准 + +Momentry Studio 的设计完全符合行业标准： +- ✅ Frame-based 定位 +- ✅ 整数运算 +- ✅ FPS 作为元数据 +- ✅ 原点对齐 +- ✅ 多轨同步 + +可以继续按照现有设计文档推进实现。 \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 91a351d..053cde5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,15 @@ # AGENTS.md +## Important: File Path Convention + +**All file paths in this document and code references must use full absolute paths.** + +Examples: +- ✅ `/Users/accusys/momentry_studio/src/views/PeopleView.vue` +- ❌ `src/views/PeopleView.vue` +- ✅ `/Users/accusys/momentry_studio/src-tauri/src/db.rs` +- ❌ `src-tauri/src/db.rs` + ## Stack - **Frontend**: Vue 3 + TypeScript + Vite (port 5173) - **Desktop**: Tauri 2 (Rust backend in `src-tauri/`) @@ -7,13 +17,15 @@ ## Commands ``` -npm run dev # Vite dev server (port 5173) +npm run proxy # Start Rust proxy binary only (port 8888) +npm run dev # Start Rust proxy + Vite dev server (port 5173) npm run build # vue-tsc --noEmit && vite build (typecheck + bundle) npm run preview # Preview production build npm run tauri # Alias for `cargo tauri` cargo tauri dev # Full Tauri desktop dev (builds frontend + opens native window) cargo tauri build # Full native app build (outputs to src-tauri/target/release/bundle/) ``` +- **Browser dev flow**: `npm run dev` → kills old proxy on 8888 → starts `momentry-proxy` → sleeps 2s → starts Vite. Vite proxy forwards `/api/v1/*` to Rust proxy at `localhost:8888`. Rust proxy handles face-crop, cluster-results, identity-matches, search-history, bookmarks, profile images locally; proxies rest to Core API at `localhost:3002`. ## Architecture - **Dual-mode API**: Frontend uses `apiCall()` from `src/api/index.ts` which detects `window.__TAURI__` and dispatches to either Tauri IPC (`invoke`) or HTTP to proxy at `http://0.0.0.0:8888`. diff --git a/docs/FACE_CLUSTERING_AGENT_REQUEST.md b/docs/FACE_CLUSTERING_AGENT_REQUEST.md new file mode 100644 index 0000000..7587d5b --- /dev/null +++ b/docs/FACE_CLUSTERING_AGENT_REQUEST.md @@ -0,0 +1,68 @@ +# Face Clustering Agent Endpoint Request + +## 需求 +新增一個 on-demand face clustering endpoint，讓前端可以手動觸發 face trace 分組。 + +## 背景 +Momentry Studio 的 People 頁面有「Face Deduplication」按鈕，目的是讓使用者可以對已處理 face 的影片執行 face 分組（clustering）。目前這個功能無法使用，因為對應的 API endpoint 不存在。 + +## 需要的 Endpoint + +### `POST /api/v1/file/:file_uuid/cluster-agent` + +**用途**：對指定檔案觸發 face clustering，將相似的 face traces 分組。 + +#### Request +``` +POST /api/v1/file/{file_uuid}/cluster-agent +Content-Type: application/json +X-API-Key: {key} + +{} +``` + +| 參數 | 類型 | 必填 | 說明 | +|------|------|------|------| +| `file_uuid` | string | ✅ (URL path) | 檔案 UUID | + +#### Response (200) +```json +{ + "success": true, + "file_uuid": "477b24d3...", + "message": "Clustering started for file", + "clusters": 0, + "total_traces": 2 +} +``` + +| 欄位 | 類型 | 說明 | +|------|------|------| +| `success` | boolean | 是否成功觸發 | +| `file_uuid` | string | 檔案 UUID | +| `message` | string | 狀態訊息 | +| `clusters` | integer | 已分組數量（剛觸發時為 0） | +| `total_traces` | integer | 該檔案總 face trace 數 | + +#### 實作邏輯（建議） +1. 檢查檔案是否存在且 face 已處理完成 +2. 讀取 face detections / face embeddings +3. 執行 face clustering（可參考 pipeline 中的 clustering 邏輯） +4. 將分組結果寫入 TKG（更新 `face_trace` node 的 `label`） +5. 回傳觸發結果（clustering 可非同步執行） + +### 前端消費流程 +``` +1. POST /api/v1/file/{uuid}/cluster-agent → 觸發 clustering +2. GET /api/v1/file/{uuid}/face-groups → 輪詢直到 face_groups 有資料 +3. 顯示 face groups 在 People 頁面 +``` + +## 相關檔案 +- **前端按鈕**: `src/views/PeopleView.vue` line 14-18 (`runClusterAgent()`) +- **API 映射**: `src/api/index.ts` line 369-371 (`run_cluster_agent`) +- **現有 face-groups**: `GET /api/v1/file/:file_uuid/face-groups` (已存在，用於讀取結果) +- **Pipeline 處理**: `docs_v1.0/doc_wasm/modules/05_process.md` (face 處理流程) + +## 優先級 +中優先級 — 目前有 2 個 unassigned traces 但無法分組，使用者無法有效的進行 face deduplication。 diff --git a/docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md b/docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md new file mode 100644 index 0000000..5714b95 --- /dev/null +++ b/docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md @@ -0,0 +1,358 @@ +# Search Improvements for Core Team + +## Overview + +This document outlines the search improvements implemented in Studio that may require Core API support or coordination. + +--- + +## 1. Keyword Search - Result Merging + +### Current Implementation (Studio-side) + +```typescript +function mergeResults(results: any[]): any[] { + // Sort by start_frame + const sorted = [...results].sort((a, b) => (a.start_frame || 0) - (b.start_frame || 0)) + + // Merge overlapping results (gap <= 30 frames) + for (const r of sorted) { + const last = merged[merged.length - 1] + if (last && last.file_uuid === r.file_uuid) { + const gap = r.start_frame - last.end_frame + if (gap <= 30) { + // Merge: extend end_frame, keep best summary + last.end_frame = Math.max(last.end_frame, r.end_frame) + if (r.similarity > last.similarity) { + last.summary = r.summary + last.similarity = r.similarity + } + continue + } + } + merged.push({ ...r }) + } + return merged.slice(0, 10) // Top 10 +} +``` + +### Behavior + +| Before | After | +|--------|-------| +| 20 results, overlapping segments | 10 merged results | +| Same content appears multiple times | Deduplicated by frame proximity | +| Short segments (1-2 seconds) | Merged into longer segments | + +### Core API Considerations + +**Option A: Server-side Merging** +- Add `merge=true` parameter to `/api/v1/search/smart` +- Server returns pre-merged results +- More efficient, less data transfer + +**Option B: Client-side (Current)** +- Studio fetches 30 results, merges locally +- More flexible, no Core API changes needed + +--- + +## 2. Minimum Duration Enforcement + +### Problem + +Some search results are too short (1-2 seconds) to provide meaningful context. + +### Proposed Solution + +Add `min_duration` parameter to search endpoints: + +```bash +POST /api/v1/search/smart +{ + "query": "Audrey Hepburn", + "min_duration": 10, // Minimum 10 seconds + "limit": 10 +} +``` + +### Core API Impact + +| Endpoint | Change | +|----------|--------| +| `/api/v1/search/smart` | Add `min_duration` param | +| `/api/v1/search/llm-smart` | Add `min_duration` param | + +**Implementation:** +1. Filter results where `end_time - start_time < min_duration` +2. Or expand short results by including adjacent chunks + +--- + +## 3. Frame-based Range Adjustment + +### Current Studio Implementation + +Users can adjust playback range by frame number: + +```vue +
+ + +
+``` + +### Core API Requirements + +No changes needed - Studio uses existing `start_frame`/`end_frame` fields. + +**Display format:** +``` +F100–F350 (4:10–14:35) +``` + +--- + +## 4. Mark & Export Feature + +### Studio Implementation + +```typescript +const markedResults = ref([]) + +function toggleMark(r: any) { + const key = `${r.file_uuid}:${r.start_frame}-${r.end_frame}` + const idx = markedResults.value.findIndex(m => + `${m.file_uuid}:${m.start_frame}-${m.end_frame}` === key + ) + if (idx >= 0) { + markedResults.value.splice(idx, 1) + } else { + markedResults.value.push({...r}) + } +} + +function exportMarked() { + const data = { + exportedAt: new Date().toISOString(), + count: markedResults.value.length, + results: markedResults.value + } + // Download as JSON +} +``` + +### Export Format + +```json +{ + "exportedAt": "2026-07-20T12:00:00.000Z", + "count": 3, + "results": [ + { + "file_uuid": "abc123", + "file_name": "Roman Holiday.mp4", + "start_frame": 100, + "end_frame": 350, + "start_time": 4.16, + "end_time": 14.58, + "summary": "Audrey Hepburn speaking...", + "similarity": 0.85 + } + ] +} +``` + +### Future Core API Integration + +**Possible endpoints:** + +``` +POST /api/v1/marks # Save marked segment +GET /api/v1/marks # List saved marks +POST /api/v1/marks/export # Export marks as EDL/JSON +``` + +--- + +## 5. Vector/Keyword Weight Ratio + +### Current State + +`/api/v1/search/llm-smart` uses RRF (Reciprocal Rank Fusion): +- Vector search: 50% +- Keyword (BM25): 50% + +### Proposed Change + +Increase vector weight for semantic relevance: + +```python +# Current +score = 0.5 / (k + rank_vector) + 0.5 / (k + rank_keyword) + +# Proposed +VECTOR_WEIGHT = 0.7 +KEYWORD_WEIGHT = 0.3 +score = VECTOR_WEIGHT / (k + rank_vector) + KEYWORD_WEIGHT / (k + rank_keyword) +``` + +### Core API Change Required + +Add `vector_weight` parameter: + +```bash +POST /api/v1/search/llm-smart +{ + "query": "two people talking", + "vector_weight": 0.7, # Default: 0.5 + "limit": 10 +} +``` + +--- + +## 6. Agent Search - Question Templates + +### Studio Implementation + +Pre-defined question templates shown below search input: + +```typescript +const QUESTION_TEMPLATES = [ + { + category: '找檔案', + examples: [ + '找出包含 {人物} 的影片', + '找出 {年份} 年的影片' + ] + }, + { + category: '找人物', + examples: [ + '{人物} 出現在哪些影片？', + '{人物} 和 {人物} 第一次同框' + ] + }, + { + category: '找內容', + examples: [ + '找出討論 {主題} 的片段', + '找出 {時間點} 發生什麼事' + ] + } +] +``` + +### Core API Requirements + +No changes needed - templates are Studio-side UI. + +--- + +## 7. VLM Search Integration (Future) + +### Proposal + +Add visual-language model search capability: + +```bash +POST /api/v1/search/vlm +{ + "query": "person wearing red dress", + "file_uuid": "optional" +} +``` + +### Response + +```json +{ + "results": [ + { + "file_uuid": "abc123", + "start_frame": 1000, + "end_frame": 1050, + "description": "Woman in red dress walking", + "confidence": 0.92 + } + ] +} +``` + +### Core Team Considerations + +1. Model selection (CLIP, BLIP, etc.) +2. GPU requirements +3. Indexing strategy +4. Latency expectations + +--- + +## Summary of Core API Changes + +| Feature | Priority | Core API Changes | +|---------|----------|------------------| +| Result merging | Low | Optional: `merge=true` param | +| Min duration | Medium | Add `min_duration` param | +| Frame adjustment | None | No changes needed | +| Mark/Export | Low | Future: marks endpoints | +| Vector weight | Medium | Add `vector_weight` param | +| Question templates | None | Studio-side only | +| VLM search | Future | New endpoint | + +--- + +## Testing Checklist for Core Team + +### 1. Search Endpoint Testing + +```bash +# Test basic search +curl -X POST http://localhost:3002/api/v1/search/smart \ + -H "X-API-Key: $KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "test", "limit": 30}' + +# Test with min_duration (if implemented) +curl -X POST http://localhost:3002/api/v1/search/smart \ + -H "X-API-Key: $KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "test", "min_duration": 10}' + +# Test vector weight (if implemented) +curl -X POST http://localhost:3002/api/v1/search/llm-smart \ + -H "X-API-Key: $KEY" \ + -H "Content-Type: application/json" \ + -d '{"query": "test", "vector_weight": 0.7}' +``` + +### 2. Performance Testing + +| Metric | Target | +|--------|--------| +| Search latency | < 500ms | +| Result count | Stable at limit=30 | +| Memory usage | No significant increase | + +### 3. Integration Testing + +- [ ] Search with file_uuid filter +- [ ] Search without file_uuid (global) +- [ ] Verify start_frame/end_frame accuracy +- [ ] Verify time/frame consistency (fps) + +--- + +## Questions for Core Team + +1. Should result merging be server-side or client-side? +2. What's the acceptable latency for search? +3. Is `min_duration` parameter feasible? +4. Can `vector_weight` be made configurable? +5. Timeline for VLM search integration? + +--- + +*Document created: 2026-07-20* +*Author: Studio Team* \ No newline at end of file diff --git a/docs/SEARCH_SOURCE_TYPE_REQUEST.md b/docs/SEARCH_SOURCE_TYPE_REQUEST.md new file mode 100644 index 0000000..e3c64a7 --- /dev/null +++ b/docs/SEARCH_SOURCE_TYPE_REQUEST.md @@ -0,0 +1,117 @@ +# Search API Source Type Request + +## Problem + +Currently, the Search API (`/api/v1/search/keyword` and `/api/v1/search/semantic`) returns results without indicating the source type: + +```json +{ + "id": 0, + "file_uuid": "...", + "start_frame": 0, + "end_frame": 128, + "start_time": 0.0, + "end_time": 5.33, + "summary": "And speaking of storage and workflow...", + "text_content": "And speaking of storage and workflow...", + ... +} +``` + +There's no field to distinguish between: +- **OCR**: Text extracted from video frames (on-screen text, titles, credits) +- **ASRX**: Speech-to-text transcription + +--- + +## Request + +Add a `source_type` field to the `SearchResult` struct: + +```rust +pub struct SearchResult { + pub id: i32, + pub file_uuid: Option, + pub parent_id: i32, + pub scene_order: Option, + pub start_frame: i64, + pub end_frame: i64, + pub fps: f64, + pub start_time: f64, + pub end_time: f64, + pub raw_text: Option, + pub summary: Option, + pub text_content: Option, + pub metadata: Option, + pub similarity: Option, + pub file_name: Option, + pub serve_url: Option, + pub thumbnail_url: Option, + pub source_type: Option, // NEW: "ocr" or "asrx" +} +``` + +--- + +## Expected Response + +```json +{ + "id": 0, + "file_uuid": "...", + "start_frame": 0, + "end_frame": 128, + "start_time": 0.0, + "end_time": 5.33, + "summary": "And speaking of storage and workflow...", + "text_content": "And speaking of storage and workflow...", + "source_type": "ocr", + ... +} +``` + +--- + +## Use Case + +Studio frontend needs to display source type tags on search result cards: + +``` +[OCR] And speaking of storage and workflow... +F0–128 (0:00–0:05) +``` + +``` +[ASRX] Today we're talking about storage solutions... +(0:10–0:20) +``` + +--- + +## Implementation Notes + +1. The database already has `chunk_type` field with values like `'sentence'`, `'cut'`, etc. +2. Need to map chunk types to source types: + - OCR chunks → `source_type: "ocr"` + - ASRX/sentence chunks → `source_type: "asrx"` +3. Update `enrich_from_pg()` function to include source_type +4. Update `SearchResult` struct serialization + +--- + +## Affected Files + +- `/Users/accusys/momentry_core/src/api/search.rs`: `SearchResult` struct and enrichment +- `/Users/accusys/momentry_core/src/core/db/postgres_db.rs`: `get_chunk_by_file_and_chunk_id()` query + +--- + +## Priority + +Medium - Required for proper search result categorization and display + +--- + +## Contact + +Studio Team \ No newline at end of file diff --git a/docs/VLM_AGENT_TOOL_REQUEST.md b/docs/VLM_AGENT_TOOL_REQUEST.md new file mode 100644 index 0000000..b0c920b --- /dev/null +++ b/docs/VLM_AGENT_TOOL_REQUEST.md @@ -0,0 +1,90 @@ +# VLM (Vision Language Model) Tool for Agent Search + +## Request +Add a `vlm_describe` tool to the Agent system that can analyze video keyframes via VLM. + +## Background +We have Ollama with LLaVA running locally (`http://localhost:11434`). The VLM can describe image content (people, objects, scenes, clothing, text, etc.). We want the Agent to be able to use this as a tool when users ask questions about visual content. + +## Use Cases +- "What is the person wearing in this scene?" +- "Find scenes where someone is wearing a red shirt" +- "Describe the background of this video" +- "What objects are visible in frame 1000?" + +## API Design + +### Agent Tool Registration +Add a `vlm_describe` tool to the Agent's tool registry in Core API. + +### Tool Definition +```json +{ + "name": "vlm_describe", + "description": "Analyze a video frame using Vision Language Model. Returns a description of the image content.", + "parameters": { + "file_uuid": "string - UUID of the video file", + "frame": "integer - Frame number to analyze", + "prompt": "string (optional) - Specific question about the image (default: 'Describe this image in 1-2 sentences.')" + } +} +``` + +### Internal Implementation +The Core API should call: +``` +POST http://localhost:11434/api/generate +{ + "model": "llava", + "prompt": "", + "images": [""], + "stream": false, + "options": { "num_predict": 80 } +} +``` + +To get the frame image, use the existing thumbnail API: +``` +GET /api/v1/file/{file_uuid}/thumbnail?api_key={key}&frame={frame} +``` +→ Returns JPEG bytes → resize to 480x270 → base64 encode → send to Ollama. + +### Result Format +```json +{ + "tool": "vlm_describe", + "result": { + "file_uuid": "...", + "frame": 1234, + "description": "A man wearing a blue suit and red tie standing at a podium.", + "time_sec": 41.13 + } +} +``` + +### Performance (Benchmarked on M5 Max) + +**Model**: LLaVA 7B Q4_0 (Ollama), 1280x720 → 480x270 resize + +| Metric | Value | +|--------|-------| +| Avg per frame | **1.02s** | +| Min | 0.81s | +| Max | 1.60s | +| Throughput | ~1 fps (sequential) | +| Concurrent (4 workers) | No speedup — Ollama queues single-GPU | + +**Recommendations**: +- Resize to 480x270 before sending reduces token count without quality loss +- Batch processing doesn't help (Ollama queues requests to single model instance) +- For large video indexing, process frames asynchronously in background +- Consider a faster model if throughput becomes critical (e.g., LLaVA-Next 8B, Moondream 1.6B) + +## Frontend Changes (for reference) +Once Core API returns `vlm_describe` in agent sources, the frontend will display: +- Tool badge: "VLM" +- Description text paired with the relevant video frame +- Click to jump to that frame in the video player + +## Timeline +Not urgent - this is an enhancement for visual search capabilities. diff --git a/docs/core-api-fix-video-range.md b/docs/core-api-fix-video-range.md new file mode 100644 index 0000000..1c1fd27 --- /dev/null +++ b/docs/core-api-fix-video-range.md @@ -0,0 +1,186 @@ +# Core API Fix Request: Video Range Requests Support + +## Issue + +Video player in Studio cannot seek (跳轉) because Core API video endpoint does not support HTTP Range requests. + +## Current Behavior + +```bash +$ curl -I "http://localhost:3002/api/v1/file/{uuid}/video?start_time=0&end_time=5" +HTTP/1.1 200 OK +content-type: video/mp4 +content-length: 1918410 +# Missing: Accept-Ranges header +# Missing: 206 Partial Content support +``` + +## Expected Behavior + +```bash +$ curl -I "http://localhost:3002/api/v1/file/{uuid}/video" -H "Range: bytes=0-1000" +HTTP/1.1 206 Partial Content +content-type: video/mp4 +content-range: bytes 0-1000/1918410 +accept-ranges: bytes +content-length: 1001 +``` + +## Why This Matters + +1. **Browser seeking requires Range support** - HTML5 `