feat: frame-based positioning and mark system foundation
Core Changes: - Fix SearchView to use start_frame/end_frame directly (no time*fps conversion) - Add hard_delete support to delete_trace API - VideoPlayer: Main timeline + Mark system foundation - Proxy: Add local routes for auth, media, identity-matches, cluster-results - Add .gitignore to exclude build artifacts and dependencies Design Documents: - Multi-track Mark system design (.opencode/plans/) - Video editing positioning standards research Files Modified: - src/views/SearchView.vue: Frame positioning, ensureMinDuration (240 frames) - src/views/PeopleView.vue: batchDeleteGroups with hard_delete - src/api/index.ts: delete_trace with hard_delete body - src/components/VideoPlayer.vue: Timeline + Mark UI - src-tauri/src/proxy.rs: New local routes - AGENTS.md: Update documentation
This commit is contained in:
358
docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md
Normal file
358
docs/SEARCH_IMPROVEMENTS_FOR_CORE_TEAM.md
Normal file
@@ -0,0 +1,358 @@
|
||||
# Search Improvements for Core Team
|
||||
|
||||
## Overview
|
||||
|
||||
This document outlines the search improvements implemented in Studio that may require Core API support or coordination.
|
||||
|
||||
---
|
||||
|
||||
## 1. Keyword Search - Result Merging
|
||||
|
||||
### Current Implementation (Studio-side)
|
||||
|
||||
```typescript
|
||||
function mergeResults(results: any[]): any[] {
|
||||
// Sort by start_frame
|
||||
const sorted = [...results].sort((a, b) => (a.start_frame || 0) - (b.start_frame || 0))
|
||||
|
||||
// Merge overlapping results (gap <= 30 frames)
|
||||
for (const r of sorted) {
|
||||
const last = merged[merged.length - 1]
|
||||
if (last && last.file_uuid === r.file_uuid) {
|
||||
const gap = r.start_frame - last.end_frame
|
||||
if (gap <= 30) {
|
||||
// Merge: extend end_frame, keep best summary
|
||||
last.end_frame = Math.max(last.end_frame, r.end_frame)
|
||||
if (r.similarity > last.similarity) {
|
||||
last.summary = r.summary
|
||||
last.similarity = r.similarity
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
merged.push({ ...r })
|
||||
}
|
||||
return merged.slice(0, 10) // Top 10
|
||||
}
|
||||
```
|
||||
|
||||
### Behavior
|
||||
|
||||
| Before | After |
|
||||
|--------|-------|
|
||||
| 20 results, overlapping segments | 10 merged results |
|
||||
| Same content appears multiple times | Deduplicated by frame proximity |
|
||||
| Short segments (1-2 seconds) | Merged into longer segments |
|
||||
|
||||
### Core API Considerations
|
||||
|
||||
**Option A: Server-side Merging**
|
||||
- Add `merge=true` parameter to `/api/v1/search/smart`
|
||||
- Server returns pre-merged results
|
||||
- More efficient, less data transfer
|
||||
|
||||
**Option B: Client-side (Current)**
|
||||
- Studio fetches 30 results, merges locally
|
||||
- More flexible, no Core API changes needed
|
||||
|
||||
---
|
||||
|
||||
## 2. Minimum Duration Enforcement
|
||||
|
||||
### Problem
|
||||
|
||||
Some search results are too short (1-2 seconds) to provide meaningful context.
|
||||
|
||||
### Proposed Solution
|
||||
|
||||
Add `min_duration` parameter to search endpoints:
|
||||
|
||||
```bash
|
||||
POST /api/v1/search/smart
|
||||
{
|
||||
"query": "Audrey Hepburn",
|
||||
"min_duration": 10, // Minimum 10 seconds
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
### Core API Impact
|
||||
|
||||
| Endpoint | Change |
|
||||
|----------|--------|
|
||||
| `/api/v1/search/smart` | Add `min_duration` param |
|
||||
| `/api/v1/search/llm-smart` | Add `min_duration` param |
|
||||
|
||||
**Implementation:**
|
||||
1. Filter results where `end_time - start_time < min_duration`
|
||||
2. Or expand short results by including adjacent chunks
|
||||
|
||||
---
|
||||
|
||||
## 3. Frame-based Range Adjustment
|
||||
|
||||
### Current Studio Implementation
|
||||
|
||||
Users can adjust playback range by frame number:
|
||||
|
||||
```vue
|
||||
<div class="card-range">
|
||||
<input type="number" v-model.number="r._startFrame">
|
||||
<input type="number" v-model.number="r._endFrame">
|
||||
</div>
|
||||
```
|
||||
|
||||
### Core API Requirements
|
||||
|
||||
No changes needed - Studio uses existing `start_frame`/`end_frame` fields.
|
||||
|
||||
**Display format:**
|
||||
```
|
||||
F100–F350 (4:10–14:35)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Mark & Export Feature
|
||||
|
||||
### Studio Implementation
|
||||
|
||||
```typescript
|
||||
const markedResults = ref<any[]>([])
|
||||
|
||||
function toggleMark(r: any) {
|
||||
const key = `${r.file_uuid}:${r.start_frame}-${r.end_frame}`
|
||||
const idx = markedResults.value.findIndex(m =>
|
||||
`${m.file_uuid}:${m.start_frame}-${m.end_frame}` === key
|
||||
)
|
||||
if (idx >= 0) {
|
||||
markedResults.value.splice(idx, 1)
|
||||
} else {
|
||||
markedResults.value.push({...r})
|
||||
}
|
||||
}
|
||||
|
||||
function exportMarked() {
|
||||
const data = {
|
||||
exportedAt: new Date().toISOString(),
|
||||
count: markedResults.value.length,
|
||||
results: markedResults.value
|
||||
}
|
||||
// Download as JSON
|
||||
}
|
||||
```
|
||||
|
||||
### Export Format
|
||||
|
||||
```json
|
||||
{
|
||||
"exportedAt": "2026-07-20T12:00:00.000Z",
|
||||
"count": 3,
|
||||
"results": [
|
||||
{
|
||||
"file_uuid": "abc123",
|
||||
"file_name": "Roman Holiday.mp4",
|
||||
"start_frame": 100,
|
||||
"end_frame": 350,
|
||||
"start_time": 4.16,
|
||||
"end_time": 14.58,
|
||||
"summary": "Audrey Hepburn speaking...",
|
||||
"similarity": 0.85
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Future Core API Integration
|
||||
|
||||
**Possible endpoints:**
|
||||
|
||||
```
|
||||
POST /api/v1/marks # Save marked segment
|
||||
GET /api/v1/marks # List saved marks
|
||||
POST /api/v1/marks/export # Export marks as EDL/JSON
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Vector/Keyword Weight Ratio
|
||||
|
||||
### Current State
|
||||
|
||||
`/api/v1/search/llm-smart` uses RRF (Reciprocal Rank Fusion):
|
||||
- Vector search: 50%
|
||||
- Keyword (BM25): 50%
|
||||
|
||||
### Proposed Change
|
||||
|
||||
Increase vector weight for semantic relevance:
|
||||
|
||||
```python
|
||||
# Current
|
||||
score = 0.5 / (k + rank_vector) + 0.5 / (k + rank_keyword)
|
||||
|
||||
# Proposed
|
||||
VECTOR_WEIGHT = 0.7
|
||||
KEYWORD_WEIGHT = 0.3
|
||||
score = VECTOR_WEIGHT / (k + rank_vector) + KEYWORD_WEIGHT / (k + rank_keyword)
|
||||
```
|
||||
|
||||
### Core API Change Required
|
||||
|
||||
Add `vector_weight` parameter:
|
||||
|
||||
```bash
|
||||
POST /api/v1/search/llm-smart
|
||||
{
|
||||
"query": "two people talking",
|
||||
"vector_weight": 0.7, # Default: 0.5
|
||||
"limit": 10
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Agent Search - Question Templates
|
||||
|
||||
### Studio Implementation
|
||||
|
||||
Pre-defined question templates shown below search input:
|
||||
|
||||
```typescript
|
||||
const QUESTION_TEMPLATES = [
|
||||
{
|
||||
category: '找檔案',
|
||||
examples: [
|
||||
'找出包含 {人物} 的影片',
|
||||
'找出 {年份} 年的影片'
|
||||
]
|
||||
},
|
||||
{
|
||||
category: '找人物',
|
||||
examples: [
|
||||
'{人物} 出現在哪些影片？',
|
||||
'{人物} 和 {人物} 第一次同框'
|
||||
]
|
||||
},
|
||||
{
|
||||
category: '找內容',
|
||||
examples: [
|
||||
'找出討論 {主題} 的片段',
|
||||
'找出 {時間點} 發生什麼事'
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Core API Requirements
|
||||
|
||||
No changes needed - templates are Studio-side UI.
|
||||
|
||||
---
|
||||
|
||||
## 7. VLM Search Integration (Future)
|
||||
|
||||
### Proposal
|
||||
|
||||
Add visual-language model search capability:
|
||||
|
||||
```bash
|
||||
POST /api/v1/search/vlm
|
||||
{
|
||||
"query": "person wearing red dress",
|
||||
"file_uuid": "optional"
|
||||
}
|
||||
```
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"file_uuid": "abc123",
|
||||
"start_frame": 1000,
|
||||
"end_frame": 1050,
|
||||
"description": "Woman in red dress walking",
|
||||
"confidence": 0.92
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Core Team Considerations
|
||||
|
||||
1. Model selection (CLIP, BLIP, etc.)
|
||||
2. GPU requirements
|
||||
3. Indexing strategy
|
||||
4. Latency expectations
|
||||
|
||||
---
|
||||
|
||||
## Summary of Core API Changes
|
||||
|
||||
| Feature | Priority | Core API Changes |
|
||||
|---------|----------|------------------|
|
||||
| Result merging | Low | Optional: `merge=true` param |
|
||||
| Min duration | Medium | Add `min_duration` param |
|
||||
| Frame adjustment | None | No changes needed |
|
||||
| Mark/Export | Low | Future: marks endpoints |
|
||||
| Vector weight | Medium | Add `vector_weight` param |
|
||||
| Question templates | None | Studio-side only |
|
||||
| VLM search | Future | New endpoint |
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist for Core Team
|
||||
|
||||
### 1. Search Endpoint Testing
|
||||
|
||||
```bash
|
||||
# Test basic search
|
||||
curl -X POST http://localhost:3002/api/v1/search/smart \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "test", "limit": 30}'
|
||||
|
||||
# Test with min_duration (if implemented)
|
||||
curl -X POST http://localhost:3002/api/v1/search/smart \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "test", "min_duration": 10}'
|
||||
|
||||
# Test vector weight (if implemented)
|
||||
curl -X POST http://localhost:3002/api/v1/search/llm-smart \
|
||||
-H "X-API-Key: $KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"query": "test", "vector_weight": 0.7}'
|
||||
```
|
||||
|
||||
### 2. Performance Testing
|
||||
|
||||
| Metric | Target |
|
||||
|--------|--------|
|
||||
| Search latency | < 500ms |
|
||||
| Result count | Stable at limit=30 |
|
||||
| Memory usage | No significant increase |
|
||||
|
||||
### 3. Integration Testing
|
||||
|
||||
- [ ] Search with file_uuid filter
|
||||
- [ ] Search without file_uuid (global)
|
||||
- [ ] Verify start_frame/end_frame accuracy
|
||||
- [ ] Verify time/frame consistency (fps)
|
||||
|
||||
---
|
||||
|
||||
## Questions for Core Team
|
||||
|
||||
1. Should result merging be server-side or client-side?
|
||||
2. What's the acceptable latency for search?
|
||||
3. Is `min_duration` parameter feasible?
|
||||
4. Can `vector_weight` be made configurable?
|
||||
5. Timeline for VLM search integration?
|
||||
|
||||
---
|
||||
|
||||
*Document created: 2026-07-20*
|
||||
*Author: Studio Team*
|
||||
Reference in New Issue
Block a user