feat: frame-based positioning and mark system foundation
Core Changes: - Fix SearchView to use start_frame/end_frame directly (no time*fps conversion) - Add hard_delete support to delete_trace API - VideoPlayer: Main timeline + Mark system foundation - Proxy: Add local routes for auth, media, identity-matches, cluster-results - Add .gitignore to exclude build artifacts and dependencies Design Documents: - Multi-track Mark system design (.opencode/plans/) - Video editing positioning standards research Files Modified: - src/views/SearchView.vue: Frame positioning, ensureMinDuration (240 frames) - src/views/PeopleView.vue: batchDeleteGroups with hard_delete - src/api/index.ts: delete_trace with hard_delete body - src/components/VideoPlayer.vue: Timeline + Mark UI - src-tauri/src/proxy.rs: New local routes - AGENTS.md: Update documentation
This commit is contained in:
90
docs/VLM_AGENT_TOOL_REQUEST.md
Normal file
90
docs/VLM_AGENT_TOOL_REQUEST.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# VLM (Vision Language Model) Tool for Agent Search
|
||||
|
||||
## Request
|
||||
Add a `vlm_describe` tool to the Agent system that can analyze video keyframes via VLM.
|
||||
|
||||
## Background
|
||||
We have Ollama with LLaVA running locally (`http://localhost:11434`). The VLM can describe image content (people, objects, scenes, clothing, text, etc.). We want the Agent to be able to use this as a tool when users ask questions about visual content.
|
||||
|
||||
## Use Cases
|
||||
- "What is the person wearing in this scene?"
|
||||
- "Find scenes where someone is wearing a red shirt"
|
||||
- "Describe the background of this video"
|
||||
- "What objects are visible in frame 1000?"
|
||||
|
||||
## API Design
|
||||
|
||||
### Agent Tool Registration
|
||||
Add a `vlm_describe` tool to the Agent's tool registry in Core API.
|
||||
|
||||
### Tool Definition
|
||||
```json
|
||||
{
|
||||
"name": "vlm_describe",
|
||||
"description": "Analyze a video frame using Vision Language Model. Returns a description of the image content.",
|
||||
"parameters": {
|
||||
"file_uuid": "string - UUID of the video file",
|
||||
"frame": "integer - Frame number to analyze",
|
||||
"prompt": "string (optional) - Specific question about the image (default: 'Describe this image in 1-2 sentences.')"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Internal Implementation
|
||||
The Core API should call:
|
||||
```
|
||||
POST http://localhost:11434/api/generate
|
||||
{
|
||||
"model": "llava",
|
||||
"prompt": "<user prompt or default>",
|
||||
"images": ["<base64 of frame thumbnail>"],
|
||||
"stream": false,
|
||||
"options": { "num_predict": 80 }
|
||||
}
|
||||
```
|
||||
|
||||
To get the frame image, use the existing thumbnail API:
|
||||
```
|
||||
GET /api/v1/file/{file_uuid}/thumbnail?api_key={key}&frame={frame}
|
||||
```
|
||||
→ Returns JPEG bytes → resize to 480x270 → base64 encode → send to Ollama.
|
||||
|
||||
### Result Format
|
||||
```json
|
||||
{
|
||||
"tool": "vlm_describe",
|
||||
"result": {
|
||||
"file_uuid": "...",
|
||||
"frame": 1234,
|
||||
"description": "A man wearing a blue suit and red tie standing at a podium.",
|
||||
"time_sec": 41.13
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Performance (Benchmarked on M5 Max)
|
||||
|
||||
**Model**: LLaVA 7B Q4_0 (Ollama), 1280x720 → 480x270 resize
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Avg per frame | **1.02s** |
|
||||
| Min | 0.81s |
|
||||
| Max | 1.60s |
|
||||
| Throughput | ~1 fps (sequential) |
|
||||
| Concurrent (4 workers) | No speedup — Ollama queues single-GPU |
|
||||
|
||||
**Recommendations**:
|
||||
- Resize to 480x270 before sending reduces token count without quality loss
|
||||
- Batch processing doesn't help (Ollama queues requests to single model instance)
|
||||
- For large video indexing, process frames asynchronously in background
|
||||
- Consider a faster model if throughput becomes critical (e.g., LLaVA-Next 8B, Moondream 1.6B)
|
||||
|
||||
## Frontend Changes (for reference)
|
||||
Once Core API returns `vlm_describe` in agent sources, the frontend will display:
|
||||
- Tool badge: "VLM"
|
||||
- Description text paired with the relevant video frame
|
||||
- Click to jump to that frame in the video player
|
||||
|
||||
## Timeline
|
||||
Not urgent - this is an enhancement for visual search capabilities.
|
||||
Reference in New Issue
Block a user