Momentry Studio
5951aca086
Core Changes: - Fix SearchView to use start_frame/end_frame directly (no time*fps conversion) - Add hard_delete support to delete_trace API - VideoPlayer: Main timeline + Mark system foundation - Proxy: Add local routes for auth, media, identity-matches, cluster-results - Add .gitignore to exclude build artifacts and dependencies Design Documents: - Multi-track Mark system design (.opencode/plans/) - Video editing positioning standards research Files Modified: - src/views/SearchView.vue: Frame positioning, ensureMinDuration (240 frames) - src/views/PeopleView.vue: batchDeleteGroups with hard_delete - src/api/index.ts: delete_trace with hard_delete body - src/components/VideoPlayer.vue: Timeline + Mark UI - src-tauri/src/proxy.rs: New local routes - AGENTS.md: Update documentation
91 lines
2.8 KiB
Markdown
91 lines
2.8 KiB
Markdown
# VLM (Vision Language Model) Tool for Agent Search
|
|
|
|
## Request
|
|
Add a `vlm_describe` tool to the Agent system that can analyze video keyframes via VLM.
|
|
|
|
## Background
|
|
We have Ollama with LLaVA running locally (`http://localhost:11434`). The VLM can describe image content (people, objects, scenes, clothing, text, etc.). We want the Agent to be able to use this as a tool when users ask questions about visual content.
|
|
|
|
## Use Cases
|
|
- "What is the person wearing in this scene?"
|
|
- "Find scenes where someone is wearing a red shirt"
|
|
- "Describe the background of this video"
|
|
- "What objects are visible in frame 1000?"
|
|
|
|
## API Design
|
|
|
|
### Agent Tool Registration
|
|
Add a `vlm_describe` tool to the Agent's tool registry in Core API.
|
|
|
|
### Tool Definition
|
|
```json
|
|
{
|
|
"name": "vlm_describe",
|
|
"description": "Analyze a video frame using Vision Language Model. Returns a description of the image content.",
|
|
"parameters": {
|
|
"file_uuid": "string - UUID of the video file",
|
|
"frame": "integer - Frame number to analyze",
|
|
"prompt": "string (optional) - Specific question about the image (default: 'Describe this image in 1-2 sentences.')"
|
|
}
|
|
}
|
|
```
|
|
|
|
### Internal Implementation
|
|
The Core API should call:
|
|
```
|
|
POST http://localhost:11434/api/generate
|
|
{
|
|
"model": "llava",
|
|
"prompt": "<user prompt or default>",
|
|
"images": ["<base64 of frame thumbnail>"],
|
|
"stream": false,
|
|
"options": { "num_predict": 80 }
|
|
}
|
|
```
|
|
|
|
To get the frame image, use the existing thumbnail API:
|
|
```
|
|
GET /api/v1/file/{file_uuid}/thumbnail?api_key={key}&frame={frame}
|
|
```
|
|
→ Returns JPEG bytes → resize to 480x270 → base64 encode → send to Ollama.
|
|
|
|
### Result Format
|
|
```json
|
|
{
|
|
"tool": "vlm_describe",
|
|
"result": {
|
|
"file_uuid": "...",
|
|
"frame": 1234,
|
|
"description": "A man wearing a blue suit and red tie standing at a podium.",
|
|
"time_sec": 41.13
|
|
}
|
|
}
|
|
```
|
|
|
|
### Performance (Benchmarked on M5 Max)
|
|
|
|
**Model**: LLaVA 7B Q4_0 (Ollama), 1280x720 → 480x270 resize
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Avg per frame | **1.02s** |
|
|
| Min | 0.81s |
|
|
| Max | 1.60s |
|
|
| Throughput | ~1 fps (sequential) |
|
|
| Concurrent (4 workers) | No speedup — Ollama queues single-GPU |
|
|
|
|
**Recommendations**:
|
|
- Resize to 480x270 before sending reduces token count without quality loss
|
|
- Batch processing doesn't help (Ollama queues requests to single model instance)
|
|
- For large video indexing, process frames asynchronously in background
|
|
- Consider a faster model if throughput becomes critical (e.g., LLaVA-Next 8B, Moondream 1.6B)
|
|
|
|
## Frontend Changes (for reference)
|
|
Once Core API returns `vlm_describe` in agent sources, the frontend will display:
|
|
- Tool badge: "VLM"
|
|
- Description text paired with the relevant video frame
|
|
- Click to jump to that frame in the video player
|
|
|
|
## Timeline
|
|
Not urgent - this is an enhancement for visual search capabilities.
|