Files

Momentry Studio 5951aca086 feat: frame-based positioning and mark system foundation

Core Changes:
- Fix SearchView to use start_frame/end_frame directly (no time*fps conversion)
- Add hard_delete support to delete_trace API
- VideoPlayer: Main timeline + Mark system foundation
- Proxy: Add local routes for auth, media, identity-matches, cluster-results
- Add .gitignore to exclude build artifacts and dependencies

Design Documents:
- Multi-track Mark system design (.opencode/plans/)
- Video editing positioning standards research

Files Modified:
- src/views/SearchView.vue: Frame positioning, ensureMinDuration (240 frames)
- src/views/PeopleView.vue: batchDeleteGroups with hard_delete
- src/api/index.ts: delete_trace with hard_delete body
- src/components/VideoPlayer.vue: Timeline + Mark UI
- src-tauri/src/proxy.rs: New local routes
- AGENTS.md: Update documentation

2026-07-24 20:19:47 +08:00

2.8 KiB

Raw Blame History

VLM (Vision Language Model) Tool for Agent Search

Request

Add a vlm_describe tool to the Agent system that can analyze video keyframes via VLM.

Background

We have Ollama with LLaVA running locally (http://localhost:11434). The VLM can describe image content (people, objects, scenes, clothing, text, etc.). We want the Agent to be able to use this as a tool when users ask questions about visual content.

Use Cases

"What is the person wearing in this scene?"
"Find scenes where someone is wearing a red shirt"
"Describe the background of this video"
"What objects are visible in frame 1000?"

API Design

Agent Tool Registration

Add a vlm_describe tool to the Agent's tool registry in Core API.

Tool Definition

{
  "name": "vlm_describe",
  "description": "Analyze a video frame using Vision Language Model. Returns a description of the image content.",
  "parameters": {
    "file_uuid": "string - UUID of the video file",
    "frame": "integer - Frame number to analyze",
    "prompt": "string (optional) - Specific question about the image (default: 'Describe this image in 1-2 sentences.')"
  }
}

Internal Implementation

The Core API should call:

POST http://localhost:11434/api/generate
{
  "model": "llava",
  "prompt": "<user prompt or default>",
  "images": ["<base64 of frame thumbnail>"],
  "stream": false,
  "options": { "num_predict": 80 }
}

To get the frame image, use the existing thumbnail API:

GET /api/v1/file/{file_uuid}/thumbnail?api_key={key}&frame={frame}

→ Returns JPEG bytes → resize to 480x270 → base64 encode → send to Ollama.

Result Format

{
  "tool": "vlm_describe",
  "result": {
    "file_uuid": "...",
    "frame": 1234,
    "description": "A man wearing a blue suit and red tie standing at a podium.",
    "time_sec": 41.13
  }
}

Performance (Benchmarked on M5 Max)

Model: LLaVA 7B Q4_0 (Ollama), 1280x720 → 480x270 resize

Metric	Value
Avg per frame	1.02s
Min	0.81s
Max	1.60s
Throughput	~1 fps (sequential)
Concurrent (4 workers)	No speedup — Ollama queues single-GPU

Recommendations:

Resize to 480x270 before sending reduces token count without quality loss
Batch processing doesn't help (Ollama queues requests to single model instance)
For large video indexing, process frames asynchronously in background
Consider a faster model if throughput becomes critical (e.g., LLaVA-Next 8B, Moondream 1.6B)

Frontend Changes (for reference)

Once Core API returns vlm_describe in agent sources, the frontend will display:

Tool badge: "VLM"
Description text paired with the relevant video frame
Click to jump to that frame in the video player

Timeline

Not urgent - this is an enhancement for visual search capabilities.

2.8 KiB Raw Blame History