# VLM (Vision Language Model) Tool for Agent Search

## Request
Add a `vlm_describe` tool to the Agent system that can analyze video keyframes via VLM.

## Background
We have Ollama with LLaVA running locally (`http://localhost:11434`). The VLM can describe image content (people, objects, scenes, clothing, text, etc.). We want the Agent to be able to use this as a tool when users ask questions about visual content.

## Use Cases
- "What is the person wearing in this scene?"
- "Find scenes where someone is wearing a red shirt"
- "Describe the background of this video"
- "What objects are visible in frame 1000?"

## API Design

### Agent Tool Registration
Add a `vlm_describe` tool to the Agent's tool registry in Core API.

### Tool Definition
```json
{
  "name": "vlm_describe",
  "description": "Analyze a video frame using Vision Language Model. Returns a description of the image content.",
  "parameters": {
    "file_uuid": "string - UUID of the video file",
    "frame": "integer - Frame number to analyze",
    "prompt": "string (optional) - Specific question about the image (default: 'Describe this image in 1-2 sentences.')"
  }
}
```

### Internal Implementation
The Core API should call:
```
POST http://localhost:11434/api/generate
{
  "model": "llava",
  "prompt": "<user prompt or default>",
  "images": ["<base64 of frame thumbnail>"],
  "stream": false,
  "options": { "num_predict": 80 }
}
```

To get the frame image, use the existing thumbnail API:
```
GET /api/v1/file/{file_uuid}/thumbnail?api_key={key}&frame={frame}
```
→ Returns JPEG bytes → resize to 480x270 → base64 encode → send to Ollama.

### Result Format
```json
{
  "tool": "vlm_describe",
  "result": {
    "file_uuid": "...",
    "frame": 1234,
    "description": "A man wearing a blue suit and red tie standing at a podium.",
    "time_sec": 41.13
  }
}
```

### Performance (Benchmarked on M5 Max)

**Model**: LLaVA 7B Q4_0 (Ollama), 1280x720 → 480x270 resize

| Metric | Value |
|--------|-------|
| Avg per frame | **1.02s** |
| Min | 0.81s |
| Max | 1.60s |
| Throughput | ~1 fps (sequential) |
| Concurrent (4 workers) | No speedup — Ollama queues single-GPU |

**Recommendations**:
- Resize to 480x270 before sending reduces token count without quality loss
- Batch processing doesn't help (Ollama queues requests to single model instance)
- For large video indexing, process frames asynchronously in background
- Consider a faster model if throughput becomes critical (e.g., LLaVA-Next 8B, Moondream 1.6B)

## Frontend Changes (for reference)
Once Core API returns `vlm_describe` in agent sources, the frontend will display:
- Tool badge: "VLM"
- Description text paired with the relevant video frame
- Click to jump to that frame in the video player

## Timeline
Not urgent - this is an enhancement for visual search capabilities.