momentry_studio/docs/core-api-pose-appearance-endpoint.md

# Pose & Appearance API Endpoints - Technical Specification

## Overview

This document specifies two new Core API endpoints needed for displaying pose skeleton and appearance colors in Momentry Studio's Face Detail Modal.

---

## Endpoints

### 1. Get Pose

```
GET /api/v1/file/:file_uuid/pose?frame=:frame_no
```

**Response:**
```json
{
  "frame": 100,
  "keypoints": [
    { "name": "nose", "x": 993, "y": 372, "confidence": 0.84 },
    { "name": "left_eye", "x": 950, "y": 350, "confidence": 0.91 },
    { "name": "right_eye", "x": 1030, "y": 352, "confidence": 0.89 },
    { "name": "left_ear", "x": 920, "y": 360, "confidence": 0.75 },
    { "name": "right_ear", "x": 1060, "y": 358, "confidence": 0.77 },
    { "name": "left_shoulder", "x": 850, "y": 480, "confidence": 0.88 },
    { "name": "right_shoulder", "x": 1100, "y": 475, "confidence": 0.90 },
    { "name": "left_elbow", "x": 780, "y": 620, "confidence": 0.82 },
    { "name": "right_elbow", "x": 1180, "y": 610, "confidence": 0.85 },
    { "name": "left_wrist", "x": 720, "y": 750, "confidence": 0.78 },
    { "name": "right_wrist", "x": 1240, "y": 740, "confidence": 0.80 },
    { "name": "left_hip", "x": 900, "y": 720, "confidence": 0.86 },
    { "name": "right_hip", "x": 1050, "y": 715, "confidence": 0.87 },
    { "name": "left_knee", "x": 870, "y": 950, "confidence": 0.83 },
    { "name": "right_knee", "x": 1080, "y": 945, "confidence": 0.84 },
    { "name": "left_ankle", "x": 850, "y": 1150, "confidence": 0.79 },
    { "name": "right_ankle", "x": 1100, "y": 1145, "confidence": 0.81 }
  ],
  "pose_class": "standing",
  "confidence": 0.95
}
```

**Fields:**
- `frame` (int): Frame number
- `keypoints` (array): 17 COCO keypoints
  - `name` (string): Keypoint name (see list below)
  - `x` (float): X coordinate in pixels
  - `y` (float): Y coordinate in pixels
  - `confidence` (float, optional): Detection confidence 0-1
- `pose_class` (string): One of `standing`, `sitting`, `kneeling`, `lying`, `prone`, `unknown`
- `confidence` (float, optional): Overall pose classification confidence

**COCO-17 Keypoint Names:**
```
nose, left_eye, right_eye, left_ear, right_ear,
left_shoulder, right_shoulder, left_elbow, right_elbow,
left_wrist, right_wrist, left_hip, right_hip,
left_knee, right_knee, left_ankle, right_ankle
```

---

### 2. Get Appearance

```
GET /api/v1/file/:file_uuid/appearance?frame=:frame_no
```

**Response:**
```json
{
  "frame": 100,
  "dominant_colors": [
    { "rgb": [255, 100, 50], "percentage": 0.35 },
    { "rgb": [50, 150, 200], "percentage": 0.25 },
    { "rgb": [100, 200, 100], "percentage": 0.15 }
  ],
  "hsv_histogram": [
    [/* 30 bins for H */],
    [/* 30 bins for S */],
    [/* 30 bins for V */]
  ]
}
```

**Fields:**
- `frame` (int): Frame number
- `dominant_colors` (array, optional): Top 3-5 dominant colors
  - `rgb` (array): [R, G, B] values 0-255
  - `percentage` (float, optional): Proportion 0-1
- `hsv_histogram` (array, optional): Raw HSV histogram for custom analysis

---

## Data Source

### Option A: Read from JSON files

**Pose file location:**
```
/momentry/output/{file_hash}/{file_hash}.pose.json
```

**Appearance file location:**
```
/momentry/output/{file_hash}/{file_hash}.appearance.json
```

**Expected JSON structure:**
```json
{
  "frames": [
    {
      "frame": 0,
      "keypoints": [...],
      "pose_class": "standing"
    },
    {
      "frame": 1,
      "keypoints": [...],
      "pose_class": "standing"
    }
  ]
}
```

### Option B: Query from TKG

If pose/appearance data is stored in TKG, implement Neo4j query:

```cypher
MATCH (p:Pose {file_uuid: $file_uuid, frame: $frame})
RETURN p.keypoints, p.pose_class, p.confidence
```

---

## Implementation Example (FastAPI)

```python
from fastapi import FastAPI, HTTPException
import json
from pathlib import Path

app = FastAPI()

OUTPUT_BASE = "/momentry/output"

@app.get("/api/v1/file/{file_uuid}/pose")
async def get_pose(file_uuid: str, frame: int):
    pose_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.pose.json")
    
    if not pose_file.exists():
        raise HTTPException(status_code=404, detail="Pose data not found")
    
    with open(pose_file) as f:
        data = json.load(f)
    
    # Find frame
    for frame_data in data.get("frames", []):
        if frame_data.get("frame") == frame:
            return {
                "frame": frame,
                "keypoints": frame_data.get("keypoints", []),
                "pose_class": frame_data.get("pose_class", "unknown"),
                "confidence": frame_data.get("confidence")
            }
    
    raise HTTPException(status_code=404, detail="Frame not found")

@app.get("/api/v1/file/{file_uuid}/appearance")
async def get_appearance(file_uuid: str, frame: int):
    appearance_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.appearance.json")
    
    if not appearance_file.exists():
        raise HTTPException(status_code=404, detail="Appearance data not found")
    
    with open(appearance_file) as f:
        data = json.load(f)
    
    for frame_data in data.get("frames", []):
        if frame_data.get("frame") == frame:
            return {
                "frame": frame,
                "dominant_colors": frame_data.get("dominant_colors", []),
                "hsv_histogram": frame_data.get("hsv_histogram")
            }
    
    raise HTTPException(status_code=404, detail="Frame not found")
```

---

## Mock Data for Testing

If real data is not yet available, use mock response:

```python
@app.get("/api/v1/file/{file_uuid}/pose")
async def get_pose(file_uuid: str, frame: int):
    return {
        "frame": frame,
        "keypoints": [
            {"name": "nose", "x": 100, "y": 50},
            {"name": "left_eye", "x": 90, "y": 45},
            {"name": "right_eye", "x": 110, "y": 45},
            {"name": "left_ear", "x": 80, "y": 50},
            {"name": "right_ear", "x": 120, "y": 50},
            {"name": "left_shoulder", "x": 60, "y": 100},
            {"name": "right_shoulder", "x": 140, "y": 100},
            {"name": "left_elbow", "x": 50, "y": 150},
            {"name": "right_elbow", "x": 150, "y": 150},
            {"name": "left_wrist", "x": 45, "y": 190},
            {"name": "right_wrist", "x": 155, "y": 190},
            {"name": "left_hip", "x": 70, "y": 200},
            {"name": "right_hip", "x": 130, "y": 200},
            {"name": "left_knee", "x": 65, "y": 280},
            {"name": "right_knee", "x": 135, "y": 280},
            {"name": "left_ankle", "x": 60, "y": 350},
            {"name": "right_ankle", "x": 140, "y": 350}
        ],
        "pose_class": "standing"
    }

@app.get("/api/v1/file/{file_uuid}/appearance")
async def get_appearance(file_uuid: str, frame: int):
    return {
        "frame": frame,
        "dominant_colors": [
            {"rgb": [255, 100, 50]},
            {"rgb": [50, 150, 200]}
        ]
    }
```

---

## Studio Integration Status

**Phase 1 (Completed):**
- ✅ TypeScript interfaces (`src/api/types.ts`)
- ✅ API builder cases (`src/api/index.ts`)
- ✅ Store functions (`src/store.ts`)
- ✅ Canvas rendering utilities (`src/utils/poseRenderer.ts`)
- ✅ Mock data for testing (`src/utils/mockPoseData.ts`)

**Phase 2 (Pending - Core Team):**
- ⏳ Implement pose endpoint
- ⏳ Implement appearance endpoint

**Phase 3-4 (After Phase 2):**
- Studio Proxy handlers
- UI integration in Face Detail Modal

---

## Testing

### Manual Test

```bash
curl "http://localhost:3002/api/v1/file/{file_uuid}/pose?frame=100"
curl "http://localhost:3002/api/v1/file/{file_uuid}/appearance?frame=100"
```

### Expected Response Time

- Target: < 100ms per request
- Cacheable: Yes (pose/appearance data doesn't change)

---

## Questions

1. **Data location**: Confirm if `.pose.json` and `.appearance.json` files exist, or if data is in TKG?

2. **Frame alignment**: Confirm that pose/appearance frame numbers align with face `best_face_frame`?

3. **Pose classification**: Do you have pose classification (`standing`, `sitting`, etc.), or just keypoints?

4. **Dominant colors**: Is color extraction already done, or need to implement?

---

## Contact

Studio Team: Ready to integrate once endpoints are available.
Expected Phase 3-4 completion: 1-2 hours after Phase 2 delivery.