momentry_studio/docs/core-api-pose-appearance-endpoint.md

Pose & Appearance API Endpoints - Technical Specification

Overview

This document specifies two new Core API endpoints needed for displaying pose skeleton and appearance colors in Momentry Studio's Face Detail Modal.

---

Endpoints

1. Get Pose

GET /api/v1/file/:file_uuid/pose?frame=:frame_no

Response:

{
  "frame": 100,
  "keypoints": [
    { "name": "nose", "x": 993, "y": 372, "confidence": 0.84 },
    { "name": "left_eye", "x": 950, "y": 350, "confidence": 0.91 },
    { "name": "right_eye", "x": 1030, "y": 352, "confidence": 0.89 },
    { "name": "left_ear", "x": 920, "y": 360, "confidence": 0.75 },
    { "name": "right_ear", "x": 1060, "y": 358, "confidence": 0.77 },
    { "name": "left_shoulder", "x": 850, "y": 480, "confidence": 0.88 },
    { "name": "right_shoulder", "x": 1100, "y": 475, "confidence": 0.90 },
    { "name": "left_elbow", "x": 780, "y": 620, "confidence": 0.82 },
    { "name": "right_elbow", "x": 1180, "y": 610, "confidence": 0.85 },
    { "name": "left_wrist", "x": 720, "y": 750, "confidence": 0.78 },
    { "name": "right_wrist", "x": 1240, "y": 740, "confidence": 0.80 },
    { "name": "left_hip", "x": 900, "y": 720, "confidence": 0.86 },
    { "name": "right_hip", "x": 1050, "y": 715, "confidence": 0.87 },
    { "name": "left_knee", "x": 870, "y": 950, "confidence": 0.83 },
    { "name": "right_knee", "x": 1080, "y": 945, "confidence": 0.84 },
    { "name": "left_ankle", "x": 850, "y": 1150, "confidence": 0.79 },
    { "name": "right_ankle", "x": 1100, "y": 1145, "confidence": 0.81 }
  ],
  "pose_class": "standing",
  "confidence": 0.95
}

Fields:

• frame (int): Frame number
• keypoints (array): 17 COCO keypoints
  • name (string): Keypoint name (see list below)
  • x (float): X coordinate in pixels
  • y (float): Y coordinate in pixels
  • confidence (float, optional): Detection confidence 0-1
• pose_class (string): One of standing, sitting, kneeling, lying, prone, unknown
• confidence (float, optional): Overall pose classification confidence

COCO-17 Keypoint Names:

nose, left_eye, right_eye, left_ear, right_ear,
left_shoulder, right_shoulder, left_elbow, right_elbow,
left_wrist, right_wrist, left_hip, right_hip,
left_knee, right_knee, left_ankle, right_ankle

---

2. Get Appearance

GET /api/v1/file/:file_uuid/appearance?frame=:frame_no

Response:

{
  "frame": 100,
  "dominant_colors": [
    { "rgb": [255, 100, 50], "percentage": 0.35 },
    { "rgb": [50, 150, 200], "percentage": 0.25 },
    { "rgb": [100, 200, 100], "percentage": 0.15 }
  ],
  "hsv_histogram": [
    [/* 30 bins for H */],
    [/* 30 bins for S */],
    [/* 30 bins for V */]
  ]
}

Fields:

• frame (int): Frame number
• dominant_colors (array, optional): Top 3-5 dominant colors
  • rgb (array): [R, G, B] values 0-255
  • percentage (float, optional): Proportion 0-1
• hsv_histogram (array, optional): Raw HSV histogram for custom analysis

---

Data Source

Option A: Read from JSON files

Pose file location:

/momentry/output/{file_hash}/{file_hash}.pose.json

Appearance file location:

/momentry/output/{file_hash}/{file_hash}.appearance.json

Expected JSON structure:

{
  "frames": [
    {
      "frame": 0,
      "keypoints": [...],
      "pose_class": "standing"
    },
    {
      "frame": 1,
      "keypoints": [...],
      "pose_class": "standing"
    }
  ]
}

Option B: Query from TKG

If pose/appearance data is stored in TKG, implement Neo4j query:

MATCH (p:Pose {file_uuid: $file_uuid, frame: $frame})
RETURN p.keypoints, p.pose_class, p.confidence

---

Implementation Example (FastAPI)

from fastapi import FastAPI, HTTPException
import json
from pathlib import Path

app = FastAPI()

OUTPUT_BASE = "/momentry/output"

@app.get("/api/v1/file/{file_uuid}/pose")
async def get_pose(file_uuid: str, frame: int):
    pose_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.pose.json")
    
    if not pose_file.exists():
        raise HTTPException(status_code=404, detail="Pose data not found")
    
    with open(pose_file) as f:
        data = json.load(f)
    
    # Find frame
    for frame_data in data.get("frames", []):
        if frame_data.get("frame") == frame:
            return {
                "frame": frame,
                "keypoints": frame_data.get("keypoints", []),
                "pose_class": frame_data.get("pose_class", "unknown"),
                "confidence": frame_data.get("confidence")
            }
    
    raise HTTPException(status_code=404, detail="Frame not found")

@app.get("/api/v1/file/{file_uuid}/appearance")
async def get_appearance(file_uuid: str, frame: int):
    appearance_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.appearance.json")
    
    if not appearance_file.exists():
        raise HTTPException(status_code=404, detail="Appearance data not found")
    
    with open(appearance_file) as f:
        data = json.load(f)
    
    for frame_data in data.get("frames", []):
        if frame_data.get("frame") == frame:
            return {
                "frame": frame,
                "dominant_colors": frame_data.get("dominant_colors", []),
                "hsv_histogram": frame_data.get("hsv_histogram")
            }
    
    raise HTTPException(status_code=404, detail="Frame not found")

---

Mock Data for Testing

If real data is not yet available, use mock response:

@app.get("/api/v1/file/{file_uuid}/pose")
async def get_pose(file_uuid: str, frame: int):
    return {
        "frame": frame,
        "keypoints": [
            {"name": "nose", "x": 100, "y": 50},
            {"name": "left_eye", "x": 90, "y": 45},
            {"name": "right_eye", "x": 110, "y": 45},
            {"name": "left_ear", "x": 80, "y": 50},
            {"name": "right_ear", "x": 120, "y": 50},
            {"name": "left_shoulder", "x": 60, "y": 100},
            {"name": "right_shoulder", "x": 140, "y": 100},
            {"name": "left_elbow", "x": 50, "y": 150},
            {"name": "right_elbow", "x": 150, "y": 150},
            {"name": "left_wrist", "x": 45, "y": 190},
            {"name": "right_wrist", "x": 155, "y": 190},
            {"name": "left_hip", "x": 70, "y": 200},
            {"name": "right_hip", "x": 130, "y": 200},
            {"name": "left_knee", "x": 65, "y": 280},
            {"name": "right_knee", "x": 135, "y": 280},
            {"name": "left_ankle", "x": 60, "y": 350},
            {"name": "right_ankle", "x": 140, "y": 350}
        ],
        "pose_class": "standing"
    }

@app.get("/api/v1/file/{file_uuid}/appearance")
async def get_appearance(file_uuid: str, frame: int):
    return {
        "frame": frame,
        "dominant_colors": [
            {"rgb": [255, 100, 50]},
            {"rgb": [50, 150, 200]}
        ]
    }

---

Studio Integration Status

Phase 1 (Completed):

• ✅ TypeScript interfaces (src/api/types.ts)
• ✅ API builder cases (src/api/index.ts)
• ✅ Store functions (src/store.ts)
• ✅ Canvas rendering utilities (src/utils/poseRenderer.ts)
• ✅ Mock data for testing (src/utils/mockPoseData.ts)

Phase 2 (Pending - Core Team):

• ⏳ Implement pose endpoint
• ⏳ Implement appearance endpoint

Phase 3-4 (After Phase 2):

• Studio Proxy handlers
• UI integration in Face Detail Modal

---

Testing

Manual Test

curl "http://localhost:3002/api/v1/file/{file_uuid}/pose?frame=100"
curl "http://localhost:3002/api/v1/file/{file_uuid}/appearance?frame=100"

Expected Response Time

• Target: < 100ms per request
• Cacheable: Yes (pose/appearance data doesn't change)

---

Questions

1. Data location: Confirm if .pose.json and .appearance.json files exist, or if data is in TKG?

2. Frame alignment: Confirm that pose/appearance frame numbers align with face best_face_frame?

3. Pose classification: Do you have pose classification (standing, sitting, etc.), or just keypoints?

4. Dominant colors: Is color extraction already done, or need to implement?

---

Contact

Studio Team: Ready to integrate once endpoints are available. Expected Phase 3-4 completion: 1-2 hours after Phase 2 delivery.