# Pose & Appearance API Endpoints - Technical Specification ## Overview This document specifies two new Core API endpoints needed for displaying pose skeleton and appearance colors in Momentry Studio's Face Detail Modal. --- ## Endpoints ### 1. Get Pose ``` GET /api/v1/file/:file_uuid/pose?frame=:frame_no ``` **Response:** ```json { "frame": 100, "keypoints": [ { "name": "nose", "x": 993, "y": 372, "confidence": 0.84 }, { "name": "left_eye", "x": 950, "y": 350, "confidence": 0.91 }, { "name": "right_eye", "x": 1030, "y": 352, "confidence": 0.89 }, { "name": "left_ear", "x": 920, "y": 360, "confidence": 0.75 }, { "name": "right_ear", "x": 1060, "y": 358, "confidence": 0.77 }, { "name": "left_shoulder", "x": 850, "y": 480, "confidence": 0.88 }, { "name": "right_shoulder", "x": 1100, "y": 475, "confidence": 0.90 }, { "name": "left_elbow", "x": 780, "y": 620, "confidence": 0.82 }, { "name": "right_elbow", "x": 1180, "y": 610, "confidence": 0.85 }, { "name": "left_wrist", "x": 720, "y": 750, "confidence": 0.78 }, { "name": "right_wrist", "x": 1240, "y": 740, "confidence": 0.80 }, { "name": "left_hip", "x": 900, "y": 720, "confidence": 0.86 }, { "name": "right_hip", "x": 1050, "y": 715, "confidence": 0.87 }, { "name": "left_knee", "x": 870, "y": 950, "confidence": 0.83 }, { "name": "right_knee", "x": 1080, "y": 945, "confidence": 0.84 }, { "name": "left_ankle", "x": 850, "y": 1150, "confidence": 0.79 }, { "name": "right_ankle", "x": 1100, "y": 1145, "confidence": 0.81 } ], "pose_class": "standing", "confidence": 0.95 } ``` **Fields:** - `frame` (int): Frame number - `keypoints` (array): 17 COCO keypoints - `name` (string): Keypoint name (see list below) - `x` (float): X coordinate in pixels - `y` (float): Y coordinate in pixels - `confidence` (float, optional): Detection confidence 0-1 - `pose_class` (string): One of `standing`, `sitting`, `kneeling`, `lying`, `prone`, `unknown` - `confidence` (float, optional): Overall pose classification confidence **COCO-17 Keypoint Names:** ``` nose, left_eye, right_eye, left_ear, right_ear, left_shoulder, right_shoulder, left_elbow, right_elbow, left_wrist, right_wrist, left_hip, right_hip, left_knee, right_knee, left_ankle, right_ankle ``` --- ### 2. Get Appearance ``` GET /api/v1/file/:file_uuid/appearance?frame=:frame_no ``` **Response:** ```json { "frame": 100, "dominant_colors": [ { "rgb": [255, 100, 50], "percentage": 0.35 }, { "rgb": [50, 150, 200], "percentage": 0.25 }, { "rgb": [100, 200, 100], "percentage": 0.15 } ], "hsv_histogram": [ [/* 30 bins for H */], [/* 30 bins for S */], [/* 30 bins for V */] ] } ``` **Fields:** - `frame` (int): Frame number - `dominant_colors` (array, optional): Top 3-5 dominant colors - `rgb` (array): [R, G, B] values 0-255 - `percentage` (float, optional): Proportion 0-1 - `hsv_histogram` (array, optional): Raw HSV histogram for custom analysis --- ## Data Source ### Option A: Read from JSON files **Pose file location:** ``` /momentry/output/{file_hash}/{file_hash}.pose.json ``` **Appearance file location:** ``` /momentry/output/{file_hash}/{file_hash}.appearance.json ``` **Expected JSON structure:** ```json { "frames": [ { "frame": 0, "keypoints": [...], "pose_class": "standing" }, { "frame": 1, "keypoints": [...], "pose_class": "standing" } ] } ``` ### Option B: Query from TKG If pose/appearance data is stored in TKG, implement Neo4j query: ```cypher MATCH (p:Pose {file_uuid: $file_uuid, frame: $frame}) RETURN p.keypoints, p.pose_class, p.confidence ``` --- ## Implementation Example (FastAPI) ```python from fastapi import FastAPI, HTTPException import json from pathlib import Path app = FastAPI() OUTPUT_BASE = "/momentry/output" @app.get("/api/v1/file/{file_uuid}/pose") async def get_pose(file_uuid: str, frame: int): pose_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.pose.json") if not pose_file.exists(): raise HTTPException(status_code=404, detail="Pose data not found") with open(pose_file) as f: data = json.load(f) # Find frame for frame_data in data.get("frames", []): if frame_data.get("frame") == frame: return { "frame": frame, "keypoints": frame_data.get("keypoints", []), "pose_class": frame_data.get("pose_class", "unknown"), "confidence": frame_data.get("confidence") } raise HTTPException(status_code=404, detail="Frame not found") @app.get("/api/v1/file/{file_uuid}/appearance") async def get_appearance(file_uuid: str, frame: int): appearance_file = Path(f"{OUTPUT_BASE}/{file_uuid}/{file_uuid}.appearance.json") if not appearance_file.exists(): raise HTTPException(status_code=404, detail="Appearance data not found") with open(appearance_file) as f: data = json.load(f) for frame_data in data.get("frames", []): if frame_data.get("frame") == frame: return { "frame": frame, "dominant_colors": frame_data.get("dominant_colors", []), "hsv_histogram": frame_data.get("hsv_histogram") } raise HTTPException(status_code=404, detail="Frame not found") ``` --- ## Mock Data for Testing If real data is not yet available, use mock response: ```python @app.get("/api/v1/file/{file_uuid}/pose") async def get_pose(file_uuid: str, frame: int): return { "frame": frame, "keypoints": [ {"name": "nose", "x": 100, "y": 50}, {"name": "left_eye", "x": 90, "y": 45}, {"name": "right_eye", "x": 110, "y": 45}, {"name": "left_ear", "x": 80, "y": 50}, {"name": "right_ear", "x": 120, "y": 50}, {"name": "left_shoulder", "x": 60, "y": 100}, {"name": "right_shoulder", "x": 140, "y": 100}, {"name": "left_elbow", "x": 50, "y": 150}, {"name": "right_elbow", "x": 150, "y": 150}, {"name": "left_wrist", "x": 45, "y": 190}, {"name": "right_wrist", "x": 155, "y": 190}, {"name": "left_hip", "x": 70, "y": 200}, {"name": "right_hip", "x": 130, "y": 200}, {"name": "left_knee", "x": 65, "y": 280}, {"name": "right_knee", "x": 135, "y": 280}, {"name": "left_ankle", "x": 60, "y": 350}, {"name": "right_ankle", "x": 140, "y": 350} ], "pose_class": "standing" } @app.get("/api/v1/file/{file_uuid}/appearance") async def get_appearance(file_uuid: str, frame: int): return { "frame": frame, "dominant_colors": [ {"rgb": [255, 100, 50]}, {"rgb": [50, 150, 200]} ] } ``` --- ## Studio Integration Status **Phase 1 (Completed):** - ✅ TypeScript interfaces (`src/api/types.ts`) - ✅ API builder cases (`src/api/index.ts`) - ✅ Store functions (`src/store.ts`) - ✅ Canvas rendering utilities (`src/utils/poseRenderer.ts`) - ✅ Mock data for testing (`src/utils/mockPoseData.ts`) **Phase 2 (Pending - Core Team):** - ⏳ Implement pose endpoint - ⏳ Implement appearance endpoint **Phase 3-4 (After Phase 2):** - Studio Proxy handlers - UI integration in Face Detail Modal --- ## Testing ### Manual Test ```bash curl "http://localhost:3002/api/v1/file/{file_uuid}/pose?frame=100" curl "http://localhost:3002/api/v1/file/{file_uuid}/appearance?frame=100" ``` ### Expected Response Time - Target: < 100ms per request - Cacheable: Yes (pose/appearance data doesn't change) --- ## Questions 1. **Data location**: Confirm if `.pose.json` and `.appearance.json` files exist, or if data is in TKG? 2. **Frame alignment**: Confirm that pose/appearance frame numbers align with face `best_face_frame`? 3. **Pose classification**: Do you have pose classification (`standing`, `sitting`, etc.), or just keypoints? 4. **Dominant colors**: Is color extraction already done, or need to implement? --- ## Contact Studio Team: Ready to integrate once endpoints are available. Expected Phase 3-4 completion: 1-2 hours after Phase 2 delivery.