diff --git a/CORE_TEAM_RESPONSE.md b/CORE_TEAM_RESPONSE.md new file mode 100644 index 0000000..6c73f6e --- /dev/null +++ b/CORE_TEAM_RESPONSE.md @@ -0,0 +1,177 @@ +# Core Team Response - Pipeline Progress Issue + +**Date**: 2026-07-25 +**Author**: Core Team (OpenCode) +**Status**: ✅ Implemented + +--- + +## Summary + +Core Team has implemented all 3 requested backend fixes: + +| # | Item | Status | +|---|------|--------| +| 1 | Add "running" stage tracking | ✅ Completed | +| 2 | Initialize stages at registration | ✅ Completed | +| 3 | sync-status TKG check | ✅ Completed | + +--- + +## Changes Made + +### 1. "Running" Stage Tracking + +**Problem**: Stages only showed `pending` → `completed`, never `running`. + +**Solution**: Added `publish_pipeline_progress` calls at the start of each stage. + +**Files Modified**: `src/worker/job_worker.rs` + +| Stage | Line | Trigger | +|-------|------|---------| +| processors | ~472 | Job picked up | +| rule1_ingestion | ~1885 | ASRX completed | +| face_tracing | ~1991 | Face completed | +| identity_agent | ~2223 | Seeds exist | +| tkg_nodes + tkg_edges | ~2372 / ~2486 | Face + ASRX completed | +| rule2_ingestion | ~2432 | TKG completed | + +**Example**: +```rust +// Mark face_tracing as running +{ + let mut pp = PipelineProgress::new(&uuid_clone); + pp.update_stage("face_tracing", 0.0, "running", None); + publish_pipeline_progress(redis_clone.as_ref(), &uuid_clone, &pp).await; +} +``` + +--- + +### 2. Initialize Stages at Registration + +**Problem**: Stages were empty when file started processing, frontend couldn't determine active stages. + +**Solution**: Initialize `PipelineProgress` immediately after `create_monitor_job`. + +**Files Modified**: `src/api/files.rs:866-874` + +```rust +if let Ok(job) = auto_state.db.create_monitor_job(&auto_uuid, Some(vp)).await { + // Initialize pipeline progress with all stages pending + if let Ok(redis) = crate::core::db::RedisClient::new() { + let pp = PipelineProgress::new(&auto_uuid); + publish_pipeline_progress(&redis, &auto_uuid, &pp).await; + } + // ... +} +``` + +--- + +### 3. sync-status TKG Check + +**Problem**: `sync_file_status` only checked processors, not TKG or chunks. + +**Solution**: Check TKG nodes, edges, and sentence chunks before marking as `completed`. + +**Files Modified**: `src/api/files.rs:1762-1856` + +**New Response Format**: +```json +{ + "success": true, + "file_uuid": "...", + "status": "completed", + "processors_complete": 5, + "processors_total": 5, + "tkg_nodes": 123, + "tkg_edges": 45, + "sentence_chunks": 67, + "is_music_only": false +} +``` + +**Logic**: +- `completed` = processors_done + tkg_nodes > 0 + sentence_chunks > 0 +- For music-only files: `tkg_edges == 0` is acceptable +- `is_music_only` = processors_done + tkg_nodes > 0 + tkg_edges == 0 + +--- + +## API Changes + +### `/api/v1/file/:file_uuid/sync-status` Response Changes + +| Field | Type | Description | +|-------|------|-------------| +| `tkg_nodes` | i64 | TKG node count | +| `tkg_edges` | i64 | TKG edge count | +| `sentence_chunks` | i64 | Rule 1 sentence chunk count | +| `is_music_only` | bool | True if music-only file (no faces) | + +--- + +## Testing + +### Test Commands + +1. **Register new file and check initial stages**: + ```bash + # After registration, immediately check + curl http://localhost:3002/api/v1/stats/pipeline/{file_uuid} + + # Expected: stages array has 7 elements, all "pending" + ``` + +2. **Check "running" status during processing**: + ```bash + # During processing, check stages + curl http://localhost:3002/api/v1/stats/pipeline/{file_uuid} + + # Expected: at least one stage shows "running" + ``` + +3. **Check sync-status response**: + ```bash + curl -X POST http://localhost:3002/api/v1/file/{file_uuid}/sync-status + + # Expected: includes tkg_nodes, tkg_edges, sentence_chunks, is_music_only + ``` + +--- + +## Deployment + +**Environment**: Production (3002) +**Status**: ✅ Compiled, ready for restart + +**Restart Command**: +```bash +./run-server-3002.sh +``` + +--- + +## Notes + +### Music-Only Files + +For files with no faces (music-only): +- `identity_agent`: marked as "completed (no faces)" +- `tkg_edges`: marked as "completed (no faces)" +- `rule2_ingestion`: marked as "completed (no faces)" + +This is handled in `src/api/scan.rs:1210-1216` and now correctly detected in `sync_file_status`. + +--- + +## Commit + +Branch: main +Commit: (pending - to be committed after verification) + +--- + +**Contact**: Core Team (OpenCode) \ No newline at end of file