在前后端开发联调前的提交20260223

2026-02-23 23:02:20 +08:00
parent 254ccb1e77
commit fafc95e64c
1142 changed files with 10366960 additions and 36957 deletions
--- a/docs/audit/prompt_logs/prompt_log_20260220_131345.md
+++ b/docs/audit/prompt_logs/prompt_log_20260220_131345.md
@@ -0,0 +1,194 @@
+- [P20260220-131345] 2026-02-20 13:13:45 +0800
+  - summary: CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary: --- ## TASK 1: Full Prod…
+  - prompt:
+```text
+CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary:
+
+---
+## TASK 1: Full Production Integration Test (api_full, verify_only, custom window)
+
+- **STATUS**: done
+- **DETAILS**: ETL run `896b9647` completed successfully. 22 ODS tasks: 21 SUCC, 1 FAIL (`ODS_ASSISTANT_LEDGER` — known JSONB bug, fix already in code). `verify_only` mode only executes ODS fetch + verification, skips DWD/DWS/INDEX.
+- **FILEPATHS**: `export/ETL-Connectors/feiqiu/REPORTS/etl_timing_20260220_091414.md`, `export/ETL-Connectors/feiqiu/REPORTS/consistency_report_20260220_091414.md`
+
+## TASK 2: Fix consistency report issues (missing DWD fields + whitelist)
+
+- **STATUS**: done
+- **DETAILS**: `principal_change_amount` fix: Added FACT_MAPPINGS expression. `update_time` fix: Added `KNOWN_NO_SOURCE` whitelist in `consistency_checker.py`.
+- **FILEPATHS**: `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py`, `apps/etl/connectors/feiqiu/quality/consistency_checker.py`
+
+## TASK 3: ODS_ASSISTANT_LEDGER JSONB fix
+
+- **STATUS**: done (code fix applied, awaiting next ETL run to verify)
+- **DETAILS**: `_mark_missing_as_deleted` in `ods_tasks.py` now detects ALL JSONB columns via `cols_info` udt_name and wraps dict/list values with `Json()`.
+- **FILEPATHS**: `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py`
+
+## TASK 4-5: Explain ETL modes and data pipeline
+
+- **STATUS**: done
+- **DETAILS**: Explained increment_only vs increment_verify vs verify_only modes, and full API→ODS→DWD→DWS→INDEX pipeline.
+
+## TASK 6: Remove `pipeline` parameter, rename to `flow` everywhere
+
+- **STATUS**: done
+- **DETAILS**: Complete removal of `pipeline` parameter across entire codebase. All tests passing (ETL unit: 727 passed, monorepo: 171 passed).
+- **KEY DECISIONS**:
+  - `--pipeline-flow` (data_source deprecated param) intentionally KEPT — separate concept
+  - `"pipeline.fetch_root"` and `"pipeline.ingest_source_dir"` are AppConfig keys — NOT renamed
+
+## TASK 7: New `full_window` processing mode
+
+- **STATUS**: done
+- **DETAILS**: Implemented `full_window` processing mode across all 6 files. All tests passing (ETL unit: 727 passed, 5 skipped; monorepo: 171 passed, 2 skipped).
+- **CHANGES MADE**:
+  - `apps/etl/connectors/feiqiu/cli/main.py`: Added `"full_window"` to `PROCESSING_MODE_CHOICES`; added `overrides["run"]["processing_mode"] = args.processing_mode` in `build_cli_overrides()`
+  - `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py`: In `_resolve_window()`, when `processing_mode == "full_window"`, skip `_get_max_fetched_at` fallback — return base window directly
+  - `apps/etl/connectors/feiqiu/orchestration/flow_runner.py`: `full_window` falls through `else` branch (same as `increment_only`), no verification triggered. Docstring updated.
+  - `apps/backend/app/services/cli_builder.py`: Added `"full_window"` to `VALID_PROCESSING_MODES`
+  - `apps/backend/app/routers/tasks.py`: Added `ProcessingModeDefinition` for `full_window`
+  - `apps/admin-web/src/pages/TaskConfig.tsx`: Added `full_window` to `FALLBACK_PROCESSING_MODES`
+- **FILEPATHS**: `apps/etl/connectors/feiqiu/cli/main.py`, `apps/etl/connectors/feiqiu/orchestration/flow_runner.py`, `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py`, `apps/backend/app/services/cli_builder.py`, `apps/backend/app/routers/tasks.py`, `apps/admin-web/src/pages/TaskConfig.tsx`
+
+## TASK 8: Sync full_window changes to docs and admin console
+
+- **STATUS**: done
+- **DETAILS**: Updated all documentation and code comments from "3 种处理模式" to "4 种处理模式", added `full_window` descriptions everywhere.
+- **FILES UPDATED**:
+  - `docs/etl-feiqiu-architecture.md`: Section 4.4 updated to "四种处理模式", added `full_window` row + description; CLI param table updated; also cleaned up stale `--pipeline` references from TASK 6
+  - `apps/backend/app/routers/tasks.py`: docstrings 3→4
+  - `apps/backend/app/schemas/tasks.py`: comment 3→4
+  - `apps/backend/app/services/cli_builder.py`: module docstring 3→4
+  - `apps/backend/tests/test_cli_builder.py`: comments 3→4
+  - `apps/etl/connectors/feiqiu/scripts/debug/debug_orchestration.py`: docstring + message 3→4
+  - `.kiro/specs/admin-web-console/tasks.md`: 3→4
+  - `.kiro/specs/admin-web-console/design.md`: 3→4, added `full_window` to mode list, API table updated
+
+## TASK 9: Web-admin front-end/back-end integration test with full_window mode
+
+- **STATUS**: in-progress
+- **USER QUERIES**: User wants full end-to-end integration test via admin-web:
+  - All stores
+  - `api_full` flow + `full_window` processing mode + "校验前从 API 获取"
+  - Custom window: 2025-11-01 00:00 ~ 2026-02-20 00:00, window split 10 days
+  - `force-full` enabled
+  - "全选常用" (select all common tasks)
+  - Timer mechanism for detailed step-by-step timing report
+  - Black-box testing: verify data source vs DB data/field consistency after completion
+- **DETAILS**:
+  - Frontend (pnpm dev) running on `http://localhost:5173` (process ID 6)
+  - Backend started via `Start-Process` with `uv run uvicorn` on `http://127.0.0.1:8000` (running in background, NOT managed by controlPwshProcess)
+  - Playwright opened admin page, confirmed "全窗口处理" option visible in UI with correct label from API ("全窗口处理 用 API 返回数据的实际时间范围处理全部层，无需校验")
+  - Selected `api_full` flow via Playwright click — confirmed layers show "ODS → DWD → DWS → INDEX"
+  - API calls returning 401 Unauthorized errors — backend requires JWT auth. The frontend uses fallback data when API fails.
+  - Processing modes loaded from API show full names ("仅增量处理" vs fallback "仅增量"), confirming backend API is partially working
+  - **STOPPED AT**: Just clicked `api_full` radio. Still need to:
+    1. Select "全窗口处理" radio
+    2. Check if "校验前从 API 获取" checkbox appears for full_window mode (it may only show for verify_only — need to check frontend logic)
+    3. Switch to "自定义范围" time window mode
+    4. Set dates 2025-11-01 ~ 2026-02-20
+    5. Select "10天" window split
+    6. Check "force-full"
+    7. Click "全选常用" button
+    8. Review CLI command preview
+    9. Click "直接执行"
+    10. Monitor execution, collect timing report
+    11. Run black-box data consistency check
+- **IMPORTANT NOTE**: `full_window` mode was designed to NOT need verification (no cursor drift risk). The user also requested "校验前从 API 获取" — this flag is typically only relevant for `verify_only` mode. The new agent should check if this combination makes sense or if the frontend hides this option for `full_window`.
+- **NEXT STEPS**:
+  1. Continue Playwright automation to configure all parameters listed above
+  2. Check `TaskConfig.tsx` for whether `fetch_before_verify` checkbox is shown for `full_window` mode
+  3. Submit task execution via "直接执行"
+  4. Monitor ETL execution progress
+  5. After completion, collect timing report from `ETL_REPORT_ROOT`
+  6. Run black-box consistency check (use `ConsistencyChecker` or manual SQL queries)
+  7. Output timing summary and black-box test report
+- **FILEPATHS**: `apps/admin-web/src/pages/TaskConfig.tsx`, `apps/etl/connectors/feiqiu/orchestration/flow_runner.py`, `apps/etl/connectors/feiqiu/quality/consistency_checker.py`
+
+## KEY ARCHITECTURE:
+
+- Frontend: React + Vite + Ant Design at `apps/admin-web/` (port 5173)
+- Backend: FastAPI at `apps/backend/` (port 8000, started via `uv run uvicorn`)
+- ETL: `apps/etl/connectors/feiqiu/`
+- ODS PK = `(id, content_hash)` — snapshot mode, content change = new row
+- DWD has no cursor; dim tables use SCD2, fact tables use `fetched_at` window upsert
+- DWS has no cursor; delete-before-insert by date range
+- Four DB connections: `mcp_pg_etl`, `mcp_pg_etl_test`, `mcp_pg_app`, `mcp_pg_app_test`
+- `--pipeline-flow` is a SEPARATE concept (data_source deprecated param) — NOT part of pipeline→flow rename
+- `"pipeline.fetch_root"` and `"pipeline.ingest_source_dir"` are AppConfig configuration keys — NOT renamed
+- `processing_mode` is now written to config via `build_cli_overrides()` so ODS tasks can read it via `self.config.get("run.processing_mode")`
+- `full_window` mode: uses base window (CLI-specified or default 24h lookback), skips `_get_max_fetched_at` fallback, processes all layers, no verification
+
+## USER CORRECTIONS AND INSTRUCTIONS:
+
+- All output text must be in simplified Chinese
+- `fetched_at` is ETL ingestion timestamp, NOT business update time
+- ODS content_hash change = NEW ROW (snapshot), not in-place update
+- `--pipeline-flow` is a separate deprecated param (for data_source), should NOT be renamed
+- Audit reminder pending for high-risk path changes
+- Python scripts preferred over PowerShell for complex operations
+- `test_audit_doc_alignment.py` has a pre-existing deadline flaky test — ignore it
+- Backend must be started with `uv run uvicorn` (not plain `python -m uvicorn`) because fastapi is in uv workspace venv
+- `controlPwshProcess` tool has trouble starting uvicorn — use `Start-Process` PowerShell command instead
+- cwd parameter in executePwsh may not work reliably — use absolute paths
+
+## Files to read
+
+- `apps/admin-web/src/pages/TaskConfig.tsx` (CRITICAL — need to check fetch_before_verify visibility logic for full_window mode, and continue Playwright automation)
+- `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` (understand how full_window flows through run() method)
+- `apps/etl/connectors/feiqiu/quality/consistency_checker.py` (for black-box testing after ETL completes)
+- `apps/backend/app/routers/tasks.py` (backend API for task execution)
+- `apps/backend/app/routers/execution.py` (backend API for submitting/monitoring execution)
+
+USER QUERIES(most recent first):
+1. web-admin 前后端联调：- 全部门店。- api_full-全窗口处理 且 校验前从 API 获取- 自定义范围 2025-11-01 0:00 到 2026-02-20 0:00  窗口切分 10天- force-full- 全选常用功能：- 调通前端逻辑和后端功能。- 保证选中的任务数据处理正确，做好回归测试和数据检验。- 启动计时器机制，对每个步骤和分步骤进行计时，输出详细颗粒度的计时结果文档。之后总结耗时长的项目或步骤。- 启动黑盒测试机制：在顺利完成所有步骤后，以一个黑盒测试者的角度，黑盒方式，检查数据源和落库数据/字段情况是否一致。输出一个黑盒测试报告。
+2. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+3. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+4. web-admin 前后端联调：- 全部门店。- api_full-全窗口处理 且 校验前从 API 获取- 自定义范围 2025-11-01 0:00 到 2026-02-20 0:00  窗口切分 10天- force-full- 全选常用功能：- 调通前端逻辑和后端功能。- 保证选中的任务数据处理正确，做好回归测试和数据检验。- 启动计时器机制，对每个步骤和分步骤进行计时，输出详细颗粒度的计时结果文档。之后总结耗时长的项目或步骤。- 启动黑盒测试机制：在顺利完成所有步骤后，以一个黑盒测试者的角度，黑盒方式，检查数据源和落库数据/字段情况是否一致。输出一个黑盒测试报告。
+5. Output:
+Hook execution failed with exit code 1.
+
+Error output:
+[AUDIT REMINDER] Pending audit detected (root-file, dir:admin-web, dir:backend, dir:db, db-schema-change). Run /audit (Manual: Run /audit hook) to sync docs & write audit artifacts. (rate limit: 15min)
+
+
+
+Exit Code: 1
+6. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+7. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+8. 同步这些改动到文档和管理后台
+9. Output:
+Hook execution failed with exit code 1.
+
+Error output:
+[AUDIT REMINDER] Pending audit detected (root-file, dir:admin-web, dir:backend, dir:db, db-schema-change). Run /audit (Manual: Run /audit hook) to sync docs & write audit artifacts. (rate limit: 15min)
+
+
+
+Exit Code: 1
+10. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+11. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+---
+
+METADATA:
+The previous conversation had 6 messages.
+
+INSTRUCTIONS:
+Continue working until the user query has been fully addressed. Do not ask for clarification - proceed with the work based on the context provided.