在前后端开发联调前的提交20260223

2026-02-23 23:02:20 +08:00
parent 254ccb1e77
commit fafc95e64c
1142 changed files with 10366960 additions and 36957 deletions
--- a/docs/audit/prompt_logs/prompt_log_20260221_211315.md
+++ b/docs/audit/prompt_logs/prompt_log_20260221_211315.md
@@ -0,0 +1,157 @@
+- [P20260221-211315] 2026-02-21 21:13:15 +0800
+  - summary: CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary: --- ## TASK 1: Execute D…
+  - prompt:
+```text
+CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary:
+
+---
+## TASK 1: Execute Dataflow Structure Analysis (Two Phases)
+
+**STATUS**: done
+**USER QUERIES**: Initial request (from context transfer)
+**DETAILS**: Completed in previous conversation. Ran `analyze_dataflow.py` and `gen_dataflow_report.py`. Output at `export/SYSTEM/REPORTS/dataflow_analysis/`.
+**FILEPATHS**: `scripts/ops/analyze_dataflow.py`, `scripts/ops/gen_dataflow_report.py`
+
+## TASK 2: Investigate Two Missing Field Mappings
+
+**STATUS**: done
+**USER QUERIES**: From context transfer + user follow-up asking to scan JSON values
+**DETAILS**:
+- `time_slot_sale` (store_goods_master): API returns it (all values = `'2'`, 692 records across 4 batches), but ODS DDL never had it
+- `commoditycode` (tenant_goods_master): API JSON does NOT return `commoditycode` field (696 records all missing). API returns `commodity_code` instead (single string values like `'10000'`, `'10000028'`). All values are single-element (no multi-value arrays found). ODS has both `commodityCode TEXT` and `commodity_code TEXT` columns.
+- One-off scan script `_scan_json_fields.py` was created and deleted
+- JSON structure: `pages[].response.data.orderGoodsList` for store_goods, `pages[].response.data.tenantGoodsList` for tenant_goods
+**FILEPATHS**: `scripts/ops/_scan_json_fields.py` (deleted)
+
+## TASK 3: Add time_slot_sale + Merge commodity_code (Task 1 from user)
+
+**STATUS**: in-progress
+**USER QUERIES**: User's main request with two sub-tasks
+**DETAILS**:
+
+What was completed:
+1. **Migration script created and executed on test DB** — all 3 ALTER TABLE operations succeeded:
+   - `ods.store_goods_master` ADD `time_slot_sale INTEGER` ✓
+   - `dwd.dim_store_goods_ex` ADD `time_slot_sale INTEGER` ✓
+   - `dwd.dim_tenant_goods_ex.commodity_code_list` changed from `VARCHAR(256)` to `TEXT[]` ✓
+   - Verification confirmed: ODS time_slot_sale=integer, DWD time_slot_sale=integer, commodity_code_list=ARRAY/_text
+
+2. **DDL files updated**:
+   - `db/etl_feiqiu/schemas/ods.sql`: Added `time_slot_sale INTEGER` to store_goods_master + COMMENT ✓
+   - `db/etl_feiqiu/schemas/dwd.sql`: Added `time_slot_sale INTEGER` to dim_store_goods_ex + COMMENT ✓
+   - `db/etl_feiqiu/schemas/dwd.sql`: Changed `commodity_code_list VARCHAR(256)` → `TEXT[]` ✓
+
+3. **FACT_MAPPINGS updated** (partially):
+   - `dwd.dim_store_goods_ex`: Added `("time_slot_sale", "time_slot_sale", None)` ✓
+   - **NOT YET DONE**: `dwd.dim_tenant_goods_ex` commodity_code_list mapping needs update — currently maps from `commodity_code` (single value) but needs to map from `"commodityCode"` (the ODS column that stores the array format `["10000028"]`) and cast to TEXT[]
+
+4. **Migration script file**: `db/etl_feiqiu/migrations/2026-02-21__add_time_slot_sale_merge_commodity_code.sql` ✓
+
+5. **One-off migration runner**: `scripts/ops/_run_migration.py` still exists (needs cleanup)
+
+**NEXT STEPS** (Task 1 remaining):
+- Update FACT_MAPPINGS for `dwd.dim_tenant_goods_ex` — the `commodity_code_list` mapping currently reads `("commodity_code_list", "commodity_code", None)` but needs to be changed to map from `"commodityCode"` column and parse the JSON array `["10000028"]` into a PostgreSQL TEXT[] array. The mapping expression needs a cast like `"string_to_array(trim(both '[]' from \"commodityCode\"), ',')"` or similar approach using the ODS `commodityCode` column which stores `{10000028}` format
+- The ODS loader is schema-aware (`_insert_records_schema_aware` in `ods_tasks.py` line 721) — it dynamically reads columns from DB, so no code change needed for ODS ingestion
+- Clean up `scripts/ops/_run_migration.py`
+- Update `docs/database/` with schema change documentation (db-docs.md rule)
+- The `gen_field_review_doc.py` skip lists may need updating (lines ~908 and ~634) to reflect these fields are now mapped
+
+**FILEPATHS**:
+- `db/etl_feiqiu/migrations/2026-02-21__add_time_slot_sale_merge_commodity_code.sql`
+- `db/etl_feiqiu/schemas/ods.sql`
+- `db/etl_feiqiu/schemas/dwd.sql`
+- `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` (FACT_MAPPINGS at ~line 113)
+- `scripts/ops/_run_migration.py` (to delete)
+
+## TASK 4: DDL vs Database Consistency Check + Migration Archive (Task 2 from user)
+
+**STATUS**: not-started
+**USER QUERIES**: User's second sub-task: "做完任务1后，检查主DDL+种子等，包含ODS DWD DWS INDEX。是否与当前数据库相同，若有遗失和偏差则合并，检查好后，将分支后续修改的DDL归档。"
+**DETAILS**: Compare all DDL files (ODS, DWD, DWS, indexes) against actual test database schema. Find discrepancies, merge missing definitions, then archive branch migration scripts.
+**NEXT STEPS**:
+- Write a script to compare DDL files vs actual DB columns for all schemas (ods, dwd, dws)
+- Check indexes match
+- Check seeds
+- Merge any discrepancies
+- Archive migration scripts from this branch
+**FILEPATHS**: `db/etl_feiqiu/schemas/ods.sql`, `db/etl_feiqiu/schemas/dwd.sql`, `db/etl_feiqiu/schemas/dws.sql`, `db/etl_feiqiu/schemas/` (index files), `db/etl_feiqiu/seeds/`, `db/etl_feiqiu/migrations/`
+
+## USER CORRECTIONS AND INSTRUCTIONS:
+- All output must go to `export/` directory structure via `.env` environment variables
+- Scripts run from `scripts/ops/` — but cwd is currently stuck at `scripts/ops/` in the shell, so use absolute paths like `python C:\NeoZQYY\scripts\ops\script.py`
+- Database testing uses `TEST_DB_DSN` (test_etl_feiqiu), never production
+- All explanatory text in simplified Chinese
+- Python scripts preferred over PowerShell for complex operations
+- ODS loader is schema-aware — reads columns from DB dynamically, no hardcoded column lists needed
+- Audit reminder is pending (root-file, dir:admin-web, dir:backend, dir:db, db-schema-change) — user will manually trigger `/audit`
+- `db-docs.md` rule: any schema change must sync `docs/database/` with change description, compatibility notes, rollback strategy, and 3+ verification SQLs
+- The locked `field_mappings/` directory in `export/SYSTEM/REPORTS/dataflow_analysis/` should be manually deleted after system restart (from previous task)
+
+## Key Architecture Notes:
+- ODS tables use `(id, content_hash)` as composite PK
+- DWD dimension tables use SCD2 with `(pk, scd2_start_time)` as composite PK
+- FACT_MAPPINGS in `dwd_load_task.py` format: `(dwd_column, ods_expression, cast_type)`
+- ODS ingestion: `BaseOdsTask._insert_records_schema_aware()` dynamically reads table columns from DB — adding a column to DDL+DB is sufficient for ODS to start capturing it
+- DWD loading: columns not in FACT_MAPPINGS are auto-mapped by matching column names between ODS and DWD
+
+## Files to read
+- `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` (FACT_MAPPINGS — need to fix commodity_code_list mapping)
+- `db/etl_feiqiu/schemas/dwd.sql` (recently modified)
+- `db/etl_feiqiu/schemas/ods.sql` (recently modified)
+- `db/etl_feiqiu/migrations/2026-02-21__add_time_slot_sale_merge_commodity_code.sql`
+- `scripts/ops/_run_migration.py` (needs cleanup)
+- `scripts/ops/gen_field_review_doc.py` (skip lists may need updating)
+
+USER QUERIES(most recent first):
+1. 任务1：time_slot_sale 在ODS和DWD中添加。commoditycode 和 commodity_code 合并。采用数组方式吧。入库，落地，做好DDL，MAPPING等处理，文档落地等工作任务2：做完任务1后，检查主DDL+种子等，包含ODS  DWD DWS INDEX。是否与当前数据库相同，若有遗失和偏差则合并，检查好后，将分支后续修改的DDL归档。
+2. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+3. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+4. 任务1：time_slot_sale 在ODS和DWD中添加。commoditycode 和 commodity_code 合并。采用数组方式吧。入库，落地，做好DDL，MAPPING等处理，文档落地等工作任务2：做完任务1后，检查主DDL+种子等，包含ODS  DWD DWS INDEX。是否与当前数据库相同，若有遗失和偏差则合并，检查好后，将分支后续修改的DDL归档。
+5. Output:
+Hook execution failed with exit code 1.
+
+Error output:
+[AUDIT REMINDER] Pending audit detected (root-file, dir:admin-web, dir:backend, dir:db, db-schema-change). Run /audit (Manual: Run /audit hook) to sync docs & write audit artifacts. (rate limit: 15min)
+
+
+
+Exit Code: 1
+6. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+7. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+8. time_slot_sale，遍历下Json这个字段，有什么值？关于commodity_code_list 和 commodity_code 问题。我打算进行合并。在操作之前，从JSON遍历检查返回的数组有多个值的情况么？
+9. Output:
+Hook execution failed with exit code 1.
+
+Error output:
+[AUDIT REMINDER] Pending audit detected (root-file, dir:admin-web, dir:backend, dir:db, db-schema-change). Run /audit (Manual: Run /audit hook) to sync docs & write audit artifacts. (rate limit: 15min)
+
+
+
+Exit Code: 1
+10. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+11. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+---
+
+METADATA:
+The previous conversation had 6 messages.
+
+INSTRUCTIONS:
+Continue working until the user query has been fully addressed. Do not ask for clarification - proceed with the work based on the context provided.