在前后端开发联调前的提交20260223

2026-02-23 23:02:20 +08:00
parent 254ccb1e77
commit fafc95e64c
1142 changed files with 10366960 additions and 36957 deletions
--- a/docs/audit/prompt_logs/prompt_log_20260221_205154.md
+++ b/docs/audit/prompt_logs/prompt_log_20260221_205154.md
@@ -0,0 +1,130 @@
+- [P20260221-205154] 2026-02-21 20:51:54 +0800
+  - summary: CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary: --- ## TASK 1: Execute D…
+  - prompt:
+```text
+CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary:
+
+---
+## TASK 1: Execute Dataflow Structure Analysis (Two Phases)
+
+**STATUS**: done
+**USER QUERIES**: 1 (initial request to run analyze_dataflow.py and gen_dataflow_report.py)
+**DETAILS**:
+- Phase 1 (Data Collection): Ran `python scripts/ops/analyze_dataflow.py` — collected 22 ODS tables, 3388 records, date range 2025-11-23 ~ 2026-02-21 (adaptive expansion: 10→30→90 days)
+- Phase 2 (Report Generation): Ran `python scripts/ops/gen_dataflow_report.py` — generated 526.1 KB Markdown report at `export/SYSTEM/REPORTS/dataflow_analysis/dataflow_2026-02-21_154548.md`
+- Major blocker: Windows file lock on `field_mappings/` directory (from previous run). Could not delete/rename/iterate. Fixed by adding fallback logic to use `field_mappings_new/` directory when original is locked.
+- Code changes made to handle Windows file locks:
+  - `scripts/ops/dataflow_analyzer.py`: Added `_ensure_writable_dir()` helper in `dump_collection_results()` that tries original dir, falls back to `{name}_new/`. Also added retry logic in `_write_json()`.
+  - `scripts/ops/gen_dataflow_report.py`: Added `_fm_dir` variable at top of `generate_report()` that detects `field_mappings_new/` and uses it. Updated `_write_field_diff_report()` and `_write_source_file_manifest()` to accept `fm_dir` parameter.
+- Output directory structure: `export/SYSTEM/REPORTS/dataflow_analysis/` contains: json_trees/, db_schemas/, field_mappings_new/ (actual data), field_mappings/ (locked empty dir), bd_descriptions/, collection_manifest.json
+- Temporary script `scripts/ops/_clean_dataflow.py` was created and deleted.
+**FILEPATHS**: `scripts/ops/analyze_dataflow.py`, `scripts/ops/gen_dataflow_report.py`, `scripts/ops/dataflow_analyzer.py`, `export/SYSTEM/REPORTS/dataflow_analysis/collection_manifest.json`
+
+## TASK 2: Investigate Two Missing Field Mappings
+
+**STATUS**: in-progress
+**USER QUERIES**: Last user message asking about `time_slot_sale` (store_goods_master) and `commoditycode` (tenant_goods_master)
+**DETAILS**:
+Database and DDL investigation completed. Results:
+
+1. **`time_slot_sale`** (store_goods_master, API→ODS unmapped):
+   - ODS column: **does NOT exist** in `ods.store_goods_master` (confirmed by DB query and DDL grep — zero hits in any .sql file)
+   - DWD column: does not exist
+   - FACT_MAPPINGS: not present
+   - The API returns `time_slot_sale` (example value: `2`) but it was never added to ODS DDL
+   - `scripts/ops/gen_field_review_doc.py` line ~908 explicitly lists it as skipped: "ODS 列不存在。需确认 API 是否返回该字段，如有则需先更新 ODS DDL"
+
+2. **`commoditycode`** (tenant_goods_master, ODS→DWD unmapped):
+   - ODS column: **EXISTS** as `text` type in `ods.tenant_goods_master` (confirmed by DB query, DDL in `ods.sql` line 1479)
+   - ODS sample values: `{10000028}`, `{10000002}`, etc. (curly-brace wrapped arrays)
+   - DWD column: **does NOT exist** in any `dwd.*tenant_goods*` table
+   - FACT_MAPPINGS: not present
+   - `scripts/ops/gen_field_review_doc.py` line ~634 explicitly lists it as skipped: "冗余字段。DWD 已有 commodity_code（自动映射）+ commodity_code_list（FACT_MAPPINGS 映射自 ODS commodity_code）覆盖该语义"
+
+- A one-off check script `scripts/ops/_check_two_fields.py` was created and still exists (should be cleaned up)
+- The user asked "这两个字段的确不存在么？之前没有被DDL加入么？" — the investigation results were gathered but **no response was delivered to the user yet**
+
+**NEXT STEPS**:
+- Deliver the investigation findings to the user in a clear summary
+- Clean up `scripts/ops/_check_two_fields.py` (one-off script)
+- If user wants to add these fields: `time_slot_sale` needs ODS DDL + DWD DDL + FACT_MAPPINGS; `commoditycode` is intentionally skipped as redundant (already covered by `commodity_code` + `commodity_code_list`)
+
+**FILEPATHS**: `scripts/ops/_check_two_fields.py`, `db/etl_feiqiu/schemas/ods.sql`, `scripts/ops/gen_field_review_doc.py`, `scripts/ops/export_dwd_field_review.py`
+
+## USER CORRECTIONS AND INSTRUCTIONS:
+- All output must go to `export/` directory structure via `.env` environment variables
+- Scripts run from `scripts/ops/` with cwd set to that directory
+- Database testing uses `TEST_DB_DSN` (test_etl_feiqiu), never production
+- All explanatory text in simplified Chinese
+- Python scripts preferred over PowerShell for complex operations
+- `scripts/ops/` scripts use `_env_paths.get_output_path()` for path resolution
+- The locked `field_mappings/` directory in `export/SYSTEM/REPORTS/dataflow_analysis/` should be manually deleted after system restart
+
+## Files to read
+- `scripts/ops/_check_two_fields.py`
+- `scripts/ops/gen_field_review_doc.py`
+- `db/etl_feiqiu/schemas/ods.sql`
+- `scripts/ops/dataflow_analyzer.py`
+- `scripts/ops/gen_dataflow_report.py`
+
+USER QUERIES(most recent first):
+1. 1.1.20 store_goods_master 字段差异明细API→ODS 未映射（平层） — 1 个#	JSON 字段	示例值	说明	状态1	time_slot_sale	2		⚠️ 未映射1.1.22 tenant_goods_master 字段差异明细ODS→DWD 未映射 — 1 个#	ODS 列	说明	状态1	commoditycode	（待补充）	⚠️ 无 DWD 目标这两个字段的确不存在么？之前没有被DDL加入么？
+2. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+3. Output:
+Command executed successfully with no output.
+
+Exit Code: 0
+4. 1.1.20 store_goods_master 字段差异明细API→ODS 未映射（平层） — 1 个#	JSON 字段	示例值	说明	状态1	time_slot_sale	2		⚠️ 未映射1.1.22 tenant_goods_master 字段差异明细ODS→DWD 未映射 — 1 个#	ODS 列	说明	状态1	commoditycode	（待补充）	⚠️ 无 DWD 目标这两个字段的确不存在么？之前没有被DDL加入么？
+5. <source-event>
+    The user manually invoked this action
+    The user is focued on the following file: docs/database/etl_feiqiu_schema_migration.md
+    The user has the following paths open: docs/database/etl_feiqiu_schema_migration.md
+</source-event>
+
+执行数据流结构分析，按以下步骤完成。若发现已完成或有历史任务痕迹则清空，重新执行：
+
+第一阶段：数据采集
+1. 运行 `python scripts/ops/analyze_dataflow.py` 完成数据采集（如需指定日期范围，加 --date-from / --date-to 参数）
+2. 确认采集结果已落盘，包括：
+   - json_trees/（含 samples 多示例值）
+   - db_schemas/
+   - field_mappings/（三层映射 + 锚点）
+   - bd_descriptions/（BD_manual 业务描述）
+   - collection_manifest.json（含 json_field_count、date_from、date_to）
+
+第二阶段：报告生成
+3. 运行 `python scripts/ops/gen_dataflow_report.py` 生成 Markdown 报告
+4. 报告包含以下增强内容：
+   - 报告头含 API 请求日期范围（date_from ~ date_to）和 JSON 数据总量
+   - 总览表含 API JSON 字段数列
+   - 1.1 API↔ODS↔DWD 字段对比差异报告（白名单字段折叠汇总，不展开详细表格行）
+   - 2.3 覆盖率表含业务描述列
+   - API 源字段表含业务描述列 + 多示例值（枚举值解释）
+   - ODS 表结构含业务描述列 + 上下游双向映射锚点链接
+   - DWD 表结构含业务描述列 + ODS 来源锚点链接
+5. 输出文件路径和关键统计摘要
+
+白名单规则（v4）：
+- ETL 元数据列（source_file, source_endpoint, fetched_at, payload, content_hash）
+- DWD 维表 SCD2 管理列（valid_from, valid_to, is_current, etl_loaded_at, etl_batch_id）
+- API siteProfile 嵌套对象字段
+- 白名单字段仍正常参与检查和统计，仅在报告中折叠显示并注明原因
+
+注意：当前仅分析飞球（feiqiu）连接器。未来新增连接器时，应自动发现并纳入分析范围。
+6. <implicit-rules>## Implicit Rules
+Focus on creating a new spec file or identifying an existing spec to update. 
+If starting a new spec, create a requirements.md file in the .kiro/specs directory with clear user stories and acceptance criteria. 
+If working with an existing spec, review the current requirements and suggest improvements if needed. 
+Do not make direct code changes yet. First establish or review the spec file that will guide our implementation.</implicit-rules>
+---
+
+METADATA:
+The previous conversation had 4 messages.
+
+INSTRUCTIONS:
+Continue working until the user query has been fully addressed. Do not ask for clarification - proceed with the work based on the context provided.
+IMPORTANT: you need to read from the files to Read section
+```