微信小程序页面迁移校验之前 P5任务处理之前
This commit is contained in:
94
docs/database/BD_Manual_fix_dim_staff_ex_rankname.md
Normal file
94
docs/database/BD_Manual_fix_dim_staff_ex_rankname.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# BD_Manual:修复 dim_staff_ex 列映射 rankname → rank_name
|
||||
|
||||
> 影响表:`dwd.dim_staff_ex`
|
||||
> ODS 源表:`ods.staff_info_master`
|
||||
> 修复日期:2026-02-26
|
||||
> 代码位置:`apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py`
|
||||
> 触发场景:`FLOW_API_FULL` 执行时 `DWD_LOAD_FROM_ODS` 阶段 dim_staff_ex 加载失败
|
||||
|
||||
---
|
||||
|
||||
## 1. 变更说明
|
||||
|
||||
DWD 加载任务 `DwdLoadTask` 中,`dwd.dim_staff_ex` 的列映射定义错误:
|
||||
|
||||
```python
|
||||
# 修复前(错误)
|
||||
("rank_name", "rankname", None)
|
||||
|
||||
# 修复后(正确)
|
||||
("rank_name", "rank_name", None)
|
||||
```
|
||||
|
||||
映射元组含义:`(dwd_列名, ods_源列名, 类型转换)`。
|
||||
|
||||
ODS 表 `ods.staff_info_master` 的实际列名为 `rank_name`(带下划线),而非 `rankname`。此错误导致 PostgreSQL 报 `UndefinedColumn: 字段 "rankname" 不存在`,dim_staff_ex 的 SCD2 合并在每个窗口段均失败(共 4 次)。
|
||||
|
||||
---
|
||||
|
||||
## 2. 兼容性影响
|
||||
|
||||
| 组件 | 影响 |
|
||||
|------|------|
|
||||
| ETL DWD 层 | dim_staff_ex 恢复正常加载,rank_name 字段将正确从 ODS 映射 |
|
||||
| DWS 层 | 无直接影响(当前无 DWS 任务依赖 dim_staff_ex.rank_name) |
|
||||
| 后端 API | 无影响(后端通过 FDW 读取,表结构未变) |
|
||||
| 小程序 | 无影响 |
|
||||
| DDL | 无变更,表结构不变 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 回滚策略
|
||||
|
||||
此修复仅涉及 Python 代码中的列映射字符串,无 DDL 变更。
|
||||
|
||||
回滚步骤:
|
||||
1. 将 `dwd_load_task.py` 中 dim_staff_ex 映射恢复为 `("rank_name", "rankname", None)`
|
||||
2. 注意:回滚后 dim_staff_ex 将再次无法加载 rank_name 字段
|
||||
|
||||
已加载的数据无需回滚——修复前该字段从未成功写入。
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证 SQL
|
||||
|
||||
```sql
|
||||
-- 验证 1:确认 ODS 源表列名为 rank_name(非 rankname)
|
||||
SELECT column_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = 'ods'
|
||||
AND table_name = 'staff_info_master'
|
||||
AND column_name IN ('rank_name', 'rankname');
|
||||
-- 预期:仅返回 rank_name
|
||||
|
||||
-- 验证 2:确认 DWD 目标表存在 rank_name 列
|
||||
SELECT column_name
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema = 'dwd'
|
||||
AND table_name = 'dim_staff_ex'
|
||||
AND column_name = 'rank_name';
|
||||
-- 预期:返回 1 行
|
||||
|
||||
-- 验证 3:修复后重跑 ETL,检查 dim_staff_ex 是否有 rank_name 非空数据
|
||||
SELECT COUNT(*) AS total,
|
||||
COUNT(rank_name) AS has_rank_name
|
||||
FROM dwd.dim_staff_ex
|
||||
WHERE scd2_is_current = 1;
|
||||
-- 预期:has_rank_name > 0(取决于上游数据是否有值)
|
||||
|
||||
-- 验证 4:对比 ODS 与 DWD 的 rank_name 一致性
|
||||
SELECT s.id AS staff_id, s.rank_name AS ods_rank_name, d.rank_name AS dwd_rank_name
|
||||
FROM ods.staff_info_master s
|
||||
JOIN dwd.dim_staff_ex d ON s.id = d.staff_id AND d.scd2_is_current = 1
|
||||
WHERE s.rank_name IS DISTINCT FROM d.rank_name
|
||||
LIMIT 10;
|
||||
-- 预期:修复并重跑后返回 0 行
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 映射修正记录
|
||||
|
||||
| 日期 | 字段 | 修正内容 |
|
||||
|------|------|---------|
|
||||
| 2026-02-26 | `rank_name` | ODS 源列名从 `rankname` 修正为 `rank_name`,与 `ods.staff_info_master` DDL 一致 |
|
||||
Reference in New Issue
Block a user