在前后端开发联调前 的提交20260223
This commit is contained in:
0
export/SYSTEM/LOGS/.gitkeep
Normal file
0
export/SYSTEM/LOGS/.gitkeep
Normal file
178
export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md
Normal file
178
export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# DWS_ASSISTANT_DAILY BUG 修复报告
|
||||
|
||||
> 生成时间:2026-02-21 19:13:11
|
||||
> 执行 run_uuid:4ba9d2d365ee4a858f1c4104b1942dc2
|
||||
> 执行开始:2026-02-21 15:29:20
|
||||
|
||||
---
|
||||
|
||||
## 1. BUG 概述
|
||||
|
||||
ETL 执行过程中 `DWS_ASSISTANT_DAILY` 任务失败,根因是 `assistant_daily_task.py` 中
|
||||
`_extract_trash_records` 方法的 SQL 引用了 `dwd.dwd_assistant_trash_event` 表中不存在的字段。
|
||||
|
||||
### 错误信息
|
||||
|
||||
```
|
||||
psycopg2.errors.UndefinedColumn: 错误: 字段 "assistant_service_id" 不存在
|
||||
LINE 3: assistant_service_id,
|
||||
^
|
||||
```
|
||||
|
||||
### 级联影响
|
||||
|
||||
`DWS_ASSISTANT_DAILY` 失败后,psycopg2 连接进入 `InFailedSqlTransaction` 状态,
|
||||
级联导致以下 8 个任务全部失败:
|
||||
|
||||
| # | 任务代码 | 失败原因 |
|
||||
|---|---------|---------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | 根因:UndefinedColumn |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | InFailedSqlTransaction(级联) |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | InFailedSqlTransaction(级联) |
|
||||
| 4 | DWS_ASSISTANT_SALARY | InFailedSqlTransaction(级联) |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | InFailedSqlTransaction(级联) |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | InFailedSqlTransaction(级联) |
|
||||
| 7 | ODS_PAYMENT | InFailedSqlTransaction(级联) |
|
||||
| 8 | ODS_REFUND | InFailedSqlTransaction(级联) |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | InFailedSqlTransaction(级联) |
|
||||
|
||||
从 `ODS_TABLE_USE` 开始,task_executor 的连接恢复机制生效,后续任务恢复正常执行。
|
||||
|
||||
---
|
||||
|
||||
## 2. 根因分析
|
||||
|
||||
### 2.1 错误 SQL(修复前)
|
||||
|
||||
```sql
|
||||
SELECT assistant_service_id, trash_seconds, trash_reason, trash_time
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s AND DATE(trash_time) >= %s AND DATE(trash_time) <= %s
|
||||
```
|
||||
|
||||
### 2.2 `dwd_assistant_trash_event` 实际表结构
|
||||
|
||||
| 字段名 | 类型 | 说明 |
|
||||
|--------|------|------|
|
||||
| assistant_trash_event_id | BIGINT (PK) | 废除事件 ID |
|
||||
| site_id | BIGINT | 门店 ID |
|
||||
| table_id | BIGINT | 台桌 ID |
|
||||
| table_area_id | BIGINT | 区域 ID |
|
||||
| assistant_no | VARCHAR(32) | 助教编号 |
|
||||
| assistant_name | VARCHAR(64) | 助教姓名 |
|
||||
| charge_minutes_raw | INTEGER | 废除时长(分钟) |
|
||||
| abolish_amount | NUMERIC(18,2) | 废除金额 |
|
||||
| trash_reason | VARCHAR(255) | 废除原因 |
|
||||
| create_time | TIMESTAMPTZ | 废除时间 |
|
||||
| tenant_id | BIGINT | 租户 ID |
|
||||
|
||||
### 2.3 字段映射错误
|
||||
|
||||
| 错误引用 | 实际字段 | 说明 |
|
||||
|----------|---------|------|
|
||||
| `assistant_service_id` | `assistant_trash_event_id` | PK 名称不同 |
|
||||
| `trash_seconds` | `charge_minutes_raw` | 单位不同(分钟 vs 秒) |
|
||||
| `trash_time` | `create_time` | 字段名不同 |
|
||||
|
||||
### 2.4 深层设计缺陷
|
||||
|
||||
废除表 `dwd_assistant_trash_event` 没有 `assistant_service_id` 外键,
|
||||
无法与服务记录表 `dwd_assistant_service_log` 做 1:1 关联。
|
||||
|
||||
原代码的 `_build_trash_index` 用 `assistant_service_id` 做 key 构建索引,
|
||||
`_aggregate_by_assistant_date` 用 `service_id in trash_index` 判断服务是否被废除。
|
||||
即使 SQL 字段名修正后,这个匹配逻辑在设计上也是无效的——两个 ID 不同源。
|
||||
|
||||
---
|
||||
|
||||
## 3. 修复方案
|
||||
|
||||
### 3.1 文件
|
||||
|
||||
`apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py`
|
||||
|
||||
### 3.2 修改点(共 4 处)
|
||||
|
||||
#### (1) `_extract_trash_records` — SQL 字段名修正
|
||||
|
||||
```sql
|
||||
-- 修复后
|
||||
SELECT
|
||||
assistant_trash_event_id,
|
||||
charge_minutes_raw * 60 AS trash_seconds,
|
||||
trash_reason,
|
||||
create_time AS trash_time,
|
||||
table_id,
|
||||
assistant_name
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s
|
||||
AND DATE(create_time) >= %s
|
||||
AND DATE(create_time) <= %s
|
||||
```
|
||||
|
||||
#### (2) `_extract_service_records` — JOIN _ex 表取 is_trash
|
||||
|
||||
```sql
|
||||
-- 新增 LEFT JOIN 和 is_trash 字段
|
||||
SELECT
|
||||
asl.assistant_service_id,
|
||||
...
|
||||
DATE(asl.start_use_time) AS service_date,
|
||||
COALESCE(ex.is_trash, 0) AS is_trash
|
||||
FROM dwd.dwd_assistant_service_log asl
|
||||
LEFT JOIN dwd.dwd_assistant_service_log_ex ex
|
||||
ON asl.assistant_service_id = ex.assistant_service_id
|
||||
WHERE asl.site_id = %s
|
||||
AND DATE(asl.start_use_time) >= %s
|
||||
AND DATE(asl.start_use_time) <= %s
|
||||
AND asl.is_delete = 0
|
||||
```
|
||||
|
||||
#### (3) `_build_trash_index` — key 改为 assistant_trash_event_id
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
service_id = record.get('assistant_service_id')
|
||||
|
||||
# 修复后
|
||||
event_id = record.get('assistant_trash_event_id')
|
||||
```
|
||||
|
||||
#### (4) `_aggregate_by_assistant_date` — 废除判断改用 is_trash
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
is_trashed = service_id in trash_index
|
||||
|
||||
# 修复后
|
||||
is_trashed = bool(record.get('is_trash', 0))
|
||||
```
|
||||
|
||||
废除时长也从 `trash_index[service_id]` 改为直接用 `income_seconds`。
|
||||
|
||||
### 3.3 设计决策说明
|
||||
|
||||
`dwd_assistant_service_log_ex` 表的 `is_trash` 字段来自上游 SaaS 系统的
|
||||
`assistant_service_records` API,是服务记录级别的废除标记,比跨表匹配更可靠。
|
||||
|
||||
废除时长统计改用服务记录自身的 `income_seconds`(即该服务的计费时长),
|
||||
而非从废除表取 `charge_minutes_raw`(废除事件的计费分钟数),
|
||||
因为两者无法 1:1 关联。
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证计划
|
||||
|
||||
修复将在下次 ETL 执行时生效。验证步骤:
|
||||
|
||||
1. 重新提交包含 `DWS_ASSISTANT_DAILY` 的执行
|
||||
2. 确认无 SQL 错误
|
||||
3. 检查 `dws.dws_assistant_daily` 表中 `trashed_count` / `trashed_seconds` 是否合理
|
||||
4. 对比 `dwd_assistant_service_log_ex.is_trash = 1` 的记录数与 DWS 汇总的 `trashed_count`
|
||||
|
||||
---
|
||||
|
||||
## 5. 回滚方案
|
||||
|
||||
如需回滚,恢复 `assistant_daily_task.py` 到修改前版本即可。
|
||||
DWS 表数据可通过重新执行 `DWS_ASSISTANT_DAILY` 任务覆盖。
|
||||
443
export/SYSTEM/LOGS/2026-02-21__etl_full_bug_report.md
Normal file
443
export/SYSTEM/LOGS/2026-02-21__etl_full_bug_report.md
Normal file
@@ -0,0 +1,443 @@
|
||||
# ETL 前后端联调 — BUG 修复全记录
|
||||
|
||||
> 日期: 2026-02-21
|
||||
> 执行轮次: v1 ~ v8(共 8 次)
|
||||
> 任务配置: api_full, full_window, 2025-11-01 ~ 2026-02-20, 30天窗口切分, force_full, 19个任务
|
||||
|
||||
---
|
||||
|
||||
## 总览
|
||||
|
||||
| 指标 | v1 (首次) | v6 (中期最佳) | v8 (最终) |
|
||||
|------|-----------|--------------|-----------|
|
||||
| 耗时 | 590.7s | 29m26s | 1m24s |
|
||||
| 成功任务 | 10/41 | 11/19 | 14/19 |
|
||||
| 失败任务 | 31/41 | 8/19 | 5/19 |
|
||||
| 累计修复 BUG | 0 | 7 | 11 |
|
||||
|
||||
最终 5 个失败均为 `InFailedSqlTransaction` 级联(根因是上游数据质量问题,非代码 BUG)。
|
||||
|
||||
---
|
||||
|
||||
## BUG 详情
|
||||
|
||||
### BUG 1 — DWS_ASSISTANT_DAILY SQL 字段引用错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v1 |
|
||||
| 验证版本 | v2 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: 错误: 字段 "assistant_service_id" 不存在`,DWS_ASSISTANT_DAILY 及其下游 31 个任务全部失败(InFailedSqlTransaction 级联) |
|
||||
| 根因 | `_extract_trash_records()` 方法的 SQL 引用了 `dwd.dwd_assistant_trash_event` 表中不存在的 3 个字段名,且废除判断逻辑依赖跨表 ID 匹配(两表无外键关联,设计上无效) |
|
||||
| 涉及表 | `dwd.dwd_assistant_trash_event`、`dwd.dwd_assistant_service_log`、`dwd.dwd_assistant_service_log_ex` |
|
||||
|
||||
#### 字段映射错误(3 处 SQL 字段名不匹配)
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 所属表 | 说明 |
|
||||
|---|-------------------|----------------------|--------|------|
|
||||
| 1 | `assistant_service_id` | `assistant_trash_event_id` | `dwd_assistant_trash_event` | PK 名称不同,废除表没有 service_id 外键 |
|
||||
| 2 | `trash_seconds` | `charge_minutes_raw * 60 AS trash_seconds` | `dwd_assistant_trash_event` | 实际字段存储分钟数,需乘 60 转秒 |
|
||||
| 3 | `trash_time` | `create_time AS trash_time` | `dwd_assistant_trash_event` | 废除时间字段名不同 |
|
||||
|
||||
#### 逻辑修复(1 处设计缺陷修正)
|
||||
|
||||
| # | 修复点 | 修复前 | 修复后 | 说明 |
|
||||
|---|--------|--------|--------|------|
|
||||
| 4 | `_aggregate_by_assistant_date` 废除判断 | `is_trashed = service_id in trash_index`(跨表 ID 匹配) | `is_trashed = bool(record.get('is_trash', 0))`(使用 _ex 表标记) | 原逻辑依赖废除表的 event_id 与服务表的 service_id 匹配,但两者无外键关联,匹配永远为空。改为 LEFT JOIN `dwd_assistant_service_log_ex` 取 `is_trash` 字段直接判断 |
|
||||
|
||||
#### 修复后 SQL(`_extract_trash_records`)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
assistant_trash_event_id,
|
||||
charge_minutes_raw * 60 AS trash_seconds,
|
||||
trash_reason,
|
||||
create_time AS trash_time,
|
||||
table_id,
|
||||
assistant_name
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s
|
||||
AND DATE(create_time) >= %s AND DATE(create_time) <= %s
|
||||
```
|
||||
|
||||
#### 修复后 SQL(`_extract_service_records` 新增 JOIN)
|
||||
|
||||
```sql
|
||||
SELECT asl.assistant_service_id, ...,
|
||||
COALESCE(ex.is_trash, 0) AS is_trash
|
||||
FROM dwd.dwd_assistant_service_log asl
|
||||
LEFT JOIN dwd.dwd_assistant_service_log_ex ex
|
||||
ON asl.assistant_service_id = ex.assistant_service_id
|
||||
WHERE asl.site_id = %s AND ...
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v2 中 DWS_ASSISTANT_DAILY 执行成功 |
|
||||
|------|------|
|
||||
| 详细修复记录 | `export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md` |
|
||||
|
||||
### BUG 2 — DWS_ASSISTANT_MONTHLY GROUP BY 聚合错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v2 |
|
||||
| 验证版本 | v3 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_monthly_task.py` |
|
||||
| 错误现象 | `UniqueViolation: duplicate key value violates unique constraint "uk_dws_assistant_monthly"` |
|
||||
| 根因 | `_extract_daily_aggregates()` 的 GROUP BY 包含了 `assistant_nickname`、`assistant_level_code`、`assistant_level_name` 三个维度字段。当助教在月内改名或升级时,同一 `(assistant_id, stat_month)` 会产出多行,INSERT 时违反 `uk_dws_assistant_monthly` 唯一约束 |
|
||||
| 涉及表 | `dws.dws_assistant_daily_detail`(源)→ `dws.dws_assistant_monthly_summary`(目标) |
|
||||
|
||||
#### 业务场景
|
||||
|
||||
助教月内改名/升级是正常业务操作。例如助教 A 在 12 月 15 日从"初级"升为"中级",则 `dws_assistant_daily_detail` 中 12 月的记录会有两种 `assistant_level_code`。原 SQL 按 `(assistant_id, stat_month, assistant_level_code)` 分组,产出 2 行,但月度汇总表的唯一约束只有 `(site_id, assistant_id, stat_month)`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
将 `assistant_nickname`、`assistant_level_code`、`assistant_level_name` 从 GROUP BY 移除,改用 `MAX()` 聚合(取月内最新值):
|
||||
|
||||
```sql
|
||||
-- 修复前 GROUP BY
|
||||
GROUP BY assistant_id, assistant_nickname, assistant_level_code, assistant_level_name, DATE_TRUNC('month', stat_date)
|
||||
|
||||
-- 修复后 GROUP BY
|
||||
GROUP BY assistant_id, DATE_TRUNC('month', stat_date)
|
||||
|
||||
-- 维度字段改为 MAX() 聚合
|
||||
MAX(assistant_nickname) AS assistant_nickname,
|
||||
MAX(assistant_level_code) AS assistant_level_code,
|
||||
MAX(assistant_level_name) AS assistant_level_name,
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v3 中 DWS_ASSISTANT_MONTHLY 执行成功(删除 9 行,插入 9 行) |
|
||||
|------|------|
|
||||
|
||||
### BUG 3 — DWS_ASSISTANT_CUSTOMER 引用不存在的 site_id 列
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v2 |
|
||||
| 验证版本 | v3 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_customer_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column dm.site_id does not exist` |
|
||||
| 根因 | `dwd.dim_member` 表没有 `site_id` 列,实际字段为 `register_site_id` |
|
||||
| 修复方式 | `dm.site_id` → `dm.register_site_id` |
|
||||
| 修复结果 | ✅ v3 中 DWS_ASSISTANT_CUSTOMER 执行成功(285 行) |
|
||||
|
||||
### BUG 4 — 多个 DWS 任务引用 dim_member/dim_member_card_account 的 site_id
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v3 |
|
||||
| 验证版本 | v4 |
|
||||
| 文件 | `assistant_customer_task.py`、`member_consumption_task.py`、`finance_recharge_task.py`(共 4 处) |
|
||||
| 错误现象 | 多个 DWS 任务因 `UndefinedColumn: site_id` 失败 |
|
||||
| 根因 | 与 BUG 3 同源 — `dim_member` 和 `dim_member_card_account` 均无 `site_id`,需用 `register_site_id` |
|
||||
| 修复方式 | 4 处 `site_id` → `register_site_id` |
|
||||
| 修复结果 | ✅ v4 中相关任务执行成功 |
|
||||
|
||||
### BUG 5 — DWS_MEMBER_VISIT 引用不存在的 birthday 字段
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v4 |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column dm.birthday does not exist` |
|
||||
| 根因 | `_extract_member_info()` 的 SQL 中 SELECT 了 `birthday` 字段,但 `dwd.dim_member` 表没有该字段 |
|
||||
| 涉及表 | `dwd.dim_member` |
|
||||
|
||||
#### 业务场景
|
||||
|
||||
`dim_member` 的数据来源是上游 SaaS 的 `member_profiles` API。该 API 返回的会员信息中不包含生日字段,因此 ODS → DWD 装载时也没有 `birthday` 列。原代码假设 `dim_member` 有 `birthday`,用于在 `dws_member_visit` 中记录会员生日,但这个字段从未存在过。
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dim_member` 实际字段:`member_id`, `system_member_id`, `tenant_id`, `register_site_id`, `mobile`, `nickname`, `member_card_grade_code`, `member_card_grade_name`, `create_time`, `update_time`, `pay_money_sum`, `recharge_money_sum` + SCD2 元数据。无 `birthday`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
1. `_extract_member_info()` SQL 中移除 `birthday`,只保留 `member_id`, `nickname`, `mobile`
|
||||
2. `transform()` 中 `member_birthday` 字段固定填 `None`(DWS 表该列保留但值为空)
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
sql = "SELECT member_id, nickname, mobile, birthday FROM dwd.dim_member ..."
|
||||
|
||||
# 修复后
|
||||
sql = "SELECT member_id, nickname, mobile FROM dwd.dim_member ..."
|
||||
|
||||
# transform 中
|
||||
'member_birthday': None, # dim_member 无 birthday 字段
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v6 中 DWS_MEMBER_VISIT 执行成功(v5 被 BUG 6 遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 6 — DWS_MEMBER_VISIT _extract_table_info() 字段名不匹配
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v5 |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` |
|
||||
| 错误现象 | `UndefinedColumn` — `_extract_table_info()` 方法中引用了 `dim_table` 中不存在的列名 |
|
||||
| 根因 | 代码中用 `site_table_id` 和 `site_table_name` 查询 `dim_table`,但该表的 PK 是 `table_id`,名称字段是 `table_name`。`site_table_id` 是事实表 `dwd_table_fee_log` 中的外键列名,不是维度表的列名 |
|
||||
| 涉及表 | `dwd.dim_table` |
|
||||
|
||||
#### 字段映射错误
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 说明 |
|
||||
|---|-------------------|----------------------|------|
|
||||
| 1 | `site_table_id` | `table_id` | `dim_table` 的 PK 是 `table_id`,不是 `site_table_id` |
|
||||
| 2 | `site_table_name` | `table_name` | `dim_table` 的名称字段是 `table_name` |
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dim_table` 实际字段:`table_id`(PK), `site_id`, `table_name`, `site_table_area_id`, `site_table_area_name`, `tenant_table_area_id`, `table_price`, `order_id` + SCD2 元数据。
|
||||
|
||||
#### 修复后 SQL
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
table_id AS table_id,
|
||||
table_name AS table_name,
|
||||
site_table_area_name AS area_name
|
||||
FROM dwd.dim_table
|
||||
WHERE site_id = %s AND scd2_is_current = 1
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后字段映射与 DDL 完全一致 ✅:
|
||||
- `table_id` — DDL 中存在,是 PK ✅
|
||||
- `table_name` — DDL 中存在 ✅
|
||||
- `site_table_area_name` — DDL 中存在 ✅
|
||||
- `site_id` 用于 WHERE 过滤 — DDL 中存在 ✅
|
||||
|
||||
| 修复结果 | ✅ v6 中 DWS_MEMBER_VISIT 执行成功 |
|
||||
|------|------|
|
||||
|
||||
### BUG 7 — DWS_FINANCE_INCOME_STRUCTURE JOIN 条件列名错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | 预防性修复(v5 代码审查发现) |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/finance_income_task.py` |
|
||||
| 错误现象 | JOIN 条件中 `dt.site_table_id` 不存在(`dim_table` 无 `site_table_id` 列) |
|
||||
| 根因 | `_extract_income_by_area()` 中 `dwd_table_fee_log` 和 `dwd_assistant_service_log` 与 `dim_table` 做 LEFT JOIN 时,JOIN 条件写成了 `dt.site_table_id = tfl.site_table_id`,但 `dim_table` 的 PK 是 `table_id` 而非 `site_table_id` |
|
||||
| 涉及表 | `dwd.dim_table`(维度)、`dwd.dwd_table_fee_log`(事实)、`dwd.dwd_assistant_service_log`(事实) |
|
||||
|
||||
#### 关联关系说明
|
||||
|
||||
事实表 `dwd_table_fee_log` 和 `dwd_assistant_service_log` 中的 `site_table_id` 是外键,指向维度表 `dim_table` 的 PK `table_id`。两个 ID 值相同,但列名不同。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
```sql
|
||||
-- 修复前(2 处)
|
||||
LEFT JOIN dwd.dim_table dt ON dt.site_table_id = tfl.site_table_id
|
||||
LEFT JOIN dwd.dim_table dt ON dt.site_table_id = asl.site_table_id
|
||||
|
||||
-- 修复后(2 处)
|
||||
LEFT JOIN dwd.dim_table dt ON dt.table_id = tfl.site_table_id
|
||||
LEFT JOIN dwd.dim_table dt ON dt.table_id = asl.site_table_id
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后 JOIN 条件与 DDL 一致 ✅:
|
||||
- `dim_table.table_id` — DDL 确认是 PK ✅
|
||||
- `dwd_table_fee_log.site_table_id` — DDL 确认存在,语义为"台桌 ID 外键" ✅
|
||||
- `dwd_assistant_service_log.site_table_id` — 同上 ✅
|
||||
- JOIN 方向正确:维度表 PK ← 事实表 FK ✅
|
||||
|
||||
| 修复结果 | ✅ v6 中未出现该错误(但被 BUG 8 级联遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 8 — DWS_FINANCE_DAILY / DWS_FINANCE_RECHARGE 字段名错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v6 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `finance_base_task.py`、`finance_recharge_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column "pay_money" does not exist`,DWS_FINANCE_DAILY 失败并级联导致 7 个下游任务失败 |
|
||||
| 根因 | 代码中用 `pay_money` / `gift_money` 查询 `dwd.dwd_recharge_order`,但该表的实际字段是 `pay_amount`(现金充值金额)/ `point_amount`(赠送金额) |
|
||||
| 涉及表 | `dwd.dwd_recharge_order` |
|
||||
|
||||
#### 字段映射错误
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 业务含义 |
|
||||
|---|-------------------|----------------------|---------|
|
||||
| 1 | `pay_money` | `pay_amount` | 现金充值金额(会员实际支付) |
|
||||
| 2 | `gift_money` | `point_amount` | 赠送金额(充值赠送的积分/赠送卡金额) |
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dwd_recharge_order` 金额相关字段:`pay_amount` NUMERIC(18,2)、`refund_amount` NUMERIC(18,2)、`point_amount` NUMERIC(18,2)、`cash_amount` NUMERIC(18,2)。无 `pay_money` 或 `gift_money`。
|
||||
|
||||
#### 修复涉及 2 个文件
|
||||
|
||||
1. `finance_base_task.py` — `_extract_recharge_summary()` 方法(被 `FinanceDailyTask` 继承调用)
|
||||
2. `finance_recharge_task.py` — `_extract_recharge_summary()` 方法(`FinanceRechargeTask` 自身的重写版本)
|
||||
|
||||
两处修复内容相同:所有 `pay_money` → `pay_amount`,所有 `gift_money` → `point_amount`。
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后字段映射与 DDL 完全一致 ✅:
|
||||
- `pay_amount` — DDL 确认存在,NUMERIC(18,2) ✅
|
||||
- `point_amount` — DDL 确认存在,NUMERIC(18,2) ✅
|
||||
- `is_first` — DDL 确认存在,INTEGER ✅
|
||||
- `member_id` — DDL 确认存在 ✅
|
||||
- `pay_time` — DDL 确认存在,TIMESTAMPTZ ✅
|
||||
- `site_id` — DDL 确认存在 ✅
|
||||
- 业务语义:`pay_amount + point_amount` = 充值总额(现金 + 赠送),`is_first = 1` 区分首充/续充 ✅
|
||||
|
||||
| 修复结果 | ✅ v8 中 DWS_FINANCE_DAILY 和 DWS_FINANCE_RECHARGE 均执行成功(v7 被 BUG 9 遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 9 — DWD_LOAD_FROM_ODS 缺少 _pick_snapshot_order_column 方法
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` |
|
||||
| 错误现象 | `AttributeError: 'DwdLoadTask' object has no attribute '_pick_snapshot_order_column'`,所有 dim 表 SCD2 装载全部失败 |
|
||||
| 根因 | `_merge_dim_scd2()` 方法内部调用了 `self._pick_snapshot_order_column(cols)`,但该方法只存在于 `quality/integrity_checker.py` 中作为模块级函数,`DwdLoadTask` 类中没有定义。这是代码重构时遗漏的问题——SCD2 装载逻辑从 integrity_checker 迁移到 DwdLoadTask 时,忘记把依赖的辅助函数一起迁移 |
|
||||
| 涉及表 | 所有 dim 表(dim_site, dim_table, dim_assistant, dim_member, dim_member_card_account, dim_tenant_goods, dim_store_goods, dim_goods_category, dim_groupbuy_package 及其 _ex 表) |
|
||||
|
||||
#### 方法功能说明
|
||||
|
||||
`_pick_snapshot_order_column(cols)` 用于从 ODS 表的列名列表中选取快照排序列,优先级为 `fetched_at` > `update_time` > `create_time`。SCD2 装载时需要按此列排序,确保同一业务主键的多条快照按时间顺序处理,最新快照覆盖旧快照。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
在 `DwdLoadTask` 类中添加 `@staticmethod` 方法,逻辑与 `integrity_checker.py` 中的同名函数一致:
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def _pick_snapshot_order_column(cols: Sequence[str]) -> str | None:
|
||||
"""从 ODS 列中选取用于快照排序的列(fetched_at > update_time > create_time)。"""
|
||||
lower = {c.lower() for c in cols}
|
||||
for candidate in ("fetched_at", "update_time", "create_time"):
|
||||
if candidate in lower:
|
||||
return candidate
|
||||
return None
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v8 中所有 15 个 dim 表 SCD2 装载成功(dim_site, dim_table, dim_assistant, dim_member, dim_member_card_account, dim_tenant_goods, dim_store_goods, dim_goods_category, dim_groupbuy_package 及其 _ex 表) |
|
||||
|------|------|
|
||||
|
||||
### BUG 10 — goods_stock 表 FACT_MAPPINGS 驼峰字段名导致 SQL 错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column "siteGoodsId" does not exist, perhaps you mean "sitegoodsid"` |
|
||||
| 根因 | `FACT_MAPPINGS` 中 `dwd_goods_stock_summary` 和 `dwd_goods_stock_movement` 的源列使用了带引号的驼峰名(如 `"siteGoodsId"`),但 ODS 表中 PostgreSQL 存储的列名是全小写的 `sitegoodsid`(ODS 入库时 `_int_col("sitegoodsid", "siteGoodsId")` 已将 JSON 驼峰键转为小写列名) |
|
||||
| 修复方式 | 将 FACT_MAPPINGS 中 2 个表共 30+ 个字段的驼峰引用全部改为小写(如 `"siteGoodsId"` → `"sitegoodsid"`) |
|
||||
| 修复结果 | ✅ v8 中 `dwd_goods_stock_summary`(716 条 INSERT)和 `dwd_goods_stock_movement`(14306 条 INSERT)装载成功 |
|
||||
|
||||
### BUG 11 — flow_runner.py sum() 类型不安全
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` |
|
||||
| 错误现象 | `TypeError: unsupported operand type(s) for +: 'int' and 'list'` |
|
||||
| 根因 | Flow 执行完成后汇总各任务的 `counts.errors` 时,某些任务返回的 `errors` 是错误详情列表(`list[str]`)而非错误计数(`int`),`sum()` 无法将 `int` 与 `list` 相加 |
|
||||
| 涉及位置 | `FlowRunner.run()` 方法末尾的 `flow_logger.set_counts(...)` 调用 |
|
||||
|
||||
#### 触发场景
|
||||
|
||||
任务执行器 `task_executor` 返回的 `result["counts"]["errors"]` 类型不统一:
|
||||
- 大部分任务返回 `int`(错误计数,如 `0` 或 `3`)
|
||||
- 部分任务返回 `list`(错误详情列表,如 `["UndefinedColumn: ...", "InFailedSqlTransaction: ..."]`)
|
||||
|
||||
当 `sum()` 遍历到 `list` 类型时,Python 尝试执行 `int + list`,抛出 `TypeError`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
在 `run()` 方法内添加 `_safe_int()` 局部函数,统一类型转换:
|
||||
|
||||
```python
|
||||
def _safe_int(val) -> int:
|
||||
"""将 int/list/None 统一转为 int 计数。"""
|
||||
if isinstance(val, int):
|
||||
return val
|
||||
if isinstance(val, list):
|
||||
return len(val) # 错误列表 → 取长度作为错误计数
|
||||
return 0
|
||||
|
||||
flow_logger.set_counts(
|
||||
fetched=sum(_safe_int(r.get("counts", {}).get("fetched", 0)) for r in results),
|
||||
inserted=sum(_safe_int(r.get("counts", {}).get("inserted", 0)) for r in results),
|
||||
updated=sum(_safe_int(r.get("counts", {}).get("updated", 0)) for r in results),
|
||||
errors=sum(_safe_int(r.get("counts", {}).get("errors", 0)) for r in results),
|
||||
)
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后类型处理正确 ✅:
|
||||
- `int` → 直接返回 ✅
|
||||
- `list` → `len()` 转为计数 ✅(语义正确:错误详情列表的长度 = 错误数量)
|
||||
- `None` → 返回 0 ✅
|
||||
- `r.get("counts", {}).get(...)` 已做双层防御,`counts` 缺失时不会报错 ✅
|
||||
- 四个计数字段(`fetched`/`inserted`/`updated`/`errors`)均经过 `_safe_int` 处理 ✅
|
||||
|
||||
| 修复结果 | ✅ v8 中不再出现 TypeError,Flow 汇总正常完成 |
|
||||
|------|------|
|
||||
|
||||
---
|
||||
|
||||
## 未修复的遗留问题
|
||||
|
||||
### 数据质量问题 — dim_assistant_ex / dim_member_card_account_ex 非法日期
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v8 |
|
||||
| 性质 | 上游数据质量问题,非代码 BUG |
|
||||
| 错误现象 | `ValueError: year -1 is out of range` |
|
||||
| 根因 | ODS 中某些记录的日期字段包含非法值(year=-1),Python `datetime` 无法解析 |
|
||||
| 影响 | `dim_assistant_ex` 和 `dim_member_card_account_ex` 装载失败 → 事务进入 `InFailedSqlTransaction` → 级联导致 5 个 DWS 任务失败(DWS_FINANCE_INCOME_STRUCTURE, DWS_FINANCE_DISCOUNT_DETAIL, DWS_WINBACK_INDEX, DWS_NEWCONV_INDEX, DWS_RELATION_INDEX) |
|
||||
| 建议 | 在 DWD 装载的日期类型转换中添加容错处理(捕获 ValueError,将非法日期置为 NULL 或哨兵值) |
|
||||
|
||||
---
|
||||
|
||||
## 修复文件清单
|
||||
|
||||
| 文件 | 修复的 BUG |
|
||||
|------|-----------|
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py` | BUG 1 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_monthly_task.py` | BUG 2 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_customer_task.py` | BUG 3, 4 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/member_consumption_task.py` | BUG 4 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` | BUG 5, 6 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_income_task.py` | BUG 7 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_base_task.py` | BUG 8 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_recharge_task.py` | BUG 4, 8 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` | BUG 9, 10 |
|
||||
| `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` | BUG 11 |
|
||||
|
||||
---
|
||||
|
||||
## 执行历史
|
||||
|
||||
| 版本 | execution_id | 耗时 | 成功 | 失败 | 修复验证 |
|
||||
|------|-------------|------|------|------|---------|
|
||||
| v1 | `dbf0c29a-...` | 590.7s | 10 | 31 | — |
|
||||
| v2 | `e21e1935-...` | 150.4s | — | — | BUG 1 ✅ |
|
||||
| v3 | `abc94b2d-...` | 681.2s | 9 | 22 | BUG 2,3 ✅ |
|
||||
| v4 | `efd4f421-...` | 11m55s | 10 | 21 | BUG 4 ✅ |
|
||||
| v5 | `fe87144a-...` | 11m37s | 10 | 21 | BUG 5 部署(被 BUG 6 遮蔽) |
|
||||
| v6 | `d9443781-...` | 29m26s | 11 | 8 | BUG 5,6,7 ✅ |
|
||||
| v7 | `0929ab3a-...` | 89.3s | — | 全部 | BUG 8 部署(被 BUG 9 遮蔽) |
|
||||
| v8 | `f943bac6-...` | 1m24s | 14 | 5 | BUG 8,9,10,11 ✅ |
|
||||
428
export/SYSTEM/LOGS/2026-02-21__etl_run_raw.json
Normal file
428
export/SYSTEM/LOGS/2026-02-21__etl_run_raw.json
Normal file
@@ -0,0 +1,428 @@
|
||||
{
|
||||
"execution": {
|
||||
"id": "dbf0c29a-253a-4705-a1ef-35cd71243d48",
|
||||
"site_id": 2790685415443269,
|
||||
"task_codes": [
|
||||
"ODS_ASSISTANT_ACCOUNT",
|
||||
"ODS_ASSISTANT_LEDGER",
|
||||
"ODS_ASSISTANT_ABOLISH",
|
||||
"DWS_ASSISTANT_DAILY",
|
||||
"DWS_ASSISTANT_MONTHLY",
|
||||
"DWS_ASSISTANT_CUSTOMER",
|
||||
"DWS_ASSISTANT_SALARY",
|
||||
"DWS_ASSISTANT_FINANCE",
|
||||
"ODS_SETTLEMENT_RECORDS",
|
||||
"ODS_PAYMENT",
|
||||
"ODS_REFUND",
|
||||
"DWS_BUILD_ORDER_SUMMARY",
|
||||
"ODS_TABLE_USE",
|
||||
"ODS_TABLE_FEE_DISCOUNT",
|
||||
"ODS_TABLES",
|
||||
"ODS_MEMBER",
|
||||
"ODS_MEMBER_CARD",
|
||||
"ODS_MEMBER_BALANCE",
|
||||
"ODS_RECHARGE_SETTLE",
|
||||
"DWS_MEMBER_CONSUMPTION",
|
||||
"DWS_MEMBER_VISIT",
|
||||
"ODS_GOODS_CATEGORY",
|
||||
"ODS_STORE_GOODS",
|
||||
"ODS_STORE_GOODS_SALES",
|
||||
"ODS_TENANT_GOODS",
|
||||
"ODS_PLATFORM_COUPON",
|
||||
"ODS_GROUP_PACKAGE",
|
||||
"ODS_GROUP_BUY_REDEMPTION",
|
||||
"ODS_INVENTORY_STOCK",
|
||||
"ODS_INVENTORY_CHANGE",
|
||||
"DWS_GOODS_STOCK_DAILY",
|
||||
"DWS_GOODS_STOCK_WEEKLY",
|
||||
"DWS_GOODS_STOCK_MONTHLY",
|
||||
"DWS_FINANCE_DAILY",
|
||||
"DWS_FINANCE_RECHARGE",
|
||||
"DWS_FINANCE_INCOME_STRUCTURE",
|
||||
"DWS_FINANCE_DISCOUNT_DETAIL",
|
||||
"DWS_WINBACK_INDEX",
|
||||
"DWS_NEWCONV_INDEX",
|
||||
"DWS_RELATION_INDEX",
|
||||
"DWD_LOAD_FROM_ODS"
|
||||
],
|
||||
"status": "success",
|
||||
"started_at": "2026-02-21T15:29:20.233302+08:00",
|
||||
"finished_at": "2026-02-21T15:39:10.909320+08:00",
|
||||
"exit_code": 0,
|
||||
"duration_ms": 590676,
|
||||
"command": "C:\\NeoZQYY\\.venv\\Scripts\\python.exe -m cli.main --flow api_full --processing-mode full_window --tasks ODS_ASSISTANT_ACCOUNT,ODS_ASSISTANT_LEDGER,ODS_ASSISTANT_ABOLISH,DWS_ASSISTANT_DAILY,DWS_ASSISTANT_MONTHLY,DWS_ASSISTANT_CUSTOMER,DWS_ASSISTANT_SALARY,DWS_ASSISTANT_FINANCE,ODS_SETTLEMENT_RECORDS,ODS_PAYMENT,ODS_REFUND,DWS_BUILD_ORDER_SUMMARY,ODS_TABLE_USE,ODS_TABLE_FEE_DISCOUNT,ODS_TABLES,ODS_MEMBER,ODS_MEMBER_CARD,ODS_MEMBER_BALANCE,ODS_RECHARGE_SETTLE,DWS_MEMBER_CONSUMPTION,DWS_MEMBER_VISIT,ODS_GOODS_CATEGORY,ODS_STORE_GOODS,ODS_STORE_GOODS_SALES,ODS_TENANT_GOODS,ODS_PLATFORM_COUPON,ODS_GROUP_PACKAGE,ODS_GROUP_BUY_REDEMPTION,ODS_INVENTORY_STOCK,ODS_INVENTORY_CHANGE,DWS_GOODS_STOCK_DAILY,DWS_GOODS_STOCK_WEEKLY,DWS_GOODS_STOCK_MONTHLY,DWS_FINANCE_DAILY,DWS_FINANCE_RECHARGE,DWS_FINANCE_INCOME_STRUCTURE,DWS_FINANCE_DISCOUNT_DETAIL,DWS_WINBACK_INDEX,DWS_NEWCONV_INDEX,DWS_RELATION_INDEX,DWD_LOAD_FROM_ODS --window-start 2025-11-01 --window-end 2026-02-20 --window-split day --window-split-days 30 --force-full --store-id 2790685415443269",
|
||||
"summary": null
|
||||
},
|
||||
"error_log_length": 66800,
|
||||
"task_results_parsed": [
|
||||
{
|
||||
"task": "ODS_ASSISTANT_ACCOUNT",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:29:21",
|
||||
"end": "2026-02-21 15:29:31",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 276, 'inserted': 0, 'updated': 276, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_ASSISTANT_LEDGER",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:29:32",
|
||||
"end": "2026-02-21 15:30:08",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 2277, 'inserted': 342, 'updated': 2277, 'skipped': 0, 'errors': 0, 'deleted': 342}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_ASSISTANT_ABOLISH",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:30:08",
|
||||
"end": "2026-02-21 15:30:11",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 78, 'inserted': 0, 'updated': 78, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:30:13",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_MONTHLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_CUSTOMER",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_SALARY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_FINANCE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_SETTLEMENT_RECORDS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_PAYMENT",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_REFUND",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_BUILD_ORDER_SUMMARY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:30:15",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLE_USE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:30:15",
|
||||
"end": "2026-02-21 15:35:01",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 36412, 'inserted': 0, 'updated': 36412, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLE_FEE_DISCOUNT",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:35:02",
|
||||
"end": "2026-02-21 15:36:04",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 6464, 'inserted': 0, 'updated': 6464, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLES",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:05",
|
||||
"end": "2026-02-21 15:36:10",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 296, 'inserted': 0, 'updated': 296, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:11",
|
||||
"end": "2026-02-21 15:36:28",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 2228, 'inserted': 0, 'updated': 2228, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER_CARD",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:29",
|
||||
"end": "2026-02-21 15:37:05",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 3784, 'inserted': 0, 'updated': 3784, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER_BALANCE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:37:06",
|
||||
"end": "2026-02-21 15:38:59",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 8740, 'inserted': 0, 'updated': 8740, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_RECHARGE_SETTLE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:39:01",
|
||||
"end": "2026-02-21 15:39:06",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 191, 'inserted': 0, 'updated': 191, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "DWS_MEMBER_CONSUMPTION",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:06",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_MEMBER_VISIT",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GOODS_CATEGORY",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_STORE_GOODS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_STORE_GOODS_SALES",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TENANT_GOODS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_PLATFORM_COUPON",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GROUP_PACKAGE",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GROUP_BUY_REDEMPTION",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_INVENTORY_STOCK",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_INVENTORY_CHANGE",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_WEEKLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_MONTHLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_RECHARGE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_INCOME_STRUCTURE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_DISCOUNT_DETAIL",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_WINBACK_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_NEWCONV_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_RELATION_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWD_LOAD_FROM_ODS",
|
||||
"layer": "DWD",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
}
|
||||
]
|
||||
}
|
||||
50
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v2.json
Normal file
50
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v2.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v3.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v3.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v4.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v4.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v5.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v5.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v6.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v6.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v7.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v7.json
Normal file
File diff suppressed because one or more lines are too long
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v8.json
Normal file
5
export/SYSTEM/LOGS/2026-02-21__etl_run_raw_v8.json
Normal file
File diff suppressed because one or more lines are too long
124
export/SYSTEM/LOGS/2026-02-21__etl_run_result.md
Normal file
124
export/SYSTEM/LOGS/2026-02-21__etl_run_result.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# ETL 执行结果报告
|
||||
|
||||
> 生成时间:2026-02-21 19:18:58
|
||||
> execution_id:dbf0c29a-253a-4705-a1ef-35cd71243d48
|
||||
> run_uuid:4ba9d2d365ee4a858f1c4104b1942dc2
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21T15:29:20.233302+08:00 |
|
||||
| 结束时间 | 2026-02-21T15:39:10.909320+08:00 |
|
||||
| 总时长 | 590.7s (9.8m) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 41 |
|
||||
| 成功 | 10 |
|
||||
| 失败 | 31 |
|
||||
|
||||
---
|
||||
|
||||
## 任务级结果
|
||||
|
||||
| # | 任务 | 层 | 状态 | 开始 | 结束 | 耗时 | 窗口数 | 备注 |
|
||||
|---|------|-----|------|------|------|------|--------|------|
|
||||
| 1 | ODS_ASSISTANT_ACCOUNT | ODS | ✅ success | 15:29:21 | 15:29:31 | 10.0s | 4 | {'fetched': 276, 'inserted': 0, 'updated': 276, 'skipped'... |
|
||||
| 2 | ODS_ASSISTANT_LEDGER | ODS | ✅ success | 15:29:32 | 15:30:08 | 36.0s | 4 | {'fetched': 2277, 'inserted': 342, 'updated': 2277, 'skip... |
|
||||
| 3 | ODS_ASSISTANT_ABOLISH | ODS | ✅ success | 15:30:08 | 15:30:11 | 3.0s | 4 | {'fetched': 78, 'inserted': 0, 'updated': 78, 'skipped': ... |
|
||||
| 4 | DWS_ASSISTANT_DAILY | DWS | ❌ failed | 15:30:13 | 15:30:14 | 1.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | DWS_ASSISTANT_MONTHLY | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | DWS_ASSISTANT_CUSTOMER | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | DWS_ASSISTANT_SALARY | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | DWS_ASSISTANT_FINANCE | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 9 | ODS_SETTLEMENT_RECORDS | ODS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 10 | ODS_PAYMENT | ODS | ❌ failed | | 15:30:15 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 11 | ODS_REFUND | ODS | ❌ failed | | 15:30:15 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | DWS_BUILD_ORDER_SUMMARY | DWS | ❌ failed | 15:30:15 | 15:30:15 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | ODS_TABLE_USE | ODS | ✅ success | 15:30:15 | 15:35:01 | 4.8m | 4 | {'fetched': 36412, 'inserted': 0, 'updated': 36412, 'skip... |
|
||||
| 14 | ODS_TABLE_FEE_DISCOUNT | ODS | ✅ success | 15:35:02 | 15:36:04 | 1.0m | 4 | {'fetched': 6464, 'inserted': 0, 'updated': 6464, 'skippe... |
|
||||
| 15 | ODS_TABLES | ODS | ✅ success | 15:36:05 | 15:36:10 | 5.0s | 4 | {'fetched': 296, 'inserted': 0, 'updated': 296, 'skipped'... |
|
||||
| 16 | ODS_MEMBER | ODS | ✅ success | 15:36:11 | 15:36:28 | 17.0s | 4 | {'fetched': 2228, 'inserted': 0, 'updated': 2228, 'skippe... |
|
||||
| 17 | ODS_MEMBER_CARD | ODS | ✅ success | 15:36:29 | 15:37:05 | 36.0s | 4 | {'fetched': 3784, 'inserted': 0, 'updated': 3784, 'skippe... |
|
||||
| 18 | ODS_MEMBER_BALANCE | ODS | ✅ success | 15:37:06 | 15:38:59 | 1.9m | 4 | {'fetched': 8740, 'inserted': 0, 'updated': 8740, 'skippe... |
|
||||
| 19 | ODS_RECHARGE_SETTLE | ODS | ✅ success | 15:39:01 | 15:39:06 | 5.0s | 4 | {'fetched': 191, 'inserted': 0, 'updated': 191, 'skipped'... |
|
||||
| 20 | DWS_MEMBER_CONSUMPTION | DWS | ❌ failed | 15:39:06 | 15:39:07 | 1.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWS_MEMBER_VISIT | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 22 | ODS_GOODS_CATEGORY | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 23 | ODS_STORE_GOODS | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 24 | ODS_STORE_GOODS_SALES | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 25 | ODS_TENANT_GOODS | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 26 | ODS_PLATFORM_COUPON | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 27 | ODS_GROUP_PACKAGE | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 28 | ODS_GROUP_BUY_REDEMPTION | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 29 | ODS_INVENTORY_STOCK | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 30 | ODS_INVENTORY_CHANGE | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 31 | DWS_GOODS_STOCK_DAILY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 32 | DWS_GOODS_STOCK_WEEKLY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 33 | DWS_GOODS_STOCK_MONTHLY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 34 | DWS_FINANCE_DAILY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 35 | DWS_FINANCE_RECHARGE | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 36 | DWS_FINANCE_INCOME_STRUCTURE | DWS | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 37 | DWS_FINANCE_DISCOUNT_DETAIL | DWS | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 38 | DWS_WINBACK_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 39 | DWS_NEWCONV_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 40 | DWS_RELATION_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 41 | DWD_LOAD_FROM_ODS | DWD | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
---
|
||||
|
||||
## 失败任务分析
|
||||
|
||||
### 根因:DWS_ASSISTANT_DAILY
|
||||
|
||||
错误:`错误: 当前事务被终止, 事务块结束之前的查询被忽略`
|
||||
|
||||
原因:`_extract_trash_records` SQL 引用了 `dwd_assistant_trash_event` 中不存在的字段 `assistant_service_id`。
|
||||
|
||||
### 级联失败
|
||||
|
||||
- DWS_ASSISTANT_MONTHLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_CUSTOMER:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_SALARY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_FINANCE:InFailedSqlTransaction(事务污染)
|
||||
- ODS_SETTLEMENT_RECORDS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_PAYMENT:InFailedSqlTransaction(事务污染)
|
||||
- ODS_REFUND:InFailedSqlTransaction(事务污染)
|
||||
- DWS_BUILD_ORDER_SUMMARY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_MEMBER_CONSUMPTION:InFailedSqlTransaction(事务污染)
|
||||
- DWS_MEMBER_VISIT:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GOODS_CATEGORY:InFailedSqlTransaction(事务污染)
|
||||
- ODS_STORE_GOODS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_STORE_GOODS_SALES:InFailedSqlTransaction(事务污染)
|
||||
- ODS_TENANT_GOODS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_PLATFORM_COUPON:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GROUP_PACKAGE:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GROUP_BUY_REDEMPTION:InFailedSqlTransaction(事务污染)
|
||||
- ODS_INVENTORY_STOCK:InFailedSqlTransaction(事务污染)
|
||||
- ODS_INVENTORY_CHANGE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_DAILY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_WEEKLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_MONTHLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_DAILY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_RECHARGE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_INCOME_STRUCTURE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_DISCOUNT_DETAIL:InFailedSqlTransaction(事务污染)
|
||||
- DWS_WINBACK_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWS_NEWCONV_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWS_RELATION_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWD_LOAD_FROM_ODS:InFailedSqlTransaction(事务污染)
|
||||
|
||||
### 修复状态
|
||||
|
||||
代码已修复(4 处改动),待下次执行验证。
|
||||
详见:`export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md`
|
||||
|
||||
---
|
||||
|
||||
## 下一步
|
||||
|
||||
1. 重新提交包含 9 个失败任务的执行,验证修复
|
||||
2. 运行 ETL Data Consistency Check
|
||||
3. 运行 /audit 审计
|
||||
136
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v2.md
Normal file
136
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v2.md
Normal file
@@ -0,0 +1,136 @@
|
||||
# ETL 回归执行结果报告(第二次)
|
||||
|
||||
> 生成时间:2026-02-21 19:33:28
|
||||
> execution_id:e21e1935-5abf-434f-9984-69c492402db7
|
||||
> 目的:验证 DWS_ASSISTANT_DAILY 修复 + 补跑上次失败的 31 个任务
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21T19:27:47.937140+08:00 |
|
||||
| 结束时间 | 2026-02-21T19:30:18.341157+08:00 |
|
||||
| 总时长 | 150.4s (2.5m) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 31 |
|
||||
|
||||
## 执行日志(error_log 末尾 100 行)
|
||||
|
||||
```
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 403, in _run_utility_task
|
||||
result = task.execute(None)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 145, in execute
|
||||
tenant_id = self._get_tenant_id()
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 688, in _get_tenant_id
|
||||
rows = self.db.query(sql)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] ERROR | etl_billiards | 任务 DWS_RELATION_INDEX 失败: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 94, in run_tasks
|
||||
task_result = self.run_single_task(
|
||||
task_code, run_uuid, store_id, data_source=data_source,
|
||||
)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 150, in run_single_task
|
||||
return self._run_utility_task(task_code_upper, store_id)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 403, in _run_utility_task
|
||||
result = task.execute(None)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 145, in execute
|
||||
tenant_id = self._get_tenant_id()
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 688, in _get_tenant_id
|
||||
rows = self.db.query(sql)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] ERROR | etl_billiards | 任务 DWD_LOAD_FROM_ODS 失败: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 94, in run_tasks
|
||||
task_result = self.run_single_task(
|
||||
task_code, run_uuid, store_id, data_source=data_source,
|
||||
)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 152, in run_single_task
|
||||
task_cfg = self._load_task_config(task_code, store_id)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 429, in _load_task_config
|
||||
rows = self.db_ops.query(sql, (store_id, task_code))
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] INFO | etl_billiards | 所有任务执行完成
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | 一致性检查报告已生成: C:\NeoZQYY\export\ETL-Connectors\feiqiu\REPORTS\consistency_report_20260221_193018.md
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | 计时报告已生成
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards |
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ 任务执行总结 ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 任务代码: FLOW_API_FULL ║
|
||||
║ 执行状态: 成功 ║
|
||||
║ 执行时间: 2026-02-21 19:27:49 ~ 19:30:18 (2分29秒) ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 数据统计 ║
|
||||
║ - 获取记录: 0 ║
|
||||
║ - 新增记录: 0 ║
|
||||
║ - 更新记录: 0 ║
|
||||
║ - 跳过记录: 0 ║
|
||||
║ - 错误记录: 0 ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards |
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ 任务执行总结 ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 任务代码: FLOW_API_FULL ║
|
||||
║ 执行状态: 成功 ║
|
||||
║ 执行时间: 2026-02-21 19:27:49 ~ 19:30:18 (2分29秒) ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 数据统计 ║
|
||||
║ - 获取记录: 0 ║
|
||||
║ - 新增记录: 0 ║
|
||||
║ - 更新记录: 0 ║
|
||||
║ - 跳过记录: 0 ║
|
||||
║ - 错误记录: 0 ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | Flow 执行完成: SUCCESS
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | ETL运行完成
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 与第一次执行的对比
|
||||
|
||||
| 项目 | 第一次 | 第二次(本次) |
|
||||
|------|--------|---------------|
|
||||
| 任务数 | 41 | 31 |
|
||||
| 状态 | success (exit_code=0) | success (exit_code=0) |
|
||||
| 耗时 | 590.7s (9.8m) | 150.4s (2.5m) |
|
||||
| 成功 | 10/41 | 待分析 |
|
||||
| 失败 | 31/41 | 待分析 |
|
||||
| 根因 | DWS_ASSISTANT_DAILY SQL 字段错误 | — |
|
||||
85
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v3.md
Normal file
85
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v3.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# ETL 回归执行结果报告(第三次)
|
||||
|
||||
> 生成时间:2026-02-21 19:54:12
|
||||
> execution_id:abc94b2d-615f-42ea-83cc-ce687524a6ea
|
||||
> 目的:验证 BUG 2(DWS_ASSISTANT_MONTHLY UniqueViolation)和 BUG 3(DWS_ASSISTANT_CUSTOMER UndefinedColumn)修复
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21 19:41:02 |
|
||||
| 结束时间 | 2026-02-21 19:52:22 |
|
||||
| 总时长 | 681.2s (11m19s) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 31 |
|
||||
| 成功 | 9 |
|
||||
| 失败 | 22 |
|
||||
| 未知 | 0 |
|
||||
| 数据统计 | 获取 52,982 / 新增 13,296 / 更新 52,982 |
|
||||
|
||||
## BUG 修复验证
|
||||
|
||||
| BUG | 任务 | 第二次结果 | 第三次结果 | 验证 |
|
||||
|-----|------|-----------|-----------|------|
|
||||
| BUG 1 | DWS_ASSISTANT_DAILY | ✅ 已修复 | ✅ 成功 | ✅ 持续通过 |
|
||||
| BUG 2 | DWS_ASSISTANT_MONTHLY | ❌ UniqueViolation | ✅ 成功 | ✅ 修复验证通过 |
|
||||
| BUG 3 | DWS_ASSISTANT_CUSTOMER | ❌ UndefinedColumn | ✅ 成功 | ✅ 修复验证通过 |
|
||||
|
||||
## 逐任务结果
|
||||
|
||||
| # | 任务 | 状态 | 统计/错误 |
|
||||
|---|------|------|----------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | ✅ 成功 | {'counts': {'fetched': 367, 'inserted': 367, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | ✅ 成功 | {'counts': {'fetched': 25, 'inserted': 25, 'updated': 0, 'skipped': 0, 'errors': |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | ✅ 成功 | {'counts': {'fetched': 486, 'inserted': 486, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 4 | DWS_ASSISTANT_SALARY | ✅ 成功 | {'counts': {'fetched': 0, 'inserted': 0, 'updated': 0, 'skipped': 0, 'errors': 0 |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | ✅ 成功 | {'counts': {'fetched': 367, 'inserted': 367, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | ✅ 成功 | fetched=10366, updated=10366 |
|
||||
| 7 | ODS_PAYMENT | ✅ 成功 | fetched=42500, updated=42500 |
|
||||
| 8 | ODS_REFUND | ✅ 成功 | fetched=116, updated=116 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | ✅ 成功 | {'fetched': 0, 'inserted': 13296, 'updated': 0, 'skipped': 0, 'errors': 0} |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION | ❌ 字段错误 | UndefinedColumn: dim_member.site_id 不存在(同 BUG 3 同类) |
|
||||
| 11 | DWS_MEMBER_VISIT | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | ODS_GOODS_CATEGORY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | ODS_STORE_GOODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 14 | ODS_STORE_GOODS_SALES | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 15 | ODS_TENANT_GOODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 16 | ODS_PLATFORM_COUPON | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 17 | ODS_GROUP_PACKAGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 18 | ODS_GROUP_BUY_REDEMPTION | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 19 | ODS_INVENTORY_STOCK | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 20 | ODS_INVENTORY_CHANGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWS_GOODS_STOCK_DAILY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 22 | DWS_GOODS_STOCK_WEEKLY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 23 | DWS_GOODS_STOCK_MONTHLY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 24 | DWS_FINANCE_DAILY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 25 | DWS_FINANCE_RECHARGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 26 | DWS_FINANCE_INCOME_STRUCTURE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 27 | DWS_FINANCE_DISCOUNT_DETAIL | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 28 | DWS_WINBACK_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 29 | DWS_NEWCONV_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 30 | DWS_RELATION_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 31 | DWD_LOAD_FROM_ODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析
|
||||
|
||||
本次新发现的根因错误:
|
||||
|
||||
- 任务:`DWS_MEMBER_CONSUMPTION`
|
||||
- 错误:错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- 影响:后续所有任务因 `InFailedSqlTransaction` 级联失败
|
||||
|
||||
|
||||
## 三次执行对比
|
||||
|
||||
| 项目 | 第一次 | 第二次 | 第三次(本次) |
|
||||
|------|--------|--------|---------------|
|
||||
| 任务数 | 41 | 31 | 31 |
|
||||
| 耗时 | 590.7s | 150.4s | 681.2s |
|
||||
| 成功 | 10/41 | 3/31 | 6/31 |
|
||||
| 失败 | 31/41 | 28/31 | 22/31 |
|
||||
| 根因 | DWS_ASSISTANT_DAILY SQL 字段 | DWS_ASSISTANT_MONTHLY UK + DWS_ASSISTANT_CUSTOMER site_id | DWS_MEMBER_CONSUMPTION site_id |
|
||||
70
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v4.md
Normal file
70
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v4.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# 第四次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `efd4f421-ee10-4244-833f-7b2d68c3c05b`
|
||||
- 时间: 2026-02-21 19:57:02 ~ 20:08:57
|
||||
- 耗时: 11 分 55 秒 (715s)
|
||||
- 整体状态: success (exit_code=0)
|
||||
- 任务总数: 31
|
||||
|
||||
## 成功任务 (10 个)
|
||||
|
||||
| # | 任务 | 耗时 | 统计 |
|
||||
|---|------|------|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | ~2m28s | fetched=367, inserted=367, deleted=367 |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | ~12s | fetched=25, inserted=25, deleted=25 |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | ~1m22s | fetched=486, inserted=486 |
|
||||
| 4 | DWS_ASSISTANT_SALARY | <1s | 非工资结算期,跳过 |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | ~1m10s | fetched=367, inserted=367, deleted=367 |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | ~1m46s | fetched=10366, updated=10366 |
|
||||
| 7 | ODS_PAYMENT | ~4m0s | fetched=42500, updated=42500 |
|
||||
| 8 | ODS_REFUND | ~3s | fetched=116, updated=116 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | ~1s | inserted=13296 |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION | ~43s | fetched=198, inserted=198 |
|
||||
|
||||
## BUG 4 修复验证
|
||||
|
||||
- DWS_MEMBER_CONSUMPTION ✅ 不再报 UndefinedColumn site_id
|
||||
- DWS_MEMBER_VISIT ❌ 新错误(BUG 5)
|
||||
- DWS_FINANCE_RECHARGE ❌ 级联失败(未能独立验证)
|
||||
|
||||
## 新发现 BUG 5
|
||||
|
||||
- 任务: `DWS_MEMBER_VISIT`
|
||||
- 错误: `UndefinedColumn: 字段 "birthday" 不存在`
|
||||
- 位置: `member_visit_task.py` → `_extract_member_info()` line ~312
|
||||
- 根因: SQL 查询 `dwd.dim_member` 时引用了 `birthday` 字段,但该表没有此字段
|
||||
- DWS 表 `dws_member_visit_detail` 设计了 `member_birthday DATE` 列,但上游 dim_member 未提供此数据
|
||||
- 级联影响: 后续 20 个任务全部 InFailedSqlTransaction
|
||||
|
||||
## 失败任务 (21 个)
|
||||
|
||||
| 类型 | 任务 | 错误 |
|
||||
|------|------|------|
|
||||
| 🔴 根因 | DWS_MEMBER_VISIT | UndefinedColumn: birthday |
|
||||
| 级联 | ODS_GOODS_CATEGORY | InFailedSqlTransaction |
|
||||
| 级联 | ODS_STORE_GOODS | InFailedSqlTransaction |
|
||||
| 级联 | ODS_STORE_GOODS_SALES | InFailedSqlTransaction |
|
||||
| 级联 | ODS_TENANT_GOODS | InFailedSqlTransaction |
|
||||
| 级联 | ODS_PLATFORM_COUPON | InFailedSqlTransaction |
|
||||
| 级联 | ODS_GROUP_PACKAGE | InFailedSqlTransaction |
|
||||
| 级联 | ODS_GROUP_BUY_REDEMPTION | InFailedSqlTransaction |
|
||||
| 级联 | ODS_INVENTORY_STOCK | InFailedSqlTransaction |
|
||||
| 级联 | ODS_INVENTORY_CHANGE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_DAILY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_WEEKLY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_MONTHLY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_DAILY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_RECHARGE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_INCOME_STRUCTURE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_DISCOUNT_DETAIL | InFailedSqlTransaction |
|
||||
| 级联 | DWS_WINBACK_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWS_NEWCONV_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWS_RELATION_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWD_LOAD_FROM_ODS | InFailedSqlTransaction |
|
||||
|
||||
## BUG 5 修复
|
||||
|
||||
- 文件: `member_visit_task.py`
|
||||
- 改动 1: `_extract_member_info` SQL 移除 `birthday` 字段
|
||||
- 改动 2: transform 中 `member_birthday` 改为 `None`
|
||||
- 已添加 CHANGE 注释
|
||||
69
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v5.md
Normal file
69
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v5.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# 第五次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `fe87144a-687d-4ce0-9b79-6bd0186b2be3`
|
||||
- 执行时间: 2026-02-21 20:19:52 ~ 20:31:29(约 11m37s)
|
||||
- exit_code: 0
|
||||
- 总任务数: 31
|
||||
|
||||
## 成功任务(10 个)
|
||||
|
||||
| # | 任务 |
|
||||
|---|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER |
|
||||
| 4 | DWS_ASSISTANT_SALARY |
|
||||
| 5 | DWS_ASSISTANT_FINANCE |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS ODS 任务完成 |
|
||||
| 7 | ODS_PAYMENT ODS 任务完成 |
|
||||
| 8 | ODS_REFUND ODS 任务完成 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION |
|
||||
|
||||
## 失败任务(21 个)
|
||||
|
||||
| # | 任务 | 错误类型 |
|
||||
|---|------|----------|
|
||||
| 1 | DWS_MEMBER_VISIT | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 2 | ODS_GOODS_CATEGORY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 3 | ODS_STORE_GOODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 4 | ODS_STORE_GOODS_SALES | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | ODS_TENANT_GOODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | ODS_PLATFORM_COUPON | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | ODS_GROUP_PACKAGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | ODS_GROUP_BUY_REDEMPTION | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 9 | ODS_INVENTORY_STOCK | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 10 | ODS_INVENTORY_CHANGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 11 | DWS_GOODS_STOCK_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | DWS_GOODS_STOCK_WEEKLY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | DWS_GOODS_STOCK_MONTHLY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 14 | DWS_FINANCE_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 15 | DWS_FINANCE_RECHARGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 16 | DWS_FINANCE_INCOME_STRUCTURE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 17 | DWS_FINANCE_DISCOUNT_DETAIL | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 18 | DWS_WINBACK_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 19 | DWS_NEWCONV_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 20 | DWS_RELATION_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWD_LOAD_FROM_ODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析
|
||||
|
||||
BUG 6: `DWS_MEMBER_VISIT` → `_extract_table_info()` 方法中 SQL 引用了 `dwd.dim_table.site_table_id`,
|
||||
但该表的主键字段实际为 `table_id`(参考 `db/etl_feiqiu/schemas/dwd.sql`)。
|
||||
|
||||
错误发生后,psycopg2 连接进入 InFailedSqlTransaction 状态,导致后续所有任务级联失败。
|
||||
|
||||
## 修复措施
|
||||
|
||||
1. `member_visit_task.py` → `_extract_table_info()`:
|
||||
- `site_table_id AS table_id` → `table_id AS table_id`
|
||||
- `site_table_name AS table_name` → `table_name AS table_name`
|
||||
|
||||
2. `finance_income_task.py` → `_extract_income_by_area()`:
|
||||
- JOIN 条件 `dt.site_table_id = tfl.site_table_id` → `dt.table_id = tfl.site_table_id`
|
||||
- JOIN 条件 `dt.site_table_id = asl.site_table_id` → `dt.table_id = asl.site_table_id`
|
||||
|
||||
## BUG 5 验证
|
||||
|
||||
BUG 5(birthday 字段)的修复已部署,但被 BUG 6 遮蔽,无法在本次执行中验证。
|
||||
需要第六次执行来同时验证 BUG 5 + BUG 6 + BUG 7。
|
||||
59
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v6.md
Normal file
59
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v6.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# 第六次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `d9443781-e4ac-4df6-9f87-11c45d72e5ba`
|
||||
- 执行时间: 2026-02-21 20:45:18 ~ 21:14:45(29 分 26 秒)
|
||||
- exit_code: 0
|
||||
- status: success
|
||||
- 总任务数: 31
|
||||
- 数据统计: 获取 171,961 / 新增 13,662 / 更新 171,595 / 跳过 0 / 错误 0
|
||||
|
||||
## 成功任务(11 个)
|
||||
|
||||
| # | 任务 |
|
||||
|---|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY: |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY: |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER: |
|
||||
| 4 | DWS_ASSISTANT_SALARY: |
|
||||
| 5 | DWS_ASSISTANT_FINANCE: |
|
||||
| 6 | {'fetched': |
|
||||
| 7 | DWS_MEMBER_CONSUMPTION: |
|
||||
| 8 | DWS_MEMBER_VISIT: |
|
||||
| 9 | DWS_GOODS_STOCK_DAILY: |
|
||||
| 10 | DWS_GOODS_STOCK_WEEKLY: |
|
||||
| 11 | DWS_GOODS_STOCK_MONTHLY: |
|
||||
|
||||
## 失败任务(8 个)
|
||||
|
||||
| # | 任务 | 错误类型 |
|
||||
|---|------|----------|
|
||||
| 1 | DWS_FINANCE_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 2 | DWS_FINANCE_RECHARGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 3 | DWS_FINANCE_INCOME_STRUCTURE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 4 | DWS_FINANCE_DISCOUNT_DETAIL | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | DWS_WINBACK_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | DWS_NEWCONV_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | DWS_RELATION_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | DWD_LOAD_FROM_ODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析(8 个非级联失败)
|
||||
|
||||
- `DWS_FINANCE_DAILY`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_RECHARGE`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_INCOME_STRUCTURE`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_DISCOUNT_DETAIL`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_WINBACK_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_NEWCONV_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_RELATION_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWD_LOAD_FROM_ODS`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
## 与前次对比
|
||||
|
||||
| 轮次 | 成功 | 失败 | 耗时 | 修复的 BUG |
|
||||
|------|------|------|------|-----------|
|
||||
| v1 | 10 | 31 | 9m51s | — |
|
||||
| v2 | — | — | 2m30s | BUG 1 |
|
||||
| v3 | 9 | 22 | 11m21s | BUG 2+3 |
|
||||
| v4 | 10 | 21 | 11m55s | BUG 4 |
|
||||
| v5 | 10 | 21 | 11m37s | BUG 5 |
|
||||
| v6 | 11 | 8 | 29m26s | BUG 5+6+7 |
|
||||
109
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v8.md
Normal file
109
export/SYSTEM/LOGS/2026-02-21__etl_run_result_v8.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# ETL 第八次执行报告 (v8)
|
||||
|
||||
- execution_id: `f943bac6-23be-45c5-8b8c-a864e85a1916`
|
||||
- 时间: 2026-02-21 21:33:37 ~ 21:35:01 (1分24秒)
|
||||
- 整体状态: success, exit_code=0
|
||||
|
||||
## 本次修复验证
|
||||
|
||||
| BUG | 修复内容 | 验证结果 |
|
||||
|-----|---------|---------|
|
||||
| BUG 8 | `finance_base_task.py` + `finance_recharge_task.py`: pay_money→pay_amount, gift_money→point_amount | ✅ DWS_FINANCE_DAILY + DWS_FINANCE_RECHARGE 均完成 |
|
||||
| BUG 9 | `dwd_load_task.py`: 添加 `_pick_snapshot_order_column` 方法 | ✅ 所有 dim 表 SCD2 装载成功 |
|
||||
| BUG 10 | `dwd_load_task.py`: FACT_MAPPINGS 驼峰字段名→小写 | ✅ dwd_goods_stock_summary(716条) + dwd_goods_stock_movement(14306条) 装载成功 |
|
||||
| BUG 11 | `flow_runner.py`: sum() 类型安全处理 | ✅ 不再出现 TypeError |
|
||||
|
||||
## DWD_LOAD_FROM_ODS 详情
|
||||
|
||||
### 维度表 (SCD2) — 全部成功
|
||||
| 表 | processed | inserted | updated |
|
||||
|----|-----------|----------|---------|
|
||||
| dim_site | 1 | 0 | 1 |
|
||||
| dim_site_ex | 1 | 0 | 1 |
|
||||
| dim_table | 74 | 0 | 74 |
|
||||
| dim_table_ex | 74 | 0 | 74 |
|
||||
| dim_assistant | 69 | 0 | 69 |
|
||||
| dim_member | 557 | 0 | 557 |
|
||||
| dim_member_ex | 557 | 0 | 557 |
|
||||
| dim_member_card_account | 946 | 0 | 946 |
|
||||
| dim_tenant_goods | 174 | 1 | 173 |
|
||||
| dim_tenant_goods_ex | 174 | 1 | 173 |
|
||||
| dim_store_goods | 173 | 1 | 172 |
|
||||
| dim_store_goods_ex | 173 | 1 | 172 |
|
||||
| dim_goods_category | 26 | 0 | 26 |
|
||||
| dim_groupbuy_package | 34 | 0 | 34 |
|
||||
| dim_groupbuy_package_ex | 34 | 0 | 34 |
|
||||
|
||||
### 事实表 (INCREMENT) — 全部成功
|
||||
| 表 | processed | inserted | updated |
|
||||
|----|-----------|----------|---------|
|
||||
| dwd_settlement_head | 10366 | 0 | 10366 |
|
||||
| dwd_settlement_head_ex | 10366 | 0 | 10366 |
|
||||
| dwd_table_fee_log | 9103 | 0 | 9103 |
|
||||
| dwd_table_fee_log_ex | 9103 | 0 | 9103 |
|
||||
| dwd_table_fee_adjust | 1616 | 0 | 1616 |
|
||||
| dwd_table_fee_adjust_ex | 1616 | 0 | 1616 |
|
||||
| dwd_assistant_service_log | 2619 | 0 | 2619 |
|
||||
| dwd_assistant_service_log_ex | 2619 | 0 | 2619 |
|
||||
| dwd_assistant_trash_event | 78 | 0 | 78 |
|
||||
| dwd_assistant_trash_event_ex | 78 | 0 | 78 |
|
||||
| dwd_member_balance_change | 2185 | 0 | 2185 |
|
||||
| dwd_member_balance_change_ex | 2185 | 0 | 2185 |
|
||||
| dwd_groupbuy_redemption | 7267 | 0 | 7267 |
|
||||
| dwd_groupbuy_redemption_ex | 7267 | 0 | 7267 |
|
||||
| dwd_platform_coupon_redemption | 18311 | 0 | 18311 |
|
||||
| dwd_platform_coupon_redemption_ex | 18311 | 0 | 18311 |
|
||||
| dwd_recharge_order | 191 | 0 | 191 |
|
||||
| dwd_recharge_order_ex | 191 | 0 | 191 |
|
||||
| dwd_payment | 10625 | 0 | 10625 |
|
||||
| dwd_refund | 29 | 0 | 29 |
|
||||
| dwd_refund_ex | 29 | 0 | 29 |
|
||||
| dwd_goods_stock_summary | 716 | 716 | 0 |
|
||||
| dwd_goods_stock_movement | 14306 | 14306 | 0 |
|
||||
|
||||
### DWD 装载错误 (2个,数据质量问题,非代码 BUG)
|
||||
| 表 | 错误 |
|
||||
|----|------|
|
||||
| dim_assistant_ex | year -1 is out of range |
|
||||
| dim_member_card_account_ex | year -1 is out of range |
|
||||
|
||||
## DWS 任务状态
|
||||
|
||||
| 任务 | 状态 | 备注 |
|
||||
|------|------|------|
|
||||
| ODS_FETCH | ✅ 完成 | |
|
||||
| DWD_LOAD_FROM_ODS | ✅ 完成 | 39表成功,2表数据质量错误 |
|
||||
| DWS_ASSISTANT_DAILY | ✅ 完成 | |
|
||||
| DWS_ASSISTANT_MONTHLY | ✅ 完成 | 删除9行,插入9行 |
|
||||
| DWS_ASSISTANT_CUSTOMER | ✅ 完成 | 删除285行,插入285行 |
|
||||
| DWS_ASSISTANT_SALARY | ✅ 完成 | |
|
||||
| DWS_ASSISTANT_FINANCE | ✅ 完成 | |
|
||||
| DWS_MEMBER_CONSUMPTION | ✅ 完成 | 删除198行,插入198行 |
|
||||
| DWS_MEMBER_VISIT | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_DAILY | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_WEEKLY | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_MONTHLY | ✅ 完成 | |
|
||||
| DWS_FINANCE_DAILY | ✅ 完成 | |
|
||||
| DWS_FINANCE_RECHARGE | ✅ 完成 | |
|
||||
| DWS_FINANCE_INCOME_STRUCTURE | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_FINANCE_DISCOUNT_DETAIL | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_WINBACK_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_NEWCONV_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_RELATION_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
|
||||
## 总结
|
||||
|
||||
- 14/19 任务成功完成
|
||||
- 5/19 任务因 InFailedSqlTransaction 级联失败
|
||||
- 级联失败根因: `dim_assistant_ex` 和 `dim_member_card_account_ex` 中存在非法日期值 (year=-1),导致事务进入失败状态
|
||||
- 这是数据质量问题,不是代码 BUG — 需要在 DWD 装载时对日期字段做容错处理
|
||||
|
||||
## 与 v6(上次最好成绩)对比
|
||||
|
||||
| 指标 | v6 | v8 |
|
||||
|------|----|----|
|
||||
| 耗时 | 29m26s | 1m24s |
|
||||
| 成功任务 | 11/19 | 14/19 |
|
||||
| 失败任务 | 8/19 | 5/19 |
|
||||
| DWD 装载 | 部分 dim 失败 | 39/41 表成功 |
|
||||
| 新增成功 | — | DWS_FINANCE_DAILY, DWS_FINANCE_RECHARGE, DWS_GOODS_STOCK_* |
|
||||
Reference in New Issue
Block a user