feat: P1-P3 全栈集成 — 数据库基础 + DWS 扩展 + 小程序鉴权 + 工程化体系
## P1 数据库基础 - zqyy_app: 创建 auth/biz schema、FDW 连接 etl_feiqiu - etl_feiqiu: 创建 app schema RLS 视图、商品库存预警表 - 清理 assistant_abolish 残留数据 ## P2 ETL/DWS 扩展 - 新增 DWS 助教订单贡献度表 (dws.assistant_order_contribution) - 新增 assistant_order_contribution_task 任务及 RLS 视图 - member_consumption 增加充值字段、assistant_daily 增加处罚字段 - 更新 ODS/DWD/DWS 任务文档及业务规则文档 - 更新 consistency_checker、flow_runner、task_registry 等核心模块 ## P3 小程序鉴权系统 - 新增 xcx_auth 路由/schema(微信登录 + JWT) - 新增 wechat/role/matching/application 服务层 - zqyy_app 鉴权表迁移 + 角色权限种子数据 - auth/dependencies.py 支持小程序 JWT 鉴权 ## 文档与审计 - 新增 DOCUMENTATION-MAP 文档导航 - 新增 7 份 BD_Manual 数据库变更文档 - 更新 DDL 基线快照(etl_feiqiu 6 schema + zqyy_app auth) - 新增全栈集成审计记录、部署检查清单更新 - 新增 BACKLOG 路线图、FDW→Core 迁移计划 ## Kiro 工程化 - 新增 5 个 Spec(P1/P2/P3/全栈集成/核心业务) - 新增审计自动化脚本(agent_on_stop/build_audit_context/compliance_prescan) - 新增 6 个 Hook(合规检查/会话日志/提交审计等) - 新增 doc-map steering 文件 ## 运维与测试 - 新增 ops 脚本:迁移验证/API 健康检查/ETL 监控/集成报告 - 新增属性测试:test_dws_contribution / test_auth_system - 清理过期 export 报告文件 - 更新 .gitignore 排除规则
This commit is contained in:
@@ -1,178 +0,0 @@
|
||||
# DWS_ASSISTANT_DAILY BUG 修复报告
|
||||
|
||||
> 生成时间:2026-02-21 19:13:11
|
||||
> 执行 run_uuid:4ba9d2d365ee4a858f1c4104b1942dc2
|
||||
> 执行开始:2026-02-21 15:29:20
|
||||
|
||||
---
|
||||
|
||||
## 1. BUG 概述
|
||||
|
||||
ETL 执行过程中 `DWS_ASSISTANT_DAILY` 任务失败,根因是 `assistant_daily_task.py` 中
|
||||
`_extract_trash_records` 方法的 SQL 引用了 `dwd.dwd_assistant_trash_event` 表中不存在的字段。
|
||||
|
||||
### 错误信息
|
||||
|
||||
```
|
||||
psycopg2.errors.UndefinedColumn: 错误: 字段 "assistant_service_id" 不存在
|
||||
LINE 3: assistant_service_id,
|
||||
^
|
||||
```
|
||||
|
||||
### 级联影响
|
||||
|
||||
`DWS_ASSISTANT_DAILY` 失败后,psycopg2 连接进入 `InFailedSqlTransaction` 状态,
|
||||
级联导致以下 8 个任务全部失败:
|
||||
|
||||
| # | 任务代码 | 失败原因 |
|
||||
|---|---------|---------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | 根因:UndefinedColumn |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | InFailedSqlTransaction(级联) |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | InFailedSqlTransaction(级联) |
|
||||
| 4 | DWS_ASSISTANT_SALARY | InFailedSqlTransaction(级联) |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | InFailedSqlTransaction(级联) |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | InFailedSqlTransaction(级联) |
|
||||
| 7 | ODS_PAYMENT | InFailedSqlTransaction(级联) |
|
||||
| 8 | ODS_REFUND | InFailedSqlTransaction(级联) |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | InFailedSqlTransaction(级联) |
|
||||
|
||||
从 `ODS_TABLE_USE` 开始,task_executor 的连接恢复机制生效,后续任务恢复正常执行。
|
||||
|
||||
---
|
||||
|
||||
## 2. 根因分析
|
||||
|
||||
### 2.1 错误 SQL(修复前)
|
||||
|
||||
```sql
|
||||
SELECT assistant_service_id, trash_seconds, trash_reason, trash_time
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s AND DATE(trash_time) >= %s AND DATE(trash_time) <= %s
|
||||
```
|
||||
|
||||
### 2.2 `dwd_assistant_trash_event` 实际表结构
|
||||
|
||||
| 字段名 | 类型 | 说明 |
|
||||
|--------|------|------|
|
||||
| assistant_trash_event_id | BIGINT (PK) | 废除事件 ID |
|
||||
| site_id | BIGINT | 门店 ID |
|
||||
| table_id | BIGINT | 台桌 ID |
|
||||
| table_area_id | BIGINT | 区域 ID |
|
||||
| assistant_no | VARCHAR(32) | 助教编号 |
|
||||
| assistant_name | VARCHAR(64) | 助教姓名 |
|
||||
| charge_minutes_raw | INTEGER | 废除时长(分钟) |
|
||||
| abolish_amount | NUMERIC(18,2) | 废除金额 |
|
||||
| trash_reason | VARCHAR(255) | 废除原因 |
|
||||
| create_time | TIMESTAMPTZ | 废除时间 |
|
||||
| tenant_id | BIGINT | 租户 ID |
|
||||
|
||||
### 2.3 字段映射错误
|
||||
|
||||
| 错误引用 | 实际字段 | 说明 |
|
||||
|----------|---------|------|
|
||||
| `assistant_service_id` | `assistant_trash_event_id` | PK 名称不同 |
|
||||
| `trash_seconds` | `charge_minutes_raw` | 单位不同(分钟 vs 秒) |
|
||||
| `trash_time` | `create_time` | 字段名不同 |
|
||||
|
||||
### 2.4 深层设计缺陷
|
||||
|
||||
废除表 `dwd_assistant_trash_event` 没有 `assistant_service_id` 外键,
|
||||
无法与服务记录表 `dwd_assistant_service_log` 做 1:1 关联。
|
||||
|
||||
原代码的 `_build_trash_index` 用 `assistant_service_id` 做 key 构建索引,
|
||||
`_aggregate_by_assistant_date` 用 `service_id in trash_index` 判断服务是否被废除。
|
||||
即使 SQL 字段名修正后,这个匹配逻辑在设计上也是无效的——两个 ID 不同源。
|
||||
|
||||
---
|
||||
|
||||
## 3. 修复方案
|
||||
|
||||
### 3.1 文件
|
||||
|
||||
`apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py`
|
||||
|
||||
### 3.2 修改点(共 4 处)
|
||||
|
||||
#### (1) `_extract_trash_records` — SQL 字段名修正
|
||||
|
||||
```sql
|
||||
-- 修复后
|
||||
SELECT
|
||||
assistant_trash_event_id,
|
||||
charge_minutes_raw * 60 AS trash_seconds,
|
||||
trash_reason,
|
||||
create_time AS trash_time,
|
||||
table_id,
|
||||
assistant_name
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s
|
||||
AND DATE(create_time) >= %s
|
||||
AND DATE(create_time) <= %s
|
||||
```
|
||||
|
||||
#### (2) `_extract_service_records` — JOIN _ex 表取 is_trash
|
||||
|
||||
```sql
|
||||
-- 新增 LEFT JOIN 和 is_trash 字段
|
||||
SELECT
|
||||
asl.assistant_service_id,
|
||||
...
|
||||
DATE(asl.start_use_time) AS service_date,
|
||||
COALESCE(ex.is_trash, 0) AS is_trash
|
||||
FROM dwd.dwd_assistant_service_log asl
|
||||
LEFT JOIN dwd.dwd_assistant_service_log_ex ex
|
||||
ON asl.assistant_service_id = ex.assistant_service_id
|
||||
WHERE asl.site_id = %s
|
||||
AND DATE(asl.start_use_time) >= %s
|
||||
AND DATE(asl.start_use_time) <= %s
|
||||
AND asl.is_delete = 0
|
||||
```
|
||||
|
||||
#### (3) `_build_trash_index` — key 改为 assistant_trash_event_id
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
service_id = record.get('assistant_service_id')
|
||||
|
||||
# 修复后
|
||||
event_id = record.get('assistant_trash_event_id')
|
||||
```
|
||||
|
||||
#### (4) `_aggregate_by_assistant_date` — 废除判断改用 is_trash
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
is_trashed = service_id in trash_index
|
||||
|
||||
# 修复后
|
||||
is_trashed = bool(record.get('is_trash', 0))
|
||||
```
|
||||
|
||||
废除时长也从 `trash_index[service_id]` 改为直接用 `income_seconds`。
|
||||
|
||||
### 3.3 设计决策说明
|
||||
|
||||
`dwd_assistant_service_log_ex` 表的 `is_trash` 字段来自上游 SaaS 系统的
|
||||
`assistant_service_records` API,是服务记录级别的废除标记,比跨表匹配更可靠。
|
||||
|
||||
废除时长统计改用服务记录自身的 `income_seconds`(即该服务的计费时长),
|
||||
而非从废除表取 `charge_minutes_raw`(废除事件的计费分钟数),
|
||||
因为两者无法 1:1 关联。
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证计划
|
||||
|
||||
修复将在下次 ETL 执行时生效。验证步骤:
|
||||
|
||||
1. 重新提交包含 `DWS_ASSISTANT_DAILY` 的执行
|
||||
2. 确认无 SQL 错误
|
||||
3. 检查 `dws.dws_assistant_daily` 表中 `trashed_count` / `trashed_seconds` 是否合理
|
||||
4. 对比 `dwd_assistant_service_log_ex.is_trash = 1` 的记录数与 DWS 汇总的 `trashed_count`
|
||||
|
||||
---
|
||||
|
||||
## 5. 回滚方案
|
||||
|
||||
如需回滚,恢复 `assistant_daily_task.py` 到修改前版本即可。
|
||||
DWS 表数据可通过重新执行 `DWS_ASSISTANT_DAILY` 任务覆盖。
|
||||
@@ -1,443 +0,0 @@
|
||||
# ETL 前后端联调 — BUG 修复全记录
|
||||
|
||||
> 日期: 2026-02-21
|
||||
> 执行轮次: v1 ~ v8(共 8 次)
|
||||
> 任务配置: api_full, full_window, 2025-11-01 ~ 2026-02-20, 30天窗口切分, force_full, 19个任务
|
||||
|
||||
---
|
||||
|
||||
## 总览
|
||||
|
||||
| 指标 | v1 (首次) | v6 (中期最佳) | v8 (最终) |
|
||||
|------|-----------|--------------|-----------|
|
||||
| 耗时 | 590.7s | 29m26s | 1m24s |
|
||||
| 成功任务 | 10/41 | 11/19 | 14/19 |
|
||||
| 失败任务 | 31/41 | 8/19 | 5/19 |
|
||||
| 累计修复 BUG | 0 | 7 | 11 |
|
||||
|
||||
最终 5 个失败均为 `InFailedSqlTransaction` 级联(根因是上游数据质量问题,非代码 BUG)。
|
||||
|
||||
---
|
||||
|
||||
## BUG 详情
|
||||
|
||||
### BUG 1 — DWS_ASSISTANT_DAILY SQL 字段引用错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v1 |
|
||||
| 验证版本 | v2 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: 错误: 字段 "assistant_service_id" 不存在`,DWS_ASSISTANT_DAILY 及其下游 31 个任务全部失败(InFailedSqlTransaction 级联) |
|
||||
| 根因 | `_extract_trash_records()` 方法的 SQL 引用了 `dwd.dwd_assistant_trash_event` 表中不存在的 3 个字段名,且废除判断逻辑依赖跨表 ID 匹配(两表无外键关联,设计上无效) |
|
||||
| 涉及表 | `dwd.dwd_assistant_trash_event`、`dwd.dwd_assistant_service_log`、`dwd.dwd_assistant_service_log_ex` |
|
||||
|
||||
#### 字段映射错误(3 处 SQL 字段名不匹配)
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 所属表 | 说明 |
|
||||
|---|-------------------|----------------------|--------|------|
|
||||
| 1 | `assistant_service_id` | `assistant_trash_event_id` | `dwd_assistant_trash_event` | PK 名称不同,废除表没有 service_id 外键 |
|
||||
| 2 | `trash_seconds` | `charge_minutes_raw * 60 AS trash_seconds` | `dwd_assistant_trash_event` | 实际字段存储分钟数,需乘 60 转秒 |
|
||||
| 3 | `trash_time` | `create_time AS trash_time` | `dwd_assistant_trash_event` | 废除时间字段名不同 |
|
||||
|
||||
#### 逻辑修复(1 处设计缺陷修正)
|
||||
|
||||
| # | 修复点 | 修复前 | 修复后 | 说明 |
|
||||
|---|--------|--------|--------|------|
|
||||
| 4 | `_aggregate_by_assistant_date` 废除判断 | `is_trashed = service_id in trash_index`(跨表 ID 匹配) | `is_trashed = bool(record.get('is_trash', 0))`(使用 _ex 表标记) | 原逻辑依赖废除表的 event_id 与服务表的 service_id 匹配,但两者无外键关联,匹配永远为空。改为 LEFT JOIN `dwd_assistant_service_log_ex` 取 `is_trash` 字段直接判断 |
|
||||
|
||||
#### 修复后 SQL(`_extract_trash_records`)
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
assistant_trash_event_id,
|
||||
charge_minutes_raw * 60 AS trash_seconds,
|
||||
trash_reason,
|
||||
create_time AS trash_time,
|
||||
table_id,
|
||||
assistant_name
|
||||
FROM dwd.dwd_assistant_trash_event
|
||||
WHERE site_id = %s
|
||||
AND DATE(create_time) >= %s AND DATE(create_time) <= %s
|
||||
```
|
||||
|
||||
#### 修复后 SQL(`_extract_service_records` 新增 JOIN)
|
||||
|
||||
```sql
|
||||
SELECT asl.assistant_service_id, ...,
|
||||
COALESCE(ex.is_trash, 0) AS is_trash
|
||||
FROM dwd.dwd_assistant_service_log asl
|
||||
LEFT JOIN dwd.dwd_assistant_service_log_ex ex
|
||||
ON asl.assistant_service_id = ex.assistant_service_id
|
||||
WHERE asl.site_id = %s AND ...
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v2 中 DWS_ASSISTANT_DAILY 执行成功 |
|
||||
|------|------|
|
||||
| 详细修复记录 | `export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md` |
|
||||
|
||||
### BUG 2 — DWS_ASSISTANT_MONTHLY GROUP BY 聚合错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v2 |
|
||||
| 验证版本 | v3 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_monthly_task.py` |
|
||||
| 错误现象 | `UniqueViolation: duplicate key value violates unique constraint "uk_dws_assistant_monthly"` |
|
||||
| 根因 | `_extract_daily_aggregates()` 的 GROUP BY 包含了 `assistant_nickname`、`assistant_level_code`、`assistant_level_name` 三个维度字段。当助教在月内改名或升级时,同一 `(assistant_id, stat_month)` 会产出多行,INSERT 时违反 `uk_dws_assistant_monthly` 唯一约束 |
|
||||
| 涉及表 | `dws.dws_assistant_daily_detail`(源)→ `dws.dws_assistant_monthly_summary`(目标) |
|
||||
|
||||
#### 业务场景
|
||||
|
||||
助教月内改名/升级是正常业务操作。例如助教 A 在 12 月 15 日从"初级"升为"中级",则 `dws_assistant_daily_detail` 中 12 月的记录会有两种 `assistant_level_code`。原 SQL 按 `(assistant_id, stat_month, assistant_level_code)` 分组,产出 2 行,但月度汇总表的唯一约束只有 `(site_id, assistant_id, stat_month)`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
将 `assistant_nickname`、`assistant_level_code`、`assistant_level_name` 从 GROUP BY 移除,改用 `MAX()` 聚合(取月内最新值):
|
||||
|
||||
```sql
|
||||
-- 修复前 GROUP BY
|
||||
GROUP BY assistant_id, assistant_nickname, assistant_level_code, assistant_level_name, DATE_TRUNC('month', stat_date)
|
||||
|
||||
-- 修复后 GROUP BY
|
||||
GROUP BY assistant_id, DATE_TRUNC('month', stat_date)
|
||||
|
||||
-- 维度字段改为 MAX() 聚合
|
||||
MAX(assistant_nickname) AS assistant_nickname,
|
||||
MAX(assistant_level_code) AS assistant_level_code,
|
||||
MAX(assistant_level_name) AS assistant_level_name,
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v3 中 DWS_ASSISTANT_MONTHLY 执行成功(删除 9 行,插入 9 行) |
|
||||
|------|------|
|
||||
|
||||
### BUG 3 — DWS_ASSISTANT_CUSTOMER 引用不存在的 site_id 列
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v2 |
|
||||
| 验证版本 | v3 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/assistant_customer_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column dm.site_id does not exist` |
|
||||
| 根因 | `dwd.dim_member` 表没有 `site_id` 列,实际字段为 `register_site_id` |
|
||||
| 修复方式 | `dm.site_id` → `dm.register_site_id` |
|
||||
| 修复结果 | ✅ v3 中 DWS_ASSISTANT_CUSTOMER 执行成功(285 行) |
|
||||
|
||||
### BUG 4 — 多个 DWS 任务引用 dim_member/dim_member_card_account 的 site_id
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v3 |
|
||||
| 验证版本 | v4 |
|
||||
| 文件 | `assistant_customer_task.py`、`member_consumption_task.py`、`finance_recharge_task.py`(共 4 处) |
|
||||
| 错误现象 | 多个 DWS 任务因 `UndefinedColumn: site_id` 失败 |
|
||||
| 根因 | 与 BUG 3 同源 — `dim_member` 和 `dim_member_card_account` 均无 `site_id`,需用 `register_site_id` |
|
||||
| 修复方式 | 4 处 `site_id` → `register_site_id` |
|
||||
| 修复结果 | ✅ v4 中相关任务执行成功 |
|
||||
|
||||
### BUG 5 — DWS_MEMBER_VISIT 引用不存在的 birthday 字段
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v4 |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column dm.birthday does not exist` |
|
||||
| 根因 | `_extract_member_info()` 的 SQL 中 SELECT 了 `birthday` 字段,但 `dwd.dim_member` 表没有该字段 |
|
||||
| 涉及表 | `dwd.dim_member` |
|
||||
|
||||
#### 业务场景
|
||||
|
||||
`dim_member` 的数据来源是上游 SaaS 的 `member_profiles` API。该 API 返回的会员信息中不包含生日字段,因此 ODS → DWD 装载时也没有 `birthday` 列。原代码假设 `dim_member` 有 `birthday`,用于在 `dws_member_visit` 中记录会员生日,但这个字段从未存在过。
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dim_member` 实际字段:`member_id`, `system_member_id`, `tenant_id`, `register_site_id`, `mobile`, `nickname`, `member_card_grade_code`, `member_card_grade_name`, `create_time`, `update_time`, `pay_money_sum`, `recharge_money_sum` + SCD2 元数据。无 `birthday`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
1. `_extract_member_info()` SQL 中移除 `birthday`,只保留 `member_id`, `nickname`, `mobile`
|
||||
2. `transform()` 中 `member_birthday` 字段固定填 `None`(DWS 表该列保留但值为空)
|
||||
|
||||
```python
|
||||
# 修复前
|
||||
sql = "SELECT member_id, nickname, mobile, birthday FROM dwd.dim_member ..."
|
||||
|
||||
# 修复后
|
||||
sql = "SELECT member_id, nickname, mobile FROM dwd.dim_member ..."
|
||||
|
||||
# transform 中
|
||||
'member_birthday': None, # dim_member 无 birthday 字段
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v6 中 DWS_MEMBER_VISIT 执行成功(v5 被 BUG 6 遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 6 — DWS_MEMBER_VISIT _extract_table_info() 字段名不匹配
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v5 |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` |
|
||||
| 错误现象 | `UndefinedColumn` — `_extract_table_info()` 方法中引用了 `dim_table` 中不存在的列名 |
|
||||
| 根因 | 代码中用 `site_table_id` 和 `site_table_name` 查询 `dim_table`,但该表的 PK 是 `table_id`,名称字段是 `table_name`。`site_table_id` 是事实表 `dwd_table_fee_log` 中的外键列名,不是维度表的列名 |
|
||||
| 涉及表 | `dwd.dim_table` |
|
||||
|
||||
#### 字段映射错误
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 说明 |
|
||||
|---|-------------------|----------------------|------|
|
||||
| 1 | `site_table_id` | `table_id` | `dim_table` 的 PK 是 `table_id`,不是 `site_table_id` |
|
||||
| 2 | `site_table_name` | `table_name` | `dim_table` 的名称字段是 `table_name` |
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dim_table` 实际字段:`table_id`(PK), `site_id`, `table_name`, `site_table_area_id`, `site_table_area_name`, `tenant_table_area_id`, `table_price`, `order_id` + SCD2 元数据。
|
||||
|
||||
#### 修复后 SQL
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
table_id AS table_id,
|
||||
table_name AS table_name,
|
||||
site_table_area_name AS area_name
|
||||
FROM dwd.dim_table
|
||||
WHERE site_id = %s AND scd2_is_current = 1
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后字段映射与 DDL 完全一致 ✅:
|
||||
- `table_id` — DDL 中存在,是 PK ✅
|
||||
- `table_name` — DDL 中存在 ✅
|
||||
- `site_table_area_name` — DDL 中存在 ✅
|
||||
- `site_id` 用于 WHERE 过滤 — DDL 中存在 ✅
|
||||
|
||||
| 修复结果 | ✅ v6 中 DWS_MEMBER_VISIT 执行成功 |
|
||||
|------|------|
|
||||
|
||||
### BUG 7 — DWS_FINANCE_INCOME_STRUCTURE JOIN 条件列名错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | 预防性修复(v5 代码审查发现) |
|
||||
| 验证版本 | v6 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dws/finance_income_task.py` |
|
||||
| 错误现象 | JOIN 条件中 `dt.site_table_id` 不存在(`dim_table` 无 `site_table_id` 列) |
|
||||
| 根因 | `_extract_income_by_area()` 中 `dwd_table_fee_log` 和 `dwd_assistant_service_log` 与 `dim_table` 做 LEFT JOIN 时,JOIN 条件写成了 `dt.site_table_id = tfl.site_table_id`,但 `dim_table` 的 PK 是 `table_id` 而非 `site_table_id` |
|
||||
| 涉及表 | `dwd.dim_table`(维度)、`dwd.dwd_table_fee_log`(事实)、`dwd.dwd_assistant_service_log`(事实) |
|
||||
|
||||
#### 关联关系说明
|
||||
|
||||
事实表 `dwd_table_fee_log` 和 `dwd_assistant_service_log` 中的 `site_table_id` 是外键,指向维度表 `dim_table` 的 PK `table_id`。两个 ID 值相同,但列名不同。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
```sql
|
||||
-- 修复前(2 处)
|
||||
LEFT JOIN dwd.dim_table dt ON dt.site_table_id = tfl.site_table_id
|
||||
LEFT JOIN dwd.dim_table dt ON dt.site_table_id = asl.site_table_id
|
||||
|
||||
-- 修复后(2 处)
|
||||
LEFT JOIN dwd.dim_table dt ON dt.table_id = tfl.site_table_id
|
||||
LEFT JOIN dwd.dim_table dt ON dt.table_id = asl.site_table_id
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后 JOIN 条件与 DDL 一致 ✅:
|
||||
- `dim_table.table_id` — DDL 确认是 PK ✅
|
||||
- `dwd_table_fee_log.site_table_id` — DDL 确认存在,语义为"台桌 ID 外键" ✅
|
||||
- `dwd_assistant_service_log.site_table_id` — 同上 ✅
|
||||
- JOIN 方向正确:维度表 PK ← 事实表 FK ✅
|
||||
|
||||
| 修复结果 | ✅ v6 中未出现该错误(但被 BUG 8 级联遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 8 — DWS_FINANCE_DAILY / DWS_FINANCE_RECHARGE 字段名错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v6 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `finance_base_task.py`、`finance_recharge_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column "pay_money" does not exist`,DWS_FINANCE_DAILY 失败并级联导致 7 个下游任务失败 |
|
||||
| 根因 | 代码中用 `pay_money` / `gift_money` 查询 `dwd.dwd_recharge_order`,但该表的实际字段是 `pay_amount`(现金充值金额)/ `point_amount`(赠送金额) |
|
||||
| 涉及表 | `dwd.dwd_recharge_order` |
|
||||
|
||||
#### 字段映射错误
|
||||
|
||||
| # | 错误引用(修复前) | 实际 DDL 字段(修复后) | 业务含义 |
|
||||
|---|-------------------|----------------------|---------|
|
||||
| 1 | `pay_money` | `pay_amount` | 现金充值金额(会员实际支付) |
|
||||
| 2 | `gift_money` | `point_amount` | 赠送金额(充值赠送的积分/赠送卡金额) |
|
||||
|
||||
#### DDL 确认
|
||||
|
||||
`dwd_recharge_order` 金额相关字段:`pay_amount` NUMERIC(18,2)、`refund_amount` NUMERIC(18,2)、`point_amount` NUMERIC(18,2)、`cash_amount` NUMERIC(18,2)。无 `pay_money` 或 `gift_money`。
|
||||
|
||||
#### 修复涉及 2 个文件
|
||||
|
||||
1. `finance_base_task.py` — `_extract_recharge_summary()` 方法(被 `FinanceDailyTask` 继承调用)
|
||||
2. `finance_recharge_task.py` — `_extract_recharge_summary()` 方法(`FinanceRechargeTask` 自身的重写版本)
|
||||
|
||||
两处修复内容相同:所有 `pay_money` → `pay_amount`,所有 `gift_money` → `point_amount`。
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后字段映射与 DDL 完全一致 ✅:
|
||||
- `pay_amount` — DDL 确认存在,NUMERIC(18,2) ✅
|
||||
- `point_amount` — DDL 确认存在,NUMERIC(18,2) ✅
|
||||
- `is_first` — DDL 确认存在,INTEGER ✅
|
||||
- `member_id` — DDL 确认存在 ✅
|
||||
- `pay_time` — DDL 确认存在,TIMESTAMPTZ ✅
|
||||
- `site_id` — DDL 确认存在 ✅
|
||||
- 业务语义:`pay_amount + point_amount` = 充值总额(现金 + 赠送),`is_first = 1` 区分首充/续充 ✅
|
||||
|
||||
| 修复结果 | ✅ v8 中 DWS_FINANCE_DAILY 和 DWS_FINANCE_RECHARGE 均执行成功(v7 被 BUG 9 遮蔽) |
|
||||
|------|------|
|
||||
|
||||
### BUG 9 — DWD_LOAD_FROM_ODS 缺少 _pick_snapshot_order_column 方法
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` |
|
||||
| 错误现象 | `AttributeError: 'DwdLoadTask' object has no attribute '_pick_snapshot_order_column'`,所有 dim 表 SCD2 装载全部失败 |
|
||||
| 根因 | `_merge_dim_scd2()` 方法内部调用了 `self._pick_snapshot_order_column(cols)`,但该方法只存在于 `quality/integrity_checker.py` 中作为模块级函数,`DwdLoadTask` 类中没有定义。这是代码重构时遗漏的问题——SCD2 装载逻辑从 integrity_checker 迁移到 DwdLoadTask 时,忘记把依赖的辅助函数一起迁移 |
|
||||
| 涉及表 | 所有 dim 表(dim_site, dim_table, dim_assistant, dim_member, dim_member_card_account, dim_tenant_goods, dim_store_goods, dim_goods_category, dim_groupbuy_package 及其 _ex 表) |
|
||||
|
||||
#### 方法功能说明
|
||||
|
||||
`_pick_snapshot_order_column(cols)` 用于从 ODS 表的列名列表中选取快照排序列,优先级为 `fetched_at` > `update_time` > `create_time`。SCD2 装载时需要按此列排序,确保同一业务主键的多条快照按时间顺序处理,最新快照覆盖旧快照。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
在 `DwdLoadTask` 类中添加 `@staticmethod` 方法,逻辑与 `integrity_checker.py` 中的同名函数一致:
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def _pick_snapshot_order_column(cols: Sequence[str]) -> str | None:
|
||||
"""从 ODS 列中选取用于快照排序的列(fetched_at > update_time > create_time)。"""
|
||||
lower = {c.lower() for c in cols}
|
||||
for candidate in ("fetched_at", "update_time", "create_time"):
|
||||
if candidate in lower:
|
||||
return candidate
|
||||
return None
|
||||
```
|
||||
|
||||
| 修复结果 | ✅ v8 中所有 15 个 dim 表 SCD2 装载成功(dim_site, dim_table, dim_assistant, dim_member, dim_member_card_account, dim_tenant_goods, dim_store_goods, dim_goods_category, dim_groupbuy_package 及其 _ex 表) |
|
||||
|------|------|
|
||||
|
||||
### BUG 10 — goods_stock 表 FACT_MAPPINGS 驼峰字段名导致 SQL 错误
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` |
|
||||
| 错误现象 | `UndefinedColumn: column "siteGoodsId" does not exist, perhaps you mean "sitegoodsid"` |
|
||||
| 根因 | `FACT_MAPPINGS` 中 `dwd_goods_stock_summary` 和 `dwd_goods_stock_movement` 的源列使用了带引号的驼峰名(如 `"siteGoodsId"`),但 ODS 表中 PostgreSQL 存储的列名是全小写的 `sitegoodsid`(ODS 入库时 `_int_col("sitegoodsid", "siteGoodsId")` 已将 JSON 驼峰键转为小写列名) |
|
||||
| 修复方式 | 将 FACT_MAPPINGS 中 2 个表共 30+ 个字段的驼峰引用全部改为小写(如 `"siteGoodsId"` → `"sitegoodsid"`) |
|
||||
| 修复结果 | ✅ v8 中 `dwd_goods_stock_summary`(716 条 INSERT)和 `dwd_goods_stock_movement`(14306 条 INSERT)装载成功 |
|
||||
|
||||
### BUG 11 — flow_runner.py sum() 类型不安全
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v7 |
|
||||
| 验证版本 | v8 |
|
||||
| 文件 | `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` |
|
||||
| 错误现象 | `TypeError: unsupported operand type(s) for +: 'int' and 'list'` |
|
||||
| 根因 | Flow 执行完成后汇总各任务的 `counts.errors` 时,某些任务返回的 `errors` 是错误详情列表(`list[str]`)而非错误计数(`int`),`sum()` 无法将 `int` 与 `list` 相加 |
|
||||
| 涉及位置 | `FlowRunner.run()` 方法末尾的 `flow_logger.set_counts(...)` 调用 |
|
||||
|
||||
#### 触发场景
|
||||
|
||||
任务执行器 `task_executor` 返回的 `result["counts"]["errors"]` 类型不统一:
|
||||
- 大部分任务返回 `int`(错误计数,如 `0` 或 `3`)
|
||||
- 部分任务返回 `list`(错误详情列表,如 `["UndefinedColumn: ...", "InFailedSqlTransaction: ..."]`)
|
||||
|
||||
当 `sum()` 遍历到 `list` 类型时,Python 尝试执行 `int + list`,抛出 `TypeError`。
|
||||
|
||||
#### 修复方式
|
||||
|
||||
在 `run()` 方法内添加 `_safe_int()` 局部函数,统一类型转换:
|
||||
|
||||
```python
|
||||
def _safe_int(val) -> int:
|
||||
"""将 int/list/None 统一转为 int 计数。"""
|
||||
if isinstance(val, int):
|
||||
return val
|
||||
if isinstance(val, list):
|
||||
return len(val) # 错误列表 → 取长度作为错误计数
|
||||
return 0
|
||||
|
||||
flow_logger.set_counts(
|
||||
fetched=sum(_safe_int(r.get("counts", {}).get("fetched", 0)) for r in results),
|
||||
inserted=sum(_safe_int(r.get("counts", {}).get("inserted", 0)) for r in results),
|
||||
updated=sum(_safe_int(r.get("counts", {}).get("updated", 0)) for r in results),
|
||||
errors=sum(_safe_int(r.get("counts", {}).get("errors", 0)) for r in results),
|
||||
)
|
||||
```
|
||||
|
||||
#### 排查结论
|
||||
|
||||
修复后类型处理正确 ✅:
|
||||
- `int` → 直接返回 ✅
|
||||
- `list` → `len()` 转为计数 ✅(语义正确:错误详情列表的长度 = 错误数量)
|
||||
- `None` → 返回 0 ✅
|
||||
- `r.get("counts", {}).get(...)` 已做双层防御,`counts` 缺失时不会报错 ✅
|
||||
- 四个计数字段(`fetched`/`inserted`/`updated`/`errors`)均经过 `_safe_int` 处理 ✅
|
||||
|
||||
| 修复结果 | ✅ v8 中不再出现 TypeError,Flow 汇总正常完成 |
|
||||
|------|------|
|
||||
|
||||
---
|
||||
|
||||
## 未修复的遗留问题
|
||||
|
||||
### 数据质量问题 — dim_assistant_ex / dim_member_card_account_ex 非法日期
|
||||
|
||||
| 项目 | 内容 |
|
||||
|------|------|
|
||||
| 发现版本 | v8 |
|
||||
| 性质 | 上游数据质量问题,非代码 BUG |
|
||||
| 错误现象 | `ValueError: year -1 is out of range` |
|
||||
| 根因 | ODS 中某些记录的日期字段包含非法值(year=-1),Python `datetime` 无法解析 |
|
||||
| 影响 | `dim_assistant_ex` 和 `dim_member_card_account_ex` 装载失败 → 事务进入 `InFailedSqlTransaction` → 级联导致 5 个 DWS 任务失败(DWS_FINANCE_INCOME_STRUCTURE, DWS_FINANCE_DISCOUNT_DETAIL, DWS_WINBACK_INDEX, DWS_NEWCONV_INDEX, DWS_RELATION_INDEX) |
|
||||
| 建议 | 在 DWD 装载的日期类型转换中添加容错处理(捕获 ValueError,将非法日期置为 NULL 或哨兵值) |
|
||||
|
||||
---
|
||||
|
||||
## 修复文件清单
|
||||
|
||||
| 文件 | 修复的 BUG |
|
||||
|------|-----------|
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_daily_task.py` | BUG 1 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_monthly_task.py` | BUG 2 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/assistant_customer_task.py` | BUG 3, 4 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/member_consumption_task.py` | BUG 4 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/member_visit_task.py` | BUG 5, 6 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_income_task.py` | BUG 7 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_base_task.py` | BUG 8 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dws/finance_recharge_task.py` | BUG 4, 8 |
|
||||
| `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` | BUG 9, 10 |
|
||||
| `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` | BUG 11 |
|
||||
|
||||
---
|
||||
|
||||
## 执行历史
|
||||
|
||||
| 版本 | execution_id | 耗时 | 成功 | 失败 | 修复验证 |
|
||||
|------|-------------|------|------|------|---------|
|
||||
| v1 | `dbf0c29a-...` | 590.7s | 10 | 31 | — |
|
||||
| v2 | `e21e1935-...` | 150.4s | — | — | BUG 1 ✅ |
|
||||
| v3 | `abc94b2d-...` | 681.2s | 9 | 22 | BUG 2,3 ✅ |
|
||||
| v4 | `efd4f421-...` | 11m55s | 10 | 21 | BUG 4 ✅ |
|
||||
| v5 | `fe87144a-...` | 11m37s | 10 | 21 | BUG 5 部署(被 BUG 6 遮蔽) |
|
||||
| v6 | `d9443781-...` | 29m26s | 11 | 8 | BUG 5,6,7 ✅ |
|
||||
| v7 | `0929ab3a-...` | 89.3s | — | 全部 | BUG 8 部署(被 BUG 9 遮蔽) |
|
||||
| v8 | `f943bac6-...` | 1m24s | 14 | 5 | BUG 8,9,10,11 ✅ |
|
||||
@@ -1,428 +0,0 @@
|
||||
{
|
||||
"execution": {
|
||||
"id": "dbf0c29a-253a-4705-a1ef-35cd71243d48",
|
||||
"site_id": 2790685415443269,
|
||||
"task_codes": [
|
||||
"ODS_ASSISTANT_ACCOUNT",
|
||||
"ODS_ASSISTANT_LEDGER",
|
||||
"ODS_ASSISTANT_ABOLISH",
|
||||
"DWS_ASSISTANT_DAILY",
|
||||
"DWS_ASSISTANT_MONTHLY",
|
||||
"DWS_ASSISTANT_CUSTOMER",
|
||||
"DWS_ASSISTANT_SALARY",
|
||||
"DWS_ASSISTANT_FINANCE",
|
||||
"ODS_SETTLEMENT_RECORDS",
|
||||
"ODS_PAYMENT",
|
||||
"ODS_REFUND",
|
||||
"DWS_BUILD_ORDER_SUMMARY",
|
||||
"ODS_TABLE_USE",
|
||||
"ODS_TABLE_FEE_DISCOUNT",
|
||||
"ODS_TABLES",
|
||||
"ODS_MEMBER",
|
||||
"ODS_MEMBER_CARD",
|
||||
"ODS_MEMBER_BALANCE",
|
||||
"ODS_RECHARGE_SETTLE",
|
||||
"DWS_MEMBER_CONSUMPTION",
|
||||
"DWS_MEMBER_VISIT",
|
||||
"ODS_GOODS_CATEGORY",
|
||||
"ODS_STORE_GOODS",
|
||||
"ODS_STORE_GOODS_SALES",
|
||||
"ODS_TENANT_GOODS",
|
||||
"ODS_PLATFORM_COUPON",
|
||||
"ODS_GROUP_PACKAGE",
|
||||
"ODS_GROUP_BUY_REDEMPTION",
|
||||
"ODS_INVENTORY_STOCK",
|
||||
"ODS_INVENTORY_CHANGE",
|
||||
"DWS_GOODS_STOCK_DAILY",
|
||||
"DWS_GOODS_STOCK_WEEKLY",
|
||||
"DWS_GOODS_STOCK_MONTHLY",
|
||||
"DWS_FINANCE_DAILY",
|
||||
"DWS_FINANCE_RECHARGE",
|
||||
"DWS_FINANCE_INCOME_STRUCTURE",
|
||||
"DWS_FINANCE_DISCOUNT_DETAIL",
|
||||
"DWS_WINBACK_INDEX",
|
||||
"DWS_NEWCONV_INDEX",
|
||||
"DWS_RELATION_INDEX",
|
||||
"DWD_LOAD_FROM_ODS"
|
||||
],
|
||||
"status": "success",
|
||||
"started_at": "2026-02-21T15:29:20.233302+08:00",
|
||||
"finished_at": "2026-02-21T15:39:10.909320+08:00",
|
||||
"exit_code": 0,
|
||||
"duration_ms": 590676,
|
||||
"command": "C:\\NeoZQYY\\.venv\\Scripts\\python.exe -m cli.main --flow api_full --processing-mode full_window --tasks ODS_ASSISTANT_ACCOUNT,ODS_ASSISTANT_LEDGER,ODS_ASSISTANT_ABOLISH,DWS_ASSISTANT_DAILY,DWS_ASSISTANT_MONTHLY,DWS_ASSISTANT_CUSTOMER,DWS_ASSISTANT_SALARY,DWS_ASSISTANT_FINANCE,ODS_SETTLEMENT_RECORDS,ODS_PAYMENT,ODS_REFUND,DWS_BUILD_ORDER_SUMMARY,ODS_TABLE_USE,ODS_TABLE_FEE_DISCOUNT,ODS_TABLES,ODS_MEMBER,ODS_MEMBER_CARD,ODS_MEMBER_BALANCE,ODS_RECHARGE_SETTLE,DWS_MEMBER_CONSUMPTION,DWS_MEMBER_VISIT,ODS_GOODS_CATEGORY,ODS_STORE_GOODS,ODS_STORE_GOODS_SALES,ODS_TENANT_GOODS,ODS_PLATFORM_COUPON,ODS_GROUP_PACKAGE,ODS_GROUP_BUY_REDEMPTION,ODS_INVENTORY_STOCK,ODS_INVENTORY_CHANGE,DWS_GOODS_STOCK_DAILY,DWS_GOODS_STOCK_WEEKLY,DWS_GOODS_STOCK_MONTHLY,DWS_FINANCE_DAILY,DWS_FINANCE_RECHARGE,DWS_FINANCE_INCOME_STRUCTURE,DWS_FINANCE_DISCOUNT_DETAIL,DWS_WINBACK_INDEX,DWS_NEWCONV_INDEX,DWS_RELATION_INDEX,DWD_LOAD_FROM_ODS --window-start 2025-11-01 --window-end 2026-02-20 --window-split day --window-split-days 30 --force-full --store-id 2790685415443269",
|
||||
"summary": null
|
||||
},
|
||||
"error_log_length": 66800,
|
||||
"task_results_parsed": [
|
||||
{
|
||||
"task": "ODS_ASSISTANT_ACCOUNT",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:29:21",
|
||||
"end": "2026-02-21 15:29:31",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 276, 'inserted': 0, 'updated': 276, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_ASSISTANT_LEDGER",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:29:32",
|
||||
"end": "2026-02-21 15:30:08",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 2277, 'inserted': 342, 'updated': 2277, 'skipped': 0, 'errors': 0, 'deleted': 342}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_ASSISTANT_ABOLISH",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:30:08",
|
||||
"end": "2026-02-21 15:30:11",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 78, 'inserted': 0, 'updated': 78, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:30:13",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_MONTHLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_CUSTOMER",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_SALARY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_ASSISTANT_FINANCE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_SETTLEMENT_RECORDS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:14",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_PAYMENT",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_REFUND",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_BUILD_ORDER_SUMMARY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:30:15",
|
||||
"end": "2026-02-21 15:30:15",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLE_USE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:30:15",
|
||||
"end": "2026-02-21 15:35:01",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 36412, 'inserted': 0, 'updated': 36412, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLE_FEE_DISCOUNT",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:35:02",
|
||||
"end": "2026-02-21 15:36:04",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 6464, 'inserted': 0, 'updated': 6464, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TABLES",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:05",
|
||||
"end": "2026-02-21 15:36:10",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 296, 'inserted': 0, 'updated': 296, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:11",
|
||||
"end": "2026-02-21 15:36:28",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 2228, 'inserted': 0, 'updated': 2228, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER_CARD",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:36:29",
|
||||
"end": "2026-02-21 15:37:05",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 3784, 'inserted': 0, 'updated': 3784, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_MEMBER_BALANCE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:37:06",
|
||||
"end": "2026-02-21 15:38:59",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 8740, 'inserted': 0, 'updated': 8740, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "ODS_RECHARGE_SETTLE",
|
||||
"layer": "ODS",
|
||||
"status": "success",
|
||||
"start": "2026-02-21 15:39:01",
|
||||
"end": "2026-02-21 15:39:06",
|
||||
"windows": 4,
|
||||
"stats": "{'fetched': 191, 'inserted': 0, 'updated': 191, 'skipped': 0, 'errors': 0, 'deleted': 0}"
|
||||
},
|
||||
{
|
||||
"task": "DWS_MEMBER_CONSUMPTION",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:06",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_MEMBER_VISIT",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GOODS_CATEGORY",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_STORE_GOODS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_STORE_GOODS_SALES",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_TENANT_GOODS",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_PLATFORM_COUPON",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GROUP_PACKAGE",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_GROUP_BUY_REDEMPTION",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_INVENTORY_STOCK",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "ODS_INVENTORY_CHANGE",
|
||||
"layer": "ODS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_WEEKLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_GOODS_STOCK_MONTHLY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_DAILY",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_RECHARGE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:07",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_INCOME_STRUCTURE",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_FINANCE_DISCOUNT_DETAIL",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_WINBACK_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_NEWCONV_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWS_RELATION_INDEX",
|
||||
"layer": "DWS",
|
||||
"status": "failed",
|
||||
"start": "2026-02-21 15:39:08",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
},
|
||||
{
|
||||
"task": "DWD_LOAD_FROM_ODS",
|
||||
"layer": "DWD",
|
||||
"status": "failed",
|
||||
"start": "",
|
||||
"end": "2026-02-21 15:39:08",
|
||||
"windows": 0,
|
||||
"error": "错误: 当前事务被终止, 事务块结束之前的查询被忽略"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,124 +0,0 @@
|
||||
# ETL 执行结果报告
|
||||
|
||||
> 生成时间:2026-02-21 19:18:58
|
||||
> execution_id:dbf0c29a-253a-4705-a1ef-35cd71243d48
|
||||
> run_uuid:4ba9d2d365ee4a858f1c4104b1942dc2
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21T15:29:20.233302+08:00 |
|
||||
| 结束时间 | 2026-02-21T15:39:10.909320+08:00 |
|
||||
| 总时长 | 590.7s (9.8m) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 41 |
|
||||
| 成功 | 10 |
|
||||
| 失败 | 31 |
|
||||
|
||||
---
|
||||
|
||||
## 任务级结果
|
||||
|
||||
| # | 任务 | 层 | 状态 | 开始 | 结束 | 耗时 | 窗口数 | 备注 |
|
||||
|---|------|-----|------|------|------|------|--------|------|
|
||||
| 1 | ODS_ASSISTANT_ACCOUNT | ODS | ✅ success | 15:29:21 | 15:29:31 | 10.0s | 4 | {'fetched': 276, 'inserted': 0, 'updated': 276, 'skipped'... |
|
||||
| 2 | ODS_ASSISTANT_LEDGER | ODS | ✅ success | 15:29:32 | 15:30:08 | 36.0s | 4 | {'fetched': 2277, 'inserted': 342, 'updated': 2277, 'skip... |
|
||||
| 3 | ODS_ASSISTANT_ABOLISH | ODS | ✅ success | 15:30:08 | 15:30:11 | 3.0s | 4 | {'fetched': 78, 'inserted': 0, 'updated': 78, 'skipped': ... |
|
||||
| 4 | DWS_ASSISTANT_DAILY | DWS | ❌ failed | 15:30:13 | 15:30:14 | 1.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | DWS_ASSISTANT_MONTHLY | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | DWS_ASSISTANT_CUSTOMER | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | DWS_ASSISTANT_SALARY | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | DWS_ASSISTANT_FINANCE | DWS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 9 | ODS_SETTLEMENT_RECORDS | ODS | ❌ failed | | 15:30:14 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 10 | ODS_PAYMENT | ODS | ❌ failed | | 15:30:15 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 11 | ODS_REFUND | ODS | ❌ failed | | 15:30:15 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | DWS_BUILD_ORDER_SUMMARY | DWS | ❌ failed | 15:30:15 | 15:30:15 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | ODS_TABLE_USE | ODS | ✅ success | 15:30:15 | 15:35:01 | 4.8m | 4 | {'fetched': 36412, 'inserted': 0, 'updated': 36412, 'skip... |
|
||||
| 14 | ODS_TABLE_FEE_DISCOUNT | ODS | ✅ success | 15:35:02 | 15:36:04 | 1.0m | 4 | {'fetched': 6464, 'inserted': 0, 'updated': 6464, 'skippe... |
|
||||
| 15 | ODS_TABLES | ODS | ✅ success | 15:36:05 | 15:36:10 | 5.0s | 4 | {'fetched': 296, 'inserted': 0, 'updated': 296, 'skipped'... |
|
||||
| 16 | ODS_MEMBER | ODS | ✅ success | 15:36:11 | 15:36:28 | 17.0s | 4 | {'fetched': 2228, 'inserted': 0, 'updated': 2228, 'skippe... |
|
||||
| 17 | ODS_MEMBER_CARD | ODS | ✅ success | 15:36:29 | 15:37:05 | 36.0s | 4 | {'fetched': 3784, 'inserted': 0, 'updated': 3784, 'skippe... |
|
||||
| 18 | ODS_MEMBER_BALANCE | ODS | ✅ success | 15:37:06 | 15:38:59 | 1.9m | 4 | {'fetched': 8740, 'inserted': 0, 'updated': 8740, 'skippe... |
|
||||
| 19 | ODS_RECHARGE_SETTLE | ODS | ✅ success | 15:39:01 | 15:39:06 | 5.0s | 4 | {'fetched': 191, 'inserted': 0, 'updated': 191, 'skipped'... |
|
||||
| 20 | DWS_MEMBER_CONSUMPTION | DWS | ❌ failed | 15:39:06 | 15:39:07 | 1.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWS_MEMBER_VISIT | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 22 | ODS_GOODS_CATEGORY | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 23 | ODS_STORE_GOODS | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 24 | ODS_STORE_GOODS_SALES | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 25 | ODS_TENANT_GOODS | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 26 | ODS_PLATFORM_COUPON | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 27 | ODS_GROUP_PACKAGE | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 28 | ODS_GROUP_BUY_REDEMPTION | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 29 | ODS_INVENTORY_STOCK | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 30 | ODS_INVENTORY_CHANGE | ODS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 31 | DWS_GOODS_STOCK_DAILY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 32 | DWS_GOODS_STOCK_WEEKLY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 33 | DWS_GOODS_STOCK_MONTHLY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 34 | DWS_FINANCE_DAILY | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 35 | DWS_FINANCE_RECHARGE | DWS | ❌ failed | | 15:39:07 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 36 | DWS_FINANCE_INCOME_STRUCTURE | DWS | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 37 | DWS_FINANCE_DISCOUNT_DETAIL | DWS | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 38 | DWS_WINBACK_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 39 | DWS_NEWCONV_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 40 | DWS_RELATION_INDEX | DWS | ❌ failed | 15:39:08 | 15:39:08 | 0.0s | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 41 | DWD_LOAD_FROM_ODS | DWD | ❌ failed | | 15:39:08 | — | — | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
---
|
||||
|
||||
## 失败任务分析
|
||||
|
||||
### 根因:DWS_ASSISTANT_DAILY
|
||||
|
||||
错误:`错误: 当前事务被终止, 事务块结束之前的查询被忽略`
|
||||
|
||||
原因:`_extract_trash_records` SQL 引用了 `dwd_assistant_trash_event` 中不存在的字段 `assistant_service_id`。
|
||||
|
||||
### 级联失败
|
||||
|
||||
- DWS_ASSISTANT_MONTHLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_CUSTOMER:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_SALARY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_ASSISTANT_FINANCE:InFailedSqlTransaction(事务污染)
|
||||
- ODS_SETTLEMENT_RECORDS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_PAYMENT:InFailedSqlTransaction(事务污染)
|
||||
- ODS_REFUND:InFailedSqlTransaction(事务污染)
|
||||
- DWS_BUILD_ORDER_SUMMARY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_MEMBER_CONSUMPTION:InFailedSqlTransaction(事务污染)
|
||||
- DWS_MEMBER_VISIT:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GOODS_CATEGORY:InFailedSqlTransaction(事务污染)
|
||||
- ODS_STORE_GOODS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_STORE_GOODS_SALES:InFailedSqlTransaction(事务污染)
|
||||
- ODS_TENANT_GOODS:InFailedSqlTransaction(事务污染)
|
||||
- ODS_PLATFORM_COUPON:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GROUP_PACKAGE:InFailedSqlTransaction(事务污染)
|
||||
- ODS_GROUP_BUY_REDEMPTION:InFailedSqlTransaction(事务污染)
|
||||
- ODS_INVENTORY_STOCK:InFailedSqlTransaction(事务污染)
|
||||
- ODS_INVENTORY_CHANGE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_DAILY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_WEEKLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_GOODS_STOCK_MONTHLY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_DAILY:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_RECHARGE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_INCOME_STRUCTURE:InFailedSqlTransaction(事务污染)
|
||||
- DWS_FINANCE_DISCOUNT_DETAIL:InFailedSqlTransaction(事务污染)
|
||||
- DWS_WINBACK_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWS_NEWCONV_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWS_RELATION_INDEX:InFailedSqlTransaction(事务污染)
|
||||
- DWD_LOAD_FROM_ODS:InFailedSqlTransaction(事务污染)
|
||||
|
||||
### 修复状态
|
||||
|
||||
代码已修复(4 处改动),待下次执行验证。
|
||||
详见:`export/SYSTEM/LOGS/2026-02-21__dws_assistant_daily_bug_fix.md`
|
||||
|
||||
---
|
||||
|
||||
## 下一步
|
||||
|
||||
1. 重新提交包含 9 个失败任务的执行,验证修复
|
||||
2. 运行 ETL Data Consistency Check
|
||||
3. 运行 /audit 审计
|
||||
@@ -1,136 +0,0 @@
|
||||
# ETL 回归执行结果报告(第二次)
|
||||
|
||||
> 生成时间:2026-02-21 19:33:28
|
||||
> execution_id:e21e1935-5abf-434f-9984-69c492402db7
|
||||
> 目的:验证 DWS_ASSISTANT_DAILY 修复 + 补跑上次失败的 31 个任务
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21T19:27:47.937140+08:00 |
|
||||
| 结束时间 | 2026-02-21T19:30:18.341157+08:00 |
|
||||
| 总时长 | 150.4s (2.5m) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 31 |
|
||||
|
||||
## 执行日志(error_log 末尾 100 行)
|
||||
|
||||
```
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 403, in _run_utility_task
|
||||
result = task.execute(None)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 145, in execute
|
||||
tenant_id = self._get_tenant_id()
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 688, in _get_tenant_id
|
||||
rows = self.db.query(sql)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] ERROR | etl_billiards | 任务 DWS_RELATION_INDEX 失败: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 94, in run_tasks
|
||||
task_result = self.run_single_task(
|
||||
task_code, run_uuid, store_id, data_source=data_source,
|
||||
)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 150, in run_single_task
|
||||
return self._run_utility_task(task_code_upper, store_id)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 403, in _run_utility_task
|
||||
result = task.execute(None)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 145, in execute
|
||||
tenant_id = self._get_tenant_id()
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\tasks\dws\index\relation_index_task.py", line 688, in _get_tenant_id
|
||||
rows = self.db.query(sql)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] ERROR | etl_billiards | 任务 DWD_LOAD_FROM_ODS 失败: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
Traceback (most recent call last):
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 94, in run_tasks
|
||||
task_result = self.run_single_task(
|
||||
task_code, run_uuid, store_id, data_source=data_source,
|
||||
)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 152, in run_single_task
|
||||
task_cfg = self._load_task_config(task_code, store_id)
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\orchestration\task_executor.py", line 429, in _load_task_config
|
||||
rows = self.db_ops.query(sql, (store_id, task_code))
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\operations.py", line 99, in query
|
||||
return self._connection.query(sql, args)
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\apps\etl\connectors\feiqiu\database\connection.py", line 50, in query
|
||||
c.execute(sql, args)
|
||||
~~~~~~~~~^^^^^^^^^^^
|
||||
File "C:\NeoZQYY\.venv\Lib\site-packages\psycopg2\extras.py", line 236, in execute
|
||||
return super().execute(query, vars)
|
||||
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
psycopg2.errors.InFailedSqlTransaction: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
[2026-02-21 19:30:15] INFO | etl_billiards | 所有任务执行完成
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | 一致性检查报告已生成: C:\NeoZQYY\export\ETL-Connectors\feiqiu\REPORTS\consistency_report_20260221_193018.md
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | 计时报告已生成
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards |
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ 任务执行总结 ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 任务代码: FLOW_API_FULL ║
|
||||
║ 执行状态: 成功 ║
|
||||
║ 执行时间: 2026-02-21 19:27:49 ~ 19:30:18 (2分29秒) ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 数据统计 ║
|
||||
║ - 获取记录: 0 ║
|
||||
║ - 新增记录: 0 ║
|
||||
║ - 更新记录: 0 ║
|
||||
║ - 跳过记录: 0 ║
|
||||
║ - 错误记录: 0 ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards |
|
||||
╔══════════════════════════════════════════════════════════════╗
|
||||
║ 任务执行总结 ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 任务代码: FLOW_API_FULL ║
|
||||
║ 执行状态: 成功 ║
|
||||
║ 执行时间: 2026-02-21 19:27:49 ~ 19:30:18 (2分29秒) ║
|
||||
╠══════════════════════════════════════════════════════════════╣
|
||||
║ 数据统计 ║
|
||||
║ - 获取记录: 0 ║
|
||||
║ - 新增记录: 0 ║
|
||||
║ - 更新记录: 0 ║
|
||||
║ - 跳过记录: 0 ║
|
||||
║ - 错误记录: 0 ║
|
||||
╚══════════════════════════════════════════════════════════════╝
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | Flow 执行完成: SUCCESS
|
||||
[2026-02-21 19:30:18] INFO | etl_billiards | ETL运行完成
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 与第一次执行的对比
|
||||
|
||||
| 项目 | 第一次 | 第二次(本次) |
|
||||
|------|--------|---------------|
|
||||
| 任务数 | 41 | 31 |
|
||||
| 状态 | success (exit_code=0) | success (exit_code=0) |
|
||||
| 耗时 | 590.7s (9.8m) | 150.4s (2.5m) |
|
||||
| 成功 | 10/41 | 待分析 |
|
||||
| 失败 | 31/41 | 待分析 |
|
||||
| 根因 | DWS_ASSISTANT_DAILY SQL 字段错误 | — |
|
||||
@@ -1,85 +0,0 @@
|
||||
# ETL 回归执行结果报告(第三次)
|
||||
|
||||
> 生成时间:2026-02-21 19:54:12
|
||||
> execution_id:abc94b2d-615f-42ea-83cc-ce687524a6ea
|
||||
> 目的:验证 BUG 2(DWS_ASSISTANT_MONTHLY UniqueViolation)和 BUG 3(DWS_ASSISTANT_CUSTOMER UndefinedColumn)修复
|
||||
|
||||
---
|
||||
|
||||
## 执行概览
|
||||
|
||||
| 项目 | 值 |
|
||||
|------|-----|
|
||||
| 状态 | success |
|
||||
| 开始时间 | 2026-02-21 19:41:02 |
|
||||
| 结束时间 | 2026-02-21 19:52:22 |
|
||||
| 总时长 | 681.2s (11m19s) |
|
||||
| 退出码 | 0 |
|
||||
| 任务总数 | 31 |
|
||||
| 成功 | 9 |
|
||||
| 失败 | 22 |
|
||||
| 未知 | 0 |
|
||||
| 数据统计 | 获取 52,982 / 新增 13,296 / 更新 52,982 |
|
||||
|
||||
## BUG 修复验证
|
||||
|
||||
| BUG | 任务 | 第二次结果 | 第三次结果 | 验证 |
|
||||
|-----|------|-----------|-----------|------|
|
||||
| BUG 1 | DWS_ASSISTANT_DAILY | ✅ 已修复 | ✅ 成功 | ✅ 持续通过 |
|
||||
| BUG 2 | DWS_ASSISTANT_MONTHLY | ❌ UniqueViolation | ✅ 成功 | ✅ 修复验证通过 |
|
||||
| BUG 3 | DWS_ASSISTANT_CUSTOMER | ❌ UndefinedColumn | ✅ 成功 | ✅ 修复验证通过 |
|
||||
|
||||
## 逐任务结果
|
||||
|
||||
| # | 任务 | 状态 | 统计/错误 |
|
||||
|---|------|------|----------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | ✅ 成功 | {'counts': {'fetched': 367, 'inserted': 367, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | ✅ 成功 | {'counts': {'fetched': 25, 'inserted': 25, 'updated': 0, 'skipped': 0, 'errors': |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | ✅ 成功 | {'counts': {'fetched': 486, 'inserted': 486, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 4 | DWS_ASSISTANT_SALARY | ✅ 成功 | {'counts': {'fetched': 0, 'inserted': 0, 'updated': 0, 'skipped': 0, 'errors': 0 |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | ✅ 成功 | {'counts': {'fetched': 367, 'inserted': 367, 'updated': 0, 'skipped': 0, 'errors |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | ✅ 成功 | fetched=10366, updated=10366 |
|
||||
| 7 | ODS_PAYMENT | ✅ 成功 | fetched=42500, updated=42500 |
|
||||
| 8 | ODS_REFUND | ✅ 成功 | fetched=116, updated=116 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | ✅ 成功 | {'fetched': 0, 'inserted': 13296, 'updated': 0, 'skipped': 0, 'errors': 0} |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION | ❌ 字段错误 | UndefinedColumn: dim_member.site_id 不存在(同 BUG 3 同类) |
|
||||
| 11 | DWS_MEMBER_VISIT | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | ODS_GOODS_CATEGORY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | ODS_STORE_GOODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 14 | ODS_STORE_GOODS_SALES | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 15 | ODS_TENANT_GOODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 16 | ODS_PLATFORM_COUPON | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 17 | ODS_GROUP_PACKAGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 18 | ODS_GROUP_BUY_REDEMPTION | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 19 | ODS_INVENTORY_STOCK | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 20 | ODS_INVENTORY_CHANGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWS_GOODS_STOCK_DAILY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 22 | DWS_GOODS_STOCK_WEEKLY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 23 | DWS_GOODS_STOCK_MONTHLY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 24 | DWS_FINANCE_DAILY | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 25 | DWS_FINANCE_RECHARGE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 26 | DWS_FINANCE_INCOME_STRUCTURE | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 27 | DWS_FINANCE_DISCOUNT_DETAIL | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 28 | DWS_WINBACK_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 29 | DWS_NEWCONV_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 30 | DWS_RELATION_INDEX | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 31 | DWD_LOAD_FROM_ODS | ❌ 失败 | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析
|
||||
|
||||
本次新发现的根因错误:
|
||||
|
||||
- 任务:`DWS_MEMBER_CONSUMPTION`
|
||||
- 错误:错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- 影响:后续所有任务因 `InFailedSqlTransaction` 级联失败
|
||||
|
||||
|
||||
## 三次执行对比
|
||||
|
||||
| 项目 | 第一次 | 第二次 | 第三次(本次) |
|
||||
|------|--------|--------|---------------|
|
||||
| 任务数 | 41 | 31 | 31 |
|
||||
| 耗时 | 590.7s | 150.4s | 681.2s |
|
||||
| 成功 | 10/41 | 3/31 | 6/31 |
|
||||
| 失败 | 31/41 | 28/31 | 22/31 |
|
||||
| 根因 | DWS_ASSISTANT_DAILY SQL 字段 | DWS_ASSISTANT_MONTHLY UK + DWS_ASSISTANT_CUSTOMER site_id | DWS_MEMBER_CONSUMPTION site_id |
|
||||
@@ -1,70 +0,0 @@
|
||||
# 第四次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `efd4f421-ee10-4244-833f-7b2d68c3c05b`
|
||||
- 时间: 2026-02-21 19:57:02 ~ 20:08:57
|
||||
- 耗时: 11 分 55 秒 (715s)
|
||||
- 整体状态: success (exit_code=0)
|
||||
- 任务总数: 31
|
||||
|
||||
## 成功任务 (10 个)
|
||||
|
||||
| # | 任务 | 耗时 | 统计 |
|
||||
|---|------|------|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY | ~2m28s | fetched=367, inserted=367, deleted=367 |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY | ~12s | fetched=25, inserted=25, deleted=25 |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER | ~1m22s | fetched=486, inserted=486 |
|
||||
| 4 | DWS_ASSISTANT_SALARY | <1s | 非工资结算期,跳过 |
|
||||
| 5 | DWS_ASSISTANT_FINANCE | ~1m10s | fetched=367, inserted=367, deleted=367 |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS | ~1m46s | fetched=10366, updated=10366 |
|
||||
| 7 | ODS_PAYMENT | ~4m0s | fetched=42500, updated=42500 |
|
||||
| 8 | ODS_REFUND | ~3s | fetched=116, updated=116 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY | ~1s | inserted=13296 |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION | ~43s | fetched=198, inserted=198 |
|
||||
|
||||
## BUG 4 修复验证
|
||||
|
||||
- DWS_MEMBER_CONSUMPTION ✅ 不再报 UndefinedColumn site_id
|
||||
- DWS_MEMBER_VISIT ❌ 新错误(BUG 5)
|
||||
- DWS_FINANCE_RECHARGE ❌ 级联失败(未能独立验证)
|
||||
|
||||
## 新发现 BUG 5
|
||||
|
||||
- 任务: `DWS_MEMBER_VISIT`
|
||||
- 错误: `UndefinedColumn: 字段 "birthday" 不存在`
|
||||
- 位置: `member_visit_task.py` → `_extract_member_info()` line ~312
|
||||
- 根因: SQL 查询 `dwd.dim_member` 时引用了 `birthday` 字段,但该表没有此字段
|
||||
- DWS 表 `dws_member_visit_detail` 设计了 `member_birthday DATE` 列,但上游 dim_member 未提供此数据
|
||||
- 级联影响: 后续 20 个任务全部 InFailedSqlTransaction
|
||||
|
||||
## 失败任务 (21 个)
|
||||
|
||||
| 类型 | 任务 | 错误 |
|
||||
|------|------|------|
|
||||
| 🔴 根因 | DWS_MEMBER_VISIT | UndefinedColumn: birthday |
|
||||
| 级联 | ODS_GOODS_CATEGORY | InFailedSqlTransaction |
|
||||
| 级联 | ODS_STORE_GOODS | InFailedSqlTransaction |
|
||||
| 级联 | ODS_STORE_GOODS_SALES | InFailedSqlTransaction |
|
||||
| 级联 | ODS_TENANT_GOODS | InFailedSqlTransaction |
|
||||
| 级联 | ODS_PLATFORM_COUPON | InFailedSqlTransaction |
|
||||
| 级联 | ODS_GROUP_PACKAGE | InFailedSqlTransaction |
|
||||
| 级联 | ODS_GROUP_BUY_REDEMPTION | InFailedSqlTransaction |
|
||||
| 级联 | ODS_INVENTORY_STOCK | InFailedSqlTransaction |
|
||||
| 级联 | ODS_INVENTORY_CHANGE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_DAILY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_WEEKLY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_GOODS_STOCK_MONTHLY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_DAILY | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_RECHARGE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_INCOME_STRUCTURE | InFailedSqlTransaction |
|
||||
| 级联 | DWS_FINANCE_DISCOUNT_DETAIL | InFailedSqlTransaction |
|
||||
| 级联 | DWS_WINBACK_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWS_NEWCONV_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWS_RELATION_INDEX | InFailedSqlTransaction |
|
||||
| 级联 | DWD_LOAD_FROM_ODS | InFailedSqlTransaction |
|
||||
|
||||
## BUG 5 修复
|
||||
|
||||
- 文件: `member_visit_task.py`
|
||||
- 改动 1: `_extract_member_info` SQL 移除 `birthday` 字段
|
||||
- 改动 2: transform 中 `member_birthday` 改为 `None`
|
||||
- 已添加 CHANGE 注释
|
||||
@@ -1,69 +0,0 @@
|
||||
# 第五次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `fe87144a-687d-4ce0-9b79-6bd0186b2be3`
|
||||
- 执行时间: 2026-02-21 20:19:52 ~ 20:31:29(约 11m37s)
|
||||
- exit_code: 0
|
||||
- 总任务数: 31
|
||||
|
||||
## 成功任务(10 个)
|
||||
|
||||
| # | 任务 |
|
||||
|---|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER |
|
||||
| 4 | DWS_ASSISTANT_SALARY |
|
||||
| 5 | DWS_ASSISTANT_FINANCE |
|
||||
| 6 | ODS_SETTLEMENT_RECORDS ODS 任务完成 |
|
||||
| 7 | ODS_PAYMENT ODS 任务完成 |
|
||||
| 8 | ODS_REFUND ODS 任务完成 |
|
||||
| 9 | DWS_BUILD_ORDER_SUMMARY |
|
||||
| 10 | DWS_MEMBER_CONSUMPTION |
|
||||
|
||||
## 失败任务(21 个)
|
||||
|
||||
| # | 任务 | 错误类型 |
|
||||
|---|------|----------|
|
||||
| 1 | DWS_MEMBER_VISIT | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 2 | ODS_GOODS_CATEGORY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 3 | ODS_STORE_GOODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 4 | ODS_STORE_GOODS_SALES | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | ODS_TENANT_GOODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | ODS_PLATFORM_COUPON | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | ODS_GROUP_PACKAGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | ODS_GROUP_BUY_REDEMPTION | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 9 | ODS_INVENTORY_STOCK | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 10 | ODS_INVENTORY_CHANGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 11 | DWS_GOODS_STOCK_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 12 | DWS_GOODS_STOCK_WEEKLY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 13 | DWS_GOODS_STOCK_MONTHLY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 14 | DWS_FINANCE_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 15 | DWS_FINANCE_RECHARGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 16 | DWS_FINANCE_INCOME_STRUCTURE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 17 | DWS_FINANCE_DISCOUNT_DETAIL | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 18 | DWS_WINBACK_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 19 | DWS_NEWCONV_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 20 | DWS_RELATION_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 21 | DWD_LOAD_FROM_ODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析
|
||||
|
||||
BUG 6: `DWS_MEMBER_VISIT` → `_extract_table_info()` 方法中 SQL 引用了 `dwd.dim_table.site_table_id`,
|
||||
但该表的主键字段实际为 `table_id`(参考 `db/etl_feiqiu/schemas/dwd.sql`)。
|
||||
|
||||
错误发生后,psycopg2 连接进入 InFailedSqlTransaction 状态,导致后续所有任务级联失败。
|
||||
|
||||
## 修复措施
|
||||
|
||||
1. `member_visit_task.py` → `_extract_table_info()`:
|
||||
- `site_table_id AS table_id` → `table_id AS table_id`
|
||||
- `site_table_name AS table_name` → `table_name AS table_name`
|
||||
|
||||
2. `finance_income_task.py` → `_extract_income_by_area()`:
|
||||
- JOIN 条件 `dt.site_table_id = tfl.site_table_id` → `dt.table_id = tfl.site_table_id`
|
||||
- JOIN 条件 `dt.site_table_id = asl.site_table_id` → `dt.table_id = asl.site_table_id`
|
||||
|
||||
## BUG 5 验证
|
||||
|
||||
BUG 5(birthday 字段)的修复已部署,但被 BUG 6 遮蔽,无法在本次执行中验证。
|
||||
需要第六次执行来同时验证 BUG 5 + BUG 6 + BUG 7。
|
||||
@@ -1,59 +0,0 @@
|
||||
# 第六次 ETL 执行结果报告
|
||||
|
||||
- execution_id: `d9443781-e4ac-4df6-9f87-11c45d72e5ba`
|
||||
- 执行时间: 2026-02-21 20:45:18 ~ 21:14:45(29 分 26 秒)
|
||||
- exit_code: 0
|
||||
- status: success
|
||||
- 总任务数: 31
|
||||
- 数据统计: 获取 171,961 / 新增 13,662 / 更新 171,595 / 跳过 0 / 错误 0
|
||||
|
||||
## 成功任务(11 个)
|
||||
|
||||
| # | 任务 |
|
||||
|---|------|
|
||||
| 1 | DWS_ASSISTANT_DAILY: |
|
||||
| 2 | DWS_ASSISTANT_MONTHLY: |
|
||||
| 3 | DWS_ASSISTANT_CUSTOMER: |
|
||||
| 4 | DWS_ASSISTANT_SALARY: |
|
||||
| 5 | DWS_ASSISTANT_FINANCE: |
|
||||
| 6 | {'fetched': |
|
||||
| 7 | DWS_MEMBER_CONSUMPTION: |
|
||||
| 8 | DWS_MEMBER_VISIT: |
|
||||
| 9 | DWS_GOODS_STOCK_DAILY: |
|
||||
| 10 | DWS_GOODS_STOCK_WEEKLY: |
|
||||
| 11 | DWS_GOODS_STOCK_MONTHLY: |
|
||||
|
||||
## 失败任务(8 个)
|
||||
|
||||
| # | 任务 | 错误类型 |
|
||||
|---|------|----------|
|
||||
| 1 | DWS_FINANCE_DAILY | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 2 | DWS_FINANCE_RECHARGE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 3 | DWS_FINANCE_INCOME_STRUCTURE | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 4 | DWS_FINANCE_DISCOUNT_DETAIL | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 5 | DWS_WINBACK_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 6 | DWS_NEWCONV_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 7 | DWS_RELATION_INDEX | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
| 8 | DWD_LOAD_FROM_ODS | 错误: 当前事务被终止, 事务块结束之前的查询被忽略 |
|
||||
|
||||
## 根因分析(8 个非级联失败)
|
||||
|
||||
- `DWS_FINANCE_DAILY`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_RECHARGE`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_INCOME_STRUCTURE`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_FINANCE_DISCOUNT_DETAIL`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_WINBACK_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_NEWCONV_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWS_RELATION_INDEX`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
- `DWD_LOAD_FROM_ODS`: 错误: 当前事务被终止, 事务块结束之前的查询被忽略
|
||||
|
||||
## 与前次对比
|
||||
|
||||
| 轮次 | 成功 | 失败 | 耗时 | 修复的 BUG |
|
||||
|------|------|------|------|-----------|
|
||||
| v1 | 10 | 31 | 9m51s | — |
|
||||
| v2 | — | — | 2m30s | BUG 1 |
|
||||
| v3 | 9 | 22 | 11m21s | BUG 2+3 |
|
||||
| v4 | 10 | 21 | 11m55s | BUG 4 |
|
||||
| v5 | 10 | 21 | 11m37s | BUG 5 |
|
||||
| v6 | 11 | 8 | 29m26s | BUG 5+6+7 |
|
||||
@@ -1,109 +0,0 @@
|
||||
# ETL 第八次执行报告 (v8)
|
||||
|
||||
- execution_id: `f943bac6-23be-45c5-8b8c-a864e85a1916`
|
||||
- 时间: 2026-02-21 21:33:37 ~ 21:35:01 (1分24秒)
|
||||
- 整体状态: success, exit_code=0
|
||||
|
||||
## 本次修复验证
|
||||
|
||||
| BUG | 修复内容 | 验证结果 |
|
||||
|-----|---------|---------|
|
||||
| BUG 8 | `finance_base_task.py` + `finance_recharge_task.py`: pay_money→pay_amount, gift_money→point_amount | ✅ DWS_FINANCE_DAILY + DWS_FINANCE_RECHARGE 均完成 |
|
||||
| BUG 9 | `dwd_load_task.py`: 添加 `_pick_snapshot_order_column` 方法 | ✅ 所有 dim 表 SCD2 装载成功 |
|
||||
| BUG 10 | `dwd_load_task.py`: FACT_MAPPINGS 驼峰字段名→小写 | ✅ dwd_goods_stock_summary(716条) + dwd_goods_stock_movement(14306条) 装载成功 |
|
||||
| BUG 11 | `flow_runner.py`: sum() 类型安全处理 | ✅ 不再出现 TypeError |
|
||||
|
||||
## DWD_LOAD_FROM_ODS 详情
|
||||
|
||||
### 维度表 (SCD2) — 全部成功
|
||||
| 表 | processed | inserted | updated |
|
||||
|----|-----------|----------|---------|
|
||||
| dim_site | 1 | 0 | 1 |
|
||||
| dim_site_ex | 1 | 0 | 1 |
|
||||
| dim_table | 74 | 0 | 74 |
|
||||
| dim_table_ex | 74 | 0 | 74 |
|
||||
| dim_assistant | 69 | 0 | 69 |
|
||||
| dim_member | 557 | 0 | 557 |
|
||||
| dim_member_ex | 557 | 0 | 557 |
|
||||
| dim_member_card_account | 946 | 0 | 946 |
|
||||
| dim_tenant_goods | 174 | 1 | 173 |
|
||||
| dim_tenant_goods_ex | 174 | 1 | 173 |
|
||||
| dim_store_goods | 173 | 1 | 172 |
|
||||
| dim_store_goods_ex | 173 | 1 | 172 |
|
||||
| dim_goods_category | 26 | 0 | 26 |
|
||||
| dim_groupbuy_package | 34 | 0 | 34 |
|
||||
| dim_groupbuy_package_ex | 34 | 0 | 34 |
|
||||
|
||||
### 事实表 (INCREMENT) — 全部成功
|
||||
| 表 | processed | inserted | updated |
|
||||
|----|-----------|----------|---------|
|
||||
| dwd_settlement_head | 10366 | 0 | 10366 |
|
||||
| dwd_settlement_head_ex | 10366 | 0 | 10366 |
|
||||
| dwd_table_fee_log | 9103 | 0 | 9103 |
|
||||
| dwd_table_fee_log_ex | 9103 | 0 | 9103 |
|
||||
| dwd_table_fee_adjust | 1616 | 0 | 1616 |
|
||||
| dwd_table_fee_adjust_ex | 1616 | 0 | 1616 |
|
||||
| dwd_assistant_service_log | 2619 | 0 | 2619 |
|
||||
| dwd_assistant_service_log_ex | 2619 | 0 | 2619 |
|
||||
| dwd_assistant_trash_event | 78 | 0 | 78 |
|
||||
| dwd_assistant_trash_event_ex | 78 | 0 | 78 |
|
||||
| dwd_member_balance_change | 2185 | 0 | 2185 |
|
||||
| dwd_member_balance_change_ex | 2185 | 0 | 2185 |
|
||||
| dwd_groupbuy_redemption | 7267 | 0 | 7267 |
|
||||
| dwd_groupbuy_redemption_ex | 7267 | 0 | 7267 |
|
||||
| dwd_platform_coupon_redemption | 18311 | 0 | 18311 |
|
||||
| dwd_platform_coupon_redemption_ex | 18311 | 0 | 18311 |
|
||||
| dwd_recharge_order | 191 | 0 | 191 |
|
||||
| dwd_recharge_order_ex | 191 | 0 | 191 |
|
||||
| dwd_payment | 10625 | 0 | 10625 |
|
||||
| dwd_refund | 29 | 0 | 29 |
|
||||
| dwd_refund_ex | 29 | 0 | 29 |
|
||||
| dwd_goods_stock_summary | 716 | 716 | 0 |
|
||||
| dwd_goods_stock_movement | 14306 | 14306 | 0 |
|
||||
|
||||
### DWD 装载错误 (2个,数据质量问题,非代码 BUG)
|
||||
| 表 | 错误 |
|
||||
|----|------|
|
||||
| dim_assistant_ex | year -1 is out of range |
|
||||
| dim_member_card_account_ex | year -1 is out of range |
|
||||
|
||||
## DWS 任务状态
|
||||
|
||||
| 任务 | 状态 | 备注 |
|
||||
|------|------|------|
|
||||
| ODS_FETCH | ✅ 完成 | |
|
||||
| DWD_LOAD_FROM_ODS | ✅ 完成 | 39表成功,2表数据质量错误 |
|
||||
| DWS_ASSISTANT_DAILY | ✅ 完成 | |
|
||||
| DWS_ASSISTANT_MONTHLY | ✅ 完成 | 删除9行,插入9行 |
|
||||
| DWS_ASSISTANT_CUSTOMER | ✅ 完成 | 删除285行,插入285行 |
|
||||
| DWS_ASSISTANT_SALARY | ✅ 完成 | |
|
||||
| DWS_ASSISTANT_FINANCE | ✅ 完成 | |
|
||||
| DWS_MEMBER_CONSUMPTION | ✅ 完成 | 删除198行,插入198行 |
|
||||
| DWS_MEMBER_VISIT | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_DAILY | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_WEEKLY | ✅ 完成 | |
|
||||
| DWS_GOODS_STOCK_MONTHLY | ✅ 完成 | |
|
||||
| DWS_FINANCE_DAILY | ✅ 完成 | |
|
||||
| DWS_FINANCE_RECHARGE | ✅ 完成 | |
|
||||
| DWS_FINANCE_INCOME_STRUCTURE | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_FINANCE_DISCOUNT_DETAIL | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_WINBACK_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_NEWCONV_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
| DWS_RELATION_INDEX | ❌ 级联失败 | InFailedSqlTransaction |
|
||||
|
||||
## 总结
|
||||
|
||||
- 14/19 任务成功完成
|
||||
- 5/19 任务因 InFailedSqlTransaction 级联失败
|
||||
- 级联失败根因: `dim_assistant_ex` 和 `dim_member_card_account_ex` 中存在非法日期值 (year=-1),导致事务进入失败状态
|
||||
- 这是数据质量问题,不是代码 BUG — 需要在 DWD 装载时对日期字段做容错处理
|
||||
|
||||
## 与 v6(上次最好成绩)对比
|
||||
|
||||
| 指标 | v6 | v8 |
|
||||
|------|----|----|
|
||||
| 耗时 | 29m26s | 1m24s |
|
||||
| 成功任务 | 11/19 | 14/19 |
|
||||
| 失败任务 | 8/19 | 5/19 |
|
||||
| DWD 装载 | 部分 dim 失败 | 39/41 表成功 |
|
||||
| 新增成功 | — | DWS_FINANCE_DAILY, DWS_FINANCE_RECHARGE, DWS_GOODS_STOCK_* |
|
||||
Reference in New Issue
Block a user