在前后端开发联调前的提交20260223

2026-02-23 23:02:20 +08:00
parent 254ccb1e77
commit fafc95e64c
1142 changed files with 10366960 additions and 36957 deletions
--- a/.kiro/specs/etl-staff-dimension/design.md
+++ b/.kiro/specs/etl-staff-dimension/design.md
@@ -0,0 +1,354 @@
+# 设计文档：ETL 员工维度表（staff_info）
+
+## 概述
+
+为飞球 ETL 连接器新增员工维度表，从 `SearchSystemStaffInfo` API 抓取球房全体员工数据（店长、主管、教练、收银员、助教管理员等），经 ODS 落地后清洗装载至 DWD 层。员工表与现有助教表（`assistant_accounts_master`）是完全独立的实体。
+
+## API 响应结构
+
+```json
+{
+  "data": {
+    "total": 15,
+    "staffProfiles": [
+      {
+        "id": 3020236636900101,
+        "cashierPointId": 2790685415443270,
+        "cashierPointName": "默认",
+        "job_num": "",
+        "staff_name": "葛芃",
+        "mobile": "13811638071",
+        "auth_code": "",
+        "avatar": "",
+        "create_time": "2025-12-24 00:03:37",
+        "entry_time": "2025-12-23 08:00:00",
+        "is_delete": 0,
+        "leave_status": 0,
+        "resign_time": "2225-12-24 00:03:37",
+        "site_id": 2790685415443269,
+        "staff_identity": 2,
+        "status": 1,
+        "system_role_id": 4,
+        "system_user_id": 3020236636293893,
+        "tenant_id": 2790683160709957,
+        "tenant_org_id": 2790685415443269,
+        "job": "店长",
+        "shop_name": "朗朗桌球",
+        "account_status": 1,
+        "is_reserve": 1,
+        "groupName": "",
+        "groupId": 0,
+        "alias_name": "葛芃",
+        "staff_profile_id": 0,
+        "site_label": "",
+        "rank_id": -1,
+        "ding_talk_synced": 1,
+        "new_rank_id": 0,
+        "new_staff_identity": 0,
+        "salary_grant_enabled": 2,
+        "rankName": "无职级",
+        "entry_type": 1,
+        "userRoles": [],
+        "entry_sign_status": 0,
+        "resign_sign_status": 0,
+        "criticism_status": 1,
+        "gender": 3
+      }
+    ]
+  },
+  "code": 0
+}
+```
+
+## 1. ODS 层设计
+
+### 1.1 ODS 任务规格
+
+```python
+OdsTaskSpec(
+    code="ODS_STAFF_INFO",
+    class_name="OdsStaffInfoTask",
+    table_name="ods.staff_info_master",
+    endpoint="/PersonnelManagement/SearchSystemStaffInfo",
+    data_path=("data",),
+    list_key="staffProfiles",
+    pk_columns=(_int_col("id", "id", required=True),),
+    extra_params={
+        "workStatusEnum": 0,
+        "dingTalkSynced": 0,
+        "staffIdentity": 0,
+        "rankId": 0,
+        "criticismStatus": 0,
+        "signStatus": -1,
+    },
+    include_source_endpoint=False,
+    include_fetched_at=False,
+    include_record_index=True,
+    requires_window=False,
+    time_fields=None,
+    snapshot_mode=SnapshotMode.FULL_TABLE,
+    description="员工档案 ODS：SearchSystemStaffInfo -> staffProfiles 原始 JSON",
+)
+```
+
+### 1.2 ODS 表 DDL：`ods.staff_info_master`
+
+```sql
+CREATE TABLE ods.staff_info_master (
+    id                    BIGINT NOT NULL,
+    tenant_id             BIGINT,
+    site_id               BIGINT,
+    tenant_org_id         BIGINT,
+    system_user_id        BIGINT,
+    staff_name            TEXT,
+    alias_name            TEXT,
+    mobile                TEXT,
+    avatar                TEXT,
+    gender                INTEGER,
+    job                   TEXT,
+    job_num               TEXT,
+    staff_identity        INTEGER,
+    status                INTEGER,
+    account_status        INTEGER,
+    system_role_id        INTEGER,
+    rank_id               INTEGER,
+    rank_name             TEXT,
+    new_rank_id           INTEGER,
+    new_staff_identity    INTEGER,
+    leave_status          INTEGER,
+    entry_time            TIMESTAMP WITHOUT TIME ZONE,
+    resign_time           TIMESTAMP WITHOUT TIME ZONE,
+    create_time           TIMESTAMP WITHOUT TIME ZONE,
+    is_delete             INTEGER,
+    is_reserve            INTEGER,
+    shop_name             TEXT,
+    site_label            TEXT,
+    cashier_point_id      BIGINT,
+    cashier_point_name    TEXT,
+    group_id              BIGINT,
+    group_name            TEXT,
+    staff_profile_id      BIGINT,
+    auth_code             TEXT,
+    auth_code_create      TIMESTAMP WITHOUT TIME ZONE,
+    ding_talk_synced      INTEGER,
+    salary_grant_enabled  INTEGER,
+    entry_type            INTEGER,
+    entry_sign_status     INTEGER,
+    resign_sign_status    INTEGER,
+    criticism_status      INTEGER,
+    user_roles            JSONB,
+    -- ETL 元数据
+    content_hash          TEXT NOT NULL,
+    source_file           TEXT,
+    fetched_at            TIMESTAMP WITH TIME ZONE DEFAULT now(),
+    payload               JSONB NOT NULL
+);
+
+COMMENT ON TABLE ods.staff_info_master IS '员工档案主数据（来源：SearchSystemStaffInfo API）';
+```
+
+### 1.3 ODS 列名映射说明
+
+API 返回的驼峰字段在 ODS 层统一转为蛇形命名（由 BaseOdsTask 自动处理）：
+- `cashierPointId` → `cashier_point_id`
+- `cashierPointName` → `cashier_point_name`
+- `staffName` / `staff_name` → `staff_name`（API 已是蛇形）
+- `systemUserId` / `system_user_id` → `system_user_id`
+- `tenantOrgId` / `tenant_org_id` → `tenant_org_id`
+- `groupName` → `group_name`（注意：API 返回驼峰 `groupName`）
+- `groupId` → `group_id`（API 返回驼峰 `groupId`）
+- `rankName` → `rank_name`（API 返回驼峰 `rankName`）
+- `userRoles` → `user_roles`（数组，存为 JSONB）
+- `authCodeCreate` / `auth_code_create` → `auth_code_create`
+
+## 2. DWD 层设计
+
+### 2.1 主表 DDL：`dwd.dim_staff`
+
+核心业务字段，高频查询使用。
+
+```sql
+CREATE TABLE dwd.dim_staff (
+    staff_id              BIGINT NOT NULL,
+    staff_name            TEXT,
+    alias_name            TEXT,
+    mobile                TEXT,
+    gender                INTEGER,
+    job                   TEXT,
+    tenant_id             BIGINT,
+    site_id               BIGINT,
+    system_role_id        INTEGER,
+    staff_identity        INTEGER,
+    status                INTEGER,
+    leave_status          INTEGER,
+    entry_time            TIMESTAMP WITH TIME ZONE,
+    resign_time           TIMESTAMP WITH TIME ZONE,
+    is_delete             INTEGER,
+    -- SCD2
+    scd2_start_time       TIMESTAMP WITH TIME ZONE NOT NULL,
+    scd2_end_time         TIMESTAMP WITH TIME ZONE,
+    scd2_is_current       INTEGER,
+    scd2_version          INTEGER,
+    PRIMARY KEY (staff_id, scd2_start_time)
+);
+
+COMMENT ON TABLE dwd.dim_staff IS '员工档案维度主表（SCD2）';
+```
+
+### 2.2 扩展表 DDL：`dwd.dim_staff_ex`
+
+次要/低频变更字段。
+
+```sql
+CREATE TABLE dwd.dim_staff_ex (
+    staff_id              BIGINT NOT NULL,
+    avatar                TEXT,
+    job_num               TEXT,
+    account_status        INTEGER,
+    rank_id               INTEGER,
+    rank_name             TEXT,
+    new_rank_id           INTEGER,
+    new_staff_identity    INTEGER,
+    is_reserve            INTEGER,
+    shop_name             TEXT,
+    site_label            TEXT,
+    tenant_org_id         BIGINT,
+    system_user_id        BIGINT,
+    cashier_point_id      BIGINT,
+    cashier_point_name    TEXT,
+    group_id              BIGINT,
+    group_name            TEXT,
+    staff_profile_id      BIGINT,
+    auth_code             TEXT,
+    auth_code_create      TIMESTAMP WITH TIME ZONE,
+    ding_talk_synced      INTEGER,
+    salary_grant_enabled  INTEGER,
+    entry_type            INTEGER,
+    entry_sign_status     INTEGER,
+    resign_sign_status    INTEGER,
+    criticism_status      INTEGER,
+    create_time           TIMESTAMP WITH TIME ZONE,
+    user_roles            JSONB,
+    -- SCD2
+    scd2_start_time       TIMESTAMP WITH TIME ZONE NOT NULL,
+    scd2_end_time         TIMESTAMP WITH TIME ZONE,
+    scd2_is_current       INTEGER,
+    scd2_version          INTEGER,
+    PRIMARY KEY (staff_id, scd2_start_time)
+);
+
+COMMENT ON TABLE dwd.dim_staff_ex IS '员工档案维度扩展表（SCD2）';
+```
+
+### 2.3 TABLE_MAP 映射
+
+```python
+# 在 DwdLoadTask.TABLE_MAP 中新增：
+"dwd.dim_staff": "ods.staff_info_master",
+"dwd.dim_staff_ex": "ods.staff_info_master",
+```
+
+### 2.4 FACT_MAPPINGS 字段映射
+
+```python
+# dim_staff 主表映射
+"dwd.dim_staff": [
+    ("staff_id", "id", None),
+    ("entry_time", "entry_time", "timestamptz"),
+    ("resign_time", "resign_time", "timestamptz"),
+],
+# dim_staff_ex 扩展表映射
+"dwd.dim_staff_ex": [
+    ("staff_id", "id", None),
+    ("rank_name", "rankname", None),
+    ("cashier_point_id", "cashierpointid", "bigint"),
+    ("cashier_point_name", "cashierpointname", None),
+    ("group_id", "groupid", "bigint"),
+    ("group_name", "groupname", None),
+    ("system_user_id", "systemuserid", "bigint"),
+    ("tenant_org_id", "tenantorgid", "bigint"),
+    ("auth_code_create", "auth_code_create", "timestamptz"),
+    ("create_time", "create_time", "timestamptz"),
+    ("user_roles", "userroles", "jsonb"),
+],
+```
+
+说明：
+- ODS 层的列名由 BaseOdsTask 自动从 API 驼峰转为蛇形（如 `cashierPointId` → `cashierpointid`，注意 PG 列名全小写无下划线）
+- DWD 主表中 `staff_name`、`alias_name`、`mobile` 等与 ODS 同名列自动映射，无需显式配置
+- `staff_id` 映射自 ODS 的 `id` 列
+
+## 3. 数据流概览
+
+```
+API: SearchSystemStaffInfo
+  ↓ (POST, 分页, extra_params 筛选)
+ODS: ods.staff_info_master
+  ↓ (SCD2 合并, FULL_TABLE 快照)
+DWD: dwd.dim_staff + dwd.dim_staff_ex
+```
+
+## 4. 测试框架
+
+- 测试框架：`pytest` + `hypothesis`
+- 单元测试使用 `FakeDB` / `FakeAPI`（`tests/unit/task_test_utils.py`）
+
+## 5. 正确性属性
+
+### P1：ODS 任务规格完整性
+对于 `ODS_STAFF_INFO` 任务规格，以下属性必须成立：
+- `code == "ODS_STAFF_INFO"`
+- `table_name == "ods.staff_info_master"`
+- `endpoint == "/PersonnelManagement/SearchSystemStaffInfo"`
+- `list_key == "staffProfiles"`
+- `snapshot_mode == SnapshotMode.FULL_TABLE`
+- `requires_window == False`
+- `time_fields is None`
+- `"staffProfiles"` 存在于 `DEFAULT_LIST_KEYS` 中
+- `"ODS_STAFF_INFO"` 存在于 `ENABLED_ODS_CODES` 中
+
+验证方式：单元测试直接断言
+
+### P2：DWD 映射完整性
+对于 DWD 装载配置，以下属性必须成立：
+- `TABLE_MAP["dwd.dim_staff"] == "ods.staff_info_master"`
+- `TABLE_MAP["dwd.dim_staff_ex"] == "ods.staff_info_master"`
+- `FACT_MAPPINGS["dwd.dim_staff"]` 包含 `staff_id` → `id` 的映射
+- `FACT_MAPPINGS["dwd.dim_staff_ex"]` 包含 `staff_id` → `id` 的映射
+
+验证方式：单元测试直接断言
+
+### P3：ODS 列名提取一致性（属性测试）
+对于任意 API 返回的员工记录（含驼峰和蛇形混合字段名），经 BaseOdsTask 处理后：
+- 所有字段名转为小写蛇形
+- `id` 字段不为空且为正整数
+- `payload` 字段包含完整原始 JSON
+
+验证方式：hypothesis 属性测试，生成随机员工记录验证转换一致性
+
+## 6. 文件变更清单
+
+### 代码变更
+| 文件 | 变更类型 | 说明 |
+|------|----------|------|
+| `apps/etl/connectors/feiqiu/api/client.py` | 修改 | `DEFAULT_LIST_KEYS` 添加 `"staffProfiles"` |
+| `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py` | 修改 | 新增 `ODS_STAFF_INFO` 任务规格 + 注册到 `ENABLED_ODS_CODES` |
+| `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py` | 修改 | `TABLE_MAP` 和 `FACT_MAPPINGS` 新增 dim_staff/dim_staff_ex 映射 |
+
+### DDL / 迁移
+| 文件 | 变更类型 | 说明 |
+|------|----------|------|
+| `db/etl_feiqiu/migrations/2026-02-22__add_staff_info_tables.sql` | 新增 | ODS + DWD 建表迁移脚本 |
+| `docs/database/ddl/etl_feiqiu__ods.sql` | 修改 | 追加 `ods.staff_info_master` DDL |
+| `docs/database/ddl/etl_feiqiu__dwd.sql` | 修改 | 追加 `dwd.dim_staff` + `dwd.dim_staff_ex` DDL |
+
+### 文档
+| 文件 | 变更类型 | 说明 |
+|------|----------|------|
+| `apps/etl/connectors/feiqiu/docs/database/ODS/mappings/mapping_SearchSystemStaffInfo_staff_info_master.md` | 新增 | API→ODS 字段映射文档 |
+| `apps/etl/connectors/feiqiu/docs/database/ODS/main/BD_manual_staff_info_master.md` | 新增 | ODS 表 BD 手册 |
+| `apps/etl/connectors/feiqiu/docs/database/DWD/main/BD_manual_dim_staff.md` | 新增 | DWD 主表 BD 手册 |
+| `apps/etl/connectors/feiqiu/docs/database/DWD/main/BD_manual_dim_staff_ex.md` | 新增 | DWD 扩展表 BD 手册 |
+| `apps/etl/connectors/feiqiu/docs/database/README.md` | 修改 | 增加员工表条目 |
+| `apps/etl/connectors/feiqiu/docs/etl_tasks/ods_tasks.md` | 修改 | 增加 ODS_STAFF_INFO 任务说明 |
+| `docs/database/README.md` | 修改 | 增加员工相关表条目 |