feat: P1-P3 全栈集成 — 数据库基础 + DWS 扩展 + 小程序鉴权 + 工程化体系
## P1 数据库基础 - zqyy_app: 创建 auth/biz schema、FDW 连接 etl_feiqiu - etl_feiqiu: 创建 app schema RLS 视图、商品库存预警表 - 清理 assistant_abolish 残留数据 ## P2 ETL/DWS 扩展 - 新增 DWS 助教订单贡献度表 (dws.assistant_order_contribution) - 新增 assistant_order_contribution_task 任务及 RLS 视图 - member_consumption 增加充值字段、assistant_daily 增加处罚字段 - 更新 ODS/DWD/DWS 任务文档及业务规则文档 - 更新 consistency_checker、flow_runner、task_registry 等核心模块 ## P3 小程序鉴权系统 - 新增 xcx_auth 路由/schema（微信登录 + JWT） - 新增 wechat/role/matching/application 服务层 - zqyy_app 鉴权表迁移 + 角色权限种子数据 - auth/dependencies.py 支持小程序 JWT 鉴权 ## 文档与审计 - 新增 DOCUMENTATION-MAP 文档导航 - 新增 7 份 BD_Manual 数据库变更文档 - 更新 DDL 基线快照（etl_feiqiu 6 schema + zqyy_app auth） - 新增全栈集成审计记录、部署检查清单更新 - 新增 BACKLOG 路线图、FDW→Core 迁移计划 ## Kiro 工程化 - 新增 5 个 Spec（P1/P2/P3/全栈集成/核心业务） - 新增审计自动化脚本（agent_on_stop/build_audit_context/compliance_prescan） - 新增 6 个 Hook（合规检查/会话日志/提交审计等） - 新增 doc-map steering 文件 ## 运维与测试 - 新增 ops 脚本：迁移验证/API 健康检查/ETL 监控/集成报告 - 新增属性测试：test_dws_contribution / test_auth_system - 清理过期 export 报告文件 - 更新 .gitignore 排除规则
This commit is contained in:
251
docs/database/BD_Manual_dws_member_spending_power_index.md
Normal file
251
docs/database/BD_Manual_dws_member_spending_power_index.md
Normal file
@@ -0,0 +1,251 @@
|
||||
# BD_Manual：dws_member_spending_power_index（SPI 消费力指数）
|
||||
|
||||
> DWS 表：`dws.dws_member_spending_power_index`
|
||||
> DWD 数据源：`dwd.dwd_settlement_head`（消费订单）、`dwd.dwd_recharge_order`（充值订单）
|
||||
> 配置表：`dws.cfg_index_parameters`（`index_type='SPI'`）
|
||||
> 任务代码：`DWS_SPENDING_POWER_INDEX`
|
||||
> 代码位置：`apps/etl/connectors/feiqiu/tasks/dws/index/spending_power_index_task.py`
|
||||
> DDL 位置：`docs/database/ddl/etl_feiqiu__dws.sql`
|
||||
> 迁移脚本：`db/etl_feiqiu/migrations/2026-02-23_create_dws_member_spending_power_index.sql`
|
||||
> 种子数据：`db/etl_feiqiu/seeds/seed_index_parameters.sql`（`index_type='SPI'` 部分）
|
||||
|
||||
---
|
||||
|
||||
## 1. 表结构
|
||||
|
||||
| 列名 | 类型 | 默认值 | 业务含义 | 取值范围/示例 |
|
||||
|------|------|--------|---------|-------------|
|
||||
| `spi_id` | BIGINT (SERIAL) | nextval 序列 | 自增主键（PK） | 自增 |
|
||||
| `site_id` | INTEGER NOT NULL | — | 门店 ID | 飞球门店 ID |
|
||||
| `member_id` | BIGINT NOT NULL | — | 会员 ID | 飞球会员 ID |
|
||||
| `spend_30` | NUMERIC(14,2) | 0 | 近 30 天消费总额（元） | `0.00` ~ 金额值 |
|
||||
| `spend_90` | NUMERIC(14,2) | 0 | 近 90 天消费总额（元） | `0.00` ~ 金额值 |
|
||||
| `recharge_90` | NUMERIC(14,2) | 0 | 近 90 天充值总额（元） | `0.00` ~ 金额值 |
|
||||
| `orders_30` | INTEGER | 0 | 近 30 天消费笔数 | `0` ~ 正整数 |
|
||||
| `orders_90` | INTEGER | 0 | 近 90 天消费笔数 | `0` ~ 正整数 |
|
||||
| `visit_days_30` | INTEGER | 0 | 近 30 天消费日数（按天去重） | `0` ~ `30` |
|
||||
| `visit_days_90` | INTEGER | 0 | 近 90 天消费日数（按天去重） | `0` ~ `90` |
|
||||
| `avg_ticket_90` | NUMERIC(14,2) | 0 | 90 天客单价（= spend_90 / max(orders_90, 1)） | `0.00` ~ 金额值 |
|
||||
| `active_weeks_90` | INTEGER | 0 | 近 90 天有消费的自然周数 | `0` ~ `13` |
|
||||
| `daily_spend_ewma_90` | NUMERIC(14,2) | 0 | 日消费 EWMA（指数加权移动平均） | `0.00` ~ 金额值 |
|
||||
| `score_level_raw` | NUMERIC(10,4) | 0 | Level 子分原始分（消费水平） | ≥ 0 |
|
||||
| `score_speed_raw` | NUMERIC(10,4) | 0 | Speed 子分原始分（消费速度） | ≥ 0 |
|
||||
| `score_stability_raw` | NUMERIC(10,4) | 0 | Stability 子分原始分（消费稳定性） | `0.0000` ~ `1.0000` |
|
||||
| `score_level_display` | NUMERIC(5,2) | 0 | Level 子分展示分 | `0.00` ~ `10.00` |
|
||||
| `score_speed_display` | NUMERIC(5,2) | 0 | Speed 子分展示分 | `0.00` ~ `10.00` |
|
||||
| `score_stability_display` | NUMERIC(5,2) | 0 | Stability 子分展示分 | `0.00` ~ `10.00` |
|
||||
| `raw_score` | NUMERIC(10,4) | 0 | SPI 总分原始分（加权合成） | ≥ 0 |
|
||||
| `display_score` | NUMERIC(5,2) | 0 | SPI 总分展示分 | `0.00` ~ `10.00` |
|
||||
| `calc_time` | TIMESTAMPTZ | NOW() | 本次计算时间 | ISO 时间戳 |
|
||||
| `created_at` | TIMESTAMPTZ | NOW() | 记录创建时间 | ISO 时间戳 |
|
||||
| `updated_at` | TIMESTAMPTZ | NOW() | 记录最后更新时间 | ISO 时间戳 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 主键与索引
|
||||
|
||||
| 名称 | 类型 | 列 | 说明 |
|
||||
|------|------|----|------|
|
||||
| `dws_member_spending_power_index_pkey` | PRIMARY KEY | `spi_id` | 物理主键（自增序列） |
|
||||
| `idx_spi_site_member` | UNIQUE INDEX | `(site_id, member_id)` | 业务主键：每个门店每个会员唯一一条记录 |
|
||||
| `idx_spi_display_score` | INDEX | `(site_id, display_score DESC)` | 按门店查询展示分排名，支持 TOP-N 查询 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据写入策略
|
||||
|
||||
- **delete-before-insert**：每次执行按 `site_id` 全量刷新
|
||||
1. `DELETE FROM dws.dws_member_spending_power_index WHERE site_id = %s`
|
||||
2. 批量 `INSERT` 新计算结果
|
||||
- 无数据时跳过（不删除、不插入），返回 `{'status': 'skipped', 'reason': 'no_data'}`
|
||||
|
||||
---
|
||||
|
||||
## 4. 算法概要
|
||||
|
||||
### 4.1 数据来源
|
||||
|
||||
| 来源表 | 筛选条件 | 提取内容 |
|
||||
|--------|---------|---------|
|
||||
| `dwd.dwd_settlement_head` | 近 90 天，`settle_type IN (1, 3)` | 消费金额、笔数、消费日数、周覆盖、日消费序列 |
|
||||
| `dwd.dwd_recharge_order` | 近 90 天，`settle_type = 5` | 充值总额 |
|
||||
|
||||
### 4.2 子分公式
|
||||
|
||||
- **Level**（消费水平，权重 0.60）：
|
||||
`L = w_s30 × ln(1 + spend_30/M30) + w_s90 × ln(1 + spend_90/M90) + w_ticket × ln(1 + avg_ticket_90/T0) + w_r90 × ln(1 + recharge_90/R90)`
|
||||
|
||||
- **Speed**（消费速度，权重 0.30）：
|
||||
`S = w_abs × V_abs + w_rel × max(0, V_rel) + w_ewma × V_ewma`
|
||||
- `V_abs = ln(1 + spend_30 / (max(visit_days_30, 1) × V0))`
|
||||
- `V_rel = ln((v_30 + ε) / (v_90 + ε))`，仅加速加分
|
||||
- `V_ewma = ln(1 + daily_spend_ewma_90 / E0)`
|
||||
|
||||
- **Stability**（消费稳定性，权重 0.10）：
|
||||
`P = active_weeks_90 / 13`，取值 [0, 1]
|
||||
|
||||
### 4.3 总分合成
|
||||
|
||||
`SPI_raw = w_L × L + w_S × S + w_P × P`（默认 0.60 / 0.30 / 0.10）
|
||||
|
||||
### 4.4 展示分映射
|
||||
|
||||
Raw → Winsorize(P5, P95) → 可选压缩(log1p/asinh) → MinMax [0, 10] → 可选 EWMA 平滑
|
||||
|
||||
子分（Level/Speed/Stability）各自独立映射，使用 `SPI_LEVEL` / `SPI_SPEED` / `SPI_STABILITY` 隔离分位历史。
|
||||
|
||||
---
|
||||
|
||||
## 5. 配置参数
|
||||
|
||||
所有参数存储在 `dws.cfg_index_parameters`（`index_type='SPI'`），缺失时回退到代码中 `DEFAULT_PARAMS`。
|
||||
|
||||
| 参数名 | 默认值 | 说明 |
|
||||
|--------|--------|------|
|
||||
| `spend_window_short_days` | 30 | 短窗口天数 |
|
||||
| `spend_window_long_days` | 90 | 长窗口天数 |
|
||||
| `ewma_alpha_daily_spend` | 0.3 | 日消费 EWMA 平滑系数 |
|
||||
| `amount_base_spend_30` | 500.0 | 30 天消费金额压缩基数 |
|
||||
| `amount_base_spend_90` | 1500.0 | 90 天消费金额压缩基数 |
|
||||
| `amount_base_ticket_90` | 200.0 | 客单价压缩基数 |
|
||||
| `amount_base_recharge_90` | 1000.0 | 充值金额压缩基数 |
|
||||
| `amount_base_speed_abs` | 100.0 | 绝对速度压缩基数 |
|
||||
| `amount_base_ewma_90` | 50.0 | EWMA 速度压缩基数 |
|
||||
| `w_level_spend_30` | 0.30 | Level 子分中 spend_30 权重 |
|
||||
| `w_level_spend_90` | 0.35 | Level 子分中 spend_90 权重 |
|
||||
| `w_level_ticket_90` | 0.20 | Level 子分中 avg_ticket_90 权重 |
|
||||
| `w_level_recharge_90` | 0.15 | Level 子分中 recharge_90 权重 |
|
||||
| `w_speed_abs` | 0.50 | Speed 子分中绝对速度权重 |
|
||||
| `w_speed_rel` | 0.30 | Speed 子分中相对速度权重 |
|
||||
| `w_speed_ewma` | 0.20 | Speed 子分中 EWMA 速度权重 |
|
||||
| `weight_level` | 0.60 | 总分中 Level 权重 |
|
||||
| `weight_speed` | 0.30 | 总分中 Speed 权重 |
|
||||
| `weight_stability` | 0.10 | 总分中 Stability 权重 |
|
||||
| `stability_window_days` | 90 | 稳定性计算窗口 |
|
||||
| `use_stability` | 1 | 是否启用稳定性子分（0=禁用） |
|
||||
| `percentile_lower` | 5 | Winsorize 下分位 |
|
||||
| `percentile_upper` | 95 | Winsorize 上分位 |
|
||||
| `compression_mode` | 1 | 压缩模式：0=无，1=log1p，2=asinh |
|
||||
| `use_smoothing` | 1 | 是否启用 EWMA 分位平滑 |
|
||||
| `ewma_alpha` | 0.2 | 分位平滑 EWMA 系数 |
|
||||
| `speed_epsilon` | 1e-6 | 速度计算防除零小量 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 前置依赖
|
||||
|
||||
- 任务依赖：`DWS_MEMBER_CONSUMPTION`（需先完成会员消费汇总）
|
||||
- 数据源表：`dwd.dwd_settlement_head`、`dwd.dwd_recharge_order` 必须已有数据
|
||||
- 配置表：`dws.cfg_index_parameters` 中 `index_type='SPI'` 种子数据已插入（缺失时使用默认值）
|
||||
- 分位历史表：`dws.dws_index_percentile_history`（首次执行时无历史，不平滑）
|
||||
|
||||
---
|
||||
|
||||
## 7. 验证 SQL
|
||||
|
||||
### 7.1 检查表是否存在且有数据
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) AS total_rows,
|
||||
COUNT(DISTINCT site_id) AS site_count,
|
||||
MIN(calc_time) AS earliest_calc,
|
||||
MAX(calc_time) AS latest_calc
|
||||
FROM dws.dws_member_spending_power_index;
|
||||
```
|
||||
|
||||
### 7.2 检查展示分范围是否合规（应全部在 [0, 10]）
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE display_score < 0 OR display_score > 10) AS spi_out_of_range,
|
||||
COUNT(*) FILTER (WHERE score_level_display < 0 OR score_level_display > 10) AS level_out_of_range,
|
||||
COUNT(*) FILTER (WHERE score_speed_display < 0 OR score_speed_display > 10) AS speed_out_of_range,
|
||||
COUNT(*) FILTER (WHERE score_stability_display < 0 OR score_stability_display > 10) AS stability_out_of_range
|
||||
FROM dws.dws_member_spending_power_index;
|
||||
-- 预期：所有列均为 0
|
||||
```
|
||||
|
||||
### 7.3 检查业务主键唯一性（不应有重复）
|
||||
|
||||
```sql
|
||||
SELECT site_id, member_id, COUNT(*) AS cnt
|
||||
FROM dws.dws_member_spending_power_index
|
||||
GROUP BY site_id, member_id
|
||||
HAVING COUNT(*) > 1;
|
||||
-- 预期：无结果返回
|
||||
```
|
||||
|
||||
### 7.4 按门店查看 SPI 分布概况
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
site_id,
|
||||
COUNT(*) AS member_count,
|
||||
ROUND(AVG(display_score), 2) AS avg_spi,
|
||||
ROUND(MIN(display_score), 2) AS min_spi,
|
||||
ROUND(MAX(display_score), 2) AS max_spi,
|
||||
ROUND(PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY display_score), 2) AS median_spi
|
||||
FROM dws.dws_member_spending_power_index
|
||||
GROUP BY site_id
|
||||
ORDER BY site_id;
|
||||
```
|
||||
|
||||
### 7.5 检查 Stability 子分原始分范围（应在 [0, 1]）
|
||||
|
||||
```sql
|
||||
SELECT COUNT(*) AS out_of_range
|
||||
FROM dws.dws_member_spending_power_index
|
||||
WHERE score_stability_raw < 0 OR score_stability_raw > 1;
|
||||
-- 预期：0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 兼容性说明
|
||||
|
||||
| 影响范围 | 说明 |
|
||||
|---------|------|
|
||||
| ETL 任务 | 新增任务 `DWS_SPENDING_POWER_INDEX`，依赖 `DWS_MEMBER_CONSUMPTION`。不影响现有 WBI/NCI/RS/OS/MS/ML 指数任务 |
|
||||
| 后端 API | 当前无 API 直接读取此表。后续如需暴露 SPI 数据，需新增接口 |
|
||||
| 管理后台 | 当前无前端页面展示 SPI。后续可在会员详情页新增 SPI 展示 |
|
||||
| 小程序 | 无影响 |
|
||||
| 其他指数 | SPI 独立于现有指数体系，不修改任何已有表或任务逻辑 |
|
||||
| 分位历史 | SPI 会向 `dws.dws_index_percentile_history` 写入 `index_type='SPI'`/`SPI_LEVEL`/`SPI_SPEED`/`SPI_STABILITY` 的分位记录 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 回滚策略
|
||||
|
||||
### 9.1 删除数据（保留表结构）
|
||||
|
||||
```sql
|
||||
DELETE FROM dws.dws_member_spending_power_index;
|
||||
DELETE FROM dws.dws_index_percentile_history WHERE index_type LIKE 'SPI%';
|
||||
DELETE FROM dws.cfg_index_parameters WHERE index_type = 'SPI';
|
||||
```
|
||||
|
||||
### 9.2 完整回滚（删除表）
|
||||
|
||||
```sql
|
||||
DROP INDEX IF EXISTS dws.idx_spi_display_score;
|
||||
DROP INDEX IF EXISTS dws.idx_spi_site_member;
|
||||
DROP TABLE IF EXISTS dws.dws_member_spending_power_index;
|
||||
DROP SEQUENCE IF EXISTS dws.dws_member_spending_power_index_spi_id_seq;
|
||||
```
|
||||
|
||||
### 9.3 回滚任务注册
|
||||
|
||||
从 `orchestration/task_registry.py` 中移除 `DWS_SPENDING_POWER_INDEX` 注册行，并从 `tasks/dws/index/__init__.py` 和 `tasks/dws/__init__.py` 中移除 `SpendingPowerIndexTask` 导出。
|
||||
|
||||
---
|
||||
|
||||
## 10. 代码引用
|
||||
|
||||
- 任务类：`tasks/dws/index/spending_power_index_task.py` → `SpendingPowerIndexTask`
|
||||
- 继承：`BaseIndexTask`（`tasks/dws/index/base_index_task.py`）
|
||||
- 任务注册：`orchestration/task_registry.py` → `DWS_SPENDING_POWER_INDEX`
|
||||
- 属性测试：`tests/test_spi_properties.py`
|
||||
- 单元测试：`apps/etl/connectors/feiqiu/tests/unit/test_spi_task.py`
|
||||
- 迁移脚本：`db/etl_feiqiu/migrations/2026-02-23_create_dws_member_spending_power_index.sql`
|
||||
- 种子数据：`db/etl_feiqiu/seeds/seed_index_parameters.sql`
|
||||
Reference in New Issue
Block a user