root/Neo-ZQYY
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: 累积功能变更 — 聊天集成、租户管理、小程序更新、ETL 增强、迁移脚本

Browse Source 
包含多个会话的累积代码变更：
- backend: AI 聊天服务、触发器调度、认证增强、WebSocket、调度器最小间隔
- admin-web: ETL 状态页、任务管理、调度配置、登录优化
- miniprogram: 看板页面、聊天集成、UI 组件、导航更新
- etl: DWS 新任务（finance_area_daily/board_cache）、连接器增强
- tenant-admin: 项目初始化
- db: 19 个迁移脚本（etl_feiqiu 11 + zqyy_app 8）
- packages/shared: 枚举和工具函数更新
- tools: 数据库工具、报表生成、健康检查
- docs: PRD/架构/部署/合约文档更新

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Neo
2026-04-06 00:03:48 +08:00
parent 70324d8542
commit 6f8f12314f
515 changed files with 76604 additions and 7456 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
apps/etl/connectors/feiqiu/.env
View File

@@ -26,7 +26,7 @@ SCHEMA_ETL=meta
# API 配置（上游 SaaS API）
# ------------------------------------------------------------------------------
API_BASE=https://pc.ficoo.vip/apiprod/admin/v1/
API_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnQtdHlwZSI6IjQiLCJ1c2VyLXR5cGUiOiIxIiwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9yb2xlIjoiMTIiLCJyb2xlLWlkIjoiMTIiLCJ0ZW5hbnQtaWQiOiIyNzkwNjgzMTYwNzA5OTU3Iiwibmlja25hbWUiOiLnp5_miLfnrqHnkIblkZjvvJrmganmgakxIiwic2l0ZS1pZCI6IjAiLCJtb2JpbGUiOiIxMzgxMDUwMjMwNCIsInNpZCI6IjI5NTA0ODk2NTgzOTU4NDUiLCJzdGFmZi1pZCI6IjMwMDk5MTg2OTE1NTkwNDUiLCJvcmctaWQiOiIwIiwicm9sZS10eXBlIjoiMyIsInJlZnJlc2hUb2tlbiI6InhZbFozZDN1ekR3cnpkaFNUeVpFYnJVQUc5ZEtrZDVrK1FUWEd0Ym9LQkU9IiwicmVmcmVzaEV4cGlyeVRpbWUiOiIyMDI2LzMvMjYg5LiK5Y2IMTozMTo1MCIsIm5lZWRDaGVja1Rva2VuIjoiZmFsc2UiLCJleHAiOjE3NzQ0NTk5MTAsImlzcyI6InRlc3QiLCJhdWQiOiJVc2VyIn0.yNtsQIIQPVoFkjPUHWVsi9nRGc_lSgFGerxOfJSDOcc
API_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnQtdHlwZSI6IjQiLCJ1c2VyLXR5cGUiOiIxIiwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9yb2xlIjoiMTIiLCJyb2xlLWlkIjoiMTIiLCJ0ZW5hbnQtaWQiOiIyNzkwNjgzMTYwNzA5OTU3Iiwibmlja25hbWUiOiLnp5_miLfnrqHnkIblkZjvvJrmganmgakxIiwic2l0ZS1pZCI6IjAiLCJtb2JpbGUiOiIxMzgxMDUwMjMwNCIsInNpZCI6IjI5NTA0ODk2NTgzOTU4NDUiLCJzdGFmZi1pZCI6IjMwMDk5MTg2OTE1NTkwNDUiLCJvcmctaWQiOiIwIiwicm9sZS10eXBlIjoiMyIsInJlZnJlc2hUb2tlbiI6IkZJUUxIWWJLSFl5QktJQlVuLzZuQVdINitqOEpLbGNHN1NScDFFaFNuMEE9IiwicmVmcmVzaEV4cGlyeVRpbWUiOiIyMDI2LzQvNyDkuIrljYg0OjU2OjA4IiwibmVlZENoZWNrVG9rZW4iOiJmYWxzZSIsImV4cCI6MTc3NTUwODk2OCwiaXNzIjoidGVzdCIsImF1ZCI6IlVzZXIifQ.fc92FpZrLBkV9tD9sNPCvMvgNePh5Y7T6g5FLx8N16A
API_TIMEOUT=20
API_PAGE_SIZE=200
API_RETRY_MAX=3
@@ -174,7 +174,7 @@ DWD_FACT_UPSERT=true
# 任务列表配置
# ------------------------------------------------------------------------------
RUN_TASKS=PRODUCTS,TABLES,MEMBERS,ASSISTANTS,PACKAGES_DEF,ORDERS,PAYMENTS,REFUNDS,COUPON_USAGE,INVENTORY_CHANGE,TOPUPS,TABLE_DISCOUNT,LEDGER
INDEX_LOOKBACK_DAYS=60
INDEX_LOOKBACK_DAYS=90
# ------------------------------------------------------------------------------
# DWS 月度/薪资配置（defaults.py → dws.*）

11
apps/etl/connectors/feiqiu/cli/main.py
View File

@@ -315,6 +315,13 @@ def parse_args():
action="store_true",
help="强制全量处理：跳过 ODS hash 去重和 DWD 变更对比，无条件写入",
)
# P19: 回测模式 — "假装今天是 X 日"重算指数
parser.add_argument(
"--as-of-date",
dest="as_of_date",
help="回测基准日期（如 2026-03-01），指数任务用此替代 NOW()",
)
return parser.parse_args()
@@ -433,6 +440,10 @@ def build_cli_overrides(args) -> dict:
if args.force_full:
overrides.setdefault("run", {})["force_full_update"] = True
# P19: 回测基准日期
if getattr(args, "as_of_date", None):
overrides.setdefault("run", {})["as_of_date"] = args.as_of_date
# Pipeline 管道参数 → pipeline.* 命名空间（供 PipelineConfig.from_app_config() 读取）
if getattr(args, "pipeline_workers", None) is not None:
overrides.setdefault("pipeline", {})["workers"] = args.pipeline_workers

2
apps/etl/connectors/feiqiu/config/defaults.py
View File

@@ -57,7 +57,7 @@ DEFAULTS = {
],
"dws_tasks": [],
"index_tasks": [],
"index_lookback_days": 60,
"index_lookback_days": 90,
"window_minutes": {
"default_busy": 30,
"default_idle": 180,

479
apps/etl/connectors/feiqiu/docs/api-reference/endpoints/tenant_config.md Normal file
View File

@@ -0,0 +1,479 @@
# 租户配置（GetTenantConfig）接口字段分析报告
> 手动分析于 2026-03-20 | 数据来源：浏览器抓包
## 基本信息
| 属性 | 值 |
| -------------- | ---------------------------------------------------------------- |
| 接口路径 | `System/GetTenantConfig` |
| 完整 URL | `https://pc.ficoo.vip/apiprod/admin/v1/System/GetTenantConfig` |
| 请求方法 | `POST` |
| Content-Type | 无 body（`body: null`） |
| 鉴权方式 | Bearer Token（`Authorization` 头） |
| 分页方式 | 无分页（单次返回全量） |
| 时间范围 | 不需要 |
| 响应结构 | `data` 对象包含 7 个子对象 + 1 个顶层字段 |
## 请求参数
无请求体。通过 Token 中的 `tenant-id` 和 `site-id` 确定返回范围。当 `site-id=0` 时返回租户级默认配置。
## 响应顶层结构
| 子对象 | 类型 | 说明 |
| ----------------------------- | -------- | ----------------------------------------------------------- |
| `showAssistant` | int | 顶层开关：是否展示助教模块（1=展示） |
| `siteConfig` | object | 门店级配置（当 site_id=0 时与 tenantConfig 字段完全一致） |
| `tenantConfig` | object | 租户级配置（品牌默认值） |
| `siteProfile` | object | 门店档案信息（site_id=0 时全部为零值/空值） |
| `tenantProfile` | object | 租户档案信息（品牌名称、图标、状态） |
| `assistantLevels` | array | 助教等级配置列表（当前为空数组） |
| `siteUserAppletConfig` | object | 小程序端功能配置（site_id=0 时全部为零值/默认值） |
| `memberCardGradeConfigList` | array | 会员卡等级配置列表（当前为空数组） |
### 关于 siteConfig 与 tenantConfig 的关系
两者字段完全一致（~150 个字段），当 `site_id=0` 时值也完全一致。业务含义：
- `tenantConfig`：品牌级默认配置
- `siteConfig`：门店级覆盖配置（门店可以覆盖品牌默认值）
- 当请求 `site_id=0` 时，siteConfig 回退到 tenantConfig 的值
**ETL 建议**：ODS 层只存一份（tenantConfig），DWD 层按需区分。如果未来需要按门店拉取，再增加 siteConfig 的独立存储。
---
## 一、tenantConfig / siteConfig 字段分析（共 ~150 字段）
> siteConfig 与 tenantConfig 字段完全一致，以下统一分析。按业务域分组。
> 开关字段惯例：1=启用/是，2=禁用/否（飞球系统通用约定），0=未设置/默认。
> 「是否保留」列留空，由人工筛选填写。
### 1.1 主键与标识
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| --- | ------------- | ------ | ------------------ | ------------------------- | ---------- |
| 1 | `id` | int | 2790683527351109 | 配置记录主键 ID | Y |
| 2 | `tenant_id` | int | 2790683160709957 | 租户 ID（品牌标识） | Y |
| 3 | `site_id` | int | 0 | 门店 ID（0=租户级默认） | Y |
### 1.2 助教模块配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | --------------------------------- | -------- | ------------ | ---------------------------------------------------------------------- | ------------------------------------ |
| 4 | `able_assistant_hang_more` | int | 2 | 助教是否允许同时挂多台（2=禁用）。控制一个助教能否同时服务多张球台 | |
| 5 | `assistant_add_clock` | int | 2 | 助教是否可以自行加钟（2=禁用）。控制助教端是否有"加钟"按钮 | |
| 6 | `assistant_global_name` | string | "助理教练" | 助教在系统中的全局称谓。不同门店可能叫"助教""教练""陪练"等 | Y |
| 7 | `assistant_live` | int | 2 | 助教直播功能开关（2=禁用） | |
| 8 | `assistant_performance_confirm` | int | 1 | 助教绩效是否需要确认（1=需要）。助教完成服务后是否需要管理员确认绩效 | |
| 9 | `assistant_ranking` | int | 1 | 助教排行榜功能开关（1=启用） | |
| 10 | `assistant_restrict_count` | int | 3 | 助教限制数量。推测为单台最多可安排的助教人数上限 | |
| 11 | `assistant_reward_name` | string | "激励" | 助教奖励/激励的自定义名称。对应 DWS 中的`assistant_reward_name` 配置 | Y |
| 12 | `assistant_salary_allocation` | int | 2 | 助教薪资分配方式开关（2=禁用自动分配） | 统一使用本项目的薪资计算结算方式。 |
| 13 | `assistant_schedule` | int | 2 | 助教排班功能开关（2=禁用） | |
| 14 | `assistant_show_type` | int | 2 | 助教展示方式。推测 1=列表 2=卡片 或类似的前端展示模式切换 | |
| 15 | `assistant_time_over` | int | 1 | 助教超时处理开关（1=启用）。服务超时后是否自动结束或提醒 | |
| 16 | `is_assistant_rotation` | int | 1 | 助教轮转/轮岗开关（1=启用）。是否按轮次自动分配助教 | |
| 17 | `is_offline_assistant` | int | 1 | 线下助教模式开关（1=启用）。是否支持线下（非系统派单）的助教服务 | |
| 18 | `virtual_playwith_limit` | int | 1 | 虚拟陪打限制（1=启用限制）。控制虚拟/模拟陪打场景的约束 | |
| 19 | `virtual_reward` | int | 2 | 虚拟奖励开关（2=禁用）。是否启用虚拟激励/奖励机制 | |
### 1.3 收银与支付配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ------------------------- | --------- | -------- | ------------------------------------------------------------------------ | ---------- |
| 20 | `able_cash_pay` | int | 2 | 现金支付开关（2=禁用）。收银台是否接受现金 | |
| 21 | `able_offline_pay` | int | 2 | 线下支付开关（2=禁用）。是否支持线下（非在线）支付方式 | |
| 22 | `able_pos_pay` | int | 2 | POS 机支付开关（2=禁用） | |
| 23 | `able_revoke_settle` | int | 2 | 撤销结算开关（2=禁用）。已结算订单是否允许撤销 | |
| 24 | `can_round_manually` | int | 2 | 手动抹零开关（2=禁用）。收银时是否允许手动调整金额（抹零） | |
| 25 | `pay_round_accuracy` | int | 0 | 支付金额舍入精度。0=不舍入，其他值表示小数位数 | |
| 26 | `pay_round_type` | int | 0 | 支付金额舍入方式。0=不舍入，1=四舍五入，2=向上取整，3=向下取整（推测） | |
| 27 | `up_round` | decimal | 0.00 | 向上取整阈值。配合 pay_round_type 使用 | |
| 28 | `down_round` | decimal | 0.00 | 向下取整阈值。配合 pay_round_type 使用 | |
| 29 | `split_real_charge` | int | 1 | 拆分实收/应收开关（1=启用）。结算时是否区分实收金额和应收金额 | |
| 30 | `currency_symbol_type` | int | 1 | 货币符号类型。1=¥（人民币），其他值对应其他货币符号 | |
| 31 | `income_inclusion_type` | int | 0 | 收入计入方式。0=默认，控制哪些收入项目计入营收统计 | |
### 1.4 会员与储值卡配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | -------------------------------- | --------- | ----------------------------- | -------------------------------------------------------------- | ---------- |
| 32 | `applet_member_recharge` | int | 2 | 小程序端会员充值开关（2=禁用） | |
| 33 | `member_app_type` | int | 6 | 会员应用类型。6 可能表示某种会员系统版本或模式 | |
| 34 | `member_bind_plural_table` | int | 1 | 会员绑定多台开关（1=允许）。一个会员是否可以同时绑定多张球台 | |
| 35 | `member_consumption_sms` | int | 1 | 会员消费短信通知开关（1=启用） | |
| 36 | `member_discount_balance_mode` | int | 1 | 会员折扣余额模式。控制折扣与余额的计算优先级 | |
| 37 | `member_discount_mode` | int | 2 | 会员折扣模式。2=某种折扣计算方式（如按等级/按消费额） | |
| 38 | `member_expense_hint` | int | 2 | 会员消费提醒开关（2=禁用）。消费达到阈值时是否提醒 | |
| 39 | `member_expense_money` | decimal | 500.00 | 会员消费提醒阈值（元）。当单次消费超过此金额时触发提醒 | |
| 40 | `member_login_method` | string | "1" | 会员登录方式。"1"=手机号登录（推测） | |
| 41 | `default_member_avatar` | string | "https://oss.ficoo.vip/..." | 会员默认头像 URL | |
| 42 | `default_member_password` | string | "123456" | ⚠️ 敏感字段：会员默认密码。ODS 层需脱敏（保留最后 2 字符） | |
| 43 | `recharge_card_included` | int | 1 | 充值卡纳入统计开关（1=纳入）。充值卡金额是否计入营收 | |
| 44 | `gift_card_included` | int | 1 | 赠送卡纳入统计开关（1=纳入）。赠送金额是否计入营收 | |
| 45 | `platform_coupon_included` | int | 1 | 平台优惠券纳入统计开关（1=纳入） | |
| 46 | `self_coupon_included` | int | 1 | 自有优惠券纳入统计开关（1=纳入） | |
| 47 | `rcp_able_chose_member_grade` | int | 1 | 收银台是否可选择会员等级（1=可选） | |
| 48 | `send_level_change_notice` | int | 1 | 会员等级变更通知开关（1=发送） | |
| 49 | `auto_rela_sales_man` | int | 2 | 自动关联销售员开关（2=禁用）。新会员是否自动关联推荐销售员 | |
| 50 | `consume_auto_rela_sales_man` | int | 2 | 消费时自动关联销售员开关（2=禁用） | |
| 51 | `is_rela_more_salesmen` | int | 1 | 允许关联多个销售员开关（1=允许） | |
### 1.5 会员等级与积分规则
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ---------------------------- | -------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| 52 | `is_calc_points` | int | 1 | 积分计算开关（1=启用）。消费时是否自动计算积分 | |
| 53 | `point_alias` | string | "积分" | 积分的自定义名称。不同品牌可能叫"积分""金币""豆子"等 | |
| 54 | `point_expire_num` | int | 0 | 积分过期数值。0=不过期 | |
| 55 | `point_expire_type` | int | 1 | 积分过期类型。1=按固定周期 | |
| 56 | `point_expire_unit` | int | 1 | 积分过期单位。1=月（推测） | |
| 57 | `point_rule` | string | "" | 积分规则描述（当前为空，可能是 JSON 或文本规则） | |
| 58 | `growth_value_expire_num` | int | 0 | 成长值过期数值。0=不过期 | |
| 59 | `growth_value_expire_type` | int | 1 | 成长值过期类型 | |
| 60 | `growth_value_expire_unit` | int | 1 | 成长值过期单位 | |
| 61 | `member_grade_expire_num` | int | 0 | 会员等级过期数值。0=不过期 | |
| 62 | `member_grade_expire_type` | int | 1 | 会员等级过期类型 | |
| 63 | `member_grade_expire_unit` | int | 1 | 会员等级过期单位 | |
| 64 | `member_rule_config_json` | string(JSON) | `{growthValueExpireNum: 12,...}` | 会员规则综合配置 JSON。包含成长值过期、等级升级类型、等级保级策略等。注意：非标准 JSON（缺引号），解析时需容错。**建议 JSONB 存储** | |
| 65 | `registration_info_rule` | string | "" | 注册信息规则（当前为空） | |
### 1.6 球台与计费配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ---------------------------- | ------ | -------- | -------------------------------------------------------------------------- | ---------- |
| 66 | `billing_rate_time_type` | int | 1 | 计费费率时间类型。控制按分钟/小时/时段计费 | |
| 67 | `business_time_type` | int | 1 | 营业时间类型。1=标准营业时间模式 | |
| 68 | `charge_time_type` | int | 1 | 计费时间类型。与 billing_rate_time_type 配合，控制计费起止时间的计算方式 | |
| 69 | `cross_region_fee` | int | 1 | 跨区域费用开关（1=启用）。不同区域球台之间转台是否收取额外费用 | |
| 70 | `table_rental_fee_enable` | int | 1 | 台费租赁费开关（1=启用）。是否收取球台租赁费 | |
| 71 | `normal_table_number` | int | -1 | 普通台数量。-1=不限制（由实际球台数决定） | |
| 72 | `rest_table_number` | int | -1 | 休息台数量。-1=不限制 | |
| 73 | `virtual_table_number` | int | -1 | 虚拟台数量。-1=不限制。虚拟台用于不占用实体球台的服务场景 | |
| 74 | `temporary_light_time` | int | 0 | 临时开灯时长（分钟）。0=不启用临时开灯 | |
| 75 | `temporary_light_max_time` | int | 0 | 临时开灯最大时长（分钟） | |
### 1.7 收银台与前台配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ---------------------------- | ------ | -------- | ------------------------------------------------------------ | ---------- |
| 76 | `bill_font_type` | int | 1 | 小票字体类型。1=标准字体 | |
| 77 | `cash_register_goods_sort` | int | 1 | 收银台商品排序方式。1=默认排序 | |
| 78 | `cashier_login_restrict` | int | 1 | 收银员登录限制（1=启用）。是否限制收银员只能在指定设备登录 | |
| 79 | `cashier_switch_light` | int | 1 | 收银台开关灯控制（1=启用）。收银台是否可以控制球台灯光 | |
| 80 | `cashier_table_show_money` | int | 1 | 收银台球台显示金额（1=显示）。球台列表是否显示当前消费金额 | |
| 81 | `cashier_temporary_light` | int | 1 | 收银台临时开灯（1=启用） | |
| 82 | `rcp_table_list_version` | int | 1 | 收银台球台列表版本。1=V1 版本布局 | |
| 83 | `coupon_goods_auto_print` | int | 2 | 优惠券商品自动打印开关（2=禁用） | |
| 84 | `is_not_paid_goods` | int | 2 | 未付款商品显示开关（2=不显示） | |
| 85 | `return_goods_settled` | int | 1 | 已结算商品退货开关（1=允许） | |
| 86 | `enable_goods_remark` | int | 1 | 商品备注功能开关（1=启用） | |
| 87 | `enable_serial_number` | int | 1 | 序列号功能开关（1=启用） | |
### 1.8 智慧屏与前厅配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ----------------------------------- | --------- | ----------------------- | -------------------------------------------------------- | ---------- |
| 88 | `enable_smart_antechamber` | int | 1 | 智慧前厅开关（1=启用）。是否启用智慧前厅（自助服务屏） | |
| 89 | `is_smart_screen_pay` | int | 1 | 智慧屏支付开关（1=启用）。智慧屏是否支持直接支付 | |
| 90 | `is_show_smart_screen_tip` | int | 1 | 智慧屏提示开关（1=显示） | |
| 91 | `smart_screen_menu_cfg` | string | "1,2,3" | 智慧屏菜单配置。逗号分隔的菜单项 ID | |
| 92 | `smart_screen_rental` | decimal | 0.00 | 智慧屏租赁费（元/月） | |
| 93 | `smart_screen_rental_receiver_id` | int | 0 | 智慧屏租赁费收款方 ID。0=未设置 | |
| 94 | `smart_screen_rental_start_time` | string | "0001-01-01 00:00:00" | 智慧屏租赁开始时间（零值=未设置） | |
| 95 | `smart_screen_rental_end_time` | string | "0001-01-01 00:00:00" | 智慧屏租赁结束时间（零值=未设置） | |
| 96 | `is_open_screen_savers` | int | 1 | 屏保开关（1=启用） | |
| 97 | `front_delivery_enabled` | int | 1 | 前台配送开关（1=启用）。前台是否支持商品配送到桌 | |
### 1.9 通知与短信配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ---------------------- | -------- | -------- | --------------------------------- | ---------- |
| 98 | `able_notify` | int | 1 | 通知总开关（1=启用） | |
| 99 | `is_sms_notified` | int | 0 | 短信通知开关（0=禁用） | |
| 100 | `sms_notice_mobile` | string | "" | 短信通知接收手机号（空=未设置） | |
| 101 | `total_sms_credit` | int | 0 | 短信余额/额度。0=无余额 | |
| 102 | `is_rcp_notice_ring` | int | 1 | 收银台通知铃声开关（1=启用） | |
| 103 | `announce_one` | int | 0 | 公告位 1 开关（0=关闭） | |
| 104 | `announce_two` | int | 0 | 公告位 2 开关（0=关闭） | |
### 1.10 考勤与排班配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ------------------------------------ | -------------- | ----------------------- | ---------------------------------------------------- | ---------- |
| 105 | `attendance_enabled` | int | 1 | 考勤功能总开关（1=启用） | |
| 106 | `attendance_sync_deadline` | string | "2000-01-01 08:00:00" | 考勤同步截止时间。零值占位=使用默认 | |
| 107 | `absent_flag` | int | 1 | 缺勤标记开关（1=启用）。是否自动标记缺勤 | |
| 108 | `absent_number` | int | 7 | 缺勤天数阈值。连续缺勤超过此天数触发告警/处理 | |
| 109 | `shift_enabled` | int | 1 | 班次功能开关（1=启用） | |
| 110 | `shift_turnover_goods` | int | 1 | 交班时商品盘点开关（1=启用） | |
| 111 | `shift_turnover_type` | int | 1 | 交班类型。1=标准交班流程 | |
| 112 | `sign_enable` | int | 1 | 签到功能开关（1=启用） | |
| 113 | `sign_config` | string(JSON) | "{}" | 签到配置 JSON（当前为空对象）。**建议 JSONB 存储** | |
| 114 | `online_sign_enable` | int | 1 | 在线签到开关（1=启用） | |
| 115 | `offline_sign_enable` | int | 1 | 离线签到开关（1=启用） | |
| 116 | `site_attendance_approval_enabled` | int | 1 | 门店考勤审批开关（1=启用） | |
| 117 | `site_staff_salary_enabled` | int | 1 | 门店员工薪资开关（1=启用） | |
### 1.11 财务与结算配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | --------------------------------- | -------- | ----------------------- | --------------------------------------------------------------------------- | ---------- |
| 118 | `cutoff_time` | string | "2000-01-05 08:00:00" | 财务截止时间（日结时间点）。只取时间部分 08:00，表示每日 8 点为财务日切点 | Y |
| 119 | `site_cutoff_time` | string | "0001-01-01 00:00:00" | 门店级财务截止时间（零值=使用租户默认） | Y |
| 120 | `daily_settle_enable` | int | 1 | 日结功能开关（1=启用）。是否启用每日自动结算 | |
| 121 | `data_analysis` | int | 1 | 数据分析功能开关（1=启用）。是否在管理端展示数据分析模块 | |
| 122 | `financial_analysis_url` | string | "" | 财务分析外链 URL（空=未配置外部 BI 链接） | |
| 123 | `performance_time` | int | 2 | 绩效计算时间维度。2=按月（推测：1=按周，2=按月，3=按季） | |
| 124 | `revenue_switch` | int | 2 | 营收统计开关（2=禁用）。是否在首页展示营收数据 | |
| 125 | `able_show_platform_receivable` | int | 1 | 平台应收展示开关（1=展示）。是否展示平台代收的应收款项 | |
| 126 | `declaration_concealment` | int | 1 | 报表隐藏开关（1=启用）。是否隐藏部分敏感财务报表 | |
| 127 | `coach_look_performance` | int | 1 | 教练查看绩效开关（1=允许）。助教/教练是否可以查看自己的绩效数据 | |
### 1.12 商品与库存配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ----------------------------- | ------ | -------- | --------------------------------- | ---------- |
| 128 | `goods_version_cfg` | int | 1 | 商品版本配置。1=V1 版本商品管理 | |
| 129 | `warehouse_type` | int | 1 | 仓库类型。1=单仓模式 | |
| 130 | `manager_applet_goods_sort` | int | 1 | 管理端小程序商品排序方式 | |
### 1.13 灯控与电力配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ------------------ | ------ | -------- | --------------------------------------------------- | ---------- |
| 131 | `is_electricity` | int | 1 | 电力管理开关（1=启用）。是否启用球台电力/灯控管理 | |
### 1.14 团队编辑配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | -------------------------------- | -------- | ----------------------- | ------------------------------------------------------------------- | ---------- |
| 132 | `edit_team_current_next_month` | int | 1 | 编辑团队当月/下月开关（1=允许）。是否允许编辑当月和下月的团队配置 | |
| 133 | `edit_team_start_time` | string | "2025-01-01 00:00:00" | 团队编辑允许的开始时间 | |
| 134 | `edit_team_end_time` | string | "2025-01-01 00:00:00" | 团队编辑允许的结束时间（与 start 相同=未限制） | |
### 1.15 小程序与线上配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ------------------------------------- | -------- | -------- | ---------------------------------------------- | ---------- |
| 135 | `enable_auto_accept` | int | 1 | 自动接单开关（1=启用）。线上订单是否自动接受 | |
| 136 | `enable_outdoor_order` | int | 2 | 外卖/外送订单开关（2=禁用） | |
| 137 | `enable_professional_member_applet` | int | 1 | 专业版会员小程序开关（1=启用） | |
| 138 | `member_applet_share_url` | string | "" | 会员小程序分享链接（空=未配置） | |
### 1.16 百能运营费配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ------------------------------------- | --------- | -------- | ----------------------------------------------------------------------- | ---------- |
| 139 | `baineng_operation_fee_enable` | int | 1 | 百能运营费开关（1=启用）。百能=第三方运营服务商，是否启用其运营费抽成 | |
| 140 | `baineng_operation_fee_ratio` | decimal | 0.00 | 百能运营费比例。0.00=不抽成（虽然开关启用但比例为 0） | |
| 141 | `baineng_operation_fee_receiver_id` | int | 0 | 百能运营费收款方 ID。0=未设置 | |
### 1.17 UI 与颜色配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | --------------------- | -------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| 142 | `color_config_json` | string(JSON) | `{"statuses":{"assistant":"#DA5F8E",...}}` | PC 端球台状态颜色配置。包含：assistant（助教服务中）、cluster（拼台）、font（字体）、leisure（空闲）、reserve（预约）、start（开台）、warning（警告）。**建议 JSONB 存储** | |
| 143 | `min_color_json` | string(JSON) | `{"statuses":{"leisure":"#BBC1C6",...}}` | 小程序端球台状态颜色配置。字段结构与 color_config_json 一致，颜色值不同（适配小程序 UI）。**建议 JSONB 存储** | |
### 1.18 安全与登录配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ------------------------- | -------- | --------------- | ------------------------------------------------------------------------ | ---------- |
| 144 | `can_many_account` | int | 1 | 多账号登录开关（1=允许）。同一用户是否可以在多设备同时登录 | |
| 145 | `single_sign_on` | int | 1 | 单点登录开关（1=启用）。是否启用 SSO | |
| 146 | `default_user_password` | string | "AFD42Zto..." | ⚠️ 敏感字段：用户默认密码（加密后）。ODS 层需脱敏（保留最后 2 字符） | |
| 147 | `needs_confirm` | int | 1 | 操作确认开关（1=需要）。关键操作是否需要二次确认 | |
| 148 | `watermark_enable` | int | 1 | 水印开关（1=启用）。管理端页面是否显示水印（防截图泄露） | |
### 1.19 营销与默认门店配置
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ----- | ----------------------------- | ------ | ------------------ | -------------------------------------------------- | ---------- |
| 149 | `default_marketing_site_id` | int | 2928823574824965 | 默认营销门店 ID。营销活动默认关联的门店 | |
| 150 | `material_id` | int | 0 | 素材 ID。0=未设置，可能关联营销素材库 | |
| 151 | `service_level` | int | 1 | 服务等级。1=基础服务等级（可能影响功能可用范围） | |
---
## 二、tenantProfile 字段分析
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| --- | ---------------- | -------- | ------------------------------------------ | ------------------------------------------------------ | ---------- |
| 1 | `tenantName` | string | "朗朗桌球" | 租户/品牌名称 | Y |
| 2 | `tenantIcon` | string | "https://oss.ficoo.vip/admin/DbtcMZ_..." | 租户/品牌图标 URL | |
| 3 | `prodEnv` | string | "Normal" | 生产环境标识。Normal=正式环境（区别于测试/演示环境） | |
| 4 | `tenantStatus` | int | 0 | 租户状态。0=正常运营（其他值可能表示停用/欠费等） | |
---
## 三、siteProfile 字段分析
> 当 site_id=0 时全部为零值/空值，以下为字段结构说明。实际按门店拉取时会有真实数据。
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | --------------------------- | -------- | ---------- | -------------------------------------- | ---------- |
| 1 | `id` | int | 0 | 门店档案主键 ID | |
| 2 | `org_id` | int | 0 | 组织 ID | |
| 3 | `shop_name` | string | "" | 门店名称 | |
| 4 | `avatar` | string | "" | 门店头像/Logo URL | |
| 5 | `business_tel` | string | "" | 门店联系电话 | |
| 6 | `full_address` | string | "" | 门店完整地址 | |
| 7 | `address` | string | "" | 门店简短地址 | |
| 8 | `longitude` | float | 0.000000 | 经度 | |
| 9 | `latitude` | float | 0.000000 | 纬度 | |
| 10 | `tenant_site_region_id` | int | 0 | 租户门店区域 ID | |
| 11 | `tenant_id` | int | 0 | 租户 ID | |
| 12 | `auto_light` | int | 1 | 自动开灯开关（1=启用） | |
| 13 | `attendance_distance` | int | 0 | 考勤打卡距离限制（米）。0=不限制 | |
| 14 | `wifi_name` | string | "" | 门店 WiFi 名称（用于考勤 WiFi 打卡） | |
| 15 | `wifi_password` | string | "" | 门店 WiFi 密码 | |
| 16 | `customer_service_qrcode` | string | "" | 客服二维码 URL | |
| 17 | `customer_service_wechat` | string | "" | 客服微信号 | |
| 18 | `fixed_pay_qrCode` | string | "" | 固定收款二维码 URL | |
| 19 | `prod_env` | int | 1 | 生产环境标识（1=正式） | |
| 20 | `light_status` | int | 1 | 灯控状态（1=正常） | |
| 21 | `light_type` | int | 0 | 灯控类型。0=默认 | |
| 22 | `site_type` | int | 1 | 门店类型。1=标准门店 | |
| 23 | `light_token` | string | "" | 灯控设备 Token | |
| 24 | `site_label` | string | "" | 门店标签 | |
| 25 | `attendance_enabled` | int | 1 | 门店考勤开关 | |
| 26 | `shop_status` | int | 1 | 门店营业状态（1=营业中） | |
---
## 四、siteUserAppletConfig 字段分析
> 小程序端功能配置。当 site_id=0 时全部为零值/默认值。
| # | 字段名 | 类型 | 示例值 | 业务含义推导 | 是否保留 |
| ---- | ----------------------------------------- | --------- | -------- | ------------------------------------ | ---------- |
| 1 | `id` | int | 0 | 配置记录主键 | |
| 2 | `applet_assistant_order` | int | 2 | 小程序助教下单开关（2=禁用） | |
| 3 | `applet_assistant_reward` | int | 2 | 小程序助教打赏开关（2=禁用） | |
| 4 | `applet_goods_order` | int | 2 | 小程序商品下单开关（2=禁用） | |
| 5 | `applet_goods_outer` | int | 2 | 小程序外卖商品开关（2=禁用） | |
| 6 | `applet_look_order` | int | 2 | 小程序查看订单开关（2=禁用） | |
| 7 | `applet_member_recharge` | int | 2 | 小程序会员充值开关（2=禁用） | |
| 8 | `applet_order_settle` | int | 2 | 小程序订单结算开关（2=禁用） | |
| 9 | `applet_use_package_coupon` | int | 2 | 小程序使用套餐券开关（2=禁用） | |
| 10 | `applet_use_table` | int | 2 | 小程序开台开关（2=禁用） | |
| 11 | `applet_use_table_with_self_coupon` | int | 2 | 小程序使用自有券开台开关（2=禁用） | |
| 12 | `applet_use_table_with_self_coupon_pay` | int | 2 | 小程序自有券开台支付开关（2=禁用） | |
| 13 | `booking_retain_time` | int | 900 | 预约保留时长（秒）。900=15 分钟 | |
| 14 | `cancel_booking_refund` | int | 2 | 取消预约退款开关（2=禁用） | |
| 15 | `delivery_fee` | decimal | 0.00 | 配送费（元） | |
| 16 | `is_delete` | int | 0 | 逻辑删除标记 | |
| 17 | `is_online_reservation` | int | 2 | 在线预约开关（2=禁用） | |
| 18 | `member_recharge_pay_channel` | int | 0 | 会员充值支付渠道。0=默认 | |
| 19 | `minimum_order_amount` | decimal | 0.00 | 最低订单金额（元） | |
| 20 | `pay_channel` | int | 0 | 支付渠道。0=默认 | |
| 21 | `reservation_deposit` | decimal | 0.00 | 预约押金（元） | |
| 22 | `reservation_expiration_time` | int | 0 | 预约过期时间（秒）。0=不过期 | |
| 23 | `site_id` | int | 0 | 门店 ID | |
| 24 | `take_out_delivery` | int | 2 | 外卖配送开关（2=禁用） | |
| 25 | `tenant_id` | int | 0 | 租户 ID | |
| 26 | `site_show_assistant` | int | 2 | 门店展示助教开关（2=不展示） | |
| 27 | `site_show_goods` | int | 2 | 门店展示商品开关（2=不展示） | |
---
## 五、assistantLevels 字段分析
> 当前返回空数组 `[]`。推测结构如下（基于系统中助教等级的已知信息）：
| # | 字段名 | 类型 | 推测含义 |
| --- | ----------------- | --------- | ---------------------------------- |
| 1 | `id` | int | 等级配置主键 |
| 2 | `level` | int | 等级代码（如 10/20/30/40） |
| 3 | `level_name` | string | 等级名称（如"初级""中级""高级"） |
| 4 | `pd_unit_price` | decimal | 陪打单价 |
| 5 | `cx_unit_price` | decimal | 超休单价 |
> 注意：此为推测结构，需要在有数据的门店验证。当前租户级请求返回空数组，可能需要按门店拉取才有数据。
---
## 六、memberCardGradeConfigList 字段分析
> 当前返回空数组 `[]`。推测结构如下（基于系统中会员卡等级的已知信息）：
| # | 字段名 | 类型 | 推测含义 |
| --- | ----------------- | --------- | ---------------- |
| 1 | `id` | int | 等级配置主键 |
| 2 | `grade_name` | string | 等级名称 |
| 3 | `grade_level` | int | 等级序号 |
| 4 | `discount_rate` | decimal | 折扣率 |
| 5 | `growth_value` | int | 升级所需成长值 |
> 注意：此为推测结构，需要在有数据的门店验证。
---
## 七、JSON 字段用途分析
| 字段名 | 所属子对象 | 结构 | 用途 | 存储建议 |
| --------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------- |
| `color_config_json` | tenantConfig | `{"statuses":{"assistant":"#DA5F8E","cluster":"#F39F01","font":"#FFFFFF","leisure":"#2B2B38","reserve":"#2B2B38","start":"#01A39B","warning":"#EC6B04"}}` | PC 端球台状态颜色主题。7 种状态各一个 HEX 颜色值 | JSONB |
| `min_color_json` | tenantConfig | 同上结构，颜色值不同 | 小程序端球台状态颜色主题 | JSONB |
| `member_rule_config_json` | tenantConfig | `{growthValueExpireNum: 12, memberGradeUpType: 1, memberGradeExpireType: 1, memberGradeExpireNum: 12, memberGradeKeepType: 1, memberGradeKeepBalanceType: 1}` | 会员等级规则综合配置：成长值过期周期、等级升级方式、等级过期策略、保级策略。⚠️ 非标准 JSON（键名无引号），解析需容错 | JSONB |
| `sign_config` | tenantConfig | `{}` | 签到规则配置（当前为空，预留） | JSONB |
---
## 八、敏感字段清单
| 字段名 | 所属子对象 | 示例值 | 脱敏方式 |
| --------------------------- | -------------- | ----------------------- | ----------------------------------------------- |
| `default_member_password` | tenantConfig | "123456" | ODS 层保留最后 2 字符 → "****56" |
| `default_user_password` | tenantConfig | "AFD42ZtoGhIn+LYp..." | ODS 层保留最后 2 字符 → "****Q==" |
| `wifi_password` | siteProfile | "" | ODS 层保留最后 2 字符（当前为空，有值时脱敏） |
---
## 九、与现有 ETL 数据的关联分析
| 本接口字段 | 关联的现有数据 | 关联方式 |
| ------------------------------------------------- | ------------------------------------ | ---------------------------------------- |
| `tenant_id` | 所有 ODS/DWD 表的`tenant_id` | 直接匹配 |
| `site_id` | 所有 ODS/DWD 表的`site_id` | 直接匹配 |
| `assistant_global_name` | DWS 层助教相关报表的称谓显示 | 替代硬编码"助教" |
| `assistant_reward_name` | DWS 层`assistant_reward_name` 配置 | 替代硬编码"激励" |
| `point_alias` | 会员积分相关展示 | 替代硬编码"积分" |
| `cutoff_time` | 财务日结时间点 | 影响日结算逻辑的时间窗口 |
| `performance_time` | 绩效计算周期 | 影响 DWS 绩效汇总的时间维度 |
| `color_config_json` / `min_color_json` | 小程序/管理端球台状态颜色 | 前端渲染配置 |
| `member_rule_config_json` | 会员等级升降级逻辑 | 影响会员维度表的等级变更判断 |
| `recharge_card_included` / `gift_card_included` | 营收统计口径 | 影响 DWS 营收汇总是否包含充值卡/赠送卡 |
| `assistant_restrict_count` | 助教排班/派单逻辑 | 单台最大助教数限制 |

2
apps/etl/connectors/feiqiu/docs/business-rules/index_algorithm_cn.md
View File

@@ -23,7 +23,7 @@
- 窗口仅回溯 **近 60 天**（`lookback_days`）。
2) **天数截断**
- 所有"天数差"在参与衰减或间隔计算时，都会被截断到 `<= lookback_days`（默认 60 天）。
- 所有"天数差"在参与衰减或间隔计算时，都会被截断到 `<= lookback_days`（默认 90 天）。
3) **半衰期衰减**
```

6
apps/etl/connectors/feiqiu/docs/etl_tasks/dws_tasks.md
View File

@@ -8,7 +8,7 @@
## 概述
DWS 层共有 19 个已注册任务（含 DWS_MAINTENANCE），按业务域分为六组：
DWS 层共有 21 个已注册任务（含 DWS_MAINTENANCE），按业务域分为六组：
### 助教业绩域（6 个）
@@ -35,7 +35,7 @@ DWS 层共有 19 个已注册任务（含 DWS_MAINTENANCE），按业务域分
| `DWS_ASSISTANT_PROJECT_TAG` | `AssistantProjectTagTask` | `dws_assistant_project_tag` | 助教+时间窗口+项目 | 全量删除重建（按 site_id） |
| `DWS_MEMBER_PROJECT_TAG` | `MemberProjectTagTask` | `dws_member_project_tag` | 会员+时间窗口+项目 | 全量删除重建（按 site_id） |
### 财务统计域（4 个）
### 财务统计域（6 个）
| 任务代码 | Python 类 | 目标表 | 粒度 | 更新策略 |
|----------|-----------|--------|------|----------|
@@ -43,6 +43,8 @@ DWS 层共有 19 个已注册任务（含 DWS_MAINTENANCE），按业务域分
| `DWS_FINANCE_RECHARGE` | `FinanceRechargeTask` | `dws_finance_recharge_summary` | 日期 | delete-before-insert |
| `DWS_FINANCE_INCOME_STRUCTURE` | `FinanceIncomeStructureTask` | `dws_finance_income_structure` | 日期+收入类型 | delete-before-insert |
| `DWS_FINANCE_DISCOUNT_DETAIL` | `FinanceDiscountDetailTask` | `dws_finance_discount_detail` | 日期+折扣类型 | delete-before-insert |
| `DWS_FINANCE_AREA_DAILY` | `FinanceAreaDailyTask` | `dws_finance_area_daily` | 日期+区域 | delete-before-insert |
| `DWS_FINANCE_BOARD_CACHE` | `FinanceBoardCacheTask` | `dws_finance_board_cache` | 时间范围+区域 | upsert（指纹对比） |
### 库存汇总域（3 个）

6
apps/etl/connectors/feiqiu/docs/etl_tasks/index_tasks.md
View File

@@ -878,7 +878,7 @@ ORDER BY effective_from DESC
| 参数名 | 默认值 | 说明 |
|--------|--------|------|
| `lookback_days` | 60 | 服务行为回溯窗口（天） |
| `lookback_days` | 90 | 服务行为回溯窗口（天） |
| `session_merge_hours` | 4 | 会话合并阈值（小时） |
| `incentive_weight` | 1.5 | 激励课权重 |
| `halflife_session` | 14 | 会话半衰期（天） |
@@ -904,7 +904,7 @@ ORDER BY effective_from DESC
| 参数名 | 默认值 | 说明 |
|--------|--------|------|
| `lookback_days` | 60 | 服务行为回溯窗口（天） |
| `lookback_days` | 90 | 服务行为回溯窗口（天） |
| `session_merge_hours` | 4 | 会话合并阈值（小时） |
| `incentive_weight` | 1.5 | 激励课权重 |
| `halflife_short` / `halflife_long` | 7 / 30 | 短期/长期半衰期（天） |
@@ -917,7 +917,7 @@ ORDER BY effective_from DESC
| 参数名 | 默认值 | 说明 |
|--------|--------|------|
| `lookback_days` | 60 | 充值行为回溯窗口（天） |
| `lookback_days` | 90 | 充值行为回溯窗口（天） |
| `amount_base` | 500 | 金额压缩基准 |
| `halflife_recharge` | 21 | 充值半衰期（天） |
| `percentile_lower` / `percentile_upper` | 5 / 95 | 归一化分位点 |

4
apps/etl/connectors/feiqiu/orchestration/flow_runner.py
View File

@@ -422,9 +422,9 @@ class FlowRunner:
verifier_kwargs: dict[str, Any] = {}
if layer.upper() == "INDEX":
try:
lookback_days = int(self.config.get("run.index_lookback_days", 60))
lookback_days = int(self.config.get("run.index_lookback_days", 90))
except (TypeError, ValueError):
lookback_days = 60
lookback_days = 90
verifier_kwargs = {
"lookback_days": lookback_days,
"config": self.config,

29
apps/etl/connectors/feiqiu/orchestration/task_executor.py
View File

@@ -412,7 +412,34 @@ class TaskExecutor:
task_code, self.config, self.db_ops, api_client, self.logger,
)
result = task.execute(None)
# P19: 回测模式 — 从配置中读取 as_of_date 构建 TaskContext
context = None
as_of_date_str = self.config.get("run.as_of_date")
if as_of_date_str:
from zoneinfo import ZoneInfo
tz = ZoneInfo(self.config.get("app.timezone", "Asia/Shanghai"))
# P19: 支持 "YYYY-MM-DD" 和 "YYYY-MM-DD HH:MM:SS" 两种格式
s = str(as_of_date_str).strip()
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M", "%Y-%m-%d"):
try:
as_of_dt = datetime.strptime(s, fmt).replace(tzinfo=tz)
break
except ValueError:
continue
else:
self.logger.warning("as_of_date 格式无法解析: %s，跳过回测模式", s)
as_of_dt = None
if as_of_dt:
from tasks.base_task import TaskContext
context = TaskContext(
store_id=store_id or int(self.config.get("app.store_id") or 0),
window_start=as_of_dt,
window_end=as_of_dt,
window_minutes=0,
as_of_date=as_of_dt,
)
result = task.execute(context)
status = (result.get("status") or "").upper() if isinstance(result, dict) else "SUCCESS"
counts = result.get("counts", {}) if isinstance(result, dict) else {}

25
apps/etl/connectors/feiqiu/orchestration/task_registry.py
View File

@@ -56,6 +56,8 @@ from tasks.dws import (
)
# CHANGE [2026-07-14] intent: 合并 MV 刷新 + 数据清理为 DWS_MAINTENANCE
from tasks.dws.maintenance_task import DwsMaintenanceTask
# CHANGE 2026-03-29 | DWS_TASK_ENGINE：编排后端任务引擎（完成检查→过期检查→任务生成）
from tasks.dws.task_engine import DwsTaskEngineTask
@dataclass
@@ -150,7 +152,7 @@ default_registry.register("DATA_INTEGRITY_CHECK", DataIntegrityTask, requires_db
# ── DWS 层业务任务 ────────────────────────────────────────────
default_registry.register("DWS_BUILD_ORDER_SUMMARY", DwsBuildOrderSummaryTask, requires_db_config=False, layer="DWS")
default_registry.register("DWS_ASSISTANT_DAILY", AssistantDailyTask, layer="DWS")
default_registry.register("DWS_ASSISTANT_ORDER_CONTRIBUTION", AssistantOrderContributionTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
default_registry.register("DWS_ASSISTANT_ORDER_CONTRIBUTION", AssistantOrderContributionTask, layer="DWS")
# CHANGE [2026-07-17] intent: 为已知依赖关系添加 depends_on 声明（需求 8.1, 8.2）
default_registry.register("DWS_ASSISTANT_MONTHLY", AssistantMonthlyTask, layer="DWS", depends_on=["DWS_ASSISTANT_DAILY"])
default_registry.register("DWS_ASSISTANT_CUSTOMER", AssistantCustomerTask, layer="DWS")
@@ -158,17 +160,17 @@ default_registry.register("DWS_ASSISTANT_SALARY", AssistantSalaryTask, layer="DW
default_registry.register("DWS_ASSISTANT_FINANCE", AssistantFinanceTask, layer="DWS", depends_on=["DWS_ASSISTANT_SALARY"])
default_registry.register("DWS_MEMBER_CONSUMPTION", MemberConsumptionTask, layer="DWS")
default_registry.register("DWS_MEMBER_VISIT", MemberVisitTask, layer="DWS")
# CHANGE [2026-03-07] intent: 注册项目标签任务，依赖 DWD 装载完成
default_registry.register("DWS_ASSISTANT_PROJECT_TAG", AssistantProjectTagTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
default_registry.register("DWS_MEMBER_PROJECT_TAG", MemberProjectTagTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
# CHANGE [2026-03-27] intent: 移除对 DWD_LOAD_FROM_ODS 的显式依赖，dwd_dws Flow 下该依赖不在批次内只产生无意义 warning
default_registry.register("DWS_ASSISTANT_PROJECT_TAG", AssistantProjectTagTask, layer="DWS")
default_registry.register("DWS_MEMBER_PROJECT_TAG", MemberProjectTagTask, layer="DWS")
default_registry.register("DWS_FINANCE_DAILY", FinanceDailyTask, layer="DWS")
default_registry.register("DWS_FINANCE_RECHARGE", FinanceRechargeTask, layer="DWS")
default_registry.register("DWS_FINANCE_INCOME_STRUCTURE", FinanceIncomeStructureTask, layer="DWS")
default_registry.register("DWS_FINANCE_DISCOUNT_DETAIL", FinanceDiscountDetailTask, layer="DWS")
# CHANGE [2026-07-20] intent: 注册 DWS 库存汇总任务（日/周/月），依赖 DWD 装载完成（需求 12.9）
default_registry.register("DWS_GOODS_STOCK_DAILY", GoodsStockDailyTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
default_registry.register("DWS_GOODS_STOCK_WEEKLY", GoodsStockWeeklyTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
default_registry.register("DWS_GOODS_STOCK_MONTHLY", GoodsStockMonthlyTask, layer="DWS", depends_on=["DWD_LOAD_FROM_ODS"])
# CHANGE [2026-03-27] intent: 移除对 DWD_LOAD_FROM_ODS 的显式依赖，dwd_dws Flow 下该依赖不在批次内只产生无意义 warning
default_registry.register("DWS_GOODS_STOCK_DAILY", GoodsStockDailyTask, layer="DWS")
default_registry.register("DWS_GOODS_STOCK_WEEKLY", GoodsStockWeeklyTask, layer="DWS")
default_registry.register("DWS_GOODS_STOCK_MONTHLY", GoodsStockMonthlyTask, layer="DWS")
# CHANGE [2026-07-14] intent: 移除 DWS_RETENTION_CLEANUP / DWS_MV_REFRESH_FINANCE_DAILY / DWS_MV_REFRESH_ASSISTANT_DAILY，
# 替换为统一维护任务 DWS_MAINTENANCE（需求 4.5）
# depends_on: 所有其他 DWS 任务——MV 刷新和清理应在数据写入后执行
@@ -191,3 +193,10 @@ default_registry.register("DWS_NEWCONV_INDEX", NewconvIndexTask, requires_db_con
default_registry.register("DWS_ML_MANUAL_IMPORT", MlManualImportTask, requires_db_config=False, layer="INDEX")
default_registry.register("DWS_RELATION_INDEX", RelationIndexTask, requires_db_config=False, layer="INDEX", depends_on=["DWS_ASSISTANT_DAILY"])
default_registry.register("DWS_SPENDING_POWER_INDEX", SpendingPowerIndexTask, requires_db_config=False, layer="INDEX", depends_on=["DWS_MEMBER_CONSUMPTION"])
# ── 任务引擎编排 ─────────────────────────────────────────────
# CHANGE 2026-03-29 | DWS_TASK_ENGINE：DWS 指数计算完成后执行后端任务引擎
# depends_on: 所有指数任务——任务生成依赖 WBI/NCI/RS 指数数据
default_registry.register("DWS_TASK_ENGINE", DwsTaskEngineTask, requires_db_config=False, layer="INDEX", depends_on=[
"DWS_WINBACK_INDEX", "DWS_NEWCONV_INDEX", "DWS_RELATION_INDEX",
])

2
apps/etl/connectors/feiqiu/tasks/base_task.py
View File

@@ -20,6 +20,8 @@ class TaskContext:
window_end: datetime
window_minutes: int
cursor: dict | None = None
# P19: 回测模式 — "假装今天是 X 日"，指数任务用此替代 datetime.now()
as_of_date: datetime | None = None
class BaseTask:

14
apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py
View File

@@ -223,6 +223,10 @@ class DwdLoadTask(BaseTask):
("person_tenant_org_id", "person_tenant_org_id", None),
("person_tenant_org_name", "person_tenant_org_name", None),
("register_source", "register_source", None),
("other_pay_money_sum", "payload->>'otherPayMoneySum'", "numeric"), # CHANGE 2026-03-26: 其他支付金额汇总（从 payload 提取，ODS 列可能为空）
("last_consume_time", "payload->>'lastConsumeTime'", "timestamptz"), # CHANGE 2026-03-26: 最后消费时间
("non_consume_day_num", "payload->>'nonConsumeDayNum'", "integer"), # CHANGE 2026-03-26: 未消费天数
("first_consumption", "payload->>'firstConsumption'", "integer"), # CHANGE 2026-03-26: 是否首次消费（1=是, 2=否）
],
"dwd.dim_member_card_account": [
("member_card_id", "id", None),
@@ -243,6 +247,8 @@ class DwdLoadTask(BaseTask):
("electricity_discount", "electricity_discount", None),
("electricity_card_deduct", "electricitycarddeduct", "boolean"),
("recharge_freeze_balance", "rechargefreezebalance", None),
("pdassisnatlevel", "pdassisnatlevel", None), # CHANGE 2026-03-26: 陪打助教等级限制
("cxassisnatlevel", "cxassisnatlevel", None), # CHANGE 2026-03-26: 促销助教等级限制
],
"dwd.dim_tenant_goods": [
("tenant_goods_id", "id", None),
@@ -352,6 +358,7 @@ class DwdLoadTask(BaseTask):
("table_fee_log_id", "id", None),
("salesman_name", "salesman_name", None),
("order_consumption_type", "order_consumption_type", None),
("order_from", "order_from", None), # CHANGE 2026-03-26: 订单来源
],
"dwd.dwd_table_fee_adjust": [
("table_fee_adjust_id", "id", None),
@@ -389,6 +396,9 @@ class DwdLoadTask(BaseTask):
("legacy_order_goods_id", "ordergoodsid", None),
("site_name", "sitename", None),
("legacy_site_id", "siteid", None),
("activity_amount", "activity_amount", None), # CHANGE 2026-03-26: 活动金额
("activity_id", "activity_id", None), # CHANGE 2026-03-26: 活动 ID
("order_from", "order_from", None), # CHANGE 2026-03-26: 订单来源
],
"dwd.dwd_assistant_service_log": [
("assistant_service_id", "id", None),
@@ -411,6 +421,8 @@ class DwdLoadTask(BaseTask):
("assistant_team_name", "assistantteamname", None),
("operator_id", "operator_id", None), # 操作员 ID
("operator_name", "operator_name", None), # 操作员姓名
("deduct_leave_seconds", "deduct_leave_seconds", None), # CHANGE 2026-03-26: 扣除请假秒数
("order_from", "order_from", None), # CHANGE 2026-03-26: 订单来源（1=线下, 4=小程序）
],
"dwd.dwd_member_balance_change": [
("balance_change_id", "id", None),
@@ -543,6 +555,7 @@ class DwdLoadTask(BaseTask):
("order_remark", "orderremark", None),
("operator_id", "operatorid", None),
("salesman_user_id", "salesmanuserid", None),
("order_from", "orderfrom", None), # CHANGE 2026-03-26: 订单来源（来自 settleList.orderFrom）
# CHANGE: intent=删除 settle_list 映射，该列已从 DWD 表中移除（与 ODS payload 冗余）
# assumptions=结算明细可随时从 ODS payload->'settleList' 按需提取
# prompt=P20260214-040000
@@ -639,6 +652,7 @@ class DwdLoadTask(BaseTask):
("range_inventory", '"rangeinventory"', "numeric"), # 盘点调整量
("current_stock", '"currentstock"', "numeric"), # 当前库存
("site_id", '"siteid"', "bigint"), # 门店 ID（ODS 入库时注入）
("create_time", '"createtime"', "timestamptz"), # CHANGE 2026-03-26: 库存记录创建时间
],
# 库存变动流水：goods_stock_movements（ODS 列名全小写）
# CHANGE 2026-02-21: BUG 10 fix — ODS 列名是小写，不是驼峰

6
apps/etl/connectors/feiqiu/tasks/dws/__init__.py
View File

@@ -26,6 +26,9 @@ from .finance_daily_task import FinanceDailyTask
from .finance_recharge_task import FinanceRechargeTask
from .finance_income_task import FinanceIncomeStructureTask
from .finance_discount_task import FinanceDiscountDetailTask
from .finance_area_daily import FinanceAreaDailyTask
from .finance_board_cache import FinanceBoardCacheTask
from .coach_area_hours_task import CoachAreaHoursTask
from .finance_base_task import FinanceBaseTask
from .maintenance_task import DwsMaintenanceTask
from .goods_stock_daily_task import GoodsStockDailyTask
@@ -63,9 +66,12 @@ __all__ = [
# 财务维度
"FinanceBaseTask",
"FinanceDailyTask",
"FinanceAreaDailyTask",
"FinanceBoardCacheTask",
"FinanceRechargeTask",
"FinanceIncomeStructureTask",
"FinanceDiscountDetailTask",
"CoachAreaHoursTask",
"DwsMaintenanceTask",
# 库存维度
"GoodsStockDailyTask",

37
apps/etl/connectors/feiqiu/tasks/dws/base_dws_task.py
View File

@@ -155,6 +155,11 @@ class BaseDwsTask(BaseTask):
# 子类声明日期列名，用于 extract 时间过滤和 load 幂等删除
# 未声明时 load() 回退到 "stat_date"
DATE_COL: str | None = None
# CHANGE 2026-03-22 | P14: AI 触发集成
# 子类设置此属性以在任务成功后触发 AI 事件
# 值为事件类型字符串（如 "dws_completed" / "consumption"），None 表示不触发
AI_TRIGGER_EVENT: str | None = None
# ==========================================================================
# 抽象方法（子类必须实现）
@@ -239,6 +244,38 @@ class BaseDwsTask(BaseTask):
"extra": {"deleted": deleted},
}
# ==========================================================================
# 主流程覆盖 — AI 触发集成
# ==========================================================================
def execute(self, cursor_data: dict | None = None) -> dict:
"""覆盖 BaseTask.execute()，成功后触发 AI 事件（如已配置）。"""
result = super().execute(cursor_data)
# CHANGE 2026-03-22 | P14: DWS 任务完成后触发 AI 事件
if self.AI_TRIGGER_EVENT and result.get("status") == "SUCCESS":
self._fire_ai_trigger()
return result
def _fire_ai_trigger(self) -> None:
"""发送 AI 触发事件，失败不中断任务流程。"""
try:
from utils.ai_trigger import trigger_ai_event
site_id = self.config.get("app.store_id")
trigger_ai_event(
event_type=self.AI_TRIGGER_EVENT,
site_id=site_id,
payload={"task_code": self.get_task_code()},
)
except Exception:
self.logger.warning(
"%s: AI 触发失败，不影响任务结果",
self.get_task_code(),
exc_info=True,
)
# ==========================================================================
# 时间计算方法
# ==========================================================================

244
apps/etl/connectors/feiqiu/tasks/dws/coach_area_hours_task.py Normal file
View File

@@ -0,0 +1,244 @@
# -*- coding: utf-8 -*-
"""
DWS_COACH_AREA_HOURS — 助教区域课时汇总表
按 (site_id, stat_month, assistant_id, area_code) 粒度聚合助教课时，
为财务看板助教分析板块的区域过滤提供数据支撑。
数据来源：dwd_assistant_service_log + dwd_assistant_service_log_ex + dim_table
更新策略：delete-before-insert（按 site_id + stat_month 删除后插入）
CHANGE 2026-03-29 | Prompt: 助教分析按区域细化 | 新建 ETL 任务
"""
from __future__ import annotations
import logging
from datetime import date
from decimal import Decimal
from typing import Any, Dict, List
from neozqyy_shared.area_mapping import (
ALL_AREA_CODES,
SPECIFIC_AREA_CODES,
resolve_area_code,
)
from tasks.dws.base_dws_task import BaseDwsTask, TaskContext
logger = logging.getLogger(__name__)
_ZERO = Decimal("0")
# 课程类型映射（skill_name → 字段前缀）
_SKILL_MAP = {
"基础课": "base",
"附加课": "bonus",
"包厢课": "room",
}
class CoachAreaHoursTask(BaseDwsTask):
"""助教区域课时汇总 ETL 任务"""
def get_task_code(self) -> str:
return "DWS_COACH_AREA_HOURS"
def get_target_table(self) -> str:
return "dws.dws_coach_area_hours"
def get_primary_keys(self) -> List[str]:
return ["site_id", "stat_month", "assistant_id", "area_code"]
def extract(self, context: TaskContext) -> Dict[str, Any]:
end_date = (
context.window_end.date()
if hasattr(context.window_end, "date")
else context.window_end
)
site_id = context.store_id
# 取当月第一天
stat_month = end_date.replace(day=1)
sql = """
SELECT
s.site_assistant_id AS assistant_id,
dt.site_table_area_name AS area_name,
s.skill_name,
s.income_seconds,
COALESCE(ex.is_trash, 0) AS is_trash
FROM dwd.dwd_assistant_service_log s
LEFT JOIN dwd.dwd_assistant_service_log_ex ex
ON s.assistant_service_id = ex.assistant_service_id
LEFT JOIN dwd.dim_table dt
ON s.site_table_id = dt.table_id
AND dt.scd2_is_current = 1
WHERE s.site_id = %s
AND s.is_delete = 0
AND DATE(s.start_use_time) >= %s
AND DATE(s.start_use_time) < %s
"""
# 下月第一天
if stat_month.month == 12:
next_month = stat_month.replace(year=stat_month.year + 1, month=1)
else:
next_month = stat_month.replace(month=stat_month.month + 1)
rows = self.db.query(sql, (site_id, stat_month, next_month))
records = [dict(r) for r in rows] if rows else []
self.logger.info(
"%s: 提取 %d 条服务记录，月份 %s",
self.get_task_code(), len(records), stat_month,
)
return {
"records": records,
"stat_month": stat_month,
"site_id": site_id,
"tenant_id": self.config.get("app.tenant_id", site_id),
}
def transform(
self, extracted: Dict[str, Any], context: TaskContext
) -> List[Dict[str, Any]]:
return transform_coach_area_hours(
records=extracted["records"],
stat_month=extracted["stat_month"],
site_id=extracted["site_id"],
tenant_id=extracted["tenant_id"],
)
def load(
self, transformed: List[Dict[str, Any]], context: TaskContext
) -> dict:
if not transformed:
return {"inserted": 0}
site_id = transformed[0]["site_id"]
stat_month = transformed[0]["stat_month"]
# delete-before-insert
self.db.execute(
"DELETE FROM dws.dws_coach_area_hours WHERE site_id = %s AND stat_month = %s",
(site_id, stat_month),
)
cols = [
"site_id", "tenant_id", "stat_month", "assistant_id", "area_code",
"base_hours", "bonus_hours", "room_hours",
"effective_hours", "trashed_hours",
"base_service_count", "bonus_service_count", "room_service_count",
]
placeholders = ", ".join(["%s"] * len(cols))
insert_sql = f"INSERT INTO dws.dws_coach_area_hours ({', '.join(cols)}) VALUES ({placeholders})"
for row in transformed:
self.db.execute(insert_sql, tuple(row[c] for c in cols))
self.logger.info(
"%s: 写入 %d 行（site_id=%s, month=%s）",
self.get_task_code(), len(transformed), site_id, stat_month,
)
return {"inserted": len(transformed)}
# ── 纯函数（可被回填脚本和属性测试直接调用） ─────────────────────────
def transform_coach_area_hours(
records: List[Dict[str, Any]],
stat_month: date,
site_id: int,
tenant_id: int,
) -> List[Dict[str, Any]]:
"""
将 DWD 服务记录按 (assistant_id, area_code) 聚合为区域课时。
返回列表，每个元素对应一行 dws_coach_area_hours。
包含 9 个 area_code（hallA~ktv + hall + all）× N 个助教。
"""
# 按 (assistant_id, area_code) 聚合
# key: (assistant_id, area_code) → {base_hours, bonus_hours, ...}
agg: Dict[tuple, Dict[str, Any]] = {}
for rec in records:
assistant_id = rec["assistant_id"]
area_name = rec.get("area_name")
skill_name = rec.get("skill_name")
income_seconds = rec.get("income_seconds") or 0
is_trash = rec.get("is_trash", 0)
hours = Decimal(str(income_seconds)) / Decimal("3600")
skill_prefix = _SKILL_MAP.get(skill_name) # base/bonus/room/None
# 解析区域
area_code = resolve_area_code(area_name) if area_name else None
if area_code is None:
# 未知区域或 NULL table_id → 不计入具体区域，只计入 all
target_areas = []
else:
target_areas = [area_code]
# 更新具体区域
for ac in target_areas:
_accumulate(agg, assistant_id, ac, skill_prefix, hours, is_trash)
# CHANGE 2026-03-29 | hall = 台球大厅（hallA+hallB+hallC），不含 vip/snooker/mahjong/ktv
if area_code in ("hallA", "hallB", "hallC"):
_accumulate(agg, assistant_id, "hall", skill_prefix, hours, is_trash)
# all 始终累加
_accumulate(agg, assistant_id, "all", skill_prefix, hours, is_trash)
# 构建输出行
result = []
for (aid, ac), data in sorted(agg.items()):
total_hours = data["base_hours"] + data["bonus_hours"] + data["room_hours"]
effective = total_hours - data["trashed_hours"]
result.append({
"site_id": site_id,
"tenant_id": tenant_id,
"stat_month": stat_month,
"assistant_id": aid,
"area_code": ac,
"base_hours": data["base_hours"],
"bonus_hours": data["bonus_hours"],
"room_hours": data["room_hours"],
"effective_hours": max(effective, _ZERO),
"trashed_hours": data["trashed_hours"],
"base_service_count": data["base_service_count"],
"bonus_service_count": data["bonus_service_count"],
"room_service_count": data["room_service_count"],
})
return result
def _accumulate(
agg: Dict[tuple, Dict[str, Any]],
assistant_id: int,
area_code: str,
skill_prefix: str | None,
hours: Decimal,
is_trash: int,
) -> None:
"""累加单条服务记录到聚合字典"""
key = (assistant_id, area_code)
if key not in agg:
agg[key] = {
"base_hours": _ZERO,
"bonus_hours": _ZERO,
"room_hours": _ZERO,
"trashed_hours": _ZERO,
"base_service_count": 0,
"bonus_service_count": 0,
"room_service_count": 0,
}
bucket = agg[key]
if is_trash == 1:
bucket["trashed_hours"] += hours
return
if skill_prefix:
bucket[f"{skill_prefix}_hours"] += hours
bucket[f"{skill_prefix}_service_count"] += 1
# skill_name 为 NULL 的记录：不计入任何课程类型的 hours，但已计入 hall/all

586
apps/etl/connectors/feiqiu/tasks/dws/finance_area_daily.py Normal file
View File

@@ -0,0 +1,586 @@
# -*- coding: utf-8 -*-
"""
财务区域日粒度汇总任务
功能说明：
以 (site_id, stat_date, area_code) 为粒度，按桌台区域聚合当日财务数据。
输出 9 行：7 个具体区域 (hallA~ktv) + hall（历史兼容）+ all。
数据来源：
- dwd_settlement_head + dim_table (scd2_is_current=1)：结算单 + 桌台区域
- dws_finance_daily_summary：全局现金流/充值/卡消费（仅 all 行使用）
目标表：
dws.dws_finance_area_daily
更新策略：
- 更新频率：每小时更新当日数据
- 幂等方式：delete-before-insert（按 site_id + stat_date 删除后插入 9 行）
业务规则：
- 收入恒等式：gross_amount = table_fee + goods + assistant_pd + assistant_cx
- 优惠恒等式：discount_total = groupbuy + vip + manual + gift_card + rounding + other
- 确认收入：confirmed_income = gross_amount - discount_total
- 非 all 行现金流/卡消费/充值字段 = 0
- hall 行 = 各具体区域之和（历史兼容）
- all 行 = 各具体区域之和（收入/优惠），现金流/充值/卡消费来自 dws_finance_daily_summary
- settle_type IN (1, 3) 过滤
- discount_gift_card 使用赠送卡消费金额口径
Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 8.1, 8.2
"""
from __future__ import annotations
import logging
from collections import defaultdict
from datetime import date
from decimal import Decimal
from typing import Any, Dict, List
from neozqyy_shared.area_mapping import (
ALL_AREA_CODES,
SPECIFIC_AREA_CODES,
resolve_area_code,
)
from neozqyy_shared.datetime_utils import biz_date_sql_expr
from .base_dws_task import TaskContext
from .finance_base_task import FinanceBaseTask
logger = logging.getLogger(__name__)
# 收入字段（按区域聚合）
_REVENUE_FIELDS = [
"table_fee_amount",
"goods_amount",
"assistant_pd_amount",
"assistant_cx_amount",
"gross_amount",
]
# 优惠字段（按区域聚合）
_DISCOUNT_FIELDS = [
"discount_groupbuy",
"discount_vip",
"discount_manual",
"discount_gift_card",
"discount_rounding",
"discount_other",
"discount_total",
]
# 现金流字段（仅 all 行有值）
_CASHFLOW_FIELDS = [
"cash_pay_amount",
"cash_paper_amount",
"scan_pay_amount",
"groupbuy_pay_amount",
"recharge_cash_inflow",
"cash_inflow_total",
"cash_outflow_total",
"cash_balance_change",
]
# 卡消费字段（仅 all 行有值）
_CARD_FIELDS = [
"card_consume_total",
"recharge_card_consume",
"gift_card_consume",
]
# 充值字段（仅 all 行有值）
_RECHARGE_FIELDS = [
"recharge_cash",
"first_recharge_cash",
"renewal_cash",
]
# 所有仅 all 行有值的字段
_ALL_ONLY_FIELDS = _CASHFLOW_FIELDS + _CARD_FIELDS + _RECHARGE_FIELDS
# 按区域聚合的字段（收入 + 优惠 + order_count）
_AREA_AGG_FIELDS = _REVENUE_FIELDS + _DISCOUNT_FIELDS + ["confirmed_income", "order_count"]
_ZERO = Decimal("0")
class FinanceAreaDailyTask(FinanceBaseTask):
"""
财务区域日粒度汇总任务
按 (site_id, stat_date, area_code) 粒度预计算收入、优惠、现金流等数据。
每次写入 9 行（hallA~ktv + hall + all）。
"""
DATE_COL = "stat_date"
def get_task_code(self) -> str:
return "DWS_FINANCE_AREA_DAILY"
def get_target_table(self) -> str:
return "dws.dws_finance_area_daily"
def get_primary_keys(self) -> List[str]:
return ["site_id", "stat_date", "area_code"]
# ======================================================================
# Extract
# ======================================================================
def extract(self, context: TaskContext) -> Dict[str, Any]:
"""提取结算单（含区域）和全局现金流/充值/卡消费数据。"""
start_date = (
context.window_start.date()
if hasattr(context.window_start, "date")
else context.window_start
)
end_date = (
context.window_end.date()
if hasattr(context.window_end, "date")
else context.window_end
)
site_id = context.store_id
self.logger.info(
"%s: 提取数据，日期范围 %s ~ %s",
self.get_task_code(),
start_date,
end_date,
)
# 1. 结算单 + dim_table 区域映射
settlement_rows = self._extract_settlement_with_area(
site_id, start_date, end_date
)
# 2. 全局现金流/充值/卡消费（从现有 dws_finance_daily_summary）
global_summary = self._extract_global_summary(site_id, start_date, end_date)
return {
"settlement_rows": settlement_rows,
"global_summary": global_summary,
"start_date": start_date,
"end_date": end_date,
"site_id": site_id,
}
def _extract_settlement_with_area(
self, site_id: int, start_date: date, end_date: date
) -> List[Dict[str, Any]]:
"""从 dwd_settlement_head + dim_table 提取结算单（含区域名称）。"""
cutoff = self.config.get("app.business_day_start_hour", 8)
biz_expr = biz_date_sql_expr("sh.pay_time", cutoff)
sql = f"""
SELECT
{biz_expr} AS stat_date,
dt.site_table_area_name AS area_name,
sh.settle_type,
-- 收入
sh.table_charge_money AS table_fee_amount,
sh.goods_money AS goods_amount,
sh.assistant_pd_money AS assistant_pd_amount,
sh.assistant_cx_money AS assistant_cx_amount,
(sh.table_charge_money + sh.goods_money
+ sh.assistant_pd_money + sh.assistant_cx_money)
AS gross_amount,
-- 优惠原始字段
sh.coupon_amount,
sh.pl_coupon_sale_amount,
sh.adjust_amount,
sh.member_discount_amount,
sh.rounding_amount,
sh.gift_card_amount
FROM dwd.dwd_settlement_head sh
LEFT JOIN dwd.dim_table dt
ON dt.table_id = sh.table_id
AND dt.site_id = sh.site_id
AND dt.scd2_is_current = 1
WHERE sh.site_id = %s
AND {biz_expr} >= %s
AND {biz_expr} <= %s
AND sh.settle_type IN (1, 3)
"""
rows = self.db.query(sql, (site_id, start_date, end_date))
return [dict(row) for row in rows] if rows else []
def _extract_global_summary(
self, site_id: int, start_date: date, end_date: date
) -> List[Dict[str, Any]]:
"""从 dws_finance_daily_summary 提取全局现金流/充值/卡消费字段。"""
sql = """
SELECT
stat_date,
cash_pay_amount,
cash_paper_amount,
scan_pay_amount,
groupbuy_pay_amount,
recharge_cash_inflow,
cash_inflow_total,
cash_outflow_total,
cash_balance_change,
card_consume_total,
recharge_card_consume,
gift_card_consume,
recharge_cash,
first_recharge_amount AS first_recharge_cash,
renewal_amount AS renewal_cash
FROM dws.dws_finance_daily_summary
WHERE site_id = %s
AND stat_date >= %s
AND stat_date <= %s
"""
rows = self.db.query(sql, (site_id, start_date, end_date))
return [dict(row) for row in rows] if rows else []
# ======================================================================
# Transform（纯函数，不依赖数据库）
# ======================================================================
def transform(
self, extracted: Dict[str, Any], context: TaskContext
) -> List[Dict[str, Any]]:
"""按区域聚合收入和优惠，构建 9 行输出。"""
settlement_rows = extracted["settlement_rows"]
global_summary = extracted["global_summary"]
site_id = extracted["site_id"]
return transform_area_daily(
settlement_rows=settlement_rows,
global_summary=global_summary,
site_id=site_id,
tenant_id=self.config.get("app.tenant_id", site_id),
safe_decimal_fn=self.safe_decimal,
logger=self.logger,
)
# ======================================================================
# Load（覆盖基类：按 site_id + stat_date 删除，而非仅按 DATE_COL）
# ======================================================================
def load(self, transformed: List[Dict[str, Any]], context: TaskContext) -> dict:
"""delete-before-insert：按 site_id + stat_date 删除后插入 9 行。"""
if not transformed:
return {
"counts": {
"fetched": 0,
"inserted": 0,
"updated": 0,
"skipped": 0,
"errors": 0,
}
}
# 目标表已含 schema 前缀
target_table = self.get_target_table()
start_date = (
context.window_start.date()
if hasattr(context.window_start, "date")
else context.window_start
)
end_date = (
context.window_end.date()
if hasattr(context.window_end, "date")
else context.window_end
)
delete_sql = f"""
DELETE FROM {target_table}
WHERE site_id = %s
AND stat_date >= %s
AND stat_date <= %s
"""
columns = list(transformed[0].keys())
cols_str = ", ".join(columns)
placeholders = ", ".join(["%s"] * len(columns))
insert_sql = f"INSERT INTO {target_table} ({cols_str}) VALUES ({placeholders})"
with self.db.conn.cursor() as cur:
cur.execute(delete_sql, (context.store_id, start_date, end_date))
deleted = cur.rowcount
inserted = 0
for row in transformed:
values = [row.get(col) for col in columns]
cur.execute(insert_sql, values)
inserted += cur.rowcount
self.logger.info(
"%s: 删除 %d 行，插入 %d 行",
self.get_task_code(),
deleted,
inserted,
)
return {
"counts": {
"fetched": len(transformed),
"inserted": inserted,
"updated": 0,
"skipped": 0,
"errors": 0,
},
"extra": {"deleted": deleted},
}
# ======================================================================
# 纯函数：transform 核心逻辑（方便属性测试直接调用）
# ======================================================================
def transform_area_daily(
settlement_rows: List[Dict[str, Any]],
global_summary: List[Dict[str, Any]],
site_id: int,
tenant_id: int,
safe_decimal_fn=None,
logger=None,
) -> List[Dict[str, Any]]:
"""按区域聚合结算单，构建 9 行输出。
这是一个纯函数（不依赖数据库），方便属性测试直接调用。
Args:
settlement_rows: 结算单行（含 area_name、settle_type 等）
global_summary: 全局现金流/充值/卡消费（按 stat_date 索引）
site_id: 门店 ID
tenant_id: 租户 ID
safe_decimal_fn: Decimal 安全转换函数（默认使用内置实现）
logger: 日志记录器（可选）
Returns:
9 × N 行字典列表（N = 涉及的日期数）
"""
if safe_decimal_fn is None:
safe_decimal_fn = _safe_decimal
if logger is None:
logger = logging.getLogger(__name__)
# 全局汇总按日期索引
global_index: Dict[date, Dict[str, Any]] = {}
for row in global_summary:
sd = row.get("stat_date")
if sd is not None:
global_index[sd] = row
# ── 第一步：按 (stat_date, area_code) 聚合结算单 ──────────────
# area_agg[stat_date][area_code] = {field: Decimal, ...}
area_agg: Dict[date, Dict[str, Dict[str, Decimal]]] = defaultdict(
lambda: defaultdict(lambda: defaultdict(lambda: _ZERO))
)
# 收集所有涉及的日期
all_dates: set[date] = set()
for row in settlement_rows:
sd = row.get("stat_date")
if sd is None:
continue
all_dates.add(sd)
# settle_type 过滤（extract 已过滤，transform 层再次防御）
settle_type = row.get("settle_type")
if settle_type not in (1, 3):
continue
area_name = row.get("area_name")
area_code = resolve_area_code(area_name)
if area_code is None:
# 未知区域：记录警告，不计入具体区域行，但仍计入 all 行
logger.warning(
"DWS_FINANCE_AREA_DAILY: 未知区域名称 '%s'，不计入具体区域",
area_name,
)
# 提取金额
table_fee = safe_decimal_fn(row.get("table_fee_amount", 0))
goods = safe_decimal_fn(row.get("goods_amount", 0))
pd_amount = safe_decimal_fn(row.get("assistant_pd_amount", 0))
cx_amount = safe_decimal_fn(row.get("assistant_cx_amount", 0))
gross = table_fee + goods + pd_amount + cx_amount
coupon = safe_decimal_fn(row.get("coupon_amount", 0))
pl_coupon_sale = safe_decimal_fn(row.get("pl_coupon_sale_amount", 0))
adjust = safe_decimal_fn(row.get("adjust_amount", 0))
member_discount = safe_decimal_fn(row.get("member_discount_amount", 0))
rounding = safe_decimal_fn(row.get("rounding_amount", 0))
gift_card_amount = safe_decimal_fn(row.get("gift_card_amount", 0))
# 团购优惠 = coupon_amount - 团购支付金额
groupbuy_pay = pl_coupon_sale if pl_coupon_sale > 0 else _ZERO
discount_groupbuy = max(coupon - groupbuy_pay, _ZERO)
# 赠送卡消费金额口径（与现有 ETL 一致）
discount_gift_card = gift_card_amount
# 手动调整 = adjust_amount（简化：不拆分大客户，区域级无法区分）
discount_manual = adjust
discount_total = (
discount_groupbuy
+ member_discount
+ discount_manual
+ discount_gift_card
+ rounding
)
# discount_other 在区域级暂为 0（adjust 全部归入 manual）
discount_other = _ZERO
# 重新计算 discount_total 确保恒等式
discount_total = (
discount_groupbuy
+ member_discount
+ discount_manual
+ discount_gift_card
+ rounding
+ discount_other
)
confirmed_income = gross - discount_total
fields = {
"table_fee_amount": table_fee,
"goods_amount": goods,
"assistant_pd_amount": pd_amount,
"assistant_cx_amount": cx_amount,
"gross_amount": gross,
"discount_groupbuy": discount_groupbuy,
"discount_vip": member_discount,
"discount_manual": discount_manual,
"discount_gift_card": discount_gift_card,
"discount_rounding": rounding,
"discount_other": discount_other,
"discount_total": discount_total,
"confirmed_income": confirmed_income,
"order_count": 1,
}
# 累加到具体区域
if area_code is not None:
bucket = area_agg[sd][area_code]
for k, v in fields.items():
bucket[k] = bucket[k] + v
# 也收集 global_summary 中的日期
for sd in global_index:
all_dates.add(sd)
# ── 第二步：构建 9 行输出 ─────────────────────────────────────
results: List[Dict[str, Any]] = []
for sd in sorted(all_dates):
day_agg = area_agg.get(sd, {})
# 计算各具体区域行
specific_rows: Dict[str, Dict[str, Any]] = {}
for ac in SPECIFIC_AREA_CODES:
bucket = day_agg.get(ac, {})
row_data = _build_area_row(
site_id=site_id,
tenant_id=tenant_id,
stat_date=sd,
area_code=ac,
agg=bucket,
)
specific_rows[ac] = row_data
# CHANGE 2026-03-29 | hall = 台球大厅（hallA+hallB+hallC），不含 vip/snooker/mahjong/ktv
_HALL_CODES = ("hallA", "hallB", "hallC")
hall_source = [specific_rows[ac] for ac in _HALL_CODES if ac in specific_rows]
hall_row = _build_sum_row(
site_id=site_id,
tenant_id=tenant_id,
stat_date=sd,
area_code="hall",
source_rows=hall_source,
)
# all 行 = 各具体区域之和（收入/优惠/order_count）+ 全局现金流/充值/卡消费
all_row = _build_sum_row(
site_id=site_id,
tenant_id=tenant_id,
stat_date=sd,
area_code="all",
source_rows=list(specific_rows.values()),
)
# 填充全局现金流/充值/卡消费
gs = global_index.get(sd, {})
for field in _ALL_ONLY_FIELDS:
all_row[field] = safe_decimal_fn(gs.get(field, 0))
# 输出顺序：all, hall, hallA, hallB, hallC, vip, snooker, mahjong, ktv
results.append(all_row)
results.append(hall_row)
for ac in SPECIFIC_AREA_CODES:
results.append(specific_rows[ac])
return results
def _build_area_row(
site_id: int,
tenant_id: int,
stat_date: date,
area_code: str,
agg: Dict[str, Decimal],
) -> Dict[str, Any]:
"""构建单个区域行。"""
row: Dict[str, Any] = {
"site_id": site_id,
"tenant_id": tenant_id,
"stat_date": stat_date,
"area_code": area_code,
}
# 收入 + 优惠 + confirmed_income + order_count
for field in _AREA_AGG_FIELDS:
if field == "order_count":
row[field] = int(agg.get(field, 0))
else:
row[field] = agg.get(field, _ZERO)
# 非 all 行：现金流/卡消费/充值 = 0
for field in _ALL_ONLY_FIELDS:
row[field] = _ZERO
return row
def _build_sum_row(
site_id: int,
tenant_id: int,
stat_date: date,
area_code: str,
source_rows: List[Dict[str, Any]],
) -> Dict[str, Any]:
"""构建 hall/all 汇总行（收入/优惠 = 各具体区域之和）。"""
row: Dict[str, Any] = {
"site_id": site_id,
"tenant_id": tenant_id,
"stat_date": stat_date,
"area_code": area_code,
}
for field in _AREA_AGG_FIELDS:
if field == "order_count":
row[field] = sum(int(r.get(field, 0)) for r in source_rows)
else:
row[field] = sum(
(r.get(field, _ZERO) for r in source_rows), _ZERO
)
# 非 all 行：现金流/卡消费/充值 = 0（all 行由调用方覆盖）
for field in _ALL_ONLY_FIELDS:
row[field] = _ZERO
return row
def _safe_decimal(value: Any, default: Decimal = _ZERO) -> Decimal:
"""内置的安全 Decimal 转换（供纯函数使用）。"""
if value is None:
return default
try:
return Decimal(str(value))
except Exception:
return default
__all__ = ["FinanceAreaDailyTask", "transform_area_daily"]

29
apps/etl/connectors/feiqiu/tasks/dws/finance_base_task.py
View File

@@ -118,6 +118,35 @@ class FinanceBaseTask(BaseDwsTask):
rows = self.db.query(sql, (site_id, start_date, end_date))
return [dict(row) for row in rows] if rows else []
# ------------------------------------------------------------------
# 支付方式拆分
# ------------------------------------------------------------------
# CHANGE 2026-03-27 | board-finance-integration T1.1 | 新增支付方式拆分
def _extract_payment_split(
self,
site_id: int,
start_date: date,
end_date: date,
) -> List[Dict[str, Any]]:
"""支付方式拆分日汇总（纸币现金 vs 扫码/离线支付）"""
cutoff = self.config.get("app.business_day_start_hour", 8)
biz_expr = biz_date_sql_expr("p.pay_time", cutoff)
sql = f"""
SELECT
{biz_expr} AS stat_date,
SUM(CASE WHEN p.payment_method = 2 THEN p.pay_amount ELSE 0 END) AS cash_paper_amount,
SUM(CASE WHEN p.payment_method = 4 THEN p.pay_amount ELSE 0 END) AS scan_pay_amount
FROM dwd.dwd_payment p
WHERE p.site_id = %s
AND p.pay_status = 2
AND p.relate_type = 2
AND {biz_expr} >= %s
AND {biz_expr} <= %s
GROUP BY {biz_expr}
"""
rows = self.db.query(sql, (site_id, start_date, end_date))
return [dict(row) for row in rows] if rows else []
# ------------------------------------------------------------------
# 团购核销汇总
# ------------------------------------------------------------------

525
apps/etl/connectors/feiqiu/tasks/dws/finance_board_cache.py Normal file
View File

@@ -0,0 +1,525 @@
# -*- coding: utf-8 -*-
"""
财务看板缓存层任务
功能说明：
基于 dws_finance_area_daily 日粒度数据，为已完成周期（lastMonth/lastWeek/
lastQuarter/quarter3/half6）× 9 个区域 = 45 组合维护缓存。
通过数据指纹（MD5）检测源数据变化，仅对指纹不一致的组合重算并 upsert。
数据来源：
- dws.dws_finance_area_daily：日粒度原子层
- dws.dws_finance_board_cache：缓存层（读取已有指纹做对比）
目标表：
dws.dws_finance_board_cache
更新策略：
- 更新频率：每天一次（营业日切点后）
- 幂等方式：ON CONFLICT (site_id, time_range, area_code) DO UPDATE
- 当期周期（month/week/quarter）不写入缓存
业务规则：
- 指纹 = MD5(sorted [(stat_date, gross_amount, discount_total), ...])
- 指纹一致 → 跳过；指纹不一致 → 从日粒度表 SUM 重算后 upsert
- overview 8 项：occurrence/discount/discount_rate/confirmed_revenue/
cash_in/cash_out/cash_balance/balance_rate
Requirements: 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7
"""
from __future__ import annotations
import hashlib
import json
import logging
from datetime import date, timedelta
from decimal import Decimal
from typing import Any, Dict, List, Optional, Tuple
from neozqyy_shared.area_mapping import ALL_AREA_CODES
from .base_dws_task import BaseDwsTask, TaskContext
logger = logging.getLogger(__name__)
# 已完成周期（当期 month/week/quarter 不缓存）
COMPLETED_PERIODS: List[str] = [
"lastMonth",
"lastWeek",
"lastQuarter",
"quarter3",
"half6",
]
# 当期周期（不写入缓存）
CURRENT_PERIODS: set[str] = {"month", "week", "quarter"}
_ZERO = Decimal("0")
# ======================================================================
# 纯函数：指纹计算（导出供属性测试直接调用）
# ======================================================================
def compute_fingerprint(rows: list[dict]) -> str:
"""对 (stat_date, gross_amount, discount_total) 排序后计算 MD5 指纹。
这是一个纯函数，不依赖数据库，方便属性测试直接调用。
Args:
rows: 日粒度行列表，每行至少包含 stat_date/gross_amount/discount_total
Returns:
MD5 十六进制字符串
"""
sorted_rows = sorted(rows, key=lambda r: str(r["stat_date"]))
payload = json.dumps(
[
(str(r["stat_date"]), str(r["gross_amount"]), str(r["discount_total"]))
for r in sorted_rows
]
)
return hashlib.md5(payload.encode()).hexdigest()
def is_current_period(time_range: str) -> bool:
"""判断 time_range 是否为当期周期（不应写入缓存）。
纯函数，导出供属性测试直接调用。
"""
return time_range in CURRENT_PERIODS
# ======================================================================
# 日期范围计算（复用 board_service 的逻辑）
# ======================================================================
def _calc_period_date_range(
time_range: str, ref_date: date | None = None
) -> Tuple[date, date]:
"""计算已完成周期的日期范围。
与 board_service._calc_date_range 保持一致。
"""
today = ref_date or date.today()
if time_range == "lastMonth":
first_of_this_month = today.replace(day=1)
last_month_end = first_of_this_month - timedelta(days=1)
last_month_start = last_month_end.replace(day=1)
return last_month_start, last_month_end
if time_range == "lastWeek":
this_monday = today - timedelta(days=today.weekday())
last_sunday = this_monday - timedelta(days=1)
last_monday = last_sunday - timedelta(days=6)
return last_monday, last_sunday
if time_range == "lastQuarter":
q_start_month = (today.month - 1) // 3 * 3 + 1
this_q_start = date(today.year, q_start_month, 1)
prev_q_end = this_q_start - timedelta(days=1)
prev_q_start_month = (prev_q_end.month - 1) // 3 * 3 + 1
prev_q_start = date(prev_q_end.year, prev_q_start_month, 1)
return prev_q_start, prev_q_end
if time_range == "quarter3":
first_of_this_month = today.replace(day=1)
end = first_of_this_month - timedelta(days=1)
start = _month_offset(first_of_this_month, -3)
return start, end
if time_range == "half6":
first_of_this_month = today.replace(day=1)
end = first_of_this_month - timedelta(days=1)
start = _month_offset(first_of_this_month, -6)
return start, end
raise ValueError(f"不支持的已完成周期: {time_range}")
def _month_offset(base_date: date, months: int) -> date:
"""按月偏移日期（保持日不越界）。"""
import calendar
total_months = base_date.year * 12 + (base_date.month - 1) + months
year = total_months // 12
month = total_months % 12 + 1
last_day = calendar.monthrange(year, month)[1]
day = min(base_date.day, last_day)
return date(year, month, day)
# ======================================================================
# ETL 任务类
# ======================================================================
class FinanceBoardCacheTask(BaseDwsTask):
"""
财务看板缓存层任务
遍历 5 个已完成周期 × 9 个区域 = 45 组合，
通过数据指纹检测变化，仅对需要重算的组合 upsert 缓存。
"""
DATE_COL = None # 缓存表无 stat_date 列
def get_task_code(self) -> str:
return "DWS_FINANCE_BOARD_CACHE"
def get_target_table(self) -> str:
return "dws.dws_finance_board_cache"
def get_primary_keys(self) -> List[str]:
return ["site_id", "time_range", "area_code"]
# ======================================================================
# Extract
# ======================================================================
def extract(self, context: TaskContext) -> Dict[str, Any]:
"""遍历 45 组合，从日粒度表读取源数据行 + 从缓存表读取已有指纹。"""
site_id = context.store_id
self.logger.info(
"%s: 开始提取，站点 %s",
self.get_task_code(),
site_id,
)
combinations: List[Dict[str, Any]] = []
for time_range in COMPLETED_PERIODS:
start_date, end_date = _calc_period_date_range(time_range)
for area_code in ALL_AREA_CODES:
# 从日粒度表读取源数据行
daily_rows = self._read_daily_rows(
site_id, start_date, end_date, area_code
)
# 从缓存表读取已有指纹
existing_fp = self._read_existing_fingerprint(
site_id, time_range, area_code
)
combinations.append(
{
"time_range": time_range,
"area_code": area_code,
"start_date": start_date,
"end_date": end_date,
"daily_rows": daily_rows,
"existing_fingerprint": existing_fp,
}
)
return {
"combinations": combinations,
"site_id": site_id,
}
def _read_daily_rows(
self,
site_id: int,
start_date: date,
end_date: date,
area_code: str,
) -> List[Dict[str, Any]]:
"""从 dws_finance_area_daily 读取日粒度行。"""
sql = """
SELECT stat_date, gross_amount, discount_total,
confirmed_income,
cash_inflow_total, cash_outflow_total, cash_balance_change
FROM dws.dws_finance_area_daily
WHERE site_id = %s
AND stat_date >= %s
AND stat_date <= %s
AND area_code = %s
"""
rows = self.db.query(sql, (site_id, start_date, end_date, area_code))
return [dict(row) for row in rows] if rows else []
def _read_existing_fingerprint(
self,
site_id: int,
time_range: str,
area_code: str,
) -> Optional[str]:
"""从缓存表读取已有指纹。"""
sql = """
SELECT data_fingerprint
FROM dws.dws_finance_board_cache
WHERE site_id = %s
AND time_range = %s
AND area_code = %s
"""
rows = self.db.query(sql, (site_id, time_range, area_code))
if rows:
return dict(rows[0]).get("data_fingerprint")
return None
# ======================================================================
# Transform（纯函数逻辑，标记需重算的组合）
# ======================================================================
def transform(
self, extracted: Dict[str, Any], context: TaskContext
) -> List[Dict[str, Any]]:
"""计算指纹，与已有指纹对比，标记需重算的组合。"""
site_id = extracted["site_id"]
combinations = extracted["combinations"]
return transform_cache_combinations(
combinations=combinations,
site_id=site_id,
logger=self.logger,
)
# ======================================================================
# Load（upsert 需重算的组合）
# ======================================================================
def load(self, transformed: List[Dict[str, Any]], context: TaskContext) -> dict:
"""对需重算的组合 upsert 到缓存表。"""
if not transformed:
return {
"counts": {
"fetched": 0,
"inserted": 0,
"updated": 0,
"skipped": 0,
"errors": 0,
}
}
site_id = context.store_id
upserted = 0
skipped = 0
for combo in transformed:
if not combo.get("needs_recompute"):
skipped += 1
continue
# 从日粒度表 SUM 计算 overview 指标
overview = self._sum_overview(
site_id,
combo["start_date"],
combo["end_date"],
combo["area_code"],
)
self._upsert_cache(
site_id=site_id,
time_range=combo["time_range"],
area_code=combo["area_code"],
start_date=combo["start_date"],
end_date=combo["end_date"],
overview=overview,
fingerprint=combo["new_fingerprint"],
)
upserted += 1
self.logger.info(
"%s: upsert %d 组合，跳过 %d 组合",
self.get_task_code(),
upserted,
skipped,
)
return {
"counts": {
"fetched": len(transformed),
"inserted": upserted,
"updated": 0,
"skipped": skipped,
"errors": 0,
}
}
def _sum_overview(
self,
site_id: int,
start_date: date,
end_date: date,
area_code: str,
) -> Dict[str, Any]:
"""从日粒度表 SUM 计算 overview 8 项指标。"""
sql = """
SELECT
COALESCE(SUM(gross_amount), 0) AS occurrence,
COALESCE(SUM(discount_total), 0) AS discount,
COALESCE(SUM(confirmed_income), 0) AS confirmed_revenue,
COALESCE(SUM(cash_inflow_total), 0) AS cash_in,
COALESCE(SUM(cash_outflow_total), 0) AS cash_out,
COALESCE(SUM(cash_balance_change), 0) AS cash_balance
FROM dws.dws_finance_area_daily
WHERE site_id = %s
AND stat_date >= %s
AND stat_date <= %s
AND area_code = %s
"""
rows = self.db.query(sql, (site_id, start_date, end_date, area_code))
if rows:
row = dict(rows[0])
occurrence = Decimal(str(row.get("occurrence", 0)))
discount = Decimal(str(row.get("discount", 0)))
confirmed_revenue = Decimal(str(row.get("confirmed_revenue", 0)))
cash_in = Decimal(str(row.get("cash_in", 0)))
cash_out = Decimal(str(row.get("cash_out", 0)))
cash_balance = Decimal(str(row.get("cash_balance", 0)))
# discount_rate = discount / occurrence（避免除零）
discount_rate = (
(discount / occurrence) if occurrence != 0 else _ZERO
)
# balance_rate = cash_balance / cash_in（避免除零）
balance_rate = (
(cash_balance / cash_in) if cash_in != 0 else _ZERO
)
return {
"occurrence": occurrence,
"discount": discount,
"discount_rate": discount_rate,
"confirmed_revenue": confirmed_revenue,
"cash_in": cash_in,
"cash_out": cash_out,
"cash_balance": cash_balance,
"balance_rate": balance_rate,
}
return {k: _ZERO for k in [
"occurrence", "discount", "discount_rate", "confirmed_revenue",
"cash_in", "cash_out", "cash_balance", "balance_rate",
]}
def _upsert_cache(
self,
site_id: int,
time_range: str,
area_code: str,
start_date: date,
end_date: date,
overview: Dict[str, Any],
fingerprint: str,
) -> None:
"""ON CONFLICT DO UPDATE 写入缓存表。"""
sql = """
INSERT INTO dws.dws_finance_board_cache (
site_id, time_range, area_code,
start_date, end_date,
occurrence, discount, discount_rate, confirmed_revenue,
cash_in, cash_out, cash_balance, balance_rate,
data_fingerprint, computed_at, updated_at
) VALUES (
%s, %s, %s,
%s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
%s, NOW(), NOW()
)
ON CONFLICT (site_id, time_range, area_code) DO UPDATE SET
start_date = EXCLUDED.start_date,
end_date = EXCLUDED.end_date,
occurrence = EXCLUDED.occurrence,
discount = EXCLUDED.discount,
discount_rate = EXCLUDED.discount_rate,
confirmed_revenue = EXCLUDED.confirmed_revenue,
cash_in = EXCLUDED.cash_in,
cash_out = EXCLUDED.cash_out,
cash_balance = EXCLUDED.cash_balance,
balance_rate = EXCLUDED.balance_rate,
data_fingerprint = EXCLUDED.data_fingerprint,
computed_at = NOW(),
updated_at = NOW()
"""
with self.db.conn.cursor() as cur:
cur.execute(
sql,
(
site_id,
time_range,
area_code,
start_date,
end_date,
overview["occurrence"],
overview["discount"],
overview["discount_rate"],
overview["confirmed_revenue"],
overview["cash_in"],
overview["cash_out"],
overview["cash_balance"],
overview["balance_rate"],
fingerprint,
),
)
# ======================================================================
# 纯函数：transform 核心逻辑（方便属性测试直接调用）
# ======================================================================
def transform_cache_combinations(
combinations: List[Dict[str, Any]],
site_id: int,
logger: Any = None,
) -> List[Dict[str, Any]]:
"""对每个组合计算指纹并与已有指纹对比，标记需重算的组合。
纯函数（不依赖数据库），方便属性测试直接调用。
Args:
combinations: extract 输出的组合列表
site_id: 门店 ID
logger: 日志记录器（可选）
Returns:
带 needs_recompute/new_fingerprint 标记的组合列表
"""
if logger is None:
logger = logging.getLogger(__name__)
results: List[Dict[str, Any]] = []
for combo in combinations:
daily_rows = combo.get("daily_rows", [])
existing_fp = combo.get("existing_fingerprint")
new_fp = compute_fingerprint(daily_rows)
needs_recompute = new_fp != existing_fp
if needs_recompute:
logger.info(
"DWS_FINANCE_BOARD_CACHE: %s/%s 指纹变化，需重算",
combo["time_range"],
combo["area_code"],
)
results.append(
{
**combo,
"site_id": site_id,
"new_fingerprint": new_fp,
"needs_recompute": needs_recompute,
}
)
return results
__all__ = [
"FinanceBoardCacheTask",
"compute_fingerprint",
"is_current_period",
"transform_cache_combinations",
"COMPLETED_PERIODS",
"CURRENT_PERIODS",
]

23
apps/etl/connectors/feiqiu/tasks/dws/finance_daily_task.py
View File

@@ -55,6 +55,9 @@ class FinanceDailyTask(FinanceBaseTask):
# CHANGE 2025-07-15 | task 1.3: 声明日期列，供基类默认 load() 使用
DATE_COL = "stat_date"
# CHANGE 2026-03-22 | P14: 财务日度任务完成后触发 AI App2 预生成
AI_TRIGGER_EVENT = "dws_completed"
def get_task_code(self) -> str:
return "DWS_FINANCE_DAILY"
@@ -93,6 +96,10 @@ class FinanceDailyTask(FinanceBaseTask):
# 3.1 获取赠送卡消费汇总（余额变动）
gift_card_summary = self._extract_gift_card_consume_summary(site_id, start_date, end_date)
# 3.2 获取支付方式拆分
# CHANGE 2026-03-27 | board-finance-integration T1.1 | 新增支付方式拆分
payment_split = self._extract_payment_split(site_id, start_date, end_date)
# 4. 获取支出汇总（来自导入表）
expense_summary = self._extract_expense_summary(site_id, start_date, end_date)
@@ -108,6 +115,7 @@ class FinanceDailyTask(FinanceBaseTask):
'groupbuy_summary': groupbuy_summary,
'recharge_summary': recharge_summary,
'gift_card_summary': gift_card_summary,
'payment_split': payment_split,
'expense_summary': expense_summary,
'platform_summary': platform_summary,
'big_customer_summary': big_customer_summary,
@@ -127,6 +135,7 @@ class FinanceDailyTask(FinanceBaseTask):
expense_summary = extracted['expense_summary']
platform_summary = extracted['platform_summary']
big_customer_summary = extracted['big_customer_summary']
payment_split = extracted['payment_split']
site_id = extracted['site_id']
self.logger.info(
@@ -149,6 +158,7 @@ class FinanceDailyTask(FinanceBaseTask):
expense_index = {e['stat_date']: e for e in expense_summary}
platform_index = {p['stat_date']: p for p in platform_summary}
big_customer_index = {b['stat_date']: b for b in big_customer_summary}
payment_split_index = {ps['stat_date']: ps for ps in payment_split}
results = []
for stat_date in sorted(dates):
@@ -159,9 +169,10 @@ class FinanceDailyTask(FinanceBaseTask):
expense = expense_index.get(stat_date, {})
platform = platform_index.get(stat_date, {})
big_customer = big_customer_index.get(stat_date, {})
ps = payment_split_index.get(stat_date, {})
record = self._build_daily_record(
stat_date, settle, groupbuy, recharge, gift_card, expense, platform, big_customer, site_id
stat_date, settle, groupbuy, recharge, gift_card, expense, platform, big_customer, site_id, ps
)
results.append(record)
@@ -185,7 +196,8 @@ class FinanceDailyTask(FinanceBaseTask):
expense: Dict[str, Any],
platform: Dict[str, Any],
big_customer: Dict[str, Any],
site_id: int
site_id: int,
payment_split: Dict[str, Any],
) -> Dict[str, Any]:
"""
构建日度财务记录
@@ -237,6 +249,11 @@ class FinanceDailyTask(FinanceBaseTask):
# 确认收入
confirmed_income = gross_amount - discount_total
# 支付方式拆分
# CHANGE 2026-03-27 | board-finance-integration T1.1 | 新增支付方式拆分
cash_paper_amount = self.safe_decimal(payment_split.get('cash_paper_amount', 0))
scan_pay_amount = self.safe_decimal(payment_split.get('scan_pay_amount', 0))
# 现金流
platform_settlement_amount = self.safe_decimal(platform.get('settlement_amount', 0))
platform_fee_amount = (
@@ -298,6 +315,8 @@ class FinanceDailyTask(FinanceBaseTask):
# 现金流
'cash_inflow_total': cash_inflow_total,
'cash_pay_amount': cash_pay_amount,
'cash_paper_amount': cash_paper_amount,
'scan_pay_amount': scan_pay_amount,
'groupbuy_pay_amount': groupbuy_pay_amount,
'platform_settlement_amount': platform_settlement_amount,
'platform_fee_amount': platform_fee_amount,

2
apps/etl/connectors/feiqiu/tasks/dws/index/base_index_task.py
View File

@@ -83,7 +83,7 @@ class BaseIndexTask(BaseDwsTask):
self._index_params_cache_by_type: Dict[str, IndexParameters] = {}
# 默认参数
DEFAULT_LOOKBACK_DAYS = 60
DEFAULT_LOOKBACK_DAYS = 90
DEFAULT_PERCENTILE_LOWER = 5
DEFAULT_PERCENTILE_UPPER = 95
DEFAULT_EWMA_ALPHA = 0.2

9
apps/etl/connectors/feiqiu/tasks/dws/index/member_index_base.py
View File

@@ -86,7 +86,9 @@ class MemberIndexBaseTask(BaseIndexTask):
tenant_id = self._get_tenant_id()
params = self._load_params()
activities = self._build_member_activity(site_id, tenant_id, params)
# P19: 回测模式用 as_of_date 替代 datetime.now()
as_of = (context.as_of_date if context and context.as_of_date else None) or datetime.now(self.tz)
activities = self._build_member_activity(site_id, tenant_id, params, as_of=as_of)
if not activities:
self.logger.warning("No member activity data available; skip calculation")
return {'status': 'skipped', 'reason': 'no_data'}
@@ -402,9 +404,12 @@ class MemberIndexBaseTask(BaseIndexTask):
site_id: int,
tenant_id: int,
params: Dict[str, float],
*,
as_of: Optional[datetime] = None,
) -> Dict[int, MemberActivityData]:
"""构建会员活动特征"""
now = datetime.now(self.tz)
# P19: 回测模式用 as_of 替代 datetime.now()
now = as_of or datetime.now(self.tz)
base_date = now.date()
visit_lookback_days = int(params.get('visit_lookback_days', self.DEFAULT_VISIT_LOOKBACK_DAYS))

38
apps/etl/connectors/feiqiu/tasks/dws/index/newconv_index_task.py
View File

@@ -202,7 +202,9 @@ class NewconvIndexTask(MemberIndexBaseTask):
avg_raw=sum(all_raw) / len(all_raw)
)
inserted = self._save_newconv_data(newconv_list)
# P19: 回测模式传入 calc_time
calc_time = (context.as_of_date if context and context.as_of_date else None)
inserted = self._save_newconv_data(newconv_list, calc_time=calc_time)
self.logger.info("NCI calculation finished, inserted %d rows", inserted)
return {
@@ -286,21 +288,30 @@ class NewconvIndexTask(MemberIndexBaseTask):
if data.raw_score < 0:
data.raw_score = 0.0
def _save_newconv_data(self, data_list: List[MemberNewconvData]) -> int:
def _save_newconv_data(self, data_list: List[MemberNewconvData], *, calc_time=None) -> int:
"""保存 NCI 数据"""
if not data_list:
return 0
site_id = data_list[0].activity.site_id
# 按门店全量刷新，避免因分群变化导致过期数据残留。
delete_sql = """
DELETE FROM dws.dws_member_newconv_index
WHERE site_id = %s
"""
# P19: 回测模式按 calc_time 删除（保留其他快照），正常模式按 site_id 全量刷新
use_param_time = calc_time is not None
with self.db.conn.cursor() as cur:
cur.execute(delete_sql, (site_id,))
if use_param_time:
cur.execute(
"DELETE FROM dws.dws_member_newconv_index WHERE site_id = %s AND calc_time = %s",
(site_id, calc_time),
)
else:
cur.execute(
"DELETE FROM dws.dws_member_newconv_index WHERE site_id = %s",
(site_id,),
)
insert_sql = """
# P19: 回测模式传入 calc_time，正常模式用 NOW()
use_param_time = calc_time is not None
time_placeholder = "%s, %s, %s" if use_param_time else "NOW(), NOW(), NOW()"
insert_sql = f"""
INSERT INTO dws.dws_member_newconv_index (
site_id, tenant_id, member_id,
status, segment,
@@ -328,7 +339,7 @@ class NewconvIndexTask(MemberIndexBaseTask):
%s, %s, %s,
%s, %s, %s,
%s,
NOW(), NOW(), NOW()
{time_placeholder}
)
"""
@@ -336,7 +347,7 @@ class NewconvIndexTask(MemberIndexBaseTask):
with self.db.conn.cursor() as cur:
for data in data_list:
activity = data.activity
cur.execute(insert_sql, (
params = (
activity.site_id, activity.tenant_id, activity.member_id,
data.status, data.segment,
activity.member_create_time, activity.first_visit_time, activity.last_visit_time, activity.last_recharge_time,
@@ -349,7 +360,10 @@ class NewconvIndexTask(MemberIndexBaseTask):
data.raw_score_welcome, data.raw_score_convert, data.raw_score,
data.display_score_welcome, data.display_score_convert, data.display_score,
None,
))
)
if use_param_time:
params = params + (calc_time, calc_time, calc_time)
cur.execute(insert_sql, params)
inserted += cur.rowcount
self.db.conn.commit()

191
apps/etl/connectors/feiqiu/tasks/dws/index/relation_index_task.py
View File

@@ -78,7 +78,7 @@ class RelationIndexTask(BaseIndexTask):
INDEX_TYPE = "RS"
DEFAULT_PARAMS_RS: Dict[str, float] = {
"lookback_days": 60,
"lookback_days": 90,
"session_merge_hours": 4,
"incentive_weight": 1.5,
"halflife_session": 14.0,
@@ -93,15 +93,13 @@ class RelationIndexTask(BaseIndexTask):
"ewma_alpha": 0.2,
}
DEFAULT_PARAMS_OS: Dict[str, float] = {
"min_rs_raw_for_ownership": 0.05,
"min_total_rs_raw": 0.10,
"ownership_main_threshold": 0.60,
"ownership_comanage_threshold": 0.35,
"ownership_gap_threshold": 0.15,
"min_rs_for_label": 0.10,
"ownership_main_ratio": 0.70,
"ownership_comanage_ratio": 0.30,
"eps": 1e-6,
}
DEFAULT_PARAMS_MS: Dict[str, float] = {
"lookback_days": 60,
"lookback_days": 90,
"session_merge_hours": 4,
"incentive_weight": 1.5,
"halflife_short": 7.0,
@@ -115,7 +113,7 @@ class RelationIndexTask(BaseIndexTask):
}
# CHANGE 2026-02-13 | intent: ML 仅使用人工台账，移除 source_mode / recharge_attribute_hours
DEFAULT_PARAMS_ML: Dict[str, float] = {
"lookback_days": 60,
"lookback_days": 90,
"amount_base": 500.0,
"halflife_recharge": 21.0,
"percentile_lower": 5.0,
@@ -143,7 +141,8 @@ class RelationIndexTask(BaseIndexTask):
site_id = self._get_site_id(context)
tenant_id = self._get_tenant_id()
now = datetime.now(self.tz)
# P19: 回测模式用 as_of_date 替代 datetime.now()
now = (context.as_of_date if context and context.as_of_date else None) or datetime.now(self.tz)
params_rs = self._load_params("RS", self.DEFAULT_PARAMS_RS)
params_os = self._load_params("OS", self.DEFAULT_PARAMS_OS)
@@ -151,8 +150,8 @@ class RelationIndexTask(BaseIndexTask):
params_ml = self._load_params("ML", self.DEFAULT_PARAMS_ML)
service_lookback_days = max(
int(params_rs.get("lookback_days", 60)),
int(params_ms.get("lookback_days", 60)),
int(params_rs.get("lookback_days", 90)),
int(params_ms.get("lookback_days", 90)),
)
service_start = now - timedelta(days=service_lookback_days)
merge_hours = max(
@@ -181,7 +180,9 @@ class RelationIndexTask(BaseIndexTask):
self._apply_display_scores(pair_map, params_rs, params_ms, params_ml, site_id)
inserted = self._save_relation_rows(site_id, list(pair_map.values()))
# P19: 仅回测模式传 calc_time（按 calc_time 删除保留其他快照），正常模式传 None（按 site_id 全量刷新）
backtest_calc_time = now if (context and context.as_of_date) else None
inserted = self._save_relation_rows(site_id, list(pair_map.values()), calc_time=backtest_calc_time)
self.logger.info("关系指数计算完成，写入 %d 条记录", inserted)
return {
@@ -313,7 +314,7 @@ class RelationIndexTask(BaseIndexTask):
params: Dict[str, float],
now: datetime,
) -> None:
lookback_days = int(params.get("lookback_days", 60))
lookback_days = int(params.get("lookback_days", 90))
halflife_session = float(params.get("halflife_session", 14.0))
halflife_last = float(params.get("halflife_last", 10.0))
weight_f = float(params.get("weight_f", 1.0))
@@ -355,7 +356,7 @@ class RelationIndexTask(BaseIndexTask):
params: Dict[str, float],
now: datetime,
) -> None:
lookback_days = int(params.get("lookback_days", 60))
lookback_days = int(params.get("lookback_days", 90))
halflife_short = float(params.get("halflife_short", 7.0))
halflife_long = float(params.get("halflife_long", 30.0))
eps = float(params.get("eps", 1e-6))
@@ -382,7 +383,7 @@ class RelationIndexTask(BaseIndexTask):
site_id: int,
now: datetime,
) -> None:
lookback_days = int(params.get("lookback_days", 60))
lookback_days = int(params.get("lookback_days", 90))
amount_base = float(params.get("amount_base", 500.0))
halflife_recharge = float(params.get("halflife_recharge", 21.0))
start_time = now - timedelta(days=lookback_days)
@@ -439,68 +440,53 @@ class RelationIndexTask(BaseIndexTask):
pair_map: Dict[Tuple[int, int], RelationPairMetrics],
params: Dict[str, float],
) -> None:
min_rs = float(params.get("min_rs_raw_for_ownership", 0.05))
min_total = float(params.get("min_total_rs_raw", 0.10))
main_threshold = float(params.get("ownership_main_threshold", 0.60))
comanage_threshold = float(params.get("ownership_comanage_threshold", 0.35))
gap_threshold = float(params.get("ownership_gap_threshold", 0.15))
"""CHANGE 2026-03-31 | 新 OS 方案：基于第一名分值比例。
规则：
- 第一名一定为 MAIN（rs_raw ≥ min_rs_for_label）
- 在第一名分值的 main_ratio_threshold 以内都为 MAIN
- 在第一名分值的 comanage_ratio_threshold 以内都为 COMANAGE
- 其余为 POOL
- MAIN/COMANAGE 前提：rs_raw ≥ min_rs_for_label
"""
min_rs_for_label = float(params.get("min_rs_for_label", 0.1))
main_ratio = float(params.get("ownership_main_ratio", 0.70))
comanage_ratio = float(params.get("ownership_comanage_ratio", 0.30))
member_groups: Dict[int, List[RelationPairMetrics]] = {}
for metrics in pair_map.values():
member_groups.setdefault(metrics.member_id, []).append(metrics)
for _, rows in member_groups.items():
eligible = [row for row in rows if row.rs_raw >= min_rs]
sum_rs = sum(row.rs_raw for row in eligible)
if sum_rs < min_total:
for row in rows:
row.os_share = 0.0
row.os_label = "UNASSIGNED"
row.os_rank = None
continue
for row in rows:
if row.rs_raw >= min_rs:
row.os_share = row.rs_raw / sum_rs
else:
row.os_share = 0.0
sorted_eligible = sorted(
eligible,
# 按 rs_raw 降序排列
sorted_rows = sorted(
rows,
key=lambda item: (
-item.os_share,
-item.rs_raw,
item.days_since_last_session if item.days_since_last_session is not None else 10**9,
item.assistant_id,
),
)
for idx, row in enumerate(sorted_eligible, start=1):
row.os_rank = idx
top1 = sorted_eligible[0]
top2_share = sorted_eligible[1].os_share if len(sorted_eligible) > 1 else 0.0
gap = top1.os_share - top2_share
has_main = top1.os_share >= main_threshold and gap >= gap_threshold
top_rs = sorted_rows[0].rs_raw if sorted_rows else 0.0
if has_main:
for row in rows:
if row is top1:
row.os_label = "MAIN"
elif row.os_share >= comanage_threshold:
row.os_label = "COMANAGE"
else:
row.os_label = "POOL"
else:
for row in rows:
if row.os_share >= comanage_threshold and row.rs_raw >= min_rs:
row.os_label = "COMANAGE"
else:
row.os_label = "POOL"
for idx, row in enumerate(sorted_rows):
row.os_rank = idx + 1
# 计算份额（保留兼容性，用于前端展示）
sum_rs = sum(r.rs_raw for r in rows if r.rs_raw > 0)
row.os_share = row.rs_raw / sum_rs if sum_rs > 0 else 0.0
# 非 eligible 不赋 rank
for row in rows:
if row.rs_raw < min_rs:
row.os_rank = None
if top_rs <= 0 or row.rs_raw < min_rs_for_label:
row.os_label = "POOL"
continue
ratio = row.rs_raw / top_rs
if ratio >= main_ratio or row is sorted_rows[0]:
row.os_label = "MAIN"
elif ratio >= comanage_ratio:
row.os_label = "COMANAGE"
else:
row.os_label = "POOL"
def _apply_display_scores(
self,
@@ -599,18 +585,27 @@ class RelationIndexTask(BaseIndexTask):
return "asinh"
return "none"
def _save_relation_rows(self, site_id: int, rows: List[RelationPairMetrics]) -> int:
def _save_relation_rows(self, site_id: int, rows: List[RelationPairMetrics], *, calc_time: Optional[datetime] = None) -> int:
# P19: 回测模式传入 calc_time，正常模式用 NOW()
use_param_time = calc_time is not None
with self.db.conn.cursor() as cur:
cur.execute(
"DELETE FROM dws.dws_member_assistant_relation_index WHERE site_id = %s",
(site_id,),
)
# P19: 回测模式按 calc_time 删除（保留其他快照），正常模式按 site_id 全量刷新
if use_param_time:
cur.execute(
"DELETE FROM dws.dws_member_assistant_relation_index WHERE site_id = %s AND calc_time = %s",
(site_id, calc_time),
)
else:
cur.execute(
"DELETE FROM dws.dws_member_assistant_relation_index WHERE site_id = %s",
(site_id,),
)
if not rows:
self.db.conn.commit()
return 0
insert_sql = """
insert_sql = f"""
INSERT INTO dws.dws_member_assistant_relation_index (
site_id, tenant_id, member_id, assistant_id,
session_count, total_duration_minutes, basic_session_count, incentive_session_count,
@@ -628,41 +623,41 @@ class RelationIndexTask(BaseIndexTask):
%s, %s, %s,
%s, %s, %s, %s,
%s, %s, %s, %s,
NOW(), NOW(), NOW()
{('%s, %s, %s' if use_param_time else 'NOW(), NOW(), NOW()')}
)
"""
inserted = 0
for row in rows:
cur.execute(
insert_sql,
(
row.site_id,
row.tenant_id,
row.member_id,
row.assistant_id,
row.session_count,
row.total_duration_minutes,
row.basic_session_count,
row.incentive_session_count,
row.days_since_last_session,
row.rs_f,
row.rs_d,
row.rs_r,
row.rs_raw,
row.rs_display,
row.os_share,
row.os_label,
row.os_rank,
row.ms_f_short,
row.ms_f_long,
row.ms_raw,
row.ms_display,
row.ml_order_count,
row.ml_allocated_amount,
row.ml_raw,
row.ml_display,
),
params = (
row.site_id,
row.tenant_id,
row.member_id,
row.assistant_id,
row.session_count,
row.total_duration_minutes,
row.basic_session_count,
row.incentive_session_count,
row.days_since_last_session,
row.rs_f,
row.rs_d,
row.rs_r,
row.rs_raw,
row.rs_display,
row.os_share,
row.os_label,
row.os_rank,
row.ms_f_short,
row.ms_f_long,
row.ms_raw,
row.ms_display,
row.ml_order_count,
row.ml_allocated_amount,
row.ml_raw,
row.ml_display,
)
if use_param_time:
params = params + (calc_time, calc_time, calc_time)
cur.execute(insert_sql, params)
inserted += max(cur.rowcount, 0)
self.db.conn.commit()
return inserted

80
apps/etl/connectors/feiqiu/tasks/dws/index/spending_power_index_task.py
View File

@@ -173,13 +173,17 @@ class SpendingPowerIndexTask(BaseIndexTask):
# 1. 获取 site_id
site_id = self._get_site_id(context)
# P19: 回测模式用 as_of_date 替代 NOW()
from datetime import datetime as _dt
as_of = (context.as_of_date if context and context.as_of_date else None) or _dt.now(self.tz)
# 2. 加载参数（配置表 + 默认值合并）
db_params = self.load_index_parameters('SPI')
params = {**self.DEFAULT_PARAMS, **db_params}
# 3. 提取特征
features = self._extract_spending_features(site_id, params)
recharge_map = self._extract_recharge_features(site_id, params)
features = self._extract_spending_features(site_id, params, as_of=as_of)
recharge_map = self._extract_recharge_features(site_id, params, as_of=as_of)
# 合并充值特征
for mid, recharge_90 in recharge_map.items():
@@ -189,7 +193,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
# 批量计算日消费 EWMA 并合并
member_ids = list(features.keys())
ewma_map = self._compute_daily_spend_ewma_batch(site_id, member_ids, params)
ewma_map = self._compute_daily_spend_ewma_batch(site_id, member_ids, params, as_of=as_of)
for mid, ewma_val in ewma_map.items():
if mid in features:
features[mid].daily_spend_ewma_90 = ewma_val
@@ -279,7 +283,9 @@ class SpendingPowerIndexTask(BaseIndexTask):
feat.score_stability_display = stability_display_map.get(mid, 0.0)
# 8. delete-before-insert 持久化（Req 9.3）
records_inserted = self._save_spi_data(feat_list, site_id)
# P19: 仅回测模式传 calc_time，正常模式传 None（按 site_id 全量刷新）
backtest_calc_time = as_of if (context and context.as_of_date) else None
records_inserted = self._save_spi_data(feat_list, site_id, calc_time=backtest_calc_time)
# 9. 保存分位点历史（Req 9.5）——SPI 总分
raw_values = [f.raw_score for f in feat_list]
@@ -323,7 +329,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
# =========================================================================
def _extract_spending_features(
self, site_id: int, params: Dict[str, float]
self, site_id: int, params: Dict[str, float], *, as_of=None
) -> Dict[int, SPIMemberFeatures]:
"""从 dwd_settlement_head 提取消费特征，按 member_id 聚合。
@@ -339,8 +345,8 @@ class SpendingPowerIndexTask(BaseIndexTask):
cutoff = self.config.get("app.business_day_start_hour", 8)
biz_expr = biz_date_sql_expr("pay_time", cutoff)
# 单条 SQL 同时聚合 30 天和 90 天窗口，避免两次扫描
# INTERVAL 天数通过 f-string 内嵌（整数，安全）；site_id 走参数化
# P19: 回测模式用 as_of 参数替代 NOW()
# 时间基准用 %s 参数化，正常模式传 NOW() 等效的 as_of
sql = f"""
WITH consume_source AS (
SELECT
@@ -356,7 +362,8 @@ class SpendingPowerIndexTask(BaseIndexTask):
AND COALESCE(mca.is_delete, 0) = 0
WHERE s.site_id = %s
AND s.settle_type IN (1, 3)
AND s.pay_time >= NOW() - INTERVAL '{long_days} days'
AND s.pay_time >= %s - INTERVAL '{long_days} days'
AND s.pay_time < %s
)
SELECT
canonical_member_id AS member_id,
@@ -367,17 +374,17 @@ class SpendingPowerIndexTask(BaseIndexTask):
COUNT(DISTINCT EXTRACT(ISOYEAR FROM pay_time)::int * 100
+ EXTRACT(WEEK FROM pay_time)::int) AS active_weeks_90,
-- 30 天窗口（子集过滤）
SUM(CASE WHEN pay_time >= NOW() - INTERVAL '{short_days} days'
SUM(CASE WHEN pay_time >= %s - INTERVAL '{short_days} days'
THEN pay_amount ELSE 0 END) AS spend_30,
SUM(CASE WHEN pay_time >= NOW() - INTERVAL '{short_days} days'
SUM(CASE WHEN pay_time >= %s - INTERVAL '{short_days} days'
THEN 1 ELSE 0 END) AS orders_30,
COUNT(DISTINCT CASE WHEN pay_time >= NOW() - INTERVAL '{short_days} days'
COUNT(DISTINCT CASE WHEN pay_time >= %s - INTERVAL '{short_days} days'
THEN {biz_expr} END) AS visit_days_30
FROM consume_source
WHERE canonical_member_id > 0
GROUP BY canonical_member_id
"""
rows = self.db.query(sql, (site_id,))
rows = self.db.query(sql, (site_id, as_of, as_of, as_of, as_of, as_of))
result: Dict[int, SPIMemberFeatures] = {}
for row in (rows or []):
@@ -412,7 +419,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
def _extract_recharge_features(
self, site_id: int, params: Dict[str, float]
self, site_id: int, params: Dict[str, float], *, as_of=None
) -> Dict[int, float]:
"""从 dwd_recharge_order 提取充值特征，返回 {member_id: recharge_90}。
@@ -421,6 +428,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
"""
long_days = int(params.get('spend_window_long_days', 90))
# P19: 回测模式用 as_of 参数替代 NOW()
sql = f"""
WITH recharge_source AS (
SELECT
@@ -435,7 +443,8 @@ class SpendingPowerIndexTask(BaseIndexTask):
AND COALESCE(mca.is_delete, 0) = 0
WHERE r.site_id = %s
AND r.settle_type = 5
AND r.pay_time >= NOW() - INTERVAL '{long_days} days'
AND r.pay_time >= %s - INTERVAL '{long_days} days'
AND r.pay_time < %s
)
SELECT
canonical_member_id AS member_id,
@@ -444,7 +453,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
WHERE canonical_member_id > 0
GROUP BY canonical_member_id
"""
rows = self.db.query(sql, (site_id,))
rows = self.db.query(sql, (site_id, as_of, as_of))
result: Dict[int, float] = {}
for row in (rows or []):
@@ -513,7 +522,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
return ewma
def _compute_daily_spend_ewma_batch(
self, site_id: int, member_ids: List[int], params: Dict[str, float]
self, site_id: int, member_ids: List[int], params: Dict[str, float], *, as_of=None
) -> Dict[int, float]:
"""批量计算多个会员的日消费 EWMA，单次 SQL 查询避免 N+1。
@@ -528,6 +537,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
cutoff = self.config.get("app.business_day_start_hour", 8)
biz_expr_s = biz_date_sql_expr("s.pay_time", cutoff)
# P19: 回测模式用 as_of 参数替代 NOW()
sql = f"""
WITH consume_source AS (
SELECT
@@ -543,7 +553,8 @@ class SpendingPowerIndexTask(BaseIndexTask):
AND COALESCE(mca.is_delete, 0) = 0
WHERE s.site_id = %s
AND s.settle_type IN (1, 3)
AND s.pay_time >= NOW() - INTERVAL '{long_days} days'
AND s.pay_time >= %s - INTERVAL '{long_days} days'
AND s.pay_time < %s
)
SELECT canonical_member_id AS member_id,
pay_date,
@@ -553,7 +564,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
GROUP BY canonical_member_id, pay_date
ORDER BY canonical_member_id, pay_date
"""
rows = self.db.query(sql, (site_id,))
rows = self.db.query(sql, (site_id, as_of, as_of))
# 按 member_id 分组，逐组计算 EWMA
result: Dict[int, float] = {}
@@ -727,21 +738,31 @@ class SpendingPowerIndexTask(BaseIndexTask):
# =========================================================================
def _save_spi_data(
self, data_list: List[SPIMemberFeatures], site_id: int
self, data_list: List[SPIMemberFeatures], site_id: int, *, calc_time=None
) -> int:
"""delete-before-insert 写入 dws_member_spending_power_index"""
with self.db.conn.cursor() as cur:
# 先删除该门店旧记录（Req 9.3）
cur.execute(
"DELETE FROM dws.dws_member_spending_power_index WHERE site_id = %s",
(site_id,),
)
# P19: 回测模式按 calc_time 删除（保留其他快照），正常模式按 site_id 全量刷新
use_param_time = calc_time is not None
if use_param_time:
cur.execute(
"DELETE FROM dws.dws_member_spending_power_index WHERE site_id = %s AND calc_time = %s",
(site_id, calc_time),
)
else:
cur.execute(
"DELETE FROM dws.dws_member_spending_power_index WHERE site_id = %s",
(site_id,),
)
if not data_list:
self.db.conn.commit()
return 0
insert_sql = """
# P19: 回测模式传入 calc_time，正常模式用 NOW()
use_param_time = calc_time is not None
time_placeholder = "%s, %s, %s" if use_param_time else "NOW(), NOW(), NOW()"
insert_sql = f"""
INSERT INTO dws.dws_member_spending_power_index (
site_id, member_id,
spend_30, spend_90, recharge_90,
@@ -761,7 +782,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
%s, %s, %s,
%s, %s, %s,
%s, %s,
NOW(), NOW(), NOW()
{time_placeholder}
)
"""
inserted = 0
@@ -773,7 +794,7 @@ class SpendingPowerIndexTask(BaseIndexTask):
return max(lo, min(hi, v))
for f in data_list:
cur.execute(insert_sql, (
params_tuple = (
f.site_id, f.member_id,
f.spend_30, f.spend_90, f.recharge_90,
f.orders_30, f.orders_90,
@@ -787,7 +808,10 @@ class SpendingPowerIndexTask(BaseIndexTask):
_clamp(f.score_stability_display, 0, DISP_MAX),
_clamp(f.raw_score, -RAW_MAX, RAW_MAX),
_clamp(f.display_score, 0, DISP_MAX),
))
)
if use_param_time:
params_tuple = params_tuple + (calc_time, calc_time, calc_time)
cur.execute(insert_sql, params_tuple)
inserted += max(cur.rowcount, 0)
self.db.conn.commit()

39
apps/etl/connectors/feiqiu/tasks/dws/index/winback_index_task.py
View File

@@ -7,7 +7,7 @@ from __future__ import annotations
import math
from dataclasses import dataclass
from datetime import date, timedelta
from datetime import date, datetime, timedelta
from typing import Any, Dict, List, Optional, Tuple
from .member_index_base import MemberActivityData, MemberIndexBaseTask
@@ -178,7 +178,9 @@ class WinbackIndexTask(MemberIndexBaseTask):
avg_raw=sum(all_raw) / len(all_raw)
)
inserted = self._save_winback_data(winback_list)
# P19: 回测模式传入 calc_time
calc_time = (context.as_of_date if context and context.as_of_date else None)
inserted = self._save_winback_data(winback_list, calc_time=calc_time)
self.logger.info("WBI calculation finished, inserted %d rows", inserted)
return {
@@ -339,21 +341,29 @@ class WinbackIndexTask(MemberIndexBaseTask):
if data.raw_score < 0:
data.raw_score = 0.0
def _save_winback_data(self, data_list: List[MemberWinbackData]) -> int:
def _save_winback_data(self, data_list: List[MemberWinbackData], *, calc_time: Optional[datetime] = None) -> int:
"""保存 WBI 数据"""
if not data_list:
return 0
site_id = data_list[0].activity.site_id
# 按门店全量刷新，避免因分群变化导致过期数据残留。
delete_sql = """
DELETE FROM dws.dws_member_winback_index
WHERE site_id = %s
"""
# P19: 回测模式传入 calc_time，正常模式用 NOW()
use_param_time = calc_time is not None
# P19: 回测模式按 calc_time 删除（保留其他快照），正常模式按 site_id 全量刷新
with self.db.conn.cursor() as cur:
cur.execute(delete_sql, (site_id,))
if use_param_time:
cur.execute(
"DELETE FROM dws.dws_member_winback_index WHERE site_id = %s AND calc_time = %s",
(site_id, calc_time),
)
else:
cur.execute(
"DELETE FROM dws.dws_member_winback_index WHERE site_id = %s",
(site_id,),
)
insert_sql = """
time_placeholder = "%s, %s, %s" if use_param_time else "NOW(), NOW(), NOW()"
insert_sql = f"""
INSERT INTO dws.dws_member_winback_index (
site_id, tenant_id, member_id,
status, segment,
@@ -379,7 +389,7 @@ class WinbackIndexTask(MemberIndexBaseTask):
%s, %s,
%s, %s,
%s,
NOW(), NOW(), NOW()
{time_placeholder}
)
"""
@@ -387,7 +397,7 @@ class WinbackIndexTask(MemberIndexBaseTask):
with self.db.conn.cursor() as cur:
for data in data_list:
activity = data.activity
cur.execute(insert_sql, (
params = (
activity.site_id, activity.tenant_id, activity.member_id,
data.status, data.segment,
activity.member_create_time, activity.first_visit_time, activity.last_visit_time, activity.last_recharge_time,
@@ -399,7 +409,10 @@ class WinbackIndexTask(MemberIndexBaseTask):
data.ideal_interval_days, data.ideal_next_visit_date,
data.raw_score, data.display_score,
None,
))
)
if use_param_time:
params = params + (calc_time, calc_time, calc_time)
cur.execute(insert_sql, params)
inserted += cur.rowcount
self.db.conn.commit()

3
apps/etl/connectors/feiqiu/tasks/dws/member_consumption_task.py
View File

@@ -53,6 +53,9 @@ class MemberConsumptionTask(BaseDwsTask):
# CHANGE 2025-07-15 | task 1.3: 声明日期列，供基类默认 load() 使用
DATE_COL = "stat_date"
# CHANGE 2026-03-22 | P14: 消费汇总完成后触发 AI 消费事件链
AI_TRIGGER_EVENT = "consumption"
def get_task_code(self) -> str:
return "DWS_MEMBER_CONSUMPTION"

124
apps/etl/connectors/feiqiu/tasks/dws/task_engine.py Normal file
View File

@@ -0,0 +1,124 @@
# AI_CHANGELOG
# - 2026-03-29 | Prompt: DWS_TASK_ENGINE ETL 任务 | 新建文件。
# 编排任务引擎全流程：完成检查 → 过期检查 → 任务生成。
# 通过 HTTP 调用后端 POST /api/internal/run-job 按 job_name 执行。
# -*- coding: utf-8 -*-
"""
DWS 任务引擎编排任务（DWS_TASK_ENGINE）
在 DWS 指数计算完成后执行，按顺序调用后端任务引擎的各个步骤：
1. recall_completion_check — 检测召回是否完成，生成回访任务
2. task_expiry_check — 标记超时未处理的任务
3. task_generator — 根据 WBI/NCI/RS 指数生成/替换任务
通过 HTTP 调用后端 POST /api/internal/run-job（Internal-Token 认证），
每步失败仅记录日志，不中断后续步骤。
"""
from __future__ import annotations
import logging
import os
from pathlib import Path
from typing import Any
import requests
from dotenv import load_dotenv
from ..base_task import BaseTask, TaskContext
# 加载根 .env（BACKEND_API_URL / INTERNAL_API_TOKEN 不在 AppConfig 映射中）
# task_engine.py → dws/ → tasks/ → feiqiu/ → connectors/ → etl/ → apps/ → root
_REPO_ROOT = Path(__file__).resolve().parents[6]
load_dotenv(_REPO_ROOT / ".env", override=False)
logger = logging.getLogger(__name__)
_TIMEOUT = (5, 30) # 连接 5s，读取 30s（任务执行可能较慢）
# 按顺序执行的后端任务列表
_JOB_SEQUENCE = [
"recall_completion_check",
"task_expiry_check",
"task_generator",
]
def _run_backend_job(backend_url: str, token: str, job_name: str) -> dict:
"""调用后端 POST /api/internal/run-job 执行指定任务。
Returns:
{"success": bool, "message": str} 或 {"success": False, "message": error}
"""
url = f"{backend_url}/api/internal/run-job"
headers = {
"Authorization": f"Internal-Token {token}",
"Content-Type": "application/json",
}
body = {"job_name": job_name}
try:
resp = requests.post(url, json=body, headers=headers, timeout=_TIMEOUT)
if resp.status_code == 200:
data = resp.json()
# 后端 ResponseWrapperMiddleware 包装：{"code": 0, "data": {...}}
inner = data.get("data", data)
return {
"success": inner.get("success", False),
"message": inner.get("message", ""),
}
else:
return {
"success": False,
"message": f"HTTP {resp.status_code}: {resp.text[:200]}",
}
except requests.RequestException as exc:
return {"success": False, "message": str(exc)}
class DwsTaskEngineTask(BaseTask):
"""DWS 任务引擎编排任务。
不读写 DWS 表，仅通过 HTTP 调用后端执行任务引擎步骤。
继承 BaseTask 而非 BaseDwsTask，因为不需要 DWS 层的数据操作方法。
"""
def get_task_code(self) -> str:
return "DWS_TASK_ENGINE"
def extract(self, context: TaskContext) -> dict[str, Any]:
"""无需提取数据，返回空上下文。"""
return {}
def load(self, extracted: dict[str, Any], context: TaskContext) -> dict[str, Any]:
"""按顺序调用后端任务引擎的各个步骤。"""
backend_url = os.environ.get("BACKEND_API_URL", "").rstrip("/")
token = os.environ.get("INTERNAL_API_TOKEN", "")
if not backend_url:
self.logger.error("DWS_TASK_ENGINE 跳过：BACKEND_API_URL 未配置")
return {"skipped": True, "reason": "BACKEND_API_URL 未配置"}
if not token:
self.logger.error("DWS_TASK_ENGINE 跳过：INTERNAL_API_TOKEN 未配置")
return {"skipped": True, "reason": "INTERNAL_API_TOKEN 未配置"}
results: dict[str, Any] = {}
for job_name in _JOB_SEQUENCE:
self.logger.info("DWS_TASK_ENGINE: 执行 %s ...", job_name)
result = _run_backend_job(backend_url, token, job_name)
success = result.get("success", False)
message = result.get("message", "")
results[job_name] = {"success": success, "message": message}
if success:
self.logger.info(
"DWS_TASK_ENGINE: %s 成功 — %s", job_name, message
)
else:
self.logger.warning(
"DWS_TASK_ENGINE: %s 失败 — %s", job_name, message
)
return results

2
apps/etl/connectors/feiqiu/tasks/verification/index_verifier.py
View File

@@ -16,7 +16,7 @@ class IndexVerifier(BaseVerifier):
self,
db_connection: Any,
logger: Optional[logging.Logger] = None,
lookback_days: int = 60,
lookback_days: int = 90,
config: Any = None,
):
super().__init__(db_connection, logger)

94
apps/etl/connectors/feiqiu/utils/ai_trigger.py Normal file
View File

@@ -0,0 +1,94 @@
# -*- coding: utf-8 -*-
"""
AI 触发工具 — DWS 任务完成后通知后端 AI 模块。
通过 POST /api/internal/ai/trigger 发送事件，
后端异步执行 AI 调用链（App2 预生成、消费事件链等）。
失败时仅记录日志，不中断 ETL 流程。
"""
from __future__ import annotations
import logging
import os
from typing import Any
import requests
logger = logging.getLogger(__name__)
# 超时设置（秒）：连接 5s，读取 10s
_TIMEOUT = (5, 10)
def trigger_ai_event(
event_type: str,
site_id: int,
member_id: int | None = None,
payload: dict[str, Any] | None = None,
connector_type: str = "feiqiu",
) -> bool:
"""向后端 AI 模块发送触发事件。
Args:
event_type: 事件类型（dws_completed / consumption / note_created / task_assigned）
site_id: 门店 ID
member_id: 会员 ID（可选）
payload: 附加数据（可选）
connector_type: 连接器类型，默认 feiqiu
Returns:
True 表示发送成功，False 表示失败（已记录日志）
"""
backend_url = os.environ.get("BACKEND_API_URL", "").rstrip("/")
token = os.environ.get("INTERNAL_API_TOKEN", "")
if not backend_url:
logger.warning("AI 触发跳过：BACKEND_API_URL 未配置")
return False
if not token:
logger.warning("AI 触发跳过：INTERNAL_API_TOKEN 未配置")
return False
url = f"{backend_url}/api/internal/ai/trigger"
headers = {
"Authorization": f"Internal-Token {token}",
"Content-Type": "application/json",
}
body = {
"event_type": event_type,
"connector_type": connector_type,
"site_id": site_id,
"member_id": member_id,
"payload": payload or {},
}
try:
resp = requests.post(url, json=body, headers=headers, timeout=_TIMEOUT)
if resp.status_code == 200:
data = resp.json()
logger.info(
"AI 触发成功：event=%s site=%s job_id=%s",
event_type,
site_id,
data.get("trigger_job_id"),
)
return True
else:
logger.warning(
"AI 触发失败：event=%s site=%s status=%s body=%s",
event_type,
site_id,
resp.status_code,
resp.text[:200],
)
return False
except requests.RequestException as exc:
logger.warning(
"AI 触发异常：event=%s site=%s error=%s",
event_type,
site_id,
exc,
)
return False