Compare commits

..

12 Commits

Author SHA1 Message Date
Neo
a6ad343092 ODS 完成 2025-11-30 07:19:05 +08:00
Neo
b9b050bb5d ODS 完成 2025-11-30 07:18:55 +08:00
Neo
cbd16a39ba 阶段性更新 2025-11-20 01:27:33 +08:00
Neo
92f219b575 阶段性更新 2025-11-20 01:27:04 +08:00
Neo
b1f64c4bac 版本更改 2025-11-19 05:35:22 +08:00
Neo
ed47754b46 版本更改 2025-11-19 05:35:10 +08:00
Neo
fbee8a751e 同步 2025-11-19 05:32:03 +08:00
Neo
cbe48c8ee7 为WinSW做准备 2025-11-19 05:05:11 +08:00
Neo
821d302243 Merge branch 'main' into dev 2025-11-19 03:38:27 +08:00
Neo
9a1df70a23 补全任务与测试 2025-11-19 03:36:44 +08:00
Neo
5bb5a8a568 迁移代码到Git 2025-11-18 21:46:46 +08:00
Neo
84e80841cd 代码迁移 2025-11-18 02:28:47 +08:00
113 changed files with 23965 additions and 183254 deletions

1360
20251121-task.txt Normal file

File diff suppressed because it is too large Load Diff

BIN
DWD层设计建议.docx Normal file

Binary file not shown.

View File

@@ -0,0 +1,78 @@
# 台球场 ETL 系统
用于台球门店业务的数据采集与入湖:从上游 API 拉取订单、支付、会员、库存等数据,先落地 ODS再清洗写入事实/维度表,并提供运行追踪、增量游标、数据质量检查与测试脚手架。
## 核心特性
- **两阶段链路**ODS 原始留痕 + DWD/事实表清洗,支持回放与重跑。
- **任务注册与调度**`TaskRegistry` 统一管理任务代码,`ETLScheduler` 负责游标、运行记录和失败隔离。
- **统一底座**:配置(默认值 + `.env` + CLI 覆盖)、分页/重试的 API 客户端、批量 Upsert 的数据库封装、SCD2 维度处理、质量检查。
- **测试与回放**ONLINE/OFFLINE 模式切换,`run_tests.py`/`test_presets.py` 支持参数化测试;`MANUAL_INGEST` 可将归档 JSON 重灌入 ODS。
- **可安装**`setup.py` / `entry_point` 提供 `etl-billiards` 命令,或直接 `python -m cli.main` 运行。
## 仓库结构(摘录)
- `etl_billiards/config`:默认配置、环境变量解析、配置加载。
- `etl_billiards/api`HTTP 客户端,内置重试/分页。
- `etl_billiards/database`:连接管理、批量 Upsert。
- `etl_billiards/tasks`业务任务ORDERS、PAYMENTS…、ODS 任务、DWD 任务、人工回放;`base_task.py`/`base_dwd_task.py` 提供模板。
- `etl_billiards/loaders`:事实/维度/ODS Loader`scd/` 为 SCD2。
- `etl_billiards/orchestration`:调度器、任务注册表、游标与运行追踪。
- `etl_billiards/scripts`:测试执行器、数据库连通性检测、预置测试指令。
- `etl_billiards/tests`:单元/集成测试与离线 JSON 归档。
## 支持的任务代码
- **事实/维度**`ORDERS``PAYMENTS``REFUNDS``INVENTORY_CHANGE``COUPON_USAGE``MEMBERS``ASSISTANTS``PRODUCTS``TABLES``PACKAGES_DEF``TOPUPS``TABLE_DISCOUNT``ASSISTANT_ABOLISH``LEDGER``TICKET_DWD``PAYMENTS_DWD``MEMBERS_DWD`
- **ODS 原始采集**`ODS_ORDER_SETTLE``ODS_TABLE_USE``ODS_ASSISTANT_LEDGER``ODS_ASSISTANT_ABOLISH``ODS_GOODS_LEDGER``ODS_PAYMENT``ODS_REFUND``ODS_COUPON_VERIFY``ODS_MEMBER``ODS_MEMBER_CARD``ODS_PACKAGE``ODS_INVENTORY_STOCK``ODS_INVENTORY_CHANGE`
- **辅助**`MANUAL_INGEST`(将归档 JSON 回放到 ODS
## 快速开始
1. **环境要求**Python 3.10+、PostgreSQL。推荐在 `etl_billiards/` 目录下执行命令。
2. **安装依赖**
```bash
cd etl_billiards
pip install -r requirements.txt
# 开发模式pip install -e .
```
3. **配置 `.env`**
```bash
cp .env.example .env
# 核心项
PG_DSN=postgresql://user:pwd@host:5432/LLZQ
API_BASE=https://api.example.com
API_TOKEN=your_token
STORE_ID=2790685415443269
EXPORT_ROOT=/path/to/export
LOG_ROOT=/path/to/logs
```
配置的生效顺序为 “默认值” < “环境变量/.env” < “CLI 参数”。
4. **运行任务**
```bash
# 运行默认任务集
python -m cli.main
# 按需选择任务(逗号分隔)
python -m cli.main --tasks ODS_ORDER_SETTLE,ORDERS,PAYMENTS
# Dry-run 示例(不提交事务)
python -m cli.main --tasks ORDERS --dry-run
# Windows 批处理
..\\run_etl.bat --tasks PAYMENTS
```
5. **查看输出**:日志目录与导出目录分别由 `LOG_ROOT`、`EXPORT_ROOT` 控制;运行追踪与游标记录写入数据库 `etl_admin.*` 表。
## 数据与运行流转
- CLI 解析参数 → `AppConfig.load()` 组装配置 → `ETLScheduler` 创建 DB/API/游标/运行追踪器。
- 调度器按任务代码实例化任务,读取/推进游标,落盘运行记录。
- 任务模板:确定时间窗口 → 调用 API/ODS 数据 → 解析校验 → Loader 批量 Upsert/SCD2 → 质量检查 → 提交事务并回写游标。
## 测试与回放
- 单元/集成测试:`pytest` 或 `python scripts/run_tests.py --suite online`。
- 预置组合:`python scripts/run_tests.py --preset offline_realdb`(见 `scripts/test_presets.py`)。
- 离线模式:`TEST_MODE=OFFLINE TEST_JSON_ARCHIVE_DIR=... pytest tests/unit/test_etl_tasks_offline.py`。
- 数据库连通性:`python scripts/test_db_connection.py --dsn postgresql://... --query "SELECT 1"`。
## 其他提示
- `.env.example` 列出了所有常用配置;`config/defaults.py` 记录默认值与任务窗口配置。
- `loaders/ods/generic.py` 支持定义主键/列名即可落 ODS`tasks/manual_ingest_task.py` 可将归档 JSON 快速灌入对应 ODS 表。
- 需要新增任务时,在 `tasks/` 中实现并在 `orchestration/task_registry.py` 注册即可复用调度能力。

9
app/etl_busy.py Normal file
View File

@@ -0,0 +1,9 @@
# app/etl_busy.py
def run():
"""
忙时抓取逻辑。
TODO: 这里写具体抓取流程API 调用 / 网页解析 / 写入 PostgreSQL 等)
"""
print("Running busy-period ETL...")
# 示例:后续在这里接 PostgreSQL 或 HTTP 抓取
# ...

8
app/etl_idle.py Normal file
View File

@@ -0,0 +1,8 @@
# app/etl_idle.py
def run():
"""
闲时抓取逻辑。
可以做全量同步、大批量历史修正等。
"""
print("Running idle-period ETL...")
# ...

31
app/runner.py Normal file
View File

@@ -0,0 +1,31 @@
# app/runner.py
import argparse
from datetime import datetime
from . import etl_busy, etl_idle
def main():
parser = argparse.ArgumentParser(description="Feiqiu ETL Runner")
parser.add_argument(
"--mode",
choices=["busy", "idle"],
required=True,
help="ETL mode: busy or idle",
)
args = parser.parse_args()
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
print(f"[{now}] Start ETL mode={args.mode}")
if args.mode == "busy":
etl_busy.run()
else:
etl_idle.run()
print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] ETL finished.")
if __name__ == "__main__":
main()

View File

@@ -1,14 +1,11 @@
# 数据库配置
PG_DSN=postgresql://local-Python:Neo-local-1991125@localhost:5432/LLZQ
# PG_HOST=localhost
# PG_PORT=5432
# PG_NAME=LLZQ
# PG_USER=local-Python
# PG_PASSWORD=your_password_here
# 数据库配置(真实库)
PG_DSN=postgresql://local-Python:Neo-local-1991125@100.64.0.4:5432/LLZQ-test
PG_CONNECT_TIMEOUT=10
# 如需拆分配置PG_HOST=... PG_PORT=... PG_NAME=... PG_USER=... PG_PASSWORD=...
# API配置(抓取时,非球的一些配置)
API_BASE=https://api.example.com # 非球URL前缀
API_TOKEN=your_token_here # 登录Token
# API配置(如需走真实接口再填写)
API_BASE=https://api.example.com
API_TOKEN=your_token_here
# API_TIMEOUT=20
# API_PAGE_SIZE=200
# API_RETRY_MAX=3
@@ -20,19 +17,37 @@ STORE_ID=2790685415443269
# SCHEMA_ETL=etl_admin
# 路径配置
EXPORT_ROOT=r"D:\LLZQ\DB\export",
LOG_ROOT=r"D:\LLZQ\DB\logs",
EXPORT_ROOT=C:\dev\LLTQ\export\JSON
LOG_ROOT=C:\dev\LLTQ\export\LOG
FETCH_ROOT=
INGEST_SOURCE_DIR=
WRITE_PRETTY_JSON=false
PGCLIENTENCODING=utf8
# ETL配置
OVERLAP_SECONDS=120 # 为了防止边界遗漏,会往前“回拨”一点的冗余秒数
OVERLAP_SECONDS=120
WINDOW_BUSY_MIN=30
WINDOW_IDLE_MIN=180
IDLE_START=04:00
IDLE_END=16:00
ALLOW_EMPTY_ADVANCE=true
ALLOW_EMPTY_RESULT_ADVANCE=true
# 清洗配置
LOG_UNKNOWN_FIELDS=true
HASH_ALGO=sha1
STRICT_NUMERIC=true
ROUND_MONEY_SCALE=2
# 测试/离线模式(真实库联调建议 ONLINE
TEST_MODE=ONLINE
TEST_JSON_ARCHIVE_DIR=tests/source-data-doc
TEST_JSON_TEMP_DIR=/tmp/etl_billiards_json_tmp
# 测试数据库
TEST_DB_DSN=postgresql://local-Python:Neo-local-1991125@100.64.0.4:5432/LLZQ-test
# ODS <20>ؽ<EFBFBD><D8BD>ű<EFBFBD><C5B1><EFBFBD><EFBFBD>ã<EFBFBD><C3A3><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ã<EFBFBD>
JSON_DOC_DIR=C:\dev\LLTQ\export\test-json-doc
ODS_INCLUDE_FILES=
ODS_DROP_SCHEMA_FIRST=true

View File

@@ -1,10 +1,11 @@
# 数据库配置
PG_DSN=postgresql://user:password@localhost:5432/LLZQ
PG_DSN=postgresql://user:password@localhost:5432/....
PG_HOST=localhost
PG_PORT=5432
PG_NAME=LLZQ
PG_USER=local-Python
PG_PASSWORD=your_password_here
PG_CONNECT_TIMEOUT=10
# API配置
API_BASE=https://api.example.com
@@ -12,6 +13,7 @@ API_TOKEN=your_token_here
API_TIMEOUT=20
API_PAGE_SIZE=200
API_RETRY_MAX=3
API_RETRY_BACKOFF=[1,2,4]
# 应用配置
STORE_ID=2790685415443269
@@ -22,6 +24,11 @@ SCHEMA_ETL=etl_admin
# 路径配置
EXPORT_ROOT=/path/to/export
LOG_ROOT=/path/to/logs
FETCH_ROOT=/path/to/json_fetch
INGEST_SOURCE_DIR=
WRITE_PRETTY_JSON=false
MANIFEST_NAME=manifest.json
INGEST_REPORT_NAME=ingest_report.json
# ETL配置
OVERLAP_SECONDS=120
@@ -29,10 +36,24 @@ WINDOW_BUSY_MIN=30
WINDOW_IDLE_MIN=180
IDLE_START=04:00
IDLE_END=16:00
ALLOW_EMPTY_ADVANCE=true
ALLOW_EMPTY_RESULT_ADVANCE=true
# 清洗配置
LOG_UNKNOWN_FIELDS=true
HASH_ALGO=sha1
STRICT_NUMERIC=true
ROUND_MONEY_SCALE=2
# 测试/离线模式
TEST_MODE=ONLINE
TEST_JSON_ARCHIVE_DIR=tests/source-data-doc
TEST_JSON_TEMP_DIR=/tmp/etl_billiards_json_tmp
# 测试数据库(可选:若设置则单元测试连入此 DSN
TEST_DB_DSN=
# ODS <20>ؽ<EFBFBD><D8BD>ű<EFBFBD><C5B1><EFBFBD><EFBFBD>ã<EFBFBD><C3A3><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ã<EFBFBD>
JSON_DOC_DIR=C:\dev\LLTQ\export\test-json-doc
ODS_INCLUDE_FILES=
ODS_DROP_SCHEMA_FIRST=true

View File

@@ -1,7 +1,40 @@
# -*- coding: UTF-8 -*-
"""Simple PostgreSQL connectivity smoke-checker."""
import os
import sys
import psycopg2
from psycopg2 import OperationalError
# Filename : helloworld.py
# author by : www.runoob.com
# 该实例输出 Hello World!
print('Hello World!')
DEFAULT_DSN = os.environ.get(
"PG_DSN", "postgresql://local-Python:Neo-local-1991125@100.64.0.4:5432/LLZQ-test"
)
DEFAULT_TIMEOUT = max(1, min(int(os.environ.get("PG_CONNECT_TIMEOUT", 10)), 20))
def check_postgres_connection(dsn: str, timeout: int = DEFAULT_TIMEOUT) -> bool:
"""Return True if connection succeeds; print diagnostics otherwise."""
try:
conn = psycopg2.connect(dsn, connect_timeout=timeout)
with conn:
with conn.cursor() as cur:
cur.execute("SELECT 1;")
_ = cur.fetchone()
print(f"PostgreSQL 连接成功 (timeout={timeout}s)")
return True
except OperationalError as exc:
print("PostgreSQL 连接失败OperationalError", exc)
except Exception as exc: # pragma: no cover - defensive
print("PostgreSQL 连接失败(其他异常):", exc)
return False
if __name__ == "__main__":
dsn = sys.argv[1] if len(sys.argv) > 1 else DEFAULT_DSN
if not dsn:
print("缺少 DSN请传入参数或设置 PG_DSN 环境变量。")
sys.exit(2)
ok = check_postgres_connection(dsn)
if not ok:
sys.exit(1)

View File

@@ -71,7 +71,7 @@ cp .env.example .env
```bash
# 数据库
PG_DSN=postgresql://user:password@localhost:5432/LLZQ
PG_DSN=postgresql://user:password@localhost:5432/....
# API
API_BASE=https://api.example.com
@@ -114,13 +114,13 @@ run_etl.bat --tasks ORDERS
- 日志目录:使用 `LOG_ROOT` 指定,例如
```bash
ls -la D:\LLZQ\DB\logs/
ls -la C:\dev\LLTQ\export\LOG/
```
- 导出目录:使用 `EXPORT_ROOT` 指定,例如
```bash
ls -la D:\LLZQ\DB\export/
ls -la C:\dev\LLTQ\export\JSON/
```
---
@@ -152,10 +152,10 @@ python -m cli.main --dry-run --tasks ORDERS
```json
{
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"python.formatting.provider": "black",
"python.testing.pytestEnabled": true
"python.linting.enabled": true,
"python.linting.pylintEnabled": true,
"python.formatting.provider": "black",
"python.testing.pytestEnabled": true
}
```
@@ -191,6 +191,70 @@ pytest --cov=. --cov-report=html
- `tests/unit/test_parsers.py` 解析器单元测试
- `tests/integration/test_database.py` 数据库集成测试
#### 3.3.1 测试模式ONLINE / OFFLINE
- `TEST_MODE=ONLINE`(默认)时,测试会模拟实时 API完整执行 E/T/L。
- `TEST_MODE=OFFLINE` 时,测试改为从 `TEST_JSON_ARCHIVE_DIR` 指定的归档 JSON 中读取数据,仅做 Transform + Load适合验证本地归档数据是否仍可回放。
- `TEST_JSON_ARCHIVE_DIR`:离线 JSON 归档目录(示例:`tests/source-data-doc` 或 CI 产出的快照)。
- `TEST_JSON_TEMP_DIR`:测试生成的临时 JSON 输出目录,便于隔离每次运行的数据。
- `TEST_DB_DSN`:可选,若设置则单元测试会连接到此 PostgreSQL DSN实打实执行写库留空时测试使用内存伪库避免依赖数据库。
示例命令:
```bash
# 在线模式覆盖所有任务
TEST_MODE=ONLINE pytest tests/unit/test_etl_tasks_online.py
# 离线模式使用归档 JSON 覆盖所有任务
TEST_MODE=OFFLINE TEST_JSON_ARCHIVE_DIR=tests/source-data-doc pytest tests/unit/test_etl_tasks_offline.py
# 使用脚本按需组合参数(示例:在线 + 仅订单用例)
python scripts/run_tests.py --suite online --mode ONLINE --keyword ORDERS
# 使用脚本连接真实测试库并回放离线模式
python scripts/run_tests.py --suite offline --mode OFFLINE --db-dsn postgresql://user:pwd@localhost:5432/testdb
# 使用“指令仓库”中的预置命令
python scripts/run_tests.py --preset offline_realdb
python scripts/run_tests.py --list-presets # 查看或自定义 scripts/test_presets.py
```
#### 3.3.2 脚本化测试组合(`run_tests.py` / `test_presets.py`
- `scripts/run_tests.py` 是 pytest 的统一入口:自动把项目根目录加入 `sys.path`,并提供 `--suite online/offline/integration`、`--tests`(自定义路径)、`--mode`、`--db-dsn`、`--json-archive`、`--json-temp`、`--keyword/-k`、`--pytest-args`、`--env KEY=VALUE` 等参数,可以像搭积木一样自由组合;
- `--preset foo` 会读取 `scripts/test_presets.py` 内 `PRESETS["foo"]` 的配置,并叠加到当前命令;`--list-presets` 与 `--dry-run` 可用来审阅或仅打印命令;
- 直接执行 `python scripts/test_presets.py` 可依次运行 `AUTO_RUN_PRESETS` 中列出的预置;传入 `--preset x --dry-run` 则只打印对应命令。
`test_presets.py` 充当“指令仓库”。每个预置都是一个字典,常用字段解释如下:
| 字段 | 作用 |
| ---------------------------- | ------------------------------------------------------------------ |
| `suite` | 复用 `run_tests.py` 内置套件online/offline/integration可多选 |
| `tests` | 追加任意 pytest 路径,例如 `tests/unit/test_config.py` |
| `mode` | 覆盖 `TEST_MODE`ONLINE / OFFLINE |
| `db_dsn` | 覆盖 `TEST_DB_DSN`,用于连入真实测试库 |
| `json_archive` / `json_temp` | 配置离线 JSON 归档与临时目录 |
| `keyword` | 映射到 `pytest -k`,用于关键字过滤 |
| `pytest_args` | 附加 pytest 参数,例 `-vv --maxfail=1` |
| `env` | 额外环境变量列表,如 `["STORE_ID=123"]` |
| `preset_meta` | 说明性文字,便于描述场景 |
示例:`offline_realdb` 预置会设置 `TEST_MODE=OFFLINE`、指定 `tests/source-data-doc` 为归档目录,并通过 `db_dsn` 连到测试库。执行 `python scripts/run_tests.py --preset offline_realdb` 或 `python scripts/test_presets.py --preset offline_realdb` 即可复用该组合保证本地、CI 与生产回放脚本一致。
#### 3.3.3 数据库连通性快速检查
`python scripts/test_db_connection.py` 提供最轻量的 PostgreSQL 连通性检测:默认使用 `TEST_DB_DSN`(也可传 `--dsn`),尝试连接并执行 `SELECT 1 AS ok`(可通过 `--query` 自定义)。典型用途:
```bash
# 读取 .env/环境变量中的 TEST_DB_DSN
python scripts/test_db_connection.py
# 临时指定 DSN并检查任务配置表
python scripts/test_db_connection.py --dsn postgresql://user:pwd@host:5432/.... --query "SELECT count(*) FROM etl_admin.etl_task"
```
脚本返回 0 代表连接与查询成功;若返回非 0可结合第 8 章“常见问题排查”的数据库章节(网络、防火墙、账号权限等)先定位问题,再运行完整 ETL。
---
## 4. 项目结构与文件说明
@@ -277,6 +341,8 @@ etl_billiards/
│ │ ├── __init__.py
│ │ ├── test_config.py
│ │ └── test_parsers.py
│ ├── testdata_json/ # 清洗入库用的测试Json文件
│ │ └── XX.json
│ └── integration/ # 集成测试
│ ├── __init__.py
│ └── test_database.py
@@ -287,29 +353,29 @@ etl_billiards/
### 4.2 各模块职责概览
- **config/**
- **config/**
- 统一配置入口,支持默认值、环境变量、命令行参数三层覆盖。
- **database/**
- **database/**
- 封装 PostgreSQL 连接与批量操作插入、更新、Upsert 等)。
- **api/**
- **api/**
- 对上游业务 API 的 HTTP 调用进行统一封装,支持重试、分页与超时控制。
- **models/**
- **models/**
- 提供类型解析器(时间戳、金额、整数等)与业务级数据校验器。
- **loaders/**
- **loaders/**
- 提供事实表与维度表的加载逻辑(包含批量 Upsert、统计写入结果等
- **scd/**
- **scd/**
- 维度型数据的 SCD2 历史管理(有效期、版本标记等)。
- **quality/**
- **quality/**
- 质量检查策略,例如余额一致性、记录数量对齐等。
- **orchestration/**
- **orchestration/**
- 任务调度、任务注册、游标管理(增量窗口)、运行记录追踪。
- **tasks/**
- **tasks/**
- 具体业务任务(订单、支付、会员等),封装了从“取数 → 处理 → 写库 → 记录结果”的完整流程。
- **cli/**
- **cli/**
- 命令行入口,解析参数并启动调度流程。
- **utils/**
- **utils/**
- 杂项工具函数。
- **tests/**
- **tests/**
- 单元测试与集成测试代码。
---
@@ -358,32 +424,39 @@ etl_billiards/
### 5.2 各层职责(当前设计)
- **CLI 层 (`cli/`)**
- 解析命令行参数指定任务列表、Dry-run、覆盖配置项等
- 初始化配置与日志后交由编排层执行。
- **编排层 (`orchestration/`)**
- `scheduler.py`:根据配置与 CLI 参数选择需要执行的任务,控制执行顺序和并行策略。
- `task_registry.py`:提供任务注册表,按任务代码创建任务实例(工厂模式)。
- `cursor_manager.py`:管理增量游标(时间窗口 / ID 游标)。
- `run_tracker.py`:记录每次任务运行的状态、统计信息和错误信息。
- **任务层 (`tasks/`)**
- `base_task.py`:定义任务执行模板流程(模板方法模式),包括获取窗口、调用上游、解析 / 校验、写库、更新游标等。
- `orders_task.py` / `payments_task.py` / `members_task.py`:实现具体任务逻辑(订单、支付、会员)。
- **加载器 / SCD / 质量层**
- `loaders/`:根据目标表封装 Upsert / Insert / Update 逻辑。
- `scd/scd2_handler.py`:为维度表提供 SCD2 历史管理能力。
- `quality/`:执行数据质量检查,如余额对账。
- **模型层 (`models/`)**
- `parsers.py`:负责数据类型转换(字符串 → 时间戳、Decimal、int 等)。
- `validators.py`:执行字段级和记录级的数据校验。
- **API 层 (`api/client.py`)**
- 封装 HTTP 调用,处理重试、超时及分页。
- **数据库层 (`database/`)**
- 管理数据库连接及上下文。
- 提供批量插入 / 更新 / Upsert 操作接口。
@@ -422,6 +495,19 @@ etl_billiards/
- API 请求失败时按配置进行重试,超过重试次数记录错误并终止该任务。
- 所有错误被记录到日志和运行追踪表,便于事后排查。
### 5.6 ODS + DWD 双阶段策略(新增)
为了支撑回溯/重放与后续 DWD 宽表构建,项目新增了 `billiards_ods` Schema 以及一组专门的 ODS 任务/Loader
- **ODS 表**`billiards_ods.ods_order_settle`、`ods_table_use_detail`、`ods_assistant_ledger`、`ods_assistant_abolish`、`ods_goods_ledger`、`ods_payment`、`ods_refund`、`ods_coupon_verify`、`ods_member`、`ods_member_card`、`ods_package_coupon`、`ods_inventory_stock`、`ods_inventory_change`。每条记录都会保存 `store_id + 源主键 + payload JSON + fetched_at + source_endpoint` 等信息。
- **通用 Loader**`loaders/ods/generic.py::GenericODSLoader` 统一封装了 `INSERT ... ON CONFLICT ...` 与批量写入逻辑,调用方只需提供列名与主键列即可。
- **ODS 任务**`tasks/ods_tasks.py` 内通过 `OdsTaskSpec` 定义了一组任务(`ODS_ORDER_SETTLE`、`ODS_PAYMENT`、`ODS_ASSISTANT_LEDGER` 等),并在 `TaskRegistry` 中自动注册,可直接通过 `python -m cli.main --tasks ODS_ORDER_SETTLE,ODS_PAYMENT` 执行。
- **双阶段链路**
1. 阶段 1ODS调用 API/离线归档 JSON将原始记录写入 ODS 表,保留分页、抓取时间、来源文件等元数据。
2. 阶段 2DWD/DIM后续订单、支付、券等事实任务将改为从 ODS 读取 payload经过解析/校验后写入 `billiards.fact_*`、`dim_*` 表,避免重复拉取上游接口。
> 新增的单元测试 `tests/unit/test_ods_tasks.py` 覆盖了 `ODS_ORDER_SETTLE`、`ODS_PAYMENT` 的入库路径,可作为扩展其他 ODS 任务的模板。
---
## 6. 迁移指南(从旧脚本到当前项目)
@@ -430,22 +516,23 @@ etl_billiards/
### 6.1 核心功能映射示意
| 旧版本函数 / 类 | 新版本位置 | 说明 |
|---------------------------|--------------------------------------------------------|----------------|
| `DEFAULTS` 字典 | `config/defaults.py` | 配置默认值 |
| `build_config()` | `config/settings.py::AppConfig.load()` | 配置加载 |
| `Pg` 类 | `database/connection.py::DatabaseConnection` | 数据库连接 |
| `http_get_json()` | `api/client.py::APIClient.get()` | API 请求 |
| `paged_get()` | `api/client.py::APIClient.get_paginated()` | 分页请求 |
| `parse_ts()` | `models/parsers.py::TypeParser.parse_timestamp()` | 时间解析 |
| `upsert_fact_order()` | `loaders/facts/order.py::OrderLoader.upsert_orders()` | 订单加载 |
| `scd2_upsert()` | `scd/scd2_handler.py::SCD2Handler.upsert()` | SCD2 处理 |
| `run_task_orders()` | `tasks/orders_task.py::OrdersTask.execute()` | 订单任务 |
| `main()` | `cli/main.py::main()` | 主入口 |
| 旧版本函数 / 类 | 新版本位置 | 说明 |
| --------------------- | ----------------------------------------------------- | ---------- |
| `DEFAULTS` 字典 | `config/defaults.py` | 配置默认值 |
| `build_config()` | `config/settings.py::AppConfig.load()` | 配置加载 |
| `Pg` 类 | `database/connection.py::DatabaseConnection` | 数据库连接 |
| `http_get_json()` | `api/client.py::APIClient.get()` | API 请求 |
| `paged_get()` | `api/client.py::APIClient.get_paginated()` | 分页请求 |
| `parse_ts()` | `models/parsers.py::TypeParser.parse_timestamp()` | 时间解析 |
| `upsert_fact_order()` | `loaders/facts/order.py::OrderLoader.upsert_orders()` | 订单加载 |
| `scd2_upsert()` | `scd/scd2_handler.py::SCD2Handler.upsert()` | SCD2 处理 |
| `run_task_orders()` | `tasks/orders_task.py::OrdersTask.execute()` | 订单任务 |
| `main()` | `cli/main.py::main()` | 主入口 |
### 6.2 典型迁移步骤
1. **配置迁移**
- 原来在 `DEFAULTS` 或脚本内硬编码的配置,迁移到 `.env` 与 `config/defaults.py`。
- 使用 `AppConfig.load()` 统一获取配置。
@@ -462,6 +549,7 @@ etl_billiards/
对比新旧版本导出的数据表和日志,确认一致性。
3. **自定义逻辑迁移**
- 原脚本中的自定义清洗逻辑 → 放入相应 `loaders/` 或任务类中。
- 自定义任务 → 在 `tasks/` 中实现并在 `task_registry` 中注册。
- 自定义 API 调用 → 扩展 `api/client.py` 或单独封装服务类。
@@ -613,3 +701,137 @@ class MyLoader(BaseLoader):
- 本文已合并原有的快速开始、项目结构、架构说明、迁移指南等内容,可作为当前项目的统一说明文档。
- 如需在此基础上拆分多份文档,可按章节拆出,例如「快速开始」「架构设计」「迁移指南」「开发扩展」等。
## 11. 运行/调试模式说明
- 生产环境仅保留“任务模式”:通过调度/CLI 执行注册的任务ETL/ODS不使用调试脚本。
- 开发/调试可使用的辅助脚本(上线前可删除或禁用):
- `python -m etl_billiards.scripts.rebuild_ods_from_json`:从本地 JSON 目录重建 `billiards_ods`,用于离线初始化/验证。环境变量:`PG_DSN`(必填)、`JSON_DOC_DIR`(可选,默认 `C:\dev\LLTQ\export\test-json-doc`)、`INCLUDE_FILES`(逗号分隔文件名)、`DROP_SCHEMA_FIRST`(默认 true
- 如需在生产环境保留脚本,请在运维手册中明确用途和禁用条件,避免误用。
## 12. ODS 任务上线指引
- 任务注册:`etl_billiards/database/seed_ods_tasks.sql` 列出了当前启用的 ODS 任务。将其中的 `store_id` 替换为实际门店后执行:
```
psql "$PG_DSN" -f etl_billiards/database/seed_ods_tasks.sql
```
`ON CONFLICT` 会保持 enabled=true避免重复。
- 调度:确认 `etl_admin.etl_task` 中已启用所需的 ODS 任务(任务代码见 seed 脚本),调度器或 CLI `--tasks` 即可调用。
- 离线回灌:开发环境可用 `rebuild_ods_from_json` 以样例 JSON 初始化 ODS生产慎用默认按 `(source_file, record_index)` 去重。
- 测试:`pytest etl_billiards/tests/unit/test_ods_tasks.py` 覆盖核心 ODS 任务;测试时可设置 `ETL_SKIP_DOTENV=1` 跳过本地 .env 读取。
## 13. ODS 表映射总览
| ODS 表名 | 接口 Path | 数据列表路径 |
| ------------------------------------ | ---------------------------------------------------- | ----------------------------- |
| `assistant_accounts_master` | `/PersonnelManagement/SearchAssistantInfo` | data.assistantInfos |
| `assistant_service_records` | `/AssistantPerformance/GetOrderAssistantDetails` | data.orderAssistantDetails |
| `assistant_cancellation_records` | `/AssistantPerformance/GetAbolitionAssistant` | data.abolitionAssistants |
| `goods_stock_movements` | `/GoodsStockManage/QueryGoodsOutboundReceipt` | data.queryDeliveryRecordsList |
| `goods_stock_summary` | `/TenantGoods/GetGoodsStockReport` | data |
| `group_buy_packages` | `/PackageCoupon/QueryPackageCouponList` | data.packageCouponList |
| `group_buy_redemption_records` | `/Site/GetSiteTableUseDetails` | data.siteTableUseDetailsList |
| `member_profiles` | `/MemberProfile/GetTenantMemberList` | data.tenantMemberInfos |
| `member_balance_changes` | `/MemberProfile/GetMemberCardBalanceChange` | data.tenantMemberCardLogs |
| `member_stored_value_cards` | `/MemberProfile/GetTenantMemberCardList` | data.tenantMemberCards |
| `payment_transactions` | `/PayLog/GetPayLogListPage` | data |
| `platform_coupon_redemption_records` | `/Promotion/GetOfflineCouponConsumePageList` | data |
| `recharge_settlements` | `/Site/GetRechargeSettleList` | data.settleList |
| `refund_transactions` | `/Order/GetRefundPayLogList` | data |
| `settlement_records` | `/Site/GetAllOrderSettleList` | data.settleList |
| `settlement_ticket_details` | `/Order/GetOrderSettleTicketNew` | (整包原始 JSON |
| `site_tables_master` | `/Table/GetSiteTables` | data.siteTables |
| `stock_goods_category_tree` | `/TenantGoodsCategory/QueryPrimarySecondaryCategory` | data.goodsCategoryList |
| `store_goods_master` | `/TenantGoods/GetGoodsInventoryList` | data.orderGoodsList |
| `store_goods_sales_records` | `/TenantGoods/GetGoodsSalesList` | data.orderGoodsLedgers |
| `table_fee_discount_records` | `/Site/GetTaiFeeAdjustList` | data.taiFeeAdjustInfos |
| `table_fee_transactions` | `/Site/GetSiteTableOrderDetails` | data.siteTableUseDetailsList |
| `tenant_goods_master` | `/TenantGoods/QueryTenantGoods` | data.tenantGoodsList |
## 14. ODS 相关环境变量/默认值
- `.env` / 环境变量:
- `JSON_DOC_DIR`ODS 样例 JSON 目录(开发/回灌用)
- `ODS_INCLUDE_FILES`:限定导入的文件名(逗号分隔,不含 .json
- `ODS_DROP_SCHEMA_FIRST`true/false是否重建 schema
- `ETL_SKIP_DOTENV`:测试/CI 时设为 1 跳过本地 .env 读取
- `config/defaults.py` 中 `ods` 默认值:
- `json_doc_dir`: `C:\dev\LLTQ\export\test-json-doc`
- `include_files`: `""`
- `drop_schema_first`: `True`
---
## 15. DWD 维度 “业务事件”
1. 粒度唯一、原子
- 一张 DWD 表只能有一种业务粒度,比如:
- 一条记录 = 一次结账;
- 一条记录 = 一段台费流水;
- 一条记录 = 一次助教服务;
- 一条记录 = 一次会员余额变动。
- 表里面不能又混“订单头”又混“订单行”,不能一部分是“汇总”,一部分是“明细”。
- 一旦粒度确定,所有字段都要跟这个粒度匹配:
- 比如“结账头表”就不要塞每一行商品明细;
- 商品明细就不要塞整单级别的总金额。
- 这是 DWD 层最重要的一条。
2. 以业务过程建模,不以 JSON 列表建模
- 先画清楚你真实的业务链路:
- 开台 / 换台 / 关台 → 台费流水
- 助教上桌 → 助教服务流水 / 废除事件
- 点单 → 商品销售流水
- 充值 / 消费 → 余额变更 / 充值单
- 结账 → 结账头表 + 支付流水 / 退款流水
- 团购 / 平台券 → 核销流水
3. 主键明确、外键统一
- 每张 DWD 表必须有业务主键(哪怕是接口给的 id不要依赖数据库自增。
- 所有“同一概念”的字段必须统一命名、统一含义:
- 门店:统一叫 site_id都对应 siteProfile.id
- 会员:统一叫 member_id 对应 member_profiles.idsystem_member_id 单独一列;
- 台桌:统一 table_id 对应 site_tables_master.id
- 结账:统一 order_settle_id
- 订单:统一 order_trade_no 等。
- 否则后面 DWS、AI 要把表拼起来会非常痛苦。
4. 保留明细,不做过度汇总
- DWD 层的事实表原则上只做“明细级”的数据:
- 不要在 DWD 就把“日汇总、周汇总、月汇总”算出来,那是 DWS 的事;
- 也不要把多个事件折成一行(例如一张表同时放日汇总+单笔流水)。
- 需要聚合时,再在 DWS 做主题宽表:
- dws_member_day_profile、dws_site_day_summary 等。
- DWD 只负责细颗粒度的真相。
5. 统一清洗、标准化,但保持可追溯
- 在 DWD 层一定要做的清洗:
- 类型转换:字符串时间 → 时间类型,金额统一为 decimal布尔统一为 0/1
- 单位统一:秒 / 分钟、元 / 分都统一;
- 枚举标准化:状态码、类型码在 DWD 里就定死含义,必要时建枚举维表。
- 同时要保证:
- 每条 DWD 记录都能追溯回 ODS
- 保留源系统主键;
- 保留原始时间 / 原始金额字段(不要覆盖掉)。
6. 扁平化、去嵌套
- JSON 里常见结构是:分页壳 + 头 + 明细数组 + 各种嵌套对象siteProfile、tableProfile、goodsLedgers…
- DWD 的原则是:
- 去掉分页壳;
- 把“数组”拆成子表(头表 / 行表);
- 把重复出现的 profile 抽出去做维度表(门店、台、商品、会员……)。
- 目标是DWD 表都是二维表结构,不存复杂嵌套 JSON。
7. 模型长期稳定,可扩展
- DWD 的表结构要尽可能稳定,新增需求尽量通过:
- 加字段;
- 新建事实表 / 维度表;
- 在 DWS 做派生指标;
- 而不是频繁重构已有 DWD 表结构。
- 这点跟你后面要喂给 LLM 也很相关AI 配的 prompt、schema 理解都要尽量少改。

View File

@@ -1,95 +1,256 @@
# -*- coding: utf-8 -*-
"""API客户端"""
"""API客户端:统一封装 POST/重试/分页与列表提取逻辑。"""
from __future__ import annotations
from typing import Iterable, Sequence, Tuple
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
DEFAULT_BROWSER_HEADERS = {
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/json",
"Origin": "https://pc.ficoo.vip",
"Referer": "https://pc.ficoo.vip/",
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
),
"Accept-Language": "zh-CN,zh;q=0.9",
"sec-ch-ua": '"Google Chrome";v="141", "Not?A_Brand";v="8", "Chromium";v="141"',
"sec-ch-ua-platform": '"Windows"',
"sec-ch-ua-mobile": "?0",
"sec-fetch-site": "same-origin",
"sec-fetch-mode": "cors",
"sec-fetch-dest": "empty",
"priority": "u=1, i",
"X-Requested-With": "XMLHttpRequest",
"DNT": "1",
}
DEFAULT_LIST_KEYS: Tuple[str, ...] = (
"list",
"rows",
"records",
"items",
"dataList",
"data_list",
"tenantMemberInfos",
"tenantMemberCardLogs",
"tenantMemberCards",
"settleList",
"orderAssistantDetails",
"assistantInfos",
"siteTables",
"taiFeeAdjustInfos",
"siteTableUseDetailsList",
"tenantGoodsList",
"packageCouponList",
"queryDeliveryRecordsList",
"goodsCategoryList",
"orderGoodsList",
"orderGoodsLedgers",
)
class APIClient:
"""HTTP API客户端"""
def __init__(self, base_url: str, token: str = None, timeout: int = 20,
retry_max: int = 3, headers_extra: dict = None):
self.base_url = base_url.rstrip("/")
self.token = token
"""HTTP API 客户端(默认使用 POST + JSON 请求体)"""
def __init__(
self,
base_url: str,
token: str | None = None,
timeout: int = 20,
retry_max: int = 3,
headers_extra: dict | None = None,
):
self.base_url = (base_url or "").rstrip("/")
self.token = self._normalize_token(token)
self.timeout = timeout
self.retry_max = retry_max
self.headers_extra = headers_extra or {}
self._session = None
def _get_session(self):
"""获取或创建会话"""
self._session: requests.Session | None = None
# ------------------------------------------------------------------ HTTP 基础
def _get_session(self) -> requests.Session:
"""获取或创建带重试的 Session。"""
if self._session is None:
self._session = requests.Session()
retries = max(0, int(self.retry_max) - 1)
retry = Retry(
total=None,
connect=retries,
read=retries,
status=retries,
allowed_methods=frozenset(["GET"]),
allowed_methods=frozenset(["GET", "POST"]),
status_forcelist=(429, 500, 502, 503, 504),
backoff_factor=1.0,
backoff_factor=0.5,
respect_retry_after_header=True,
raise_on_status=False,
)
adapter = HTTPAdapter(max_retries=retry)
self._session.mount("http://", adapter)
self._session.mount("https://", adapter)
if self.headers_extra:
self._session.headers.update(self.headers_extra)
self._session.headers.update(self._build_headers())
return self._session
def get(self, endpoint: str, params: dict = None) -> dict:
"""执行GET请求"""
def get(self, endpoint: str, params: dict | None = None) -> dict:
"""
兼容旧名的请求入口(实际以 POST JSON 方式请求)。
"""
return self._post_json(endpoint, params)
def _post_json(self, endpoint: str, payload: dict | None = None) -> dict:
if not self.base_url:
raise ValueError("API base_url 未配置")
url = f"{self.base_url}/{endpoint.lstrip('/')}"
headers = {"Authorization": self.token} if self.token else {}
headers.update(self.headers_extra)
sess = self._get_session()
resp = sess.get(url, headers=headers, params=params, timeout=self.timeout)
resp = sess.post(url, json=payload or {}, timeout=self.timeout)
resp.raise_for_status()
return resp.json()
def get_paginated(self, endpoint: str, params: dict, page_size: int = 200,
page_field: str = "pageIndex", size_field: str = "pageSize",
data_path: tuple = ("data",), list_key: str = None) -> tuple:
"""分页获取数据"""
records, pages_meta = [], []
page = 1
data = resp.json()
self._ensure_success(data)
return data
def _build_headers(self) -> dict:
headers = dict(DEFAULT_BROWSER_HEADERS)
headers.update(self.headers_extra)
if self.token:
headers["Authorization"] = self.token
return headers
@staticmethod
def _normalize_token(token: str | None) -> str | None:
if not token:
return None
t = str(token).strip()
if not t.lower().startswith("bearer "):
t = f"Bearer {t}"
return t
@staticmethod
def _ensure_success(payload: dict):
"""API 返回 code 非 0 时主动抛错,便于上层重试/记录。"""
if isinstance(payload, dict) and "code" in payload:
code = payload.get("code")
if code not in (0, "0", None):
msg = payload.get("msg") or payload.get("message") or ""
raise ValueError(f"API 返回错误 code={code} msg={msg}")
# ------------------------------------------------------------------ 分页
def iter_paginated(
self,
endpoint: str,
params: dict | None,
page_size: int | None = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | Sequence[str] | None = None,
page_start: int = 1,
page_end: int | None = None,
) -> Iterable[tuple[int, list, dict, dict]]:
"""
分页迭代器:逐页拉取数据并产出 (page_no, records, request_params, raw_response)。
page_size=None 时不附带分页参数,仅拉取一次。
"""
base_params = dict(params or {})
page = page_start
while True:
p = dict(params)
p[page_field] = page
p[size_field] = page_size
obj = self.get(endpoint, p)
# 解析数据路径
cur = obj
for k in data_path:
if isinstance(cur, dict) and k in cur:
cur = cur[k]
if list_key:
cur = (cur or {}).get(list_key, [])
if not isinstance(cur, list):
cur = []
records.extend(cur)
if len(cur) == 0:
page_params = dict(base_params)
if page_size is not None:
page_params[page_field] = page
page_params[size_field] = page_size
payload = self._post_json(endpoint, page_params)
records = self._extract_list(payload, data_path, list_key)
yield page, records, page_params, payload
if page_size is None:
break
pages_meta.append({"page": page, "request": p, "response": obj})
if len(cur) < page_size:
if page_end is not None and page >= page_end:
break
if len(records) < (page_size or 0):
break
if len(records) == 0:
break
page += 1
def get_paginated(
self,
endpoint: str,
params: dict,
page_size: int | None = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | Sequence[str] | None = None,
page_start: int = 1,
page_end: int | None = None,
) -> tuple[list, list]:
"""分页获取数据并将所有记录汇总在一个列表中。"""
records, pages_meta = [], []
for page_no, page_records, request_params, response in self.iter_paginated(
endpoint=endpoint,
params=params,
page_size=page_size,
page_field=page_field,
size_field=size_field,
data_path=data_path,
list_key=list_key,
page_start=page_start,
page_end=page_end,
):
records.extend(page_records)
pages_meta.append(
{"page": page_no, "request": request_params, "response": response}
)
return records, pages_meta
# ------------------------------------------------------------------ 响应解析
@classmethod
def _extract_list(
cls, payload: dict | list, data_path: tuple, list_key: str | Sequence[str] | None
) -> list:
"""根据 data_path/list_key 提取列表结构,兼容常见字段名。"""
cur: object = payload
if isinstance(cur, list):
return cur
for key in data_path:
if isinstance(cur, dict):
cur = cur.get(key)
else:
cur = None
if cur is None:
break
if isinstance(cur, list):
return cur
if isinstance(cur, dict):
if list_key:
keys = (list_key,) if isinstance(list_key, str) else tuple(list_key)
for k in keys:
if isinstance(cur.get(k), list):
return cur[k]
for k in DEFAULT_LIST_KEYS:
if isinstance(cur.get(k), list):
return cur[k]
for v in cur.values():
if isinstance(v, list):
return v
return []

View File

@@ -0,0 +1,74 @@
# -*- coding: utf-8 -*-
"""本地 JSON 客户端,模拟 APIClient 的分页接口,从落盘的 JSON 回放数据。"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Iterable, Tuple
from api.client import APIClient
from utils.json_store import endpoint_to_filename
class LocalJsonClient:
"""
读取 RecordingAPIClient 生成的 JSON提供 iter_paginated/get_paginated 接口。
"""
def __init__(self, base_dir: str | Path):
self.base_dir = Path(base_dir)
if not self.base_dir.exists():
raise FileNotFoundError(f"JSON 目录不存在: {self.base_dir}")
def iter_paginated(
self,
endpoint: str,
params: dict | None,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | None = None,
) -> Iterable[Tuple[int, list, dict, dict]]:
file_path = self.base_dir / endpoint_to_filename(endpoint)
if not file_path.exists():
raise FileNotFoundError(f"未找到匹配的 JSON 文件: {file_path}")
with file_path.open("r", encoding="utf-8") as fp:
payload = json.load(fp)
pages = payload.get("pages")
if not isinstance(pages, list) or not pages:
pages = [{"page": 1, "request": params or {}, "response": payload}]
for idx, page in enumerate(pages, start=1):
response = page.get("response", {})
request_params = page.get("request") or {}
page_no = page.get("page") or idx
records = APIClient._extract_list(response, data_path, list_key) # type: ignore[attr-defined]
yield page_no, records, request_params, response
def get_paginated(
self,
endpoint: str,
params: dict,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | None = None,
) -> tuple[list, list]:
records: list = []
pages_meta: list = []
for page_no, page_records, request_params, response in self.iter_paginated(
endpoint=endpoint,
params=params,
page_size=page_size,
page_field=page_field,
size_field=size_field,
data_path=data_path,
list_key=list_key,
):
records.extend(page_records)
pages_meta.append({"page": page_no, "request": request_params, "response": response})
return records, pages_meta

View File

@@ -0,0 +1,118 @@
# -*- coding: utf-8 -*-
"""包装 APIClient将分页响应落盘便于后续本地清洗。"""
from __future__ import annotations
from datetime import datetime
from pathlib import Path
from typing import Any, Iterable, Tuple
from api.client import APIClient
from utils.json_store import dump_json, endpoint_to_filename
class RecordingAPIClient:
"""
代理 APIClient在调用 iter_paginated/get_paginated 时同时把响应写入 JSON 文件。
文件名根据 endpoint 生成,写入到指定 output_dir。
"""
def __init__(
self,
base_client: APIClient,
output_dir: Path | str,
task_code: str,
run_id: int,
write_pretty: bool = False,
):
self.base = base_client
self.output_dir = Path(output_dir)
self.output_dir.mkdir(parents=True, exist_ok=True)
self.task_code = task_code
self.run_id = run_id
self.write_pretty = write_pretty
self.last_dump: dict[str, Any] | None = None
# ------------------------------------------------------------------ public API
def iter_paginated(
self,
endpoint: str,
params: dict | None,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | None = None,
) -> Iterable[Tuple[int, list, dict, dict]]:
pages: list[dict[str, Any]] = []
total_records = 0
for page_no, records, request_params, response in self.base.iter_paginated(
endpoint=endpoint,
params=params,
page_size=page_size,
page_field=page_field,
size_field=size_field,
data_path=data_path,
list_key=list_key,
):
pages.append({"page": page_no, "request": request_params, "response": response})
total_records += len(records)
yield page_no, records, request_params, response
self._dump(endpoint, params, page_size, pages, total_records)
def get_paginated(
self,
endpoint: str,
params: dict,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: tuple = ("data",),
list_key: str | None = None,
) -> tuple[list, list]:
records: list = []
pages_meta: list = []
for page_no, page_records, request_params, response in self.iter_paginated(
endpoint=endpoint,
params=params,
page_size=page_size,
page_field=page_field,
size_field=size_field,
data_path=data_path,
list_key=list_key,
):
records.extend(page_records)
pages_meta.append({"page": page_no, "request": request_params, "response": response})
return records, pages_meta
# ------------------------------------------------------------------ internal
def _dump(
self,
endpoint: str,
params: dict | None,
page_size: int,
pages: list[dict[str, Any]],
total_records: int,
):
filename = endpoint_to_filename(endpoint)
path = self.output_dir / filename
payload = {
"task_code": self.task_code,
"run_id": self.run_id,
"endpoint": endpoint,
"params": params or {},
"page_size": page_size,
"pages": pages,
"total_records": total_records,
"dumped_at": datetime.utcnow().isoformat() + "Z",
}
dump_json(path, payload, pretty=self.write_pretty)
self.last_dump = {
"file": str(path),
"endpoint": endpoint,
"pages": len(pages),
"records": total_records,
}

View File

@@ -36,13 +36,25 @@ def parse_args():
# API参数
parser.add_argument("--api-base", help="API基础URL")
parser.add_argument("--api-token", help="API令牌")
parser.add_argument("--api-token", "--token", dest="api_token", help="API令牌Bearer Token")
parser.add_argument("--api-timeout", type=int, help="API超时(秒)")
parser.add_argument("--api-page-size", type=int, help="分页大小")
parser.add_argument("--api-retry-max", type=int, help="API重试最大次数")
# 目录参数
parser.add_argument("--export-root", help="导出根目录")
parser.add_argument("--log-root", help="日志根目录")
# 抓取/清洗管线
parser.add_argument("--pipeline-flow", choices=["FULL", "FETCH_ONLY", "INGEST_ONLY"], help="流水线模式")
parser.add_argument("--fetch-root", help="抓取JSON输出根目录")
parser.add_argument("--ingest-source", help="本地清洗入库源目录")
parser.add_argument("--write-pretty-json", action="store_true", help="抓取JSON美化输出")
# 运行窗口
parser.add_argument("--idle-start", help="闲时窗口开始(HH:MM)")
parser.add_argument("--idle-end", help="闲时窗口结束(HH:MM)")
parser.add_argument("--allow-empty-advance", action="store_true", help="允许空结果推进窗口")
return parser.parse_args()
@@ -77,12 +89,32 @@ def build_cli_overrides(args) -> dict:
overrides.setdefault("api", {})["timeout_sec"] = args.api_timeout
if args.api_page_size:
overrides.setdefault("api", {})["page_size"] = args.api_page_size
if args.api_retry_max:
overrides.setdefault("api", {}).setdefault("retries", {})["max_attempts"] = args.api_retry_max
# 目录
if args.export_root:
overrides.setdefault("io", {})["export_root"] = args.export_root
if args.log_root:
overrides.setdefault("io", {})["log_root"] = args.log_root
# 抓取/清洗管线
if args.pipeline_flow:
overrides.setdefault("pipeline", {})["flow"] = args.pipeline_flow.upper()
if args.fetch_root:
overrides.setdefault("pipeline", {})["fetch_root"] = args.fetch_root
if args.ingest_source:
overrides.setdefault("pipeline", {})["ingest_source_dir"] = args.ingest_source
if args.write_pretty_json:
overrides.setdefault("io", {})["write_pretty_json"] = True
# 运行窗口
if args.idle_start:
overrides.setdefault("run", {}).setdefault("idle_window", {})["start"] = args.idle_start
if args.idle_end:
overrides.setdefault("run", {}).setdefault("idle_window", {})["end"] = args.idle_end
if args.allow_empty_advance:
overrides.setdefault("run", {})["allow_empty_result_advance"] = True
# 任务
if args.tasks:

View File

@@ -1,12 +1,12 @@
# -*- coding: utf-8 -*-
"""配置默认值"""
"""配置默认值定义"""
DEFAULTS = {
"app": {
"timezone": "Asia/Taipei",
"store_id": "",
"schema_oltp": "billiards",
"schema_etl": "etl_admin",
"schema_etl": "etl_admin",
},
"db": {
"dsn": "",
@@ -15,20 +15,21 @@ DEFAULTS = {
"name": "",
"user": "",
"password": "",
"connect_timeout_sec": 5,
"connect_timeout_sec": 20,
"batch_size": 1000,
"session": {
"timezone": "Asia/Taipei",
"statement_timeout_ms": 30000,
"lock_timeout_ms": 5000,
"idle_in_tx_timeout_ms": 600000
"idle_in_tx_timeout_ms": 600000,
},
},
"api": {
"base_url": None,
"base_url": "https://pc.ficoo.vip/apiprod/admin/v1",
"token": None,
"timeout_sec": 20,
"page_size": 200,
"params": {},
"retries": {
"max_attempts": 3,
"backoff_sec": [1, 2, 4],
@@ -37,9 +38,19 @@ DEFAULTS = {
},
"run": {
"tasks": [
"PRODUCTS", "TABLES", "MEMBERS", "ASSISTANTS", "PACKAGES_DEF",
"ORDERS", "PAYMENTS", "REFUNDS", "COUPON_USAGE", "INVENTORY_CHANGE",
"TOPUPS", "TABLE_DISCOUNT", "ASSISTANT_ABOLISH",
"PRODUCTS",
"TABLES",
"MEMBERS",
"ASSISTANTS",
"PACKAGES_DEF",
"ORDERS",
"PAYMENTS",
"REFUNDS",
"COUPON_USAGE",
"INVENTORY_CHANGE",
"TOPUPS",
"TABLE_DISCOUNT",
"ASSISTANT_ABOLISH",
"LEDGER",
],
"window_minutes": {
@@ -49,18 +60,26 @@ DEFAULTS = {
"overlap_seconds": 120,
"idle_window": {
"start": "04:00",
"end": "16:00",
"end": "16:00",
},
"allow_empty_result_advance": True,
},
"io": {
"export_root": r"D:\LLZQ\DB\export",
"log_root": r"D:\LLZQ\DB\logs",
"export_root": r"C:\dev\LLTQ\export\JSON",
"log_root": r"C:\dev\LLTQ\export\LOG",
"manifest_name": "manifest.json",
"ingest_report_name": "ingest_report.json",
"write_pretty_json": False,
"write_pretty_json": True,
"max_file_bytes": 50 * 1024 * 1024,
},
"pipeline": {
# 运行流程FETCH_ONLY仅在线抓取落盘、INGEST_ONLY本地清洗入库、FULL抓取 + 清洗入库)
"flow": "FULL",
# 在线抓取 JSON 输出根目录按任务、run_id 与时间自动创建子目录)
"fetch_root": r"C:\dev\LLTQ\export\JSON",
# 本地清洗入库时的 JSON 输入目录(为空则默认使用本次抓取目录)
"ingest_source_dir": "",
},
"clean": {
"log_unknown_fields": True,
"unknown_fields_limit": 50,
@@ -76,20 +95,26 @@ DEFAULTS = {
"redact_keys": ["token", "password", "Authorization"],
"echo_token_in_logs": False,
},
"ods": {
# ODS 离线重建/回放相关(仅开发/运维使用)
"json_doc_dir": r"C:\dev\LLTQ\export\test-json-doc",
"include_files": "",
"drop_schema_first": True,
},
}
# 任务代码常量
TASK_ORDERS = "ORDERS"
TASK_PAYMENTS = "PAYMENTS"
TASK_REFUNDS = "REFUNDS"
TASK_INVENTORY_CHANGE = "INVENTORY_CHANGE"
TASK_COUPON_USAGE = "COUPON_USAGE"
TASK_MEMBERS = "MEMBERS"
TASK_ASSISTANTS = "ASSISTANTS"
TASK_PRODUCTS = "PRODUCTS"
TASK_TABLES = "TABLES"
TASK_PACKAGES_DEF = "PACKAGES_DEF"
TASK_TOPUPS = "TOPUPS"
TASK_TABLE_DISCOUNT = "TABLE_DISCOUNT"
TASK_ORDERS = "ORDERS"
TASK_PAYMENTS = "PAYMENTS"
TASK_REFUNDS = "REFUNDS"
TASK_INVENTORY_CHANGE = "INVENTORY_CHANGE"
TASK_COUPON_USAGE = "COUPON_USAGE"
TASK_MEMBERS = "MEMBERS"
TASK_ASSISTANTS = "ASSISTANTS"
TASK_PRODUCTS = "PRODUCTS"
TASK_TABLES = "TABLES"
TASK_PACKAGES_DEF = "PACKAGES_DEF"
TASK_TOPUPS = "TOPUPS"
TASK_TABLE_DISCOUNT = "TABLE_DISCOUNT"
TASK_ASSISTANT_ABOLISH = "ASSISTANT_ABOLISH"
TASK_LEDGER = "LEDGER"
TASK_LEDGER = "LEDGER"

View File

@@ -2,36 +2,59 @@
"""环境变量解析"""
import os
import json
from pathlib import Path
from copy import deepcopy
ENV_MAP = {
"TIMEZONE": ("app.timezone",),
"STORE_ID": ("app.store_id",),
"SCHEMA_OLTP": ("app.schema_oltp",),
"SCHEMA_ETL": ("app.schema_etl",),
"PG_DSN": ("db.dsn",),
"PG_HOST": ("db.host",),
"PG_PORT": ("db.port",),
"PG_NAME": ("db.name",),
"PG_USER": ("db.user",),
"PG_PASSWORD": ("db.password",),
"API_BASE": ("api.base_url",),
"API_TOKEN": ("api.token",),
"API_TIMEOUT": ("api.timeout_sec",),
"API_PAGE_SIZE": ("api.page_size",),
"EXPORT_ROOT": ("io.export_root",),
"LOG_ROOT": ("io.log_root",),
"OVERLAP_SECONDS": ("run.overlap_seconds",),
"WINDOW_BUSY_MIN": ("run.window_minutes.default_busy",),
"WINDOW_IDLE_MIN": ("run.window_minutes.default_idle",),
"TIMEZONE": ("app.timezone",),
"STORE_ID": ("app.store_id",),
"SCHEMA_OLTP": ("app.schema_oltp",),
"SCHEMA_ETL": ("app.schema_etl",),
"PG_DSN": ("db.dsn",),
"PG_HOST": ("db.host",),
"PG_PORT": ("db.port",),
"PG_NAME": ("db.name",),
"PG_USER": ("db.user",),
"PG_PASSWORD": ("db.password",),
"PG_CONNECT_TIMEOUT": ("db.connect_timeout_sec",),
"API_BASE": ("api.base_url",),
"API_TOKEN": ("api.token",),
"FICOO_TOKEN": ("api.token",),
"API_TIMEOUT": ("api.timeout_sec",),
"API_PAGE_SIZE": ("api.page_size",),
"API_RETRY_MAX": ("api.retries.max_attempts",),
"API_RETRY_BACKOFF": ("api.retries.backoff_sec",),
"API_PARAMS": ("api.params",),
"EXPORT_ROOT": ("io.export_root",),
"LOG_ROOT": ("io.log_root",),
"MANIFEST_NAME": ("io.manifest_name",),
"INGEST_REPORT_NAME": ("io.ingest_report_name",),
"WRITE_PRETTY_JSON": ("io.write_pretty_json",),
"RUN_TASKS": ("run.tasks",),
"OVERLAP_SECONDS": ("run.overlap_seconds",),
"WINDOW_BUSY_MIN": ("run.window_minutes.default_busy",),
"WINDOW_IDLE_MIN": ("run.window_minutes.default_idle",),
"IDLE_START": ("run.idle_window.start",),
"IDLE_END": ("run.idle_window.end",),
"IDLE_WINDOW_START": ("run.idle_window.start",),
"IDLE_WINDOW_END": ("run.idle_window.end",),
"ALLOW_EMPTY_RESULT_ADVANCE": ("run.allow_empty_result_advance",),
"ALLOW_EMPTY_ADVANCE": ("run.allow_empty_result_advance",),
"PIPELINE_FLOW": ("pipeline.flow",),
"JSON_FETCH_ROOT": ("pipeline.fetch_root",),
"JSON_SOURCE_DIR": ("pipeline.ingest_source_dir",),
"FETCH_ROOT": ("pipeline.fetch_root",),
"INGEST_SOURCE_DIR": ("pipeline.ingest_source_dir",),
}
def _deep_set(d, dotted_keys, value):
cur = d
for k in dotted_keys[:-1]:
cur = cur.setdefault(k, {})
cur[dotted_keys[-1]] = value
def _coerce_env(v: str):
if v is None:
return None
@@ -50,13 +73,103 @@ def _coerce_env(v: str):
return s
return s
def load_env_overrides(defaults: dict) -> dict:
cfg = deepcopy(defaults)
def _strip_inline_comment(value: str) -> str:
"""去掉未被引号包裹的内联注释"""
result = []
in_quote = False
quote_char = ""
escape = False
for ch in value:
if escape:
result.append(ch)
escape = False
continue
if ch == "\\":
escape = True
result.append(ch)
continue
if ch in ("'", '"'):
if not in_quote:
in_quote = True
quote_char = ch
elif quote_char == ch:
in_quote = False
quote_char = ""
result.append(ch)
continue
if ch == "#" and not in_quote:
break
result.append(ch)
return "".join(result).rstrip()
def _unquote_value(value: str) -> str:
"""处理引号/原始字符串以及尾随逗号"""
trimmed = value.strip()
trimmed = _strip_inline_comment(trimmed)
trimmed = trimmed.rstrip(",").rstrip()
if not trimmed:
return trimmed
if len(trimmed) >= 2 and trimmed[0] in ("'", '"') and trimmed[-1] == trimmed[0]:
return trimmed[1:-1]
if (
len(trimmed) >= 3
and trimmed[0] in ("r", "R")
and trimmed[1] in ("'", '"')
and trimmed[-1] == trimmed[1]
):
return trimmed[2:-1]
return trimmed
def _parse_dotenv_line(line: str) -> tuple[str, str] | None:
"""解析 .env 文件中的单行"""
stripped = line.strip()
if not stripped or stripped.startswith("#"):
return None
if stripped.startswith("export "):
stripped = stripped[len("export ") :].strip()
if "=" not in stripped:
return None
key, value = stripped.split("=", 1)
key = key.strip()
value = _unquote_value(value)
return key, value
def _load_dotenv_values() -> dict:
"""从项目根目录读取 .env 文件键值"""
if os.environ.get("ETL_SKIP_DOTENV") in ("1", "true", "TRUE", "True"):
return {}
root = Path(__file__).resolve().parents[1]
dotenv_path = root / ".env"
if not dotenv_path.exists():
return {}
values: dict[str, str] = {}
for line in dotenv_path.read_text(encoding="utf-8", errors="ignore").splitlines():
parsed = _parse_dotenv_line(line)
if parsed:
key, value = parsed
values[key] = value
return values
def _apply_env_values(cfg: dict, source: dict):
for env_key, dotted in ENV_MAP.items():
val = os.environ.get(env_key)
val = source.get(env_key)
if val is None:
continue
v2 = _coerce_env(val)
for path in dotted:
if path == "run.tasks" and isinstance(v2, str):
v2 = [item.strip() for item in v2.split(",") if item.strip()]
_deep_set(cfg, path.split("."), v2)
def load_env_overrides(defaults: dict) -> dict:
cfg = deepcopy(defaults)
# 先读取 .env再读取真实环境变量确保 CLI 仍然最高优先级
_apply_env_values(cfg, _load_dotenv_values())
_apply_env_values(cfg, os.environ)
return cfg

View File

@@ -49,6 +49,13 @@ class AppConfig:
f"@{cfg['db']['host']}:{cfg['db']['port']}/{cfg['db']['name']}"
)
# connect_timeout 限定 1-20 秒
try:
timeout_sec = int(cfg["db"].get("connect_timeout_sec") or 5)
except Exception:
raise SystemExit("db.connect_timeout_sec 必须为整数")
cfg["db"]["connect_timeout_sec"] = max(1, min(timeout_sec, 20))
# 会话参数
cfg["db"].setdefault("session", {})
sess = cfg["db"]["session"]

View File

@@ -1,49 +1,62 @@
# -*- coding: utf-8 -*-
"""数据库连接管理"""
"""Database connection manager with capped connect_timeout."""
import psycopg2
import psycopg2.extras
class DatabaseConnection:
"""数据库连接管理器"""
"""Wrap psycopg2 connection with session parameters and timeout guard."""
def __init__(self, dsn: str, session: dict = None, connect_timeout: int = None):
self.conn = psycopg2.connect(dsn, connect_timeout=(connect_timeout or 5))
timeout_val = connect_timeout if connect_timeout is not None else 5
# PRD: database connect_timeout must not exceed 20 seconds.
timeout_val = max(1, min(int(timeout_val), 20))
self.conn = psycopg2.connect(dsn, connect_timeout=timeout_val)
self.conn.autocommit = False
# 设置会话参数
# Session parameters (timezone, statement timeout, etc.)
if session:
with self.conn.cursor() as c:
if session.get("timezone"):
c.execute("SET TIME ZONE %s", (session["timezone"],))
if session.get("statement_timeout_ms") is not None:
c.execute("SET statement_timeout = %s", (int(session["statement_timeout_ms"]),))
c.execute(
"SET statement_timeout = %s",
(int(session["statement_timeout_ms"]),),
)
if session.get("lock_timeout_ms") is not None:
c.execute("SET lock_timeout = %s", (int(session["lock_timeout_ms"]),))
c.execute(
"SET lock_timeout = %s", (int(session["lock_timeout_ms"]),)
)
if session.get("idle_in_tx_timeout_ms") is not None:
c.execute("SET idle_in_transaction_session_timeout = %s",
(int(session["idle_in_tx_timeout_ms"]),))
c.execute(
"SET idle_in_transaction_session_timeout = %s",
(int(session["idle_in_tx_timeout_ms"]),),
)
def query(self, sql: str, args=None):
"""执行查询并返回结果"""
"""Execute a query and fetch all rows."""
with self.conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as c:
c.execute(sql, args)
return c.fetchall()
def execute(self, sql: str, args=None):
"""执行SQL语句"""
"""Execute a SQL statement without returning rows."""
with self.conn.cursor() as c:
c.execute(sql, args)
def commit(self):
"""提交事务"""
"""Commit current transaction."""
self.conn.commit()
def rollback(self):
"""回滚事务"""
"""Rollback current transaction."""
self.conn.rollback()
def close(self):
"""关闭连接"""
"""Safely close the connection."""
try:
self.conn.close()
except Exception:

View File

@@ -7,6 +7,7 @@ class DatabaseOperations:
"""数据库批量操作封装"""
def __init__(self, connection):
self._connection = connection
self.conn = connection.conn
def batch_execute(self, sql: str, rows: list, page_size: int = 1000):
@@ -75,3 +76,24 @@ class DatabaseOperations:
if isinstance(rec, dict):
return bool(rec.get("inserted"))
return False
# --- pass-through helpers -------------------------------------------------
def commit(self):
"""提交事务(委托给底层连接)"""
self._connection.commit()
def rollback(self):
"""回滚事务(委托给底层连接)"""
self._connection.rollback()
def query(self, sql: str, args=None):
"""执行查询并返回结果"""
return self._connection.query(sql, args)
def execute(self, sql: str, args=None):
"""执行任意 SQL"""
self._connection.execute(sql, args)
def cursor(self):
"""暴露原生 cursor供特殊操作使用"""
return self.conn.cursor()

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,39 @@
-- 将新的 ODS 任务注册到 etl_admin.etl_task根据需要替换 store_id
-- 使用方式(示例):
-- psql "$PG_DSN" -f etl_billiards/database/seed_ods_tasks.sql
-- 或者在 psql 中执行本文件内容。
WITH target_store AS (
SELECT 2790685415443269::bigint AS store_id -- TODO: 替换为实际 store_id
),
task_codes AS (
SELECT unnest(ARRAY[
'ODS_ASSISTANT_ACCOUNTS',
'ODS_ASSISTANT_LEDGER',
'ODS_ASSISTANT_ABOLISH',
'ODS_INVENTORY_CHANGE',
'ODS_INVENTORY_STOCK',
'ODS_PACKAGE',
'ODS_GROUP_BUY_REDEMPTION',
'ODS_MEMBER',
'ODS_MEMBER_BALANCE',
'ODS_MEMBER_CARD',
'ODS_PAYMENT',
'ODS_REFUND',
'ODS_COUPON_VERIFY',
'ODS_RECHARGE_SETTLE',
'ODS_TABLES',
'ODS_GOODS_CATEGORY',
'ODS_STORE_GOODS',
'ODS_TABLE_DISCOUNT',
'ODS_TENANT_GOODS',
'ODS_SETTLEMENT_TICKET',
'ODS_ORDER_SETTLE'
]) AS task_code
)
INSERT INTO etl_admin.etl_task (task_code, store_id, enabled)
SELECT t.task_code, s.store_id, TRUE
FROM task_codes t CROSS JOIN target_store s
ON CONFLICT (task_code, store_id) DO UPDATE
SET enabled = EXCLUDED.enabled;

View File

@@ -0,0 +1,16 @@
{
"folders": [
{
"path": ".."
},
{
"name": "LLZQ-server",
"path": "../../../LLZQ-server"
},
{
"name": "feiqiu-ETL-reload",
"path": "../../feiqiu-ETL-reload"
}
],
"settings": {}
}

View File

@@ -1,19 +1,23 @@
# -*- coding: utf-8 -*-
"""数据加载器基类"""
import logging
class BaseLoader:
"""数据加载器基类"""
def __init__(self, db_ops):
def __init__(self, db_ops, logger=None):
self.db = db_ops
self.logger = logger or logging.getLogger(self.__class__.__name__)
def upsert(self, records: list) -> tuple:
"""
执行UPSERT操作
执行 UPSERT 操作
返回: (inserted_count, updated_count, skipped_count)
"""
raise NotImplementedError("子类需实现 upsert 方法")
def _batch_size(self) -> int:
"""批次大小"""
return 1000

View File

@@ -0,0 +1,114 @@
# -*- coding: utf-8 -*-
"""助教维度加载器"""
from ..base_loader import BaseLoader
class AssistantLoader(BaseLoader):
"""写入 dim_assistant"""
def upsert_assistants(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.dim_assistant (
store_id,
assistant_id,
assistant_no,
nickname,
real_name,
gender,
mobile,
level,
team_id,
team_name,
assistant_status,
work_status,
entry_time,
resign_time,
start_time,
end_time,
create_time,
update_time,
system_role_id,
online_status,
allow_cx,
charge_way,
pd_unit_price,
cx_unit_price,
is_guaranteed,
is_team_leader,
serial_number,
show_sort,
is_delete,
raw_data
)
VALUES (
%(store_id)s,
%(assistant_id)s,
%(assistant_no)s,
%(nickname)s,
%(real_name)s,
%(gender)s,
%(mobile)s,
%(level)s,
%(team_id)s,
%(team_name)s,
%(assistant_status)s,
%(work_status)s,
%(entry_time)s,
%(resign_time)s,
%(start_time)s,
%(end_time)s,
%(create_time)s,
%(update_time)s,
%(system_role_id)s,
%(online_status)s,
%(allow_cx)s,
%(charge_way)s,
%(pd_unit_price)s,
%(cx_unit_price)s,
%(is_guaranteed)s,
%(is_team_leader)s,
%(serial_number)s,
%(show_sort)s,
%(is_delete)s,
%(raw_data)s
)
ON CONFLICT (store_id, assistant_id) DO UPDATE SET
assistant_no = EXCLUDED.assistant_no,
nickname = EXCLUDED.nickname,
real_name = EXCLUDED.real_name,
gender = EXCLUDED.gender,
mobile = EXCLUDED.mobile,
level = EXCLUDED.level,
team_id = EXCLUDED.team_id,
team_name = EXCLUDED.team_name,
assistant_status= EXCLUDED.assistant_status,
work_status = EXCLUDED.work_status,
entry_time = EXCLUDED.entry_time,
resign_time = EXCLUDED.resign_time,
start_time = EXCLUDED.start_time,
end_time = EXCLUDED.end_time,
update_time = COALESCE(EXCLUDED.update_time, now()),
system_role_id = EXCLUDED.system_role_id,
online_status = EXCLUDED.online_status,
allow_cx = EXCLUDED.allow_cx,
charge_way = EXCLUDED.charge_way,
pd_unit_price = EXCLUDED.pd_unit_price,
cx_unit_price = EXCLUDED.cx_unit_price,
is_guaranteed = EXCLUDED.is_guaranteed,
is_team_leader = EXCLUDED.is_team_leader,
serial_number = EXCLUDED.serial_number,
show_sort = EXCLUDED.show_sort,
is_delete = EXCLUDED.is_delete,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,91 @@
# -*- coding: utf-8 -*-
"""团购/套餐定义加载器"""
from ..base_loader import BaseLoader
class PackageDefinitionLoader(BaseLoader):
"""写入 dim_package_coupon"""
def upsert_packages(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.dim_package_coupon (
store_id,
package_id,
package_code,
package_name,
table_area_id,
table_area_name,
selling_price,
duration_seconds,
start_time,
end_time,
type,
is_enabled,
is_delete,
usable_count,
creator_name,
date_type,
group_type,
coupon_money,
area_tag_type,
system_group_type,
card_type_ids,
raw_data
)
VALUES (
%(store_id)s,
%(package_id)s,
%(package_code)s,
%(package_name)s,
%(table_area_id)s,
%(table_area_name)s,
%(selling_price)s,
%(duration_seconds)s,
%(start_time)s,
%(end_time)s,
%(type)s,
%(is_enabled)s,
%(is_delete)s,
%(usable_count)s,
%(creator_name)s,
%(date_type)s,
%(group_type)s,
%(coupon_money)s,
%(area_tag_type)s,
%(system_group_type)s,
%(card_type_ids)s,
%(raw_data)s
)
ON CONFLICT (store_id, package_id) DO UPDATE SET
package_code = EXCLUDED.package_code,
package_name = EXCLUDED.package_name,
table_area_id = EXCLUDED.table_area_id,
table_area_name = EXCLUDED.table_area_name,
selling_price = EXCLUDED.selling_price,
duration_seconds = EXCLUDED.duration_seconds,
start_time = EXCLUDED.start_time,
end_time = EXCLUDED.end_time,
type = EXCLUDED.type,
is_enabled = EXCLUDED.is_enabled,
is_delete = EXCLUDED.is_delete,
usable_count = EXCLUDED.usable_count,
creator_name = EXCLUDED.creator_name,
date_type = EXCLUDED.date_type,
group_type = EXCLUDED.group_type,
coupon_money = EXCLUDED.coupon_money,
area_tag_type = EXCLUDED.area_tag_type,
system_group_type = EXCLUDED.system_group_type,
card_type_ids = EXCLUDED.card_type_ids,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,80 @@
# -*- coding: utf-8 -*-
"""台桌维度加载器"""
from ..base_loader import BaseLoader
class TableLoader(BaseLoader):
"""将台桌档案写入 dim_table"""
def upsert_tables(self, records: list) -> tuple:
"""批量写入台桌档案"""
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.dim_table (
store_id,
table_id,
site_id,
area_id,
area_name,
table_name,
table_price,
table_status,
table_status_name,
light_status,
is_rest_area,
show_status,
virtual_table,
charge_free,
only_allow_groupon,
is_online_reservation,
created_time,
raw_data
)
VALUES (
%(store_id)s,
%(table_id)s,
%(site_id)s,
%(area_id)s,
%(area_name)s,
%(table_name)s,
%(table_price)s,
%(table_status)s,
%(table_status_name)s,
%(light_status)s,
%(is_rest_area)s,
%(show_status)s,
%(virtual_table)s,
%(charge_free)s,
%(only_allow_groupon)s,
%(is_online_reservation)s,
%(created_time)s,
%(raw_data)s
)
ON CONFLICT (store_id, table_id) DO UPDATE SET
site_id = EXCLUDED.site_id,
area_id = EXCLUDED.area_id,
area_name = EXCLUDED.area_name,
table_name = EXCLUDED.table_name,
table_price = EXCLUDED.table_price,
table_status = EXCLUDED.table_status,
table_status_name = EXCLUDED.table_status_name,
light_status = EXCLUDED.light_status,
is_rest_area = EXCLUDED.is_rest_area,
show_status = EXCLUDED.show_status,
virtual_table = EXCLUDED.virtual_table,
charge_free = EXCLUDED.charge_free,
only_allow_groupon = EXCLUDED.only_allow_groupon,
is_online_reservation = EXCLUDED.is_online_reservation,
created_time = COALESCE(EXCLUDED.created_time, dim_table.created_time),
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,64 @@
# -*- coding: utf-8 -*-
"""助教作废事实表"""
from ..base_loader import BaseLoader
class AssistantAbolishLoader(BaseLoader):
"""写入 fact_assistant_abolish"""
def upsert_records(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_assistant_abolish (
store_id,
abolish_id,
table_id,
table_name,
table_area_id,
table_area,
assistant_no,
assistant_name,
charge_minutes,
abolish_amount,
create_time,
trash_reason,
raw_data
)
VALUES (
%(store_id)s,
%(abolish_id)s,
%(table_id)s,
%(table_name)s,
%(table_area_id)s,
%(table_area)s,
%(assistant_no)s,
%(assistant_name)s,
%(charge_minutes)s,
%(abolish_amount)s,
%(create_time)s,
%(trash_reason)s,
%(raw_data)s
)
ON CONFLICT (store_id, abolish_id) DO UPDATE SET
table_id = EXCLUDED.table_id,
table_name = EXCLUDED.table_name,
table_area_id = EXCLUDED.table_area_id,
table_area = EXCLUDED.table_area,
assistant_no = EXCLUDED.assistant_no,
assistant_name = EXCLUDED.assistant_name,
charge_minutes = EXCLUDED.charge_minutes,
abolish_amount = EXCLUDED.abolish_amount,
create_time = EXCLUDED.create_time,
trash_reason = EXCLUDED.trash_reason,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,136 @@
# -*- coding: utf-8 -*-
"""助教流水事实表"""
from ..base_loader import BaseLoader
class AssistantLedgerLoader(BaseLoader):
"""写入 fact_assistant_ledger"""
def upsert_ledgers(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_assistant_ledger (
store_id,
ledger_id,
assistant_no,
assistant_name,
nickname,
level_name,
table_name,
ledger_unit_price,
ledger_count,
ledger_amount,
projected_income,
service_money,
member_discount_amount,
manual_discount_amount,
coupon_deduct_money,
order_trade_no,
order_settle_id,
operator_id,
operator_name,
assistant_team_id,
assistant_level,
site_table_id,
order_assistant_id,
site_assistant_id,
user_id,
ledger_start_time,
ledger_end_time,
start_use_time,
last_use_time,
income_seconds,
real_use_seconds,
is_trash,
trash_reason,
is_confirm,
ledger_status,
create_time,
raw_data
)
VALUES (
%(store_id)s,
%(ledger_id)s,
%(assistant_no)s,
%(assistant_name)s,
%(nickname)s,
%(level_name)s,
%(table_name)s,
%(ledger_unit_price)s,
%(ledger_count)s,
%(ledger_amount)s,
%(projected_income)s,
%(service_money)s,
%(member_discount_amount)s,
%(manual_discount_amount)s,
%(coupon_deduct_money)s,
%(order_trade_no)s,
%(order_settle_id)s,
%(operator_id)s,
%(operator_name)s,
%(assistant_team_id)s,
%(assistant_level)s,
%(site_table_id)s,
%(order_assistant_id)s,
%(site_assistant_id)s,
%(user_id)s,
%(ledger_start_time)s,
%(ledger_end_time)s,
%(start_use_time)s,
%(last_use_time)s,
%(income_seconds)s,
%(real_use_seconds)s,
%(is_trash)s,
%(trash_reason)s,
%(is_confirm)s,
%(ledger_status)s,
%(create_time)s,
%(raw_data)s
)
ON CONFLICT (store_id, ledger_id) DO UPDATE SET
assistant_no = EXCLUDED.assistant_no,
assistant_name = EXCLUDED.assistant_name,
nickname = EXCLUDED.nickname,
level_name = EXCLUDED.level_name,
table_name = EXCLUDED.table_name,
ledger_unit_price = EXCLUDED.ledger_unit_price,
ledger_count = EXCLUDED.ledger_count,
ledger_amount = EXCLUDED.ledger_amount,
projected_income = EXCLUDED.projected_income,
service_money = EXCLUDED.service_money,
member_discount_amount = EXCLUDED.member_discount_amount,
manual_discount_amount = EXCLUDED.manual_discount_amount,
coupon_deduct_money = EXCLUDED.coupon_deduct_money,
order_trade_no = EXCLUDED.order_trade_no,
order_settle_id = EXCLUDED.order_settle_id,
operator_id = EXCLUDED.operator_id,
operator_name = EXCLUDED.operator_name,
assistant_team_id = EXCLUDED.assistant_team_id,
assistant_level = EXCLUDED.assistant_level,
site_table_id = EXCLUDED.site_table_id,
order_assistant_id = EXCLUDED.order_assistant_id,
site_assistant_id = EXCLUDED.site_assistant_id,
user_id = EXCLUDED.user_id,
ledger_start_time = EXCLUDED.ledger_start_time,
ledger_end_time = EXCLUDED.ledger_end_time,
start_use_time = EXCLUDED.start_use_time,
last_use_time = EXCLUDED.last_use_time,
income_seconds = EXCLUDED.income_seconds,
real_use_seconds = EXCLUDED.real_use_seconds,
is_trash = EXCLUDED.is_trash,
trash_reason = EXCLUDED.trash_reason,
is_confirm = EXCLUDED.is_confirm,
ledger_status = EXCLUDED.ledger_status,
create_time = EXCLUDED.create_time,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,91 @@
# -*- coding: utf-8 -*-
"""券核销事实表"""
from ..base_loader import BaseLoader
class CouponUsageLoader(BaseLoader):
"""写入 fact_coupon_usage"""
def upsert_coupon_usage(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_coupon_usage (
store_id,
usage_id,
coupon_code,
coupon_channel,
coupon_name,
sale_price,
coupon_money,
coupon_free_time,
use_status,
create_time,
consume_time,
operator_id,
operator_name,
table_id,
site_order_id,
group_package_id,
coupon_remark,
deal_id,
certificate_id,
verify_id,
is_delete,
raw_data
)
VALUES (
%(store_id)s,
%(usage_id)s,
%(coupon_code)s,
%(coupon_channel)s,
%(coupon_name)s,
%(sale_price)s,
%(coupon_money)s,
%(coupon_free_time)s,
%(use_status)s,
%(create_time)s,
%(consume_time)s,
%(operator_id)s,
%(operator_name)s,
%(table_id)s,
%(site_order_id)s,
%(group_package_id)s,
%(coupon_remark)s,
%(deal_id)s,
%(certificate_id)s,
%(verify_id)s,
%(is_delete)s,
%(raw_data)s
)
ON CONFLICT (store_id, usage_id) DO UPDATE SET
coupon_code = EXCLUDED.coupon_code,
coupon_channel = EXCLUDED.coupon_channel,
coupon_name = EXCLUDED.coupon_name,
sale_price = EXCLUDED.sale_price,
coupon_money = EXCLUDED.coupon_money,
coupon_free_time = EXCLUDED.coupon_free_time,
use_status = EXCLUDED.use_status,
create_time = EXCLUDED.create_time,
consume_time = EXCLUDED.consume_time,
operator_id = EXCLUDED.operator_id,
operator_name = EXCLUDED.operator_name,
table_id = EXCLUDED.table_id,
site_order_id = EXCLUDED.site_order_id,
group_package_id = EXCLUDED.group_package_id,
coupon_remark = EXCLUDED.coupon_remark,
deal_id = EXCLUDED.deal_id,
certificate_id = EXCLUDED.certificate_id,
verify_id = EXCLUDED.verify_id,
is_delete = EXCLUDED.is_delete,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,73 @@
# -*- coding: utf-8 -*-
"""库存变动事实表"""
from ..base_loader import BaseLoader
class InventoryChangeLoader(BaseLoader):
"""写入 fact_inventory_change"""
def upsert_changes(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_inventory_change (
store_id,
change_id,
site_goods_id,
stock_type,
goods_name,
change_time,
start_qty,
end_qty,
change_qty,
unit,
price,
operator_name,
remark,
goods_category_id,
goods_second_category_id,
raw_data
)
VALUES (
%(store_id)s,
%(change_id)s,
%(site_goods_id)s,
%(stock_type)s,
%(goods_name)s,
%(change_time)s,
%(start_qty)s,
%(end_qty)s,
%(change_qty)s,
%(unit)s,
%(price)s,
%(operator_name)s,
%(remark)s,
%(goods_category_id)s,
%(goods_second_category_id)s,
%(raw_data)s
)
ON CONFLICT (store_id, change_id) DO UPDATE SET
site_goods_id = EXCLUDED.site_goods_id,
stock_type = EXCLUDED.stock_type,
goods_name = EXCLUDED.goods_name,
change_time = EXCLUDED.change_time,
start_qty = EXCLUDED.start_qty,
end_qty = EXCLUDED.end_qty,
change_qty = EXCLUDED.change_qty,
unit = EXCLUDED.unit,
price = EXCLUDED.price,
operator_name = EXCLUDED.operator_name,
remark = EXCLUDED.remark,
goods_category_id = EXCLUDED.goods_category_id,
goods_second_category_id = EXCLUDED.goods_second_category_id,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -1,29 +1,55 @@
# -*- coding: utf-8 -*-
"""支付记录加载器"""
"""支付事实表加载器"""
from ..base_loader import BaseLoader
class PaymentLoader(BaseLoader):
"""支付记录加载器"""
"""支付数据加载器"""
def upsert_payments(self, records: list, store_id: int) -> tuple:
"""加载支付记录"""
"""加载支付数据"""
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_payment (
store_id, pay_id, order_id, pay_time, pay_amount,
pay_type, pay_status, remark, raw_data
store_id, pay_id, order_id,
site_id, tenant_id,
order_settle_id, order_trade_no,
relate_type, relate_id,
create_time, pay_time,
pay_amount, fee_amount, discount_amount,
payment_method, pay_type,
online_pay_channel, pay_terminal,
pay_status, remark, raw_data
)
VALUES (
%(store_id)s, %(pay_id)s, %(order_id)s, %(pay_time)s, %(pay_amount)s,
%(pay_type)s, %(pay_status)s, %(remark)s, %(raw_data)s
%(store_id)s, %(pay_id)s, %(order_id)s,
%(site_id)s, %(tenant_id)s,
%(order_settle_id)s, %(order_trade_no)s,
%(relate_type)s, %(relate_id)s,
%(create_time)s, %(pay_time)s,
%(pay_amount)s, %(fee_amount)s, %(discount_amount)s,
%(payment_method)s, %(pay_type)s,
%(online_pay_channel)s, %(pay_terminal)s,
%(pay_status)s, %(remark)s, %(raw_data)s
)
ON CONFLICT (store_id, pay_id) DO UPDATE SET
order_settle_id = EXCLUDED.order_settle_id,
order_trade_no = EXCLUDED.order_trade_no,
relate_type = EXCLUDED.relate_type,
relate_id = EXCLUDED.relate_id,
order_id = EXCLUDED.order_id,
site_id = EXCLUDED.site_id,
tenant_id = EXCLUDED.tenant_id,
create_time = EXCLUDED.create_time,
pay_time = EXCLUDED.pay_time,
pay_amount = EXCLUDED.pay_amount,
fee_amount = EXCLUDED.fee_amount,
discount_amount = EXCLUDED.discount_amount,
payment_method = EXCLUDED.payment_method,
pay_type = EXCLUDED.pay_type,
online_pay_channel = EXCLUDED.online_pay_channel,
pay_terminal = EXCLUDED.pay_terminal,
pay_status = EXCLUDED.pay_status,
remark = EXCLUDED.remark,
raw_data = EXCLUDED.raw_data,

View File

@@ -0,0 +1,88 @@
# -*- coding: utf-8 -*-
"""退款事实表加载器"""
from ..base_loader import BaseLoader
class RefundLoader(BaseLoader):
"""写入 fact_refund"""
def upsert_refunds(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_refund (
store_id,
refund_id,
site_id,
tenant_id,
pay_amount,
pay_status,
pay_time,
create_time,
relate_type,
relate_id,
payment_method,
refund_amount,
action_type,
pay_terminal,
operator_id,
channel_pay_no,
channel_fee,
is_delete,
member_id,
member_card_id,
raw_data
)
VALUES (
%(store_id)s,
%(refund_id)s,
%(site_id)s,
%(tenant_id)s,
%(pay_amount)s,
%(pay_status)s,
%(pay_time)s,
%(create_time)s,
%(relate_type)s,
%(relate_id)s,
%(payment_method)s,
%(refund_amount)s,
%(action_type)s,
%(pay_terminal)s,
%(operator_id)s,
%(channel_pay_no)s,
%(channel_fee)s,
%(is_delete)s,
%(member_id)s,
%(member_card_id)s,
%(raw_data)s
)
ON CONFLICT (store_id, refund_id) DO UPDATE SET
site_id = EXCLUDED.site_id,
tenant_id = EXCLUDED.tenant_id,
pay_amount = EXCLUDED.pay_amount,
pay_status = EXCLUDED.pay_status,
pay_time = EXCLUDED.pay_time,
create_time = EXCLUDED.create_time,
relate_type = EXCLUDED.relate_type,
relate_id = EXCLUDED.relate_id,
payment_method = EXCLUDED.payment_method,
refund_amount = EXCLUDED.refund_amount,
action_type = EXCLUDED.action_type,
pay_terminal = EXCLUDED.pay_terminal,
operator_id = EXCLUDED.operator_id,
channel_pay_no = EXCLUDED.channel_pay_no,
channel_fee = EXCLUDED.channel_fee,
is_delete = EXCLUDED.is_delete,
member_id = EXCLUDED.member_id,
member_card_id = EXCLUDED.member_card_id,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,82 @@
# -*- coding: utf-8 -*-
"""台费打折事实表"""
from ..base_loader import BaseLoader
class TableDiscountLoader(BaseLoader):
"""写入 fact_table_discount"""
def upsert_discounts(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_table_discount (
store_id,
discount_id,
adjust_type,
applicant_id,
applicant_name,
operator_id,
operator_name,
ledger_amount,
ledger_count,
ledger_name,
ledger_status,
order_settle_id,
order_trade_no,
site_table_id,
table_area_id,
table_area_name,
create_time,
is_delete,
raw_data
)
VALUES (
%(store_id)s,
%(discount_id)s,
%(adjust_type)s,
%(applicant_id)s,
%(applicant_name)s,
%(operator_id)s,
%(operator_name)s,
%(ledger_amount)s,
%(ledger_count)s,
%(ledger_name)s,
%(ledger_status)s,
%(order_settle_id)s,
%(order_trade_no)s,
%(site_table_id)s,
%(table_area_id)s,
%(table_area_name)s,
%(create_time)s,
%(is_delete)s,
%(raw_data)s
)
ON CONFLICT (store_id, discount_id) DO UPDATE SET
adjust_type = EXCLUDED.adjust_type,
applicant_id = EXCLUDED.applicant_id,
applicant_name = EXCLUDED.applicant_name,
operator_id = EXCLUDED.operator_id,
operator_name = EXCLUDED.operator_name,
ledger_amount = EXCLUDED.ledger_amount,
ledger_count = EXCLUDED.ledger_count,
ledger_name = EXCLUDED.ledger_name,
ledger_status = EXCLUDED.ledger_status,
order_settle_id = EXCLUDED.order_settle_id,
order_trade_no = EXCLUDED.order_trade_no,
site_table_id = EXCLUDED.site_table_id,
table_area_id = EXCLUDED.table_area_id,
table_area_name = EXCLUDED.table_area_name,
create_time = EXCLUDED.create_time,
is_delete = EXCLUDED.is_delete,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,188 @@
# -*- coding: utf-8 -*-
"""小票详情加载器"""
from ..base_loader import BaseLoader
import json
class TicketLoader(BaseLoader):
"""
Loader for parsing Ticket Detail JSON and populating DWD fact tables.
Handles:
- fact_order (Header)
- fact_order_goods (Items)
- fact_table_usage (Items)
- fact_assistant_service (Items)
"""
def process_tickets(self, tickets: list, store_id: int) -> tuple:
"""
Process a batch of ticket JSONs.
Returns (inserted_count, error_count)
"""
inserted_count = 0
error_count = 0
# Prepare batch lists
orders = []
goods_list = []
table_usages = []
assistant_services = []
for ticket in tickets:
try:
# 1. Parse Header (fact_order)
root_data = ticket.get("data", {}).get("data", {})
if not root_data:
continue
order_settle_id = root_data.get("orderSettleId")
if not order_settle_id:
continue
orders.append({
"store_id": store_id,
"order_settle_id": order_settle_id,
"order_trade_no": 0,
"order_no": str(root_data.get("orderSettleNumber", "")),
"member_id": 0,
"pay_time": root_data.get("payTime"),
"total_amount": root_data.get("consumeMoney", 0),
"pay_amount": root_data.get("actualPayment", 0),
"discount_amount": root_data.get("memberOfferAmount", 0),
"coupon_amount": root_data.get("couponAmount", 0),
"status": "PAID",
"cashier_name": root_data.get("cashierName", ""),
"remark": root_data.get("orderRemark", ""),
"raw_data": json.dumps(ticket, ensure_ascii=False)
})
# 2. Parse Items (orderItem list)
order_items = root_data.get("orderItem", [])
for item in order_items:
order_trade_no = item.get("siteOrderId")
# 2.1 Table Ledger
table_ledger = item.get("tableLedger")
if table_ledger:
table_usages.append({
"store_id": store_id,
"order_ledger_id": table_ledger.get("orderTableLedgerId"),
"order_settle_id": order_settle_id,
"table_id": table_ledger.get("siteTableId"),
"table_name": table_ledger.get("tableName"),
"start_time": table_ledger.get("chargeStartTime"),
"end_time": table_ledger.get("chargeEndTime"),
"duration_minutes": table_ledger.get("useDuration", 0),
"total_amount": table_ledger.get("consumptionAmount", 0),
"pay_amount": table_ledger.get("consumptionAmount", 0) - table_ledger.get("memberDiscountAmount", 0)
})
# 2.2 Goods Ledgers
goods_ledgers = item.get("goodsLedgers", [])
for g in goods_ledgers:
goods_list.append({
"store_id": store_id,
"order_goods_id": g.get("orderGoodsLedgerId"),
"order_settle_id": order_settle_id,
"order_trade_no": order_trade_no,
"goods_id": g.get("siteGoodsId"),
"goods_name": g.get("goodsName"),
"quantity": g.get("goodsCount", 0),
"unit_price": g.get("goodsPrice", 0),
"total_amount": g.get("ledgerAmount", 0),
"pay_amount": g.get("realGoodsMoney", 0)
})
# 2.3 Assistant Services
assistant_ledgers = item.get("assistantPlayWith", [])
for a in assistant_ledgers:
assistant_services.append({
"store_id": store_id,
"ledger_id": a.get("orderAssistantLedgerId"),
"order_settle_id": order_settle_id,
"assistant_id": a.get("assistantId"),
"assistant_name": a.get("ledgerName"),
"service_type": a.get("skillName", "Play"),
"start_time": a.get("ledgerStartTime"),
"end_time": a.get("ledgerEndTime"),
"duration_minutes": int(a.get("ledgerCount", 0) / 60) if a.get("ledgerCount") else 0,
"total_amount": a.get("ledgerAmount", 0),
"pay_amount": a.get("ledgerAmount", 0)
})
inserted_count += 1
except Exception as e:
self.logger.error(f"Error parsing ticket: {e}", exc_info=True)
error_count += 1
# 3. Batch Insert/Upsert
if orders:
self._upsert_orders(orders)
if goods_list:
self._upsert_goods(goods_list)
if table_usages:
self._upsert_table_usages(table_usages)
if assistant_services:
self._upsert_assistant_services(assistant_services)
return inserted_count, error_count
def _upsert_orders(self, rows):
sql = """
INSERT INTO billiards.fact_order (
store_id, order_settle_id, order_trade_no, order_no, member_id,
pay_time, total_amount, pay_amount, discount_amount, coupon_amount,
status, cashier_name, remark, raw_data
) VALUES (
%(store_id)s, %(order_settle_id)s, %(order_trade_no)s, %(order_no)s, %(member_id)s,
%(pay_time)s, %(total_amount)s, %(pay_amount)s, %(discount_amount)s, %(coupon_amount)s,
%(status)s, %(cashier_name)s, %(remark)s, %(raw_data)s
)
ON CONFLICT (store_id, order_settle_id) DO UPDATE SET
pay_time = EXCLUDED.pay_time,
pay_amount = EXCLUDED.pay_amount,
updated_at = now()
"""
self.db.batch_execute(sql, rows)
def _upsert_goods(self, rows):
sql = """
INSERT INTO billiards.fact_order_goods (
store_id, order_goods_id, order_settle_id, order_trade_no,
goods_id, goods_name, quantity, unit_price, total_amount, pay_amount
) VALUES (
%(store_id)s, %(order_goods_id)s, %(order_settle_id)s, %(order_trade_no)s,
%(goods_id)s, %(goods_name)s, %(quantity)s, %(unit_price)s, %(total_amount)s, %(pay_amount)s
)
ON CONFLICT (store_id, order_goods_id) DO UPDATE SET
pay_amount = EXCLUDED.pay_amount
"""
self.db.batch_execute(sql, rows)
def _upsert_table_usages(self, rows):
sql = """
INSERT INTO billiards.fact_table_usage (
store_id, order_ledger_id, order_settle_id, table_id, table_name,
start_time, end_time, duration_minutes, total_amount, pay_amount
) VALUES (
%(store_id)s, %(order_ledger_id)s, %(order_settle_id)s, %(table_id)s, %(table_name)s,
%(start_time)s, %(end_time)s, %(duration_minutes)s, %(total_amount)s, %(pay_amount)s
)
ON CONFLICT (store_id, order_ledger_id) DO UPDATE SET
pay_amount = EXCLUDED.pay_amount
"""
self.db.batch_execute(sql, rows)
def _upsert_assistant_services(self, rows):
sql = """
INSERT INTO billiards.fact_assistant_service (
store_id, ledger_id, order_settle_id, assistant_id, assistant_name,
service_type, start_time, end_time, duration_minutes, total_amount, pay_amount
) VALUES (
%(store_id)s, %(ledger_id)s, %(order_settle_id)s, %(assistant_id)s, %(assistant_name)s,
%(service_type)s, %(start_time)s, %(end_time)s, %(duration_minutes)s, %(total_amount)s, %(pay_amount)s
)
ON CONFLICT (store_id, ledger_id) DO UPDATE SET
pay_amount = EXCLUDED.pay_amount
"""
self.db.batch_execute(sql, rows)

View File

@@ -0,0 +1,118 @@
# -*- coding: utf-8 -*-
"""充值记录事实表"""
from ..base_loader import BaseLoader
class TopupLoader(BaseLoader):
"""写入 fact_topup"""
def upsert_topups(self, records: list) -> tuple:
if not records:
return (0, 0, 0)
sql = """
INSERT INTO billiards.fact_topup (
store_id,
topup_id,
member_id,
member_name,
member_phone,
card_id,
card_type_name,
pay_amount,
consume_money,
settle_status,
settle_type,
settle_name,
settle_relate_id,
pay_time,
create_time,
operator_id,
operator_name,
payment_method,
refund_amount,
cash_amount,
card_amount,
balance_amount,
online_amount,
rounding_amount,
adjust_amount,
goods_money,
table_charge_money,
service_money,
coupon_amount,
order_remark,
raw_data
)
VALUES (
%(store_id)s,
%(topup_id)s,
%(member_id)s,
%(member_name)s,
%(member_phone)s,
%(card_id)s,
%(card_type_name)s,
%(pay_amount)s,
%(consume_money)s,
%(settle_status)s,
%(settle_type)s,
%(settle_name)s,
%(settle_relate_id)s,
%(pay_time)s,
%(create_time)s,
%(operator_id)s,
%(operator_name)s,
%(payment_method)s,
%(refund_amount)s,
%(cash_amount)s,
%(card_amount)s,
%(balance_amount)s,
%(online_amount)s,
%(rounding_amount)s,
%(adjust_amount)s,
%(goods_money)s,
%(table_charge_money)s,
%(service_money)s,
%(coupon_amount)s,
%(order_remark)s,
%(raw_data)s
)
ON CONFLICT (store_id, topup_id) DO UPDATE SET
member_id = EXCLUDED.member_id,
member_name = EXCLUDED.member_name,
member_phone = EXCLUDED.member_phone,
card_id = EXCLUDED.card_id,
card_type_name = EXCLUDED.card_type_name,
pay_amount = EXCLUDED.pay_amount,
consume_money = EXCLUDED.consume_money,
settle_status = EXCLUDED.settle_status,
settle_type = EXCLUDED.settle_type,
settle_name = EXCLUDED.settle_name,
settle_relate_id = EXCLUDED.settle_relate_id,
pay_time = EXCLUDED.pay_time,
create_time = EXCLUDED.create_time,
operator_id = EXCLUDED.operator_id,
operator_name = EXCLUDED.operator_name,
payment_method = EXCLUDED.payment_method,
refund_amount = EXCLUDED.refund_amount,
cash_amount = EXCLUDED.cash_amount,
card_amount = EXCLUDED.card_amount,
balance_amount = EXCLUDED.balance_amount,
online_amount = EXCLUDED.online_amount,
rounding_amount = EXCLUDED.rounding_amount,
adjust_amount = EXCLUDED.adjust_amount,
goods_money = EXCLUDED.goods_money,
table_charge_money = EXCLUDED.table_charge_money,
service_money = EXCLUDED.service_money,
coupon_amount = EXCLUDED.coupon_amount,
order_remark = EXCLUDED.order_remark,
raw_data = EXCLUDED.raw_data,
updated_at = now()
RETURNING (xmax = 0) AS inserted
"""
inserted, updated = self.db.batch_upsert_with_returning(
sql, records, page_size=self._batch_size()
)
return (inserted, updated, 0)

View File

@@ -0,0 +1,6 @@
# -*- coding: utf-8 -*-
"""ODS loader helpers."""
from .generic import GenericODSLoader
__all__ = ["GenericODSLoader"]

View File

@@ -0,0 +1,67 @@
# -*- coding: utf-8 -*-
"""Generic ODS loader that keeps raw payload + primary keys."""
from __future__ import annotations
import json
from datetime import datetime, timezone
from typing import Iterable, Sequence
from ..base_loader import BaseLoader
class GenericODSLoader(BaseLoader):
"""Insert/update helper for ODS tables that share the same pattern."""
def __init__(
self,
db_ops,
table_name: str,
columns: Sequence[str],
conflict_columns: Sequence[str],
):
super().__init__(db_ops)
if not conflict_columns:
raise ValueError("conflict_columns must not be empty for ODS loader")
self.table_name = table_name
self.columns = list(columns)
self.conflict_columns = list(conflict_columns)
self._sql = self._build_sql()
def upsert_rows(self, rows: Iterable[dict]) -> tuple[int, int, int]:
"""Insert/update the provided iterable of dictionaries."""
rows = list(rows)
if not rows:
return (0, 0, 0)
normalized = [self._normalize_row(row) for row in rows]
inserted, updated = self.db.batch_upsert_with_returning(
self._sql, normalized, page_size=self._batch_size()
)
return inserted, updated, 0
def _build_sql(self) -> str:
col_list = ", ".join(self.columns)
placeholders = ", ".join(f"%({col})s" for col in self.columns)
conflict_clause = ", ".join(self.conflict_columns)
update_columns = [c for c in self.columns if c not in self.conflict_columns]
set_clause = ", ".join(f"{col} = EXCLUDED.{col}" for col in update_columns)
return (
f"INSERT INTO {self.table_name} ({col_list}) "
f"VALUES ({placeholders}) "
f"ON CONFLICT ({conflict_clause}) DO UPDATE SET {set_clause} "
f"RETURNING (xmax = 0) AS inserted"
)
def _normalize_row(self, row: dict) -> dict:
normalized = {}
for col in self.columns:
value = row.get(col)
if col == "payload" and value is not None and not isinstance(value, str):
normalized[col] = json.dumps(value, ensure_ascii=False)
else:
normalized[col] = value
if "fetched_at" in normalized and normalized["fetched_at"] is None:
normalized["fetched_at"] = datetime.now(timezone.utc)
return normalized

View File

@@ -1,131 +1,234 @@
# -*- coding: utf-8 -*-
"""ETL调度"""
"""ETL 调度:支持在线抓取、离线清洗入库、全流程三种模式。"""
from __future__ import annotations
import uuid
from datetime import datetime
from pathlib import Path
from zoneinfo import ZoneInfo
from api.client import APIClient
from api.local_json_client import LocalJsonClient
from api.recording_client import RecordingAPIClient
from database.connection import DatabaseConnection
from database.operations import DatabaseOperations
from api.client import APIClient
from orchestration.cursor_manager import CursorManager
from orchestration.run_tracker import RunTracker
from orchestration.task_registry import default_registry
class ETLScheduler:
"""ETL任务调度器"""
"""调度多个任务,按 pipeline.flow 执行抓取/清洗入库。"""
def __init__(self, config, logger):
self.config = config
self.logger = logger
self.tz = ZoneInfo(config.get("app.timezone", "Asia/Taipei"))
# 初始化组件
self.pipeline_flow = str(config.get("pipeline.flow", "FULL") or "FULL").upper()
self.fetch_root = Path(config.get("pipeline.fetch_root") or config["io"]["export_root"])
self.ingest_source_dir = config.get("pipeline.ingest_source_dir") or ""
self.write_pretty_json = bool(config.get("io.write_pretty_json", False))
# 组件
self.db_conn = DatabaseConnection(
dsn=config["db"]["dsn"],
session=config["db"].get("session"),
connect_timeout=config["db"].get("connect_timeout_sec")
connect_timeout=config["db"].get("connect_timeout_sec"),
)
self.db_ops = DatabaseOperations(self.db_conn)
self.api_client = APIClient(
base_url=config["api"]["base_url"],
token=config["api"]["token"],
timeout=config["api"]["timeout_sec"],
retry_max=config["api"]["retries"]["max_attempts"],
headers_extra=config["api"].get("headers_extra")
headers_extra=config["api"].get("headers_extra"),
)
self.cursor_mgr = CursorManager(self.db_conn)
self.run_tracker = RunTracker(self.db_conn)
self.task_registry = default_registry
def run_tasks(self, task_codes: list = None):
"""运行任务列表"""
# ------------------------------------------------------------------ public
def run_tasks(self, task_codes: list | None = None):
"""按配置或传入列表执行任务。"""
run_uuid = uuid.uuid4().hex
store_id = self.config.get("app.store_id")
if not task_codes:
task_codes = self.config.get("run.tasks", [])
self.logger.info(f"开始运行任务: {task_codes}, run_uuid={run_uuid}")
self.logger.info("开始运行任务: %s, run_uuid=%s", task_codes, run_uuid)
for task_code in task_codes:
try:
self._run_single_task(task_code, run_uuid, store_id)
except Exception as e:
self.logger.error(f"任务 {task_code} 失败: {e}", exc_info=True)
except Exception as exc: # noqa: BLE001
self.logger.error("任务 %s 失败: %s", task_code, exc, exc_info=True)
continue
self.logger.info("所有任务执行完成")
# ------------------------------------------------------------------ internals
def _run_single_task(self, task_code: str, run_uuid: str, store_id: int):
"""运行单个任务"""
# 创建任务实例
task = self.task_registry.create_task(
task_code, self.config, self.db_ops, self.api_client, self.logger
)
# 获取任务配置(从数据库)
"""单个任务的抓取/清洗编排。"""
task_cfg = self._load_task_config(task_code, store_id)
if not task_cfg:
self.logger.warning(f"任务 {task_code} 未启用或不存在")
self.logger.warning("任务 %s 未启用或不存在", task_code)
return
task_id = task_cfg["task_id"]
# 创建运行记录
cursor_data = self.cursor_mgr.get_or_create(task_id, store_id)
# run 记录
export_dir = Path(self.config["io"]["export_root"]) / datetime.now(self.tz).strftime("%Y%m%d")
log_path = str(Path(self.config["io"]["log_root"]) / f"{run_uuid}.log")
run_id = self.run_tracker.create_run(
task_id=task_id,
store_id=store_id,
run_uuid=run_uuid,
export_dir=str(export_dir),
log_path=log_path,
status="RUNNING"
status=self._map_run_status("RUNNING"),
)
# 执行任务
# 为抓取阶段准备目录
fetch_dir = self._build_fetch_dir(task_code, run_id)
fetch_stats = None
try:
result = task.execute()
# 更新运行记录
self.run_tracker.update_run(
run_id=run_id,
counts=result["counts"],
status=result["status"],
ended_at=datetime.now(self.tz)
)
# 推进游标
if result["status"] == "SUCCESS":
# TODO: 从任务结果中获取窗口信息
pass
except Exception as e:
if self._flow_includes_fetch():
fetch_stats = self._execute_fetch(task_code, cursor_data, fetch_dir, run_id)
if self.pipeline_flow == "FETCH_ONLY":
counts = self._counts_from_fetch(fetch_stats)
self.run_tracker.update_run(
run_id=run_id,
counts=counts,
status=self._map_run_status("SUCCESS"),
ended_at=datetime.now(self.tz),
)
return
if self._flow_includes_ingest():
source_dir = self._resolve_ingest_source(fetch_dir, fetch_stats)
result = self._execute_ingest(task_code, cursor_data, source_dir)
self.run_tracker.update_run(
run_id=run_id,
counts=result["counts"],
status=self._map_run_status(result["status"]),
ended_at=datetime.now(self.tz),
)
if (result.get("status") or "").upper() == "SUCCESS":
window = result.get("window")
if window:
self.cursor_mgr.advance(
task_id=task_id,
store_id=store_id,
window_start=window.get("start"),
window_end=window.get("end"),
run_id=run_id,
)
except Exception as exc: # noqa: BLE001
self.run_tracker.update_run(
run_id=run_id,
counts={},
status="FAIL",
status=self._map_run_status("FAIL"),
ended_at=datetime.now(self.tz),
error_message=str(e)
error_message=str(exc),
)
raise
def _load_task_config(self, task_code: str, store_id: int) -> dict:
"""从数据库加载任务配置"""
def _execute_fetch(self, task_code: str, cursor_data: dict | None, fetch_dir: Path, run_id: int):
"""在线抓取阶段:用 RecordingAPIClient 拉取并落盘,不做 Transform/Load。"""
recording_client = RecordingAPIClient(
base_client=self.api_client,
output_dir=fetch_dir,
task_code=task_code,
run_id=run_id,
write_pretty=self.write_pretty_json,
)
task = self.task_registry.create_task(task_code, self.config, self.db_ops, recording_client, self.logger)
context = task._build_context(cursor_data) # type: ignore[attr-defined]
self.logger.info("%s: 抓取阶段开始,目录=%s", task_code, fetch_dir)
extracted = task.extract(context)
# 抓取结束,不执行 transform/load
stats = recording_client.last_dump or {}
fetched_count = stats.get("records") or len(extracted.get("records", [])) if isinstance(extracted, dict) else 0
self.logger.info(
"%s: 抓取完成,文件=%s,记录数=%s",
task_code,
stats.get("file"),
fetched_count,
)
return {"file": stats.get("file"), "records": fetched_count, "pages": stats.get("pages")}
def _execute_ingest(self, task_code: str, cursor_data: dict | None, source_dir: Path):
"""本地清洗入库:使用 LocalJsonClient 回放 JSON走原有任务 ETL。"""
local_client = LocalJsonClient(source_dir)
task = self.task_registry.create_task(task_code, self.config, self.db_ops, local_client, self.logger)
self.logger.info("%s: 本地清洗入库开始,源目录=%s", task_code, source_dir)
return task.execute(cursor_data)
def _build_fetch_dir(self, task_code: str, run_id: int) -> Path:
ts = datetime.now(self.tz).strftime("%Y%m%d-%H%M%S")
return Path(self.fetch_root) / f"{task_code.upper()}-{run_id}-{ts}"
def _resolve_ingest_source(self, fetch_dir: Path, fetch_stats: dict | None) -> Path:
if fetch_stats and fetch_dir.exists():
return fetch_dir
if self.ingest_source_dir:
return Path(self.ingest_source_dir)
raise FileNotFoundError("未提供本地清洗入库所需的 JSON 目录")
def _counts_from_fetch(self, stats: dict | None) -> dict:
fetched = (stats or {}).get("records") or 0
return {
"fetched": fetched,
"inserted": 0,
"updated": 0,
"skipped": 0,
"errors": 0,
}
def _flow_includes_fetch(self) -> bool:
return self.pipeline_flow in {"FETCH_ONLY", "FULL"}
def _flow_includes_ingest(self) -> bool:
return self.pipeline_flow in {"INGEST_ONLY", "FULL"}
def _load_task_config(self, task_code: str, store_id: int) -> dict | None:
"""从数据库加载任务配置。"""
sql = """
SELECT task_id, task_code, store_id, enabled, cursor_field,
window_minutes_default, overlap_seconds, page_size, retry_max, params
FROM etl_admin.etl_task
WHERE store_id = %s AND task_code = %s AND enabled = TRUE
"""
rows = self.db_conn.query(sql, (store_id, task_code))
return rows[0] if rows else None
def close(self):
"""关闭连接"""
"""关闭连接"""
self.db_conn.close()
@staticmethod
def _map_run_status(status: str) -> str:
"""
将任务返回的状态转换为 etl_admin.run_status_enum
(SUCC / FAIL / PARTIAL)
"""
normalized = (status or "").upper()
if normalized in {"SUCCESS", "SUCC"}:
return "SUCC"
if normalized in {"FAIL", "FAILED", "ERROR"}:
return "FAIL"
if normalized in {"RUNNING", "PARTIAL", "PENDING", "IN_PROGRESS"}:
return "PARTIAL"
# 未知状态默认标记为 FAIL便于排查
return "FAIL"

View File

@@ -3,6 +3,22 @@
from tasks.orders_task import OrdersTask
from tasks.payments_task import PaymentsTask
from tasks.members_task import MembersTask
from tasks.products_task import ProductsTask
from tasks.tables_task import TablesTask
from tasks.assistants_task import AssistantsTask
from tasks.packages_task import PackagesDefTask
from tasks.refunds_task import RefundsTask
from tasks.coupon_usage_task import CouponUsageTask
from tasks.inventory_change_task import InventoryChangeTask
from tasks.topups_task import TopupsTask
from tasks.table_discount_task import TableDiscountTask
from tasks.assistant_abolish_task import AssistantAbolishTask
from tasks.ledger_task import LedgerTask
from tasks.ods_tasks import ODS_TASK_CLASSES
from tasks.ticket_dwd_task import TicketDwdTask
from tasks.manual_ingest_task import ManualIngestTask
from tasks.payments_dwd_task import PaymentsDwdTask
from tasks.members_dwd_task import MembersDwdTask
class TaskRegistry:
"""任务注册和工厂"""
@@ -30,7 +46,23 @@ class TaskRegistry:
# 默认注册表
default_registry = TaskRegistry()
default_registry.register("PRODUCTS", ProductsTask)
default_registry.register("TABLES", TablesTask)
default_registry.register("MEMBERS", MembersTask)
default_registry.register("ASSISTANTS", AssistantsTask)
default_registry.register("PACKAGES_DEF", PackagesDefTask)
default_registry.register("ORDERS", OrdersTask)
default_registry.register("PAYMENTS", PaymentsTask)
default_registry.register("MEMBERS", MembersTask)
# 可以继续注册其他任务...
default_registry.register("REFUNDS", RefundsTask)
default_registry.register("COUPON_USAGE", CouponUsageTask)
default_registry.register("INVENTORY_CHANGE", InventoryChangeTask)
default_registry.register("TOPUPS", TopupsTask)
default_registry.register("TABLE_DISCOUNT", TableDiscountTask)
default_registry.register("ASSISTANT_ABOLISH", AssistantAbolishTask)
default_registry.register("LEDGER", LedgerTask)
default_registry.register("TICKET_DWD", TicketDwdTask)
default_registry.register("MANUAL_INGEST", ManualIngestTask)
default_registry.register("PAYMENTS_DWD", PaymentsDwdTask)
default_registry.register("MEMBERS_DWD", MembersDwdTask)
for code, task_cls in ODS_TASK_CLASSES.items():
default_registry.register(code, task_cls)

View File

@@ -1,92 +1,89 @@
# -*- coding: utf-8 -*-
"""SCD2 (Slowly Changing Dimension Type 2) 处理"""
"""SCD2 (Slowly Changing Dimension Type 2) 处理逻辑"""
from datetime import datetime
def _row_to_dict(cursor, row):
if row is None:
return None
columns = [desc[0] for desc in cursor.description]
return {col: row[idx] for idx, col in enumerate(columns)}
class SCD2Handler:
"""SCD2历史记录处理"""
"""SCD2历史记录处理"""
def __init__(self, db_ops):
self.db = db_ops
def upsert(self, table_name: str, natural_key: list, tracked_fields: list,
record: dict, effective_date: datetime = None) -> str:
def upsert(
self,
table_name: str,
natural_key: list,
tracked_fields: list,
record: dict,
effective_date: datetime = None,
) -> str:
"""
处理SCD2更新
Args:
table_name: 表名
natural_key: 自然键字段列表
tracked_fields: 需要跟踪变化的字段列表
record: 记录数据
effective_date: 生效日期
Returns:
操作类型: 'INSERT', 'UPDATE', 'UNCHANGED'
"""
effective_date = effective_date or datetime.now()
# 查找当前有效记录
where_clause = " AND ".join([f"{k} = %({k})s" for k in natural_key])
sql_select = f"""
SELECT * FROM {table_name}
WHERE {where_clause}
AND valid_to IS NULL
"""
# 使用 db 的 connection
current = self.db.conn.cursor()
current.execute(sql_select, record)
existing = current.fetchone()
if not existing:
# 新记录:直接插入
with self.db.conn.cursor() as current:
current.execute(sql_select, record)
existing = _row_to_dict(current, current.fetchone())
if not existing:
record["valid_from"] = effective_date
record["valid_to"] = None
record["is_current"] = True
fields = list(record.keys())
placeholders = ", ".join([f"%({f})s" for f in fields])
sql_insert = f"""
INSERT INTO {table_name} ({', '.join(fields)})
VALUES ({placeholders})
"""
current.execute(sql_insert, record)
return "INSERT"
has_changes = any(existing.get(field) != record.get(field) for field in tracked_fields)
if not has_changes:
return "UNCHANGED"
update_where = " AND ".join([f"{k} = %({k})s" for k in natural_key])
sql_close = f"""
UPDATE {table_name}
SET valid_to = %(effective_date)s,
is_current = FALSE
WHERE {update_where}
AND valid_to IS NULL
"""
record["effective_date"] = effective_date
current.execute(sql_close, record)
record["valid_from"] = effective_date
record["valid_to"] = None
record["is_current"] = True
fields = list(record.keys())
if "effective_date" in fields:
fields.remove("effective_date")
placeholders = ", ".join([f"%({f})s" for f in fields])
sql_insert = f"""
INSERT INTO {table_name} ({', '.join(fields)})
VALUES ({placeholders})
"""
current.execute(sql_insert, record)
return 'INSERT'
# 检查是否有变化
has_changes = any(
existing.get(field) != record.get(field)
for field in tracked_fields
)
if not has_changes:
return 'UNCHANGED'
# 有变化:关闭旧记录,插入新记录
update_where = " AND ".join([f"{k} = %({k})s" for k in natural_key])
sql_close = f"""
UPDATE {table_name}
SET valid_to = %(effective_date)s,
is_current = FALSE
WHERE {update_where}
AND valid_to IS NULL
"""
record["effective_date"] = effective_date
current.execute(sql_close, record)
# 插入新记录
record["valid_from"] = effective_date
record["valid_to"] = None
record["is_current"] = True
fields = list(record.keys())
if "effective_date" in fields:
fields.remove("effective_date")
placeholders = ", ".join([f"%({f})s" for f in fields])
sql_insert = f"""
INSERT INTO {table_name} ({', '.join(fields)})
VALUES ({placeholders})
"""
current.execute(sql_insert, record)
return 'UPDATE'
return "UPDATE"

View File

View File

@@ -0,0 +1,76 @@
# -*- coding: utf-8 -*-
"""Apply the PRD-aligned warehouse schema (ODS/DWD/DWS) to PostgreSQL."""
from __future__ import annotations
import argparse
import os
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[1]
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from database.connection import DatabaseConnection # noqa: E402
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Create/upgrade warehouse schemas using schema_v2.sql"
)
parser.add_argument(
"--dsn",
help="PostgreSQL DSN (fallback to PG_DSN env)",
default=os.environ.get("PG_DSN"),
)
parser.add_argument(
"--file",
help="Path to schema SQL",
default=str(PROJECT_ROOT / "database" / "schema_v2.sql"),
)
parser.add_argument(
"--timeout",
type=int,
default=int(os.environ.get("PG_CONNECT_TIMEOUT", 10) or 10),
help="connect_timeout seconds (capped at 20, default 10)",
)
return parser.parse_args()
def apply_schema(dsn: str, sql_path: Path, timeout: int) -> None:
if not sql_path.exists():
raise FileNotFoundError(f"Schema file not found: {sql_path}")
sql_text = sql_path.read_text(encoding="utf-8")
timeout_val = max(1, min(timeout, 20))
conn = DatabaseConnection(dsn, connect_timeout=timeout_val)
try:
with conn.conn.cursor() as cur:
cur.execute(sql_text)
conn.commit()
except Exception:
conn.rollback()
raise
finally:
conn.close()
def main() -> int:
args = parse_args()
if not args.dsn:
print("Missing DSN. Set PG_DSN or pass --dsn.", file=sys.stderr)
return 2
try:
apply_schema(args.dsn, Path(args.file), args.timeout)
except Exception as exc: # pragma: no cover - utility script
print(f"Schema apply failed: {exc}", file=sys.stderr)
return 1
print("Schema applied successfully.")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,425 @@
# -*- coding: utf-8 -*-
"""Populate PRD DWD tables from ODS payload snapshots."""
from __future__ import annotations
import argparse
import os
import sys
import psycopg2
SQL_STEPS: list[tuple[str, str]] = [
(
"dim_tenant",
"""
INSERT INTO billiards_dwd.dim_tenant (tenant_id, tenant_name, status)
SELECT DISTINCT tenant_id, 'default' AS tenant_name, 'active' AS status
FROM (
SELECT tenant_id FROM billiards_ods.ods_order_settle
UNION SELECT tenant_id FROM billiards_ods.ods_order_receipt_detail
UNION SELECT tenant_id FROM billiards_ods.ods_member_profile
) s
WHERE tenant_id IS NOT NULL
ON CONFLICT (tenant_id) DO UPDATE SET updated_at = now();
""",
),
(
"dim_site",
"""
INSERT INTO billiards_dwd.dim_site (site_id, tenant_id, site_name, status)
SELECT DISTINCT site_id, MAX(tenant_id) AS tenant_id, 'default' AS site_name, 'active' AS status
FROM (
SELECT site_id, tenant_id FROM billiards_ods.ods_order_settle
UNION SELECT site_id, tenant_id FROM billiards_ods.ods_order_receipt_detail
UNION SELECT site_id, tenant_id FROM billiards_ods.ods_table_info
) s
WHERE site_id IS NOT NULL
GROUP BY site_id
ON CONFLICT (site_id) DO UPDATE SET updated_at = now();
""",
),
(
"dim_product_category",
"""
INSERT INTO billiards_dwd.dim_product_category (category_id, category_name, parent_id, level_no, status)
SELECT DISTINCT category_id, category_name, parent_id, level_no, status
FROM billiards_ods.ods_goods_category
WHERE category_id IS NOT NULL
ON CONFLICT (category_id) DO UPDATE SET
category_name = EXCLUDED.category_name,
parent_id = EXCLUDED.parent_id,
level_no = EXCLUDED.level_no,
status = EXCLUDED.status;
""",
),
(
"dim_product",
"""
INSERT INTO billiards_dwd.dim_product (goods_id, goods_name, goods_code, category_id, category_name, unit, default_price, status)
SELECT DISTINCT goods_id, goods_name, NULL::TEXT AS goods_code, category_id, category_name, NULL::TEXT AS unit, sale_price AS default_price, status
FROM billiards_ods.ods_store_product
WHERE goods_id IS NOT NULL
ON CONFLICT (goods_id) DO UPDATE SET
goods_name = EXCLUDED.goods_name,
category_id = EXCLUDED.category_id,
category_name = EXCLUDED.category_name,
default_price = EXCLUDED.default_price,
status = EXCLUDED.status,
updated_at = now();
""",
),
(
"dim_product_from_sales",
"""
INSERT INTO billiards_dwd.dim_product (goods_id, goods_name)
SELECT DISTINCT goods_id, goods_name
FROM billiards_ods.ods_store_sale_item
WHERE goods_id IS NOT NULL
ON CONFLICT (goods_id) DO NOTHING;
""",
),
(
"dim_member_card_type",
"""
INSERT INTO billiards_dwd.dim_member_card_type (card_type_id, card_type_name, discount_rate)
SELECT DISTINCT card_type_id, card_type_name, discount_rate
FROM billiards_ods.ods_member_card
WHERE card_type_id IS NOT NULL
ON CONFLICT (card_type_id) DO UPDATE SET
card_type_name = EXCLUDED.card_type_name,
discount_rate = EXCLUDED.discount_rate;
""",
),
(
"dim_member",
"""
INSERT INTO billiards_dwd.dim_member (
site_id, member_id, tenant_id, member_name, nickname, gender, birthday, mobile,
member_type_id, member_type_name, status, register_time, last_visit_time,
balance, total_recharge_amount, total_consumed_amount, wechat_id, alipay_id, remark
)
SELECT DISTINCT
prof.site_id,
prof.member_id,
prof.tenant_id,
prof.member_name,
prof.nickname,
prof.gender,
prof.birthday,
prof.mobile,
card.member_type_id,
card.member_type_name,
prof.status,
prof.register_time,
prof.last_visit_time,
prof.balance,
NULL::NUMERIC AS total_recharge_amount,
NULL::NUMERIC AS total_consumed_amount,
prof.wechat_id,
prof.alipay_id,
prof.remarks
FROM billiards_ods.ods_member_profile prof
LEFT JOIN (
SELECT DISTINCT site_id, member_id, card_type_id AS member_type_id, card_type_name AS member_type_name
FROM billiards_ods.ods_member_card
) card
ON prof.site_id = card.site_id AND prof.member_id = card.member_id
WHERE prof.member_id IS NOT NULL
ON CONFLICT (site_id, member_id) DO UPDATE SET
member_name = EXCLUDED.member_name,
nickname = EXCLUDED.nickname,
gender = EXCLUDED.gender,
birthday = EXCLUDED.birthday,
mobile = EXCLUDED.mobile,
member_type_id = EXCLUDED.member_type_id,
member_type_name = EXCLUDED.member_type_name,
status = EXCLUDED.status,
register_time = EXCLUDED.register_time,
last_visit_time = EXCLUDED.last_visit_time,
balance = EXCLUDED.balance,
wechat_id = EXCLUDED.wechat_id,
alipay_id = EXCLUDED.alipay_id,
remark = EXCLUDED.remark,
updated_at = now();
""",
),
(
"dim_table",
"""
INSERT INTO billiards_dwd.dim_table (table_id, site_id, table_code, table_name, table_type, area_name, status, created_time, updated_time)
SELECT DISTINCT table_id, site_id, table_code, table_name, table_type, area_name, status, created_time, updated_time
FROM billiards_ods.ods_table_info
WHERE table_id IS NOT NULL
ON CONFLICT (table_id) DO UPDATE SET
site_id = EXCLUDED.site_id,
table_code = EXCLUDED.table_code,
table_name = EXCLUDED.table_name,
table_type = EXCLUDED.table_type,
area_name = EXCLUDED.area_name,
status = EXCLUDED.status,
created_time = EXCLUDED.created_time,
updated_time = EXCLUDED.updated_time;
""",
),
(
"dim_assistant",
"""
INSERT INTO billiards_dwd.dim_assistant (assistant_id, assistant_name, mobile, status)
SELECT DISTINCT assistant_id, assistant_name, mobile, status
FROM billiards_ods.ods_assistant_account
WHERE assistant_id IS NOT NULL
ON CONFLICT (assistant_id) DO UPDATE SET
assistant_name = EXCLUDED.assistant_name,
mobile = EXCLUDED.mobile,
status = EXCLUDED.status,
updated_at = now();
""",
),
(
"dim_pay_method",
"""
INSERT INTO billiards_dwd.dim_pay_method (pay_method_code, pay_method_name, is_stored_value, status)
SELECT DISTINCT pay_method_code, pay_method_name, FALSE AS is_stored_value, 'active' AS status
FROM billiards_ods.ods_payment_record
WHERE pay_method_code IS NOT NULL
ON CONFLICT (pay_method_code) DO UPDATE SET
pay_method_name = EXCLUDED.pay_method_name,
status = EXCLUDED.status,
updated_at = now();
""",
),
(
"dim_coupon_platform",
"""
INSERT INTO billiards_dwd.dim_coupon_platform (platform_code, platform_name)
SELECT DISTINCT platform_code, platform_code AS platform_name
FROM billiards_ods.ods_platform_coupon_log
WHERE platform_code IS NOT NULL
ON CONFLICT (platform_code) DO NOTHING;
""",
),
(
"fact_sale_item",
"""
INSERT INTO billiards_dwd.fact_sale_item (
site_id, sale_item_id, order_trade_no, order_settle_id, member_id,
goods_id, category_id, quantity, original_amount, discount_amount,
final_amount, is_gift, sale_time
)
SELECT
site_id,
sale_item_id,
order_trade_no,
order_settle_id,
NULL::BIGINT AS member_id,
goods_id,
category_id,
quantity,
original_amount,
discount_amount,
final_amount,
COALESCE(is_gift, FALSE),
sale_time
FROM billiards_ods.ods_store_sale_item
ON CONFLICT (site_id, sale_item_id) DO NOTHING;
""",
),
(
"fact_table_usage",
"""
INSERT INTO billiards_dwd.fact_table_usage (
site_id, ledger_id, order_trade_no, order_settle_id, table_id,
member_id, start_time, end_time, duration_minutes,
original_table_fee, member_discount_amount, manual_discount_amount,
final_table_fee, is_canceled, cancel_time
)
SELECT
site_id,
ledger_id,
order_trade_no,
order_settle_id,
table_id,
member_id,
start_time,
end_time,
duration_minutes,
original_table_fee,
0::NUMERIC AS member_discount_amount,
discount_amount AS manual_discount_amount,
final_table_fee,
FALSE AS is_canceled,
NULL::TIMESTAMPTZ AS cancel_time
FROM billiards_ods.ods_table_use_log
ON CONFLICT (site_id, ledger_id) DO NOTHING;
""",
),
(
"fact_assistant_service",
"""
INSERT INTO billiards_dwd.fact_assistant_service (
site_id, ledger_id, order_trade_no, order_settle_id, assistant_id,
assist_type_code, member_id, start_time, end_time, duration_minutes,
original_fee, member_discount_amount, manual_discount_amount,
final_fee, is_canceled, cancel_time
)
SELECT
site_id,
ledger_id,
order_trade_no,
order_settle_id,
assistant_id,
NULL::TEXT AS assist_type_code,
member_id,
start_time,
end_time,
duration_minutes,
original_fee,
0::NUMERIC AS member_discount_amount,
discount_amount AS manual_discount_amount,
final_fee,
FALSE AS is_canceled,
NULL::TIMESTAMPTZ AS cancel_time
FROM billiards_ods.ods_assistant_service_log
ON CONFLICT (site_id, ledger_id) DO NOTHING;
""",
),
(
"fact_coupon_usage",
"""
INSERT INTO billiards_dwd.fact_coupon_usage (
site_id, coupon_id, package_id, order_trade_no, order_settle_id,
member_id, platform_code, status, deduct_amount, settle_price, used_time
)
SELECT
site_id,
coupon_id,
NULL::BIGINT AS package_id,
order_trade_no,
order_settle_id,
member_id,
platform_code,
status,
deduct_amount,
settle_price,
used_time
FROM billiards_ods.ods_platform_coupon_log
ON CONFLICT (site_id, coupon_id) DO NOTHING;
""",
),
(
"fact_payment",
"""
INSERT INTO billiards_dwd.fact_payment (
site_id, pay_id, order_trade_no, order_settle_id, member_id,
pay_method_code, pay_amount, pay_time, relate_type, relate_id
)
SELECT
site_id,
pay_id,
order_trade_no,
order_settle_id,
member_id,
pay_method_code,
pay_amount,
pay_time,
relate_type,
relate_id
FROM billiards_ods.ods_payment_record
ON CONFLICT (site_id, pay_id) DO NOTHING;
""",
),
(
"fact_refund",
"""
INSERT INTO billiards_dwd.fact_refund (
site_id, refund_id, order_trade_no, order_settle_id, member_id,
pay_method_code, refund_amount, refund_time, status
)
SELECT
site_id,
refund_id,
order_trade_no,
order_settle_id,
member_id,
pay_method_code,
refund_amount,
refund_time,
status
FROM billiards_ods.ods_refund_record
ON CONFLICT (site_id, refund_id) DO NOTHING;
""",
),
(
"fact_balance_change",
"""
INSERT INTO billiards_dwd.fact_balance_change (
site_id, change_id, member_id, change_type, relate_type, relate_id,
pay_method_code, change_amount, balance_before, balance_after, change_time
)
SELECT
site_id,
change_id,
member_id,
change_type,
NULL::TEXT AS relate_type,
relate_id,
NULL::TEXT AS pay_method_code,
change_amount,
balance_before,
balance_after,
change_time
FROM billiards_ods.ods_balance_change
ON CONFLICT (site_id, change_id) DO NOTHING;
""",
),
]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Build DWD tables from ODS payloads (PRD schema).")
parser.add_argument(
"--dsn",
default=os.environ.get("PG_DSN"),
help="PostgreSQL DSN (fallback PG_DSN env)",
)
parser.add_argument(
"--timeout",
type=int,
default=int(os.environ.get("PG_CONNECT_TIMEOUT", 10) or 10),
help="connect_timeout seconds (capped at 20, default 10)",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if not args.dsn:
print("Missing DSN. Use --dsn or PG_DSN.", file=sys.stderr)
return 2
timeout_val = max(1, min(args.timeout, 20))
conn = psycopg2.connect(args.dsn, connect_timeout=timeout_val)
conn.autocommit = False
try:
with conn.cursor() as cur:
for name, sql in SQL_STEPS:
cur.execute(sql)
print(f"[OK] {name}")
conn.commit()
except Exception as exc: # pragma: no cover - operational script
conn.rollback()
print(f"[FAIL] {exc}", file=sys.stderr)
return 1
finally:
try:
conn.close()
except Exception:
pass
print("DWD build complete.")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,322 @@
# -*- coding: utf-8 -*-
"""Recompute billiards_dws.dws_order_summary from DWD fact tables."""
from __future__ import annotations
import argparse
import os
import sys
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parents[1]
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from database.connection import DatabaseConnection # noqa: E402
SQL_BUILD_SUMMARY = r"""
WITH table_fee AS (
SELECT
site_id,
order_settle_id,
order_trade_no,
MIN(member_id) AS member_id,
SUM(COALESCE(final_table_fee, 0)) AS table_fee_amount,
SUM(COALESCE(member_discount_amount, 0)) AS member_discount_amount,
SUM(COALESCE(manual_discount_amount, 0)) AS manual_discount_amount,
SUM(COALESCE(original_table_fee, 0)) AS original_table_fee,
MIN(start_time) AS first_time
FROM billiards_dwd.fact_table_usage
WHERE (%(site_id)s IS NULL OR site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR start_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR start_time::date <= %(end_date)s)
AND COALESCE(is_canceled, FALSE) = FALSE
GROUP BY site_id, order_settle_id, order_trade_no
),
assistant_fee AS (
SELECT
site_id,
order_settle_id,
order_trade_no,
MIN(member_id) AS member_id,
SUM(COALESCE(final_fee, 0)) AS assistant_service_amount,
SUM(COALESCE(member_discount_amount, 0)) AS member_discount_amount,
SUM(COALESCE(manual_discount_amount, 0)) AS manual_discount_amount,
SUM(COALESCE(original_fee, 0)) AS original_fee,
MIN(start_time) AS first_time
FROM billiards_dwd.fact_assistant_service
WHERE (%(site_id)s IS NULL OR site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR start_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR start_time::date <= %(end_date)s)
AND COALESCE(is_canceled, FALSE) = FALSE
GROUP BY site_id, order_settle_id, order_trade_no
),
goods_fee AS (
SELECT
site_id,
order_settle_id,
order_trade_no,
MIN(member_id) AS member_id,
SUM(COALESCE(final_amount, 0)) FILTER (WHERE COALESCE(is_gift, FALSE) = FALSE) AS goods_amount,
SUM(COALESCE(discount_amount, 0)) FILTER (WHERE COALESCE(is_gift, FALSE) = FALSE) AS goods_discount_amount,
SUM(COALESCE(original_amount, 0)) FILTER (WHERE COALESCE(is_gift, FALSE) = FALSE) AS goods_original_amount,
COUNT(*) FILTER (WHERE COALESCE(is_gift, FALSE) = FALSE) AS item_count,
SUM(COALESCE(quantity, 0)) FILTER (WHERE COALESCE(is_gift, FALSE) = FALSE) AS total_item_quantity,
MIN(sale_time) AS first_time
FROM billiards_dwd.fact_sale_item
WHERE (%(site_id)s IS NULL OR site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR sale_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR sale_time::date <= %(end_date)s)
GROUP BY site_id, order_settle_id, order_trade_no
),
coupon_usage AS (
SELECT
site_id,
order_settle_id,
order_trade_no,
MIN(member_id) AS member_id,
SUM(COALESCE(deduct_amount, 0)) AS coupon_deduction,
SUM(COALESCE(settle_price, 0)) AS settle_price,
MIN(used_time) AS first_time
FROM billiards_dwd.fact_coupon_usage
WHERE (%(site_id)s IS NULL OR site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR used_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR used_time::date <= %(end_date)s)
GROUP BY site_id, order_settle_id, order_trade_no
),
payments AS (
SELECT
fp.site_id,
fp.order_settle_id,
fp.order_trade_no,
MIN(fp.member_id) AS member_id,
SUM(COALESCE(fp.pay_amount, 0)) AS total_paid_amount,
SUM(COALESCE(fp.pay_amount, 0)) FILTER (WHERE COALESCE(pm.is_stored_value, FALSE)) AS stored_card_deduct,
SUM(COALESCE(fp.pay_amount, 0)) FILTER (WHERE NOT COALESCE(pm.is_stored_value, FALSE)) AS external_paid_amount,
MIN(fp.pay_time) AS first_time
FROM billiards_dwd.fact_payment fp
LEFT JOIN billiards_dwd.dim_pay_method pm ON fp.pay_method_code = pm.pay_method_code
WHERE (%(site_id)s IS NULL OR fp.site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR fp.pay_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR fp.pay_time::date <= %(end_date)s)
GROUP BY fp.site_id, fp.order_settle_id, fp.order_trade_no
),
refunds AS (
SELECT
site_id,
order_settle_id,
order_trade_no,
SUM(COALESCE(refund_amount, 0)) AS refund_amount
FROM billiards_dwd.fact_refund
WHERE (%(site_id)s IS NULL OR site_id = %(site_id)s)
AND (%(start_date)s IS NULL OR refund_time::date >= %(start_date)s)
AND (%(end_date)s IS NULL OR refund_time::date <= %(end_date)s)
GROUP BY site_id, order_settle_id, order_trade_no
),
combined_ids AS (
SELECT site_id, order_settle_id, order_trade_no FROM table_fee
UNION
SELECT site_id, order_settle_id, order_trade_no FROM assistant_fee
UNION
SELECT site_id, order_settle_id, order_trade_no FROM goods_fee
UNION
SELECT site_id, order_settle_id, order_trade_no FROM coupon_usage
UNION
SELECT site_id, order_settle_id, order_trade_no FROM payments
UNION
SELECT site_id, order_settle_id, order_trade_no FROM refunds
),
site_dim AS (
SELECT site_id, tenant_id FROM billiards_dwd.dim_site
)
INSERT INTO billiards_dws.dws_order_summary (
site_id,
order_settle_id,
order_trade_no,
order_date,
tenant_id,
member_id,
member_flag,
recharge_order_flag,
item_count,
total_item_quantity,
table_fee_amount,
assistant_service_amount,
goods_amount,
group_amount,
total_coupon_deduction,
member_discount_amount,
manual_discount_amount,
order_original_amount,
order_final_amount,
stored_card_deduct,
external_paid_amount,
total_paid_amount,
book_table_flow,
book_assistant_flow,
book_goods_flow,
book_group_flow,
book_order_flow,
order_effective_consume_cash,
order_effective_recharge_cash,
order_effective_flow,
refund_amount,
net_income,
created_at,
updated_at
)
SELECT
c.site_id,
c.order_settle_id,
c.order_trade_no,
COALESCE(tf.first_time, af.first_time, gf.first_time, pay.first_time, cu.first_time)::date AS order_date,
sd.tenant_id,
COALESCE(tf.member_id, af.member_id, gf.member_id, cu.member_id, pay.member_id) AS member_id,
COALESCE(tf.member_id, af.member_id, gf.member_id, cu.member_id, pay.member_id) IS NOT NULL AS member_flag,
-- recharge flag: no consumption side but has payments
(COALESCE(tf.table_fee_amount, 0) + COALESCE(af.assistant_service_amount, 0) + COALESCE(gf.goods_amount, 0) + COALESCE(cu.settle_price, 0) = 0)
AND COALESCE(pay.total_paid_amount, 0) > 0 AS recharge_order_flag,
COALESCE(gf.item_count, 0) AS item_count,
COALESCE(gf.total_item_quantity, 0) AS total_item_quantity,
COALESCE(tf.table_fee_amount, 0) AS table_fee_amount,
COALESCE(af.assistant_service_amount, 0) AS assistant_service_amount,
COALESCE(gf.goods_amount, 0) AS goods_amount,
COALESCE(cu.settle_price, 0) AS group_amount,
COALESCE(cu.coupon_deduction, 0) AS total_coupon_deduction,
COALESCE(tf.member_discount_amount, 0) + COALESCE(af.member_discount_amount, 0) + COALESCE(gf.goods_discount_amount, 0) AS member_discount_amount,
COALESCE(tf.manual_discount_amount, 0) + COALESCE(af.manual_discount_amount, 0) AS manual_discount_amount,
COALESCE(tf.original_table_fee, 0) + COALESCE(af.original_fee, 0) + COALESCE(gf.goods_original_amount, 0) AS order_original_amount,
COALESCE(tf.table_fee_amount, 0) + COALESCE(af.assistant_service_amount, 0) + COALESCE(gf.goods_amount, 0) + COALESCE(cu.settle_price, 0) - COALESCE(cu.coupon_deduction, 0) AS order_final_amount,
COALESCE(pay.stored_card_deduct, 0) AS stored_card_deduct,
COALESCE(pay.external_paid_amount, 0) AS external_paid_amount,
COALESCE(pay.total_paid_amount, 0) AS total_paid_amount,
COALESCE(tf.table_fee_amount, 0) AS book_table_flow,
COALESCE(af.assistant_service_amount, 0) AS book_assistant_flow,
COALESCE(gf.goods_amount, 0) AS book_goods_flow,
COALESCE(cu.settle_price, 0) AS book_group_flow,
COALESCE(tf.table_fee_amount, 0) + COALESCE(af.assistant_service_amount, 0) + COALESCE(gf.goods_amount, 0) + COALESCE(cu.settle_price, 0) AS book_order_flow,
CASE
WHEN (COALESCE(tf.table_fee_amount, 0) + COALESCE(af.assistant_service_amount, 0) + COALESCE(gf.goods_amount, 0) + COALESCE(cu.settle_price, 0) = 0)
THEN 0
ELSE COALESCE(pay.external_paid_amount, 0)
END AS order_effective_consume_cash,
CASE
WHEN (COALESCE(tf.table_fee_amount, 0) + COALESCE(af.assistant_service_amount, 0) + COALESCE(gf.goods_amount, 0) + COALESCE(cu.settle_price, 0) = 0)
THEN COALESCE(pay.external_paid_amount, 0)
ELSE 0
END AS order_effective_recharge_cash,
COALESCE(pay.external_paid_amount, 0) + COALESCE(cu.settle_price, 0) AS order_effective_flow,
COALESCE(rf.refund_amount, 0) AS refund_amount,
(COALESCE(pay.external_paid_amount, 0) + COALESCE(cu.settle_price, 0)) - COALESCE(rf.refund_amount, 0) AS net_income,
now() AS created_at,
now() AS updated_at
FROM combined_ids c
LEFT JOIN table_fee tf ON c.site_id = tf.site_id AND c.order_settle_id = tf.order_settle_id
LEFT JOIN assistant_fee af ON c.site_id = af.site_id AND c.order_settle_id = af.order_settle_id
LEFT JOIN goods_fee gf ON c.site_id = gf.site_id AND c.order_settle_id = gf.order_settle_id
LEFT JOIN coupon_usage cu ON c.site_id = cu.site_id AND c.order_settle_id = cu.order_settle_id
LEFT JOIN payments pay ON c.site_id = pay.site_id AND c.order_settle_id = pay.order_settle_id
LEFT JOIN refunds rf ON c.site_id = rf.site_id AND c.order_settle_id = rf.order_settle_id
LEFT JOIN site_dim sd ON c.site_id = sd.site_id
ON CONFLICT (site_id, order_settle_id) DO UPDATE SET
order_trade_no = EXCLUDED.order_trade_no,
order_date = EXCLUDED.order_date,
tenant_id = EXCLUDED.tenant_id,
member_id = EXCLUDED.member_id,
member_flag = EXCLUDED.member_flag,
recharge_order_flag = EXCLUDED.recharge_order_flag,
item_count = EXCLUDED.item_count,
total_item_quantity = EXCLUDED.total_item_quantity,
table_fee_amount = EXCLUDED.table_fee_amount,
assistant_service_amount = EXCLUDED.assistant_service_amount,
goods_amount = EXCLUDED.goods_amount,
group_amount = EXCLUDED.group_amount,
total_coupon_deduction = EXCLUDED.total_coupon_deduction,
member_discount_amount = EXCLUDED.member_discount_amount,
manual_discount_amount = EXCLUDED.manual_discount_amount,
order_original_amount = EXCLUDED.order_original_amount,
order_final_amount = EXCLUDED.order_final_amount,
stored_card_deduct = EXCLUDED.stored_card_deduct,
external_paid_amount = EXCLUDED.external_paid_amount,
total_paid_amount = EXCLUDED.total_paid_amount,
book_table_flow = EXCLUDED.book_table_flow,
book_assistant_flow = EXCLUDED.book_assistant_flow,
book_goods_flow = EXCLUDED.book_goods_flow,
book_group_flow = EXCLUDED.book_group_flow,
book_order_flow = EXCLUDED.book_order_flow,
order_effective_consume_cash = EXCLUDED.order_effective_consume_cash,
order_effective_recharge_cash = EXCLUDED.order_effective_recharge_cash,
order_effective_flow = EXCLUDED.order_effective_flow,
refund_amount = EXCLUDED.refund_amount,
net_income = EXCLUDED.net_income,
updated_at = now();
"""
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Build/update dws_order_summary from DWD fact tables."
)
parser.add_argument(
"--dsn",
default=os.environ.get("PG_DSN"),
help="PostgreSQL DSN (fallback: PG_DSN env)",
)
parser.add_argument(
"--site-id",
type=int,
default=None,
help="Filter by site_id (optional, default all sites)",
)
parser.add_argument(
"--start-date",
dest="start_date",
default=None,
help="Filter facts from this date (YYYY-MM-DD, optional)",
)
parser.add_argument(
"--end-date",
dest="end_date",
default=None,
help="Filter facts until this date (YYYY-MM-DD, optional)",
)
parser.add_argument(
"--timeout",
type=int,
default=int(os.environ.get("PG_CONNECT_TIMEOUT", 10) or 10),
help="connect_timeout seconds (capped at 20, default 10)",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if not args.dsn:
print("Missing DSN. Set PG_DSN or pass --dsn.", file=sys.stderr)
return 2
params = {
"site_id": args.site_id,
"start_date": args.start_date,
"end_date": args.end_date,
}
timeout_val = max(1, min(args.timeout, 20))
conn = DatabaseConnection(args.dsn, connect_timeout=timeout_val)
try:
with conn.conn.cursor() as cur:
cur.execute(SQL_BUILD_SUMMARY, params)
conn.commit()
except Exception as exc: # pragma: no cover - operational script
conn.rollback()
print(f"DWS build failed: {exc}", file=sys.stderr)
return 1
finally:
conn.close()
print("dws_order_summary refreshed.")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,258 @@
# -*- coding: utf-8 -*-
"""
从本地 JSON 示例目录重建 billiards_ods.* 表,并导入样例数据。
用法:
PYTHONPATH=. python -m etl_billiards.scripts.rebuild_ods_from_json [--dsn ...] [--json-dir ...] [--include ...] [--drop-schema-first]
依赖环境变量:
PG_DSN PostgreSQL 连接串(必填)
PG_CONNECT_TIMEOUT 可选,秒,默认 10
JSON_DOC_DIR 可选JSON 目录,默认 C:\\dev\\LLTQ\\export\\test-json-doc
ODS_INCLUDE_FILES 可选,逗号分隔文件名(不含 .json
ODS_DROP_SCHEMA_FIRST 可选true/false默认 true
"""
from __future__ import annotations
import argparse
import os
import re
import sys
import json
from pathlib import Path
from typing import Iterable, List, Tuple
import psycopg2
from psycopg2 import sql
from psycopg2.extras import Json, execute_values
DEFAULT_JSON_DIR = r"C:\dev\LLTQ\export\test-json-doc"
SPECIAL_LIST_PATHS: dict[str, tuple[str, ...]] = {
"assistant_accounts_master": ("data", "assistantInfos"),
"assistant_cancellation_records": ("data", "abolitionAssistants"),
"assistant_service_records": ("data", "orderAssistantDetails"),
"goods_stock_movements": ("data", "queryDeliveryRecordsList"),
"goods_stock_summary": ("data",),
"group_buy_packages": ("data", "packageCouponList"),
"group_buy_redemption_records": ("data", "siteTableUseDetailsList"),
"member_balance_changes": ("data", "tenantMemberCardLogs"),
"member_profiles": ("data", "tenantMemberInfos"),
"member_stored_value_cards": ("data", "tenantMemberCards"),
"recharge_settlements": ("data", "settleList"),
"settlement_records": ("data", "settleList"),
"site_tables_master": ("data", "siteTables"),
"stock_goods_category_tree": ("data", "goodsCategoryList"),
"store_goods_master": ("data", "orderGoodsList"),
"store_goods_sales_records": ("data", "orderGoodsLedgers"),
"table_fee_discount_records": ("data", "taiFeeAdjustInfos"),
"table_fee_transactions": ("data", "siteTableUseDetailsList"),
"tenant_goods_master": ("data", "tenantGoodsList"),
}
def sanitize_identifier(name: str) -> str:
"""将任意字符串转为可用的 SQL identifier小写、非字母数字转下划线"""
cleaned = re.sub(r"[^0-9a-zA-Z_]", "_", name.strip())
if not cleaned:
cleaned = "col"
if cleaned[0].isdigit():
cleaned = f"_{cleaned}"
return cleaned.lower()
def _extract_list_via_path(node, path: tuple[str, ...]):
cur = node
for key in path:
if isinstance(cur, dict):
cur = cur.get(key)
else:
return []
return cur if isinstance(cur, list) else []
def load_records(payload, list_path: tuple[str, ...] | None = None) -> list:
"""
尝试从 JSON 结构中提取记录列表:
- 直接是 list -> 返回
- dict 中 data 是 list -> 返回
- dict 中 data 是 dict取第一个 list 字段
- dict 中任意值是 list -> 返回
- 其余情况,包装为单条记录
"""
if list_path:
if isinstance(payload, list):
merged: list = []
for item in payload:
merged.extend(_extract_list_via_path(item, list_path))
if merged:
return merged
elif isinstance(payload, dict):
lst = _extract_list_via_path(payload, list_path)
if lst:
return lst
if isinstance(payload, list):
return payload
if isinstance(payload, dict):
data_node = payload.get("data")
if isinstance(data_node, list):
return data_node
if isinstance(data_node, dict):
for v in data_node.values():
if isinstance(v, list):
return v
for v in payload.values():
if isinstance(v, list):
return v
return [payload]
def collect_columns(records: Iterable[dict]) -> List[str]:
"""汇总所有顶层键,作为表字段;仅处理 dict 记录。"""
cols: set[str] = set()
for rec in records:
if isinstance(rec, dict):
cols.update(rec.keys())
return sorted(cols)
def create_table(cur, schema: str, table: str, columns: List[Tuple[str, str]]):
"""
创建表:字段全部 jsonb外加 source_file、record_index、payload、ingested_at。
columns: [(col_name, original_key)]
"""
fields = [sql.SQL("{} jsonb").format(sql.Identifier(col)) for col, _ in columns]
constraint_name = f"uq_{table}_source_record"
ddl = sql.SQL(
"CREATE TABLE IF NOT EXISTS {schema}.{table} ("
"source_file text,"
"record_index integer,"
"{cols},"
"payload jsonb,"
"ingested_at timestamptz default now(),"
"CONSTRAINT {constraint} UNIQUE (source_file, record_index)"
");"
).format(
schema=sql.Identifier(schema),
table=sql.Identifier(table),
cols=sql.SQL(",").join(fields),
constraint=sql.Identifier(constraint_name),
)
cur.execute(ddl)
def insert_records(cur, schema: str, table: str, columns: List[Tuple[str, str]], records: list, source_file: str):
"""批量插入记录。"""
col_idents = [sql.Identifier(col) for col, _ in columns]
col_names = [col for col, _ in columns]
orig_keys = [orig for _, orig in columns]
all_cols = [sql.Identifier("source_file"), sql.Identifier("record_index")] + col_idents + [
sql.Identifier("payload")
]
rows = []
for idx, rec in enumerate(records):
if not isinstance(rec, dict):
rec = {"value": rec}
row_values = [source_file, idx]
for key in orig_keys:
row_values.append(Json(rec.get(key)))
row_values.append(Json(rec))
rows.append(row_values)
insert_sql = sql.SQL("INSERT INTO {}.{} ({}) VALUES %s ON CONFLICT DO NOTHING").format(
sql.Identifier(schema),
sql.Identifier(table),
sql.SQL(",").join(all_cols),
)
execute_values(cur, insert_sql, rows, page_size=500)
def rebuild(schema: str = "billiards_ods", data_dir: str | Path = DEFAULT_JSON_DIR):
parser = argparse.ArgumentParser(description="重建 billiards_ods.* 表并导入 JSON 样例")
parser.add_argument("--dsn", dest="dsn", help="PostgreSQL DSN默认读取环境变量 PG_DSN")
parser.add_argument("--json-dir", dest="json_dir", help=f"JSON 目录,默认 {DEFAULT_JSON_DIR}")
parser.add_argument(
"--include",
dest="include_files",
help="限定导入的文件名(逗号分隔,不含 .json默认全部",
)
parser.add_argument(
"--drop-schema-first",
dest="drop_schema_first",
action="store_true",
help="先删除并重建 schema默认 true",
)
parser.add_argument(
"--no-drop-schema-first",
dest="drop_schema_first",
action="store_false",
help="保留现有 schema仅按冲突去重导入",
)
parser.set_defaults(drop_schema_first=None)
args = parser.parse_args()
dsn = args.dsn or os.environ.get("PG_DSN")
if not dsn:
print("缺少参数/环境变量 PG_DSN无法连接数据库。")
sys.exit(1)
timeout = max(1, min(int(os.environ.get("PG_CONNECT_TIMEOUT", 10)), 60))
env_drop = os.environ.get("ODS_DROP_SCHEMA_FIRST") or os.environ.get("DROP_SCHEMA_FIRST")
drop_schema_first = (
args.drop_schema_first
if args.drop_schema_first is not None
else str(env_drop or "true").lower() in ("1", "true", "yes")
)
include_files_env = args.include_files or os.environ.get("ODS_INCLUDE_FILES") or os.environ.get("INCLUDE_FILES")
include_files = set()
if include_files_env:
include_files = {p.strip().lower() for p in include_files_env.split(",") if p.strip()}
base_dir = Path(args.json_dir or data_dir or DEFAULT_JSON_DIR)
if not base_dir.exists():
print(f"JSON 目录不存在: {base_dir}")
sys.exit(1)
conn = psycopg2.connect(dsn, connect_timeout=timeout)
conn.autocommit = False
cur = conn.cursor()
if drop_schema_first:
print(f"Dropping schema {schema} ...")
cur.execute(sql.SQL("DROP SCHEMA IF EXISTS {} CASCADE;").format(sql.Identifier(schema)))
cur.execute(sql.SQL("CREATE SCHEMA {};").format(sql.Identifier(schema)))
else:
cur.execute(
sql.SQL("SELECT schema_name FROM information_schema.schemata WHERE schema_name=%s"),
(schema,),
)
if not cur.fetchone():
cur.execute(sql.SQL("CREATE SCHEMA {};").format(sql.Identifier(schema)))
json_files = sorted(base_dir.glob("*.json"))
for path in json_files:
stem_lower = path.stem.lower()
if include_files and stem_lower not in include_files:
continue
print(f"Processing {path.name} ...")
payload = json.loads(path.read_text(encoding="utf-8"))
list_path = SPECIAL_LIST_PATHS.get(stem_lower)
records = load_records(payload, list_path=list_path)
columns_raw = collect_columns(records)
columns = [(sanitize_identifier(c), c) for c in columns_raw]
table_name = sanitize_identifier(path.stem)
create_table(cur, schema, table_name, columns)
if records:
insert_records(cur, schema, table_name, columns, records, path.name)
print(f" -> rows: {len(records)}, columns: {len(columns)}")
conn.commit()
cur.close()
conn.close()
print("Rebuild done.")
if __name__ == "__main__":
rebuild()

View File

@@ -0,0 +1,195 @@
# -*- coding: utf-8 -*-
"""
灵活的测试执行脚本,可像搭积木一样组合不同参数或预置命令(模式/数据库/归档路径等),
直接运行本文件即可触发 pytest。
示例:
python scripts/run_tests.py --suite online --flow FULL --keyword ORDERS
python scripts/run_tests.py --preset fetch_only
python scripts/run_tests.py --suite online --json-source tmp/archives
"""
from __future__ import annotations
import argparse
import importlib.util
import os
import shlex
import sys
from typing import Dict, List
import pytest
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
# 确保项目根目录在 sys.path便于 tests 内部 import config / tasks 等模块
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
SUITE_MAP: Dict[str, str] = {
"online": "tests/unit/test_etl_tasks_online.py",
"integration": "tests/integration/test_database.py",
}
PRESETS: Dict[str, Dict] = {}
def _load_presets():
preset_path = os.path.join(os.path.dirname(__file__), "test_presets.py")
if not os.path.exists(preset_path):
return
spec = importlib.util.spec_from_file_location("test_presets", preset_path)
if not spec or not spec.loader:
return
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # type: ignore[attr-defined]
presets = getattr(module, "PRESETS", {})
if isinstance(presets, dict):
PRESETS.update(presets)
_load_presets()
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="ETL 测试执行器(支持参数化调配)")
parser.add_argument(
"--suite",
choices=sorted(SUITE_MAP.keys()),
nargs="+",
help="预置测试套件,可多选(默认全部 online/offline",
)
parser.add_argument(
"--tests",
nargs="+",
help="自定义测试路径(可与 --suite 混用),例如 tests/unit/test_config.py",
)
parser.add_argument(
"--flow",
choices=["FETCH_ONLY", "INGEST_ONLY", "FULL"],
help="覆盖 PIPELINE_FLOW在线抓取/本地清洗/全流程)",
)
parser.add_argument("--json-source", help="设置 JSON_SOURCE_DIR本地清洗入库使用的 JSON 目录)")
parser.add_argument("--json-fetch-root", help="设置 JSON_FETCH_ROOT在线抓取输出根目录")
parser.add_argument(
"--keyword",
"-k",
help="pytest -k 关键字过滤(例如 ORDERS只运行包含该字符串的用例",
)
parser.add_argument(
"--pytest-args",
help="附加 pytest 参数,格式与命令行一致(例如 \"-vv --maxfail=1\"",
)
parser.add_argument(
"--env",
action="append",
metavar="KEY=VALUE",
help="自定义环境变量,可重复传入,例如 --env STORE_ID=123",
)
parser.add_argument("--preset", choices=sorted(PRESETS.keys()) if PRESETS else None, nargs="+",
help="从 scripts/test_presets.py 中选择一个或多个组合命令")
parser.add_argument("--list-presets", action="store_true", help="列出可用预置命令后退出")
parser.add_argument("--dry-run", action="store_true", help="仅打印将要执行的命令与环境,不真正运行 pytest")
return parser.parse_args()
def apply_presets_to_args(args: argparse.Namespace):
if not args.preset:
return
for name in args.preset:
preset = PRESETS.get(name, {})
if not preset:
continue
for key, value in preset.items():
if key in ("suite", "tests"):
if not value:
continue
existing = getattr(args, key)
if existing is None:
setattr(args, key, list(value))
else:
existing.extend(value)
elif key == "env":
args.env = (args.env or []) + list(value)
elif key == "pytest_args":
args.pytest_args = " ".join(filter(None, [value, args.pytest_args or ""]))
elif key == "keyword":
if args.keyword is None:
args.keyword = value
else:
if getattr(args, key, None) is None:
setattr(args, key, value)
def apply_env(args: argparse.Namespace) -> Dict[str, str]:
env_updates = {}
if args.flow:
env_updates["PIPELINE_FLOW"] = args.flow
if args.json_source:
env_updates["JSON_SOURCE_DIR"] = args.json_source
if args.json_fetch_root:
env_updates["JSON_FETCH_ROOT"] = args.json_fetch_root
if args.env:
for item in args.env:
if "=" not in item:
raise SystemExit(f"--env 参数格式错误: {item!r},应为 KEY=VALUE")
key, value = item.split("=", 1)
env_updates[key.strip()] = value.strip()
for key, value in env_updates.items():
os.environ[key] = value
return env_updates
def build_pytest_args(args: argparse.Namespace) -> List[str]:
targets: List[str] = []
if args.suite:
for suite in args.suite:
targets.append(SUITE_MAP[suite])
if args.tests:
targets.extend(args.tests)
if not targets:
targets = list(SUITE_MAP.values())
pytest_args: List[str] = targets
if args.keyword:
pytest_args += ["-k", args.keyword]
if args.pytest_args:
pytest_args += shlex.split(args.pytest_args)
return pytest_args
def main() -> int:
os.chdir(PROJECT_ROOT)
args = parse_args()
if args.list_presets:
print("可用预置命令:")
if not PRESETS:
print("(暂无,可编辑 scripts/test_presets.py 添加)")
else:
for name in sorted(PRESETS):
print(f"- {name}")
return 0
apply_presets_to_args(args)
env_updates = apply_env(args)
pytest_args = build_pytest_args(args)
print("=== 环境变量覆盖 ===")
if env_updates:
for k, v in env_updates.items():
print(f"{k}={v}")
else:
print("(无覆盖,沿用系统默认)")
print("\n=== Pytest 参数 ===")
print(" ".join(pytest_args))
print()
if args.dry_run:
print("Dry-run 模式,未真正执行 pytest")
return 0
exit_code = pytest.main(pytest_args)
return int(exit_code)
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,64 @@
# -*- coding: utf-8 -*-
"""Quick utility for validating PostgreSQL connectivity (ASCII-only output)."""
from __future__ import annotations
import argparse
import os
import sys
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
from database.connection import DatabaseConnection
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="PostgreSQL connectivity smoke test")
parser.add_argument("--dsn", help="Override TEST_DB_DSN / env value")
parser.add_argument(
"--query",
default="SELECT 1 AS ok",
help="Custom SQL to run after connection (default: SELECT 1 AS ok)",
)
parser.add_argument(
"--timeout",
type=int,
default=10,
help="connect_timeout seconds passed to psycopg2 (capped at 20, default: 10)",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
dsn = args.dsn or os.environ.get("TEST_DB_DSN")
if not dsn:
print("Missing DSN. Use --dsn or TEST_DB_DSN.", file=sys.stderr)
return 2
print(f"Trying connection: {dsn}")
try:
timeout = max(1, min(args.timeout, 20))
conn = DatabaseConnection(dsn, connect_timeout=timeout)
except Exception as exc: # pragma: no cover - diagnostic output
print("Connection failed:", exc, file=sys.stderr)
return 1
try:
result = conn.query(args.query)
print("Connection OK, query result:")
for row in result:
print(row)
conn.close()
return 0
except Exception as exc: # pragma: no cover - diagnostic output
print("Connection succeeded but query failed:", exc, file=sys.stderr)
try:
conn.close()
finally:
return 3
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,122 @@
# -*- coding: utf-8 -*-
"""测试命令仓库:集中维护 run_tests.py 的常用组合,支持一键执行。"""
from __future__ import annotations
import argparse
import os
import subprocess
import sys
from typing import List
RUN_TESTS_SCRIPT = os.path.join(os.path.dirname(__file__), "run_tests.py")
# 默认自动运行的预置(可根据需要修改顺序/条目)
AUTO_RUN_PRESETS = ["fetch_only"]
PRESETS = {
"fetch_only": {
"suite": ["online"],
"flow": "FETCH_ONLY",
"json_fetch_root": "tmp/json_fetch",
"keyword": "ORDERS",
"pytest_args": "-vv",
"preset_meta": "仅在线抓取阶段,输出到本地目录",
},
"ingest_local": {
"suite": ["online"],
"flow": "INGEST_ONLY",
"json_source": "tests/source-data-doc",
"keyword": "ORDERS",
"preset_meta": "从指定 JSON 目录做本地清洗入库",
},
"full_pipeline": {
"suite": ["online"],
"flow": "FULL",
"json_fetch_root": "tmp/json_fetch",
"keyword": "ORDERS",
"preset_meta": "先抓取再清洗入库的全流程",
},
}
def print_parameter_help() -> None:
print("=== 参数键说明 ===")
print("suite : 预置套件列表,如 ['online','integration']")
print("tests : 自定义 pytest 路径列表")
print("flow : PIPELINE_FLOWFETCH_ONLY / INGEST_ONLY / FULL")
print("json_source : JSON_SOURCE_DIR本地清洗入库使用的 JSON 目录")
print("json_fetch_root : JSON_FETCH_ROOT在线抓取输出根目录")
print("keyword : pytest -k 过滤关键字")
print("pytest_args : 额外 pytest 参数(字符串)")
print("env : 附加环境变量,例如 ['KEY=VALUE']")
print("preset_meta : 仅用于注释说明")
print()
def print_presets() -> None:
if not PRESETS:
print("当前未定义任何预置,请在 PRESETS 中添加。")
return
for idx, (name, payload) in enumerate(PRESETS.items(), start=1):
comment = payload.get("preset_meta", "")
print(f"{idx}. {name}")
if comment:
print(f" 说明: {comment}")
for key, value in payload.items():
if key == "preset_meta":
continue
print(f" {key}: {value}")
print()
def resolve_targets(requested: List[str] | None) -> List[str]:
if not PRESETS:
raise SystemExit("预置为空,请先在 PRESETS 中定义测试组合。")
def valid(names: List[str]) -> List[str]:
return [name for name in names if name in PRESETS]
if requested:
candidates = valid(requested)
missing = [name for name in requested if name not in PRESETS]
if missing:
print(f"警告:忽略未定义的预置 {missing}")
if candidates:
return candidates
auto = valid(AUTO_RUN_PRESETS)
if auto:
return auto
return list(PRESETS.keys())
def run_presets(preset_names: List[str], dry_run: bool) -> None:
for name in preset_names:
cmd = [sys.executable, RUN_TESTS_SCRIPT, "--preset", name]
printable = " ".join(cmd)
if dry_run:
print(f"[Dry-Run] {printable}")
else:
print(f"\n>>> 执行: {printable}")
subprocess.run(cmd, check=False)
def main() -> None:
parser = argparse.ArgumentParser(description="测试预置仓库(集中配置即可批量触发 run_tests")
parser.add_argument("--preset", choices=sorted(PRESETS.keys()), nargs="+", help="指定要运行的预置命令")
parser.add_argument("--list", action="store_true", help="仅列出参数说明与所有预置")
parser.add_argument("--dry-run", action="store_true", help="仅打印命令,不执行 pytest")
args = parser.parse_args()
if args.list:
print_parameter_help()
print_presets()
return
targets = resolve_targets(args.preset)
run_presets(targets, dry_run=args.dry_run)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,81 @@
# -*- coding: utf-8 -*-
"""助教作废任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.assistant_abolish import AssistantAbolishLoader
from models.parsers import TypeParser
class AssistantAbolishTask(BaseTask):
"""同步助教作废记录"""
def get_task_code(self) -> str:
return "ASSISTANT_ABOLISH"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/AssistantPerformance/GetAbolitionAssistant",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="abolitionAssistants",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_record(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = AssistantAbolishLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_records(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_record(self, raw: dict, store_id: int) -> dict | None:
abolish_id = TypeParser.parse_int(raw.get("id"))
if not abolish_id:
self.logger.warning("跳过缺少作废ID的记录: %s", raw)
return None
return {
"store_id": store_id,
"abolish_id": abolish_id,
"table_id": TypeParser.parse_int(raw.get("tableId")),
"table_name": raw.get("tableName"),
"table_area_id": TypeParser.parse_int(raw.get("tableAreaId")),
"table_area": raw.get("tableArea"),
"assistant_no": raw.get("assistantOn"),
"assistant_name": raw.get("assistantName"),
"charge_minutes": TypeParser.parse_int(raw.get("pdChargeMinutes")),
"abolish_amount": TypeParser.parse_decimal(raw.get("assistantAbolishAmount")),
"create_time": TypeParser.parse_timestamp(
raw.get("createTime") or raw.get("create_time"), self.tz
),
"trash_reason": raw.get("trashReason"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,102 @@
# -*- coding: utf-8 -*-
"""助教账号任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.dimensions.assistant import AssistantLoader
from models.parsers import TypeParser
class AssistantsTask(BaseTask):
"""同步助教账号资料"""
def get_task_code(self) -> str:
return "ASSISTANTS"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params({"siteId": context.store_id})
records, _ = self.api.get_paginated(
endpoint="/PersonnelManagement/SearchAssistantInfo",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="assistantInfos",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_assistant(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = AssistantLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_assistants(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_assistant(self, raw: dict, store_id: int) -> dict | None:
assistant_id = TypeParser.parse_int(raw.get("id"))
if not assistant_id:
self.logger.warning("跳过缺少助教ID的数据: %s", raw)
return None
return {
"store_id": store_id,
"assistant_id": assistant_id,
"assistant_no": raw.get("assistant_no") or raw.get("assistantNo"),
"nickname": raw.get("nickname"),
"real_name": raw.get("real_name") or raw.get("realName"),
"gender": raw.get("gender"),
"mobile": raw.get("mobile"),
"level": raw.get("level"),
"team_id": TypeParser.parse_int(raw.get("team_id") or raw.get("teamId")),
"team_name": raw.get("team_name"),
"assistant_status": raw.get("assistant_status"),
"work_status": raw.get("work_status"),
"entry_time": TypeParser.parse_timestamp(
raw.get("entry_time") or raw.get("entryTime"), self.tz
),
"resign_time": TypeParser.parse_timestamp(
raw.get("resign_time") or raw.get("resignTime"), self.tz
),
"start_time": TypeParser.parse_timestamp(
raw.get("start_time") or raw.get("startTime"), self.tz
),
"end_time": TypeParser.parse_timestamp(
raw.get("end_time") or raw.get("endTime"), self.tz
),
"create_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"update_time": TypeParser.parse_timestamp(
raw.get("update_time") or raw.get("updateTime"), self.tz
),
"system_role_id": raw.get("system_role_id"),
"online_status": raw.get("online_status"),
"allow_cx": raw.get("allow_cx"),
"charge_way": raw.get("charge_way"),
"pd_unit_price": TypeParser.parse_decimal(raw.get("pd_unit_price")),
"cx_unit_price": TypeParser.parse_decimal(raw.get("cx_unit_price")),
"is_guaranteed": raw.get("is_guaranteed"),
"is_team_leader": raw.get("is_team_leader"),
"serial_number": raw.get("serial_number"),
"show_sort": raw.get("show_sort"),
"is_delete": raw.get("is_delete"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,79 @@
# -*- coding: utf-8 -*-
"""DWD任务基类"""
import json
from typing import Any, Dict, Iterator, List, Optional, Tuple
from datetime import datetime
from .base_task import BaseTask
from models.parsers import TypeParser
class BaseDwdTask(BaseTask):
"""
DWD 层任务基类
负责从 ODS 表读取数据,供子类清洗和写入事实/维度表
"""
def _get_ods_cursor(self, task_code: str) -> datetime:
"""
获取上次处理的 ODS 数据的时间点 (fetched_at)
这里简化处理,实际应该从 etl_cursor 表读取
目前先依赖 BaseTask 的时间窗口逻辑,或者子类自己管理
"""
# TODO: 对接真正的 CursorManager
# 暂时返回一个较早的时间,或者由子类通过 _get_time_window 获取
return None
def iter_ods_rows(
self,
table_name: str,
columns: List[str],
start_time: datetime,
end_time: datetime,
time_col: str = "fetched_at",
batch_size: int = 1000
) -> Iterator[List[Dict[str, Any]]]:
"""
分批迭代读取 ODS 表数据
Args:
table_name: ODS 表名
columns: 需要查询的字段列表 (必须包含 payload)
start_time: 开始时间 (包含)
end_time: 结束时间 (包含)
time_col: 时间过滤字段,默认 fetched_at
batch_size: 批次大小
"""
offset = 0
cols_str = ", ".join(columns)
while True:
sql = f"""
SELECT {cols_str}
FROM {table_name}
WHERE {time_col} >= %s AND {time_col} <= %s
ORDER BY {time_col} ASC
LIMIT %s OFFSET %s
"""
rows = self.db.query(sql, (start_time, end_time, batch_size, offset))
if not rows:
break
yield rows
if len(rows) < batch_size:
break
offset += batch_size
def parse_payload(self, row: Dict[str, Any]) -> Dict[str, Any]:
"""
解析 ODS 行中的 payload JSON
"""
payload = row.get("payload")
if isinstance(payload, str):
return json.loads(payload)
elif isinstance(payload, dict):
return payload
return {}

View File

@@ -1,62 +1,141 @@
# -*- coding: utf-8 -*-
"""ETL任务基类"""
"""ETL任务基类(引入 Extract/Transform/Load 模板方法)"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
@dataclass(frozen=True)
class TaskContext:
"""统一透传给 Extract/Transform/Load 的运行期信息。"""
store_id: int
window_start: datetime
window_end: datetime
window_minutes: int
cursor: dict | None = None
class BaseTask:
"""ETL任务基类"""
"""提供 E/T/L 模板的任务基类"""
def __init__(self, config, db_connection, api_client, logger):
self.config = config
self.db = db_connection
self.api = api_client
self.logger = logger
self.tz = ZoneInfo(config.get("app.timezone", "Asia/Taipei"))
# ------------------------------------------------------------------ 基本信息
def get_task_code(self) -> str:
"""获取任务代码"""
raise NotImplementedError("子类需实现 get_task_code 方法")
def execute(self) -> dict:
"""执行任务"""
raise NotImplementedError("子类需实现 execute 方法")
# ------------------------------------------------------------------ E/T/L 钩子
def extract(self, context: TaskContext):
"""提取数据"""
raise NotImplementedError("子类需实现 extract 方法")
def transform(self, extracted, context: TaskContext):
"""转换数据"""
return extracted
def load(self, transformed, context: TaskContext) -> dict:
"""加载数据并返回统计信息"""
raise NotImplementedError("子类需实现 load 方法")
# ------------------------------------------------------------------ 主流程
def execute(self, cursor_data: dict | None = None) -> dict:
"""统一 orchestrate Extract → Transform → Load"""
context = self._build_context(cursor_data)
task_code = self.get_task_code()
self.logger.info(
"%s: 开始执行,窗口[%s ~ %s]",
task_code,
context.window_start,
context.window_end,
)
try:
extracted = self.extract(context)
transformed = self.transform(extracted, context)
counts = self.load(transformed, context) or {}
self.db.commit()
except Exception:
self.db.rollback()
self.logger.error("%s: 执行失败", task_code, exc_info=True)
raise
result = self._build_result("SUCCESS", counts)
result["window"] = {
"start": context.window_start,
"end": context.window_end,
"minutes": context.window_minutes,
}
self.logger.info("%s: 完成,统计=%s", task_code, result["counts"])
return result
# ------------------------------------------------------------------ 辅助方法
def _build_context(self, cursor_data: dict | None) -> TaskContext:
window_start, window_end, window_minutes = self._get_time_window(cursor_data)
return TaskContext(
store_id=self.config.get("app.store_id"),
window_start=window_start,
window_end=window_end,
window_minutes=window_minutes,
cursor=cursor_data,
)
def _get_time_window(self, cursor_data: dict = None) -> tuple:
"""计算时间窗口"""
now = datetime.now(self.tz)
# 判断是否在闲时窗口
idle_start = self.config.get("run.idle_window.start", "04:00")
idle_end = self.config.get("run.idle_window.end", "16:00")
is_idle = self._is_in_idle_window(now, idle_start, idle_end)
# 获取窗口大小
if is_idle:
window_minutes = self.config.get("run.window_minutes.default_idle", 180)
else:
window_minutes = self.config.get("run.window_minutes.default_busy", 30)
# 计算窗口
overlap_seconds = self.config.get("run.overlap_seconds", 120)
if cursor_data and cursor_data.get("last_end"):
window_start = cursor_data["last_end"] - timedelta(seconds=overlap_seconds)
else:
window_start = now - timedelta(minutes=window_minutes)
window_end = now
return window_start, window_end, window_minutes
def _is_in_idle_window(self, dt: datetime, start_time: str, end_time: str) -> bool:
"""判断是否在闲时窗口"""
current_time = dt.strftime("%H:%M")
return start_time <= current_time <= end_time
def _merge_common_params(self, base: dict) -> dict:
"""
合并全局/任务级参数池便于在配置中统一覆<E4B880>?/追加过滤条件。
支持:
- api.params 下的通用键<E794A8>?
- api.params.<task_code_lower> 下的任务级键<E7BAA7>?
"""
merged: dict = {}
common = self.config.get("api.params", {}) or {}
if isinstance(common, dict):
merged.update(common)
task_key = f"api.params.{self.get_task_code().lower()}"
scoped = self.config.get(task_key, {}) or {}
if isinstance(scoped, dict):
merged.update(scoped)
merged.update(base)
return merged
def _build_result(self, status: str, counts: dict) -> dict:
"""构建结果字典"""
return {
"status": status,
"counts": counts
}
return {"status": status, "counts": counts}

View File

@@ -0,0 +1,93 @@
# -*- coding: utf-8 -*-
"""平台券核销任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.coupon_usage import CouponUsageLoader
from models.parsers import TypeParser
class CouponUsageTask(BaseTask):
"""同步平台券验券/核销记录"""
def get_task_code(self) -> str:
return "COUPON_USAGE"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/Promotion/GetOfflineCouponConsumePageList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_usage(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = CouponUsageLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_coupon_usage(
transformed["records"]
)
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_usage(self, raw: dict, store_id: int) -> dict | None:
usage_id = TypeParser.parse_int(raw.get("id"))
if not usage_id:
self.logger.warning("跳过缺少券核销ID的记录: %s", raw)
return None
return {
"store_id": store_id,
"usage_id": usage_id,
"coupon_code": raw.get("coupon_code"),
"coupon_channel": raw.get("coupon_channel"),
"coupon_name": raw.get("coupon_name"),
"sale_price": TypeParser.parse_decimal(raw.get("sale_price")),
"coupon_money": TypeParser.parse_decimal(raw.get("coupon_money")),
"coupon_free_time": TypeParser.parse_int(raw.get("coupon_free_time")),
"use_status": raw.get("use_status"),
"create_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"consume_time": TypeParser.parse_timestamp(
raw.get("consume_time") or raw.get("consumeTime"), self.tz
),
"operator_id": TypeParser.parse_int(raw.get("operator_id")),
"operator_name": raw.get("operator_name"),
"table_id": TypeParser.parse_int(raw.get("table_id")),
"site_order_id": TypeParser.parse_int(raw.get("site_order_id")),
"group_package_id": TypeParser.parse_int(raw.get("group_package_id")),
"coupon_remark": raw.get("coupon_remark"),
"deal_id": raw.get("deal_id"),
"certificate_id": raw.get("certificate_id"),
"verify_id": raw.get("verify_id"),
"is_delete": raw.get("is_delete"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,90 @@
# -*- coding: utf-8 -*-
"""库存变更任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.inventory_change import InventoryChangeLoader
from models.parsers import TypeParser
class InventoryChangeTask(BaseTask):
"""同步库存变化记录"""
def get_task_code(self) -> str:
return "INVENTORY_CHANGE"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/GoodsStockManage/QueryGoodsOutboundReceipt",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="queryDeliveryRecordsList",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_change(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = InventoryChangeLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_changes(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_change(self, raw: dict, store_id: int) -> dict | None:
change_id = TypeParser.parse_int(
raw.get("siteGoodsStockId") or raw.get("site_goods_stock_id")
)
if not change_id:
self.logger.warning("跳过缺少库存变动ID的记录: %s", raw)
return None
return {
"store_id": store_id,
"change_id": change_id,
"site_goods_id": TypeParser.parse_int(
raw.get("siteGoodsId") or raw.get("site_goods_id")
),
"stock_type": raw.get("stockType") or raw.get("stock_type"),
"goods_name": raw.get("goodsName"),
"change_time": TypeParser.parse_timestamp(
raw.get("createTime") or raw.get("create_time"), self.tz
),
"start_qty": TypeParser.parse_int(raw.get("startNum")),
"end_qty": TypeParser.parse_int(raw.get("endNum")),
"change_qty": TypeParser.parse_int(raw.get("changeNum")),
"unit": raw.get("unit"),
"price": TypeParser.parse_decimal(raw.get("price")),
"operator_name": raw.get("operatorName"),
"remark": raw.get("remark"),
"goods_category_id": TypeParser.parse_int(raw.get("goodsCategoryId")),
"goods_second_category_id": TypeParser.parse_int(
raw.get("goodsSecondCategoryId")
),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,115 @@
# -*- coding: utf-8 -*-
"""助教流水任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.assistant_ledger import AssistantLedgerLoader
from models.parsers import TypeParser
class LedgerTask(BaseTask):
"""同步助教服务台账"""
def get_task_code(self) -> str:
return "LEDGER"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/AssistantPerformance/GetOrderAssistantDetails",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="orderAssistantDetails",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_ledger(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = AssistantLedgerLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_ledgers(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_ledger(self, raw: dict, store_id: int) -> dict | None:
ledger_id = TypeParser.parse_int(raw.get("id"))
if not ledger_id:
self.logger.warning("跳过缺少助教流水ID的记录: %s", raw)
return None
return {
"store_id": store_id,
"ledger_id": ledger_id,
"assistant_no": raw.get("assistantNo"),
"assistant_name": raw.get("assistantName"),
"nickname": raw.get("nickname"),
"level_name": raw.get("levelName"),
"table_name": raw.get("tableName"),
"ledger_unit_price": TypeParser.parse_decimal(raw.get("ledger_unit_price")),
"ledger_count": TypeParser.parse_int(raw.get("ledger_count")),
"ledger_amount": TypeParser.parse_decimal(raw.get("ledger_amount")),
"projected_income": TypeParser.parse_decimal(raw.get("projected_income")),
"service_money": TypeParser.parse_decimal(raw.get("service_money")),
"member_discount_amount": TypeParser.parse_decimal(
raw.get("member_discount_amount")
),
"manual_discount_amount": TypeParser.parse_decimal(
raw.get("manual_discount_amount")
),
"coupon_deduct_money": TypeParser.parse_decimal(
raw.get("coupon_deduct_money")
),
"order_trade_no": TypeParser.parse_int(raw.get("order_trade_no")),
"order_settle_id": TypeParser.parse_int(raw.get("order_settle_id")),
"operator_id": TypeParser.parse_int(raw.get("operator_id")),
"operator_name": raw.get("operator_name"),
"assistant_team_id": TypeParser.parse_int(raw.get("assistant_team_id")),
"assistant_level": raw.get("assistant_level"),
"site_table_id": TypeParser.parse_int(raw.get("site_table_id")),
"order_assistant_id": TypeParser.parse_int(raw.get("order_assistant_id")),
"site_assistant_id": TypeParser.parse_int(raw.get("site_assistant_id")),
"user_id": TypeParser.parse_int(raw.get("user_id")),
"ledger_start_time": TypeParser.parse_timestamp(
raw.get("ledger_start_time"), self.tz
),
"ledger_end_time": TypeParser.parse_timestamp(
raw.get("ledger_end_time"), self.tz
),
"start_use_time": TypeParser.parse_timestamp(raw.get("start_use_time"), self.tz),
"last_use_time": TypeParser.parse_timestamp(raw.get("last_use_time"), self.tz),
"income_seconds": TypeParser.parse_int(raw.get("income_seconds")),
"real_use_seconds": TypeParser.parse_int(raw.get("real_use_seconds")),
"is_trash": raw.get("is_trash"),
"trash_reason": raw.get("trash_reason"),
"is_confirm": raw.get("is_confirm"),
"ledger_status": raw.get("ledger_status"),
"create_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,89 @@
# -*- coding: utf-8 -*-
from .base_dwd_task import BaseDwdTask
from loaders.dimensions.member import MemberLoader
from models.parsers import TypeParser
import json
class MembersDwdTask(BaseDwdTask):
"""
DWD Task: Process Member Records from ODS to Dimension Table
Source: billiards_ods.ods_member_profile
Target: billiards.dim_member
"""
def get_task_code(self) -> str:
return "MEMBERS_DWD"
def execute(self) -> dict:
self.logger.info(f"Starting {self.get_task_code()} task")
window_start, window_end, _ = self._get_time_window()
self.logger.info(f"Processing window: {window_start} to {window_end}")
loader = MemberLoader(self.db)
store_id = self.config.get("app.store_id")
total_inserted = 0
total_updated = 0
total_errors = 0
# Iterate ODS Data
batches = self.iter_ods_rows(
table_name="billiards_ods.ods_member_profile",
columns=["site_id", "member_id", "payload", "fetched_at"],
start_time=window_start,
end_time=window_end
)
for batch in batches:
if not batch:
continue
parsed_rows = []
for row in batch:
payload = self.parse_payload(row)
if not payload:
continue
parsed = self._parse_member(payload, store_id)
if parsed:
parsed_rows.append(parsed)
if parsed_rows:
inserted, updated, skipped = loader.upsert_members(parsed_rows, store_id)
total_inserted += inserted
total_updated += updated
self.db.commit()
self.logger.info(f"Task {self.get_task_code()} completed. Inserted: {total_inserted}, Updated: {total_updated}")
return {
"status": "success",
"inserted": total_inserted,
"updated": total_updated,
"window_start": window_start.isoformat(),
"window_end": window_end.isoformat()
}
def _parse_member(self, raw: dict, store_id: int) -> dict:
"""Parse ODS payload into Dim structure"""
try:
# Handle both API structure (camelCase) and manual structure
member_id = raw.get("id") or raw.get("memberId")
if not member_id:
return None
return {
"store_id": store_id,
"member_id": member_id,
"member_name": raw.get("name") or raw.get("memberName"),
"phone": raw.get("phone") or raw.get("mobile"),
"balance": raw.get("balance", 0),
"status": str(raw.get("status", "NORMAL")),
"register_time": raw.get("createTime") or raw.get("registerTime"),
"raw_data": json.dumps(raw, ensure_ascii=False)
}
except Exception as e:
self.logger.warning(f"Error parsing member: {e}")
return None

View File

@@ -1,73 +1,72 @@
# -*- coding: utf-8 -*-
"""会员ETL任务"""
import json
from .base_task import BaseTask
from .base_task import BaseTask, TaskContext
from loaders.dimensions.member import MemberLoader
from models.parsers import TypeParser
class MembersTask(BaseTask):
"""会员ETL任务"""
def get_task_code(self) -> str:
return "MEMBERS"
def execute(self) -> dict:
"""执行会员ETL"""
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
params = {
"storeId": self.config.get("app.store_id"),
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params({"siteId": context.store_id})
records, _ = self.api.get_paginated(
endpoint="/MemberProfile/GetTenantMemberList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="tenantMemberInfos",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
parsed_row = self._parse_member(raw, context.store_id)
if parsed_row:
parsed.append(parsed_row)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
try:
records, pages_meta = self.api.get_paginated(
endpoint="/MemberProfile/GetTenantMemberList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",)
)
parsed_records = []
for rec in records:
parsed = self._parse_member(rec)
if parsed:
parsed_records.append(parsed)
loader = MemberLoader(self.db)
store_id = self.config.get("app.store_id")
inserted, updated, skipped = loader.upsert_members(parsed_records, store_id)
self.db.commit()
counts = {
"fetched": len(records),
"inserted": inserted,
"updated": updated,
"skipped": skipped,
"errors": 0
}
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
return self._build_result("SUCCESS", counts)
except Exception as e:
self.db.rollback()
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
raise
def _parse_member(self, raw: dict) -> dict:
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = MemberLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_members(
transformed["records"], context.store_id
)
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_member(self, raw: dict, store_id: int) -> dict | None:
"""解析会员记录"""
try:
member_id = TypeParser.parse_int(raw.get("memberId"))
if not member_id:
return None
return {
"store_id": self.config.get("app.store_id"),
"member_id": TypeParser.parse_int(raw.get("memberId")),
"store_id": store_id,
"member_id": member_id,
"member_name": raw.get("memberName"),
"phone": raw.get("phone"),
"balance": TypeParser.parse_decimal(raw.get("balance")),
"status": raw.get("status"),
"register_time": TypeParser.parse_timestamp(raw.get("registerTime"), self.tz),
"raw_data": json.dumps(raw, ensure_ascii=False)
"raw_data": json.dumps(raw, ensure_ascii=False),
}
except Exception as e:
self.logger.warning(f"解析会员记录失败: {e}, 原始数据: {raw}")
except Exception as exc:
self.logger.warning("解析会员记录失败: %s, 原始数据: %s", exc, raw)
return None

View File

@@ -0,0 +1,933 @@
# -*- coding: utf-8 -*-
"""ODS ingestion tasks."""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Callable, Dict, Iterable, List, Sequence, Tuple, Type
from loaders.ods import GenericODSLoader
from models.parsers import TypeParser
from .base_task import BaseTask
ColumnTransform = Callable[[Any], Any]
@dataclass(frozen=True)
class ColumnSpec:
"""Mapping between DB column and source JSON field."""
column: str
sources: Tuple[str, ...] = ()
required: bool = False
default: Any = None
transform: ColumnTransform | None = None
@dataclass(frozen=True)
class OdsTaskSpec:
"""Definition of a single ODS ingestion task."""
code: str
class_name: str
table_name: str
endpoint: str
data_path: Tuple[str, ...] = ("data",)
list_key: str | None = None
pk_columns: Tuple[ColumnSpec, ...] = ()
extra_columns: Tuple[ColumnSpec, ...] = ()
include_page_size: bool = False
include_page_no: bool = False
include_source_file: bool = True
include_source_endpoint: bool = True
include_record_index: bool = False
include_site_column: bool = True
include_fetched_at: bool = True
requires_window: bool = True
time_fields: Tuple[str, str] | None = ("startTime", "endTime")
include_site_id: bool = True
description: str = ""
extra_params: Dict[str, Any] = field(default_factory=dict)
conflict_columns_override: Tuple[str, ...] | None = None
class BaseOdsTask(BaseTask):
"""Shared functionality for ODS ingestion tasks."""
SPEC: OdsTaskSpec
def get_task_code(self) -> str:
return self.SPEC.code
def execute(self) -> dict:
spec = self.SPEC
self.logger.info("开始执行 %s (ODS)", spec.code)
store_id = TypeParser.parse_int(self.config.get("app.store_id"))
if not store_id:
raise ValueError("app.store_id 未配置,无法执行 ODS 任务")
page_size = self.config.get("api.page_size", 200)
params = self._build_params(spec, store_id)
columns = self._resolve_columns(spec)
if spec.conflict_columns_override:
conflict_columns = list(spec.conflict_columns_override)
else:
conflict_columns = []
if spec.include_site_column:
conflict_columns.append("site_id")
conflict_columns += [col.column for col in spec.pk_columns]
loader = GenericODSLoader(
self.db,
spec.table_name,
columns,
conflict_columns,
)
counts = {"fetched": 0, "inserted": 0, "updated": 0, "skipped": 0, "errors": 0}
source_file = self._resolve_source_file_hint(spec)
try:
global_index = 0
for page_no, page_records, _, _ in self.api.iter_paginated(
endpoint=spec.endpoint,
params=params,
page_size=page_size,
data_path=spec.data_path,
list_key=spec.list_key,
):
rows: List[dict] = []
for raw in page_records:
row = self._build_row(
spec=spec,
store_id=store_id,
record=raw,
page_no=page_no if spec.include_page_no else None,
page_size_value=len(page_records)
if spec.include_page_size
else None,
source_file=source_file,
record_index=global_index if spec.include_record_index else None,
)
if row is None:
counts["skipped"] += 1
continue
rows.append(row)
global_index += 1
inserted, updated, _ = loader.upsert_rows(rows)
counts["inserted"] += inserted
counts["updated"] += updated
counts["fetched"] += len(page_records)
self.db.commit()
self.logger.info("%s ODS 任务完成: %s", spec.code, counts)
return self._build_result("SUCCESS", counts)
except Exception:
self.db.rollback()
counts["errors"] += 1
self.logger.error("%s ODS 任务失败", spec.code, exc_info=True)
raise
def _build_params(self, spec: OdsTaskSpec, store_id: int) -> dict:
base: dict[str, Any] = {}
if spec.include_site_id:
base["siteId"] = store_id
if spec.requires_window and spec.time_fields:
window_start, window_end, _ = self._get_time_window()
start_key, end_key = spec.time_fields
base[start_key] = TypeParser.format_timestamp(window_start, self.tz)
base[end_key] = TypeParser.format_timestamp(window_end, self.tz)
params = self._merge_common_params(base)
params.update(spec.extra_params)
return params
def _resolve_columns(self, spec: OdsTaskSpec) -> List[str]:
columns: List[str] = []
if spec.include_site_column:
columns.append("site_id")
seen = set(columns)
for col_spec in list(spec.pk_columns) + list(spec.extra_columns):
if col_spec.column not in seen:
columns.append(col_spec.column)
seen.add(col_spec.column)
if spec.include_record_index and "record_index" not in seen:
columns.append("record_index")
seen.add("record_index")
if spec.include_page_no and "page_no" not in seen:
columns.append("page_no")
seen.add("page_no")
if spec.include_page_size and "page_size" not in seen:
columns.append("page_size")
seen.add("page_size")
if spec.include_source_file and "source_file" not in seen:
columns.append("source_file")
seen.add("source_file")
if spec.include_source_endpoint and "source_endpoint" not in seen:
columns.append("source_endpoint")
seen.add("source_endpoint")
if spec.include_fetched_at and "fetched_at" not in seen:
columns.append("fetched_at")
seen.add("fetched_at")
if "payload" not in seen:
columns.append("payload")
return columns
def _build_row(
self,
spec: OdsTaskSpec,
store_id: int,
record: dict,
page_no: int | None,
page_size_value: int | None,
source_file: str | None,
record_index: int | None = None,
) -> dict | None:
row: dict[str, Any] = {}
if spec.include_site_column:
row["site_id"] = store_id
for col_spec in spec.pk_columns + spec.extra_columns:
value = self._extract_value(record, col_spec)
if value is None and col_spec.required:
self.logger.warning(
"%s 缺少必填字段 %s,原始记录: %s",
spec.code,
col_spec.column,
record,
)
return None
row[col_spec.column] = value
if spec.include_page_no:
row["page_no"] = page_no
if spec.include_page_size:
row["page_size"] = page_size_value
if spec.include_record_index:
row["record_index"] = record_index
if spec.include_source_file:
row["source_file"] = source_file
if spec.include_source_endpoint:
row["source_endpoint"] = spec.endpoint
if spec.include_fetched_at:
row["fetched_at"] = datetime.now(self.tz)
row["payload"] = record
return row
def _extract_value(self, record: dict, spec: ColumnSpec):
value = None
for key in spec.sources:
value = self._dig(record, key)
if value is not None:
break
if value is None and spec.default is not None:
value = spec.default
if value is not None and spec.transform:
value = spec.transform(value)
return value
@staticmethod
def _dig(record: Any, path: str | None):
if not path:
return None
current = record
for part in path.split("."):
if isinstance(current, dict):
current = current.get(part)
else:
return None
return current
def _resolve_source_file_hint(self, spec: OdsTaskSpec) -> str | None:
resolver = getattr(self.api, "get_source_hint", None)
if callable(resolver):
return resolver(spec.endpoint)
return None
def _int_col(name: str, *sources: str, required: bool = False) -> ColumnSpec:
return ColumnSpec(
column=name,
sources=sources,
required=required,
transform=TypeParser.parse_int,
)
ODS_TASK_SPECS: Tuple[OdsTaskSpec, ...] = (
OdsTaskSpec(
code="ODS_ASSISTANT_ACCOUNTS",
class_name="OdsAssistantAccountsTask",
table_name="billiards_ods.assistant_accounts_master",
endpoint="/PersonnelManagement/SearchAssistantInfo",
data_path=("data",),
list_key="assistantInfos",
pk_columns=(_int_col("id", "id", required=True),),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
description="助教账号档案 ODSSearchAssistantInfo -> assistantInfos 原始 JSON",
),
OdsTaskSpec(
code="ODS_ORDER_SETTLE",
class_name="OdsOrderSettleTask",
table_name="billiards_ods.settlement_records",
endpoint="/Site/GetAllOrderSettleList",
data_path=("data",),
list_key="settleList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="结账记录 ODSGetAllOrderSettleList -> settleList 原始 JSON",
),
OdsTaskSpec(
code="ODS_TABLE_USE",
class_name="OdsTableUseTask",
table_name="billiards_ods.table_fee_transactions",
endpoint="/Site/GetSiteTableOrderDetails",
data_path=("data",),
list_key="siteTableUseDetailsList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="台费计费流水 ODSGetSiteTableOrderDetails -> siteTableUseDetailsList 原始 JSON",
),
OdsTaskSpec(
code="ODS_ASSISTANT_LEDGER",
class_name="OdsAssistantLedgerTask",
table_name="billiards_ods.assistant_service_records",
endpoint="/AssistantPerformance/GetOrderAssistantDetails",
data_path=("data",),
list_key="orderAssistantDetails",
pk_columns=(_int_col("id", "id", required=True),),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
description="助教服务流水 ODSGetOrderAssistantDetails -> orderAssistantDetails 原始 JSON",
),
OdsTaskSpec(
code="ODS_ASSISTANT_ABOLISH",
class_name="OdsAssistantAbolishTask",
table_name="billiards_ods.assistant_cancellation_records",
endpoint="/AssistantPerformance/GetAbolitionAssistant",
data_path=("data",),
list_key="abolitionAssistants",
pk_columns=(_int_col("id", "id", required=True),),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
description="助教废除记录 ODSGetAbolitionAssistant -> abolitionAssistants 原始 JSON",
),
OdsTaskSpec(
code="ODS_GOODS_LEDGER",
class_name="OdsGoodsLedgerTask",
table_name="billiards_ods.store_goods_sales_records",
endpoint="/TenantGoods/GetGoodsSalesList",
data_path=("data",),
list_key="orderGoodsLedgers",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="门店商品销售流水 ODSGetGoodsSalesList -> orderGoodsLedgers 原始 JSON",
),
OdsTaskSpec(
code="ODS_PAYMENT",
class_name="OdsPaymentTask",
table_name="billiards_ods.payment_transactions",
endpoint="/PayLog/GetPayLogListPage",
data_path=("data",),
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="支付流水 ODSGetPayLogListPage 原始 JSON",
),
OdsTaskSpec(
code="ODS_REFUND",
class_name="OdsRefundTask",
table_name="billiards_ods.refund_transactions",
endpoint="/Order/GetRefundPayLogList",
data_path=("data",),
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="退款流水 ODSGetRefundPayLogList 原始 JSON",
),
OdsTaskSpec(
code="ODS_COUPON_VERIFY",
class_name="OdsCouponVerifyTask",
table_name="billiards_ods.platform_coupon_redemption_records",
endpoint="/Promotion/GetOfflineCouponConsumePageList",
data_path=("data",),
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="平台/团购券核销 ODSGetOfflineCouponConsumePageList 原始 JSON",
),
OdsTaskSpec(
code="ODS_MEMBER",
class_name="OdsMemberTask",
table_name="billiards_ods.member_profiles",
endpoint="/MemberProfile/GetTenantMemberList",
data_path=("data",),
list_key="tenantMemberInfos",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="会员档案 ODSGetTenantMemberList -> tenantMemberInfos 原始 JSON",
),
OdsTaskSpec(
code="ODS_MEMBER_CARD",
class_name="OdsMemberCardTask",
table_name="billiards_ods.member_stored_value_cards",
endpoint="/MemberProfile/GetTenantMemberCardList",
data_path=("data",),
list_key="tenantMemberCards",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="会员储值卡 ODSGetTenantMemberCardList -> tenantMemberCards 原始 JSON",
),
OdsTaskSpec(
code="ODS_MEMBER_BALANCE",
class_name="OdsMemberBalanceTask",
table_name="billiards_ods.member_balance_changes",
endpoint="/MemberProfile/GetMemberCardBalanceChange",
data_path=("data",),
list_key="tenantMemberCardLogs",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="会员余额变动 ODSGetMemberCardBalanceChange -> tenantMemberCardLogs 原始 JSON",
),
OdsTaskSpec(
code="ODS_RECHARGE_SETTLE",
class_name="OdsRechargeSettleTask",
table_name="billiards_ods.recharge_settlements",
endpoint="/Site/GetRechargeSettleList",
data_path=("data",),
list_key="settleList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="会员充值结算 ODSGetRechargeSettleList -> settleList 原始 JSON",
),
OdsTaskSpec(
code="ODS_PACKAGE",
class_name="OdsPackageTask",
table_name="billiards_ods.group_buy_packages",
endpoint="/PackageCoupon/QueryPackageCouponList",
data_path=("data",),
list_key="packageCouponList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="团购套餐定义 ODSQueryPackageCouponList -> packageCouponList 原始 JSON",
),
OdsTaskSpec(
code="ODS_GROUP_BUY_REDEMPTION",
class_name="OdsGroupBuyRedemptionTask",
table_name="billiards_ods.group_buy_redemption_records",
endpoint="/Site/GetSiteTableUseDetails",
data_path=("data",),
list_key="siteTableUseDetailsList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="团购套餐核销 ODSGetSiteTableUseDetails -> siteTableUseDetailsList 原始 JSON",
),
OdsTaskSpec(
code="ODS_INVENTORY_STOCK",
class_name="OdsInventoryStockTask",
table_name="billiards_ods.goods_stock_summary",
endpoint="/TenantGoods/GetGoodsStockReport",
data_path=("data",),
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="库存汇总 ODSGetGoodsStockReport 原始 JSON",
),
OdsTaskSpec(
code="ODS_INVENTORY_CHANGE",
class_name="OdsInventoryChangeTask",
table_name="billiards_ods.goods_stock_movements",
endpoint="/GoodsStockManage/QueryGoodsOutboundReceipt",
data_path=("data",),
list_key="queryDeliveryRecordsList",
pk_columns=(_int_col("sitegoodsstockid", "siteGoodsStockId", required=True),),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
description="库存变化记录 ODSQueryGoodsOutboundReceipt -> queryDeliveryRecordsList 原始 JSON",
),
OdsTaskSpec(
code="ODS_TABLES",
class_name="OdsTablesTask",
table_name="billiards_ods.site_tables_master",
endpoint="/Table/GetSiteTables",
data_path=("data",),
list_key="siteTables",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="台桌维表 ODSGetSiteTables -> siteTables 原始 JSON",
),
OdsTaskSpec(
code="ODS_GOODS_CATEGORY",
class_name="OdsGoodsCategoryTask",
table_name="billiards_ods.stock_goods_category_tree",
endpoint="/TenantGoodsCategory/QueryPrimarySecondaryCategory",
data_path=("data",),
list_key="goodsCategoryList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="库存商品分类树 ODSQueryPrimarySecondaryCategory -> goodsCategoryList 原始 JSON",
),
OdsTaskSpec(
code="ODS_STORE_GOODS",
class_name="OdsStoreGoodsTask",
table_name="billiards_ods.store_goods_master",
endpoint="/TenantGoods/GetGoodsInventoryList",
data_path=("data",),
list_key="orderGoodsList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="门店商品档案 ODSGetGoodsInventoryList -> orderGoodsList 原始 JSON",
),
OdsTaskSpec(
code="ODS_TABLE_DISCOUNT",
class_name="OdsTableDiscountTask",
table_name="billiards_ods.table_fee_discount_records",
endpoint="/Site/GetTaiFeeAdjustList",
data_path=("data",),
list_key="taiFeeAdjustInfos",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="台费折扣/调账 ODSGetTaiFeeAdjustList -> taiFeeAdjustInfos 原始 JSON",
),
OdsTaskSpec(
code="ODS_TENANT_GOODS",
class_name="OdsTenantGoodsTask",
table_name="billiards_ods.tenant_goods_master",
endpoint="/TenantGoods/QueryTenantGoods",
data_path=("data",),
list_key="tenantGoodsList",
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=False,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
description="租户商品档案 ODSQueryTenantGoods -> tenantGoodsList 原始 JSON",
),
OdsTaskSpec(
code="ODS_SETTLEMENT_TICKET",
class_name="OdsSettlementTicketTask",
table_name="billiards_ods.settlement_ticket_details",
endpoint="/Order/GetOrderSettleTicketNew",
data_path=(),
list_key=None,
pk_columns=(),
include_site_column=False,
include_source_endpoint=False,
include_page_no=False,
include_page_size=False,
include_fetched_at=True,
include_record_index=True,
conflict_columns_override=("source_file", "record_index"),
requires_window=False,
include_site_id=False,
description="结账小票详情 ODSGetOrderSettleTicketNew 原始 JSON",
),
)
def _get_spec(code: str) -> OdsTaskSpec:
for spec in ODS_TASK_SPECS:
if spec.code == code:
return spec
raise KeyError(f"Spec not found for code {code}")
_SETTLEMENT_TICKET_SPEC = _get_spec("ODS_SETTLEMENT_TICKET")
class OdsSettlementTicketTask(BaseOdsTask):
"""Special handling: fetch ticket details per payment relate_id/orderSettleId."""
SPEC = _SETTLEMENT_TICKET_SPEC
def extract(self, context) -> dict:
"""Fetch ticket payloads only (used by fetch-only pipeline)."""
existing_ids = self._fetch_existing_ticket_ids()
candidates = self._collect_settlement_ids(
context.store_id or 0, existing_ids, context.window_start, context.window_end
)
candidates = [cid for cid in candidates if cid and cid not in existing_ids]
payloads, skipped = self._fetch_ticket_payloads(candidates)
return {"records": payloads, "skipped": skipped, "fetched": len(candidates)}
def execute(self, cursor_data: dict | None = None) -> dict:
spec = self.SPEC
context = self._build_context(cursor_data)
store_id = TypeParser.parse_int(self.config.get("app.store_id")) or 0
counts = {"fetched": 0, "inserted": 0, "updated": 0, "skipped": 0, "errors": 0}
loader = GenericODSLoader(
self.db,
spec.table_name,
self._resolve_columns(spec),
list(spec.conflict_columns_override or ("source_file", "record_index")),
)
source_file = self._resolve_source_file_hint(spec)
try:
existing_ids = self._fetch_existing_ticket_ids()
candidates = self._collect_settlement_ids(
store_id, existing_ids, context.window_start, context.window_end
)
candidates = [cid for cid in candidates if cid and cid not in existing_ids]
counts["fetched"] = len(candidates)
if not candidates:
self.logger.info(
"%s: 窗口[%s ~ %s] 未发现需要抓取的小票",
spec.code,
context.window_start,
context.window_end,
)
return self._build_result("SUCCESS", counts)
payloads, skipped = self._fetch_ticket_payloads(candidates)
counts["skipped"] += skipped
rows: list[dict] = []
for idx, payload in enumerate(payloads):
row = self._build_row(
spec=spec,
store_id=store_id,
record=payload,
page_no=None,
page_size_value=None,
source_file=source_file,
record_index=idx if spec.include_record_index else None,
)
if row is None:
counts["skipped"] += 1
continue
rows.append(row)
inserted, updated, _ = loader.upsert_rows(rows)
counts["inserted"] += inserted
counts["updated"] += updated
self.db.commit()
self.logger.info(
"%s: 小票抓取完成,候选=%s 插入=%s 更新=%s 跳过=%s",
spec.code,
len(candidates),
inserted,
updated,
counts["skipped"],
)
return self._build_result("SUCCESS", counts)
except Exception:
counts["errors"] += 1
self.db.rollback()
self.logger.error("%s: 小票抓取失败", spec.code, exc_info=True)
raise
# ------------------------------------------------------------------ helpers
def _fetch_existing_ticket_ids(self) -> set[int]:
sql = """
SELECT DISTINCT
CASE WHEN (payload ->> 'orderSettleId') ~ '^[0-9]+$'
THEN (payload ->> 'orderSettleId')::bigint
END AS order_settle_id
FROM billiards_ods.settlement_ticket_details
"""
try:
rows = self.db.query(sql)
except Exception:
self.logger.warning("查询已有小票失败,按空集处理", exc_info=True)
return set()
return {
TypeParser.parse_int(row.get("order_settle_id"))
for row in rows
if row.get("order_settle_id") is not None
}
def _collect_settlement_ids(
self, store_id: int, existing_ids: set[int], window_start, window_end
) -> list[int]:
ids = self._fetch_from_payment_table(store_id)
if not ids:
ids = self._fetch_from_payment_api(store_id, window_start, window_end)
return sorted(i for i in ids if i is not None and i not in existing_ids)
def _fetch_from_payment_table(self, store_id: int) -> set[int]:
sql = """
SELECT DISTINCT COALESCE(
CASE WHEN (payload ->> 'orderSettleId') ~ '^[0-9]+$'
THEN (payload ->> 'orderSettleId')::bigint END,
CASE WHEN (payload ->> 'relateId') ~ '^[0-9]+$'
THEN (payload ->> 'relateId')::bigint END
) AS order_settle_id
FROM billiards_ods.payment_transactions
WHERE (payload ->> 'orderSettleId') ~ '^[0-9]+$'
OR (payload ->> 'relateId') ~ '^[0-9]+$'
"""
params = None
if store_id:
sql += " AND COALESCE((payload ->> 'siteId')::bigint, %s) = %s"
params = (store_id, store_id)
try:
rows = self.db.query(sql, params)
except Exception:
self.logger.warning("读取支付流水以获取结算单ID失败将尝试调用支付接口回退", exc_info=True)
return set()
return {
TypeParser.parse_int(row.get("order_settle_id"))
for row in rows
if row.get("order_settle_id") is not None
}
def _fetch_from_payment_api(self, store_id: int, window_start, window_end) -> set[int]:
params = self._merge_common_params(
{
"siteId": store_id,
"StartPayTime": TypeParser.format_timestamp(window_start, self.tz),
"EndPayTime": TypeParser.format_timestamp(window_end, self.tz),
}
)
candidate_ids: set[int] = set()
try:
for _, records, _, _ in self.api.iter_paginated(
endpoint="/PayLog/GetPayLogListPage",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
):
for rec in records:
relate_id = TypeParser.parse_int(
(rec or {}).get("relateId")
or (rec or {}).get("orderSettleId")
or (rec or {}).get("order_settle_id")
)
if relate_id:
candidate_ids.add(relate_id)
except Exception:
self.logger.warning("调用支付接口获取结算单ID失败当前批次将跳过回退来源", exc_info=True)
return candidate_ids
def _fetch_ticket_payload(self, order_settle_id: int):
payload = None
try:
for _, _, _, response in self.api.iter_paginated(
endpoint=self.SPEC.endpoint,
params={"orderSettleId": order_settle_id},
page_size=None,
data_path=self.SPEC.data_path,
list_key=self.SPEC.list_key,
):
payload = response
except Exception:
self.logger.warning(
"调用小票接口失败 orderSettleId=%s", order_settle_id, exc_info=True
)
if isinstance(payload, dict) and isinstance(payload.get("data"), list) and len(payload["data"]) == 1:
# 本地桩/回放可能把响应包装成单元素 list这里展开以贴近真实结构
payload = payload["data"][0]
return payload
def _fetch_ticket_payloads(self, candidates: list[int]) -> tuple[list, int]:
"""Fetch ticket payloads for a set of orderSettleIds; returns (payloads, skipped)."""
payloads: list = []
skipped = 0
for order_settle_id in candidates:
payload = self._fetch_ticket_payload(order_settle_id)
if payload:
payloads.append(payload)
else:
skipped += 1
return payloads, skipped
def _build_task_class(spec: OdsTaskSpec) -> Type[BaseOdsTask]:
attrs = {
"SPEC": spec,
"__doc__": spec.description or f"ODS ingestion task {spec.code}",
"__module__": __name__,
}
return type(spec.class_name, (BaseOdsTask,), attrs)
ENABLED_ODS_CODES = {
"ODS_ASSISTANT_ACCOUNTS",
"ODS_ASSISTANT_LEDGER",
"ODS_ASSISTANT_ABOLISH",
"ODS_INVENTORY_CHANGE",
"ODS_INVENTORY_STOCK",
"ODS_PACKAGE",
"ODS_GROUP_BUY_REDEMPTION",
"ODS_MEMBER",
"ODS_MEMBER_BALANCE",
"ODS_MEMBER_CARD",
"ODS_PAYMENT",
"ODS_REFUND",
"ODS_COUPON_VERIFY",
"ODS_RECHARGE_SETTLE",
"ODS_TABLES",
"ODS_GOODS_CATEGORY",
"ODS_STORE_GOODS",
"ODS_TABLE_DISCOUNT",
"ODS_TENANT_GOODS",
"ODS_SETTLEMENT_TICKET",
"ODS_ORDER_SETTLE",
}
ODS_TASK_CLASSES: Dict[str, Type[BaseOdsTask]] = {
spec.code: _build_task_class(spec)
for spec in ODS_TASK_SPECS
if spec.code in ENABLED_ODS_CODES
}
# Override with specialized settlement ticket implementation
ODS_TASK_CLASSES["ODS_SETTLEMENT_TICKET"] = OdsSettlementTicketTask
__all__ = ["ODS_TASK_CLASSES", "ODS_TASK_SPECS", "BaseOdsTask", "ENABLED_ODS_CODES"]

View File

@@ -1,80 +1,77 @@
# -*- coding: utf-8 -*-
"""订单ETL任务"""
import json
from .base_task import BaseTask
from .base_task import BaseTask, TaskContext
from loaders.facts.order import OrderLoader
from models.parsers import TypeParser
class OrdersTask(BaseTask):
"""订单数据ETL任务"""
def get_task_code(self) -> str:
return "ORDERS"
def execute(self) -> dict:
"""执行订单数据ETL"""
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
# 1. 获取时间窗口
window_start, window_end, window_minutes = self._get_time_window()
# 2. 调用API获取数据
params = {
"storeId": self.config.get("app.store_id"),
"startTime": TypeParser.format_timestamp(window_start, self.tz),
"endTime": TypeParser.format_timestamp(window_end, self.tz),
}
try:
records, pages_meta = self.api.get_paginated(
endpoint="/order/list",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",)
)
# 3. 解析并清洗数据
parsed_records = []
for rec in records:
parsed = self._parse_order(rec)
if parsed:
parsed_records.append(parsed)
# 4. 加载数据
loader = OrderLoader(self.db)
store_id = self.config.get("app.store_id")
inserted, updated, skipped = loader.upsert_orders(
parsed_records,
store_id
)
# 5. 提交事务
self.db.commit()
counts = {
"fetched": len(records),
"inserted": inserted,
"updated": updated,
"skipped": skipped,
"errors": 0
# ------------------------------------------------------------------ E/T/L hooks
def extract(self, context: TaskContext) -> dict:
"""调用 API 拉取订单记录"""
params = self._merge_common_params(
{
"siteId": context.store_id,
"rangeStartTime": TypeParser.format_timestamp(context.window_start, self.tz),
"rangeEndTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
self.logger.info(
f"{self.get_task_code()} 完成: {counts}"
)
return self._build_result("SUCCESS", counts)
except Exception as e:
self.db.rollback()
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
raise
def _parse_order(self, raw: dict) -> dict:
)
records, pages_meta = self.api.get_paginated(
endpoint="/Site/GetAllOrderSettleList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="settleList",
)
return {"records": records, "meta": pages_meta}
def transform(self, extracted: dict, context: TaskContext) -> dict:
"""解析原始订单 JSON"""
parsed_records = []
skipped = 0
for rec in extracted.get("records", []):
parsed = self._parse_order(rec, context.store_id)
if parsed:
parsed_records.append(parsed)
else:
skipped += 1
return {
"records": parsed_records,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
"""写入 fact_order"""
loader = OrderLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_orders(
transformed["records"], context.store_id
)
counts = {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
return counts
# ------------------------------------------------------------------ helpers
def _parse_order(self, raw: dict, store_id: int) -> dict | None:
"""解析单条订单记录"""
try:
return {
"store_id": self.config.get("app.store_id"),
"store_id": store_id,
"order_id": TypeParser.parse_int(raw.get("orderId")),
"order_no": raw.get("orderNo"),
"member_id": TypeParser.parse_int(raw.get("memberId")),
@@ -87,8 +84,8 @@ class OrdersTask(BaseTask):
"pay_status": raw.get("payStatus"),
"order_status": raw.get("orderStatus"),
"remark": raw.get("remark"),
"raw_data": json.dumps(raw, ensure_ascii=False)
"raw_data": json.dumps(raw, ensure_ascii=False),
}
except Exception as e:
self.logger.warning(f"解析订单失败: {e}, 原始数据: {raw}")
except Exception as exc:
self.logger.warning("解析订单失败: %s, 原始数据: %s", exc, raw)
return None

View File

@@ -0,0 +1,90 @@
# -*- coding: utf-8 -*-
"""团购/套餐定义任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.dimensions.package import PackageDefinitionLoader
from models.parsers import TypeParser
class PackagesDefTask(BaseTask):
"""同步团购套餐定义"""
def get_task_code(self) -> str:
return "PACKAGES_DEF"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params({"siteId": context.store_id})
records, _ = self.api.get_paginated(
endpoint="/PackageCoupon/QueryPackageCouponList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="packageCouponList",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_package(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = PackageDefinitionLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_packages(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_package(self, raw: dict, store_id: int) -> dict | None:
package_id = TypeParser.parse_int(raw.get("id"))
if not package_id:
self.logger.warning("跳过缺少 package id 的套餐记录: %s", raw)
return None
return {
"store_id": store_id,
"package_id": package_id,
"package_code": raw.get("package_id") or raw.get("packageId"),
"package_name": raw.get("package_name"),
"table_area_id": raw.get("table_area_id"),
"table_area_name": raw.get("table_area_name"),
"selling_price": TypeParser.parse_decimal(
raw.get("selling_price") or raw.get("sellingPrice")
),
"duration_seconds": TypeParser.parse_int(raw.get("duration")),
"start_time": TypeParser.parse_timestamp(
raw.get("start_time") or raw.get("startTime"), self.tz
),
"end_time": TypeParser.parse_timestamp(
raw.get("end_time") or raw.get("endTime"), self.tz
),
"type": raw.get("type"),
"is_enabled": raw.get("is_enabled"),
"is_delete": raw.get("is_delete"),
"usable_count": TypeParser.parse_int(raw.get("usable_count")),
"creator_name": raw.get("creator_name"),
"date_type": raw.get("date_type"),
"group_type": raw.get("group_type"),
"coupon_money": TypeParser.parse_decimal(
raw.get("coupon_money") or raw.get("couponMoney")
),
"area_tag_type": raw.get("area_tag_type"),
"system_group_type": raw.get("system_group_type"),
"card_type_ids": raw.get("card_type_ids"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,138 @@
# -*- coding: utf-8 -*-
from .base_dwd_task import BaseDwdTask
from loaders.facts.payment import PaymentLoader
from models.parsers import TypeParser
import json
class PaymentsDwdTask(BaseDwdTask):
"""
DWD Task: Process Payment Records from ODS to Fact Table
Source: billiards_ods.ods_payment
Target: billiards.fact_payment
"""
def get_task_code(self) -> str:
return "PAYMENTS_DWD"
def execute(self) -> dict:
self.logger.info(f"Starting {self.get_task_code()} task")
window_start, window_end, _ = self._get_time_window()
self.logger.info(f"Processing window: {window_start} to {window_end}")
loader = PaymentLoader(self.db, logger=self.logger)
store_id = self.config.get("app.store_id")
total_inserted = 0
total_updated = 0
total_skipped = 0
# Iterate ODS Data
batches = self.iter_ods_rows(
table_name="billiards_ods.ods_payment_record",
columns=["site_id", "pay_id", "payload", "fetched_at"],
start_time=window_start,
end_time=window_end
)
for batch in batches:
if not batch:
continue
parsed_rows = []
for row in batch:
payload = self.parse_payload(row)
if not payload:
continue
parsed = self._parse_payment(payload, store_id)
if parsed:
parsed_rows.append(parsed)
if parsed_rows:
inserted, updated, skipped = loader.upsert_payments(parsed_rows, store_id)
total_inserted += inserted
total_updated += updated
total_skipped += skipped
self.db.commit()
self.logger.info(
"Task %s completed. inserted=%s updated=%s skipped=%s",
self.get_task_code(),
total_inserted,
total_updated,
total_skipped,
)
return {
"status": "SUCCESS",
"counts": {
"inserted": total_inserted,
"updated": total_updated,
"skipped": total_skipped,
},
"window_start": window_start,
"window_end": window_end,
}
def _parse_payment(self, raw: dict, store_id: int) -> dict:
"""Parse ODS payload into Fact structure"""
try:
pay_id = TypeParser.parse_int(raw.get("payId") or raw.get("id"))
if not pay_id:
return None
relate_type = str(raw.get("relateType") or raw.get("relate_type") or "")
relate_id = TypeParser.parse_int(raw.get("relateId") or raw.get("relate_id"))
# Attempt to populate settlement / trade identifiers
order_settle_id = TypeParser.parse_int(
raw.get("orderSettleId") or raw.get("order_settle_id")
)
order_trade_no = TypeParser.parse_int(
raw.get("orderTradeNo") or raw.get("order_trade_no")
)
if relate_type in {"1", "SETTLE", "ORDER"}:
order_settle_id = order_settle_id or relate_id
return {
"store_id": store_id,
"pay_id": pay_id,
"order_id": TypeParser.parse_int(raw.get("orderId") or raw.get("order_id")),
"order_settle_id": order_settle_id,
"order_trade_no": order_trade_no,
"relate_type": relate_type,
"relate_id": relate_id,
"site_id": TypeParser.parse_int(
raw.get("siteId") or raw.get("site_id") or store_id
),
"tenant_id": TypeParser.parse_int(raw.get("tenantId") or raw.get("tenant_id")),
"create_time": TypeParser.parse_timestamp(
raw.get("createTime") or raw.get("create_time"), self.tz
),
"pay_time": TypeParser.parse_timestamp(raw.get("payTime"), self.tz),
"pay_amount": TypeParser.parse_decimal(raw.get("payAmount")),
"fee_amount": TypeParser.parse_decimal(
raw.get("feeAmount")
or raw.get("serviceFee")
or raw.get("channelFee")
or raw.get("fee_amount")
),
"discount_amount": TypeParser.parse_decimal(
raw.get("discountAmount")
or raw.get("couponAmount")
or raw.get("discount_amount")
),
"payment_method": str(raw.get("paymentMethod") or raw.get("payment_method") or ""),
"pay_type": raw.get("payType") or raw.get("pay_type"),
"online_pay_channel": raw.get("onlinePayChannel") or raw.get("online_pay_channel"),
"pay_terminal": raw.get("payTerminal") or raw.get("pay_terminal"),
"pay_status": str(raw.get("payStatus") or raw.get("pay_status") or ""),
"remark": raw.get("remark"),
"raw_data": json.dumps(raw, ensure_ascii=False)
}
except Exception as e:
self.logger.warning(f"Error parsing payment: {e}")
return None

View File

@@ -1,78 +1,111 @@
# -*- coding: utf-8 -*-
"""支付记录ETL任务"""
import json
from .base_task import BaseTask
from .base_task import BaseTask, TaskContext
from loaders.facts.payment import PaymentLoader
from models.parsers import TypeParser
class PaymentsTask(BaseTask):
"""支付记录ETL任务"""
"""支付记录 E/T/L 任务"""
def get_task_code(self) -> str:
return "PAYMENTS"
def execute(self) -> dict:
"""执行支付记录ETL"""
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
window_start, window_end, window_minutes = self._get_time_window()
params = {
"storeId": self.config.get("app.store_id"),
"startTime": TypeParser.format_timestamp(window_start, self.tz),
"endTime": TypeParser.format_timestamp(window_end, self.tz),
}
try:
records, pages_meta = self.api.get_paginated(
endpoint="/pay/records",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",)
)
parsed_records = []
for rec in records:
parsed = self._parse_payment(rec)
if parsed:
parsed_records.append(parsed)
loader = PaymentLoader(self.db)
store_id = self.config.get("app.store_id")
inserted, updated, skipped = loader.upsert_payments(parsed_records, store_id)
self.db.commit()
counts = {
"fetched": len(records),
"inserted": inserted,
"updated": updated,
"skipped": skipped,
"errors": 0
# ------------------------------------------------------------------ E/T/L hooks
def extract(self, context: TaskContext) -> dict:
"""调用 API 抓取支付记录"""
params = self._merge_common_params(
{
"siteId": context.store_id,
"StartPayTime": TypeParser.format_timestamp(context.window_start, self.tz),
"EndPayTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
return self._build_result("SUCCESS", counts)
except Exception as e:
self.db.rollback()
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
raise
def _parse_payment(self, raw: dict) -> dict:
)
records, pages_meta = self.api.get_paginated(
endpoint="/PayLog/GetPayLogListPage",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
)
return {"records": records, "meta": pages_meta}
def transform(self, extracted: dict, context: TaskContext) -> dict:
"""解析支付 JSON"""
parsed, skipped = [], 0
for rec in extracted.get("records", []):
cleaned = self._parse_payment(rec, context.store_id)
if cleaned:
parsed.append(cleaned)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
"""写入 fact_payment"""
loader = PaymentLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_payments(
transformed["records"], context.store_id
)
counts = {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
return counts
# ------------------------------------------------------------------ helpers
def _parse_payment(self, raw: dict, store_id: int) -> dict | None:
"""解析支付记录"""
try:
return {
"store_id": self.config.get("app.store_id"),
"pay_id": TypeParser.parse_int(raw.get("payId")),
"store_id": store_id,
"pay_id": TypeParser.parse_int(raw.get("payId") or raw.get("id")),
"order_id": TypeParser.parse_int(raw.get("orderId")),
"order_settle_id": TypeParser.parse_int(
raw.get("orderSettleId") or raw.get("order_settle_id")
),
"order_trade_no": TypeParser.parse_int(
raw.get("orderTradeNo") or raw.get("order_trade_no")
),
"relate_type": raw.get("relateType") or raw.get("relate_type"),
"relate_id": TypeParser.parse_int(raw.get("relateId") or raw.get("relate_id")),
"site_id": TypeParser.parse_int(
raw.get("siteId") or raw.get("site_id") or store_id
),
"tenant_id": TypeParser.parse_int(raw.get("tenantId") or raw.get("tenant_id")),
"pay_time": TypeParser.parse_timestamp(raw.get("payTime"), self.tz),
"create_time": TypeParser.parse_timestamp(
raw.get("createTime") or raw.get("create_time"), self.tz
),
"pay_amount": TypeParser.parse_decimal(raw.get("payAmount")),
"fee_amount": TypeParser.parse_decimal(
raw.get("feeAmount")
or raw.get("serviceFee")
or raw.get("channelFee")
or raw.get("fee_amount")
),
"discount_amount": TypeParser.parse_decimal(
raw.get("discountAmount")
or raw.get("couponAmount")
or raw.get("discount_amount")
),
"pay_type": raw.get("payType"),
"payment_method": raw.get("paymentMethod") or raw.get("payment_method"),
"online_pay_channel": raw.get("onlinePayChannel")
or raw.get("online_pay_channel"),
"pay_status": raw.get("payStatus"),
"pay_terminal": raw.get("payTerminal") or raw.get("pay_terminal"),
"remark": raw.get("remark"),
"raw_data": json.dumps(raw, ensure_ascii=False)
"raw_data": json.dumps(raw, ensure_ascii=False),
}
except Exception as e:
self.logger.warning(f"解析支付记录失败: {e}, 原始数据: {raw}")
except Exception as exc:
self.logger.warning("解析支付记录失败: %s, 原始数据: %s", exc, raw)
return None

View File

@@ -3,7 +3,7 @@
import json
from .base_task import BaseTask
from .base_task import BaseTask, TaskContext
from loaders.dimensions.product import ProductLoader
from models.parsers import TypeParser
@@ -12,95 +12,56 @@ class ProductsTask(BaseTask):
"""商品维度 ETL 任务"""
def get_task_code(self) -> str:
"""任务代码,应与 etl_admin.etl_task.task_code 一致"""
return "PRODUCTS"
def execute(self) -> dict:
"""
执行商品档案 ETL
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params({"siteId": context.store_id})
records, _ = self.api.get_paginated(
endpoint="/TenantGoods/QueryTenantGoods",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="tenantGoodsList",
)
return {"records": records}
流程:
1. 调用上游 /TenantGoods/QueryTenantGoods 分页拉取商品列表
2. 解析/清洗字段
3. 通过 ProductLoader 写入 dim_product 和 dim_product_price_scd
"""
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
params = {
"storeId": self.config.get("app.store_id"),
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
parsed_row = self._parse_product(raw, context.store_id)
if parsed_row:
parsed.append(parsed_row)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = ProductLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_products(
transformed["records"], context.store_id
)
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_product(self, raw: dict, store_id: int) -> dict | None:
try:
# 1. 分页拉取数据
records, pages_meta = self.api.get_paginated(
endpoint="/TenantGoods/QueryTenantGoods",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
)
# 2. 解析/清洗
parsed_records = []
for raw in records:
parsed = self._parse_product(raw)
if parsed:
parsed_records.append(parsed)
# 3. 加载入库(维度主表 + 价格SCD2
loader = ProductLoader(self.db)
store_id = self.config.get("app.store_id")
inserted, updated, skipped = loader.upsert_products(
parsed_records, store_id
)
# 4. 提交事务
self.db.commit()
counts = {
"fetched": len(records),
"inserted": inserted,
"updated": updated,
"skipped": skipped,
"errors": 0,
}
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
return self._build_result("SUCCESS", counts)
except Exception:
# 明确回滚,避免部分成功
self.db.rollback()
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
raise
def _parse_product(self, raw: dict) -> dict | None:
"""
解析单条商品记录,字段映射参考旧版 upsert_dim_product_and_price_scd
上游字段示例:
- siteGoodsId / tenantGoodsId / productId
- goodsName / productName
- tenantGoodsCategoryId / goodsCategoryId / categoryName / goodsCategorySecondId
- goodsUnit
- costPrice / goodsPrice / salePrice
- goodsState / status
- supplierId / barcode / isCombo
- createTime / updateTime
"""
try:
product_id = (
TypeParser.parse_int(
raw.get("siteGoodsId")
or raw.get("tenantGoodsId")
or raw.get("productId")
)
product_id = TypeParser.parse_int(
raw.get("siteGoodsId") or raw.get("tenantGoodsId") or raw.get("productId")
)
if not product_id:
# 主键缺失,直接跳过
return None
return {
"store_id": self.config.get("app.store_id"),
"store_id": store_id,
"product_id": product_id,
"site_product_id": TypeParser.parse_int(raw.get("siteGoodsId")),
"product_name": raw.get("goodsName") or raw.get("productName"),
@@ -108,15 +69,12 @@ class ProductsTask(BaseTask):
raw.get("tenantGoodsCategoryId") or raw.get("goodsCategoryId")
),
"category_name": raw.get("categoryName"),
"second_category_id": TypeParser.parse_int(
raw.get("goodsCategorySecondId")
),
"second_category_id": TypeParser.parse_int(raw.get("goodsCategorySecondId")),
"unit": raw.get("goodsUnit"),
"cost_price": TypeParser.parse_decimal(raw.get("costPrice")),
"sale_price": TypeParser.parse_decimal(
raw.get("goodsPrice") or raw.get("salePrice")
),
# 旧版这里就是 None如后面有明确字段可以再补
"allow_discount": None,
"status": raw.get("goodsState") or raw.get("status"),
"supplier_id": TypeParser.parse_int(raw.get("supplierId"))
@@ -126,14 +84,10 @@ class ProductsTask(BaseTask):
"is_combo": bool(raw.get("isCombo"))
if raw.get("isCombo") is not None
else None,
"created_time": TypeParser.parse_timestamp(
raw.get("createTime"), self.tz
),
"updated_time": TypeParser.parse_timestamp(
raw.get("updateTime"), self.tz
),
"created_time": TypeParser.parse_timestamp(raw.get("createTime"), self.tz),
"updated_time": TypeParser.parse_timestamp(raw.get("updateTime"), self.tz),
"raw_data": json.dumps(raw, ensure_ascii=False),
}
except Exception as e:
self.logger.warning(f"解析商品记录失败: {e}, 原始数据: {raw}")
return None
except Exception as exc:
self.logger.warning("解析商品记录失败: %s, 原始数据: %s", exc, raw)
return None

View File

@@ -0,0 +1,90 @@
# -*- coding: utf-8 -*-
"""退款记录任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.refund import RefundLoader
from models.parsers import TypeParser
class RefundsTask(BaseTask):
"""同步支付退款流水"""
def get_task_code(self) -> str:
return "REFUNDS"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/Order/GetRefundPayLogList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_refund(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = RefundLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_refunds(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_refund(self, raw: dict, store_id: int) -> dict | None:
refund_id = TypeParser.parse_int(raw.get("id"))
if not refund_id:
self.logger.warning("跳过缺少退款ID的数据: %s", raw)
return None
return {
"store_id": store_id,
"refund_id": refund_id,
"site_id": TypeParser.parse_int(raw.get("site_id") or raw.get("siteId")),
"tenant_id": TypeParser.parse_int(raw.get("tenant_id") or raw.get("tenantId")),
"pay_amount": TypeParser.parse_decimal(raw.get("pay_amount")),
"pay_status": raw.get("pay_status"),
"pay_time": TypeParser.parse_timestamp(
raw.get("pay_time") or raw.get("payTime"), self.tz
),
"create_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"relate_type": raw.get("relate_type"),
"relate_id": TypeParser.parse_int(raw.get("relate_id")),
"payment_method": raw.get("payment_method"),
"refund_amount": TypeParser.parse_decimal(raw.get("refund_amount")),
"action_type": raw.get("action_type"),
"pay_terminal": raw.get("pay_terminal"),
"operator_id": TypeParser.parse_int(raw.get("operator_id")),
"channel_pay_no": raw.get("channel_pay_no"),
"channel_fee": TypeParser.parse_decimal(raw.get("channel_fee")),
"is_delete": raw.get("is_delete"),
"member_id": TypeParser.parse_int(raw.get("member_id")),
"member_card_id": TypeParser.parse_int(raw.get("member_card_id")),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,92 @@
# -*- coding: utf-8 -*-
"""台费折扣任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.table_discount import TableDiscountLoader
from models.parsers import TypeParser
class TableDiscountTask(BaseTask):
"""同步台费折扣/调价记录"""
def get_task_code(self) -> str:
return "TABLE_DISCOUNT"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"startTime": TypeParser.format_timestamp(context.window_start, self.tz),
"endTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/Site/GetTaiFeeAdjustList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="taiFeeAdjustInfos",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_discount(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = TableDiscountLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_discounts(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_discount(self, raw: dict, store_id: int) -> dict | None:
discount_id = TypeParser.parse_int(raw.get("id"))
if not discount_id:
self.logger.warning("跳过缺少折扣ID的记录: %s", raw)
return None
table_profile = raw.get("tableProfile") or {}
return {
"store_id": store_id,
"discount_id": discount_id,
"adjust_type": raw.get("adjust_type") or raw.get("adjustType"),
"applicant_id": TypeParser.parse_int(raw.get("applicant_id")),
"applicant_name": raw.get("applicant_name"),
"operator_id": TypeParser.parse_int(raw.get("operator_id")),
"operator_name": raw.get("operator_name"),
"ledger_amount": TypeParser.parse_decimal(raw.get("ledger_amount")),
"ledger_count": TypeParser.parse_int(raw.get("ledger_count")),
"ledger_name": raw.get("ledger_name"),
"ledger_status": raw.get("ledger_status"),
"order_settle_id": TypeParser.parse_int(raw.get("order_settle_id")),
"order_trade_no": TypeParser.parse_int(raw.get("order_trade_no")),
"site_table_id": TypeParser.parse_int(
raw.get("site_table_id") or table_profile.get("id")
),
"table_area_id": TypeParser.parse_int(
raw.get("tableAreaId") or table_profile.get("site_table_area_id")
),
"table_area_name": table_profile.get("site_table_area_name"),
"create_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"is_delete": raw.get("is_delete"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -1,4 +1,84 @@
# -*- coding: utf-8 -*-
"""台桌档案任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.dimensions.table import TableLoader
from models.parsers import TypeParser
class TablesTask(BaseTask):
def get_task_code(self) -> str: # 返回 "TABLES"
def execute(self) -> dict: # 拉取 /Table/GetSiteTables
def _parse_table(self, raw: dict) -> dict | None:
"""同步门店台桌列表"""
def get_task_code(self) -> str:
return "TABLES"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params({"siteId": context.store_id})
records, _ = self.api.get_paginated(
endpoint="/Table/GetSiteTables",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="siteTables",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_table(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = TableLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_tables(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_table(self, raw: dict, store_id: int) -> dict | None:
table_id = TypeParser.parse_int(raw.get("id"))
if not table_id:
self.logger.warning("跳过缺少 table_id 的台桌记录: %s", raw)
return None
return {
"store_id": store_id,
"table_id": table_id,
"site_id": TypeParser.parse_int(raw.get("site_id") or raw.get("siteId")),
"area_id": TypeParser.parse_int(
raw.get("site_table_area_id") or raw.get("siteTableAreaId")
),
"area_name": raw.get("areaName") or raw.get("site_table_area_name"),
"table_name": raw.get("table_name") or raw.get("tableName"),
"table_price": TypeParser.parse_decimal(
raw.get("table_price") or raw.get("tablePrice")
),
"table_status": raw.get("table_status") or raw.get("tableStatus"),
"table_status_name": raw.get("tableStatusName"),
"light_status": raw.get("light_status"),
"is_rest_area": raw.get("is_rest_area"),
"show_status": raw.get("show_status"),
"virtual_table": raw.get("virtual_table"),
"charge_free": raw.get("charge_free"),
"only_allow_groupon": raw.get("only_allow_groupon"),
"is_online_reservation": raw.get("is_online_reservation"),
"created_time": TypeParser.parse_timestamp(
raw.get("create_time") or raw.get("createTime"), self.tz
),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -0,0 +1,69 @@
# -*- coding: utf-8 -*-
from .base_dwd_task import BaseDwdTask
from loaders.facts.ticket import TicketLoader
class TicketDwdTask(BaseDwdTask):
"""
DWD Task: Process Ticket Details from ODS to Fact Tables
Source: billiards_ods.ods_ticket_detail
Targets:
- billiards.fact_order
- billiards.fact_order_goods
- billiards.fact_table_usage
- billiards.fact_assistant_service
"""
def get_task_code(self) -> str:
return "TICKET_DWD"
def execute(self) -> dict:
self.logger.info(f"Starting {self.get_task_code()} task")
# 1. Get Time Window (Incremental Load)
window_start, window_end, _ = self._get_time_window()
self.logger.info(f"Processing window: {window_start} to {window_end}")
# 2. Initialize Loader
loader = TicketLoader(self.db, logger=self.logger)
store_id = self.config.get("app.store_id")
total_inserted = 0
total_errors = 0
# 3. Iterate ODS Data
# We query ods_ticket_detail based on fetched_at
batches = self.iter_ods_rows(
table_name="billiards_ods.settlement_ticket_details",
columns=["payload", "fetched_at", "source_file", "record_index"],
start_time=window_start,
end_time=window_end
)
for batch in batches:
if not batch:
continue
# Extract payloads
tickets = []
for row in batch:
payload = self.parse_payload(row)
if payload:
tickets.append(payload)
# Process Batch
inserted, errors = loader.process_tickets(tickets, store_id)
total_inserted += inserted
total_errors += errors
# 4. Commit
self.db.commit()
self.logger.info(f"Task {self.get_task_code()} completed. Inserted: {total_inserted}, Errors: {total_errors}")
return {
"status": "success",
"inserted": total_inserted,
"errors": total_errors,
"window_start": window_start.isoformat(),
"window_end": window_end.isoformat()
}

View File

@@ -0,0 +1,102 @@
# -*- coding: utf-8 -*-
"""充值记录任务"""
import json
from .base_task import BaseTask, TaskContext
from loaders.facts.topup import TopupLoader
from models.parsers import TypeParser
class TopupsTask(BaseTask):
"""同步储值充值结算记录"""
def get_task_code(self) -> str:
return "TOPUPS"
def extract(self, context: TaskContext) -> dict:
params = self._merge_common_params(
{
"siteId": context.store_id,
"rangeStartTime": TypeParser.format_timestamp(context.window_start, self.tz),
"rangeEndTime": TypeParser.format_timestamp(context.window_end, self.tz),
}
)
records, _ = self.api.get_paginated(
endpoint="/Site/GetRechargeSettleList",
params=params,
page_size=self.config.get("api.page_size", 200),
data_path=("data",),
list_key="settleList",
)
return {"records": records}
def transform(self, extracted: dict, context: TaskContext) -> dict:
parsed, skipped = [], 0
for raw in extracted.get("records", []):
mapped = self._parse_topup(raw, context.store_id)
if mapped:
parsed.append(mapped)
else:
skipped += 1
return {
"records": parsed,
"fetched": len(extracted.get("records", [])),
"skipped": skipped,
}
def load(self, transformed: dict, context: TaskContext) -> dict:
loader = TopupLoader(self.db)
inserted, updated, loader_skipped = loader.upsert_topups(transformed["records"])
return {
"fetched": transformed["fetched"],
"inserted": inserted,
"updated": updated,
"skipped": transformed["skipped"] + loader_skipped,
"errors": 0,
}
def _parse_topup(self, raw: dict, store_id: int) -> dict | None:
node = raw.get("settleList") if isinstance(raw.get("settleList"), dict) else raw
topup_id = TypeParser.parse_int(node.get("id"))
if not topup_id:
self.logger.warning("跳过缺少充值ID的记录: %s", raw)
return None
return {
"store_id": store_id,
"topup_id": topup_id,
"member_id": TypeParser.parse_int(node.get("memberId")),
"member_name": node.get("memberName"),
"member_phone": node.get("memberPhone"),
"card_id": TypeParser.parse_int(node.get("tenantMemberCardId")),
"card_type_name": node.get("memberCardTypeName"),
"pay_amount": TypeParser.parse_decimal(node.get("payAmount")),
"consume_money": TypeParser.parse_decimal(node.get("consumeMoney")),
"settle_status": node.get("settleStatus"),
"settle_type": node.get("settleType"),
"settle_name": node.get("settleName"),
"settle_relate_id": TypeParser.parse_int(node.get("settleRelateId")),
"pay_time": TypeParser.parse_timestamp(
node.get("payTime") or node.get("pay_time"), self.tz
),
"create_time": TypeParser.parse_timestamp(
node.get("createTime") or node.get("create_time"), self.tz
),
"operator_id": TypeParser.parse_int(node.get("operatorId")),
"operator_name": node.get("operatorName"),
"payment_method": node.get("paymentMethod"),
"refund_amount": TypeParser.parse_decimal(node.get("refundAmount")),
"cash_amount": TypeParser.parse_decimal(node.get("cashAmount")),
"card_amount": TypeParser.parse_decimal(node.get("cardAmount")),
"balance_amount": TypeParser.parse_decimal(node.get("balanceAmount")),
"online_amount": TypeParser.parse_decimal(node.get("onlineAmount")),
"rounding_amount": TypeParser.parse_decimal(node.get("roundingAmount")),
"adjust_amount": TypeParser.parse_decimal(node.get("adjustAmount")),
"goods_money": TypeParser.parse_decimal(node.get("goodsMoney")),
"table_charge_money": TypeParser.parse_decimal(node.get("tableChargeMoney")),
"service_money": TypeParser.parse_decimal(node.get("serviceMoney")),
"coupon_amount": TypeParser.parse_decimal(node.get("couponAmount")),
"order_remark": node.get("orderRemark"),
"raw_data": json.dumps(raw, ensure_ascii=False),
}

View File

@@ -1,647 +0,0 @@
[
{
"data": {
"total": 15,
"abolitionAssistants": [
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-11-09 19:23:29",
"id": 2957675849518789,
"siteId": 2790685415443269,
"tableAreaId": 2791963816579205,
"tableId": 2793016660660357,
"tableArea": "C区",
"tableName": "C1",
"assistantOn": "27",
"assistantName": "泡芙",
"pdChargeMinutes": 214,
"assistantAbolishAmount": 5.83,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-11-06 17:42:09",
"id": 2953329501898373,
"siteId": 2790685415443269,
"tableAreaId": 2802006170324037,
"tableId": 2851642357976581,
"tableArea": "补时长",
"tableName": "补时长5",
"assistantOn": "23",
"assistantName": "婉婉",
"pdChargeMinutes": 10800,
"assistantAbolishAmount": 570.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-11-06 17:42:09",
"id": 2953329502357125,
"siteId": 2790685415443269,
"tableAreaId": 2802006170324037,
"tableId": 2851642357976581,
"tableArea": "补时长",
"tableName": "补时长5",
"assistantOn": "52",
"assistantName": "小柔",
"pdChargeMinutes": 10800,
"assistantAbolishAmount": 570.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-30 01:17:22",
"id": 2942452375932869,
"siteId": 2790685415443269,
"tableAreaId": 2791963825803397,
"tableId": 2793018776604805,
"tableArea": "VIP包厢",
"tableName": "VIP1",
"assistantOn": "2",
"assistantName": "佳怡",
"pdChargeMinutes": 0,
"assistantAbolishAmount": 0.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-29 06:57:22",
"id": 2941371032964997,
"siteId": 2790685415443269,
"tableAreaId": 2791963848527941,
"tableId": 2793021451292741,
"tableArea": "666",
"tableName": "董事办",
"assistantOn": "4",
"assistantName": "璇子",
"pdChargeMinutes": 0,
"assistantAbolishAmount": 0.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-28 02:13:18",
"id": 2939676194180229,
"siteId": 2790685415443269,
"tableAreaId": 2791963887030341,
"tableId": 2793023960551493,
"tableArea": "麻将房",
"tableName": "1",
"assistantOn": "2",
"assistantName": "佳怡",
"pdChargeMinutes": 3602,
"assistantAbolishAmount": 108.06,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-26 21:06:37",
"id": 2937959143262725,
"siteId": 2790685415443269,
"tableAreaId": 2791963855982661,
"tableId": 2793022145302597,
"tableArea": "K包",
"tableName": "888",
"assistantOn": "16",
"assistantName": "周周",
"pdChargeMinutes": 0,
"assistantAbolishAmount": 0.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-25 21:36:22",
"id": 2936572806285765,
"siteId": 2790685415443269,
"tableAreaId": 2791963816579205,
"tableId": 2793017278451845,
"tableArea": "C区",
"tableName": "C2",
"assistantOn": "4",
"assistantName": "璇子",
"pdChargeMinutes": 0,
"assistantAbolishAmount": 0.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-23 19:05:48",
"id": 2933593641256581,
"siteId": 2790685415443269,
"tableAreaId": 2791963807682693,
"tableId": 2793012902318213,
"tableArea": "B区",
"tableName": "B9",
"assistantOn": "16",
"assistantName": "周周",
"pdChargeMinutes": 3600,
"assistantAbolishAmount": 190.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-18 20:25:50",
"id": 2926594431305093,
"siteId": 2790685415443269,
"tableAreaId": 2791963794329671,
"tableId": 2793001904918661,
"tableArea": "A区",
"tableName": "A4",
"assistantOn": "15",
"assistantName": "七七",
"pdChargeMinutes": 2379,
"assistantAbolishAmount": 71.37,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-14 14:20:32",
"id": 2920573007709573,
"siteId": 2790685415443269,
"tableAreaId": 2791963855982661,
"tableId": 2793022145302597,
"tableArea": "K包",
"tableName": "888",
"assistantOn": "9",
"assistantName": "球球",
"pdChargeMinutes": 14400,
"assistantAbolishAmount": 392.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-03 01:21:59",
"id": 2904236313234373,
"siteId": 2790685415443269,
"tableAreaId": 2791963848527941,
"tableId": 2793020955840645,
"tableArea": "666",
"tableName": "666",
"assistantOn": "9",
"assistantName": "球球",
"pdChargeMinutes": 0,
"assistantAbolishAmount": 0.0,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-01 00:27:29",
"id": 2901351579143365,
"siteId": 2790685415443269,
"tableAreaId": 2791963855982661,
"tableId": 2793022145302597,
"tableArea": "K包",
"tableName": "888",
"assistantOn": "99",
"assistantName": "Amy",
"pdChargeMinutes": 10605,
"assistantAbolishAmount": 465.44,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-01 00:27:29",
"id": 2901351578864837,
"siteId": 2790685415443269,
"tableAreaId": 2791963855982661,
"tableId": 2793022145302597,
"tableArea": "K包",
"tableName": "888",
"assistantOn": "4",
"assistantName": "璇子",
"pdChargeMinutes": 10608,
"assistantAbolishAmount": 318.24,
"trashReason": ""
},
{
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 1,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"createTime": "2025-10-01 00:27:29",
"id": 2901351578602693,
"siteId": 2790685415443269,
"tableAreaId": 2791963855982661,
"tableId": 2793022145302597,
"tableArea": "K包",
"tableName": "888",
"assistantOn": "2",
"assistantName": "佳怡",
"pdChargeMinutes": 10611,
"assistantAbolishAmount": 318.33,
"trashReason": ""
}
]
},
"code": 0
},
{
"data": {
"total": 15,
"abolitionAssistants": []
},
"code": 0
}
]

View File

@@ -1,673 +0,0 @@
[
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2955202296416389,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -5000.0,
"pay_status": 2,
"pay_time": "2025-11-08 01:27:16",
"create_time": "2025-11-08 01:27:16",
"relate_type": 5,
"relate_id": 2955078219057349,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2955171790194821,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -10000.0,
"pay_status": 2,
"pay_time": "2025-11-08 00:56:14",
"create_time": "2025-11-08 00:56:14",
"relate_type": 5,
"relate_id": 2955153380001861,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2951883030513413,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -12.0,
"pay_status": 2,
"pay_time": "2025-11-05 17:10:44",
"create_time": "2025-11-05 17:10:44",
"relate_type": 2,
"relate_id": 2951881496577861,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2948959062542597,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -65.0,
"pay_status": 2,
"pay_time": "2025-11-03 15:36:19",
"create_time": "2025-11-03 15:36:19",
"relate_type": 2,
"relate_id": 2948934289967557,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2948630468005509,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -88.33,
"pay_status": 2,
"pay_time": "2025-11-03 10:02:03",
"create_time": "2025-11-03 10:02:03",
"relate_type": 2,
"relate_id": 2948246513454661,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2948269239095045,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -0.67,
"pay_status": 2,
"pay_time": "2025-11-03 03:54:36",
"create_time": "2025-11-03 03:54:36",
"relate_type": 2,
"relate_id": 2948246513454661,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2944743812581445,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -44000.0,
"pay_status": 2,
"pay_time": "2025-10-31 16:08:21",
"create_time": "2025-10-31 16:08:21",
"relate_type": 5,
"relate_id": 2944743413958789,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2931109065131653,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -10.0,
"pay_status": 2,
"pay_time": "2025-10-22 00:58:22",
"create_time": "2025-10-22 00:58:22",
"relate_type": 2,
"relate_id": 2931108522378885,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2921195994465669,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -2.0,
"pay_status": 2,
"pay_time": "2025-10-15 00:54:16",
"create_time": "2025-10-15 00:54:16",
"relate_type": 2,
"relate_id": 2920440691344901,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2919690732146181,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -3000.0,
"pay_status": 2,
"pay_time": "2025-10-13 23:23:02",
"create_time": "2025-10-13 23:23:02",
"relate_type": 5,
"relate_id": 2919519811440261,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
},
{
"tenantName": "朗朗桌球",
"siteProfile": {
"id": 2790685415443269,
"org_id": 2790684179467077,
"shop_name": "朗朗桌球",
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
"business_tel": "13316068642",
"full_address": "广东省广州市天河区丽阳街12号",
"address": "广东省广州市天河区天园街道朗朗桌球",
"longitude": 113.360321,
"latitude": 23.133629,
"tenant_site_region_id": 156440100,
"tenant_id": 2790683160709957,
"auto_light": 1,
"attendance_distance": 0,
"wifi_name": "",
"wifi_password": "",
"customer_service_qrcode": "",
"customer_service_wechat": "",
"fixed_pay_qrCode": "",
"prod_env": 1,
"light_status": 2,
"light_type": 0,
"site_type": 1,
"light_token": "",
"site_label": "A",
"attendance_enabled": 1,
"shop_status": 1
},
"id": 2914039374956165,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"pay_sn": 0,
"pay_amount": -8.0,
"pay_status": 2,
"pay_time": "2025-10-09 23:34:11",
"create_time": "2025-10-09 23:34:11",
"relate_type": 2,
"relate_id": 2914030720124357,
"is_revoke": 0,
"is_delete": 0,
"online_pay_channel": 0,
"payment_method": 4,
"balance_frozen_amount": 0.0,
"card_frozen_amount": 0.0,
"member_id": 0,
"member_card_id": 0,
"round_amount": 0.0,
"online_pay_type": 0,
"action_type": 2,
"refund_amount": 0.0,
"cashier_point_id": 0,
"operator_id": 0,
"pay_terminal": 1,
"pay_config_id": 0,
"channel_payer_id": "",
"channel_pay_no": "",
"check_status": 1,
"channel_fee": 0.0
}
]

View File

@@ -1,712 +0,0 @@
[
{
"data": {
"total": 0,
"goodsCategoryList": [
{
"id": 2790683528350533,
"tenant_id": 2790683160709957,
"category_name": "槟榔",
"alias_name": "",
"pid": 0,
"business_name": "槟榔",
"tenant_goods_business_id": 2790683528317766,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350534,
"tenant_id": 2790683160709957,
"category_name": "槟榔",
"alias_name": "",
"pid": 2790683528350533,
"business_name": "槟榔",
"tenant_goods_business_id": 2790683528317766,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 1,
"is_warehousing": 1
},
{
"id": 2790683528350535,
"tenant_id": 2790683160709957,
"category_name": "器材",
"alias_name": "",
"pid": 0,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350536,
"tenant_id": 2790683160709957,
"category_name": "皮头",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350537,
"tenant_id": 2790683160709957,
"category_name": "球杆",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350538,
"tenant_id": 2790683160709957,
"category_name": "其他",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350539,
"tenant_id": 2790683160709957,
"category_name": "酒水",
"alias_name": "",
"pid": 0,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350540,
"tenant_id": 2790683160709957,
"category_name": "饮料",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350541,
"tenant_id": 2790683160709957,
"category_name": "酒水",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350542,
"tenant_id": 2790683160709957,
"category_name": "茶水",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350543,
"tenant_id": 2790683160709957,
"category_name": "咖啡",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350544,
"tenant_id": 2790683160709957,
"category_name": "加料",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793221553489733,
"tenant_id": 2790683160709957,
"category_name": "洋酒",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350545,
"tenant_id": 2790683160709957,
"category_name": "果盘",
"alias_name": "",
"pid": 0,
"business_name": "水果",
"tenant_goods_business_id": 2790683528317769,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792050275864453,
"tenant_id": 2790683160709957,
"category_name": "果盘",
"alias_name": "",
"pid": 2790683528350545,
"business_name": "水果",
"tenant_goods_business_id": 2790683528317769,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2791941988405125,
"tenant_id": 2790683160709957,
"category_name": "零食",
"alias_name": "",
"pid": 0,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2791948300259205,
"tenant_id": 2790683160709957,
"category_name": "零食",
"alias_name": "",
"pid": 2791941988405125,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793236829620037,
"tenant_id": 2790683160709957,
"category_name": "面",
"alias_name": "",
"pid": 2791941988405125,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2791942087561093,
"tenant_id": 2790683160709957,
"category_name": "雪糕",
"alias_name": "",
"pid": 0,
"business_name": "雪糕",
"tenant_goods_business_id": 2791931866402693,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792035069284229,
"tenant_id": 2790683160709957,
"category_name": "雪糕",
"alias_name": "",
"pid": 2791942087561093,
"business_name": "雪糕",
"tenant_goods_business_id": 2791931866402693,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2792062778003333,
"tenant_id": 2790683160709957,
"category_name": "香烟",
"alias_name": "",
"pid": 0,
"business_name": "香烟",
"tenant_goods_business_id": 2790683528317765,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792063209623429,
"tenant_id": 2790683160709957,
"category_name": "香烟",
"alias_name": "",
"pid": 2792062778003333,
"business_name": "香烟",
"tenant_goods_business_id": 2790683528317765,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 1,
"is_warehousing": 1
}
],
"sort": 1,
"is_warehousing": 1
},
{
"id": 2793217944864581,
"tenant_id": 2790683160709957,
"category_name": "其他",
"alias_name": "",
"pid": 0,
"business_name": "其他",
"tenant_goods_business_id": 2793217599407941,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2793218343257925,
"tenant_id": 2790683160709957,
"category_name": "其他2",
"alias_name": "",
"pid": 2793217944864581,
"business_name": "其他",
"tenant_goods_business_id": 2793217599407941,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793220945250117,
"tenant_id": 2790683160709957,
"category_name": "小吃",
"alias_name": "",
"pid": 0,
"business_name": "小吃",
"tenant_goods_business_id": 2793220268902213,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2793221283104581,
"tenant_id": 2790683160709957,
"category_name": "小吃",
"alias_name": "",
"pid": 2793220945250117,
"business_name": "小吃",
"tenant_goods_business_id": 2793220268902213,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
}
]
},
"code": 0
},
{
"data": {
"total": 0,
"goodsCategoryList": [
{
"id": 2790683528350533,
"tenant_id": 2790683160709957,
"category_name": "槟榔",
"alias_name": "",
"pid": 0,
"business_name": "槟榔",
"tenant_goods_business_id": 2790683528317766,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350534,
"tenant_id": 2790683160709957,
"category_name": "槟榔",
"alias_name": "",
"pid": 2790683528350533,
"business_name": "槟榔",
"tenant_goods_business_id": 2790683528317766,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 1,
"is_warehousing": 1
},
{
"id": 2790683528350535,
"tenant_id": 2790683160709957,
"category_name": "器材",
"alias_name": "",
"pid": 0,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350536,
"tenant_id": 2790683160709957,
"category_name": "皮头",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350537,
"tenant_id": 2790683160709957,
"category_name": "球杆",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350538,
"tenant_id": 2790683160709957,
"category_name": "其他",
"alias_name": "",
"pid": 2790683528350535,
"business_name": "器材",
"tenant_goods_business_id": 2790683528317767,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350539,
"tenant_id": 2790683160709957,
"category_name": "酒水",
"alias_name": "",
"pid": 0,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2790683528350540,
"tenant_id": 2790683160709957,
"category_name": "饮料",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350541,
"tenant_id": 2790683160709957,
"category_name": "酒水",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350542,
"tenant_id": 2790683160709957,
"category_name": "茶水",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350543,
"tenant_id": 2790683160709957,
"category_name": "咖啡",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350544,
"tenant_id": 2790683160709957,
"category_name": "加料",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793221553489733,
"tenant_id": 2790683160709957,
"category_name": "洋酒",
"alias_name": "",
"pid": 2790683528350539,
"business_name": "酒水",
"tenant_goods_business_id": 2790683528317768,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2790683528350545,
"tenant_id": 2790683160709957,
"category_name": "果盘",
"alias_name": "",
"pid": 0,
"business_name": "水果",
"tenant_goods_business_id": 2790683528317769,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792050275864453,
"tenant_id": 2790683160709957,
"category_name": "果盘",
"alias_name": "",
"pid": 2790683528350545,
"business_name": "水果",
"tenant_goods_business_id": 2790683528317769,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2791941988405125,
"tenant_id": 2790683160709957,
"category_name": "零食",
"alias_name": "",
"pid": 0,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2791948300259205,
"tenant_id": 2790683160709957,
"category_name": "零食",
"alias_name": "",
"pid": 2791941988405125,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793236829620037,
"tenant_id": 2790683160709957,
"category_name": "面",
"alias_name": "",
"pid": 2791941988405125,
"business_name": "零食",
"tenant_goods_business_id": 2791932037238661,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2791942087561093,
"tenant_id": 2790683160709957,
"category_name": "雪糕",
"alias_name": "",
"pid": 0,
"business_name": "雪糕",
"tenant_goods_business_id": 2791931866402693,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792035069284229,
"tenant_id": 2790683160709957,
"category_name": "雪糕",
"alias_name": "",
"pid": 2791942087561093,
"business_name": "雪糕",
"tenant_goods_business_id": 2791931866402693,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2792062778003333,
"tenant_id": 2790683160709957,
"category_name": "香烟",
"alias_name": "",
"pid": 0,
"business_name": "香烟",
"tenant_goods_business_id": 2790683528317765,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2792063209623429,
"tenant_id": 2790683160709957,
"category_name": "香烟",
"alias_name": "",
"pid": 2792062778003333,
"business_name": "香烟",
"tenant_goods_business_id": 2790683528317765,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 1,
"is_warehousing": 1
}
],
"sort": 1,
"is_warehousing": 1
},
{
"id": 2793217944864581,
"tenant_id": 2790683160709957,
"category_name": "其他",
"alias_name": "",
"pid": 0,
"business_name": "其他",
"tenant_goods_business_id": 2793217599407941,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2793218343257925,
"tenant_id": 2790683160709957,
"category_name": "其他2",
"alias_name": "",
"pid": 2793217944864581,
"business_name": "其他",
"tenant_goods_business_id": 2793217599407941,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
},
{
"id": 2793220945250117,
"tenant_id": 2790683160709957,
"category_name": "小吃",
"alias_name": "",
"pid": 0,
"business_name": "小吃",
"tenant_goods_business_id": 2793220268902213,
"open_salesman": 2,
"categoryBoxes": [
{
"id": 2793221283104581,
"tenant_id": 2790683160709957,
"category_name": "小吃",
"alias_name": "",
"pid": 2793220945250117,
"business_name": "小吃",
"tenant_goods_business_id": 2793220268902213,
"open_salesman": 2,
"categoryBoxes": [],
"sort": 0,
"is_warehousing": 1
}
],
"sort": 0,
"is_warehousing": 1
}
]
},
"code": 0
}
]

View File

@@ -1,646 +0,0 @@
[
{
"data": {
"total": 17,
"packageCouponList": [
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2939215004469573,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "早场特惠一小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-10-27 00:00:00",
"end_time": "2026-10-28 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 2,
"package_id": 1814707240811572,
"usable_count": 9999999,
"create_time": "2025-10-27 18:24:09",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2861343275830405,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "B区桌球一小时",
"table_area_id": "0",
"table_area_name": "B区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-09-02 00:00:00",
"end_time": "2026-09-03 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1370841337,
"usable_count": 9999999,
"create_time": "2025-09-02 18:08:56",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960521691013",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 3,
"id": 2836713896429317,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "午夜一小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-08-16 00:00:00",
"end_time": "2026-08-17 00:00:00",
"is_enabled": 2,
"is_delete": 0,
"type": 1,
"package_id": 1370841337,
"usable_count": 9999999,
"create_time": "2025-08-16 08:34:38",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2801876691340293,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "中八、斯诺克包厢两小时",
"table_area_id": "0",
"table_area_name": "VIP包厢",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-22 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1126976372,
"usable_count": 9999999,
"create_time": "2025-07-22 17:56:24",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961060364165",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 3,
"id": 2801875268668357,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "中八、斯诺克包厢两小时",
"table_area_id": "0",
"table_area_name": "VIP包厢",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-22 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 2,
"is_delete": 0,
"type": 1,
"package_id": 1126976372,
"usable_count": 9999999,
"create_time": "2025-07-22 17:54:57",
"creator_name": "管理员:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961060364165",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "0",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 3,
"id": 2800772613934149,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "午夜场一小时A区",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 2,
"is_delete": 0,
"type": 1,
"package_id": 1370841337,
"usable_count": 9999999,
"create_time": "2025-07-21 23:13:16",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798905767676933,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "中八、斯诺克包厢两小时",
"table_area_id": "0",
"table_area_name": "VIP包厢",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 2,
"package_id": 1812429097416714,
"usable_count": 9999999,
"create_time": "2025-07-20 15:34:13",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961060364165",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 3,
"id": 2798901295615045,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "新人特惠A区中八一小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 2,
"is_delete": 0,
"type": 2,
"package_id": 1814707240811572,
"usable_count": 9999999,
"create_time": "2025-07-20 15:29:40",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798898826300485,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "斯诺克两小时",
"table_area_id": "0",
"table_area_name": "斯诺克区",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 2,
"package_id": 1814983609169019,
"usable_count": 9999999,
"create_time": "2025-07-20 15:27:09",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961347968901",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798734170983493,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "助理教练竞技教学两小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1173128804,
"usable_count": 9999999,
"create_time": "2025-07-20 12:39:39",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798732571167749,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "全天斯诺克一小时",
"table_area_id": "0",
"table_area_name": "斯诺克区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-30 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1147633733,
"usable_count": 9999999,
"create_time": "2025-07-20 12:38:02",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961347968901",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798731703045189,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "KTV欢唱四小时",
"table_area_id": "0",
"table_area_name": "888",
"selling_price": 0.0,
"duration": 14400,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1137882866,
"usable_count": 9999999,
"create_time": "2025-07-20 12:37:09",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791961709907845",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "10:00:00",
"add_end_clock": "18:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798729978514501,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "全天A区中八两小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1130465371,
"usable_count": 9999999,
"create_time": "2025-07-20 12:35:24",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798728823213061,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "全天B区中八两小时",
"table_area_id": "0",
"table_area_name": "B区",
"selling_price": 0.0,
"duration": 7200,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1137872168,
"usable_count": 9999999,
"create_time": "2025-07-20 12:34:13",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960521691013",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798727423528005,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "全天A区中八一小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1128411555,
"usable_count": 9999999,
"create_time": "2025-07-20 12:32:48",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798723640069125,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "中八A区新人特惠一小时",
"table_area_id": "0",
"table_area_name": "A区",
"selling_price": 0.0,
"duration": 3600,
"start_time": "2025-07-20 00:00:00",
"end_time": "2025-12-31 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1203035334,
"usable_count": 9999999,
"create_time": "2025-07-20 12:28:57",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791960001957765",
"start_clock": "00:00:00",
"end_clock": "1.00:00:00",
"add_start_clock": "00:00:00",
"add_end_clock": "1.00:00:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
},
{
"site_name": "朗朗桌球",
"effective_status": 1,
"id": 2798713926290437,
"site_id": 2790685415443269,
"tenant_id": 2790683160709957,
"package_name": "麻将 、掼蛋包厢四小时",
"table_area_id": "0",
"table_area_name": "麻将房",
"selling_price": 0.0,
"duration": 14400,
"start_time": "2025-07-21 00:00:00",
"end_time": "2025-12-30 00:00:00",
"is_enabled": 1,
"is_delete": 0,
"type": 1,
"package_id": 1134269810,
"usable_count": 9999999,
"create_time": "2025-07-20 12:19:04",
"creator_name": "店长:郑丽珊",
"tenant_table_area_id": "0",
"table_area_id_list": "",
"tenant_table_area_id_list": "2791962314215301",
"start_clock": "10:00:00",
"end_clock": "1.02:00:00",
"add_start_clock": "10:00:00",
"add_end_clock": "23:59:00",
"date_info": "",
"date_type": 1,
"group_type": 1,
"usable_range": "",
"coupon_money": 0.0,
"area_tag_type": 1,
"system_group_type": 1,
"max_selectable_categories": 0,
"card_type_ids": "0"
}
]
},
"code": 0
},
{
"data": {
"total": 17,
"packageCouponList": []
},
"code": 0
}
]

View File

@@ -1,20 +0,0 @@
[
{
"data": {
"goodsStockA": 0,
"goodsStockB": 6252,
"goodsSaleNum": 210.29,
"stockSumMoney": 1461.28
},
"code": 0
},
{
"data": {
"goodsStockA": 0,
"goodsStockB": 6252,
"goodsSaleNum": 210.29,
"stockSumMoney": 1461.28
},
"code": 0
}
]

View File

@@ -0,0 +1,719 @@
# -*- coding: utf-8 -*-
"""ETL 任务测试的共用辅助模块,涵盖在线/离线模式所需的伪造数据、客户端与配置等工具函数。"""
from __future__ import annotations
import json
import os
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, List, Sequence, Tuple, Type
from config.settings import AppConfig
from database.connection import DatabaseConnection
from database.operations import DatabaseOperations as PgDBOperations
from tasks.assistant_abolish_task import AssistantAbolishTask
from tasks.assistants_task import AssistantsTask
from tasks.coupon_usage_task import CouponUsageTask
from tasks.inventory_change_task import InventoryChangeTask
from tasks.ledger_task import LedgerTask
from tasks.members_task import MembersTask
from tasks.orders_task import OrdersTask
from tasks.packages_task import PackagesDefTask
from tasks.payments_task import PaymentsTask
from tasks.products_task import ProductsTask
from tasks.refunds_task import RefundsTask
from tasks.table_discount_task import TableDiscountTask
from tasks.tables_task import TablesTask
from tasks.topups_task import TopupsTask
from utils.json_store import endpoint_to_filename
DEFAULT_STORE_ID = 2790685415443269
BASE_TS = "2025-01-01 10:00:00"
END_TS = "2025-01-01 12:00:00"
@dataclass(frozen=True)
class TaskSpec:
"""描述单个任务在测试中如何被驱动的元数据包含任务代码、API 路径、数据路径与样例记录。"""
code: str
task_cls: Type
endpoint: str
data_path: Tuple[str, ...]
sample_records: List[Dict]
@property
def archive_filename(self) -> str:
return endpoint_to_filename(self.endpoint)
def wrap_records(records: List[Dict], data_path: Sequence[str]):
"""按照 data_path 逐层包裹记录列表,使其结构与真实 API 返回体一致,方便离线回放。"""
payload = records
for key in reversed(data_path):
payload = {key: payload}
return payload
def create_test_config(mode: str, archive_dir: Path, temp_dir: Path) -> AppConfig:
"""构建一份适合测试的 AppConfig自动填充存储、日志、归档目录等参数并保证目录存在。"""
archive_dir = Path(archive_dir)
temp_dir = Path(temp_dir)
archive_dir.mkdir(parents=True, exist_ok=True)
temp_dir.mkdir(parents=True, exist_ok=True)
flow = "FULL" if str(mode or "").upper() == "ONLINE" else "INGEST_ONLY"
overrides = {
"app": {"store_id": DEFAULT_STORE_ID, "timezone": "Asia/Taipei"},
"db": {"dsn": "postgresql://user:pass@localhost:5432/etl_billiards_test"},
"api": {
"base_url": "https://api.example.com",
"token": "test-token",
"timeout_sec": 3,
"page_size": 50,
},
"pipeline": {
"flow": flow,
"fetch_root": str(temp_dir / "json_fetch"),
"ingest_source_dir": str(archive_dir),
},
"io": {
"export_root": str(temp_dir / "export"),
"log_root": str(temp_dir / "logs"),
},
}
return AppConfig.load(overrides)
def dump_offline_payload(spec: TaskSpec, archive_dir: Path) -> Path:
"""将 TaskSpec 的样例数据写入指定归档目录,供离线测试回放使用,并返回生成文件的完整路径。"""
archive_dir = Path(archive_dir)
payload = wrap_records(spec.sample_records, spec.data_path)
file_path = archive_dir / spec.archive_filename
with file_path.open("w", encoding="utf-8") as fp:
json.dump(payload, fp, ensure_ascii=False)
return file_path
class FakeCursor:
"""极简游标桩对象,记录 SQL/参数并支持上下文管理,供 FakeDBOperations 与 SCD2Handler 使用。"""
def __init__(self, recorder: List[Dict]):
self.recorder = recorder
# pylint: disable=unused-argument
def execute(self, sql: str, params=None):
self.recorder.append({"sql": sql.strip(), "params": params})
def fetchone(self):
return None
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
class FakeConnection:
"""仿 psycopg 连接对象,仅满足 SCD2Handler 对 cursor 的最小需求,并缓存执行过的语句。"""
def __init__(self):
self.statements: List[Dict] = []
def cursor(self):
return FakeCursor(self.statements)
class FakeDBOperations:
"""拦截并记录批量 upsert/事务操作,避免触碰真实数据库,同时提供 commit/rollback 计数。"""
def __init__(self):
self.upserts: List[Dict] = []
self.executes: List[Dict] = []
self.commits = 0
self.rollbacks = 0
self.conn = FakeConnection()
# Pre-seeded query results (FIFO) to let tests control DB-returned rows
self.query_results: List[List[Dict]] = []
def batch_upsert_with_returning(self, sql: str, rows: List[Dict], page_size: int = 1000):
self.upserts.append(
{
"sql": sql.strip(),
"count": len(rows),
"page_size": page_size,
"rows": [dict(row) for row in rows],
}
)
return len(rows), 0
def batch_execute(self, sql: str, rows: List[Dict], page_size: int = 1000):
self.executes.append(
{
"sql": sql.strip(),
"count": len(rows),
"page_size": page_size,
"rows": [dict(row) for row in rows],
}
)
def execute(self, sql: str, params=None):
self.executes.append({"sql": sql.strip(), "params": params})
def query(self, sql: str, params=None):
self.executes.append({"sql": sql.strip(), "params": params, "type": "query"})
if self.query_results:
return self.query_results.pop(0)
return []
def cursor(self):
return self.conn.cursor()
def commit(self):
self.commits += 1
def rollback(self):
self.rollbacks += 1
class FakeAPIClient:
"""在线模式使用的伪 API Client直接返回预置的内存数据并记录调用以确保任务参数正确传递。"""
def __init__(self, data_map: Dict[str, List[Dict]]):
self.data_map = data_map
self.calls: List[Dict] = []
# pylint: disable=unused-argument
def iter_paginated(
self,
endpoint: str,
params=None,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: Tuple[str, ...] = (),
list_key: str | None = None,
):
self.calls.append({"endpoint": endpoint, "params": params})
if endpoint not in self.data_map:
raise AssertionError(f"Missing fixture for endpoint {endpoint}")
records = list(self.data_map[endpoint])
yield 1, records, dict(params or {}), {"data": records}
def get_paginated(self, endpoint: str, params=None, **kwargs):
records = []
pages = []
for page_no, page_records, req, resp in self.iter_paginated(endpoint, params, **kwargs):
records.extend(page_records)
pages.append({"page": page_no, "request": req, "response": resp})
return records, pages
def get_source_hint(self, endpoint: str) -> str | None:
return None
class OfflineAPIClient:
"""离线模式专用 API Client根据 endpoint 读取归档 JSON、套入 data_path 并回放列表数据。"""
def __init__(self, file_map: Dict[str, Path]):
self.file_map = {k: Path(v) for k, v in file_map.items()}
self.calls: List[Dict] = []
# pylint: disable=unused-argument
def iter_paginated(
self,
endpoint: str,
params=None,
page_size: int = 200,
page_field: str = "page",
size_field: str = "limit",
data_path: Tuple[str, ...] = (),
list_key: str | None = None,
):
self.calls.append({"endpoint": endpoint, "params": params})
if endpoint not in self.file_map:
raise AssertionError(f"Missing archive for endpoint {endpoint}")
with self.file_map[endpoint].open("r", encoding="utf-8") as fp:
payload = json.load(fp)
data = payload
for key in data_path:
if isinstance(data, dict):
data = data.get(key, [])
if list_key and isinstance(data, dict):
data = data.get(list_key, [])
if not isinstance(data, list):
data = []
total = len(data)
start = 0
page = 1
while start < total or (start == 0 and total == 0):
chunk = data[start : start + page_size]
if not chunk and total != 0:
break
yield page, list(chunk), dict(params or {}), payload
if len(chunk) < page_size:
break
start += page_size
page += 1
def get_paginated(self, endpoint: str, params=None, **kwargs):
records = []
pages = []
for page_no, page_records, req, resp in self.iter_paginated(endpoint, params, **kwargs):
records.extend(page_records)
pages.append({"page": page_no, "request": req, "response": resp})
return records, pages
def get_source_hint(self, endpoint: str) -> str | None:
if endpoint not in self.file_map:
return None
return str(self.file_map[endpoint])
class RealDBOperationsAdapter:
"""连接真实 PostgreSQL 的适配器,为任务提供 batch_upsert + 事务能力。"""
def __init__(self, dsn: str):
self._conn = DatabaseConnection(dsn)
self._ops = PgDBOperations(self._conn)
# SCD2Handler 会访问 db.conn.cursor(),因此暴露底层连接
self.conn = self._conn.conn
def batch_upsert_with_returning(self, sql: str, rows: List[Dict], page_size: int = 1000):
return self._ops.batch_upsert_with_returning(sql, rows, page_size=page_size)
def batch_execute(self, sql: str, rows: List[Dict], page_size: int = 1000):
return self._ops.batch_execute(sql, rows, page_size=page_size)
def commit(self):
self._conn.commit()
def rollback(self):
self._conn.rollback()
def close(self):
self._conn.close()
@contextmanager
def get_db_operations():
"""
测试专用的 DB 操作上下文:
- 若设置 TEST_DB_DSN则连接真实 PostgreSQL
- 否则回退到 FakeDBOperations内存桩
"""
dsn = os.environ.get("TEST_DB_DSN")
if dsn:
adapter = RealDBOperationsAdapter(dsn)
try:
yield adapter
finally:
adapter.close()
else:
fake = FakeDBOperations()
yield fake
TASK_SPECS: List[TaskSpec] = [
TaskSpec(
code="PRODUCTS",
task_cls=ProductsTask,
endpoint="/TenantGoods/QueryTenantGoods",
data_path=("data", "tenantGoodsList"),
sample_records=[
{
"siteGoodsId": 101,
"tenantGoodsId": 101,
"goodsName": "测试球杆",
"goodsCategoryId": 201,
"categoryName": "器材",
"goodsCategorySecondId": 202,
"goodsUnit": "",
"costPrice": "100.00",
"goodsPrice": "150.00",
"goodsState": "ON",
"supplierId": 20,
"barcode": "PRD001",
"isCombo": False,
"createTime": BASE_TS,
"updateTime": END_TS,
}
],
),
TaskSpec(
code="TABLES",
task_cls=TablesTask,
endpoint="/Table/GetSiteTables",
data_path=("data", "siteTables"),
sample_records=[
{
"id": 301,
"site_id": 30,
"site_table_area_id": 40,
"areaName": "大厅",
"table_name": "1号桌",
"table_price": "50.00",
"table_status": "FREE",
"tableStatusName": "空闲",
"light_status": "OFF",
"is_rest_area": False,
"show_status": True,
"virtual_table": False,
"charge_free": False,
"only_allow_groupon": False,
"is_online_reservation": True,
"createTime": BASE_TS,
}
],
),
TaskSpec(
code="MEMBERS",
task_cls=MembersTask,
endpoint="/MemberProfile/GetTenantMemberList",
data_path=("data", "tenantMemberInfos"),
sample_records=[
{
"memberId": 401,
"memberName": "张三",
"phone": "13800000000",
"balance": "88.88",
"status": "ACTIVE",
"registerTime": BASE_TS,
}
],
),
TaskSpec(
code="ASSISTANTS",
task_cls=AssistantsTask,
endpoint="/PersonnelManagement/SearchAssistantInfo",
data_path=("data", "assistantInfos"),
sample_records=[
{
"id": 501,
"assistant_no": "AS001",
"nickname": "小李",
"real_name": "李雷",
"gender": "M",
"mobile": "13900000000",
"level": "A",
"team_id": 10,
"team_name": "先锋队",
"assistant_status": "ON",
"work_status": "BUSY",
"entry_time": BASE_TS,
"resign_time": END_TS,
"start_time": BASE_TS,
"end_time": END_TS,
"create_time": BASE_TS,
"update_time": END_TS,
"system_role_id": 1,
"online_status": "ONLINE",
"allow_cx": True,
"charge_way": "TIME",
"pd_unit_price": "30.00",
"cx_unit_price": "20.00",
"is_guaranteed": True,
"is_team_leader": False,
"serial_number": "SN001",
"show_sort": 1,
"is_delete": False,
}
],
),
TaskSpec(
code="PACKAGES_DEF",
task_cls=PackagesDefTask,
endpoint="/PackageCoupon/QueryPackageCouponList",
data_path=("data", "packageCouponList"),
sample_records=[
{
"id": 601,
"package_id": "PKG001",
"package_name": "白天特惠",
"table_area_id": 70,
"table_area_name": "大厅",
"selling_price": "199.00",
"duration": 120,
"start_time": BASE_TS,
"end_time": END_TS,
"type": "Groupon",
"is_enabled": True,
"is_delete": False,
"usable_count": 3,
"creator_name": "系统",
"date_type": "WEEKDAY",
"group_type": "DINE_IN",
"coupon_money": "30.00",
"area_tag_type": "VIP",
"system_group_type": "BASIC",
"card_type_ids": "1,2,3",
}
],
),
TaskSpec(
code="ORDERS",
task_cls=OrdersTask,
endpoint="/Site/GetAllOrderSettleList",
data_path=("data", "settleList"),
sample_records=[
{
"orderId": 701,
"orderNo": "ORD001",
"memberId": 401,
"tableId": 301,
"orderTime": BASE_TS,
"endTime": END_TS,
"totalAmount": "300.00",
"discountAmount": "20.00",
"finalAmount": "280.00",
"payStatus": "PAID",
"orderStatus": "CLOSED",
"remark": "测试订单",
}
],
),
TaskSpec(
code="PAYMENTS",
task_cls=PaymentsTask,
endpoint="/PayLog/GetPayLogListPage",
data_path=("data",),
sample_records=[
{
"payId": 801,
"orderId": 701,
"payTime": END_TS,
"payAmount": "280.00",
"payType": "CARD",
"payStatus": "SUCCESS",
"remark": "测试支付",
}
],
),
TaskSpec(
code="REFUNDS",
task_cls=RefundsTask,
endpoint="/Order/GetRefundPayLogList",
data_path=("data",),
sample_records=[
{
"id": 901,
"site_id": 1,
"tenant_id": 2,
"pay_amount": "100.00",
"pay_status": "SUCCESS",
"pay_time": END_TS,
"create_time": END_TS,
"relate_type": "ORDER",
"relate_id": 701,
"payment_method": "CARD",
"refund_amount": "20.00",
"action_type": "PARTIAL",
"pay_terminal": "POS",
"operator_id": 11,
"channel_pay_no": "CH001",
"channel_fee": "1.00",
"is_delete": False,
"member_id": 401,
"member_card_id": 501,
}
],
),
TaskSpec(
code="COUPON_USAGE",
task_cls=CouponUsageTask,
endpoint="/Promotion/GetOfflineCouponConsumePageList",
data_path=("data",),
sample_records=[
{
"id": 1001,
"coupon_code": "CP001",
"coupon_channel": "MEITUAN",
"coupon_name": "双人券",
"sale_price": "50.00",
"coupon_money": "30.00",
"coupon_free_time": 60,
"use_status": "USED",
"create_time": BASE_TS,
"consume_time": END_TS,
"operator_id": 11,
"operator_name": "操作员",
"table_id": 301,
"site_order_id": 701,
"group_package_id": 601,
"coupon_remark": "备注",
"deal_id": "DEAL001",
"certificate_id": "CERT001",
"verify_id": "VERIFY001",
"is_delete": False,
}
],
),
TaskSpec(
code="INVENTORY_CHANGE",
task_cls=InventoryChangeTask,
endpoint="/GoodsStockManage/QueryGoodsOutboundReceipt",
data_path=("data", "queryDeliveryRecordsList"),
sample_records=[
{
"siteGoodsStockId": 1101,
"siteGoodsId": 101,
"stockType": "OUT",
"goodsName": "测试球杆",
"createTime": END_TS,
"startNum": 10,
"endNum": 8,
"changeNum": -2,
"unit": "",
"price": "120.00",
"operatorName": "仓管",
"remark": "测试出库",
"goodsCategoryId": 201,
"goodsSecondCategoryId": 202,
}
],
),
TaskSpec(
code="TOPUPS",
task_cls=TopupsTask,
endpoint="/Site/GetRechargeSettleList",
data_path=("data", "settleList"),
sample_records=[
{
"id": 1201,
"memberId": 401,
"memberName": "张三",
"memberPhone": "13800000000",
"tenantMemberCardId": 1301,
"memberCardTypeName": "金卡",
"payAmount": "500.00",
"consumeMoney": "100.00",
"settleStatus": "DONE",
"settleType": "AUTO",
"settleName": "日结",
"settleRelateId": 1501,
"payTime": BASE_TS,
"createTime": END_TS,
"operatorId": 11,
"operatorName": "收银员",
"paymentMethod": "CASH",
"refundAmount": "0",
"cashAmount": "500.00",
"cardAmount": "0",
"balanceAmount": "0",
"onlineAmount": "0",
"roundingAmount": "0",
"adjustAmount": "0",
"goodsMoney": "0",
"tableChargeMoney": "0",
"serviceMoney": "0",
"couponAmount": "0",
"orderRemark": "首次充值",
}
],
),
TaskSpec(
code="TABLE_DISCOUNT",
task_cls=TableDiscountTask,
endpoint="/Site/GetTaiFeeAdjustList",
data_path=("data", "taiFeeAdjustInfos"),
sample_records=[
{
"id": 1301,
"adjust_type": "DISCOUNT",
"applicant_id": 11,
"applicant_name": "店长",
"operator_id": 22,
"operator_name": "值班",
"ledger_amount": "50.00",
"ledger_count": 2,
"ledger_name": "调价",
"ledger_status": "APPROVED",
"order_settle_id": 7010,
"order_trade_no": 8001,
"site_table_id": 301,
"create_time": END_TS,
"is_delete": False,
"tableProfile": {
"id": 301,
"site_table_area_id": 40,
"site_table_area_name": "大厅",
},
}
],
),
TaskSpec(
code="ASSISTANT_ABOLISH",
task_cls=AssistantAbolishTask,
endpoint="/AssistantPerformance/GetAbolitionAssistant",
data_path=("data", "abolitionAssistants"),
sample_records=[
{
"id": 1401,
"tableId": 301,
"tableName": "1号桌",
"tableAreaId": 40,
"tableArea": "大厅",
"assistantOn": "AS001",
"assistantName": "小李",
"pdChargeMinutes": 30,
"assistantAbolishAmount": "15.00",
"createTime": END_TS,
"trashReason": "测试",
}
],
),
TaskSpec(
code="LEDGER",
task_cls=LedgerTask,
endpoint="/AssistantPerformance/GetOrderAssistantDetails",
data_path=("data", "orderAssistantDetails"),
sample_records=[
{
"id": 1501,
"assistantNo": "AS001",
"assistantName": "小李",
"nickname": "",
"levelName": "L1",
"tableName": "1号桌",
"ledger_unit_price": "30.00",
"ledger_count": 2,
"ledger_amount": "60.00",
"projected_income": "80.00",
"service_money": "5.00",
"member_discount_amount": "2.00",
"manual_discount_amount": "1.00",
"coupon_deduct_money": "3.00",
"order_trade_no": 8001,
"order_settle_id": 7010,
"operator_id": 22,
"operator_name": "值班",
"assistant_team_id": 10,
"assistant_level": "A",
"site_table_id": 301,
"order_assistant_id": 1601,
"site_assistant_id": 501,
"user_id": 5010,
"ledger_start_time": BASE_TS,
"ledger_end_time": END_TS,
"start_use_time": BASE_TS,
"last_use_time": END_TS,
"income_seconds": 3600,
"real_use_seconds": 3300,
"is_trash": False,
"trash_reason": "",
"is_confirm": True,
"ledger_status": "CLOSED",
"create_time": END_TS,
}
],
),
]

View File

@@ -0,0 +1,39 @@
# -*- coding: utf-8 -*-
"""离线模式任务测试,通过回放归档 JSON 来验证 T+L 链路可用。"""
import logging
from pathlib import Path
import pytest
from .task_test_utils import (
TASK_SPECS,
OfflineAPIClient,
create_test_config,
dump_offline_payload,
get_db_operations,
)
@pytest.mark.parametrize("spec", TASK_SPECS, ids=lambda spec: spec.code)
def test_task_offline_mode(spec, tmp_path):
"""确保每个任务都能读取归档 JSON 并完成 Transform + Load 操作。"""
archive_dir = tmp_path / "archive"
temp_dir = tmp_path / "tmp"
archive_dir.mkdir()
temp_dir.mkdir()
file_path = dump_offline_payload(spec, archive_dir)
config = create_test_config("OFFLINE", archive_dir, temp_dir)
offline_api = OfflineAPIClient({spec.endpoint: Path(file_path)})
logger = logging.getLogger(f"test_offline_{spec.code.lower()}")
with get_db_operations() as db_ops:
task = spec.task_cls(config, db_ops, offline_api, logger)
result = task.execute()
assert result["status"] == "SUCCESS"
assert result["counts"]["fetched"] == len(spec.sample_records)
assert result["counts"]["inserted"] == len(spec.sample_records)
if hasattr(db_ops, "commits"):
assert db_ops.commits == 1
assert db_ops.rollbacks == 0

View File

@@ -0,0 +1,32 @@
# -*- coding: utf-8 -*-
"""在线模式下的端到端任务测试,验证所有任务在模拟 API 下能顺利执行。"""
import logging
import pytest
from .task_test_utils import (
TASK_SPECS,
FakeAPIClient,
create_test_config,
get_db_operations,
)
@pytest.mark.parametrize("spec", TASK_SPECS, ids=lambda spec: spec.code)
def test_task_online_mode(spec, tmp_path):
"""针对每个 TaskSpec 验证:模拟 API 数据下依旧能完整跑完 ETL并正确统计。"""
archive_dir = tmp_path / "archive"
temp_dir = tmp_path / "tmp"
config = create_test_config("ONLINE", archive_dir, temp_dir)
fake_api = FakeAPIClient({spec.endpoint: spec.sample_records})
logger = logging.getLogger(f"test_online_{spec.code.lower()}")
with get_db_operations() as db_ops:
task = spec.task_cls(config, db_ops, fake_api, logger)
result = task.execute()
assert result["status"] == "SUCCESS"
assert result["counts"]["fetched"] == len(spec.sample_records)
assert result["counts"]["inserted"] == len(spec.sample_records)
if hasattr(db_ops, "commits"):
assert db_ops.commits == 1
assert db_ops.rollbacks == 0

View File

@@ -0,0 +1,59 @@
# -*- coding: utf-8 -*-
"""验证 14 个任务的 E/T/L 分阶段调用FakeDB/FakeAPI不访问真实接口或数据库"""
import logging
import sys
from pathlib import Path
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import pytest
PROJECT_ROOT = Path(__file__).resolve().parents[2]
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
from tasks.base_task import TaskContext
from tests.unit.task_test_utils import (
TASK_SPECS,
create_test_config,
get_db_operations,
FakeAPIClient,
)
def _build_context(store_id: int) -> TaskContext:
now = datetime.now(ZoneInfo("Asia/Taipei"))
return TaskContext(
store_id=store_id,
window_start=now - timedelta(minutes=30),
window_end=now,
window_minutes=30,
cursor=None,
)
@pytest.mark.parametrize("spec", TASK_SPECS)
def test_etl_stage_flow(spec, tmp_path):
"""对每个任务,单独调用 transform/load验证 counts 结构与 FakeDB 写入。"""
config = create_test_config("ONLINE", tmp_path / "archive", tmp_path / "temp")
api = FakeAPIClient({spec.endpoint: spec.sample_records})
logger = logging.getLogger(f"test_{spec.code.lower()}")
task_cls = spec.task_cls
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logger)
ctx = _build_context(config.get("app.store_id"))
# 跳过 extract直接验证 transform + load
extracted = {"records": spec.sample_records}
transformed = task.transform(extracted, ctx)
counts = task.load(transformed, ctx)
assert set(counts.keys()) == {"fetched", "inserted", "updated", "skipped", "errors"}
assert counts["fetched"] == len(spec.sample_records)
assert counts["errors"] == 0
# FakeDB 记录upserts/executes至少有一条
upserts = getattr(db_ops, "upserts", [])
executes = getattr(db_ops, "executes", [])
assert upserts or executes, "expected db operations to be recorded"

View File

@@ -0,0 +1,160 @@
# -*- coding: utf-8 -*-
"""Unit tests for the new ODS ingestion tasks."""
import logging
import os
import sys
from pathlib import Path
# Ensure project root is resolvable when running tests in isolation
PROJECT_ROOT = Path(__file__).resolve().parents[2]
if str(PROJECT_ROOT) not in sys.path:
sys.path.insert(0, str(PROJECT_ROOT))
os.environ.setdefault("ETL_SKIP_DOTENV", "1")
from tasks.ods_tasks import ODS_TASK_CLASSES
from .task_test_utils import create_test_config, get_db_operations, FakeAPIClient
def _build_config(tmp_path):
archive_dir = tmp_path / "archive"
temp_dir = tmp_path / "temp"
return create_test_config("ONLINE", archive_dir, temp_dir)
def test_ods_assistant_accounts_ingest(tmp_path):
"""Ensure ODS_ASSISTANT_ACCOUNTS task stores raw payload with record_index dedup keys."""
config = _build_config(tmp_path)
sample = [
{
"id": 5001,
"assistant_no": "A01",
"nickname": "小张",
}
]
api = FakeAPIClient({"/PersonnelManagement/SearchAssistantInfo": sample})
task_cls = ODS_TASK_CLASSES["ODS_ASSISTANT_ACCOUNTS"]
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_assistant_accounts"))
result = task.execute()
assert result["status"] == "SUCCESS"
assert result["counts"]["fetched"] == 1
assert db_ops.commits == 1
row = db_ops.upserts[0]["rows"][0]
assert row["id"] == 5001
assert row["record_index"] == 0
assert row["source_file"] is None or row["source_file"] == ""
assert '"id": 5001' in row["payload"]
def test_ods_inventory_change_ingest(tmp_path):
"""Ensure ODS_INVENTORY_CHANGE task stores raw payload with record_index dedup keys."""
config = _build_config(tmp_path)
sample = [
{
"siteGoodsStockId": 123456,
"stockType": 1,
"goodsName": "测试商品",
}
]
api = FakeAPIClient({"/GoodsStockManage/QueryGoodsOutboundReceipt": sample})
task_cls = ODS_TASK_CLASSES["ODS_INVENTORY_CHANGE"]
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_inventory_change"))
result = task.execute()
assert result["status"] == "SUCCESS"
assert result["counts"]["fetched"] == 1
assert db_ops.commits == 1
row = db_ops.upserts[0]["rows"][0]
assert row["sitegoodsstockid"] == 123456
assert row["record_index"] == 0
assert '"siteGoodsStockId": 123456' in row["payload"]
def test_ods_member_profiles_ingest(tmp_path):
"""Ensure ODS_MEMBER task stores tenantMemberInfos raw JSON."""
config = _build_config(tmp_path)
sample = [{"tenantMemberInfos": [{"id": 101, "mobile": "13800000000"}]}]
api = FakeAPIClient({"/MemberProfile/GetTenantMemberList": sample})
task_cls = ODS_TASK_CLASSES["ODS_MEMBER"]
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_member"))
result = task.execute()
assert result["status"] == "SUCCESS"
row = db_ops.upserts[0]["rows"][0]
assert row["record_index"] == 0
assert '"id": 101' in row["payload"]
def test_ods_payment_ingest(tmp_path):
"""Ensure ODS_PAYMENT task stores payment_transactions raw JSON."""
config = _build_config(tmp_path)
sample = [{"payId": 901, "payAmount": "100.00"}]
api = FakeAPIClient({"/PayLog/GetPayLogListPage": sample})
task_cls = ODS_TASK_CLASSES["ODS_PAYMENT"]
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_payment"))
result = task.execute()
assert result["status"] == "SUCCESS"
row = db_ops.upserts[0]["rows"][0]
assert row["record_index"] == 0
assert '"payId": 901' in row["payload"]
def test_ods_settlement_records_ingest(tmp_path):
"""Ensure ODS_ORDER_SETTLE task stores settleList raw JSON."""
config = _build_config(tmp_path)
sample = [{"data": {"settleList": [{"id": 701, "orderTradeNo": 8001}]}}]
api = FakeAPIClient({"/Site/GetAllOrderSettleList": sample})
task_cls = ODS_TASK_CLASSES["ODS_ORDER_SETTLE"]
with get_db_operations() as db_ops:
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_order_settle"))
result = task.execute()
assert result["status"] == "SUCCESS"
row = db_ops.upserts[0]["rows"][0]
assert row["record_index"] == 0
assert '"orderTradeNo": 8001' in row["payload"]
def test_ods_settlement_ticket_by_payment_relate_ids(tmp_path):
"""Ensure settlement tickets are fetched per payment relate_id and skip existing ones."""
config = _build_config(tmp_path)
ticket_payload = {"data": {"data": {"orderSettleId": 9001, "orderSettleNumber": "T001"}}}
api = FakeAPIClient({"/Order/GetOrderSettleTicketNew": [ticket_payload]})
task_cls = ODS_TASK_CLASSES["ODS_SETTLEMENT_TICKET"]
with get_db_operations() as db_ops:
# First query: existing ticket ids; Second query: payment relate_ids
db_ops.query_results = [
[{"order_settle_id": 9002}],
[
{"order_settle_id": 9001},
{"order_settle_id": 9002},
{"order_settle_id": None},
],
]
task = task_cls(config, db_ops, api, logging.getLogger("test_ods_settlement_ticket"))
result = task.execute()
assert result["status"] == "SUCCESS"
counts = result["counts"]
assert counts["fetched"] == 1
assert counts["inserted"] == 1
assert counts["updated"] == 0
assert counts["skipped"] == 0
assert '"orderSettleId": 9001' in db_ops.upserts[0]["rows"][0]["payload"]
assert any(
call["endpoint"] == "/Order/GetOrderSettleTicketNew"
and call.get("params", {}).get("orderSettleId") == 9001
for call in api.calls
)

Some files were not shown because too many files have changed in this diff Show More