Compare commits
2 Commits
13d853c3f5
...
c3749474c6
| Author | SHA1 | Date | |
|---|---|---|---|
| c3749474c6 | |||
| 7f87421678 |
48
.gitignore
vendored
Normal file
48
.gitignore
vendored
Normal file
@@ -0,0 +1,48 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# 虚拟环境
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# 日志和导出
|
||||
*.log
|
||||
*.jsonl
|
||||
export/
|
||||
logs/
|
||||
|
||||
# 环境变量
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# 测试
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
38
etl_billiards/.env
Normal file
38
etl_billiards/.env
Normal file
@@ -0,0 +1,38 @@
|
||||
# 数据库配置
|
||||
PG_DSN=postgresql://local-Python:Neo-local-1991125@localhost:5432/LLZQ
|
||||
# PG_HOST=localhost
|
||||
# PG_PORT=5432
|
||||
# PG_NAME=LLZQ
|
||||
# PG_USER=local-Python
|
||||
# PG_PASSWORD=your_password_here
|
||||
|
||||
# API配置(抓取时，非球的一些配置)
|
||||
API_BASE=https://api.example.com # 非球URL前缀
|
||||
API_TOKEN=your_token_here # 登录Token
|
||||
# API_TIMEOUT=20
|
||||
# API_PAGE_SIZE=200
|
||||
# API_RETRY_MAX=3
|
||||
|
||||
# 应用配置
|
||||
STORE_ID=2790685415443269
|
||||
# TIMEZONE=Asia/Taipei
|
||||
# SCHEMA_OLTP=billiards
|
||||
# SCHEMA_ETL=etl_admin
|
||||
|
||||
# 路径配置
|
||||
EXPORT_ROOT=r"D:\LLZQ\DB\export",
|
||||
LOG_ROOT=r"D:\LLZQ\DB\logs",
|
||||
|
||||
# ETL配置
|
||||
OVERLAP_SECONDS=120 # 为了防止边界遗漏，会往前“回拨”一点的冗余秒数
|
||||
WINDOW_BUSY_MIN=30
|
||||
WINDOW_IDLE_MIN=180
|
||||
IDLE_START=04:00
|
||||
IDLE_END=16:00
|
||||
ALLOW_EMPTY_ADVANCE=true
|
||||
|
||||
# 清洗配置
|
||||
LOG_UNKNOWN_FIELDS=true
|
||||
HASH_ALGO=sha1
|
||||
STRICT_NUMERIC=true
|
||||
ROUND_MONEY_SCALE=2
|
||||
38
etl_billiards/.env.example
Normal file
38
etl_billiards/.env.example
Normal file
@@ -0,0 +1,38 @@
|
||||
# 数据库配置
|
||||
PG_DSN=postgresql://user:password@localhost:5432/LLZQ
|
||||
PG_HOST=localhost
|
||||
PG_PORT=5432
|
||||
PG_NAME=LLZQ
|
||||
PG_USER=local-Python
|
||||
PG_PASSWORD=your_password_here
|
||||
|
||||
# API配置
|
||||
API_BASE=https://api.example.com
|
||||
API_TOKEN=your_token_here
|
||||
API_TIMEOUT=20
|
||||
API_PAGE_SIZE=200
|
||||
API_RETRY_MAX=3
|
||||
|
||||
# 应用配置
|
||||
STORE_ID=2790685415443269
|
||||
TIMEZONE=Asia/Taipei
|
||||
SCHEMA_OLTP=billiards
|
||||
SCHEMA_ETL=etl_admin
|
||||
|
||||
# 路径配置
|
||||
EXPORT_ROOT=/path/to/export
|
||||
LOG_ROOT=/path/to/logs
|
||||
|
||||
# ETL配置
|
||||
OVERLAP_SECONDS=120
|
||||
WINDOW_BUSY_MIN=30
|
||||
WINDOW_IDLE_MIN=180
|
||||
IDLE_START=04:00
|
||||
IDLE_END=16:00
|
||||
ALLOW_EMPTY_ADVANCE=true
|
||||
|
||||
# 清洗配置
|
||||
LOG_UNKNOWN_FIELDS=true
|
||||
HASH_ALGO=sha1
|
||||
STRICT_NUMERIC=true
|
||||
ROUND_MONEY_SCALE=2
|
||||
7
etl_billiards/0.py
Normal file
7
etl_billiards/0.py
Normal file
@@ -0,0 +1,7 @@
|
||||
# -*- coding: UTF-8 -*-
|
||||
|
||||
# Filename : helloworld.py
|
||||
# author by : www.runoob.com
|
||||
|
||||
# 该实例输出 Hello World!
|
||||
print('Hello World!')
|
||||
615
etl_billiards/README.md
Normal file
615
etl_billiards/README.md
Normal file
@@ -0,0 +1,615 @@
|
||||
# 台球场 ETL 系统（模块化版本）合并文档
|
||||
|
||||
本文为原多份文档（如 `INDEX.md`、`QUICK_START.md`、`ARCHITECTURE.md`、`MIGRATION_GUIDE.md`、`PROJECT_STRUCTURE.md`、`README.md` 等）的合并版，只保留与**当前项目本身**相关的内容：项目说明、目录结构、架构设计、数据与控制流程、迁移与扩展指南等，不包含修改历史和重构过程描述。
|
||||
|
||||
---
|
||||
|
||||
## 1. 项目概述
|
||||
|
||||
台球场 ETL 系统是一个面向门店业务的专业 ETL 工程项目，用于从外部业务 API 拉取订单、支付、会员等数据，经过解析、校验、SCD2 处理、质量检查后写入 PostgreSQL 数据库，并支持增量同步和任务运行追踪。
|
||||
|
||||
系统采用模块化、分层架构设计，核心特性包括：
|
||||
|
||||
- 模块化目录结构（配置、数据库、API、模型、加载器、SCD2、质量检查、编排、任务、CLI、工具、测试等分层清晰）。
|
||||
- 完整的配置管理：默认值 + 环境变量 + CLI 参数多层覆盖。
|
||||
- 可复用的数据库访问层（连接管理、批量 Upsert 封装）。
|
||||
- 支持重试与分页的 API 客户端。
|
||||
- 类型安全的数据解析与校验模块。
|
||||
- SCD2 维度历史管理。
|
||||
- 数据质量检查（例如余额一致性检查）。
|
||||
- 任务编排层统一调度、游标管理与运行追踪。
|
||||
- 命令行入口统一管理任务执行，支持筛选任务、Dry-run 等模式。
|
||||
|
||||
---
|
||||
|
||||
## 2. 快速开始
|
||||
|
||||
### 2.1 环境准备
|
||||
|
||||
- Python 版本：建议 3.10+
|
||||
- 数据库：PostgreSQL
|
||||
- 操作系统：Windows / Linux / macOS 均可
|
||||
|
||||
```bash
|
||||
# 克隆/下载代码后进入项目目录
|
||||
cd etl_billiards/
|
||||
ls -la
|
||||
```
|
||||
|
||||
你会看到下述目录结构的顶层部分（详细见第 4 章）：
|
||||
|
||||
- `config/` - 配置管理
|
||||
- `database/` - 数据库访问
|
||||
- `api/` - API 客户端
|
||||
- `tasks/` - ETL 任务实现
|
||||
- `cli/` - 命令行入口
|
||||
- `docs/` - 技术文档
|
||||
|
||||
### 2.2 安装依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
主要依赖示例（按实际 `requirements.txt` 为准）：
|
||||
|
||||
- `psycopg2-binary`：PostgreSQL 驱动
|
||||
- `requests`：HTTP 客户端
|
||||
- `python-dateutil`：时间处理
|
||||
- `tzdata`：时区数据
|
||||
|
||||
### 2.3 配置环境变量
|
||||
|
||||
复制并修改环境变量模板：
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# 使用你习惯的编辑器修改 .env
|
||||
```
|
||||
|
||||
`.env` 示例（最小配置）：
|
||||
|
||||
```bash
|
||||
# 数据库
|
||||
PG_DSN=postgresql://user:password@localhost:5432/LLZQ
|
||||
|
||||
# API
|
||||
API_BASE=https://api.example.com
|
||||
API_TOKEN=your_token_here
|
||||
|
||||
# 门店/应用
|
||||
STORE_ID=2790685415443269
|
||||
TIMEZONE=Asia/Taipei
|
||||
|
||||
# 目录
|
||||
EXPORT_ROOT=/path/to/export
|
||||
LOG_ROOT=/path/to/logs
|
||||
```
|
||||
|
||||
> 所有配置项的默认值见 `config/defaults.py`，最终生效配置由「默认值 + 环境变量 + CLI 参数」三层叠加。
|
||||
|
||||
### 2.4 运行第一个任务
|
||||
|
||||
通过 CLI 入口运行：
|
||||
|
||||
```bash
|
||||
# 运行所有任务
|
||||
python -m cli.main
|
||||
|
||||
# 仅运行订单任务
|
||||
python -m cli.main --tasks ORDERS
|
||||
|
||||
# 运行订单 + 支付
|
||||
python -m cli.main --tasks ORDERS,PAYMENTS
|
||||
|
||||
# Windows 使用脚本
|
||||
run_etl.bat --tasks ORDERS
|
||||
|
||||
# Linux / macOS 使用脚本
|
||||
./run_etl.sh --tasks ORDERS
|
||||
```
|
||||
|
||||
### 2.5 查看结果
|
||||
|
||||
- 日志目录：使用 `LOG_ROOT` 指定，例如
|
||||
|
||||
```bash
|
||||
ls -la D:\LLZQ\DB\logs/
|
||||
```
|
||||
|
||||
- 导出目录：使用 `EXPORT_ROOT` 指定，例如
|
||||
|
||||
```bash
|
||||
ls -la D:\LLZQ\DB\export/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 常用命令与开发工具
|
||||
|
||||
### 3.1 CLI 常用命令
|
||||
|
||||
```bash
|
||||
# 运行所有任务
|
||||
python -m cli.main
|
||||
|
||||
# 运行指定任务
|
||||
python -m cli.main --tasks ORDERS,PAYMENTS,MEMBERS
|
||||
|
||||
# 使用自定义数据库
|
||||
python -m cli.main --pg-dsn "postgresql://user:password@host:5432/db"
|
||||
|
||||
# 使用自定义 API 端点
|
||||
python -m cli.main --api-base "https://api.example.com" --api-token "..."
|
||||
|
||||
# 试运行（不写入数据库）
|
||||
python -m cli.main --dry-run --tasks ORDERS
|
||||
```
|
||||
|
||||
### 3.2 IDE / 代码质量工具（示例：VSCode）
|
||||
|
||||
`.vscode/settings.json` 示例：
|
||||
|
||||
```json
|
||||
{
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.pylintEnabled": true,
|
||||
"python.formatting.provider": "black",
|
||||
"python.testing.pytestEnabled": true
|
||||
}
|
||||
```
|
||||
|
||||
代码格式化与检查：
|
||||
|
||||
```bash
|
||||
pip install black isort pylint
|
||||
|
||||
black .
|
||||
isort .
|
||||
pylint etl_billiards/
|
||||
```
|
||||
|
||||
### 3.3 测试
|
||||
|
||||
```bash
|
||||
# 安装测试依赖（按需）
|
||||
pip install pytest pytest-cov
|
||||
|
||||
# 运行全部测试
|
||||
pytest
|
||||
|
||||
# 仅运行单元测试
|
||||
pytest tests/unit/
|
||||
|
||||
# 生成覆盖率报告
|
||||
pytest --cov=. --cov-report=html
|
||||
```
|
||||
|
||||
测试示例（按实际项目为准）：
|
||||
|
||||
- `tests/unit/test_config.py` – 配置管理单元测试
|
||||
- `tests/unit/test_parsers.py` – 解析器单元测试
|
||||
- `tests/integration/test_database.py` – 数据库集成测试
|
||||
|
||||
---
|
||||
|
||||
## 4. 项目结构与文件说明
|
||||
|
||||
### 4.1 总体目录结构（树状图）
|
||||
|
||||
```text
|
||||
etl_billiards/
|
||||
│
|
||||
├── README.md # 项目总览和使用说明
|
||||
├── MIGRATION_GUIDE.md # 从旧版本迁移指南
|
||||
├── requirements.txt # Python 依赖列表
|
||||
├── setup.py # 项目安装配置
|
||||
├── .env.example # 环境变量配置模板
|
||||
├── .gitignore # Git 忽略文件配置
|
||||
├── run_etl.sh # Linux/Mac 运行脚本
|
||||
├── run_etl.bat # Windows 运行脚本
|
||||
│
|
||||
├── config/ # 配置管理模块
|
||||
│ ├── __init__.py
|
||||
│ ├── defaults.py # 默认配置值定义
|
||||
│ ├── env_parser.py # 环境变量解析器
|
||||
│ └── settings.py # 配置管理主类
|
||||
│
|
||||
├── database/ # 数据库访问层
|
||||
│ ├── __init__.py
|
||||
│ ├── connection.py # 数据库连接管理
|
||||
│ └── operations.py # 批量操作封装
|
||||
│
|
||||
├── api/ # HTTP API 客户端
|
||||
│ ├── __init__.py
|
||||
│ └── client.py # API 客户端（重试 + 分页）
|
||||
│
|
||||
├── models/ # 数据模型层
|
||||
│ ├── __init__.py
|
||||
│ ├── parsers.py # 类型解析器
|
||||
│ └── validators.py # 数据验证器
|
||||
│
|
||||
├── loaders/ # 数据加载器层
|
||||
│ ├── __init__.py
|
||||
│ ├── base_loader.py # 加载器基类
|
||||
│ ├── dimensions/ # 维度表加载器
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── member.py # 会员维度加载器
|
||||
│ └── facts/ # 事实表加载器
|
||||
│ ├── __init__.py
|
||||
│ ├── order.py # 订单事实表加载器
|
||||
│ └── payment.py # 支付记录加载器
|
||||
│
|
||||
├── scd/ # SCD2 处理层
|
||||
│ ├── __init__.py
|
||||
│ └── scd2_handler.py # SCD2 历史记录处理器
|
||||
│
|
||||
├── quality/ # 数据质量检查层
|
||||
│ ├── __init__.py
|
||||
│ ├── base_checker.py # 质量检查器基类
|
||||
│ └── balance_checker.py # 余额一致性检查器
|
||||
│
|
||||
├── orchestration/ # ETL 编排层
|
||||
│ ├── __init__.py
|
||||
│ ├── scheduler.py # ETL 调度器
|
||||
│ ├── task_registry.py # 任务注册表（工厂模式）
|
||||
│ ├── cursor_manager.py # 游标管理器
|
||||
│ └── run_tracker.py # 运行记录追踪器
|
||||
│
|
||||
├── tasks/ # ETL 任务层
|
||||
│ ├── __init__.py
|
||||
│ ├── base_task.py # 任务基类（模板方法）
|
||||
│ ├── orders_task.py # 订单 ETL 任务
|
||||
│ ├── payments_task.py # 支付 ETL 任务
|
||||
│ └── members_task.py # 会员 ETL 任务
|
||||
│
|
||||
├── cli/ # 命令行接口层
|
||||
│ ├── __init__.py
|
||||
│ └── main.py # CLI 主入口
|
||||
│
|
||||
├── utils/ # 工具函数
|
||||
│ ├── __init__.py
|
||||
│ └── helpers.py # 通用工具函数
|
||||
│
|
||||
├── tests/ # 测试代码
|
||||
│ ├── __init__.py
|
||||
│ ├── unit/ # 单元测试
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── test_config.py
|
||||
│ │ └── test_parsers.py
|
||||
│ └── integration/ # 集成测试
|
||||
│ ├── __init__.py
|
||||
│ └── test_database.py
|
||||
│
|
||||
└── docs/ # 文档
|
||||
└── ARCHITECTURE.md # 架构设计文档
|
||||
```
|
||||
|
||||
### 4.2 各模块职责概览
|
||||
|
||||
- **config/**
|
||||
- 统一配置入口，支持默认值、环境变量、命令行参数三层覆盖。
|
||||
- **database/**
|
||||
- 封装 PostgreSQL 连接与批量操作（插入、更新、Upsert 等）。
|
||||
- **api/**
|
||||
- 对上游业务 API 的 HTTP 调用进行统一封装，支持重试、分页与超时控制。
|
||||
- **models/**
|
||||
- 提供类型解析器（时间戳、金额、整数等）与业务级数据校验器。
|
||||
- **loaders/**
|
||||
- 提供事实表与维度表的加载逻辑（包含批量 Upsert、统计写入结果等）。
|
||||
- **scd/**
|
||||
- 维度型数据的 SCD2 历史管理（有效期、版本标记等）。
|
||||
- **quality/**
|
||||
- 质量检查策略，例如余额一致性、记录数量对齐等。
|
||||
- **orchestration/**
|
||||
- 任务调度、任务注册、游标管理（增量窗口）、运行记录追踪。
|
||||
- **tasks/**
|
||||
- 具体业务任务（订单、支付、会员等），封装了从“取数 → 处理 → 写库 → 记录结果”的完整流程。
|
||||
- **cli/**
|
||||
- 命令行入口，解析参数并启动调度流程。
|
||||
- **utils/**
|
||||
- 杂项工具函数。
|
||||
- **tests/**
|
||||
- 单元测试与集成测试代码。
|
||||
|
||||
---
|
||||
|
||||
## 5. 架构设计与流程说明
|
||||
|
||||
### 5.1 分层架构图
|
||||
|
||||
```text
|
||||
┌─────────────────────────────────────┐
|
||||
│ CLI 命令行接口 │ <- cli/main.py
|
||||
└─────────────┬───────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────┐
|
||||
│ Orchestration 编排层 │ <- orchestration/
|
||||
│ (Scheduler, TaskRegistry, ...) │
|
||||
└─────────────┬───────────────────────┘
|
||||
│
|
||||
┌─────────────▼───────────────────────┐
|
||||
│ Tasks 任务层 │ <- tasks/
|
||||
│ (OrdersTask, PaymentsTask, ...) │
|
||||
└───┬─────────┬─────────┬─────────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────┐ ┌─────┐ ┌──────────┐
|
||||
│Loaders │ │ SCD │ │ Quality │ <- loaders/, scd/, quality/
|
||||
└────────┘ └─────┘ └──────────┘
|
||||
│
|
||||
┌───────▼────────┐
|
||||
│ Models 模型 │ <- models/
|
||||
└───────┬────────┘
|
||||
│
|
||||
┌───────▼────────┐
|
||||
│ API 客户端 │ <- api/
|
||||
└───────┬────────┘
|
||||
│
|
||||
┌───────▼────────┐
|
||||
│ Database 访问 │ <- database/
|
||||
└───────┬────────┘
|
||||
│
|
||||
┌───────▼────────┐
|
||||
│ Config 配置 │ <- config/
|
||||
└────────────────┘
|
||||
```
|
||||
|
||||
### 5.2 各层职责（当前设计）
|
||||
|
||||
- **CLI 层 (`cli/`)**
|
||||
- 解析命令行参数（指定任务列表、Dry-run、覆盖配置项等）。
|
||||
- 初始化配置与日志后交由编排层执行。
|
||||
|
||||
- **编排层 (`orchestration/`)**
|
||||
- `scheduler.py`：根据配置与 CLI 参数选择需要执行的任务，控制执行顺序和并行策略。
|
||||
- `task_registry.py`：提供任务注册表，按任务代码创建任务实例（工厂模式）。
|
||||
- `cursor_manager.py`：管理增量游标（时间窗口 / ID 游标）。
|
||||
- `run_tracker.py`：记录每次任务运行的状态、统计信息和错误信息。
|
||||
|
||||
- **任务层 (`tasks/`)**
|
||||
- `base_task.py`：定义任务执行模板流程（模板方法模式），包括获取窗口、调用上游、解析 / 校验、写库、更新游标等。
|
||||
- `orders_task.py` / `payments_task.py` / `members_task.py`：实现具体任务逻辑（订单、支付、会员）。
|
||||
|
||||
- **加载器 / SCD / 质量层**
|
||||
- `loaders/`：根据目标表封装 Upsert / Insert / Update 逻辑。
|
||||
- `scd/scd2_handler.py`：为维度表提供 SCD2 历史管理能力。
|
||||
- `quality/`：执行数据质量检查，如余额对账。
|
||||
|
||||
- **模型层 (`models/`)**
|
||||
- `parsers.py`：负责数据类型转换（字符串 → 时间戳、Decimal、int 等）。
|
||||
- `validators.py`：执行字段级和记录级的数据校验。
|
||||
|
||||
- **API 层 (`api/client.py`)**
|
||||
- 封装 HTTP 调用，处理重试、超时及分页。
|
||||
|
||||
- **数据库层 (`database/`)**
|
||||
- 管理数据库连接及上下文。
|
||||
- 提供批量插入 / 更新 / Upsert 操作接口。
|
||||
|
||||
- **配置层 (`config/`)**
|
||||
- 定义配置项默认值。
|
||||
- 解析环境变量并进行类型转换。
|
||||
- 对外提供统一配置对象。
|
||||
|
||||
### 5.3 设计模式（当前使用）
|
||||
|
||||
- 工厂模式：任务注册 / 创建（`TaskRegistry`）。
|
||||
- 模板方法模式：任务执行流程（`BaseTask`）。
|
||||
- 策略模式：不同 Loader / Checker 实现不同策略。
|
||||
- 依赖注入：通过构造函数向任务传入 `db`、`api`、`config` 等依赖。
|
||||
|
||||
### 5.4 数据与控制流程
|
||||
|
||||
整体流程：
|
||||
|
||||
1. CLI 解析参数并加载配置。
|
||||
2. Scheduler 构建数据库连接、API 客户端等依赖。
|
||||
3. Scheduler 遍历任务配置，从 `TaskRegistry` 获取任务类并实例化。
|
||||
4. 每个任务按统一模板执行：
|
||||
- 读取游标 / 时间窗口。
|
||||
- 调用 API 拉取数据（可分页）。
|
||||
- 解析、验证数据。
|
||||
- 通过 Loader 写入数据库（事实表 / 维度表 / SCD2）。
|
||||
- 执行质量检查。
|
||||
- 更新游标与运行记录。
|
||||
5. 所有任务执行完成后，释放连接并退出进程。
|
||||
|
||||
### 5.5 错误处理策略
|
||||
|
||||
- 单个任务失败不影响其他任务执行。
|
||||
- 数据库操作异常自动回滚当前事务。
|
||||
- API 请求失败时按配置进行重试，超过重试次数记录错误并终止该任务。
|
||||
- 所有错误被记录到日志和运行追踪表，便于事后排查。
|
||||
|
||||
---
|
||||
|
||||
## 6. 迁移指南（从旧脚本到当前项目）
|
||||
|
||||
本节用于说明如何从旧的单文件脚本（如 `task_merged.py`）迁移到当前模块化项目，属于当前项目的使用说明，不涉及历史对比细节。
|
||||
|
||||
### 6.1 核心功能映射示意
|
||||
|
||||
| 旧版本函数 / 类 | 新版本位置 | 说明 |
|
||||
|---------------------------|--------------------------------------------------------|----------------|
|
||||
| `DEFAULTS` 字典 | `config/defaults.py` | 配置默认值 |
|
||||
| `build_config()` | `config/settings.py::AppConfig.load()` | 配置加载 |
|
||||
| `Pg` 类 | `database/connection.py::DatabaseConnection` | 数据库连接 |
|
||||
| `http_get_json()` | `api/client.py::APIClient.get()` | API 请求 |
|
||||
| `paged_get()` | `api/client.py::APIClient.get_paginated()` | 分页请求 |
|
||||
| `parse_ts()` | `models/parsers.py::TypeParser.parse_timestamp()` | 时间解析 |
|
||||
| `upsert_fact_order()` | `loaders/facts/order.py::OrderLoader.upsert_orders()` | 订单加载 |
|
||||
| `scd2_upsert()` | `scd/scd2_handler.py::SCD2Handler.upsert()` | SCD2 处理 |
|
||||
| `run_task_orders()` | `tasks/orders_task.py::OrdersTask.execute()` | 订单任务 |
|
||||
| `main()` | `cli/main.py::main()` | 主入口 |
|
||||
|
||||
### 6.2 典型迁移步骤
|
||||
|
||||
1. **配置迁移**
|
||||
- 原来在 `DEFAULTS` 或脚本内硬编码的配置，迁移到 `.env` 与 `config/defaults.py`。
|
||||
- 使用 `AppConfig.load()` 统一获取配置。
|
||||
|
||||
2. **并行运行验证**
|
||||
|
||||
```bash
|
||||
# 旧脚本
|
||||
python task_merged.py --tasks ORDERS
|
||||
|
||||
# 新项目
|
||||
python -m cli.main --tasks ORDERS
|
||||
```
|
||||
|
||||
对比新旧版本导出的数据表和日志，确认一致性。
|
||||
|
||||
3. **自定义逻辑迁移**
|
||||
- 原脚本中的自定义清洗逻辑 → 放入相应 `loaders/` 或任务类中。
|
||||
- 自定义任务 → 在 `tasks/` 中实现并在 `task_registry` 中注册。
|
||||
- 自定义 API 调用 → 扩展 `api/client.py` 或单独封装服务类。
|
||||
|
||||
4. **逐步切换**
|
||||
- 先在测试环境并行运行。
|
||||
- 再逐步切换生产任务到新版本。
|
||||
|
||||
---
|
||||
|
||||
## 7. 开发与扩展指南（当前项目）
|
||||
|
||||
### 7.1 添加新任务
|
||||
|
||||
1. 在 `tasks/` 目录创建任务类：
|
||||
|
||||
```python
|
||||
from .base_task import BaseTask
|
||||
|
||||
class MyTask(BaseTask):
|
||||
def get_task_code(self) -> str:
|
||||
return "MY_TASK"
|
||||
|
||||
def execute(self) -> dict:
|
||||
# 1. 获取时间窗口
|
||||
window_start, window_end, _ = self._get_time_window()
|
||||
|
||||
# 2. 调用 API 获取数据
|
||||
records, _ = self.api.get_paginated(...)
|
||||
|
||||
# 3. 解析 / 校验
|
||||
parsed = [self._parse(r) for r in records]
|
||||
|
||||
# 4. 加载数据
|
||||
loader = MyLoader(self.db)
|
||||
inserted, updated, _ = loader.upsert(parsed)
|
||||
|
||||
# 5. 提交并返回结果
|
||||
self.db.commit()
|
||||
return self._build_result("SUCCESS", {
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
})
|
||||
```
|
||||
|
||||
2. 在 `orchestration/task_registry.py` 中注册：
|
||||
|
||||
```python
|
||||
from tasks.my_task import MyTask
|
||||
|
||||
default_registry.register("MY_TASK", MyTask)
|
||||
```
|
||||
|
||||
3. 在任务配置表中启用（示例）：
|
||||
|
||||
```sql
|
||||
INSERT INTO etl_admin.etl_task (task_code, store_id, enabled)
|
||||
VALUES ('MY_TASK', 123456, TRUE);
|
||||
```
|
||||
|
||||
### 7.2 添加新加载器
|
||||
|
||||
```python
|
||||
from loaders.base_loader import BaseLoader
|
||||
|
||||
class MyLoader(BaseLoader):
|
||||
def upsert(self, records: list) -> tuple:
|
||||
sql = "INSERT INTO table_name (...) VALUES (...) ON CONFLICT (...) DO UPDATE SET ... RETURNING (xmax = 0) AS inserted"
|
||||
inserted, updated = self.db.batch_upsert_with_returning(
|
||||
sql, records, page_size=self._batch_size()
|
||||
)
|
||||
return (inserted, updated, 0)
|
||||
```
|
||||
|
||||
### 7.3 添加新质量检查器
|
||||
|
||||
1. 在 `quality/` 中实现检查器，继承 `base_checker.py`。
|
||||
2. 在任务或调度流程中调用该检查器，在写库后进行验证。
|
||||
|
||||
### 7.4 类型解析与校验扩展
|
||||
|
||||
- 在 `models/parsers.py` 中添加新类型解析方法。
|
||||
- 在 `models/validators.py` 中添加新规则（如枚举校验、跨字段校验等）。
|
||||
|
||||
---
|
||||
|
||||
## 8. 常见问题排查
|
||||
|
||||
### 8.1 数据库连接失败
|
||||
|
||||
```text
|
||||
错误: could not connect to server
|
||||
```
|
||||
|
||||
排查要点：
|
||||
|
||||
- 检查 `PG_DSN` 或相关数据库配置是否正确。
|
||||
- 确认数据库服务是否启动、网络是否可达。
|
||||
|
||||
### 8.2 API 请求超时
|
||||
|
||||
```text
|
||||
错误: requests.exceptions.Timeout
|
||||
```
|
||||
|
||||
排查要点：
|
||||
|
||||
- 检查 `API_BASE` 地址与网络连通性。
|
||||
- 适当提高超时与重试次数（在配置中调整）。
|
||||
|
||||
### 8.3 模块导入错误
|
||||
|
||||
```text
|
||||
错误: ModuleNotFoundError
|
||||
```
|
||||
|
||||
排查要点：
|
||||
|
||||
- 确认在项目根目录下运行（包含 `etl_billiards/` 包）。
|
||||
- 或通过 `pip install -e .` 以可编辑模式安装项目。
|
||||
|
||||
### 8.4 权限相关问题
|
||||
|
||||
```text
|
||||
错误: Permission denied
|
||||
```
|
||||
|
||||
排查要点：
|
||||
|
||||
- 脚本无执行权限：`chmod +x run_etl.sh`。
|
||||
- Windows 需要以管理员身份运行，或修改日志 / 导出目录权限。
|
||||
|
||||
---
|
||||
|
||||
## 9. 使用前检查清单
|
||||
|
||||
在正式运行前建议确认：
|
||||
|
||||
- [ ] 已安装 Python 3.10+。
|
||||
- [ ] 已执行 `pip install -r requirements.txt`。
|
||||
- [ ] `.env` 已配置正确（数据库、API、门店 ID、路径等）。
|
||||
- [ ] PostgreSQL 数据库可连接。
|
||||
- [ ] API 服务可访问且凭证有效。
|
||||
- [ ] `LOG_ROOT`、`EXPORT_ROOT` 目录存在且拥有写权限。
|
||||
|
||||
---
|
||||
|
||||
## 10. 参考说明
|
||||
|
||||
- 本文已合并原有的快速开始、项目结构、架构说明、迁移指南等内容，可作为当前项目的统一说明文档。
|
||||
- 如需在此基础上拆分多份文档，可按章节拆出，例如「快速开始」「架构设计」「迁移指南」「开发扩展」等。
|
||||
0
etl_billiards/__init__.py
Normal file
0
etl_billiards/__init__.py
Normal file
0
etl_billiards/api/__init__.py
Normal file
0
etl_billiards/api/__init__.py
Normal file
95
etl_billiards/api/client.py
Normal file
95
etl_billiards/api/client.py
Normal file
@@ -0,0 +1,95 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""API客户端"""
|
||||
import requests
|
||||
from urllib3.util.retry import Retry
|
||||
from requests.adapters import HTTPAdapter
|
||||
|
||||
class APIClient:
|
||||
"""HTTP API客户端"""
|
||||
|
||||
def __init__(self, base_url: str, token: str = None, timeout: int = 20,
|
||||
retry_max: int = 3, headers_extra: dict = None):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.token = token
|
||||
self.timeout = timeout
|
||||
self.retry_max = retry_max
|
||||
self.headers_extra = headers_extra or {}
|
||||
self._session = None
|
||||
|
||||
def _get_session(self):
|
||||
"""获取或创建会话"""
|
||||
if self._session is None:
|
||||
self._session = requests.Session()
|
||||
|
||||
retries = max(0, int(self.retry_max) - 1)
|
||||
retry = Retry(
|
||||
total=None,
|
||||
connect=retries,
|
||||
read=retries,
|
||||
status=retries,
|
||||
allowed_methods=frozenset(["GET"]),
|
||||
status_forcelist=(429, 500, 502, 503, 504),
|
||||
backoff_factor=1.0,
|
||||
respect_retry_after_header=True,
|
||||
raise_on_status=False,
|
||||
)
|
||||
|
||||
adapter = HTTPAdapter(max_retries=retry)
|
||||
self._session.mount("http://", adapter)
|
||||
self._session.mount("https://", adapter)
|
||||
|
||||
if self.headers_extra:
|
||||
self._session.headers.update(self.headers_extra)
|
||||
|
||||
return self._session
|
||||
|
||||
def get(self, endpoint: str, params: dict = None) -> dict:
|
||||
"""执行GET请求"""
|
||||
url = f"{self.base_url}/{endpoint.lstrip('/')}"
|
||||
headers = {"Authorization": self.token} if self.token else {}
|
||||
headers.update(self.headers_extra)
|
||||
|
||||
sess = self._get_session()
|
||||
resp = sess.get(url, headers=headers, params=params, timeout=self.timeout)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
def get_paginated(self, endpoint: str, params: dict, page_size: int = 200,
|
||||
page_field: str = "pageIndex", size_field: str = "pageSize",
|
||||
data_path: tuple = ("data",), list_key: str = None) -> tuple:
|
||||
"""分页获取数据"""
|
||||
records, pages_meta = [], []
|
||||
page = 1
|
||||
|
||||
while True:
|
||||
p = dict(params)
|
||||
p[page_field] = page
|
||||
p[size_field] = page_size
|
||||
|
||||
obj = self.get(endpoint, p)
|
||||
|
||||
# 解析数据路径
|
||||
cur = obj
|
||||
for k in data_path:
|
||||
if isinstance(cur, dict) and k in cur:
|
||||
cur = cur[k]
|
||||
|
||||
if list_key:
|
||||
cur = (cur or {}).get(list_key, [])
|
||||
|
||||
if not isinstance(cur, list):
|
||||
cur = []
|
||||
|
||||
records.extend(cur)
|
||||
|
||||
if len(cur) == 0:
|
||||
break
|
||||
|
||||
pages_meta.append({"page": page, "request": p, "response": obj})
|
||||
|
||||
if len(cur) < page_size:
|
||||
break
|
||||
|
||||
page += 1
|
||||
|
||||
return records, pages_meta
|
||||
0
etl_billiards/cli/__init__.py
Normal file
0
etl_billiards/cli/__init__.py
Normal file
126
etl_billiards/cli/main.py
Normal file
126
etl_billiards/cli/main.py
Normal file
@@ -0,0 +1,126 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""CLI主入口"""
|
||||
import sys
|
||||
import argparse
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from config.settings import AppConfig
|
||||
from orchestration.scheduler import ETLScheduler
|
||||
|
||||
def setup_logging():
|
||||
"""设置日志"""
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(name)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
)
|
||||
return logging.getLogger("etl_billiards")
|
||||
|
||||
def parse_args():
|
||||
"""解析命令行参数"""
|
||||
parser = argparse.ArgumentParser(description="台球场ETL系统")
|
||||
|
||||
# 基本参数
|
||||
parser.add_argument("--store-id", type=int, help="门店ID")
|
||||
parser.add_argument("--tasks", help="任务列表，逗号分隔")
|
||||
parser.add_argument("--dry-run", action="store_true", help="试运行（不提交）")
|
||||
|
||||
# 数据库参数
|
||||
parser.add_argument("--pg-dsn", help="PostgreSQL DSN")
|
||||
parser.add_argument("--pg-host", help="PostgreSQL主机")
|
||||
parser.add_argument("--pg-port", type=int, help="PostgreSQL端口")
|
||||
parser.add_argument("--pg-name", help="PostgreSQL数据库名")
|
||||
parser.add_argument("--pg-user", help="PostgreSQL用户名")
|
||||
parser.add_argument("--pg-password", help="PostgreSQL密码")
|
||||
|
||||
# API参数
|
||||
parser.add_argument("--api-base", help="API基础URL")
|
||||
parser.add_argument("--api-token", help="API令牌")
|
||||
parser.add_argument("--api-timeout", type=int, help="API超时(秒)")
|
||||
parser.add_argument("--api-page-size", type=int, help="分页大小")
|
||||
|
||||
# 目录参数
|
||||
parser.add_argument("--export-root", help="导出根目录")
|
||||
parser.add_argument("--log-root", help="日志根目录")
|
||||
|
||||
return parser.parse_args()
|
||||
|
||||
def build_cli_overrides(args) -> dict:
|
||||
"""从命令行参数构建配置覆盖"""
|
||||
overrides = {}
|
||||
|
||||
# 基本信息
|
||||
if args.store_id is not None:
|
||||
overrides.setdefault("app", {})["store_id"] = args.store_id
|
||||
|
||||
# 数据库
|
||||
if args.pg_dsn:
|
||||
overrides.setdefault("db", {})["dsn"] = args.pg_dsn
|
||||
if args.pg_host:
|
||||
overrides.setdefault("db", {})["host"] = args.pg_host
|
||||
if args.pg_port:
|
||||
overrides.setdefault("db", {})["port"] = args.pg_port
|
||||
if args.pg_name:
|
||||
overrides.setdefault("db", {})["name"] = args.pg_name
|
||||
if args.pg_user:
|
||||
overrides.setdefault("db", {})["user"] = args.pg_user
|
||||
if args.pg_password:
|
||||
overrides.setdefault("db", {})["password"] = args.pg_password
|
||||
|
||||
# API
|
||||
if args.api_base:
|
||||
overrides.setdefault("api", {})["base_url"] = args.api_base
|
||||
if args.api_token:
|
||||
overrides.setdefault("api", {})["token"] = args.api_token
|
||||
if args.api_timeout:
|
||||
overrides.setdefault("api", {})["timeout_sec"] = args.api_timeout
|
||||
if args.api_page_size:
|
||||
overrides.setdefault("api", {})["page_size"] = args.api_page_size
|
||||
|
||||
# 目录
|
||||
if args.export_root:
|
||||
overrides.setdefault("io", {})["export_root"] = args.export_root
|
||||
if args.log_root:
|
||||
overrides.setdefault("io", {})["log_root"] = args.log_root
|
||||
|
||||
# 任务
|
||||
if args.tasks:
|
||||
tasks = [t.strip().upper() for t in args.tasks.split(",") if t.strip()]
|
||||
overrides.setdefault("run", {})["tasks"] = tasks
|
||||
|
||||
return overrides
|
||||
|
||||
def main():
|
||||
"""主函数"""
|
||||
logger = setup_logging()
|
||||
args = parse_args()
|
||||
|
||||
try:
|
||||
# 加载配置
|
||||
cli_overrides = build_cli_overrides(args)
|
||||
config = AppConfig.load(cli_overrides)
|
||||
|
||||
logger.info("配置加载完成")
|
||||
logger.info(f"门店ID: {config.get('app.store_id')}")
|
||||
logger.info(f"任务列表: {config.get('run.tasks')}")
|
||||
|
||||
# 创建调度器
|
||||
scheduler = ETLScheduler(config, logger)
|
||||
|
||||
# 运行任务
|
||||
task_codes = config.get("run.tasks")
|
||||
scheduler.run_tasks(task_codes)
|
||||
|
||||
# 关闭连接
|
||||
scheduler.close()
|
||||
|
||||
logger.info("ETL运行完成")
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"ETL运行失败: {e}", exc_info=True)
|
||||
return 1
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
0
etl_billiards/config/__init__.py
Normal file
0
etl_billiards/config/__init__.py
Normal file
95
etl_billiards/config/defaults.py
Normal file
95
etl_billiards/config/defaults.py
Normal file
@@ -0,0 +1,95 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""配置默认值"""
|
||||
|
||||
DEFAULTS = {
|
||||
"app": {
|
||||
"timezone": "Asia/Taipei",
|
||||
"store_id": "",
|
||||
"schema_oltp": "billiards",
|
||||
"schema_etl": "etl_admin",
|
||||
},
|
||||
"db": {
|
||||
"dsn": "",
|
||||
"host": "",
|
||||
"port": "",
|
||||
"name": "",
|
||||
"user": "",
|
||||
"password": "",
|
||||
"connect_timeout_sec": 5,
|
||||
"batch_size": 1000,
|
||||
"session": {
|
||||
"timezone": "Asia/Taipei",
|
||||
"statement_timeout_ms": 30000,
|
||||
"lock_timeout_ms": 5000,
|
||||
"idle_in_tx_timeout_ms": 600000
|
||||
},
|
||||
},
|
||||
"api": {
|
||||
"base_url": None,
|
||||
"token": None,
|
||||
"timeout_sec": 20,
|
||||
"page_size": 200,
|
||||
"retries": {
|
||||
"max_attempts": 3,
|
||||
"backoff_sec": [1, 2, 4],
|
||||
},
|
||||
"headers_extra": {},
|
||||
},
|
||||
"run": {
|
||||
"tasks": [
|
||||
"PRODUCTS", "TABLES", "MEMBERS", "ASSISTANTS", "PACKAGES_DEF",
|
||||
"ORDERS", "PAYMENTS", "REFUNDS", "COUPON_USAGE", "INVENTORY_CHANGE",
|
||||
"TOPUPS", "TABLE_DISCOUNT", "ASSISTANT_ABOLISH",
|
||||
"LEDGER",
|
||||
],
|
||||
"window_minutes": {
|
||||
"default_busy": 30,
|
||||
"default_idle": 180,
|
||||
},
|
||||
"overlap_seconds": 120,
|
||||
"idle_window": {
|
||||
"start": "04:00",
|
||||
"end": "16:00",
|
||||
},
|
||||
"allow_empty_result_advance": True,
|
||||
},
|
||||
"io": {
|
||||
"export_root": r"D:\LLZQ\DB\export",
|
||||
"log_root": r"D:\LLZQ\DB\logs",
|
||||
"manifest_name": "manifest.json",
|
||||
"ingest_report_name": "ingest_report.json",
|
||||
"write_pretty_json": False,
|
||||
"max_file_bytes": 50 * 1024 * 1024,
|
||||
},
|
||||
"clean": {
|
||||
"log_unknown_fields": True,
|
||||
"unknown_fields_limit": 50,
|
||||
"hash_key": {
|
||||
"algo": "sha1",
|
||||
"salt": "",
|
||||
},
|
||||
"strict_numeric": True,
|
||||
"round_money_scale": 2,
|
||||
},
|
||||
"security": {
|
||||
"redact_in_logs": True,
|
||||
"redact_keys": ["token", "password", "Authorization"],
|
||||
"echo_token_in_logs": False,
|
||||
},
|
||||
}
|
||||
|
||||
# 任务代码常量
|
||||
TASK_ORDERS = "ORDERS"
|
||||
TASK_PAYMENTS = "PAYMENTS"
|
||||
TASK_REFUNDS = "REFUNDS"
|
||||
TASK_INVENTORY_CHANGE = "INVENTORY_CHANGE"
|
||||
TASK_COUPON_USAGE = "COUPON_USAGE"
|
||||
TASK_MEMBERS = "MEMBERS"
|
||||
TASK_ASSISTANTS = "ASSISTANTS"
|
||||
TASK_PRODUCTS = "PRODUCTS"
|
||||
TASK_TABLES = "TABLES"
|
||||
TASK_PACKAGES_DEF = "PACKAGES_DEF"
|
||||
TASK_TOPUPS = "TOPUPS"
|
||||
TASK_TABLE_DISCOUNT = "TABLE_DISCOUNT"
|
||||
TASK_ASSISTANT_ABOLISH = "ASSISTANT_ABOLISH"
|
||||
TASK_LEDGER = "LEDGER"
|
||||
62
etl_billiards/config/env_parser.py
Normal file
62
etl_billiards/config/env_parser.py
Normal file
@@ -0,0 +1,62 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""环境变量解析"""
|
||||
import os
|
||||
import json
|
||||
from copy import deepcopy
|
||||
|
||||
ENV_MAP = {
|
||||
"TIMEZONE": ("app.timezone",),
|
||||
"STORE_ID": ("app.store_id",),
|
||||
"SCHEMA_OLTP": ("app.schema_oltp",),
|
||||
"SCHEMA_ETL": ("app.schema_etl",),
|
||||
"PG_DSN": ("db.dsn",),
|
||||
"PG_HOST": ("db.host",),
|
||||
"PG_PORT": ("db.port",),
|
||||
"PG_NAME": ("db.name",),
|
||||
"PG_USER": ("db.user",),
|
||||
"PG_PASSWORD": ("db.password",),
|
||||
"API_BASE": ("api.base_url",),
|
||||
"API_TOKEN": ("api.token",),
|
||||
"API_TIMEOUT": ("api.timeout_sec",),
|
||||
"API_PAGE_SIZE": ("api.page_size",),
|
||||
"EXPORT_ROOT": ("io.export_root",),
|
||||
"LOG_ROOT": ("io.log_root",),
|
||||
"OVERLAP_SECONDS": ("run.overlap_seconds",),
|
||||
"WINDOW_BUSY_MIN": ("run.window_minutes.default_busy",),
|
||||
"WINDOW_IDLE_MIN": ("run.window_minutes.default_idle",),
|
||||
}
|
||||
|
||||
def _deep_set(d, dotted_keys, value):
|
||||
cur = d
|
||||
for k in dotted_keys[:-1]:
|
||||
cur = cur.setdefault(k, {})
|
||||
cur[dotted_keys[-1]] = value
|
||||
|
||||
def _coerce_env(v: str):
|
||||
if v is None:
|
||||
return None
|
||||
s = v.strip()
|
||||
if s.lower() in ("true", "false"):
|
||||
return s.lower() == "true"
|
||||
try:
|
||||
if s.isdigit() or (s.startswith("-") and s[1:].isdigit()):
|
||||
return int(s)
|
||||
except Exception:
|
||||
pass
|
||||
if (s.startswith("{") and s.endswith("}")) or (s.startswith("[") and s.endswith("]")):
|
||||
try:
|
||||
return json.loads(s)
|
||||
except Exception:
|
||||
return s
|
||||
return s
|
||||
|
||||
def load_env_overrides(defaults: dict) -> dict:
|
||||
cfg = deepcopy(defaults)
|
||||
for env_key, dotted in ENV_MAP.items():
|
||||
val = os.environ.get(env_key)
|
||||
if val is None:
|
||||
continue
|
||||
v2 = _coerce_env(val)
|
||||
for path in dotted:
|
||||
_deep_set(cfg, path.split("."), v2)
|
||||
return cfg
|
||||
85
etl_billiards/config/settings.py
Normal file
85
etl_billiards/config/settings.py
Normal file
@@ -0,0 +1,85 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""配置管理主类"""
|
||||
from copy import deepcopy
|
||||
from .defaults import DEFAULTS
|
||||
from .env_parser import load_env_overrides
|
||||
|
||||
class AppConfig:
|
||||
"""应用配置管理器"""
|
||||
|
||||
def __init__(self, config_dict: dict):
|
||||
self.config = config_dict
|
||||
|
||||
@classmethod
|
||||
def load(cls, cli_overrides: dict = None):
|
||||
"""加载配置: DEFAULTS < ENV < CLI"""
|
||||
cfg = load_env_overrides(DEFAULTS)
|
||||
|
||||
if cli_overrides:
|
||||
cls._deep_merge(cfg, cli_overrides)
|
||||
|
||||
# 规范化
|
||||
cls._normalize(cfg)
|
||||
cls._validate(cfg)
|
||||
|
||||
return cls(cfg)
|
||||
|
||||
@staticmethod
|
||||
def _deep_merge(dst, src):
|
||||
"""深度合并字典"""
|
||||
for k, v in src.items():
|
||||
if isinstance(v, dict) and isinstance(dst.get(k), dict):
|
||||
AppConfig._deep_merge(dst[k], v)
|
||||
else:
|
||||
dst[k] = v
|
||||
|
||||
@staticmethod
|
||||
def _normalize(cfg):
|
||||
"""规范化配置"""
|
||||
# 转换 store_id 为整数
|
||||
try:
|
||||
cfg["app"]["store_id"] = int(str(cfg["app"]["store_id"]).strip())
|
||||
except Exception:
|
||||
raise SystemExit("app.store_id 必须为整数")
|
||||
|
||||
# DSN 组装
|
||||
if not cfg["db"]["dsn"]:
|
||||
cfg["db"]["dsn"] = (
|
||||
f"postgresql://{cfg['db']['user']}:{cfg['db']['password']}"
|
||||
f"@{cfg['db']['host']}:{cfg['db']['port']}/{cfg['db']['name']}"
|
||||
)
|
||||
|
||||
# 会话参数
|
||||
cfg["db"].setdefault("session", {})
|
||||
sess = cfg["db"]["session"]
|
||||
sess.setdefault("timezone", cfg["app"]["timezone"])
|
||||
|
||||
for k in ("statement_timeout_ms", "lock_timeout_ms", "idle_in_tx_timeout_ms"):
|
||||
if k in sess and sess[k] is not None:
|
||||
try:
|
||||
sess[k] = int(sess[k])
|
||||
except Exception:
|
||||
raise SystemExit(f"db.session.{k} 需为整数毫秒")
|
||||
|
||||
@staticmethod
|
||||
def _validate(cfg):
|
||||
"""验证必填配置"""
|
||||
missing = []
|
||||
if not cfg["app"]["store_id"]:
|
||||
missing.append("app.store_id")
|
||||
if missing:
|
||||
raise SystemExit("缺少必需配置: " + ", ".join(missing))
|
||||
|
||||
def get(self, key: str, default=None):
|
||||
"""获取配置值（支持点号路径）"""
|
||||
keys = key.split(".")
|
||||
val = self.config
|
||||
for k in keys:
|
||||
if isinstance(val, dict):
|
||||
val = val.get(k)
|
||||
else:
|
||||
return default
|
||||
return val if val is not None else default
|
||||
|
||||
def __getitem__(self, key):
|
||||
return self.config[key]
|
||||
0
etl_billiards/database/__init__.py
Normal file
0
etl_billiards/database/__init__.py
Normal file
112
etl_billiards/database/base.py
Normal file
112
etl_billiards/database/base.py
Normal file
@@ -0,0 +1,112 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
数据库操作（批量、RETURNING支持）
|
||||
"""
|
||||
import re
|
||||
from typing import List, Dict, Tuple
|
||||
import psycopg2.extras
|
||||
from .connection import DatabaseConnection
|
||||
|
||||
|
||||
class DatabaseOperations(DatabaseConnection):
|
||||
"""扩展数据库操作（包含批量upsert和returning支持）"""
|
||||
|
||||
def batch_execute(self, sql: str, rows: List[Dict], page_size: int = 1000):
|
||||
"""批量执行SQL（不带RETURNING）"""
|
||||
if not rows:
|
||||
return
|
||||
with self.conn.cursor() as c:
|
||||
psycopg2.extras.execute_batch(c, sql, rows, page_size=page_size)
|
||||
|
||||
def batch_upsert_with_returning(self, sql: str, rows: List[Dict], page_size: int = 1000) -> Tuple[int, int]:
|
||||
"""
|
||||
批量 UPSERT 并统计插入/更新数
|
||||
|
||||
Args:
|
||||
sql: 包含RETURNING子句的SQL
|
||||
rows: 数据行列表
|
||||
page_size: 批次大小
|
||||
|
||||
Returns:
|
||||
(inserted_count, updated_count) 元组
|
||||
"""
|
||||
if not rows:
|
||||
return (0, 0)
|
||||
|
||||
use_returning = "RETURNING" in sql.upper()
|
||||
|
||||
with self.conn.cursor() as c:
|
||||
if not use_returning:
|
||||
psycopg2.extras.execute_batch(c, sql, rows, page_size=page_size)
|
||||
return (0, 0)
|
||||
|
||||
# 优先尝试向量化执行
|
||||
try:
|
||||
inserted, updated = self._execute_with_returning_vectorized(c, sql, rows, page_size)
|
||||
return (inserted, updated)
|
||||
except Exception:
|
||||
# 回退到逐行执行
|
||||
return self._execute_with_returning_row_by_row(c, sql, rows)
|
||||
|
||||
def _execute_with_returning_vectorized(self, cursor, sql: str, rows: List[Dict], page_size: int) -> Tuple[int, int]:
|
||||
"""向量化执行（使用execute_values）"""
|
||||
# 解析VALUES子句
|
||||
m = re.search(r"VALUES\s*\((.*?)\)", sql, flags=re.IGNORECASE | re.DOTALL)
|
||||
if not m:
|
||||
raise ValueError("Cannot parse VALUES clause")
|
||||
|
||||
tpl = "(" + m.group(1) + ")"
|
||||
base_sql = sql[:m.start()] + "VALUES %s" + sql[m.end():]
|
||||
|
||||
ret = psycopg2.extras.execute_values(
|
||||
cursor, base_sql, rows, template=tpl, page_size=page_size, fetch=True
|
||||
)
|
||||
|
||||
if not ret:
|
||||
return (0, 0)
|
||||
|
||||
inserted = 0
|
||||
for rec in ret:
|
||||
flag = self._extract_inserted_flag(rec)
|
||||
if flag:
|
||||
inserted += 1
|
||||
|
||||
return (inserted, len(ret) - inserted)
|
||||
|
||||
def _execute_with_returning_row_by_row(self, cursor, sql: str, rows: List[Dict]) -> Tuple[int, int]:
|
||||
"""逐行执行（回退方案）"""
|
||||
inserted = 0
|
||||
updated = 0
|
||||
|
||||
for r in rows:
|
||||
cursor.execute(sql, r)
|
||||
try:
|
||||
rec = cursor.fetchone()
|
||||
except Exception:
|
||||
rec = None
|
||||
|
||||
flag = self._extract_inserted_flag(rec) if rec else None
|
||||
|
||||
if flag:
|
||||
inserted += 1
|
||||
else:
|
||||
updated += 1
|
||||
|
||||
return (inserted, updated)
|
||||
|
||||
@staticmethod
|
||||
def _extract_inserted_flag(rec) -> bool:
|
||||
"""从返回记录中提取inserted标志"""
|
||||
if isinstance(rec, tuple):
|
||||
return bool(rec[0])
|
||||
elif isinstance(rec, dict):
|
||||
return bool(rec.get("inserted"))
|
||||
else:
|
||||
try:
|
||||
return bool(rec["inserted"])
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# 为了向后兼容，提供Pg别名
|
||||
Pg = DatabaseOperations
|
||||
50
etl_billiards/database/connection.py
Normal file
50
etl_billiards/database/connection.py
Normal file
@@ -0,0 +1,50 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据库连接管理"""
|
||||
import psycopg2
|
||||
import psycopg2.extras
|
||||
|
||||
class DatabaseConnection:
|
||||
"""数据库连接管理器"""
|
||||
|
||||
def __init__(self, dsn: str, session: dict = None, connect_timeout: int = None):
|
||||
self.conn = psycopg2.connect(dsn, connect_timeout=(connect_timeout or 5))
|
||||
self.conn.autocommit = False
|
||||
|
||||
# 设置会话参数
|
||||
if session:
|
||||
with self.conn.cursor() as c:
|
||||
if session.get("timezone"):
|
||||
c.execute("SET TIME ZONE %s", (session["timezone"],))
|
||||
if session.get("statement_timeout_ms") is not None:
|
||||
c.execute("SET statement_timeout = %s", (int(session["statement_timeout_ms"]),))
|
||||
if session.get("lock_timeout_ms") is not None:
|
||||
c.execute("SET lock_timeout = %s", (int(session["lock_timeout_ms"]),))
|
||||
if session.get("idle_in_tx_timeout_ms") is not None:
|
||||
c.execute("SET idle_in_transaction_session_timeout = %s",
|
||||
(int(session["idle_in_tx_timeout_ms"]),))
|
||||
|
||||
def query(self, sql: str, args=None):
|
||||
"""执行查询并返回结果"""
|
||||
with self.conn.cursor(cursor_factory=psycopg2.extras.RealDictCursor) as c:
|
||||
c.execute(sql, args)
|
||||
return c.fetchall()
|
||||
|
||||
def execute(self, sql: str, args=None):
|
||||
"""执行SQL语句"""
|
||||
with self.conn.cursor() as c:
|
||||
c.execute(sql, args)
|
||||
|
||||
def commit(self):
|
||||
"""提交事务"""
|
||||
self.conn.commit()
|
||||
|
||||
def rollback(self):
|
||||
"""回滚事务"""
|
||||
self.conn.rollback()
|
||||
|
||||
def close(self):
|
||||
"""关闭连接"""
|
||||
try:
|
||||
self.conn.close()
|
||||
except Exception:
|
||||
pass
|
||||
77
etl_billiards/database/operations.py
Normal file
77
etl_billiards/database/operations.py
Normal file
@@ -0,0 +1,77 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据库批量操作"""
|
||||
import psycopg2.extras
|
||||
import re
|
||||
|
||||
class DatabaseOperations:
|
||||
"""数据库批量操作封装"""
|
||||
|
||||
def __init__(self, connection):
|
||||
self.conn = connection.conn
|
||||
|
||||
def batch_execute(self, sql: str, rows: list, page_size: int = 1000):
|
||||
"""批量执行SQL"""
|
||||
if not rows:
|
||||
return
|
||||
with self.conn.cursor() as c:
|
||||
psycopg2.extras.execute_batch(c, sql, rows, page_size=page_size)
|
||||
|
||||
def batch_upsert_with_returning(self, sql: str, rows: list,
|
||||
page_size: int = 1000) -> tuple:
|
||||
"""批量UPSERT并返回插入/更新计数"""
|
||||
if not rows:
|
||||
return (0, 0)
|
||||
|
||||
use_returning = "RETURNING" in sql.upper()
|
||||
|
||||
with self.conn.cursor() as c:
|
||||
if not use_returning:
|
||||
psycopg2.extras.execute_batch(c, sql, rows, page_size=page_size)
|
||||
return (0, 0)
|
||||
|
||||
# 尝试向量化执行
|
||||
try:
|
||||
m = re.search(r"VALUES\s*\((.*?)\)", sql, flags=re.IGNORECASE | re.DOTALL)
|
||||
if m:
|
||||
tpl = "(" + m.group(1) + ")"
|
||||
base_sql = sql[:m.start()] + "VALUES %s" + sql[m.end():]
|
||||
|
||||
ret = psycopg2.extras.execute_values(
|
||||
c, base_sql, rows, template=tpl, page_size=page_size, fetch=True
|
||||
)
|
||||
|
||||
if not ret:
|
||||
return (0, 0)
|
||||
|
||||
inserted = sum(1 for rec in ret if self._is_inserted(rec))
|
||||
return (inserted, len(ret) - inserted)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 回退：逐行执行
|
||||
inserted = 0
|
||||
updated = 0
|
||||
for r in rows:
|
||||
c.execute(sql, r)
|
||||
try:
|
||||
rec = c.fetchone()
|
||||
except Exception:
|
||||
rec = None
|
||||
|
||||
if self._is_inserted(rec):
|
||||
inserted += 1
|
||||
else:
|
||||
updated += 1
|
||||
|
||||
return (inserted, updated)
|
||||
|
||||
@staticmethod
|
||||
def _is_inserted(rec) -> bool:
|
||||
"""判断是否为插入操作"""
|
||||
if rec is None:
|
||||
return False
|
||||
if isinstance(rec, tuple):
|
||||
return bool(rec[0])
|
||||
if isinstance(rec, dict):
|
||||
return bool(rec.get("inserted"))
|
||||
return False
|
||||
0
etl_billiards/loaders/__init__.py
Normal file
0
etl_billiards/loaders/__init__.py
Normal file
19
etl_billiards/loaders/base_loader.py
Normal file
19
etl_billiards/loaders/base_loader.py
Normal file
@@ -0,0 +1,19 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据加载器基类"""
|
||||
|
||||
class BaseLoader:
|
||||
"""数据加载器基类"""
|
||||
|
||||
def __init__(self, db_ops):
|
||||
self.db = db_ops
|
||||
|
||||
def upsert(self, records: list) -> tuple:
|
||||
"""
|
||||
执行UPSERT操作
|
||||
返回: (inserted_count, updated_count, skipped_count)
|
||||
"""
|
||||
raise NotImplementedError("子类需实现 upsert 方法")
|
||||
|
||||
def _batch_size(self) -> int:
|
||||
"""批次大小"""
|
||||
return 1000
|
||||
0
etl_billiards/loaders/dimensions/__init__.py
Normal file
0
etl_billiards/loaders/dimensions/__init__.py
Normal file
34
etl_billiards/loaders/dimensions/member.py
Normal file
34
etl_billiards/loaders/dimensions/member.py
Normal file
@@ -0,0 +1,34 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""会员维度表加载器"""
|
||||
from ..base_loader import BaseLoader
|
||||
|
||||
class MemberLoader(BaseLoader):
|
||||
"""会员维度加载器"""
|
||||
|
||||
def upsert_members(self, records: list, store_id: int) -> tuple:
|
||||
"""加载会员数据"""
|
||||
if not records:
|
||||
return (0, 0, 0)
|
||||
|
||||
sql = """
|
||||
INSERT INTO billiards.dim_member (
|
||||
store_id, member_id, member_name, phone, balance,
|
||||
status, register_time, raw_data
|
||||
)
|
||||
VALUES (
|
||||
%(store_id)s, %(member_id)s, %(member_name)s, %(phone)s, %(balance)s,
|
||||
%(status)s, %(register_time)s, %(raw_data)s
|
||||
)
|
||||
ON CONFLICT (store_id, member_id) DO UPDATE SET
|
||||
member_name = EXCLUDED.member_name,
|
||||
phone = EXCLUDED.phone,
|
||||
balance = EXCLUDED.balance,
|
||||
status = EXCLUDED.status,
|
||||
register_time = EXCLUDED.register_time,
|
||||
raw_data = EXCLUDED.raw_data,
|
||||
updated_at = now()
|
||||
RETURNING (xmax = 0) AS inserted
|
||||
"""
|
||||
|
||||
inserted, updated = self.db.batch_upsert_with_returning(sql, records, page_size=self._batch_size())
|
||||
return (inserted, updated, 0)
|
||||
134
etl_billiards/loaders/dimensions/product.py
Normal file
134
etl_billiards/loaders/dimensions/product.py
Normal file
@@ -0,0 +1,134 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""商品维度 + 价格SCD2 加载器"""
|
||||
|
||||
from ..base_loader import BaseLoader
|
||||
from scd.scd2_handler import SCD2Handler
|
||||
|
||||
|
||||
class ProductLoader(BaseLoader):
|
||||
"""商品维度加载器（dim_product + dim_product_price_scd）"""
|
||||
|
||||
def __init__(self, db_ops):
|
||||
super().__init__(db_ops)
|
||||
# SCD2 处理器，复用通用逻辑
|
||||
self.scd_handler = SCD2Handler(db_ops)
|
||||
|
||||
def upsert_products(self, records: list, store_id: int) -> tuple:
|
||||
"""
|
||||
加载商品维度及价格SCD
|
||||
|
||||
返回: (inserted_count, updated_count, skipped_count)
|
||||
"""
|
||||
if not records:
|
||||
return (0, 0, 0)
|
||||
|
||||
# 1) 维度主表：billiards.dim_product
|
||||
sql_base = """
|
||||
INSERT INTO billiards.dim_product (
|
||||
store_id,
|
||||
product_id,
|
||||
site_product_id,
|
||||
product_name,
|
||||
category_id,
|
||||
category_name,
|
||||
second_category_id,
|
||||
unit,
|
||||
cost_price,
|
||||
sale_price,
|
||||
allow_discount,
|
||||
status,
|
||||
supplier_id,
|
||||
barcode,
|
||||
is_combo,
|
||||
created_time,
|
||||
updated_time,
|
||||
raw_data
|
||||
)
|
||||
VALUES (
|
||||
%(store_id)s,
|
||||
%(product_id)s,
|
||||
%(site_product_id)s,
|
||||
%(product_name)s,
|
||||
%(category_id)s,
|
||||
%(category_name)s,
|
||||
%(second_category_id)s,
|
||||
%(unit)s,
|
||||
%(cost_price)s,
|
||||
%(sale_price)s,
|
||||
%(allow_discount)s,
|
||||
%(status)s,
|
||||
%(supplier_id)s,
|
||||
%(barcode)s,
|
||||
%(is_combo)s,
|
||||
%(created_time)s,
|
||||
%(updated_time)s,
|
||||
%(raw_data)s
|
||||
)
|
||||
ON CONFLICT (store_id, product_id) DO UPDATE SET
|
||||
site_product_id = EXCLUDED.site_product_id,
|
||||
product_name = EXCLUDED.product_name,
|
||||
category_id = EXCLUDED.category_id,
|
||||
category_name = EXCLUDED.category_name,
|
||||
second_category_id = EXCLUDED.second_category_id,
|
||||
unit = EXCLUDED.unit,
|
||||
cost_price = EXCLUDED.cost_price,
|
||||
sale_price = EXCLUDED.sale_price,
|
||||
allow_discount = EXCLUDED.allow_discount,
|
||||
status = EXCLUDED.status,
|
||||
supplier_id = EXCLUDED.supplier_id,
|
||||
barcode = EXCLUDED.barcode,
|
||||
is_combo = EXCLUDED.is_combo,
|
||||
updated_time = COALESCE(EXCLUDED.updated_time, now()),
|
||||
raw_data = EXCLUDED.raw_data
|
||||
RETURNING (xmax = 0) AS inserted
|
||||
"""
|
||||
|
||||
inserted, updated = self.db.batch_upsert_with_returning(
|
||||
sql_base,
|
||||
records,
|
||||
page_size=self._batch_size(),
|
||||
)
|
||||
|
||||
# 2) 价格 SCD2：billiards.dim_product_price_scd
|
||||
# 只追踪 price + 类目 + 名称等字段的历史
|
||||
tracked_fields = [
|
||||
"product_name",
|
||||
"category_id",
|
||||
"category_name",
|
||||
"second_category_id",
|
||||
"cost_price",
|
||||
"sale_price",
|
||||
"allow_discount",
|
||||
"status",
|
||||
]
|
||||
natural_key = ["store_id", "product_id"]
|
||||
|
||||
for rec in records:
|
||||
effective_date = rec.get("updated_time") or rec.get("created_time")
|
||||
|
||||
scd_record = {
|
||||
"store_id": rec["store_id"],
|
||||
"product_id": rec["product_id"],
|
||||
"product_name": rec.get("product_name"),
|
||||
"category_id": rec.get("category_id"),
|
||||
"category_name": rec.get("category_name"),
|
||||
"second_category_id": rec.get("second_category_id"),
|
||||
"cost_price": rec.get("cost_price"),
|
||||
"sale_price": rec.get("sale_price"),
|
||||
"allow_discount": rec.get("allow_discount"),
|
||||
"status": rec.get("status"),
|
||||
# 原表中有 raw_data jsonb 字段，这里直接复用 task 传入的 raw_data
|
||||
"raw_data": rec.get("raw_data"),
|
||||
}
|
||||
|
||||
# 这里我们不强行区分 INSERT/UPDATE/SKIP，对 ETL 统计来说意义不大
|
||||
self.scd_handler.upsert(
|
||||
table_name="billiards.dim_product_price_scd",
|
||||
natural_key=natural_key,
|
||||
tracked_fields=tracked_fields,
|
||||
record=scd_record,
|
||||
effective_date=effective_date,
|
||||
)
|
||||
|
||||
# skipped_count 统一按 0 返回（真正被丢弃的记录在 Task 端已经过滤）
|
||||
return (inserted, updated, 0)
|
||||
0
etl_billiards/loaders/dimensions/table.py
Normal file
0
etl_billiards/loaders/dimensions/table.py
Normal file
0
etl_billiards/loaders/facts/__init__.py
Normal file
0
etl_billiards/loaders/facts/__init__.py
Normal file
42
etl_billiards/loaders/facts/order.py
Normal file
42
etl_billiards/loaders/facts/order.py
Normal file
@@ -0,0 +1,42 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""订单事实表加载器"""
|
||||
from ..base_loader import BaseLoader
|
||||
|
||||
class OrderLoader(BaseLoader):
|
||||
"""订单数据加载器"""
|
||||
|
||||
def upsert_orders(self, records: list, store_id: int) -> tuple:
|
||||
"""加载订单数据"""
|
||||
if not records:
|
||||
return (0, 0, 0)
|
||||
|
||||
sql = """
|
||||
INSERT INTO billiards.fact_order (
|
||||
store_id, order_id, order_no, member_id, table_id,
|
||||
order_time, end_time, total_amount, discount_amount,
|
||||
final_amount, pay_status, order_status, remark, raw_data
|
||||
)
|
||||
VALUES (
|
||||
%(store_id)s, %(order_id)s, %(order_no)s, %(member_id)s, %(table_id)s,
|
||||
%(order_time)s, %(end_time)s, %(total_amount)s, %(discount_amount)s,
|
||||
%(final_amount)s, %(pay_status)s, %(order_status)s, %(remark)s, %(raw_data)s
|
||||
)
|
||||
ON CONFLICT (store_id, order_id) DO UPDATE SET
|
||||
order_no = EXCLUDED.order_no,
|
||||
member_id = EXCLUDED.member_id,
|
||||
table_id = EXCLUDED.table_id,
|
||||
order_time = EXCLUDED.order_time,
|
||||
end_time = EXCLUDED.end_time,
|
||||
total_amount = EXCLUDED.total_amount,
|
||||
discount_amount = EXCLUDED.discount_amount,
|
||||
final_amount = EXCLUDED.final_amount,
|
||||
pay_status = EXCLUDED.pay_status,
|
||||
order_status = EXCLUDED.order_status,
|
||||
remark = EXCLUDED.remark,
|
||||
raw_data = EXCLUDED.raw_data,
|
||||
updated_at = now()
|
||||
RETURNING (xmax = 0) AS inserted
|
||||
"""
|
||||
|
||||
inserted, updated = self.db.batch_upsert_with_returning(sql, records, page_size=self._batch_size())
|
||||
return (inserted, updated, 0)
|
||||
35
etl_billiards/loaders/facts/payment.py
Normal file
35
etl_billiards/loaders/facts/payment.py
Normal file
@@ -0,0 +1,35 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""支付记录加载器"""
|
||||
from ..base_loader import BaseLoader
|
||||
|
||||
class PaymentLoader(BaseLoader):
|
||||
"""支付记录加载器"""
|
||||
|
||||
def upsert_payments(self, records: list, store_id: int) -> tuple:
|
||||
"""加载支付记录"""
|
||||
if not records:
|
||||
return (0, 0, 0)
|
||||
|
||||
sql = """
|
||||
INSERT INTO billiards.fact_payment (
|
||||
store_id, pay_id, order_id, pay_time, pay_amount,
|
||||
pay_type, pay_status, remark, raw_data
|
||||
)
|
||||
VALUES (
|
||||
%(store_id)s, %(pay_id)s, %(order_id)s, %(pay_time)s, %(pay_amount)s,
|
||||
%(pay_type)s, %(pay_status)s, %(remark)s, %(raw_data)s
|
||||
)
|
||||
ON CONFLICT (store_id, pay_id) DO UPDATE SET
|
||||
order_id = EXCLUDED.order_id,
|
||||
pay_time = EXCLUDED.pay_time,
|
||||
pay_amount = EXCLUDED.pay_amount,
|
||||
pay_type = EXCLUDED.pay_type,
|
||||
pay_status = EXCLUDED.pay_status,
|
||||
remark = EXCLUDED.remark,
|
||||
raw_data = EXCLUDED.raw_data,
|
||||
updated_at = now()
|
||||
RETURNING (xmax = 0) AS inserted
|
||||
"""
|
||||
|
||||
inserted, updated = self.db.batch_upsert_with_returning(sql, records, page_size=self._batch_size())
|
||||
return (inserted, updated, 0)
|
||||
0
etl_billiards/models/__init__.py
Normal file
0
etl_billiards/models/__init__.py
Normal file
50
etl_billiards/models/parsers.py
Normal file
50
etl_billiards/models/parsers.py
Normal file
@@ -0,0 +1,50 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据类型解析器"""
|
||||
from datetime import datetime
|
||||
from decimal import Decimal, ROUND_HALF_UP
|
||||
from dateutil import parser as dtparser
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
class TypeParser:
|
||||
"""类型解析工具"""
|
||||
|
||||
@staticmethod
|
||||
def parse_timestamp(s: str, tz: ZoneInfo) -> datetime | None:
|
||||
"""解析时间戳"""
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
dt = dtparser.parse(s)
|
||||
if dt.tzinfo is None:
|
||||
return dt.replace(tzinfo=tz)
|
||||
return dt.astimezone(tz)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def parse_decimal(value, scale: int = 2) -> Decimal | None:
|
||||
"""解析金额"""
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
d = Decimal(str(value))
|
||||
return d.quantize(Decimal(10) ** -scale, rounding=ROUND_HALF_UP)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def parse_int(value) -> int | None:
|
||||
"""解析整数"""
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def format_timestamp(dt: datetime | None, tz: ZoneInfo) -> str | None:
|
||||
"""格式化时间戳"""
|
||||
if not dt:
|
||||
return None
|
||||
return dt.astimezone(tz).strftime("%Y-%m-%d %H:%M:%S")
|
||||
25
etl_billiards/models/validators.py
Normal file
25
etl_billiards/models/validators.py
Normal file
@@ -0,0 +1,25 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据验证器"""
|
||||
from decimal import Decimal
|
||||
|
||||
class DataValidator:
|
||||
"""数据验证工具"""
|
||||
|
||||
@staticmethod
|
||||
def validate_positive_amount(value: Decimal | None, field_name: str = "amount"):
|
||||
"""验证金额为正数"""
|
||||
if value is not None and value < 0:
|
||||
raise ValueError(f"{field_name} 不能为负数: {value}")
|
||||
|
||||
@staticmethod
|
||||
def validate_required(value, field_name: str):
|
||||
"""验证必填字段"""
|
||||
if value is None or value == "":
|
||||
raise ValueError(f"{field_name} 是必填字段")
|
||||
|
||||
@staticmethod
|
||||
def validate_range(value, min_val, max_val, field_name: str):
|
||||
"""验证值范围"""
|
||||
if value is not None:
|
||||
if value < min_val or value > max_val:
|
||||
raise ValueError(f"{field_name} 必须在 {min_val} 到 {max_val} 之间")
|
||||
0
etl_billiards/orchestration/__init__.py
Normal file
0
etl_billiards/orchestration/__init__.py
Normal file
62
etl_billiards/orchestration/cursor_manager.py
Normal file
62
etl_billiards/orchestration/cursor_manager.py
Normal file
@@ -0,0 +1,62 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""游标管理器"""
|
||||
from datetime import datetime
|
||||
|
||||
class CursorManager:
|
||||
"""ETL游标管理"""
|
||||
|
||||
def __init__(self, db_connection):
|
||||
self.db = db_connection
|
||||
|
||||
def get_or_create(self, task_id: int, store_id: int) -> dict:
|
||||
"""获取或创建游标"""
|
||||
rows = self.db.query(
|
||||
"SELECT * FROM etl_admin.etl_cursor WHERE task_id=%s AND store_id=%s",
|
||||
(task_id, store_id)
|
||||
)
|
||||
|
||||
if rows:
|
||||
return rows[0]
|
||||
|
||||
# 创建新游标
|
||||
self.db.execute(
|
||||
"""
|
||||
INSERT INTO etl_admin.etl_cursor(task_id, store_id, last_start, last_end, last_id, extra)
|
||||
VALUES(%s, %s, NULL, NULL, NULL, '{}'::jsonb)
|
||||
""",
|
||||
(task_id, store_id)
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
rows = self.db.query(
|
||||
"SELECT * FROM etl_admin.etl_cursor WHERE task_id=%s AND store_id=%s",
|
||||
(task_id, store_id)
|
||||
)
|
||||
return rows[0] if rows else None
|
||||
|
||||
def advance(self, task_id: int, store_id: int, window_start: datetime,
|
||||
window_end: datetime, run_id: int, last_id: int = None):
|
||||
"""推进游标"""
|
||||
if last_id is not None:
|
||||
sql = """
|
||||
UPDATE etl_admin.etl_cursor
|
||||
SET last_start = %s,
|
||||
last_end = %s,
|
||||
last_id = GREATEST(COALESCE(last_id, 0), %s),
|
||||
last_run_id = %s,
|
||||
updated_at = now()
|
||||
WHERE task_id = %s AND store_id = %s
|
||||
"""
|
||||
self.db.execute(sql, (window_start, window_end, last_id, run_id, task_id, store_id))
|
||||
else:
|
||||
sql = """
|
||||
UPDATE etl_admin.etl_cursor
|
||||
SET last_start = %s,
|
||||
last_end = %s,
|
||||
last_run_id = %s,
|
||||
updated_at = now()
|
||||
WHERE task_id = %s AND store_id = %s
|
||||
"""
|
||||
self.db.execute(sql, (window_start, window_end, run_id, task_id, store_id))
|
||||
|
||||
self.db.commit()
|
||||
70
etl_billiards/orchestration/run_tracker.py
Normal file
70
etl_billiards/orchestration/run_tracker.py
Normal file
@@ -0,0 +1,70 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""运行记录追踪器"""
|
||||
import json
|
||||
from datetime import datetime
|
||||
|
||||
class RunTracker:
|
||||
"""ETL运行记录管理"""
|
||||
|
||||
def __init__(self, db_connection):
|
||||
self.db = db_connection
|
||||
|
||||
def create_run(self, task_id: int, store_id: int, run_uuid: str,
|
||||
export_dir: str, log_path: str, status: str,
|
||||
window_start: datetime = None, window_end: datetime = None,
|
||||
window_minutes: int = None, overlap_seconds: int = None,
|
||||
request_params: dict = None) -> int:
|
||||
"""创建运行记录"""
|
||||
sql = """
|
||||
INSERT INTO etl_admin.etl_run(
|
||||
run_uuid, task_id, store_id, status, started_at, window_start, window_end,
|
||||
window_minutes, overlap_seconds, fetched_count, loaded_count, updated_count,
|
||||
skipped_count, error_count, unknown_fields, export_dir, log_path,
|
||||
request_params, manifest, error_message, extra
|
||||
) VALUES (
|
||||
%s, %s, %s, %s, now(), %s, %s, %s, %s, 0, 0, 0, 0, 0, 0, %s, %s, %s,
|
||||
'{}'::jsonb, NULL, '{}'::jsonb
|
||||
)
|
||||
RETURNING run_id
|
||||
"""
|
||||
|
||||
result = self.db.query(
|
||||
sql,
|
||||
(run_uuid, task_id, store_id, status, window_start, window_end,
|
||||
window_minutes, overlap_seconds, export_dir, log_path,
|
||||
json.dumps(request_params or {}, ensure_ascii=False))
|
||||
)
|
||||
|
||||
run_id = result[0]["run_id"]
|
||||
self.db.commit()
|
||||
return run_id
|
||||
|
||||
def update_run(self, run_id: int, counts: dict, status: str,
|
||||
ended_at: datetime = None, manifest: dict = None,
|
||||
error_message: str = None):
|
||||
"""更新运行记录"""
|
||||
sql = """
|
||||
UPDATE etl_admin.etl_run
|
||||
SET fetched_count = %s,
|
||||
loaded_count = %s,
|
||||
updated_count = %s,
|
||||
skipped_count = %s,
|
||||
error_count = %s,
|
||||
unknown_fields = %s,
|
||||
status = %s,
|
||||
ended_at = %s,
|
||||
manifest = %s,
|
||||
error_message = %s
|
||||
WHERE run_id = %s
|
||||
"""
|
||||
|
||||
self.db.execute(
|
||||
sql,
|
||||
(counts.get("fetched", 0), counts.get("inserted", 0),
|
||||
counts.get("updated", 0), counts.get("skipped", 0),
|
||||
counts.get("errors", 0), counts.get("unknown_fields", 0),
|
||||
status, ended_at,
|
||||
json.dumps(manifest or {}, ensure_ascii=False),
|
||||
error_message, run_id)
|
||||
)
|
||||
self.db.commit()
|
||||
131
etl_billiards/orchestration/scheduler.py
Normal file
131
etl_billiards/orchestration/scheduler.py
Normal file
@@ -0,0 +1,131 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""ETL调度器"""
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from database.connection import DatabaseConnection
|
||||
from database.operations import DatabaseOperations
|
||||
from api.client import APIClient
|
||||
from orchestration.cursor_manager import CursorManager
|
||||
from orchestration.run_tracker import RunTracker
|
||||
from orchestration.task_registry import default_registry
|
||||
|
||||
class ETLScheduler:
|
||||
"""ETL任务调度器"""
|
||||
|
||||
def __init__(self, config, logger):
|
||||
self.config = config
|
||||
self.logger = logger
|
||||
self.tz = ZoneInfo(config.get("app.timezone", "Asia/Taipei"))
|
||||
|
||||
# 初始化组件
|
||||
self.db_conn = DatabaseConnection(
|
||||
dsn=config["db"]["dsn"],
|
||||
session=config["db"].get("session"),
|
||||
connect_timeout=config["db"].get("connect_timeout_sec")
|
||||
)
|
||||
self.db_ops = DatabaseOperations(self.db_conn)
|
||||
|
||||
self.api_client = APIClient(
|
||||
base_url=config["api"]["base_url"],
|
||||
token=config["api"]["token"],
|
||||
timeout=config["api"]["timeout_sec"],
|
||||
retry_max=config["api"]["retries"]["max_attempts"],
|
||||
headers_extra=config["api"].get("headers_extra")
|
||||
)
|
||||
|
||||
self.cursor_mgr = CursorManager(self.db_conn)
|
||||
self.run_tracker = RunTracker(self.db_conn)
|
||||
self.task_registry = default_registry
|
||||
|
||||
def run_tasks(self, task_codes: list = None):
|
||||
"""运行任务列表"""
|
||||
run_uuid = uuid.uuid4().hex
|
||||
store_id = self.config.get("app.store_id")
|
||||
|
||||
if not task_codes:
|
||||
task_codes = self.config.get("run.tasks", [])
|
||||
|
||||
self.logger.info(f"开始运行任务: {task_codes}, run_uuid={run_uuid}")
|
||||
|
||||
for task_code in task_codes:
|
||||
try:
|
||||
self._run_single_task(task_code, run_uuid, store_id)
|
||||
except Exception as e:
|
||||
self.logger.error(f"任务 {task_code} 失败: {e}", exc_info=True)
|
||||
continue
|
||||
|
||||
self.logger.info("所有任务执行完成")
|
||||
|
||||
def _run_single_task(self, task_code: str, run_uuid: str, store_id: int):
|
||||
"""运行单个任务"""
|
||||
# 创建任务实例
|
||||
task = self.task_registry.create_task(
|
||||
task_code, self.config, self.db_ops, self.api_client, self.logger
|
||||
)
|
||||
|
||||
# 获取任务配置（从数据库）
|
||||
task_cfg = self._load_task_config(task_code, store_id)
|
||||
if not task_cfg:
|
||||
self.logger.warning(f"任务 {task_code} 未启用或不存在")
|
||||
return
|
||||
|
||||
task_id = task_cfg["task_id"]
|
||||
|
||||
# 创建运行记录
|
||||
export_dir = Path(self.config["io"]["export_root"]) / datetime.now(self.tz).strftime("%Y%m%d")
|
||||
log_path = str(Path(self.config["io"]["log_root"]) / f"{run_uuid}.log")
|
||||
|
||||
run_id = self.run_tracker.create_run(
|
||||
task_id=task_id,
|
||||
store_id=store_id,
|
||||
run_uuid=run_uuid,
|
||||
export_dir=str(export_dir),
|
||||
log_path=log_path,
|
||||
status="RUNNING"
|
||||
)
|
||||
|
||||
# 执行任务
|
||||
try:
|
||||
result = task.execute()
|
||||
|
||||
# 更新运行记录
|
||||
self.run_tracker.update_run(
|
||||
run_id=run_id,
|
||||
counts=result["counts"],
|
||||
status=result["status"],
|
||||
ended_at=datetime.now(self.tz)
|
||||
)
|
||||
|
||||
# 推进游标
|
||||
if result["status"] == "SUCCESS":
|
||||
# TODO: 从任务结果中获取窗口信息
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
self.run_tracker.update_run(
|
||||
run_id=run_id,
|
||||
counts={},
|
||||
status="FAIL",
|
||||
ended_at=datetime.now(self.tz),
|
||||
error_message=str(e)
|
||||
)
|
||||
raise
|
||||
|
||||
def _load_task_config(self, task_code: str, store_id: int) -> dict:
|
||||
"""从数据库加载任务配置"""
|
||||
sql = """
|
||||
SELECT task_id, task_code, store_id, enabled, cursor_field,
|
||||
window_minutes_default, overlap_seconds, page_size, retry_max, params
|
||||
FROM etl_admin.etl_task
|
||||
WHERE store_id = %s AND task_code = %s AND enabled = TRUE
|
||||
"""
|
||||
|
||||
rows = self.db_conn.query(sql, (store_id, task_code))
|
||||
return rows[0] if rows else None
|
||||
|
||||
def close(self):
|
||||
"""关闭连接"""
|
||||
self.db_conn.close()
|
||||
36
etl_billiards/orchestration/task_registry.py
Normal file
36
etl_billiards/orchestration/task_registry.py
Normal file
@@ -0,0 +1,36 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""任务注册表"""
|
||||
from tasks.orders_task import OrdersTask
|
||||
from tasks.payments_task import PaymentsTask
|
||||
from tasks.members_task import MembersTask
|
||||
|
||||
class TaskRegistry:
|
||||
"""任务注册和工厂"""
|
||||
|
||||
def __init__(self):
|
||||
self._tasks = {}
|
||||
|
||||
def register(self, task_code: str, task_class):
|
||||
"""注册任务类"""
|
||||
self._tasks[task_code.upper()] = task_class
|
||||
|
||||
def create_task(self, task_code: str, config, db_connection, api_client, logger):
|
||||
"""创建任务实例"""
|
||||
task_code = task_code.upper()
|
||||
if task_code not in self._tasks:
|
||||
raise ValueError(f"未知的任务类型: {task_code}")
|
||||
|
||||
task_class = self._tasks[task_code]
|
||||
return task_class(config, db_connection, api_client, logger)
|
||||
|
||||
def get_all_task_codes(self) -> list:
|
||||
"""获取所有已注册的任务代码"""
|
||||
return list(self._tasks.keys())
|
||||
|
||||
|
||||
# 默认注册表
|
||||
default_registry = TaskRegistry()
|
||||
default_registry.register("ORDERS", OrdersTask)
|
||||
default_registry.register("PAYMENTS", PaymentsTask)
|
||||
default_registry.register("MEMBERS", MembersTask)
|
||||
# 可以继续注册其他任务...
|
||||
0
etl_billiards/quality/__init__.py
Normal file
0
etl_billiards/quality/__init__.py
Normal file
73
etl_billiards/quality/balance_checker.py
Normal file
73
etl_billiards/quality/balance_checker.py
Normal file
@@ -0,0 +1,73 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""余额一致性检查器"""
|
||||
from .base_checker import BaseDataQualityChecker
|
||||
|
||||
class BalanceChecker(BaseDataQualityChecker):
|
||||
"""检查订单、支付、退款的金额一致性"""
|
||||
|
||||
def check(self, store_id: int, start_date: str, end_date: str) -> dict:
|
||||
"""
|
||||
检查指定时间范围内的余额一致性
|
||||
|
||||
验证: 订单总额 = 支付总额 - 退款总额
|
||||
"""
|
||||
checks = []
|
||||
|
||||
# 查询订单总额
|
||||
sql_orders = """
|
||||
SELECT COALESCE(SUM(final_amount), 0) AS total
|
||||
FROM billiards.fact_order
|
||||
WHERE store_id = %s
|
||||
AND order_time >= %s
|
||||
AND order_time < %s
|
||||
AND order_status = 'COMPLETED'
|
||||
"""
|
||||
order_total = self.db.query(sql_orders, (store_id, start_date, end_date))[0]["total"]
|
||||
|
||||
# 查询支付总额
|
||||
sql_payments = """
|
||||
SELECT COALESCE(SUM(pay_amount), 0) AS total
|
||||
FROM billiards.fact_payment
|
||||
WHERE store_id = %s
|
||||
AND pay_time >= %s
|
||||
AND pay_time < %s
|
||||
AND pay_status = 'SUCCESS'
|
||||
"""
|
||||
payment_total = self.db.query(sql_payments, (store_id, start_date, end_date))[0]["total"]
|
||||
|
||||
# 查询退款总额
|
||||
sql_refunds = """
|
||||
SELECT COALESCE(SUM(refund_amount), 0) AS total
|
||||
FROM billiards.fact_refund
|
||||
WHERE store_id = %s
|
||||
AND refund_time >= %s
|
||||
AND refund_time < %s
|
||||
AND refund_status = 'SUCCESS'
|
||||
"""
|
||||
refund_total = self.db.query(sql_refunds, (store_id, start_date, end_date))[0]["total"]
|
||||
|
||||
# 验证余额
|
||||
expected_total = payment_total - refund_total
|
||||
diff = abs(float(order_total) - float(expected_total))
|
||||
threshold = 0.01 # 1分钱的容差
|
||||
|
||||
passed = diff < threshold
|
||||
|
||||
checks.append({
|
||||
"name": "balance_consistency",
|
||||
"passed": passed,
|
||||
"message": f"订单总额: {order_total}, 支付-退款: {expected_total}, 差异: {diff}",
|
||||
"details": {
|
||||
"order_total": float(order_total),
|
||||
"payment_total": float(payment_total),
|
||||
"refund_total": float(refund_total),
|
||||
"diff": diff
|
||||
}
|
||||
})
|
||||
|
||||
all_passed = all(c["passed"] for c in checks)
|
||||
|
||||
return {
|
||||
"passed": all_passed,
|
||||
"checks": checks
|
||||
}
|
||||
19
etl_billiards/quality/base_checker.py
Normal file
19
etl_billiards/quality/base_checker.py
Normal file
@@ -0,0 +1,19 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据质量检查器基类"""
|
||||
|
||||
class BaseDataQualityChecker:
|
||||
"""数据质量检查器基类"""
|
||||
|
||||
def __init__(self, db_connection, logger):
|
||||
self.db = db_connection
|
||||
self.logger = logger
|
||||
|
||||
def check(self) -> dict:
|
||||
"""
|
||||
执行质量检查
|
||||
返回: {
|
||||
"passed": bool,
|
||||
"checks": [{"name": str, "passed": bool, "message": str}]
|
||||
}
|
||||
"""
|
||||
raise NotImplementedError("子类需实现 check 方法")
|
||||
5
etl_billiards/requirements.txt
Normal file
5
etl_billiards/requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
# Python依赖包
|
||||
psycopg2-binary>=2.9.0
|
||||
requests>=2.28.0
|
||||
python-dateutil>=2.8.0
|
||||
tzdata>=2023.0
|
||||
0
etl_billiards/scd/__init__.py
Normal file
0
etl_billiards/scd/__init__.py
Normal file
92
etl_billiards/scd/scd2_handler.py
Normal file
92
etl_billiards/scd/scd2_handler.py
Normal file
@@ -0,0 +1,92 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""SCD2 (Slowly Changing Dimension Type 2) 处理器"""
|
||||
from datetime import datetime
|
||||
|
||||
class SCD2Handler:
|
||||
"""SCD2历史记录处理器"""
|
||||
|
||||
def __init__(self, db_ops):
|
||||
self.db = db_ops
|
||||
|
||||
def upsert(self, table_name: str, natural_key: list, tracked_fields: list,
|
||||
record: dict, effective_date: datetime = None) -> str:
|
||||
"""
|
||||
处理SCD2更新
|
||||
|
||||
Args:
|
||||
table_name: 表名
|
||||
natural_key: 自然键字段列表
|
||||
tracked_fields: 需要跟踪变化的字段列表
|
||||
record: 记录数据
|
||||
effective_date: 生效日期
|
||||
|
||||
Returns:
|
||||
操作类型: 'INSERT', 'UPDATE', 'UNCHANGED'
|
||||
"""
|
||||
effective_date = effective_date or datetime.now()
|
||||
|
||||
# 查找当前有效记录
|
||||
where_clause = " AND ".join([f"{k} = %({k})s" for k in natural_key])
|
||||
sql_select = f"""
|
||||
SELECT * FROM {table_name}
|
||||
WHERE {where_clause}
|
||||
AND valid_to IS NULL
|
||||
"""
|
||||
|
||||
# 使用 db 的 connection
|
||||
current = self.db.conn.cursor()
|
||||
current.execute(sql_select, record)
|
||||
existing = current.fetchone()
|
||||
|
||||
if not existing:
|
||||
# 新记录：直接插入
|
||||
record["valid_from"] = effective_date
|
||||
record["valid_to"] = None
|
||||
record["is_current"] = True
|
||||
|
||||
fields = list(record.keys())
|
||||
placeholders = ", ".join([f"%({f})s" for f in fields])
|
||||
sql_insert = f"""
|
||||
INSERT INTO {table_name} ({', '.join(fields)})
|
||||
VALUES ({placeholders})
|
||||
"""
|
||||
current.execute(sql_insert, record)
|
||||
return 'INSERT'
|
||||
|
||||
# 检查是否有变化
|
||||
has_changes = any(
|
||||
existing.get(field) != record.get(field)
|
||||
for field in tracked_fields
|
||||
)
|
||||
|
||||
if not has_changes:
|
||||
return 'UNCHANGED'
|
||||
|
||||
# 有变化：关闭旧记录，插入新记录
|
||||
update_where = " AND ".join([f"{k} = %({k})s" for k in natural_key])
|
||||
sql_close = f"""
|
||||
UPDATE {table_name}
|
||||
SET valid_to = %(effective_date)s,
|
||||
is_current = FALSE
|
||||
WHERE {update_where}
|
||||
AND valid_to IS NULL
|
||||
"""
|
||||
record["effective_date"] = effective_date
|
||||
current.execute(sql_close, record)
|
||||
|
||||
# 插入新记录
|
||||
record["valid_from"] = effective_date
|
||||
record["valid_to"] = None
|
||||
record["is_current"] = True
|
||||
|
||||
fields = list(record.keys())
|
||||
if "effective_date" in fields:
|
||||
fields.remove("effective_date")
|
||||
placeholders = ", ".join([f"%({f})s" for f in fields])
|
||||
sql_insert = f"""
|
||||
INSERT INTO {table_name} ({', '.join(fields)})
|
||||
VALUES ({placeholders})
|
||||
"""
|
||||
current.execute(sql_insert, record)
|
||||
|
||||
return 'UPDATE'
|
||||
30
etl_billiards/setup.py
Normal file
30
etl_billiards/setup.py
Normal file
@@ -0,0 +1,30 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
Setup script for ETL Billiards
|
||||
"""
|
||||
from setuptools import setup, find_packages
|
||||
|
||||
with open("requirements.txt") as f:
|
||||
requirements = f.read().splitlines()
|
||||
|
||||
setup(
|
||||
name="etl-billiards",
|
||||
version="2.0.0",
|
||||
description="Modular ETL system for billiards business data",
|
||||
author="Data Platform Team",
|
||||
author_email="data-platform@example.com",
|
||||
packages=find_packages(),
|
||||
install_requires=requirements,
|
||||
python_requires=">=3.10",
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"etl-billiards=cli.main:main",
|
||||
],
|
||||
},
|
||||
classifiers=[
|
||||
"Development Status :: 4 - Beta",
|
||||
"Intended Audience :: Developers",
|
||||
"Programming Language :: Python :: 3.10",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
],
|
||||
)
|
||||
0
etl_billiards/tasks/__init__.py
Normal file
0
etl_billiards/tasks/__init__.py
Normal file
62
etl_billiards/tasks/base_task.py
Normal file
62
etl_billiards/tasks/base_task.py
Normal file
@@ -0,0 +1,62 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""ETL任务基类"""
|
||||
from datetime import datetime, timedelta
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
class BaseTask:
|
||||
"""ETL任务基类"""
|
||||
|
||||
def __init__(self, config, db_connection, api_client, logger):
|
||||
self.config = config
|
||||
self.db = db_connection
|
||||
self.api = api_client
|
||||
self.logger = logger
|
||||
self.tz = ZoneInfo(config.get("app.timezone", "Asia/Taipei"))
|
||||
|
||||
def get_task_code(self) -> str:
|
||||
"""获取任务代码"""
|
||||
raise NotImplementedError("子类需实现 get_task_code 方法")
|
||||
|
||||
def execute(self) -> dict:
|
||||
"""执行任务"""
|
||||
raise NotImplementedError("子类需实现 execute 方法")
|
||||
|
||||
def _get_time_window(self, cursor_data: dict = None) -> tuple:
|
||||
"""计算时间窗口"""
|
||||
now = datetime.now(self.tz)
|
||||
|
||||
# 判断是否在闲时窗口
|
||||
idle_start = self.config.get("run.idle_window.start", "04:00")
|
||||
idle_end = self.config.get("run.idle_window.end", "16:00")
|
||||
|
||||
is_idle = self._is_in_idle_window(now, idle_start, idle_end)
|
||||
|
||||
# 获取窗口大小
|
||||
if is_idle:
|
||||
window_minutes = self.config.get("run.window_minutes.default_idle", 180)
|
||||
else:
|
||||
window_minutes = self.config.get("run.window_minutes.default_busy", 30)
|
||||
|
||||
# 计算窗口
|
||||
overlap_seconds = self.config.get("run.overlap_seconds", 120)
|
||||
|
||||
if cursor_data and cursor_data.get("last_end"):
|
||||
window_start = cursor_data["last_end"] - timedelta(seconds=overlap_seconds)
|
||||
else:
|
||||
window_start = now - timedelta(minutes=window_minutes)
|
||||
|
||||
window_end = now
|
||||
|
||||
return window_start, window_end, window_minutes
|
||||
|
||||
def _is_in_idle_window(self, dt: datetime, start_time: str, end_time: str) -> bool:
|
||||
"""判断是否在闲时窗口"""
|
||||
current_time = dt.strftime("%H:%M")
|
||||
return start_time <= current_time <= end_time
|
||||
|
||||
def _build_result(self, status: str, counts: dict) -> dict:
|
||||
"""构建结果字典"""
|
||||
return {
|
||||
"status": status,
|
||||
"counts": counts
|
||||
}
|
||||
73
etl_billiards/tasks/members_task.py
Normal file
73
etl_billiards/tasks/members_task.py
Normal file
@@ -0,0 +1,73 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""会员ETL任务"""
|
||||
import json
|
||||
from .base_task import BaseTask
|
||||
from loaders.dimensions.member import MemberLoader
|
||||
from models.parsers import TypeParser
|
||||
|
||||
class MembersTask(BaseTask):
|
||||
"""会员ETL任务"""
|
||||
|
||||
def get_task_code(self) -> str:
|
||||
return "MEMBERS"
|
||||
|
||||
def execute(self) -> dict:
|
||||
"""执行会员ETL"""
|
||||
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
|
||||
|
||||
params = {
|
||||
"storeId": self.config.get("app.store_id"),
|
||||
}
|
||||
|
||||
try:
|
||||
records, pages_meta = self.api.get_paginated(
|
||||
endpoint="/MemberProfile/GetTenantMemberList",
|
||||
params=params,
|
||||
page_size=self.config.get("api.page_size", 200),
|
||||
data_path=("data",)
|
||||
)
|
||||
|
||||
parsed_records = []
|
||||
for rec in records:
|
||||
parsed = self._parse_member(rec)
|
||||
if parsed:
|
||||
parsed_records.append(parsed)
|
||||
|
||||
loader = MemberLoader(self.db)
|
||||
store_id = self.config.get("app.store_id")
|
||||
inserted, updated, skipped = loader.upsert_members(parsed_records, store_id)
|
||||
|
||||
self.db.commit()
|
||||
|
||||
counts = {
|
||||
"fetched": len(records),
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"errors": 0
|
||||
}
|
||||
|
||||
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
|
||||
return self._build_result("SUCCESS", counts)
|
||||
|
||||
except Exception as e:
|
||||
self.db.rollback()
|
||||
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
|
||||
raise
|
||||
|
||||
def _parse_member(self, raw: dict) -> dict:
|
||||
"""解析会员记录"""
|
||||
try:
|
||||
return {
|
||||
"store_id": self.config.get("app.store_id"),
|
||||
"member_id": TypeParser.parse_int(raw.get("memberId")),
|
||||
"member_name": raw.get("memberName"),
|
||||
"phone": raw.get("phone"),
|
||||
"balance": TypeParser.parse_decimal(raw.get("balance")),
|
||||
"status": raw.get("status"),
|
||||
"register_time": TypeParser.parse_timestamp(raw.get("registerTime"), self.tz),
|
||||
"raw_data": json.dumps(raw, ensure_ascii=False)
|
||||
}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"解析会员记录失败: {e}, 原始数据: {raw}")
|
||||
return None
|
||||
94
etl_billiards/tasks/orders_task.py
Normal file
94
etl_billiards/tasks/orders_task.py
Normal file
@@ -0,0 +1,94 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""订单ETL任务"""
|
||||
import json
|
||||
from .base_task import BaseTask
|
||||
from loaders.facts.order import OrderLoader
|
||||
from models.parsers import TypeParser
|
||||
|
||||
class OrdersTask(BaseTask):
|
||||
"""订单数据ETL任务"""
|
||||
|
||||
def get_task_code(self) -> str:
|
||||
return "ORDERS"
|
||||
|
||||
def execute(self) -> dict:
|
||||
"""执行订单数据ETL"""
|
||||
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
|
||||
|
||||
# 1. 获取时间窗口
|
||||
window_start, window_end, window_minutes = self._get_time_window()
|
||||
|
||||
# 2. 调用API获取数据
|
||||
params = {
|
||||
"storeId": self.config.get("app.store_id"),
|
||||
"startTime": TypeParser.format_timestamp(window_start, self.tz),
|
||||
"endTime": TypeParser.format_timestamp(window_end, self.tz),
|
||||
}
|
||||
|
||||
try:
|
||||
records, pages_meta = self.api.get_paginated(
|
||||
endpoint="/order/list",
|
||||
params=params,
|
||||
page_size=self.config.get("api.page_size", 200),
|
||||
data_path=("data",)
|
||||
)
|
||||
|
||||
# 3. 解析并清洗数据
|
||||
parsed_records = []
|
||||
for rec in records:
|
||||
parsed = self._parse_order(rec)
|
||||
if parsed:
|
||||
parsed_records.append(parsed)
|
||||
|
||||
# 4. 加载数据
|
||||
loader = OrderLoader(self.db)
|
||||
store_id = self.config.get("app.store_id")
|
||||
inserted, updated, skipped = loader.upsert_orders(
|
||||
parsed_records,
|
||||
store_id
|
||||
)
|
||||
|
||||
# 5. 提交事务
|
||||
self.db.commit()
|
||||
|
||||
counts = {
|
||||
"fetched": len(records),
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"errors": 0
|
||||
}
|
||||
|
||||
self.logger.info(
|
||||
f"{self.get_task_code()} 完成: {counts}"
|
||||
)
|
||||
|
||||
return self._build_result("SUCCESS", counts)
|
||||
|
||||
except Exception as e:
|
||||
self.db.rollback()
|
||||
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
|
||||
raise
|
||||
|
||||
def _parse_order(self, raw: dict) -> dict:
|
||||
"""解析单条订单记录"""
|
||||
try:
|
||||
return {
|
||||
"store_id": self.config.get("app.store_id"),
|
||||
"order_id": TypeParser.parse_int(raw.get("orderId")),
|
||||
"order_no": raw.get("orderNo"),
|
||||
"member_id": TypeParser.parse_int(raw.get("memberId")),
|
||||
"table_id": TypeParser.parse_int(raw.get("tableId")),
|
||||
"order_time": TypeParser.parse_timestamp(raw.get("orderTime"), self.tz),
|
||||
"end_time": TypeParser.parse_timestamp(raw.get("endTime"), self.tz),
|
||||
"total_amount": TypeParser.parse_decimal(raw.get("totalAmount")),
|
||||
"discount_amount": TypeParser.parse_decimal(raw.get("discountAmount")),
|
||||
"final_amount": TypeParser.parse_decimal(raw.get("finalAmount")),
|
||||
"pay_status": raw.get("payStatus"),
|
||||
"order_status": raw.get("orderStatus"),
|
||||
"remark": raw.get("remark"),
|
||||
"raw_data": json.dumps(raw, ensure_ascii=False)
|
||||
}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"解析订单失败: {e}, 原始数据: {raw}")
|
||||
return None
|
||||
78
etl_billiards/tasks/payments_task.py
Normal file
78
etl_billiards/tasks/payments_task.py
Normal file
@@ -0,0 +1,78 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""支付记录ETL任务"""
|
||||
import json
|
||||
from .base_task import BaseTask
|
||||
from loaders.facts.payment import PaymentLoader
|
||||
from models.parsers import TypeParser
|
||||
|
||||
class PaymentsTask(BaseTask):
|
||||
"""支付记录ETL任务"""
|
||||
|
||||
def get_task_code(self) -> str:
|
||||
return "PAYMENTS"
|
||||
|
||||
def execute(self) -> dict:
|
||||
"""执行支付记录ETL"""
|
||||
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
|
||||
|
||||
window_start, window_end, window_minutes = self._get_time_window()
|
||||
|
||||
params = {
|
||||
"storeId": self.config.get("app.store_id"),
|
||||
"startTime": TypeParser.format_timestamp(window_start, self.tz),
|
||||
"endTime": TypeParser.format_timestamp(window_end, self.tz),
|
||||
}
|
||||
|
||||
try:
|
||||
records, pages_meta = self.api.get_paginated(
|
||||
endpoint="/pay/records",
|
||||
params=params,
|
||||
page_size=self.config.get("api.page_size", 200),
|
||||
data_path=("data",)
|
||||
)
|
||||
|
||||
parsed_records = []
|
||||
for rec in records:
|
||||
parsed = self._parse_payment(rec)
|
||||
if parsed:
|
||||
parsed_records.append(parsed)
|
||||
|
||||
loader = PaymentLoader(self.db)
|
||||
store_id = self.config.get("app.store_id")
|
||||
inserted, updated, skipped = loader.upsert_payments(parsed_records, store_id)
|
||||
|
||||
self.db.commit()
|
||||
|
||||
counts = {
|
||||
"fetched": len(records),
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"errors": 0
|
||||
}
|
||||
|
||||
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
|
||||
return self._build_result("SUCCESS", counts)
|
||||
|
||||
except Exception as e:
|
||||
self.db.rollback()
|
||||
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
|
||||
raise
|
||||
|
||||
def _parse_payment(self, raw: dict) -> dict:
|
||||
"""解析支付记录"""
|
||||
try:
|
||||
return {
|
||||
"store_id": self.config.get("app.store_id"),
|
||||
"pay_id": TypeParser.parse_int(raw.get("payId")),
|
||||
"order_id": TypeParser.parse_int(raw.get("orderId")),
|
||||
"pay_time": TypeParser.parse_timestamp(raw.get("payTime"), self.tz),
|
||||
"pay_amount": TypeParser.parse_decimal(raw.get("payAmount")),
|
||||
"pay_type": raw.get("payType"),
|
||||
"pay_status": raw.get("payStatus"),
|
||||
"remark": raw.get("remark"),
|
||||
"raw_data": json.dumps(raw, ensure_ascii=False)
|
||||
}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"解析支付记录失败: {e}, 原始数据: {raw}")
|
||||
return None
|
||||
139
etl_billiards/tasks/products_task.py
Normal file
139
etl_billiards/tasks/products_task.py
Normal file
@@ -0,0 +1,139 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""商品档案（PRODUCTS）ETL任务"""
|
||||
|
||||
import json
|
||||
|
||||
from .base_task import BaseTask
|
||||
from loaders.dimensions.product import ProductLoader
|
||||
from models.parsers import TypeParser
|
||||
|
||||
|
||||
class ProductsTask(BaseTask):
|
||||
"""商品维度 ETL 任务"""
|
||||
|
||||
def get_task_code(self) -> str:
|
||||
"""任务代码，应与 etl_admin.etl_task.task_code 一致"""
|
||||
return "PRODUCTS"
|
||||
|
||||
def execute(self) -> dict:
|
||||
"""
|
||||
执行商品档案 ETL
|
||||
|
||||
流程：
|
||||
1. 调用上游 /TenantGoods/QueryTenantGoods 分页拉取商品列表
|
||||
2. 解析/清洗字段
|
||||
3. 通过 ProductLoader 写入 dim_product 和 dim_product_price_scd
|
||||
"""
|
||||
self.logger.info(f"开始执行 {self.get_task_code()} 任务")
|
||||
|
||||
params = {
|
||||
"storeId": self.config.get("app.store_id"),
|
||||
}
|
||||
|
||||
try:
|
||||
# 1. 分页拉取数据
|
||||
records, pages_meta = self.api.get_paginated(
|
||||
endpoint="/TenantGoods/QueryTenantGoods",
|
||||
params=params,
|
||||
page_size=self.config.get("api.page_size", 200),
|
||||
data_path=("data",),
|
||||
)
|
||||
|
||||
# 2. 解析/清洗
|
||||
parsed_records = []
|
||||
for raw in records:
|
||||
parsed = self._parse_product(raw)
|
||||
if parsed:
|
||||
parsed_records.append(parsed)
|
||||
|
||||
# 3. 加载入库（维度主表 + 价格SCD2）
|
||||
loader = ProductLoader(self.db)
|
||||
store_id = self.config.get("app.store_id")
|
||||
inserted, updated, skipped = loader.upsert_products(
|
||||
parsed_records, store_id
|
||||
)
|
||||
|
||||
# 4. 提交事务
|
||||
self.db.commit()
|
||||
|
||||
counts = {
|
||||
"fetched": len(records),
|
||||
"inserted": inserted,
|
||||
"updated": updated,
|
||||
"skipped": skipped,
|
||||
"errors": 0,
|
||||
}
|
||||
|
||||
self.logger.info(f"{self.get_task_code()} 完成: {counts}")
|
||||
return self._build_result("SUCCESS", counts)
|
||||
|
||||
except Exception:
|
||||
# 明确回滚，避免部分成功
|
||||
self.db.rollback()
|
||||
self.logger.error(f"{self.get_task_code()} 失败", exc_info=True)
|
||||
raise
|
||||
|
||||
def _parse_product(self, raw: dict) -> dict | None:
|
||||
"""
|
||||
解析单条商品记录，字段映射参考旧版 upsert_dim_product_and_price_scd
|
||||
|
||||
上游字段示例：
|
||||
- siteGoodsId / tenantGoodsId / productId
|
||||
- goodsName / productName
|
||||
- tenantGoodsCategoryId / goodsCategoryId / categoryName / goodsCategorySecondId
|
||||
- goodsUnit
|
||||
- costPrice / goodsPrice / salePrice
|
||||
- goodsState / status
|
||||
- supplierId / barcode / isCombo
|
||||
- createTime / updateTime
|
||||
"""
|
||||
try:
|
||||
product_id = (
|
||||
TypeParser.parse_int(
|
||||
raw.get("siteGoodsId")
|
||||
or raw.get("tenantGoodsId")
|
||||
or raw.get("productId")
|
||||
)
|
||||
)
|
||||
if not product_id:
|
||||
# 主键缺失，直接跳过
|
||||
return None
|
||||
|
||||
return {
|
||||
"store_id": self.config.get("app.store_id"),
|
||||
"product_id": product_id,
|
||||
"site_product_id": TypeParser.parse_int(raw.get("siteGoodsId")),
|
||||
"product_name": raw.get("goodsName") or raw.get("productName"),
|
||||
"category_id": TypeParser.parse_int(
|
||||
raw.get("tenantGoodsCategoryId") or raw.get("goodsCategoryId")
|
||||
),
|
||||
"category_name": raw.get("categoryName"),
|
||||
"second_category_id": TypeParser.parse_int(
|
||||
raw.get("goodsCategorySecondId")
|
||||
),
|
||||
"unit": raw.get("goodsUnit"),
|
||||
"cost_price": TypeParser.parse_decimal(raw.get("costPrice")),
|
||||
"sale_price": TypeParser.parse_decimal(
|
||||
raw.get("goodsPrice") or raw.get("salePrice")
|
||||
),
|
||||
# 旧版这里就是 None，如后面有明确字段可以再补
|
||||
"allow_discount": None,
|
||||
"status": raw.get("goodsState") or raw.get("status"),
|
||||
"supplier_id": TypeParser.parse_int(raw.get("supplierId"))
|
||||
if raw.get("supplierId")
|
||||
else None,
|
||||
"barcode": raw.get("barcode"),
|
||||
"is_combo": bool(raw.get("isCombo"))
|
||||
if raw.get("isCombo") is not None
|
||||
else None,
|
||||
"created_time": TypeParser.parse_timestamp(
|
||||
raw.get("createTime"), self.tz
|
||||
),
|
||||
"updated_time": TypeParser.parse_timestamp(
|
||||
raw.get("updateTime"), self.tz
|
||||
),
|
||||
"raw_data": json.dumps(raw, ensure_ascii=False),
|
||||
}
|
||||
except Exception as e:
|
||||
self.logger.warning(f"解析商品记录失败: {e}, 原始数据: {raw}")
|
||||
return None
|
||||
4
etl_billiards/tasks/tables_task.py
Normal file
4
etl_billiards/tasks/tables_task.py
Normal file
@@ -0,0 +1,4 @@
|
||||
class TablesTask(BaseTask):
|
||||
def get_task_code(self) -> str: # 返回 "TABLES"
|
||||
def execute(self) -> dict: # 拉取 /Table/GetSiteTables
|
||||
def _parse_table(self, raw: dict) -> dict | None:
|
||||
0
etl_billiards/tests/__init__.py
Normal file
0
etl_billiards/tests/__init__.py
Normal file
0
etl_billiards/tests/integration/__init__.py
Normal file
0
etl_billiards/tests/integration/__init__.py
Normal file
33
etl_billiards/tests/integration/test_database.py
Normal file
33
etl_billiards/tests/integration/test_database.py
Normal file
@@ -0,0 +1,33 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""数据库集成测试"""
|
||||
import pytest
|
||||
from database.connection import DatabaseConnection
|
||||
from database.operations import DatabaseOperations
|
||||
|
||||
# 注意：这些测试需要实际的数据库连接
|
||||
# 在CI/CD环境中应使用测试数据库
|
||||
|
||||
@pytest.fixture
|
||||
def db_connection():
|
||||
"""数据库连接fixture"""
|
||||
# 从环境变量获取测试数据库DSN
|
||||
import os
|
||||
dsn = os.environ.get("TEST_DB_DSN")
|
||||
if not dsn:
|
||||
pytest.skip("未配置测试数据库")
|
||||
|
||||
conn = DatabaseConnection(dsn)
|
||||
yield conn
|
||||
conn.close()
|
||||
|
||||
def test_database_query(db_connection):
|
||||
"""测试数据库查询"""
|
||||
result = db_connection.query("SELECT 1 AS test")
|
||||
assert len(result) == 1
|
||||
assert result[0]["test"] == 1
|
||||
|
||||
def test_database_operations(db_connection):
|
||||
"""测试数据库操作"""
|
||||
ops = DatabaseOperations(db_connection)
|
||||
# 添加实际的测试用例
|
||||
pass
|
||||
18618
etl_billiards/tests/testdata_json/20251110_034959_助教流水.json
Normal file
18618
etl_billiards/tests/testdata_json/20251110_034959_助教流水.json
Normal file
File diff suppressed because it is too large
Load Diff
647
etl_billiards/tests/testdata_json/20251110_035004_助教废除.json
Normal file
647
etl_billiards/tests/testdata_json/20251110_035004_助教废除.json
Normal file
@@ -0,0 +1,647 @@
|
||||
[
|
||||
{
|
||||
"data": {
|
||||
"total": 15,
|
||||
"abolitionAssistants": [
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-11-09 19:23:29",
|
||||
"id": 2957675849518789,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963816579205,
|
||||
"tableId": 2793016660660357,
|
||||
"tableArea": "C区",
|
||||
"tableName": "C1",
|
||||
"assistantOn": "27",
|
||||
"assistantName": "泡芙",
|
||||
"pdChargeMinutes": 214,
|
||||
"assistantAbolishAmount": 5.83,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-11-06 17:42:09",
|
||||
"id": 2953329501898373,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2802006170324037,
|
||||
"tableId": 2851642357976581,
|
||||
"tableArea": "补时长",
|
||||
"tableName": "补时长5",
|
||||
"assistantOn": "23",
|
||||
"assistantName": "婉婉",
|
||||
"pdChargeMinutes": 10800,
|
||||
"assistantAbolishAmount": 570.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-11-06 17:42:09",
|
||||
"id": 2953329502357125,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2802006170324037,
|
||||
"tableId": 2851642357976581,
|
||||
"tableArea": "补时长",
|
||||
"tableName": "补时长5",
|
||||
"assistantOn": "52",
|
||||
"assistantName": "小柔",
|
||||
"pdChargeMinutes": 10800,
|
||||
"assistantAbolishAmount": 570.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-30 01:17:22",
|
||||
"id": 2942452375932869,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963825803397,
|
||||
"tableId": 2793018776604805,
|
||||
"tableArea": "VIP包厢",
|
||||
"tableName": "VIP1",
|
||||
"assistantOn": "2",
|
||||
"assistantName": "佳怡",
|
||||
"pdChargeMinutes": 0,
|
||||
"assistantAbolishAmount": 0.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-29 06:57:22",
|
||||
"id": 2941371032964997,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963848527941,
|
||||
"tableId": 2793021451292741,
|
||||
"tableArea": "666",
|
||||
"tableName": "董事办",
|
||||
"assistantOn": "4",
|
||||
"assistantName": "璇子",
|
||||
"pdChargeMinutes": 0,
|
||||
"assistantAbolishAmount": 0.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-28 02:13:18",
|
||||
"id": 2939676194180229,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963887030341,
|
||||
"tableId": 2793023960551493,
|
||||
"tableArea": "麻将房",
|
||||
"tableName": "Ｍ1",
|
||||
"assistantOn": "2",
|
||||
"assistantName": "佳怡",
|
||||
"pdChargeMinutes": 3602,
|
||||
"assistantAbolishAmount": 108.06,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-26 21:06:37",
|
||||
"id": 2937959143262725,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963855982661,
|
||||
"tableId": 2793022145302597,
|
||||
"tableArea": "K包",
|
||||
"tableName": "888",
|
||||
"assistantOn": "16",
|
||||
"assistantName": "周周",
|
||||
"pdChargeMinutes": 0,
|
||||
"assistantAbolishAmount": 0.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-25 21:36:22",
|
||||
"id": 2936572806285765,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963816579205,
|
||||
"tableId": 2793017278451845,
|
||||
"tableArea": "C区",
|
||||
"tableName": "C2",
|
||||
"assistantOn": "4",
|
||||
"assistantName": "璇子",
|
||||
"pdChargeMinutes": 0,
|
||||
"assistantAbolishAmount": 0.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-23 19:05:48",
|
||||
"id": 2933593641256581,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963807682693,
|
||||
"tableId": 2793012902318213,
|
||||
"tableArea": "B区",
|
||||
"tableName": "B9",
|
||||
"assistantOn": "16",
|
||||
"assistantName": "周周",
|
||||
"pdChargeMinutes": 3600,
|
||||
"assistantAbolishAmount": 190.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-18 20:25:50",
|
||||
"id": 2926594431305093,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963794329671,
|
||||
"tableId": 2793001904918661,
|
||||
"tableArea": "A区",
|
||||
"tableName": "A4",
|
||||
"assistantOn": "15",
|
||||
"assistantName": "七七",
|
||||
"pdChargeMinutes": 2379,
|
||||
"assistantAbolishAmount": 71.37,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-14 14:20:32",
|
||||
"id": 2920573007709573,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963855982661,
|
||||
"tableId": 2793022145302597,
|
||||
"tableArea": "K包",
|
||||
"tableName": "888",
|
||||
"assistantOn": "9",
|
||||
"assistantName": "球球",
|
||||
"pdChargeMinutes": 14400,
|
||||
"assistantAbolishAmount": 392.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-03 01:21:59",
|
||||
"id": 2904236313234373,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963848527941,
|
||||
"tableId": 2793020955840645,
|
||||
"tableArea": "666",
|
||||
"tableName": "666",
|
||||
"assistantOn": "9",
|
||||
"assistantName": "球球",
|
||||
"pdChargeMinutes": 0,
|
||||
"assistantAbolishAmount": 0.0,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-01 00:27:29",
|
||||
"id": 2901351579143365,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963855982661,
|
||||
"tableId": 2793022145302597,
|
||||
"tableArea": "K包",
|
||||
"tableName": "888",
|
||||
"assistantOn": "99",
|
||||
"assistantName": "Amy",
|
||||
"pdChargeMinutes": 10605,
|
||||
"assistantAbolishAmount": 465.44,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-01 00:27:29",
|
||||
"id": 2901351578864837,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963855982661,
|
||||
"tableId": 2793022145302597,
|
||||
"tableArea": "K包",
|
||||
"tableName": "888",
|
||||
"assistantOn": "4",
|
||||
"assistantName": "璇子",
|
||||
"pdChargeMinutes": 10608,
|
||||
"assistantAbolishAmount": 318.24,
|
||||
"trashReason": ""
|
||||
},
|
||||
{
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 1,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"createTime": "2025-10-01 00:27:29",
|
||||
"id": 2901351578602693,
|
||||
"siteId": 2790685415443269,
|
||||
"tableAreaId": 2791963855982661,
|
||||
"tableId": 2793022145302597,
|
||||
"tableArea": "K包",
|
||||
"tableName": "888",
|
||||
"assistantOn": "2",
|
||||
"assistantName": "佳怡",
|
||||
"pdChargeMinutes": 10611,
|
||||
"assistantAbolishAmount": 318.33,
|
||||
"trashReason": ""
|
||||
}
|
||||
]
|
||||
},
|
||||
"code": 0
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"total": 15,
|
||||
"abolitionAssistants": []
|
||||
},
|
||||
"code": 0
|
||||
}
|
||||
]
|
||||
13618
etl_billiards/tests/testdata_json/20251110_035011_台费流水.json
Normal file
13618
etl_billiards/tests/testdata_json/20251110_035011_台费流水.json
Normal file
File diff suppressed because it is too large
Load Diff
20561
etl_billiards/tests/testdata_json/20251110_035904_小票详情.json
Normal file
20561
etl_billiards/tests/testdata_json/20251110_035904_小票详情.json
Normal file
File diff suppressed because it is too large
Load Diff
12218
etl_billiards/tests/testdata_json/20251110_035908_台费打折.json
Normal file
12218
etl_billiards/tests/testdata_json/20251110_035908_台费打折.json
Normal file
File diff suppressed because it is too large
Load Diff
18618
etl_billiards/tests/testdata_json/20251110_035916_结账记录.json
Normal file
18618
etl_billiards/tests/testdata_json/20251110_035916_结账记录.json
Normal file
File diff suppressed because it is too large
Load Diff
8002
etl_billiards/tests/testdata_json/20251110_035923_支付记录.json
Normal file
8002
etl_billiards/tests/testdata_json/20251110_035923_支付记录.json
Normal file
File diff suppressed because it is too large
Load Diff
673
etl_billiards/tests/testdata_json/20251110_035929_退款记录.json
Normal file
673
etl_billiards/tests/testdata_json/20251110_035929_退款记录.json
Normal file
@@ -0,0 +1,673 @@
|
||||
[
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2955202296416389,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -5000.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-08 01:27:16",
|
||||
"create_time": "2025-11-08 01:27:16",
|
||||
"relate_type": 5,
|
||||
"relate_id": 2955078219057349,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2955171790194821,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -10000.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-08 00:56:14",
|
||||
"create_time": "2025-11-08 00:56:14",
|
||||
"relate_type": 5,
|
||||
"relate_id": 2955153380001861,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2951883030513413,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -12.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-05 17:10:44",
|
||||
"create_time": "2025-11-05 17:10:44",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2951881496577861,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2948959062542597,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -65.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-03 15:36:19",
|
||||
"create_time": "2025-11-03 15:36:19",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2948934289967557,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2948630468005509,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -88.33,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-03 10:02:03",
|
||||
"create_time": "2025-11-03 10:02:03",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2948246513454661,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2948269239095045,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -0.67,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-11-03 03:54:36",
|
||||
"create_time": "2025-11-03 03:54:36",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2948246513454661,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2944743812581445,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -44000.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-10-31 16:08:21",
|
||||
"create_time": "2025-10-31 16:08:21",
|
||||
"relate_type": 5,
|
||||
"relate_id": 2944743413958789,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2931109065131653,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -10.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-10-22 00:58:22",
|
||||
"create_time": "2025-10-22 00:58:22",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2931108522378885,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2921195994465669,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -2.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-10-15 00:54:16",
|
||||
"create_time": "2025-10-15 00:54:16",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2920440691344901,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2919690732146181,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -3000.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-10-13 23:23:02",
|
||||
"create_time": "2025-10-13 23:23:02",
|
||||
"relate_type": 5,
|
||||
"relate_id": 2919519811440261,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
},
|
||||
{
|
||||
"tenantName": "朗朗桌球",
|
||||
"siteProfile": {
|
||||
"id": 2790685415443269,
|
||||
"org_id": 2790684179467077,
|
||||
"shop_name": "朗朗桌球",
|
||||
"avatar": "https://oss.ficoo.vip/admin/hXcE4E_1752495052016.jpg",
|
||||
"business_tel": "13316068642",
|
||||
"full_address": "广东省广州市天河区丽阳街12号",
|
||||
"address": "广东省广州市天河区天园街道朗朗桌球",
|
||||
"longitude": 113.360321,
|
||||
"latitude": 23.133629,
|
||||
"tenant_site_region_id": 156440100,
|
||||
"tenant_id": 2790683160709957,
|
||||
"auto_light": 1,
|
||||
"attendance_distance": 0,
|
||||
"wifi_name": "",
|
||||
"wifi_password": "",
|
||||
"customer_service_qrcode": "",
|
||||
"customer_service_wechat": "",
|
||||
"fixed_pay_qrCode": "",
|
||||
"prod_env": 1,
|
||||
"light_status": 2,
|
||||
"light_type": 0,
|
||||
"site_type": 1,
|
||||
"light_token": "",
|
||||
"site_label": "A",
|
||||
"attendance_enabled": 1,
|
||||
"shop_status": 1
|
||||
},
|
||||
"id": 2914039374956165,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"pay_sn": 0,
|
||||
"pay_amount": -8.0,
|
||||
"pay_status": 2,
|
||||
"pay_time": "2025-10-09 23:34:11",
|
||||
"create_time": "2025-10-09 23:34:11",
|
||||
"relate_type": 2,
|
||||
"relate_id": 2914030720124357,
|
||||
"is_revoke": 0,
|
||||
"is_delete": 0,
|
||||
"online_pay_channel": 0,
|
||||
"payment_method": 4,
|
||||
"balance_frozen_amount": 0.0,
|
||||
"card_frozen_amount": 0.0,
|
||||
"member_id": 0,
|
||||
"member_card_id": 0,
|
||||
"round_amount": 0.0,
|
||||
"online_pay_type": 0,
|
||||
"action_type": 2,
|
||||
"refund_amount": 0.0,
|
||||
"cashier_point_id": 0,
|
||||
"operator_id": 0,
|
||||
"pay_terminal": 1,
|
||||
"pay_config_id": 0,
|
||||
"channel_payer_id": "",
|
||||
"channel_pay_no": "",
|
||||
"check_status": 1,
|
||||
"channel_fee": 0.0
|
||||
}
|
||||
]
|
||||
11002
etl_billiards/tests/testdata_json/20251110_035934_平台验券记录.json
Normal file
11002
etl_billiards/tests/testdata_json/20251110_035934_平台验券记录.json
Normal file
File diff suppressed because it is too large
Load Diff
5450
etl_billiards/tests/testdata_json/20251110_035941_商品档案.json
Normal file
5450
etl_billiards/tests/testdata_json/20251110_035941_商品档案.json
Normal file
File diff suppressed because it is too large
Load Diff
10418
etl_billiards/tests/testdata_json/20251110_035948_门店销售记录.json
Normal file
10418
etl_billiards/tests/testdata_json/20251110_035948_门店销售记录.json
Normal file
File diff suppressed because it is too large
Load Diff
4218
etl_billiards/tests/testdata_json/20251110_043159_库存变化记录1.json
Normal file
4218
etl_billiards/tests/testdata_json/20251110_043159_库存变化记录1.json
Normal file
File diff suppressed because it is too large
Load Diff
712
etl_billiards/tests/testdata_json/20251110_043204_库存变化记录2.json
Normal file
712
etl_billiards/tests/testdata_json/20251110_043204_库存变化记录2.json
Normal file
@@ -0,0 +1,712 @@
|
||||
[
|
||||
{
|
||||
"data": {
|
||||
"total": 0,
|
||||
"goodsCategoryList": [
|
||||
{
|
||||
"id": 2790683528350533,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "槟榔",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "槟榔",
|
||||
"tenant_goods_business_id": 2790683528317766,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350534,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "槟榔",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350533,
|
||||
"business_name": "槟榔",
|
||||
"tenant_goods_business_id": 2790683528317766,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350535,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "器材",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350536,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "皮头",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350537,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "球杆",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350538,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350539,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "酒水",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350540,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "饮料",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350541,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "酒水",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350542,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "茶水",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350543,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "咖啡",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350544,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "加料",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793221553489733,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "洋酒",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350545,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "果盘",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "水果",
|
||||
"tenant_goods_business_id": 2790683528317769,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792050275864453,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "果盘",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350545,
|
||||
"business_name": "水果",
|
||||
"tenant_goods_business_id": 2790683528317769,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2791941988405125,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "零食",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2791948300259205,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "零食",
|
||||
"alias_name": "",
|
||||
"pid": 2791941988405125,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793236829620037,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "面",
|
||||
"alias_name": "",
|
||||
"pid": 2791941988405125,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2791942087561093,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "雪糕",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "雪糕",
|
||||
"tenant_goods_business_id": 2791931866402693,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792035069284229,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "雪糕",
|
||||
"alias_name": "",
|
||||
"pid": 2791942087561093,
|
||||
"business_name": "雪糕",
|
||||
"tenant_goods_business_id": 2791931866402693,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2792062778003333,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "香烟",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "香烟",
|
||||
"tenant_goods_business_id": 2790683528317765,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792063209623429,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "香烟",
|
||||
"alias_name": "",
|
||||
"pid": 2792062778003333,
|
||||
"business_name": "香烟",
|
||||
"tenant_goods_business_id": 2790683528317765,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793217944864581,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "其他",
|
||||
"tenant_goods_business_id": 2793217599407941,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2793218343257925,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他2",
|
||||
"alias_name": "",
|
||||
"pid": 2793217944864581,
|
||||
"business_name": "其他",
|
||||
"tenant_goods_business_id": 2793217599407941,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793220945250117,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "小吃",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "小吃",
|
||||
"tenant_goods_business_id": 2793220268902213,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2793221283104581,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "小吃",
|
||||
"alias_name": "",
|
||||
"pid": 2793220945250117,
|
||||
"business_name": "小吃",
|
||||
"tenant_goods_business_id": 2793220268902213,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"code": 0
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"total": 0,
|
||||
"goodsCategoryList": [
|
||||
{
|
||||
"id": 2790683528350533,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "槟榔",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "槟榔",
|
||||
"tenant_goods_business_id": 2790683528317766,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350534,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "槟榔",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350533,
|
||||
"business_name": "槟榔",
|
||||
"tenant_goods_business_id": 2790683528317766,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350535,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "器材",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350536,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "皮头",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350537,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "球杆",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350538,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350535,
|
||||
"business_name": "器材",
|
||||
"tenant_goods_business_id": 2790683528317767,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350539,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "酒水",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2790683528350540,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "饮料",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350541,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "酒水",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350542,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "茶水",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350543,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "咖啡",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350544,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "加料",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793221553489733,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "洋酒",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350539,
|
||||
"business_name": "酒水",
|
||||
"tenant_goods_business_id": 2790683528317768,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2790683528350545,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "果盘",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "水果",
|
||||
"tenant_goods_business_id": 2790683528317769,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792050275864453,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "果盘",
|
||||
"alias_name": "",
|
||||
"pid": 2790683528350545,
|
||||
"business_name": "水果",
|
||||
"tenant_goods_business_id": 2790683528317769,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2791941988405125,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "零食",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2791948300259205,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "零食",
|
||||
"alias_name": "",
|
||||
"pid": 2791941988405125,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793236829620037,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "面",
|
||||
"alias_name": "",
|
||||
"pid": 2791941988405125,
|
||||
"business_name": "零食",
|
||||
"tenant_goods_business_id": 2791932037238661,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2791942087561093,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "雪糕",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "雪糕",
|
||||
"tenant_goods_business_id": 2791931866402693,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792035069284229,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "雪糕",
|
||||
"alias_name": "",
|
||||
"pid": 2791942087561093,
|
||||
"business_name": "雪糕",
|
||||
"tenant_goods_business_id": 2791931866402693,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2792062778003333,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "香烟",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "香烟",
|
||||
"tenant_goods_business_id": 2790683528317765,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2792063209623429,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "香烟",
|
||||
"alias_name": "",
|
||||
"pid": 2792062778003333,
|
||||
"business_name": "香烟",
|
||||
"tenant_goods_business_id": 2790683528317765,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 1,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793217944864581,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "其他",
|
||||
"tenant_goods_business_id": 2793217599407941,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2793218343257925,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "其他2",
|
||||
"alias_name": "",
|
||||
"pid": 2793217944864581,
|
||||
"business_name": "其他",
|
||||
"tenant_goods_business_id": 2793217599407941,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
},
|
||||
{
|
||||
"id": 2793220945250117,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "小吃",
|
||||
"alias_name": "",
|
||||
"pid": 0,
|
||||
"business_name": "小吃",
|
||||
"tenant_goods_business_id": 2793220268902213,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [
|
||||
{
|
||||
"id": 2793221283104581,
|
||||
"tenant_id": 2790683160709957,
|
||||
"category_name": "小吃",
|
||||
"alias_name": "",
|
||||
"pid": 2793220945250117,
|
||||
"business_name": "小吃",
|
||||
"tenant_goods_business_id": 2793220268902213,
|
||||
"open_salesman": 2,
|
||||
"categoryBoxes": [],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
],
|
||||
"sort": 0,
|
||||
"is_warehousing": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"code": 0
|
||||
}
|
||||
]
|
||||
3418
etl_billiards/tests/testdata_json/20251110_043209_会员档案.json
Normal file
3418
etl_billiards/tests/testdata_json/20251110_043209_会员档案.json
Normal file
File diff suppressed because it is too large
Load Diff
5418
etl_billiards/tests/testdata_json/20251110_043217_余额变更记录.json
Normal file
5418
etl_billiards/tests/testdata_json/20251110_043217_余额变更记录.json
Normal file
File diff suppressed because it is too large
Load Diff
13980
etl_billiards/tests/testdata_json/20251110_043223_储值卡列表.json
Normal file
13980
etl_billiards/tests/testdata_json/20251110_043223_储值卡列表.json
Normal file
File diff suppressed because it is too large
Load Diff
6899
etl_billiards/tests/testdata_json/20251110_043231_充值记录.json
Normal file
6899
etl_billiards/tests/testdata_json/20251110_043231_充值记录.json
Normal file
File diff suppressed because it is too large
Load Diff
3186
etl_billiards/tests/testdata_json/20251110_043237_助教账号1.json
Normal file
3186
etl_billiards/tests/testdata_json/20251110_043237_助教账号1.json
Normal file
File diff suppressed because it is too large
Load Diff
3186
etl_billiards/tests/testdata_json/20251110_043243_助教账号2.json
Normal file
3186
etl_billiards/tests/testdata_json/20251110_043243_助教账号2.json
Normal file
File diff suppressed because it is too large
Load Diff
1934
etl_billiards/tests/testdata_json/20251110_043250_台桌列表.json
Normal file
1934
etl_billiards/tests/testdata_json/20251110_043250_台桌列表.json
Normal file
File diff suppressed because it is too large
Load Diff
646
etl_billiards/tests/testdata_json/20251110_043255_团购套餐.json
Normal file
646
etl_billiards/tests/testdata_json/20251110_043255_团购套餐.json
Normal file
@@ -0,0 +1,646 @@
|
||||
[
|
||||
{
|
||||
"data": {
|
||||
"total": 17,
|
||||
"packageCouponList": [
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2939215004469573,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "早场特惠一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-10-27 00:00:00",
|
||||
"end_time": "2026-10-28 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 2,
|
||||
"package_id": 1814707240811572,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-10-27 18:24:09",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2861343275830405,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "B区桌球一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "B区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-09-02 00:00:00",
|
||||
"end_time": "2026-09-03 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1370841337,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-09-02 18:08:56",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960521691013",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 3,
|
||||
"id": 2836713896429317,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "午夜一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-08-16 00:00:00",
|
||||
"end_time": "2026-08-17 00:00:00",
|
||||
"is_enabled": 2,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1370841337,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-08-16 08:34:38",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2801876691340293,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "中八、斯诺克包厢两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "VIP包厢",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-22 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1126976372,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-22 17:56:24",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961060364165",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 3,
|
||||
"id": 2801875268668357,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "中八、斯诺克包厢两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "VIP包厢",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-22 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 2,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1126976372,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-22 17:54:57",
|
||||
"creator_name": "管理员：郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961060364165",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "0",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 3,
|
||||
"id": 2800772613934149,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "午夜场一小时A区",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 2,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1370841337,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-21 23:13:16",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798905767676933,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "中八、斯诺克包厢两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "VIP包厢",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 2,
|
||||
"package_id": 1812429097416714,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 15:34:13",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961060364165",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 3,
|
||||
"id": 2798901295615045,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "新人特惠A区中八一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 2,
|
||||
"is_delete": 0,
|
||||
"type": 2,
|
||||
"package_id": 1814707240811572,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 15:29:40",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798898826300485,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "斯诺克两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "斯诺克区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 2,
|
||||
"package_id": 1814983609169019,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 15:27:09",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961347968901",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798734170983493,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "助理教练竞技教学两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1173128804,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:39:39",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798732571167749,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "全天斯诺克一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "斯诺克区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-30 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1147633733,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:38:02",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961347968901",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798731703045189,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "KTV欢唱四小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "888",
|
||||
"selling_price": 0.0,
|
||||
"duration": 14400,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1137882866,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:37:09",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791961709907845",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "10:00:00",
|
||||
"add_end_clock": "18:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798729978514501,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "全天A区中八两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1130465371,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:35:24",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798728823213061,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "全天B区中八两小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "B区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 7200,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1137872168,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:34:13",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960521691013",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798727423528005,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "全天A区中八一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1128411555,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:32:48",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798723640069125,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "中八A区新人特惠一小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "A区",
|
||||
"selling_price": 0.0,
|
||||
"duration": 3600,
|
||||
"start_time": "2025-07-20 00:00:00",
|
||||
"end_time": "2025-12-31 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1203035334,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:28:57",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791960001957765",
|
||||
"start_clock": "00:00:00",
|
||||
"end_clock": "1.00:00:00",
|
||||
"add_start_clock": "00:00:00",
|
||||
"add_end_clock": "1.00:00:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
},
|
||||
{
|
||||
"site_name": "朗朗桌球",
|
||||
"effective_status": 1,
|
||||
"id": 2798713926290437,
|
||||
"site_id": 2790685415443269,
|
||||
"tenant_id": 2790683160709957,
|
||||
"package_name": "麻将 、掼蛋包厢四小时",
|
||||
"table_area_id": "0",
|
||||
"table_area_name": "麻将房",
|
||||
"selling_price": 0.0,
|
||||
"duration": 14400,
|
||||
"start_time": "2025-07-21 00:00:00",
|
||||
"end_time": "2025-12-30 00:00:00",
|
||||
"is_enabled": 1,
|
||||
"is_delete": 0,
|
||||
"type": 1,
|
||||
"package_id": 1134269810,
|
||||
"usable_count": 9999999,
|
||||
"create_time": "2025-07-20 12:19:04",
|
||||
"creator_name": "店长:郑丽珊",
|
||||
"tenant_table_area_id": "0",
|
||||
"table_area_id_list": "",
|
||||
"tenant_table_area_id_list": "2791962314215301",
|
||||
"start_clock": "10:00:00",
|
||||
"end_clock": "1.02:00:00",
|
||||
"add_start_clock": "10:00:00",
|
||||
"add_end_clock": "23:59:00",
|
||||
"date_info": "",
|
||||
"date_type": 1,
|
||||
"group_type": 1,
|
||||
"usable_range": "",
|
||||
"coupon_money": 0.0,
|
||||
"area_tag_type": 1,
|
||||
"system_group_type": 1,
|
||||
"max_selectable_categories": 0,
|
||||
"card_type_ids": "0"
|
||||
}
|
||||
]
|
||||
},
|
||||
"code": 0
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"total": 17,
|
||||
"packageCouponList": []
|
||||
},
|
||||
"code": 0
|
||||
}
|
||||
]
|
||||
9020
etl_billiards/tests/testdata_json/20251110_043302_团购套餐流水.json
Normal file
9020
etl_billiards/tests/testdata_json/20251110_043302_团购套餐流水.json
Normal file
File diff suppressed because it is too large
Load Diff
2578
etl_billiards/tests/testdata_json/20251110_043308_库存汇总.json
Normal file
2578
etl_billiards/tests/testdata_json/20251110_043308_库存汇总.json
Normal file
File diff suppressed because it is too large
Load Diff
7585
etl_billiards/tests/testdata_json/20251110_051132_门店商品档案1.json
Normal file
7585
etl_billiards/tests/testdata_json/20251110_051132_门店商品档案1.json
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,20 @@
|
||||
[
|
||||
{
|
||||
"data": {
|
||||
"goodsStockA": 0,
|
||||
"goodsStockB": 6252,
|
||||
"goodsSaleNum": 210.29,
|
||||
"stockSumMoney": 1461.28
|
||||
},
|
||||
"code": 0
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"goodsStockA": 0,
|
||||
"goodsStockB": 6252,
|
||||
"goodsSaleNum": 210.29,
|
||||
"stockSumMoney": 1461.28
|
||||
},
|
||||
"code": 0
|
||||
}
|
||||
]
|
||||
0
etl_billiards/tests/unit/__init__.py
Normal file
0
etl_billiards/tests/unit/__init__.py
Normal file
24
etl_billiards/tests/unit/test_config.py
Normal file
24
etl_billiards/tests/unit/test_config.py
Normal file
@@ -0,0 +1,24 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""配置管理测试"""
|
||||
import pytest
|
||||
from config.settings import AppConfig
|
||||
from config.defaults import DEFAULTS
|
||||
|
||||
def test_config_load():
|
||||
"""测试配置加载"""
|
||||
config = AppConfig.load()
|
||||
assert config.get("app.timezone") == DEFAULTS["app"]["timezone"]
|
||||
|
||||
def test_config_override():
|
||||
"""测试配置覆盖"""
|
||||
overrides = {
|
||||
"app": {"store_id": 12345}
|
||||
}
|
||||
config = AppConfig.load(overrides)
|
||||
assert config.get("app.store_id") == 12345
|
||||
|
||||
def test_config_get_nested():
|
||||
"""测试嵌套配置获取"""
|
||||
config = AppConfig.load()
|
||||
assert config.get("db.batch_size") == 1000
|
||||
assert config.get("nonexistent.key", "default") == "default"
|
||||
29
etl_billiards/tests/unit/test_parsers.py
Normal file
29
etl_billiards/tests/unit/test_parsers.py
Normal file
@@ -0,0 +1,29 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""解析器测试"""
|
||||
import pytest
|
||||
from decimal import Decimal
|
||||
from datetime import datetime
|
||||
from zoneinfo import ZoneInfo
|
||||
from models.parsers import TypeParser
|
||||
|
||||
def test_parse_decimal():
|
||||
"""测试金额解析"""
|
||||
assert TypeParser.parse_decimal("100.555", 2) == Decimal("100.56")
|
||||
assert TypeParser.parse_decimal(None) is None
|
||||
assert TypeParser.parse_decimal("invalid") is None
|
||||
|
||||
def test_parse_int():
|
||||
"""测试整数解析"""
|
||||
assert TypeParser.parse_int("123") == 123
|
||||
assert TypeParser.parse_int(456) == 456
|
||||
assert TypeParser.parse_int(None) is None
|
||||
assert TypeParser.parse_int("abc") is None
|
||||
|
||||
def test_parse_timestamp():
|
||||
"""测试时间戳解析"""
|
||||
tz = ZoneInfo("Asia/Taipei")
|
||||
dt = TypeParser.parse_timestamp("2025-01-15 10:30:00", tz)
|
||||
assert dt is not None
|
||||
assert dt.year == 2025
|
||||
assert dt.month == 1
|
||||
assert dt.day == 15
|
||||
0
etl_billiards/utils/__init__.py
Normal file
0
etl_billiards/utils/__init__.py
Normal file
22
etl_billiards/utils/helpers.py
Normal file
22
etl_billiards/utils/helpers.py
Normal file
@@ -0,0 +1,22 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""通用工具函数"""
|
||||
import hashlib
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
def ensure_dir(path: Path):
|
||||
"""确保目录存在"""
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def make_surrogate_key(*parts) -> int:
|
||||
"""
|
||||
生成代理键
|
||||
将多个字段值拼接后计算SHA1，取前8字节转为无符号64位整数
|
||||
"""
|
||||
raw = "|".join("" if p is None else str(p) for p in parts)
|
||||
h = hashlib.sha1(raw.encode("utf-8")).digest()[:8]
|
||||
return int.from_bytes(h, byteorder="big", signed=False)
|
||||
|
||||
def now_local(tz) -> datetime:
|
||||
"""获取本地当前时间"""
|
||||
return datetime.now(tz)
|
||||
5
run_etl.bat
Normal file
5
run_etl.bat
Normal file
@@ -0,0 +1,5 @@
|
||||
@echo off
|
||||
REM ETL运行脚本 (Windows)
|
||||
|
||||
REM 运行ETL
|
||||
python -m cli.main %*
|
||||
10
run_etl.sh
Normal file
10
run_etl.sh
Normal file
@@ -0,0 +1,10 @@
|
||||
#!/bin/bash
|
||||
# ETL运行脚本
|
||||
|
||||
# 加载环境变量
|
||||
if [ -f .env ]; then
|
||||
export $(cat .env | grep -v '^#' | xargs)
|
||||
fi
|
||||
|
||||
# 运行ETL
|
||||
python -m cli.main "$@"
|
||||
13
开发笔记/记录.md
Normal file
13
开发笔记/记录.md
Normal file
@@ -0,0 +1,13 @@
|
||||
原来在 task_merged.py 里配置的 14 个任务中，有 11 个目前还没有在新项目里实现，对应的 loader / task 类也不存在。
|
||||
|
||||
|
||||
|
||||
原脚本里“导出请求/响应 JSON 到本地目录、生成 manifest.json / ingest_report.json 并支持 offline 模式”的那一块逻辑，在新代码里还没有真正落地，只保留了配置字段和数据库字段，但没有实际写文件和离线装载的实现。
|
||||
|
||||
|
||||
丰富Pytest，进行分模块.分任务测试
|
||||
|
||||
|
||||
质量检查目前没有被“接入主流程”，内容也待完善，入库等问题？
|
||||
|
||||
|
||||
Reference in New Issue
Block a user