Compare commits
13 Commits
7ca19a4a2c
...
dev
| Author | SHA1 | Date | |
|---|---|---|---|
| 4fafc80254 | |||
| 1a76108209 | |||
| c42b516895 | |||
| 7e67bc4218 | |||
| 64a3159f9e | |||
| ba00654ac5 | |||
| 8b1200383e | |||
| d05c98784f | |||
| ff9e993ec2 | |||
| 9f8976e75a | |||
| 04c064793a | |||
| 7c7280917a | |||
| 8b98fcea1f |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -6,7 +6,6 @@ __pycache__/
|
|||||||
.Python
|
.Python
|
||||||
build/
|
build/
|
||||||
develop-eggs/
|
develop-eggs/
|
||||||
dist/
|
|
||||||
downloads/
|
downloads/
|
||||||
eggs/
|
eggs/
|
||||||
.eggs/
|
.eggs/
|
||||||
@@ -39,8 +38,6 @@ export/
|
|||||||
logs/
|
logs/
|
||||||
|
|
||||||
# 环境变量
|
# 环境变量
|
||||||
.env
|
|
||||||
.env.local
|
|
||||||
|
|
||||||
# 测试
|
# 测试
|
||||||
.pytest_cache/
|
.pytest_cache/
|
||||||
|
|||||||
43
README.md
43
README.md
@@ -110,9 +110,6 @@ python -m cli.main \
|
|||||||
- `run_etl.sh`:Linux/macOS 启动脚本(需先 `cd etl_billiards`;会加载当前目录 `.env`)。
|
- `run_etl.sh`:Linux/macOS 启动脚本(需先 `cd etl_billiards`;会加载当前目录 `.env`)。
|
||||||
|
|
||||||
### app/
|
### app/
|
||||||
- `app/etl_busy.py`:忙时 ETL 示例函数(TODO,占位)。
|
|
||||||
- `app/etl_idle.py`:闲时 ETL 示例函数(TODO,占位)。
|
|
||||||
- `app/runner.py`:简易 Runner:按 `--mode busy/idle` 调用 `app/etl_busy.py` 或 `app/etl_idle.py`(示例/未接入主 ETL)。
|
|
||||||
|
|
||||||
### etl_billiards/
|
### etl_billiards/
|
||||||
- `etl_billiards/.env`:本地运行环境变量(含敏感信息,勿提交/勿外传)。
|
- `etl_billiards/.env`:本地运行环境变量(含敏感信息,勿提交/勿外传)。
|
||||||
@@ -220,13 +217,12 @@ python -m cli.main \
|
|||||||
- `etl_billiards/scripts/bootstrap_schema.py`:Apply the PRD-aligned warehouse schema (ODS/DWD/DWS) to PostgreSQL.
|
- `etl_billiards/scripts/bootstrap_schema.py`:Apply the PRD-aligned warehouse schema (ODS/DWD/DWS) to PostgreSQL.
|
||||||
- `etl_billiards/scripts/build_dwd_from_ods.py`:Populate PRD DWD tables from ODS payload snapshots.
|
- `etl_billiards/scripts/build_dwd_from_ods.py`:Populate PRD DWD tables from ODS payload snapshots.
|
||||||
- `etl_billiards/scripts/build_dws_order_summary.py`:Recompute billiards_dws.dws_order_summary from DWD fact tables.
|
- `etl_billiards/scripts/build_dws_order_summary.py`:Recompute billiards_dws.dws_order_summary from DWD fact tables.
|
||||||
- `etl_billiards/scripts/check_ods_json_vs_table.py`:ODS JSON 字段核对脚本:对照当前数据库中的 ODS 表字段,检查示例 JSON(默认目录 C:\dev\LLTQ\export\test-json-doc)
|
- `etl_billiards/scripts/check_ods_json_vs_table.py`:ODS JSON 字段核对脚本:对照当前数据库中的 ODS 表字段,检查示例 JSON(默认目录 export/test-json-doc)
|
||||||
- `etl_billiards/scripts/check_ods_gaps.py`:ODS 缺失校验脚本:API 主键 vs ODS 主键逐条比对,输出缺失明细样例。
|
- `etl_billiards/scripts/check_ods_gaps.py`:ODS 缺失校验脚本:API 主键 vs ODS 主键逐条比对,输出缺失明细样例。
|
||||||
- `etl_billiards/scripts/reload_ods_windowed.py`:ODS 窗口化补跑脚本:按时间切片重跑 ODS 任务,并可配置窗口粒度与延时。
|
- `etl_billiards/scripts/reload_ods_windowed.py`:ODS 窗口化补跑脚本:按时间切片重跑 ODS 任务,并可配置窗口粒度与延时。
|
||||||
- `etl_billiards/scripts/rebuild_db_and_run_ods_to_dwd.py`:一键重建 ETL 相关 Schema,并执行 ODS → DWD。
|
- `etl_billiards/scripts/rebuild_db_and_run_ods_to_dwd.py`:一键重建 ETL 相关 Schema,并执行 ODS → DWD。
|
||||||
- `etl_billiards/scripts/rebuild_ods_from_json.py`:从本地 JSON 示例目录重建 billiards_ods.* 表,并导入样例数据。
|
- `etl_billiards/scripts/rebuild_ods_from_json.py`:从本地 JSON 示例目录重建 billiards_ods.* 表,并导入样例数据。
|
||||||
- `etl_billiards/scripts/run_tests.py`:灵活的测试执行脚本,可像搭积木一样组合不同参数或预置命令(模式/数据库/归档路径等),
|
- `etl_billiards/scripts/run_tests.py`:灵活的测试执行脚本,可像搭积木一样组合不同参数或预置命令(模式/数据库/归档路径等),
|
||||||
- `etl_billiards/scripts/Temp1.py`:空 Python 文件(占位/临时)。
|
|
||||||
- `etl_billiards/scripts/test_db_connection.py`:Quick utility for validating PostgreSQL connectivity (ASCII-only output).
|
- `etl_billiards/scripts/test_db_connection.py`:Quick utility for validating PostgreSQL connectivity (ASCII-only output).
|
||||||
- `etl_billiards/scripts/test_presets.py`:测试命令仓库:集中维护 run_tests.py 的常用组合,支持一键执行。
|
- `etl_billiards/scripts/test_presets.py`:测试命令仓库:集中维护 run_tests.py 的常用组合,支持一键执行。
|
||||||
|
|
||||||
@@ -388,6 +384,33 @@ python -m cli.main --pg-dsn "postgresql://user:pwd@host:5432/db" --store-id 123
|
|||||||
python -m cli.main --dry-run --tasks DWD_LOAD_FROM_ODS
|
python -m cli.main --dry-run --tasks DWD_LOAD_FROM_ODS
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## 窗口切分与补偿
|
||||||
|
用于 ETL 任务、ODS 缺失校验、数据一致性检查等“带时间窗口”的执行场景。逻辑如下:
|
||||||
|
- 仅当传入窗口参数(如 CLI `--window-start/--window-end` 或脚本 `--start/--end`)时启用切分。
|
||||||
|
- 先对整体窗口前后补偿 N 小时,再按月切分(`month` 为最大单位)。不需要切分时设为 `none`。
|
||||||
|
- 分段窗口将依次执行并汇总结果。
|
||||||
|
|
||||||
|
配置项(默认值见 `config/defaults.py`):
|
||||||
|
- `run.window_split.unit`:`month` / `none`(默认 `month`)
|
||||||
|
- `run.window_split.compensation_hours`:整数小时(默认 2)
|
||||||
|
|
||||||
|
环境变量:
|
||||||
|
- `WINDOW_SPLIT_UNIT`
|
||||||
|
- `WINDOW_COMPENSATION_HOURS`
|
||||||
|
|
||||||
|
CLI 参数(覆盖配置):
|
||||||
|
- `python -m cli.main`:`--window-split-unit`,`--window-compensation-hours`
|
||||||
|
- `scripts/check_ods_gaps.py`:`--window-split-unit`,`--window-compensation-hours`
|
||||||
|
- `scripts/check_data_integrity.py`:`--window-split-unit`,`--window-compensation-hours`
|
||||||
|
- `scripts/reload_ods_windowed.py`:`--window-split-unit`,`--window-compensation-hours`
|
||||||
|
|
||||||
|
示例(`unit=month`,`compensation_hours=2`):
|
||||||
|
- 传入窗口:`2025/11/10 10:00` - `2026/1/19 10:15`
|
||||||
|
- 实际处理窗口切分:
|
||||||
|
- `2025/11/10 08:00` - `2025/12/01 00:00`
|
||||||
|
- `2025/12/01 00:00` - `2026/01/01 00:00`
|
||||||
|
- `2026/01/01 00:00` - `2026/01/19 12:15`
|
||||||
|
|
||||||
## 测试
|
## 测试
|
||||||
说明:仓库未固定 pytest 版本(运行测试需自行安装 `pytest`)。
|
说明:仓库未固定 pytest 版本(运行测试需自行安装 `pytest`)。
|
||||||
|
|
||||||
@@ -419,7 +442,7 @@ python scripts/test_db_connection.py --dsn "postgresql://user:pwd@host:5432/db"
|
|||||||
- 离线回放/重建 ODS(开发/运维):
|
- 离线回放/重建 ODS(开发/运维):
|
||||||
```bash
|
```bash
|
||||||
cd etl_billiards
|
cd etl_billiards
|
||||||
python scripts/rebuild_ods_from_json.py --dsn "$PG_DSN" --json-dir "C:\\path\\to\\json-doc"
|
python scripts/rebuild_ods_from_json.py --dsn "$PG_DSN" --json-dir "export/test-json-doc"
|
||||||
```
|
```
|
||||||
|
|
||||||
## ODS 表概览(数据路径)
|
## ODS 表概览(数据路径)
|
||||||
@@ -584,10 +607,10 @@ python -m cli.main --pipeline-flow INGEST_ONLY --tasks DWS_BUILD_ORDER_SUMMARY
|
|||||||
- `--log-level` 日志级别(`INFO`/`DEBUG`)。
|
- `--log-level` 日志级别(`INFO`/`DEBUG`)。
|
||||||
- `--no-log-console` 禁用控制台日志记录(仅写入文件)。
|
- `--no-log-console` 禁用控制台日志记录(仅写入文件)。
|
||||||
|
|
||||||
示例(按桌、按天设置窗口):
|
示例(按月切分 + 前后补偿 2h):
|
||||||
```bash
|
```bash
|
||||||
cd etl_billiards
|
cd etl_billiards
|
||||||
python scripts/check_ods_gaps.py --start 2025-07-01 --window-days 1 --task-codes ODS_PAYMENT --sleep-per-window-seconds 0.5
|
python scripts/check_ods_gaps.py --start 2025-07-01 --end 2025-09-30 --window-split-unit month --window-compensation-hours 2 --task-codes ODS_PAYMENT --sleep-per-window-seconds 0.5
|
||||||
python scripts/reload_ods_windowed.py --tasks ODS_PAYMENT,ODS_TABLE_USE --start 2025-07-01 --window-days 1 --sleep-seconds 1
|
python scripts/reload_ods_windowed.py --tasks ODS_PAYMENT,ODS_TABLE_USE --start 2025-07-01 --end 2025-09-30 --window-split-unit month --window-compensation-hours 2 --sleep-seconds 1
|
||||||
python run_update.py --check-ods-gaps --check-ods-window-days 1 --check-ods-sleep-per-window-seconds 0.5
|
python run_update.py --check-ods-gaps --check-ods-window-days 1 --check-ods-sleep-per-window-seconds 0.5
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -1,9 +0,0 @@
|
|||||||
# app/etl_busy.py
|
|
||||||
def run():
|
|
||||||
"""
|
|
||||||
忙时抓取逻辑。
|
|
||||||
TODO: 这里写具体抓取流程(API 调用 / 网页解析 / 写入 PostgreSQL 等)
|
|
||||||
"""
|
|
||||||
print("Running busy-period ETL...")
|
|
||||||
# 示例:后续在这里接 PostgreSQL 或 HTTP 抓取
|
|
||||||
# ...
|
|
||||||
@@ -1,8 +0,0 @@
|
|||||||
# app/etl_idle.py
|
|
||||||
def run():
|
|
||||||
"""
|
|
||||||
闲时抓取逻辑。
|
|
||||||
可以做全量同步、大批量历史修正等。
|
|
||||||
"""
|
|
||||||
print("Running idle-period ETL...")
|
|
||||||
# ...
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
# app/runner.py
|
|
||||||
import argparse
|
|
||||||
from datetime import datetime
|
|
||||||
|
|
||||||
from . import etl_busy, etl_idle
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description="Feiqiu ETL Runner")
|
|
||||||
parser.add_argument(
|
|
||||||
"--mode",
|
|
||||||
choices=["busy", "idle"],
|
|
||||||
required=True,
|
|
||||||
help="ETL mode: busy or idle",
|
|
||||||
)
|
|
||||||
|
|
||||||
args = parser.parse_args()
|
|
||||||
now = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
|
||||||
|
|
||||||
print(f"[{now}] Start ETL mode={args.mode}")
|
|
||||||
|
|
||||||
if args.mode == "busy":
|
|
||||||
etl_busy.run()
|
|
||||||
else:
|
|
||||||
etl_idle.run()
|
|
||||||
|
|
||||||
print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] ETL finished.")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,53 +1,132 @@
|
|||||||
# -*- coding: utf-8 -*-
|
# ==============================================================================
|
||||||
# 文件说明:ETL 环境变量(config/env_parser.py 读取),用于数据库连接、目录与运行参数。
|
# ETL 系统配置文件
|
||||||
|
# ==============================================================================
|
||||||
|
# 配置优先级:DEFAULTS < .env < CLI 参数
|
||||||
|
|
||||||
# 数据库连接字符串,config/env_parser.py -> db.dsn,所有任务必需
|
# ------------------------------------------------------------------------------
|
||||||
|
# 数据库配置
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 完整 DSN(优先使用,如果设置了则忽略下面的 host/port/name/user/password)
|
||||||
PG_DSN=postgresql://local-Python:Neo-local-1991125@100.64.0.4:5432/LLZQ-test
|
PG_DSN=postgresql://local-Python:Neo-local-1991125@100.64.0.4:5432/LLZQ-test
|
||||||
# 数据库连接超时秒,config/env_parser.py -> db.connect_timeout_sec
|
|
||||||
|
# 分离式配置(如果不使用 DSN,可以单独配置以下参数)
|
||||||
|
# PG_HOST=localhost
|
||||||
|
# PG_PORT=5432
|
||||||
|
# PG_NAME=your_database
|
||||||
|
# PG_USER=your_user
|
||||||
|
# PG_PASSWORD=your_password
|
||||||
|
|
||||||
|
# 连接超时(秒,范围 1-20)
|
||||||
PG_CONNECT_TIMEOUT=10
|
PG_CONNECT_TIMEOUT=10
|
||||||
|
|
||||||
# 门店/租户ID,config/env_parser.py -> app.store_id,任务调度记录使用
|
# ------------------------------------------------------------------------------
|
||||||
STORE_ID=2790685415443269
|
# 数据库 Schema 配置
|
||||||
# 时区标识,config/env_parser.py -> app.timezone
|
# ------------------------------------------------------------------------------
|
||||||
TIMEZONE=Asia/Taipei
|
# OLTP 业务数据 schema(默认 billiards)
|
||||||
|
SCHEMA_OLTP=billiards
|
||||||
|
|
||||||
# API 基础地址,config/env_parser.py -> api.base_url,FETCH 类任务调用
|
# ETL 管理数据 schema(默认 etl_admin)
|
||||||
|
SCHEMA_ETL=etl_admin
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# API 配置
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
API_BASE=https://pc.ficoo.vip/apiprod/admin/v1/
|
API_BASE=https://pc.ficoo.vip/apiprod/admin/v1/
|
||||||
# API 鉴权 Token,config/env_parser.py -> api.token,FETCH 类任务调用
|
API_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnQtdHlwZSI6IjQiLCJ1c2VyLXR5cGUiOiIxIiwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9yb2xlIjoiMTIiLCJyb2xlLWlkIjoiMTIiLCJ0ZW5hbnQtaWQiOiIyNzkwNjgzMTYwNzA5OTU3Iiwibmlja25hbWUiOiLnp5_miLfnrqHnkIblkZjvvJrmganmgakxIiwic2l0ZS1pZCI6IjAiLCJtb2JpbGUiOiIxMzgxMDUwMjMwNCIsInNpZCI6IjI5NTA0ODk2NTgzOTU4NDUiLCJzdGFmZi1pZCI6IjMwMDk5MTg2OTE1NTkwNDUiLCJvcmctaWQiOiIwIiwicm9sZS10eXBlIjoiMyIsInJlZnJlc2hUb2tlbiI6Iks1ZnBhYlRTNkFsR0FpMmN4WGYrMHdJVkk0L2UvTVQrSVBHM3V5VWRrSjg9IiwicmVmcmVzaEV4cGlyeVRpbWUiOiIyMDI2LzEvMzEg5LiL5Y2IMTA6MTQ6NTEiLCJuZWVkQ2hlY2tUb2tlbiI6ImZhbHNlIiwiZXhwIjoxNzY5ODY4ODkxLCJpc3MiOiJ0ZXN0IiwiYXVkIjoiVXNlciJ9.BH3-iwwrBczb8aFfI__6kwe3AIsEPacN9TruaTrQ3nY
|
||||||
API_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnQtdHlwZSI6IjQiLCJ1c2VyLXR5cGUiOiIxIiwiaHR0cDovL3NjaGVtYXMubWljcm9zb2Z0LmNvbS93cy8yMDA4LzA2L2lkZW50aXR5L2NsYWltcy9yb2xlIjoiMTIiLCJyb2xlLWlkIjoiMTIiLCJ0ZW5hbnQtaWQiOiIyNzkwNjgzMTYwNzA5OTU3Iiwibmlja25hbWUiOiLnp5_miLfnrqHnkIblkZjvvJrmganmgakxIiwic2l0ZS1pZCI6IjAiLCJtb2JpbGUiOiIxMzgxMDUwMjMwNCIsInNpZCI6IjI5NTA0ODk2NTgzOTU4NDUiLCJzdGFmZi1pZCI6IjMwMDk5MTg2OTE1NTkwNDUiLCJvcmctaWQiOiIwIiwicm9sZS10eXBlIjoiMyIsInJlZnJlc2hUb2tlbiI6IlpWV3grVThBc2FYekFJeTRiaXF6MktwNjMxbTFNRlozV3pLaXNjOHREY289IiwicmVmcmVzaEV4cGlyeVRpbWUiOiIyMDI2LzEvMTcg5LiL5Y2INDoyMjo1OSIsIm5lZWRDaGVja1Rva2VuIjoiZmFsc2UiLCJleHAiOjE3Njg2MzgxNzksImlzcyI6InRlc3QiLCJhdWQiOiJVc2VyIn0.PVeAIx1iHqmHRNaQ4OMFPuOlHBoE47bR5TGJjZP-eOE
|
|
||||||
# API 请求超时秒,config/env_parser.py -> api.timeout_sec
|
# API 请求超时(秒)
|
||||||
API_TIMEOUT=20
|
API_TIMEOUT=20
|
||||||
# API 分页大小,config/env_parser.py -> api.page_size
|
|
||||||
|
# 分页大小
|
||||||
API_PAGE_SIZE=200
|
API_PAGE_SIZE=200
|
||||||
# API 最大重试次数,config/env_parser.py -> api.retries.max_attempts
|
|
||||||
|
# 最大重试次数
|
||||||
API_RETRY_MAX=3
|
API_RETRY_MAX=3
|
||||||
|
|
||||||
# 日志根目录,config/env_parser.py -> io.log_root,Init/任务运行写日志
|
# 重试退避时间(JSON 数组格式,单位秒)
|
||||||
LOG_ROOT=C:\dev\LLTQ\export\LOG
|
# API_RETRY_BACKOFF=[1, 2, 4]
|
||||||
# JSON 导出根目录,config/env_parser.py -> io.export_root,FETCH 产出及 INIT 准备
|
|
||||||
EXPORT_ROOT=C:\dev\LLTQ\export\JSON
|
|
||||||
|
|
||||||
# FETCH 模式本地输出目录,config/env_parser.py -> pipeline.fetch_root
|
# ------------------------------------------------------------------------------
|
||||||
FETCH_ROOT=C:\dev\LLTQ\export\JSON
|
# 门店配置
|
||||||
# 本地入库 JSON 目录,config/env_parser.py -> pipeline.ingest_source_dir,MANUAL_INGEST/INGEST_ONLY 使用
|
# ------------------------------------------------------------------------------
|
||||||
INGEST_SOURCE_DIR=C:\dev\LLTQ\export\test-json-doc
|
STORE_ID=2790685415443269
|
||||||
|
TIMEZONE=Asia/Taipei
|
||||||
|
|
||||||
# JSON 漂亮格式输出开关,config/env_parser.py -> io.write_pretty_json
|
# ------------------------------------------------------------------------------
|
||||||
|
# 路径配置
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 导出根目录
|
||||||
|
EXPORT_ROOT=export/JSON
|
||||||
|
|
||||||
|
# 日志根目录
|
||||||
|
LOG_ROOT=export/LOG
|
||||||
|
|
||||||
|
# 在线抓取 JSON 输出目录
|
||||||
|
FETCH_ROOT=export/JSON
|
||||||
|
|
||||||
|
# 本地入库数据源目录(INGEST_ONLY 模式使用)
|
||||||
|
INGEST_SOURCE_DIR=export/test-json-doc
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 流水线配置
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 运行模式:FULL(抓取+入库)、FETCH_ONLY(仅抓取)、INGEST_ONLY(仅入库)
|
||||||
|
PIPELINE_FLOW=FULL
|
||||||
|
|
||||||
|
# JSON 美化输出(调试用,会增加文件大小)
|
||||||
WRITE_PRETTY_JSON=false
|
WRITE_PRETTY_JSON=false
|
||||||
|
|
||||||
# 运行流程:FULL / FETCH_ONLY / INGEST_ONLY,config/env_parser.py -> pipeline.flow
|
# ------------------------------------------------------------------------------
|
||||||
PIPELINE_FLOW=FULL
|
# 时间窗口配置
|
||||||
# 指定任务列表(逗号分隔,覆盖默认),config/env_parser.py -> run.tasks
|
# ------------------------------------------------------------------------------
|
||||||
# RUN_TASKS=INIT_ODS_SCHEMA,MANUAL_INGEST
|
# 冗余窗口(秒),向前多抓取的时间避免边界数据丢失
|
||||||
|
|
||||||
# 固定回溯窗口(可选):同时设置 WINDOW_START + WINDOW_END,将覆盖游标/当前时间窗口
|
|
||||||
# WINDOW_START=2025-07-01 00:00:00
|
|
||||||
# WINDOW_END=2025-08-01 00:00:00
|
|
||||||
|
|
||||||
# 窗口/补偿参数,config/env_parser.py -> run.*
|
|
||||||
OVERLAP_SECONDS=120
|
OVERLAP_SECONDS=120
|
||||||
|
|
||||||
|
# 忙时窗口大小(分钟)
|
||||||
WINDOW_BUSY_MIN=30
|
WINDOW_BUSY_MIN=30
|
||||||
|
|
||||||
|
# 闲时窗口大小(分钟)
|
||||||
WINDOW_IDLE_MIN=180
|
WINDOW_IDLE_MIN=180
|
||||||
|
|
||||||
|
# 闲时窗口定义(HH:MM 格式)
|
||||||
IDLE_START=04:00
|
IDLE_START=04:00
|
||||||
IDLE_END=16:00
|
IDLE_END=16:00
|
||||||
|
|
||||||
|
# 窗口切分单位(month/none),用于长时间回溯任务按月切分
|
||||||
|
WINDOW_SPLIT_UNIT=month
|
||||||
|
|
||||||
|
# 窗口前后补偿小时数,用于捕获边界数据
|
||||||
|
WINDOW_COMPENSATION_HOURS=2
|
||||||
|
|
||||||
|
# 允许空结果推进窗口
|
||||||
ALLOW_EMPTY_RESULT_ADVANCE=true
|
ALLOW_EMPTY_RESULT_ADVANCE=true
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 数据完整性检查配置
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 检查模式:history(历史全量)、recent(最近增量)
|
||||||
|
INTEGRITY_MODE=history
|
||||||
|
|
||||||
|
# 历史检查起始日期(history 模式使用)
|
||||||
|
INTEGRITY_HISTORY_START=2025-07-01
|
||||||
|
|
||||||
|
# 历史检查结束日期(留空表示到当前)
|
||||||
|
INTEGRITY_HISTORY_END=
|
||||||
|
|
||||||
|
# 是否包含维度表校验
|
||||||
|
INTEGRITY_INCLUDE_DIMENSIONS=true
|
||||||
|
|
||||||
|
# 发现丢失数据时是否自动补全
|
||||||
|
INTEGRITY_AUTO_BACKFILL=true
|
||||||
|
|
||||||
|
# 自动执行完整性检查(ETL 完成后自动触发)
|
||||||
|
INTEGRITY_AUTO_CHECK=false
|
||||||
|
|
||||||
|
# 指定要校验的 ODS 任务代码(逗号分隔,留空表示全部)
|
||||||
|
# INTEGRITY_ODS_TASK_CODES=ODS_PAYMENT,ODS_MEMBER
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# 默认任务列表(逗号分隔,可被 CLI --tasks 参数覆盖)
|
||||||
|
# ------------------------------------------------------------------------------
|
||||||
|
# RUN_TASKS=ODS_PAYMENT,ODS_MEMBER,ODS_SETTLEMENT_RECORDS,DWD_LOAD_FROM_ODS
|
||||||
|
|
||||||
|
|||||||
BIN
etl_billiards/1.zip
Normal file
BIN
etl_billiards/1.zip
Normal file
Binary file not shown.
BIN
etl_billiards/ETL_Manager.exe - 快捷方式.lnk
Normal file
BIN
etl_billiards/ETL_Manager.exe - 快捷方式.lnk
Normal file
Binary file not shown.
257
etl_billiards/build_exe.py
Normal file
257
etl_billiards/build_exe.py
Normal file
@@ -0,0 +1,257 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""
|
||||||
|
ETL GUI 打包脚本
|
||||||
|
|
||||||
|
使用 PyInstaller 将 GUI 应用打包为 Windows EXE
|
||||||
|
|
||||||
|
用法:
|
||||||
|
python build_exe.py [--onefile] [--console] [--clean]
|
||||||
|
|
||||||
|
参数:
|
||||||
|
--onefile 打包为单个 EXE 文件(默认为目录模式)
|
||||||
|
--console 显示控制台窗口(调试用)
|
||||||
|
--clean 打包前清理旧的构建文件
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def get_project_root() -> Path:
|
||||||
|
"""获取项目根目录"""
|
||||||
|
return Path(__file__).resolve().parent
|
||||||
|
|
||||||
|
|
||||||
|
def clean_build():
|
||||||
|
"""清理旧的构建文件"""
|
||||||
|
project_root = get_project_root()
|
||||||
|
|
||||||
|
dirs_to_clean = [
|
||||||
|
project_root / "build",
|
||||||
|
project_root / "dist",
|
||||||
|
]
|
||||||
|
|
||||||
|
files_to_clean = [
|
||||||
|
project_root / "etl_gui.spec",
|
||||||
|
]
|
||||||
|
|
||||||
|
for d in dirs_to_clean:
|
||||||
|
if d.exists():
|
||||||
|
print(f"清理目录: {d}")
|
||||||
|
shutil.rmtree(d)
|
||||||
|
|
||||||
|
for f in files_to_clean:
|
||||||
|
if f.exists():
|
||||||
|
print(f"清理文件: {f}")
|
||||||
|
f.unlink()
|
||||||
|
|
||||||
|
|
||||||
|
def build_exe(onefile: bool = False, console: bool = False):
|
||||||
|
"""构建 EXE"""
|
||||||
|
project_root = get_project_root()
|
||||||
|
|
||||||
|
# 主入口
|
||||||
|
main_script = project_root / "gui" / "main.py"
|
||||||
|
|
||||||
|
# 资源文件
|
||||||
|
resources_dir = project_root / "gui" / "resources"
|
||||||
|
database_dir = project_root / "database"
|
||||||
|
|
||||||
|
# 构建 PyInstaller 命令
|
||||||
|
# 使用 ASCII 名称避免 Windows 控制台编码问题
|
||||||
|
cmd = [
|
||||||
|
sys.executable, "-m", "PyInstaller",
|
||||||
|
"--name", "ETL_Manager",
|
||||||
|
"--noconfirm",
|
||||||
|
]
|
||||||
|
|
||||||
|
# 单文件或目录模式
|
||||||
|
if onefile:
|
||||||
|
cmd.append("--onefile")
|
||||||
|
else:
|
||||||
|
cmd.append("--onedir")
|
||||||
|
|
||||||
|
# 窗口模式
|
||||||
|
if not console:
|
||||||
|
cmd.append("--windowed")
|
||||||
|
|
||||||
|
# 添加数据文件
|
||||||
|
# 样式表
|
||||||
|
if resources_dir.exists():
|
||||||
|
cmd.extend(["--add-data", f"{resources_dir};gui/resources"])
|
||||||
|
|
||||||
|
# 数据库 SQL 文件
|
||||||
|
if database_dir.exists():
|
||||||
|
for sql_file in database_dir.glob("*.sql"):
|
||||||
|
cmd.extend(["--add-data", f"{sql_file};database"])
|
||||||
|
|
||||||
|
# 隐式导入
|
||||||
|
hidden_imports = [
|
||||||
|
# PySide6 核心模块
|
||||||
|
"PySide6.QtCore",
|
||||||
|
"PySide6.QtGui",
|
||||||
|
"PySide6.QtWidgets",
|
||||||
|
# 数据库
|
||||||
|
"psycopg2",
|
||||||
|
"psycopg2.extras",
|
||||||
|
"psycopg2.extensions",
|
||||||
|
# GUI 模块
|
||||||
|
"gui.models.task_model",
|
||||||
|
"gui.models.schedule_model",
|
||||||
|
"gui.utils.cli_builder",
|
||||||
|
"gui.utils.config_helper",
|
||||||
|
"gui.utils.app_settings",
|
||||||
|
"gui.workers.task_worker",
|
||||||
|
"gui.workers.db_worker",
|
||||||
|
"gui.widgets.settings_dialog",
|
||||||
|
]
|
||||||
|
for imp in hidden_imports:
|
||||||
|
cmd.extend(["--hidden-import", imp])
|
||||||
|
|
||||||
|
# 排除不需要的模块(减小体积)
|
||||||
|
excludes = [
|
||||||
|
"matplotlib",
|
||||||
|
"numpy",
|
||||||
|
"pandas",
|
||||||
|
"scipy",
|
||||||
|
"PIL",
|
||||||
|
"cv2",
|
||||||
|
"tkinter",
|
||||||
|
]
|
||||||
|
for exc in excludes:
|
||||||
|
cmd.extend(["--exclude-module", exc])
|
||||||
|
|
||||||
|
# 工作目录
|
||||||
|
cmd.extend(["--workpath", str(project_root / "build")])
|
||||||
|
cmd.extend(["--distpath", str(project_root / "dist")])
|
||||||
|
cmd.extend(["--specpath", str(project_root)])
|
||||||
|
|
||||||
|
# 主脚本
|
||||||
|
cmd.append(str(main_script))
|
||||||
|
|
||||||
|
print("执行命令:")
|
||||||
|
print(" ".join(cmd))
|
||||||
|
print()
|
||||||
|
|
||||||
|
# 执行打包
|
||||||
|
result = subprocess.run(cmd, cwd=str(project_root))
|
||||||
|
|
||||||
|
if result.returncode == 0:
|
||||||
|
# 打包后精简:删除不需要的文件
|
||||||
|
slim_dist(project_root / "dist" / "ETL_Manager" / "_internal")
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("=" * 50)
|
||||||
|
print("打包成功!")
|
||||||
|
print(f"输出目录: {project_root / 'dist'}")
|
||||||
|
print("=" * 50)
|
||||||
|
else:
|
||||||
|
print()
|
||||||
|
print("打包失败,请检查错误信息")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def slim_dist(internal_dir: Path):
|
||||||
|
"""精简打包后的文件,删除不需要的内容"""
|
||||||
|
if not internal_dir.exists():
|
||||||
|
return
|
||||||
|
|
||||||
|
print()
|
||||||
|
print("精简打包文件...")
|
||||||
|
|
||||||
|
removed_size = 0
|
||||||
|
|
||||||
|
# 1. 删除不需要的翻译文件(只保留中文和英文)
|
||||||
|
translations_dir = internal_dir / "PySide6" / "translations"
|
||||||
|
if translations_dir.exists():
|
||||||
|
keep_langs = {"zh_CN", "zh_TW", "en"}
|
||||||
|
for qm_file in translations_dir.glob("*.qm"):
|
||||||
|
# 检查是否是需要保留的语言
|
||||||
|
keep = False
|
||||||
|
for lang in keep_langs:
|
||||||
|
if lang in qm_file.name:
|
||||||
|
keep = True
|
||||||
|
break
|
||||||
|
if not keep:
|
||||||
|
size = qm_file.stat().st_size
|
||||||
|
qm_file.unlink()
|
||||||
|
removed_size += size
|
||||||
|
|
||||||
|
# 2. 删除 opengl32sw.dll(软件渲染,20MB,通常不需要)
|
||||||
|
opengl_sw = internal_dir / "PySide6" / "opengl32sw.dll"
|
||||||
|
if opengl_sw.exists():
|
||||||
|
size = opengl_sw.stat().st_size
|
||||||
|
opengl_sw.unlink()
|
||||||
|
removed_size += size
|
||||||
|
print(f" 删除: opengl32sw.dll ({size / 1024 / 1024:.1f} MB)")
|
||||||
|
|
||||||
|
# 3. 删除不需要的 Qt 模块 DLL(如果存在)
|
||||||
|
unnecessary_dlls = [
|
||||||
|
"Qt6Pdf.dll", # PDF 支持
|
||||||
|
"Qt6Qml.dll", # QML 引擎
|
||||||
|
"Qt6QmlMeta.dll",
|
||||||
|
"Qt6QmlModels.dll",
|
||||||
|
"Qt6QmlWorkerScript.dll",
|
||||||
|
"Qt6Quick.dll", # Quick UI
|
||||||
|
"Qt6VirtualKeyboard.dll", # 虚拟键盘
|
||||||
|
]
|
||||||
|
pyside6_dir = internal_dir / "PySide6"
|
||||||
|
for dll_name in unnecessary_dlls:
|
||||||
|
dll_path = pyside6_dir / dll_name
|
||||||
|
if dll_path.exists():
|
||||||
|
size = dll_path.stat().st_size
|
||||||
|
dll_path.unlink()
|
||||||
|
removed_size += size
|
||||||
|
print(f" 删除: {dll_name} ({size / 1024 / 1024:.1f} MB)")
|
||||||
|
|
||||||
|
# 4. 删除不需要的插件目录
|
||||||
|
unnecessary_plugins = [
|
||||||
|
"networkinformation", # 网络信息
|
||||||
|
"tls", # TLS 支持(数据库已有)
|
||||||
|
]
|
||||||
|
plugins_dir = pyside6_dir / "plugins"
|
||||||
|
if plugins_dir.exists():
|
||||||
|
for plugin_name in unnecessary_plugins:
|
||||||
|
plugin_path = plugins_dir / plugin_name
|
||||||
|
if plugin_path.exists():
|
||||||
|
size = sum(f.stat().st_size for f in plugin_path.rglob("*") if f.is_file())
|
||||||
|
shutil.rmtree(plugin_path)
|
||||||
|
removed_size += size
|
||||||
|
print(f" 删除插件: {plugin_name}")
|
||||||
|
|
||||||
|
print(f"共节省: {removed_size / 1024 / 1024:.1f} MB")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""主函数"""
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description="ETL GUI 打包工具")
|
||||||
|
parser.add_argument("--onefile", action="store_true", help="打包为单个 EXE")
|
||||||
|
parser.add_argument("--console", action="store_true", help="显示控制台窗口")
|
||||||
|
parser.add_argument("--clean", action="store_true", help="打包前清理")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# 检查 PyInstaller
|
||||||
|
try:
|
||||||
|
import PyInstaller
|
||||||
|
print(f"PyInstaller 版本: {PyInstaller.__version__}")
|
||||||
|
except ImportError:
|
||||||
|
print("错误: 未安装 PyInstaller")
|
||||||
|
print("请运行: pip install pyinstaller")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# 清理
|
||||||
|
if args.clean:
|
||||||
|
clean_build()
|
||||||
|
|
||||||
|
# 构建
|
||||||
|
build_exe(onefile=args.onefile, console=args.console)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -57,6 +57,17 @@ def parse_args():
|
|||||||
action="store_true",
|
action="store_true",
|
||||||
help="强制使用 window_start/window_end,不走 MAX(fetched_at) 兜底",
|
help="强制使用 window_start/window_end,不走 MAX(fetched_at) 兜底",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--window-split-unit",
|
||||||
|
dest="window_split_unit",
|
||||||
|
help="窗口切分单位(month/none),默认来自配置 run.window_split.unit",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--window-compensation-hours",
|
||||||
|
dest="window_compensation_hours",
|
||||||
|
type=int,
|
||||||
|
help="窗口前后补偿小时数,默认来自配置 run.window_split.compensation_hours",
|
||||||
|
)
|
||||||
|
|
||||||
# 目录参数
|
# 目录参数
|
||||||
parser.add_argument("--export-root", help="导出根目录")
|
parser.add_argument("--export-root", help="导出根目录")
|
||||||
@@ -134,6 +145,12 @@ def build_cli_overrides(args) -> dict:
|
|||||||
overrides["run"]["window_override"]["end"] = args.window_end
|
overrides["run"]["window_override"]["end"] = args.window_end
|
||||||
if args.force_window_override:
|
if args.force_window_override:
|
||||||
overrides.setdefault("run", {})["force_window_override"] = True
|
overrides.setdefault("run", {})["force_window_override"] = True
|
||||||
|
if args.window_split_unit:
|
||||||
|
overrides.setdefault("run", {}).setdefault("window_split", {})["unit"] = args.window_split_unit
|
||||||
|
if args.window_compensation_hours is not None:
|
||||||
|
overrides.setdefault("run", {}).setdefault("window_split", {})[
|
||||||
|
"compensation_hours"
|
||||||
|
] = args.window_compensation_hours
|
||||||
|
|
||||||
# 运行窗口
|
# 运行窗口
|
||||||
if args.idle_start:
|
if args.idle_start:
|
||||||
|
|||||||
@@ -58,6 +58,10 @@ DEFAULTS = {
|
|||||||
"default_idle": 180,
|
"default_idle": 180,
|
||||||
},
|
},
|
||||||
"overlap_seconds": 120,
|
"overlap_seconds": 120,
|
||||||
|
"window_split": {
|
||||||
|
"unit": "month",
|
||||||
|
"compensation_hours": 2,
|
||||||
|
},
|
||||||
"idle_window": {
|
"idle_window": {
|
||||||
"start": "04:00",
|
"start": "04:00",
|
||||||
"end": "16:00",
|
"end": "16:00",
|
||||||
@@ -65,8 +69,8 @@ DEFAULTS = {
|
|||||||
"allow_empty_result_advance": True,
|
"allow_empty_result_advance": True,
|
||||||
},
|
},
|
||||||
"io": {
|
"io": {
|
||||||
"export_root": r"C:\dev\LLTQ\export\JSON",
|
"export_root": "export/JSON",
|
||||||
"log_root": r"C:\dev\LLTQ\export\LOG",
|
"log_root": "export/LOG",
|
||||||
"manifest_name": "manifest.json",
|
"manifest_name": "manifest.json",
|
||||||
"ingest_report_name": "ingest_report.json",
|
"ingest_report_name": "ingest_report.json",
|
||||||
"write_pretty_json": True,
|
"write_pretty_json": True,
|
||||||
@@ -76,7 +80,7 @@ DEFAULTS = {
|
|||||||
# 运行流程:FETCH_ONLY(仅在线抓取落盘)、INGEST_ONLY(本地清洗入库)、FULL(抓取 + 清洗入库)
|
# 运行流程:FETCH_ONLY(仅在线抓取落盘)、INGEST_ONLY(本地清洗入库)、FULL(抓取 + 清洗入库)
|
||||||
"flow": "FULL",
|
"flow": "FULL",
|
||||||
# 在线抓取 JSON 输出根目录(按任务、run_id 与时间自动创建子目录)
|
# 在线抓取 JSON 输出根目录(按任务、run_id 与时间自动创建子目录)
|
||||||
"fetch_root": r"C:\dev\LLTQ\export\JSON",
|
"fetch_root": "export/JSON",
|
||||||
# 本地清洗入库时的 JSON 输入目录(为空则默认使用本次抓取目录)
|
# 本地清洗入库时的 JSON 输入目录(为空则默认使用本次抓取目录)
|
||||||
"ingest_source_dir": "",
|
"ingest_source_dir": "",
|
||||||
},
|
},
|
||||||
@@ -97,10 +101,19 @@ DEFAULTS = {
|
|||||||
},
|
},
|
||||||
"ods": {
|
"ods": {
|
||||||
# ODS 离线重建/回放相关(仅开发/运维使用)
|
# ODS 离线重建/回放相关(仅开发/运维使用)
|
||||||
"json_doc_dir": r"C:\dev\LLTQ\export\test-json-doc",
|
"json_doc_dir": "export/test-json-doc",
|
||||||
"include_files": "",
|
"include_files": "",
|
||||||
"drop_schema_first": True,
|
"drop_schema_first": True,
|
||||||
},
|
},
|
||||||
|
"integrity": {
|
||||||
|
"mode": "history",
|
||||||
|
"history_start": "2025-07-01",
|
||||||
|
"history_end": "",
|
||||||
|
"include_dimensions": False,
|
||||||
|
"auto_check": False,
|
||||||
|
"ods_task_codes": "",
|
||||||
|
},
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# 任务代码常量
|
# 任务代码常量
|
||||||
|
|||||||
@@ -42,11 +42,20 @@ ENV_MAP = {
|
|||||||
"ALLOW_EMPTY_ADVANCE": ("run.allow_empty_result_advance",),
|
"ALLOW_EMPTY_ADVANCE": ("run.allow_empty_result_advance",),
|
||||||
"WINDOW_START": ("run.window_override.start",),
|
"WINDOW_START": ("run.window_override.start",),
|
||||||
"WINDOW_END": ("run.window_override.end",),
|
"WINDOW_END": ("run.window_override.end",),
|
||||||
|
"WINDOW_SPLIT_UNIT": ("run.window_split.unit",),
|
||||||
|
"WINDOW_COMPENSATION_HOURS": ("run.window_split.compensation_hours",),
|
||||||
"PIPELINE_FLOW": ("pipeline.flow",),
|
"PIPELINE_FLOW": ("pipeline.flow",),
|
||||||
"JSON_FETCH_ROOT": ("pipeline.fetch_root",),
|
"JSON_FETCH_ROOT": ("pipeline.fetch_root",),
|
||||||
"JSON_SOURCE_DIR": ("pipeline.ingest_source_dir",),
|
"JSON_SOURCE_DIR": ("pipeline.ingest_source_dir",),
|
||||||
"FETCH_ROOT": ("pipeline.fetch_root",),
|
"FETCH_ROOT": ("pipeline.fetch_root",),
|
||||||
"INGEST_SOURCE_DIR": ("pipeline.ingest_source_dir",),
|
"INGEST_SOURCE_DIR": ("pipeline.ingest_source_dir",),
|
||||||
|
"INTEGRITY_MODE": ("integrity.mode",),
|
||||||
|
"INTEGRITY_HISTORY_START": ("integrity.history_start",),
|
||||||
|
"INTEGRITY_HISTORY_END": ("integrity.history_end",),
|
||||||
|
"INTEGRITY_INCLUDE_DIMENSIONS": ("integrity.include_dimensions",),
|
||||||
|
"INTEGRITY_AUTO_CHECK": ("integrity.auto_check",),
|
||||||
|
"INTEGRITY_AUTO_BACKFILL": ("integrity.auto_backfill",),
|
||||||
|
"INTEGRITY_ODS_TASK_CODES": ("integrity.ods_task_codes",),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.member_profiles (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.member_profiles IS 'ODS 原始明细表:会员档案/会员账户信息。来源:C:/dev/LLTQ/export/test-json-doc/member_profiles.json;分析:member_profiles-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.member_profiles IS 'ODS 原始明细表:会员档案/会员账户信息。来源:export/test-json-doc/member_profiles.json;分析:member_profiles-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.register_site_id IS '【说明】会员的注册门店 ID。 【示例】2790685415443269(用于会员的注册门店 ID)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - register_site_id。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.register_site_id IS '【说明】会员的注册门店 ID。 【示例】2790685415443269(用于会员的注册门店 ID)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - register_site_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.site_name IS '【说明】注册门店名称,属于冗余字段,用于直接展示。 【示例】朗朗桌球(注册门店名称,属于冗余字段,用于直接展示)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - site_name。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.site_name IS '【说明】注册门店名称,属于冗余字段,用于直接展示。 【示例】朗朗桌球(注册门店名称,属于冗余字段,用于直接展示)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - site_name。';
|
||||||
@@ -42,7 +42,7 @@ COMMENT ON COLUMN billiards_ods.member_profiles.status IS '【说明】帐户状
|
|||||||
COMMENT ON COLUMN billiards_ods.member_profiles.user_status IS '【说明】用户账号状态(偏“用户逻辑”层面的状态)。 【示例】1(用于用户账号状态(偏“用户逻辑”层面的状态))。 【JSON字段】member_profiles.json - data.tenantMemberInfos - user_status。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.user_status IS '【说明】用户账号状态(偏“用户逻辑”层面的状态)。 【示例】1(用于用户账号状态(偏“用户逻辑”层面的状态))。 【JSON字段】member_profiles.json - data.tenantMemberInfos - user_status。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.create_time IS '【说明】会员账户的创建时间(即这条档案/这张卡在系统中被创建的时间)。 【示例】2025-11-08 01:29:33(用于会员账户的创建时间(即这条档案/这张卡在系统中被创建的时间))。 【JSON字段】member_profiles.json - data.tenantMemberInfos - create_time。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.create_time IS '【说明】会员账户的创建时间(即这条档案/这张卡在系统中被创建的时间)。 【示例】2025-11-08 01:29:33(用于会员账户的创建时间(即这条档案/这张卡在系统中被创建的时间))。 【JSON字段】member_profiles.json - data.tenantMemberInfos - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_profiles.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_profiles.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/member_profiles.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/member_profiles.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_profiles.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_profiles.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - $。';
|
COMMENT ON COLUMN billiards_ods.member_profiles.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_profiles.json - data.tenantMemberInfos - $。';
|
||||||
|
|
||||||
@@ -79,7 +79,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.member_balance_changes (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.member_balance_changes IS 'ODS 原始明细表:会员余额变更流水。来源:C:/dev/LLTQ/export/test-json-doc/member_balance_changes.json;分析:member_balance_changes-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.member_balance_changes IS 'ODS 原始明细表:会员余额变更流水。来源:export/test-json-doc/member_balance_changes.json;分析:member_balance_changes-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.tenant_id IS '【说明】租户/商户 ID,本数据中是固定值(同一品牌/商户)。 【示例】2790683160709957(用于租户/商户 ID,本数据中是固定值(同一品牌/商户))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.tenant_id IS '【说明】租户/商户 ID,本数据中是固定值(同一品牌/商户)。 【示例】2790683160709957(用于租户/商户 ID,本数据中是固定值(同一品牌/商户))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.site_id IS '【说明】非 0:记录所属的具体门店 ID(与其他 JSON 内的 site_id 一致)。 【示例】2790685415443269(用于非 0:记录所属的具体门店 ID(与其他 JSON 内的 site_id 一致))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - site_id。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.site_id IS '【说明】非 0:记录所属的具体门店 ID(与其他 JSON 内的 site_id 一致)。 【示例】2790685415443269(用于非 0:记录所属的具体门店 ID(与其他 JSON 内的 site_id 一致))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - site_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.register_site_id IS '【说明】会员卡的“注册门店 ID”,即办卡所在门店。 【示例】2790685415443269(用于会员卡的“注册门店 ID”,即办卡所在门店)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - register_site_id。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.register_site_id IS '【说明】会员卡的“注册门店 ID”,即办卡所在门店。 【示例】2790685415443269(用于会员卡的“注册门店 ID”,即办卡所在门店)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - register_site_id。';
|
||||||
@@ -106,7 +106,7 @@ COMMENT ON COLUMN billiards_ods.member_balance_changes.operator_name IS '【说
|
|||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.is_delete IS '【说明】逻辑删除标记(0=否,1=是)。 【示例】0(用于逻辑删除标记(0=否,1=是))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - is_delete。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.is_delete IS '【说明】逻辑删除标记(0=否,1=是)。 【示例】0(用于逻辑删除标记(0=否,1=是))。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - is_delete。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.create_time IS '【说明】本条余额变更记录的创建时间,通常接近交易发生时间。 【示例】2025-11-09 22:52:48(用于本条余额变更记录的创建时间,通常接近交易发生时间)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - create_time。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.create_time IS '【说明】本条余额变更记录的创建时间,通常接近交易发生时间。 【示例】2025-11-09 22:52:48(用于本条余额变更记录的创建时间,通常接近交易发生时间)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_balance_changes.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_balance_changes.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/member_balance_changes.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/member_balance_changes.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_balance_changes.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_balance_changes.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - $。';
|
COMMENT ON COLUMN billiards_ods.member_balance_changes.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_balance_changes.json - data.tenantMemberCardLogs - $。';
|
||||||
|
|
||||||
@@ -187,7 +187,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.member_stored_value_cards (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.member_stored_value_cards IS 'ODS 原始明细表:会员储值/卡券账户列表。来源:C:/dev/LLTQ/export/test-json-doc/member_stored_value_cards.json;分析:member_stored_value_cards-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.member_stored_value_cards IS 'ODS 原始明细表:会员储值/卡券账户列表。来源:export/test-json-doc/member_stored_value_cards.json;分析:member_stored_value_cards-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.tenant_id IS '【说明】租户/品牌 ID,与其他 JSON 中 tenant_id 一致。 【示例】2790683160709957(用于租户/品牌 ID,与其他 JSON 中 tenant_id 一致)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.tenant_id IS '【说明】租户/品牌 ID,与其他 JSON 中 tenant_id 一致。 【示例】2790683160709957(用于租户/品牌 ID,与其他 JSON 中 tenant_id 一致)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.tenant_member_id IS '【说明】当前商户(品牌/租户)中会员的主键 ID。 【示例】2955204541320325(用于当前商户(品牌/租户)中会员的主键 ID)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - tenant_member_id。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.tenant_member_id IS '【说明】当前商户(品牌/租户)中会员的主键 ID。 【示例】2955204541320325(用于当前商户(品牌/租户)中会员的主键 ID)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - tenant_member_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.system_member_id IS '【说明】系统级会员 ID(跨门店统一主键)。 【示例】2955204540009605(用于系统级会员 ID(跨门店统一主键))。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - system_member_id。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.system_member_id IS '【说明】系统级会员 ID(跨门店统一主键)。 【示例】2955204540009605(用于系统级会员 ID(跨门店统一主键))。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - system_member_id。';
|
||||||
@@ -258,7 +258,7 @@ COMMENT ON COLUMN billiards_ods.member_stored_value_cards.tenantName IS '【说
|
|||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.pdAssisnatLevel IS '【说明】允许使用的“陪打/助教等级”列表。 【示例】[](用于允许使用的“陪打/助教等级”列表)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - pdAssisnatLevel。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.pdAssisnatLevel IS '【说明】允许使用的“陪打/助教等级”列表。 【示例】[](用于允许使用的“陪打/助教等级”列表)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - pdAssisnatLevel。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.cxAssisnatLevel IS '【说明】可能是“促销活动中的助教等级限制”(命名中 cx 多为“促销”缩写)。 【示例】[](用于可能是“促销活动中的助教等级限制”(命名中 cx 多为“促销”缩写))。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - cxAssisnatLevel。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.cxAssisnatLevel IS '【说明】可能是“促销活动中的助教等级限制”(命名中 cx 多为“促销”缩写)。 【示例】[](用于可能是“促销活动中的助教等级限制”(命名中 cx 多为“促销”缩写))。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - cxAssisnatLevel。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_stored_value_cards.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】member_stored_value_cards.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/member_stored_value_cards.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/member_stored_value_cards.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】member_stored_value_cards.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - $。';
|
COMMENT ON COLUMN billiards_ods.member_stored_value_cards.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】member_stored_value_cards.json - data.tenantMemberCards - $。';
|
||||||
|
|
||||||
@@ -331,7 +331,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.recharge_settlements (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.recharge_settlements IS 'ODS 原始明细表:充值结算记录。来源:C:/dev/LLTQ/export/test-json-doc/recharge_settlements.json;分析:recharge_settlements-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.recharge_settlements IS 'ODS 原始明细表:充值结算记录。来源:export/test-json-doc/recharge_settlements.json;分析:recharge_settlements-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.id IS '【说明】门店 ID。 【示例】NULL(用于门店 ID)。 【JSON字段】recharge_settlements.json - $ - id。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.id IS '【说明】门店 ID。 【示例】NULL(用于门店 ID)。 【JSON字段】recharge_settlements.json - $ - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.tenantid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】recharge_settlements.json - $ - tenantid。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.tenantid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】recharge_settlements.json - $ - tenantid。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.siteid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】recharge_settlements.json - $ - siteid。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.siteid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】recharge_settlements.json - $ - siteid。';
|
||||||
@@ -394,7 +394,7 @@ COMMENT ON COLUMN billiards_ods.recharge_settlements.isfirst IS '【说明】来
|
|||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.rechargecardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】recharge_settlements.json - $ - rechargecardamount。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.rechargecardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】recharge_settlements.json - $ - rechargecardamount。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.giftcardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】recharge_settlements.json - $ - giftcardamount。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.giftcardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】recharge_settlements.json - $ - giftcardamount。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】recharge_settlements.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】recharge_settlements.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/recharge_settlements.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/recharge_settlements.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】recharge_settlements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.recharge_settlements.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】recharge_settlements.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.recharge_settlements.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】recharge_settlements.json - $ - $。';
|
||||||
|
|
||||||
@@ -467,7 +467,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.settlement_records (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.settlement_records IS 'ODS 原始明细表:结账/结算记录。来源:C:/dev/LLTQ/export/test-json-doc/settlement_records.json;分析:settlement_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.settlement_records IS 'ODS 原始明细表:结账/结算记录。来源:export/test-json-doc/settlement_records.json;分析:settlement_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.id IS '【说明】结账记录主键 ID(订单结算 ID)。 【示例】NULL(用于结账记录主键 ID(订单结算 ID))。 【JSON字段】settlement_records.json - $ - id。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.id IS '【说明】结账记录主键 ID(订单结算 ID)。 【示例】NULL(用于结账记录主键 ID(订单结算 ID))。 【JSON字段】settlement_records.json - $ - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.tenantid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_records.json - $ - tenantid。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.tenantid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_records.json - $ - tenantid。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.siteid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_records.json - $ - siteid。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.siteid IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_records.json - $ - siteid。';
|
||||||
@@ -530,7 +530,7 @@ COMMENT ON COLUMN billiards_ods.settlement_records.isfirst IS '【说明】来
|
|||||||
COMMENT ON COLUMN billiards_ods.settlement_records.rechargecardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】settlement_records.json - $ - rechargecardamount。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.rechargecardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】settlement_records.json - $ - rechargecardamount。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.giftcardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】settlement_records.json - $ - giftcardamount。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.giftcardamount IS '【说明】金额字段,用于计费/结算/分摊等金额计算。 【示例】NULL(金额字段,用于计费/结算/分摊等金额计算)。 【JSON字段】settlement_records.json - $ - giftcardamount。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】settlement_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】settlement_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/settlement_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/settlement_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】settlement_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】settlement_records.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.settlement_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】settlement_records.json - $ - $。';
|
||||||
|
|
||||||
@@ -555,7 +555,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.assistant_cancellation_records (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.assistant_cancellation_records IS 'ODS 原始明细表:助教作废/取消记录。来源:C:/dev/LLTQ/export/test-json-doc/assistant_cancellation_records.json;分析:assistant_cancellation_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.assistant_cancellation_records IS 'ODS 原始明细表:助教作废/取消记录。来源:export/test-json-doc/assistant_cancellation_records.json;分析:assistant_cancellation_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.id IS '【说明】本表主键 ID,用于唯一标识一条记录。 【示例】2957675849518789(本表主键 ID,用于唯一标识一条记录)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - id。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.id IS '【说明】本表主键 ID,用于唯一标识一条记录。 【示例】2957675849518789(本表主键 ID,用于唯一标识一条记录)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.siteId IS '【说明】门店 ID,即该废除记录所在门店。 【示例】2790685415443269(用于门店 ID,即该废除记录所在门店)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - siteId。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.siteId IS '【说明】门店 ID,即该废除记录所在门店。 【示例】2790685415443269(用于门店 ID,即该废除记录所在门店)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - siteId。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.siteProfile IS '【说明】门店信息快照。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - siteProfile。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.siteProfile IS '【说明】门店信息快照。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - siteProfile。';
|
||||||
@@ -570,7 +570,7 @@ COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.tableName IS '【
|
|||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.trashReason IS '【说明】用于记录“废除原因”的文本描述,例如“顾客临时有事取消”“录入错误”“更换助教”等。 【示例】NULL(用于记录“废除原因”的文本描述,例如“顾客临时有事取消”“录入错误”“更换助教”等)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - trashReason。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.trashReason IS '【说明】用于记录“废除原因”的文本描述,例如“顾客临时有事取消”“录入错误”“更换助教”等。 【示例】NULL(用于记录“废除原因”的文本描述,例如“顾客临时有事取消”“录入错误”“更换助教”等)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - trashReason。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.createTime IS '【说明】这条“助教废除记录”被创建的时间,即系统正式记录“废除”操作的时刻。 【示例】2025-11-09 19:23:29(用于这条“助教废除记录”被创建的时间,即系统正式记录“废除”操作的时刻)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - createTime。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.createTime IS '【说明】这条“助教废除记录”被创建的时间,即系统正式记录“废除”操作的时刻。 【示例】2025-11-09 19:23:29(用于这条“助教废除记录”被创建的时间,即系统正式记录“废除”操作的时刻)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - createTime。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_cancellation_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_cancellation_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/assistant_cancellation_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/assistant_cancellation_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_cancellation_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - $。';
|
COMMENT ON COLUMN billiards_ods.assistant_cancellation_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_cancellation_records.json - data.abolitionAssistants - $。';
|
||||||
|
|
||||||
@@ -644,7 +644,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.assistant_accounts_master (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.assistant_accounts_master IS 'ODS 原始明细表:助教档案主数据。来源:C:/dev/LLTQ/export/test-json-doc/assistant_accounts_master.json;分析:assistant_accounts_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.assistant_accounts_master IS 'ODS 原始明细表:助教档案主数据。来源:export/test-json-doc/assistant_accounts_master.json;分析:assistant_accounts_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.id IS '【说明】助教账号主键 ID,在“助教流水.json”中对应 site_assistant_id。 【示例】2947562271297029(用于助教账号主键 ID,在“助教流水.json”中对应 site_assistant_id)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - id。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.id IS '【说明】助教账号主键 ID,在“助教流水.json”中对应 site_assistant_id。 【示例】2947562271297029(用于助教账号主键 ID,在“助教流水.json”中对应 site_assistant_id)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.tenant_id IS '【说明】品牌/租户 ID,对应“非球科技”系统中该商户的唯一标识。 【示例】2790683160709957(用于品牌/租户 ID,对应“非球科技”系统中该商户的唯一标识)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.tenant_id IS '【说明】品牌/租户 ID,对应“非球科技”系统中该商户的唯一标识。 【示例】2790683160709957(用于品牌/租户 ID,对应“非球科技”系统中该商户的唯一标识)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.site_id IS '【说明】门店 ID,对应本次数据的这家球房(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,对应本次数据的这家球房(朗朗桌球))。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - site_id。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.site_id IS '【说明】门店 ID,对应本次数据的这家球房(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,对应本次数据的这家球房(朗朗桌球))。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - site_id。';
|
||||||
@@ -708,7 +708,7 @@ COMMENT ON COLUMN billiards_ods.assistant_accounts_master.light_equipment_id IS
|
|||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.entry_sign_status IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】0(来自 JSON 导出的原始字段,用于保留业务取值。)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - entry_sign_status。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.entry_sign_status IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】0(来自 JSON 导出的原始字段,用于保留业务取值。)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - entry_sign_status。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.resign_sign_status IS '【说明】离职协议签署状态,类似上面。 【示例】0(用于离职协议签署状态,类似上面)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - resign_sign_status。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.resign_sign_status IS '【说明】离职协议签署状态,类似上面。 【示例】0(用于离职协议签署状态,类似上面)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - resign_sign_status。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_accounts_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_accounts_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/assistant_accounts_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/assistant_accounts_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_accounts_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - $。';
|
COMMENT ON COLUMN billiards_ods.assistant_accounts_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_accounts_master.json - data.assistantInfos - $。';
|
||||||
|
|
||||||
@@ -784,7 +784,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.assistant_service_records (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.assistant_service_records IS 'ODS 原始明细表:助教服务流水。来源:C:/dev/LLTQ/export/test-json-doc/assistant_service_records.json;分析:assistant_service_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.assistant_service_records IS 'ODS 原始明细表:助教服务流水。来源:export/test-json-doc/assistant_service_records.json;分析:assistant_service_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.id IS '【说明】本条助教流水记录的主键 ID(流水唯一标识)。 【示例】2957913441292165(用于本条助教流水记录的主键 ID(流水唯一标识))。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - id。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.id IS '【说明】本条助教流水记录的主键 ID(流水唯一标识)。 【示例】2957913441292165(用于本条助教流水记录的主键 ID(流水唯一标识))。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.site_id IS '【说明】门店 ID,本数据中指“朗朗桌球”这一家门店。 【示例】2790685415443269(用于门店 ID,本数据中指“朗朗桌球”这一家门店)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - site_id。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.site_id IS '【说明】门店 ID,本数据中指“朗朗桌球”这一家门店。 【示例】2790685415443269(用于门店 ID,本数据中指“朗朗桌球”这一家门店)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - site_id。';
|
||||||
@@ -851,7 +851,7 @@ COMMENT ON COLUMN billiards_ods.assistant_service_records.is_not_responding IS '
|
|||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.is_confirm IS '【说明】布尔/开关字段,用于表示权限、可用性或状态开关。 【示例】2(布尔/开关字段,用于表示权限、可用性或状态开关。)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - is_confirm。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.is_confirm IS '【说明】布尔/开关字段,用于表示权限、可用性或状态开关。 【示例】2(布尔/开关字段,用于表示权限、可用性或状态开关。)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - is_confirm。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - $。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】assistant_service_records.json - data.orderAssistantDetails - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_service_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】assistant_service_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/assistant_service_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/assistant_service_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.assistant_service_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.assistant_service_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】assistant_service_records.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -887,7 +887,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.site_tables_master (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.site_tables_master IS 'ODS 原始明细表:门店桌台主数据。来源:C:/dev/LLTQ/export/test-json-doc/site_tables_master.json;分析:site_tables_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.site_tables_master IS 'ODS 原始明细表:门店桌台主数据。来源:export/test-json-doc/site_tables_master.json;分析:site_tables_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.id IS '【说明】台桌主键 ID。 【示例】2791964216463493(用于台桌主键 ID)。 【JSON字段】site_tables_master.json - data.siteTables - id。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.id IS '【说明】台桌主键 ID。 【示例】2791964216463493(用于台桌主键 ID)。 【JSON字段】site_tables_master.json - data.siteTables - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.site_id IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】site_tables_master.json - data.siteTables - site_id。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.site_id IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】site_tables_master.json - data.siteTables - site_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.siteName IS '【说明】门店名称快照,冗余字段,配合 site_id 使用。 【示例】朗朗桌球(用于门店名称快照,冗余字段,配合 site_id 使用)。 【JSON字段】site_tables_master.json - data.siteTables - siteName。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.siteName IS '【说明】门店名称快照,冗余字段,配合 site_id 使用。 【示例】朗朗桌球(用于门店名称快照,冗余字段,配合 site_id 使用)。 【JSON字段】site_tables_master.json - data.siteTables - siteName。';
|
||||||
@@ -913,7 +913,7 @@ COMMENT ON COLUMN billiards_ods.site_tables_master.table_status IS '【说明】
|
|||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.temporary_light_second IS '【说明】临时点灯时长(秒),例如手动临时开灯一段时间。 【示例】0(用于临时点灯时长(秒),例如手动临时开灯一段时间)。 【JSON字段】site_tables_master.json - data.siteTables - temporary_light_second。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.temporary_light_second IS '【说明】临时点灯时长(秒),例如手动临时开灯一段时间。 【示例】0(用于临时点灯时长(秒),例如手动临时开灯一段时间)。 【JSON字段】site_tables_master.json - data.siteTables - temporary_light_second。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.virtual_table IS '【说明】当前值:全部为 0。 【示例】0(用于当前值:全部为 0)。 【JSON字段】site_tables_master.json - data.siteTables - virtual_table。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.virtual_table IS '【说明】当前值:全部为 0。 【示例】0(用于当前值:全部为 0)。 【JSON字段】site_tables_master.json - data.siteTables - virtual_table。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】site_tables_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】site_tables_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/site_tables_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/site_tables_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】site_tables_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.site_tables_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】site_tables_master.json - data.siteTables - $。';
|
COMMENT ON COLUMN billiards_ods.site_tables_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】site_tables_master.json - data.siteTables - $。';
|
||||||
|
|
||||||
@@ -945,7 +945,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.table_fee_discount_records (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.table_fee_discount_records IS 'ODS 原始明细表:台费折扣记录。来源:C:/dev/LLTQ/export/test-json-doc/table_fee_discount_records.json;分析:table_fee_discount_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.table_fee_discount_records IS 'ODS 原始明细表:台费折扣记录。来源:export/test-json-doc/table_fee_discount_records.json;分析:table_fee_discount_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.id IS '【说明】台费打折 / 调整流水主键 ID。 【示例】2957913441881989(用于台费打折 / 调整流水主键 ID)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.id IS '【说明】台费打折 / 调整流水主键 ID。 【示例】2957913441881989(用于台费打折 / 调整流水主键 ID)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.site_id IS '【说明】门店 ID,本批数据全部为同一家门店(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,本批数据全部为同一家门店(朗朗桌球))。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - site_id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.site_id IS '【说明】门店 ID,本批数据全部为同一家门店(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,本批数据全部为同一家门店(朗朗桌球))。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - site_id。';
|
||||||
@@ -967,7 +967,7 @@ COMMENT ON COLUMN billiards_ods.table_fee_discount_records.order_trade_no IS '
|
|||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.is_delete IS '【说明】逻辑删除标记(0=否,1=是)。 【示例】0(用于逻辑删除标记(0=否,1=是))。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - is_delete。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.is_delete IS '【说明】逻辑删除标记(0=否,1=是)。 【示例】0(用于逻辑删除标记(0=否,1=是))。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - is_delete。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.create_time IS '【说明】台费调整记录的创建时间,即打折操作被执行的时间戳。 【示例】2025-11-09 23:25:11(用于台费调整记录的创建时间,即打折操作被执行的时间戳)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - create_time。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.create_time IS '【说明】台费调整记录的创建时间,即打折操作被执行的时间戳。 【示例】2025-11-09 23:25:11(用于台费调整记录的创建时间,即打折操作被执行的时间戳)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】table_fee_discount_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】table_fee_discount_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/table_fee_discount_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/table_fee_discount_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】table_fee_discount_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - $。';
|
COMMENT ON COLUMN billiards_ods.table_fee_discount_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】table_fee_discount_records.json - data.taiFeeAdjustInfos - $。';
|
||||||
|
|
||||||
@@ -1018,7 +1018,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.table_fee_transactions (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.table_fee_transactions IS 'ODS 原始明细表:台费流水。来源:C:/dev/LLTQ/export/test-json-doc/table_fee_transactions.json;分析:table_fee_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.table_fee_transactions IS 'ODS 原始明细表:台费流水。来源:export/test-json-doc/table_fee_transactions.json;分析:table_fee_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.id IS '【说明】台费流水记录主键(事实表主键)。 【示例】2957924029058885(用于台费流水记录主键(事实表主键))。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.id IS '【说明】台费流水记录主键(事实表主键)。 【示例】2957924029058885(用于台费流水记录主键(事实表主键))。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.site_id IS '【说明】门店 ID,本次数据全部来自同一门店(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,本次数据全部来自同一门店(朗朗桌球))。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - site_id。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.site_id IS '【说明】门店 ID,本次数据全部来自同一门店(朗朗桌球)。 【示例】2790685415443269(用于门店 ID,本次数据全部来自同一门店(朗朗桌球))。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - site_id。';
|
||||||
@@ -1060,7 +1060,7 @@ COMMENT ON COLUMN billiards_ods.table_fee_transactions.salesman_user_id IS '【
|
|||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.create_time IS '【说明】这条台费流水记录的创建时间,通常接近结账时间。 【示例】2025-11-09 23:35:57(用于这条台费流水记录的创建时间,通常接近结账时间)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - create_time。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.create_time IS '【说明】这条台费流水记录的创建时间,通常接近结账时间。 【示例】2025-11-09 23:35:57(用于这条台费流水记录的创建时间,通常接近结账时间)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - $。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】table_fee_transactions.json - data.siteTableUseDetailsList - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】table_fee_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】table_fee_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/table_fee_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/table_fee_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.table_fee_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.table_fee_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】table_fee_transactions.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -1090,7 +1090,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.goods_stock_movements (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.goods_stock_movements IS 'ODS 原始明细表:商品库存变动流水。来源:C:/dev/LLTQ/export/test-json-doc/goods_stock_movements.json;分析:goods_stock_movements-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.goods_stock_movements IS 'ODS 原始明细表:商品库存变动流水。来源:export/test-json-doc/goods_stock_movements.json;分析:goods_stock_movements-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.siteGoodsStockId IS '【说明】门店某个“商品库存记录”的主键 ID。 【示例】2957911857581957(用于门店某个“商品库存记录”的主键 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - siteGoodsStockId。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.siteGoodsStockId IS '【说明】门店某个“商品库存记录”的主键 ID。 【示例】2957911857581957(用于门店某个“商品库存记录”的主键 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - siteGoodsStockId。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.tenantId IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - tenantId。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.tenantId IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - tenantId。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.siteId IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - siteId。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.siteId IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - siteId。';
|
||||||
@@ -1111,7 +1111,7 @@ COMMENT ON COLUMN billiards_ods.goods_stock_movements.remark IS '【说明】备
|
|||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.operatorName IS '【说明】执行此次库存变动的操作人。 【示例】收银员:郑丽珊(用于执行此次库存变动的操作人)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - operatorName。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.operatorName IS '【说明】执行此次库存变动的操作人。 【示例】收银员:郑丽珊(用于执行此次库存变动的操作人)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - operatorName。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.createTime IS '【说明】这条库存变动记录的创建时间,即发生库存变更的时间点。 【示例】2025-11-09 23:23:34(用于这条库存变动记录的创建时间,即发生库存变更的时间点)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - createTime。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.createTime IS '【说明】这条库存变动记录的创建时间,即发生库存变更的时间点。 【示例】2025-11-09 23:23:34(用于这条库存变动记录的创建时间,即发生库存变更的时间点)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - createTime。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】goods_stock_movements.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】goods_stock_movements.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/goods_stock_movements.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/goods_stock_movements.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】goods_stock_movements.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_movements.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - $。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_movements.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】goods_stock_movements.json - data.queryDeliveryRecordsList - $。';
|
||||||
|
|
||||||
@@ -1134,7 +1134,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.stock_goods_category_tree (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.stock_goods_category_tree IS 'ODS 原始明细表:商品分类树。来源:C:/dev/LLTQ/export/test-json-doc/stock_goods_category_tree.json;分析:stock_goods_category_tree-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.stock_goods_category_tree IS 'ODS 原始明细表:商品分类树。来源:export/test-json-doc/stock_goods_category_tree.json;分析:stock_goods_category_tree-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.id IS '【说明】分类节点主键 ID(在商品分类维度中的唯一标识)。 【示例】2790683528350533(用于分类节点主键 ID(在商品分类维度中的唯一标识))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - id。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.id IS '【说明】分类节点主键 ID(在商品分类维度中的唯一标识)。 【示例】2790683528350533(用于分类节点主键 ID(在商品分类维度中的唯一标识))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.tenant_id IS '【说明】租户 ID(品牌/商户 ID)。 【示例】2790683160709957(用于租户 ID(品牌/商户 ID))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.tenant_id IS '【说明】租户 ID(品牌/商户 ID)。 【示例】2790683160709957(用于租户 ID(品牌/商户 ID))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.category_name IS '【说明】分类名称(实际业务分类名称)。 【示例】槟榔(用于分类名称(实际业务分类名称))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - category_name。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.category_name IS '【说明】分类名称(实际业务分类名称)。 【示例】槟榔(用于分类名称(实际业务分类名称))。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - category_name。';
|
||||||
@@ -1147,7 +1147,7 @@ COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.categoryBoxes IS '【
|
|||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.sort IS '【说明】分类的排序序号,用于前端展示顺序的控制。 【示例】1(分类的排序序号,用于前端展示顺序的控制)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - sort。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.sort IS '【说明】分类的排序序号,用于前端展示顺序的控制。 【示例】1(分类的排序序号,用于前端展示顺序的控制)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - sort。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.is_warehousing IS '【说明】本文件可视为“所有参与库存管理的商品分类清单”,因此均为 1。 【示例】1(用于本文件可视为“所有参与库存管理的商品分类清单”,因此均为 1)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - is_warehousing。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.is_warehousing IS '【说明】本文件可视为“所有参与库存管理的商品分类清单”,因此均为 1。 【示例】1(用于本文件可视为“所有参与库存管理的商品分类清单”,因此均为 1)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - is_warehousing。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】stock_goods_category_tree.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】stock_goods_category_tree.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/stock_goods_category_tree.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/stock_goods_category_tree.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】stock_goods_category_tree.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - $。';
|
COMMENT ON COLUMN billiards_ods.stock_goods_category_tree.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】stock_goods_category_tree.json - data.goodsCategoryList - $。';
|
||||||
|
|
||||||
@@ -1173,7 +1173,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.goods_stock_summary (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.goods_stock_summary IS 'ODS 原始明细表:商品库存汇总。来源:C:/dev/LLTQ/export/test-json-doc/goods_stock_summary.json;分析:goods_stock_summary-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.goods_stock_summary IS 'ODS 原始明细表:商品库存汇总。来源:export/test-json-doc/goods_stock_summary.json;分析:goods_stock_summary-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.siteGoodsId IS '【说明】门店商品 ID,本库存汇总表的主键,对应某个具体商品在本店的唯一标识。 【示例】2791953867886725(用于门店商品 ID,本库存汇总表的主键,对应某个具体商品在本店的唯一标识)。 【JSON字段】goods_stock_summary.json - $ - siteGoodsId。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.siteGoodsId IS '【说明】门店商品 ID,本库存汇总表的主键,对应某个具体商品在本店的唯一标识。 【示例】2791953867886725(用于门店商品 ID,本库存汇总表的主键,对应某个具体商品在本店的唯一标识)。 【JSON字段】goods_stock_summary.json - $ - siteGoodsId。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.goodsName IS '【说明】商品名称,冗余于门店商品档案的 goods_name。 【示例】东方树叶(用于商品名称,冗余于门店商品档案的 goods_name)。 【JSON字段】goods_stock_summary.json - $ - goodsName。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.goodsName IS '【说明】商品名称,冗余于门店商品档案的 goods_name。 【示例】东方树叶(用于商品名称,冗余于门店商品档案的 goods_name)。 【JSON字段】goods_stock_summary.json - $ - goodsName。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.goodsUnit IS '【说明】商品的计量单位(售卖单位)。 【示例】瓶(用于商品的计量单位(售卖单位))。 【JSON字段】goods_stock_summary.json - $ - goodsUnit。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.goodsUnit IS '【说明】商品的计量单位(售卖单位)。 【示例】瓶(用于商品的计量单位(售卖单位))。 【JSON字段】goods_stock_summary.json - $ - goodsUnit。';
|
||||||
@@ -1189,7 +1189,7 @@ COMMENT ON COLUMN billiards_ods.goods_stock_summary.rangeSaleMoney IS '【说明
|
|||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.rangeInventory IS '【说明】查询区间内的 盘点调整净变动量(盘盈–盘亏)。 【示例】0(用于查询区间内的 盘点调整净变动量(盘盈–盘亏))。 【JSON字段】goods_stock_summary.json - $ - rangeInventory。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.rangeInventory IS '【说明】查询区间内的 盘点调整净变动量(盘盈–盘亏)。 【示例】0(用于查询区间内的 盘点调整净变动量(盘盈–盘亏))。 【JSON字段】goods_stock_summary.json - $ - rangeInventory。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.currentStock IS '【说明】导出时刻的实时库存数量。 【示例】118(用于导出时刻的实时库存数量)。 【JSON字段】goods_stock_summary.json - $ - currentStock。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.currentStock IS '【说明】导出时刻的实时库存数量。 【示例】118(用于导出时刻的实时库存数量)。 【JSON字段】goods_stock_summary.json - $ - currentStock。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】goods_stock_summary.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】goods_stock_summary.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/goods_stock_summary.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/goods_stock_summary.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】goods_stock_summary.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.goods_stock_summary.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】goods_stock_summary.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.goods_stock_summary.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】goods_stock_summary.json - $ - $。';
|
||||||
|
|
||||||
@@ -1212,7 +1212,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.payment_transactions (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.payment_transactions IS 'ODS 原始明细表:支付流水。来源:C:/dev/LLTQ/export/test-json-doc/payment_transactions.json;分析:payment_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.payment_transactions IS 'ODS 原始明细表:支付流水。来源:export/test-json-doc/payment_transactions.json;分析:payment_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.id IS '【说明】支付流水记录的主键 ID。 【示例】2957924026486597(用于支付流水记录的主键 ID)。 【JSON字段】payment_transactions.json - $ - id。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.id IS '【说明】支付流水记录的主键 ID。 【示例】2957924026486597(用于支付流水记录的主键 ID)。 【JSON字段】payment_transactions.json - $ - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.site_id IS '【说明】支付记录所属的门店 ID。 【示例】2790685415443269(用于支付记录所属的门店 ID)。 【JSON字段】payment_transactions.json - $ - site_id。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.site_id IS '【说明】支付记录所属的门店 ID。 【示例】2790685415443269(用于支付记录所属的门店 ID)。 【JSON字段】payment_transactions.json - $ - site_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.siteProfile IS '【说明】门店信息快照,与其他 JSON 中的 siteProfile 结构一致。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照,与其他 JSON 中的 siteProfile 结构一致)。 【JSON字段】payment_transactions.json - $ - siteProfile。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.siteProfile IS '【说明】门店信息快照,与其他 JSON 中的 siteProfile 结构一致。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照,与其他 JSON 中的 siteProfile 结构一致)。 【JSON字段】payment_transactions.json - $ - siteProfile。';
|
||||||
@@ -1225,7 +1225,7 @@ COMMENT ON COLUMN billiards_ods.payment_transactions.create_time IS '【说明
|
|||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.payment_method IS '【说明】支付方式枚举,例如微信、支付宝、现金、银行卡、储值卡等某一种。 【示例】4(用于支付方式枚举,例如微信、支付宝、现金、银行卡、储值卡等某一种)。 【JSON字段】payment_transactions.json - $ - payment_method。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.payment_method IS '【说明】支付方式枚举,例如微信、支付宝、现金、银行卡、储值卡等某一种。 【示例】4(用于支付方式枚举,例如微信、支付宝、现金、银行卡、储值卡等某一种)。 【JSON字段】payment_transactions.json - $ - payment_method。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.online_pay_channel IS '【说明】每一笔结账单(settleList.id)对应一条支付记录(当前样本中是一条记录,relate_id 唯一)。 【示例】0(用于每一笔结账单(settleList.id)对应一条支付记录(当前样本中是一条记录,relate_id 唯一))。 【JSON字段】payment_transactions.json - $ - online_pay_channel。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.online_pay_channel IS '【说明】每一笔结账单(settleList.id)对应一条支付记录(当前样本中是一条记录,relate_id 唯一)。 【示例】0(用于每一笔结账单(settleList.id)对应一条支付记录(当前样本中是一条记录,relate_id 唯一))。 【JSON字段】payment_transactions.json - $ - online_pay_channel。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】payment_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】payment_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/payment_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/payment_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】payment_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.payment_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】payment_transactions.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.payment_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】payment_transactions.json - $ - $。';
|
||||||
|
|
||||||
@@ -1269,7 +1269,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.refund_transactions (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.refund_transactions IS 'ODS 原始明细表:退款流水。来源:C:/dev/LLTQ/export/test-json-doc/refund_transactions.json;分析:refund_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.refund_transactions IS 'ODS 原始明细表:退款流水。来源:export/test-json-doc/refund_transactions.json;分析:refund_transactions-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.id IS '【说明】本条 退款流水 的唯一 ID。 【示例】2955202296416389(用于本条 退款流水 的唯一 ID)。 【JSON字段】refund_transactions.json - $ - id。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.id IS '【说明】本条 退款流水 的唯一 ID。 【示例】2955202296416389(用于本条 退款流水 的唯一 ID)。 【JSON字段】refund_transactions.json - $ - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.tenant_id IS '【说明】租户/品牌 ID,全系统维度标识该商户。 【示例】2790683160709957(用于租户/品牌 ID,全系统维度标识该商户)。 【JSON字段】refund_transactions.json - $ - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.tenant_id IS '【说明】租户/品牌 ID,全系统维度标识该商户。 【示例】2790683160709957(用于租户/品牌 ID,全系统维度标识该商户)。 【JSON字段】refund_transactions.json - $ - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.tenantName IS '【说明】租户(商户)名称。 【示例】朗朗桌球(用于租户(商户)名称)。 【JSON字段】refund_transactions.json - $ - tenantName。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.tenantName IS '【说明】租户(商户)名称。 【示例】朗朗桌球(用于租户(商户)名称)。 【JSON字段】refund_transactions.json - $ - tenantName。';
|
||||||
@@ -1303,7 +1303,7 @@ COMMENT ON COLUMN billiards_ods.refund_transactions.is_delete IS '【说明】
|
|||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.balance_frozen_amount IS '【说明】涉及会员储值卡退款时,暂时冻结的余额金额。 【示例】0.0(用于涉及会员储值卡退款时,暂时冻结的余额金额)。 【JSON字段】refund_transactions.json - $ - balance_frozen_amount。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.balance_frozen_amount IS '【说明】涉及会员储值卡退款时,暂时冻结的余额金额。 【示例】0.0(用于涉及会员储值卡退款时,暂时冻结的余额金额)。 【JSON字段】refund_transactions.json - $ - balance_frozen_amount。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.card_frozen_amount IS '【说明】与上一个类似,偏向“某张卡的被冻结金额”,也与会员卡/储值账户相关。 【示例】0.0(用于与上一个类似,偏向“某张卡的被冻结金额”,也与会员卡/储值账户相关)。 【JSON字段】refund_transactions.json - $ - card_frozen_amount。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.card_frozen_amount IS '【说明】与上一个类似,偏向“某张卡的被冻结金额”,也与会员卡/储值账户相关。 【示例】0.0(用于与上一个类似,偏向“某张卡的被冻结金额”,也与会员卡/储值账户相关)。 【JSON字段】refund_transactions.json - $ - card_frozen_amount。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】refund_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】refund_transactions.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/refund_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/refund_transactions.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】refund_transactions.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.refund_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】refund_transactions.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.refund_transactions.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】refund_transactions.json - $ - $。';
|
||||||
|
|
||||||
@@ -1341,7 +1341,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.platform_coupon_redemption_records (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.platform_coupon_redemption_records IS 'ODS 原始明细表:平台券核销/使用记录。来源:C:/dev/LLTQ/export/test-json-doc/platform_coupon_redemption_records.json;分析:platform_coupon_redemption_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.platform_coupon_redemption_records IS 'ODS 原始明细表:平台券核销/使用记录。来源:export/test-json-doc/platform_coupon_redemption_records.json;分析:platform_coupon_redemption_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.id IS '【说明】本条平台验券记录在本系统内的主键 ID。 【示例】2957929042218501(用于本条平台验券记录在本系统内的主键 ID)。 【JSON字段】platform_coupon_redemption_records.json - $ - id。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.id IS '【说明】本条平台验券记录在本系统内的主键 ID。 【示例】2957929042218501(用于本条平台验券记录在本系统内的主键 ID)。 【JSON字段】platform_coupon_redemption_records.json - $ - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.verify_id IS '【说明】平台核销记录 ID(某些平台会为每一次核销生成一个唯一 ID)。 【示例】7570689090418149418(用于平台核销记录 ID(某些平台会为每一次核销生成一个唯一 ID))。 【JSON字段】platform_coupon_redemption_records.json - $ - verify_id。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.verify_id IS '【说明】平台核销记录 ID(某些平台会为每一次核销生成一个唯一 ID)。 【示例】7570689090418149418(用于平台核销记录 ID(某些平台会为每一次核销生成一个唯一 ID))。 【JSON字段】platform_coupon_redemption_records.json - $ - verify_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.certificate_id IS '【说明】平台侧的凭证 ID(通常由第三方团购平台生成的券实例 ID)。 【示例】5008024789379597447(用于平台侧的凭证 ID(通常由第三方团购平台生成的券实例 ID))。 【JSON字段】platform_coupon_redemption_records.json - $ - certificate_id。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.certificate_id IS '【说明】平台侧的凭证 ID(通常由第三方团购平台生成的券实例 ID)。 【示例】5008024789379597447(用于平台侧的凭证 ID(通常由第三方团购平台生成的券实例 ID))。 【JSON字段】platform_coupon_redemption_records.json - $ - certificate_id。';
|
||||||
@@ -1369,7 +1369,7 @@ COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.operator_name
|
|||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.is_delete IS '【说明】把平台验券记录挂到本门店的一条订单上。 【示例】0(用于把平台验券记录挂到本门店的一条订单上)。 【JSON字段】platform_coupon_redemption_records.json - $ - is_delete。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.is_delete IS '【说明】把平台验券记录挂到本门店的一条订单上。 【示例】0(用于把平台验券记录挂到本门店的一条订单上)。 【JSON字段】platform_coupon_redemption_records.json - $ - is_delete。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.siteProfile IS '【说明】门店信息快照。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照)。 【JSON字段】platform_coupon_redemption_records.json - $ - siteProfile。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.siteProfile IS '【说明】门店信息快照。 【示例】{"id": 2790685415443269, "org_id": 2790684179467077, "shop_name": "朗朗桌球", "avatar": "https://oss.ficoo.vip/admin/hXcE4E…(用于门店信息快照)。 【JSON字段】platform_coupon_redemption_records.json - $ - siteProfile。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】platform_coupon_redemption_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】platform_coupon_redemption_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/platform_coupon_redemption_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/platform_coupon_redemption_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】platform_coupon_redemption_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】platform_coupon_redemption_records.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.platform_coupon_redemption_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】platform_coupon_redemption_records.json - $ - $。';
|
||||||
|
|
||||||
@@ -1412,7 +1412,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.tenant_goods_master (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.tenant_goods_master IS 'ODS 原始明细表:租户商品主数据。来源:C:/dev/LLTQ/export/test-json-doc/tenant_goods_master.json;分析:tenant_goods_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.tenant_goods_master IS 'ODS 原始明细表:租户商品主数据。来源:export/test-json-doc/tenant_goods_master.json;分析:tenant_goods_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.id IS '【说明】商品档案主键 ID,唯一标识一条商品。 【示例】2791925230096261(用于商品档案主键 ID,唯一标识一条商品)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - id。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.id IS '【说明】商品档案主键 ID,唯一标识一条商品。 【示例】2791925230096261(用于商品档案主键 ID,唯一标识一条商品)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.goods_name IS '【说明】商品名称(前台展示名称)。 【示例】东方树叶(用于商品名称(前台展示名称))。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - goods_name。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.goods_name IS '【说明】商品名称(前台展示名称)。 【示例】东方树叶(用于商品名称(前台展示名称))。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - goods_name。';
|
||||||
@@ -1446,7 +1446,7 @@ COMMENT ON COLUMN billiards_ods.tenant_goods_master.create_time IS '【说明】
|
|||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.update_time IS '【说明】商品档案最近一次修改时间。 【示例】2025-10-29 23:51:38(用于商品档案最近一次修改时间)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - update_time。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.update_time IS '【说明】商品档案最近一次修改时间。 【示例】2025-10-29 23:51:38(用于商品档案最近一次修改时间)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - update_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - $。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】tenant_goods_master.json - data.tenantGoodsList - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】tenant_goods_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】tenant_goods_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/tenant_goods_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/tenant_goods_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.tenant_goods_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.tenant_goods_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】tenant_goods_master.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -1492,7 +1492,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.group_buy_packages (
|
|||||||
payload JSONB NOT NULL
|
payload JSONB NOT NULL
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.group_buy_packages IS 'ODS 原始明细表:团购套餐主数据。来源:C:/dev/LLTQ/export/test-json-doc/group_buy_packages.json;分析:group_buy_packages-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.group_buy_packages IS 'ODS 原始明细表:团购套餐主数据。来源:export/test-json-doc/group_buy_packages.json;分析:group_buy_packages-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.id IS '【说明】门店侧套餐 ID,本文件内部的主键。 【示例】2939215004469573(用于门店侧套餐 ID,本文件内部的主键)。 【JSON字段】group_buy_packages.json - data.packageCouponList - id。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.id IS '【说明】门店侧套餐 ID,本文件内部的主键。 【示例】2939215004469573(用于门店侧套餐 ID,本文件内部的主键)。 【JSON字段】group_buy_packages.json - data.packageCouponList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.package_id IS '【说明】“上层套餐 ID” 或“总部/系统级套餐 ID”。 【示例】1814707240811572(用于“上层套餐 ID” 或“总部/系统级套餐 ID”)。 【JSON字段】group_buy_packages.json - data.packageCouponList - package_id。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.package_id IS '【说明】“上层套餐 ID” 或“总部/系统级套餐 ID”。 【示例】1814707240811572(用于“上层套餐 ID” 或“总部/系统级套餐 ID”)。 【JSON字段】group_buy_packages.json - data.packageCouponList - package_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.package_name IS '【说明】团购套餐名称,用于前台展示和核销界面。 【示例】早场特惠一小时(团购套餐名称,用于前台展示和核销界面)。 【JSON字段】group_buy_packages.json - data.packageCouponList - package_name。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.package_name IS '【说明】团购套餐名称,用于前台展示和核销界面。 【示例】早场特惠一小时(团购套餐名称,用于前台展示和核销界面)。 【JSON字段】group_buy_packages.json - data.packageCouponList - package_name。';
|
||||||
@@ -1529,7 +1529,7 @@ COMMENT ON COLUMN billiards_ods.group_buy_packages.area_tag_type IS '【说明
|
|||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.creator_name IS '【说明】创建人信息,一般包含“角色:姓名”。 【示例】店长:郑丽珊(用于创建人信息,一般包含“角色:姓名”)。 【JSON字段】group_buy_packages.json - data.packageCouponList - creator_name。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.creator_name IS '【说明】创建人信息,一般包含“角色:姓名”。 【示例】店长:郑丽珊(用于创建人信息,一般包含“角色:姓名”)。 【JSON字段】group_buy_packages.json - data.packageCouponList - creator_name。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.create_time IS '【说明】该套餐在系统中创建的时间。 【示例】2025-10-27 18:24:09(用于该套餐在系统中创建的时间)。 【JSON字段】group_buy_packages.json - data.packageCouponList - create_time。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.create_time IS '【说明】该套餐在系统中创建的时间。 【示例】2025-10-27 18:24:09(用于该套餐在系统中创建的时间)。 【JSON字段】group_buy_packages.json - data.packageCouponList - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】group_buy_packages.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】group_buy_packages.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/group_buy_packages.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/group_buy_packages.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】group_buy_packages.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_packages.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】group_buy_packages.json - data.packageCouponList - $。';
|
COMMENT ON COLUMN billiards_ods.group_buy_packages.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】group_buy_packages.json - data.packageCouponList - $。';
|
||||||
|
|
||||||
@@ -1584,7 +1584,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.group_buy_redemption_records (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.group_buy_redemption_records IS 'ODS 原始明细表:团购核销记录。来源:C:/dev/LLTQ/export/test-json-doc/group_buy_redemption_records.json;分析:group_buy_redemption_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.group_buy_redemption_records IS 'ODS 原始明细表:团购核销记录。来源:export/test-json-doc/group_buy_redemption_records.json;分析:group_buy_redemption_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.id IS '【说明】本条“团购套餐流水”记录的 主键 ID。 【示例】2957924029615941(用于本条“团购套餐流水”记录的 主键 ID)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - id。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.id IS '【说明】本条“团购套餐流水”记录的 主键 ID。 【示例】2957924029615941(用于本条“团购套餐流水”记录的 主键 ID)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.site_id IS '【说明】门店 ID,与其它 JSON 中一致。 【示例】2790685415443269(用于门店 ID,与其它 JSON 中一致)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - site_id。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.site_id IS '【说明】门店 ID,与其它 JSON 中一致。 【示例】2790685415443269(用于门店 ID,与其它 JSON 中一致)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - site_id。';
|
||||||
@@ -1630,7 +1630,7 @@ COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.is_delete IS '【
|
|||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.create_time IS '【说明】本条团购套餐使用流水创建时间(即券核销时间,或与结账时间接近)。 【示例】2025-11-09 23:35:57(用于本条团购套餐使用流水创建时间(即券核销时间,或与结账时间接近))。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - create_time。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.create_time IS '【说明】本条团购套餐使用流水创建时间(即券核销时间,或与结账时间接近)。 【示例】2025-11-09 23:35:57(用于本条团购套餐使用流水创建时间(即券核销时间,或与结账时间接近))。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - $。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】group_buy_redemption_records.json - data.siteTableUseDetailsList - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】group_buy_redemption_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】group_buy_redemption_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/group_buy_redemption_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/group_buy_redemption_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.group_buy_redemption_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】group_buy_redemption_records.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -1679,7 +1679,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.settlement_ticket_details (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.settlement_ticket_details IS 'ODS 原始明细表:结算小票明细。来源:C:/dev/LLTQ/export/test-json-doc/settlement_ticket_details.json;分析:settlement_ticket_details-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.settlement_ticket_details IS 'ODS 原始明细表:结算小票明细。来源:export/test-json-doc/settlement_ticket_details.json;分析:settlement_ticket_details-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.orderSettleId IS '【说明】结算单 ID(和顶层字段相同,再次冗余)。 【示例】2957922914357125(用于结算单 ID(和顶层字段相同,再次冗余))。 【JSON字段】settlement_ticket_details.json - $ - orderSettleId。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.orderSettleId IS '【说明】结算单 ID(和顶层字段相同,再次冗余)。 【示例】2957922914357125(用于结算单 ID(和顶层字段相同,再次冗余))。 【JSON字段】settlement_ticket_details.json - $ - orderSettleId。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.actualPayment IS '【说明】本单实际支付金额总和(顾客本次实际付出:现金 + 线上 + 会员余额等)。 【示例】NULL(用于本单实际支付金额总和(顾客本次实际付出:现金 + 线上 + 会员余额等))。 【JSON字段】settlement_ticket_details.json - $ - actualPayment。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.actualPayment IS '【说明】本单实际支付金额总和(顾客本次实际付出:现金 + 线上 + 会员余额等)。 【示例】NULL(用于本单实际支付金额总和(顾客本次实际付出:现金 + 线上 + 会员余额等))。 【JSON字段】settlement_ticket_details.json - $ - actualPayment。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.adjustAmount IS '【说明】人工调价/整单调整金额(例如手工改价、折扣调整),是所有类型的手工调整合计。 【示例】NULL(用于人工调价/整单调整金额(例如手工改价、折扣调整),是所有类型的手工调整合计)。 【JSON字段】settlement_ticket_details.json - $ - adjustAmount。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.adjustAmount IS '【说明】人工调价/整单调整金额(例如手工改价、折扣调整),是所有类型的手工调整合计。 【示例】NULL(用于人工调价/整单调整金额(例如手工改价、折扣调整),是所有类型的手工调整合计)。 【JSON字段】settlement_ticket_details.json - $ - adjustAmount。';
|
||||||
@@ -1720,7 +1720,7 @@ COMMENT ON COLUMN billiards_ods.settlement_ticket_details.orderItem IS '【说
|
|||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.tenantMemberCardLogs IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_ticket_details.json - $ - tenantMemberCardLogs。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.tenantMemberCardLogs IS '【说明】来自 JSON 导出的原始字段,用于保留业务取值。 【示例】NULL(来自 JSON 导出的原始字段,用于保留业务取值)。 【JSON字段】settlement_ticket_details.json - $ - tenantMemberCardLogs。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】settlement_ticket_details.json - $ - $。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】settlement_ticket_details.json - $ - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】settlement_ticket_details.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】settlement_ticket_details.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/settlement_ticket_details.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/settlement_ticket_details.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.settlement_ticket_details.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】settlement_ticket_details.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -1776,7 +1776,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.store_goods_master (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.store_goods_master IS 'ODS 原始明细表:门店商品主数据。来源:C:/dev/LLTQ/export/test-json-doc/store_goods_master.json;分析:store_goods_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.store_goods_master IS 'ODS 原始明细表:门店商品主数据。来源:export/test-json-doc/store_goods_master.json;分析:store_goods_master-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.id IS '【说明】门店商品 ID,门店维度的商品主键。 【示例】2793025851560005(用于门店商品 ID,门店维度的商品主键)。 【JSON字段】store_goods_master.json - data.orderGoodsList - id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.id IS '【说明】门店商品 ID,门店维度的商品主键。 【示例】2793025851560005(用于门店商品 ID,门店维度的商品主键)。 【JSON字段】store_goods_master.json - data.orderGoodsList - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】store_goods_master.json - data.orderGoodsList - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】store_goods_master.json - data.orderGoodsList - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.site_id IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】store_goods_master.json - data.orderGoodsList - site_id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.site_id IS '【说明】门店 ID。 【示例】2790685415443269(用于门店 ID)。 【JSON字段】store_goods_master.json - data.orderGoodsList - site_id。';
|
||||||
@@ -1823,7 +1823,7 @@ COMMENT ON COLUMN billiards_ods.store_goods_master.create_time IS '【说明】
|
|||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.update_time IS '【说明】最后一次修改该商品档案的时间(包括价格调整、状态变更等)。 【示例】2025-11-09 07:23:47(用于最后一次修改该商品档案的时间(包括价格调整、状态变更等))。 【JSON字段】store_goods_master.json - data.orderGoodsList - update_time。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.update_time IS '【说明】最后一次修改该商品档案的时间(包括价格调整、状态变更等)。 【示例】2025-11-09 07:23:47(用于最后一次修改该商品档案的时间(包括价格调整、状态变更等))。 【JSON字段】store_goods_master.json - data.orderGoodsList - update_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】store_goods_master.json - data.orderGoodsList - $。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】store_goods_master.json - data.orderGoodsList - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】store_goods_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】store_goods_master.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/store_goods_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/store_goods_master.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_master.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】store_goods_master.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
@@ -1885,7 +1885,7 @@ CREATE TABLE IF NOT EXISTS billiards_ods.store_goods_sales_records (
|
|||||||
fetched_at TIMESTAMPTZ DEFAULT now()
|
fetched_at TIMESTAMPTZ DEFAULT now()
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_ods.store_goods_sales_records IS 'ODS 原始明细表:门店商品销售流水。来源:C:/dev/LLTQ/export/test-json-doc/store_goods_sales_records.json;分析:store_goods_sales_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
COMMENT ON TABLE billiards_ods.store_goods_sales_records IS 'ODS 原始明细表:门店商品销售流水。来源:export/test-json-doc/store_goods_sales_records.json;分析:store_goods_sales_records-Analysis.md。字段以导出原样为主;ETL 补充 source_file/source_endpoint/fetched_at,并保留 payload 为原始记录快照。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.id IS '【说明】本条「门店销售流水」记录的主键 ID。 【示例】2957924029550406(用于本条「门店销售流水」记录的主键 ID)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.id IS '【说明】本条「门店销售流水」记录的主键 ID。 【示例】2957924029550406(用于本条「门店销售流水」记录的主键 ID)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - id。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - tenant_id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.tenant_id IS '【说明】租户/品牌 ID。 【示例】2790683160709957(用于租户/品牌 ID)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - tenant_id。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.site_id IS '【说明】门店 ID(系统主键)。 【示例】2790685415443269(用于门店 ID(系统主键))。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - site_id。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.site_id IS '【说明】门店 ID(系统主键)。 【示例】2790685415443269(用于门店 ID(系统主键))。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - site_id。';
|
||||||
@@ -1939,7 +1939,7 @@ COMMENT ON COLUMN billiards_ods.store_goods_sales_records.tenant_goods_category_
|
|||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.create_time IS '【说明】销售记录创建时间,通常就是结账时间或录入时间。 【示例】2025-11-09 23:35:57(用于销售记录创建时间,通常就是结账时间或录入时间)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - create_time。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.create_time IS '【说明】销售记录创建时间,通常就是结账时间或录入时间。 【示例】2025-11-09 23:35:57(用于销售记录创建时间,通常就是结账时间或录入时间)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - create_time。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - $。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.payload IS '【说明】完整原始 JSON 记录快照,用于回溯与二次解析。 【示例】{...}(完整原始 JSON 记录快照,用于回溯与二次解析)。 【JSON字段】store_goods_sales_records.json - data.orderGoodsLedgers - $。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】store_goods_sales_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.source_file IS '【说明】ETL 元数据:原始导出文件名,用于数据追溯。 【示例】store_goods_sales_records.json(ETL 元数据:原始导出文件名,用于数据追溯)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】C:/dev/LLTQ/export/test-json-doc/store_goods_sales_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.source_endpoint IS '【说明】ETL 元数据:采集来源(接口/文件路径),用于数据追溯。 【示例】export/test-json-doc/store_goods_sales_records.json(ETL 元数据:采集来源(接口/文件路径),用于数据追溯)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
||||||
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
COMMENT ON COLUMN billiards_ods.store_goods_sales_records.fetched_at IS '【说明】ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理。 【示例】2025-11-10T00:00:00+08:00(ETL 元数据:采集/入库时间戳,用于口径对齐与增量处理)。 【JSON字段】store_goods_sales_records.json - ETL元数据 - 无。';
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1324,7 +1324,8 @@ CREATE TABLE IF NOT EXISTS dwd_store_goods_sale_Ex (
|
|||||||
push_money NUMERIC(18,2),
|
push_money NUMERIC(18,2),
|
||||||
is_single_order INTEGER,
|
is_single_order INTEGER,
|
||||||
sales_type INTEGER,
|
sales_type INTEGER,
|
||||||
operator_id BIGINT
|
operator_id BIGINT,
|
||||||
|
PRIMARY KEY (store_goods_sale_id)
|
||||||
);
|
);
|
||||||
|
|
||||||
COMMENT ON TABLE billiards_dwd.dwd_store_goods_sale_ex IS 'DWD 明细事实表(扩展字段表):dwd_store_goods_sale_ex。ODS 来源表:billiards_ods.store_goods_sales_records(对应 JSON:store_goods_sales_records.json;分析:store_goods_sales_records-Analysis.md)。装载/清洗逻辑参考:etl_billiards/tasks/dwd_load_task.py(DwdLoadTask)。';
|
COMMENT ON TABLE billiards_dwd.dwd_store_goods_sale_ex IS 'DWD 明细事实表(扩展字段表):dwd_store_goods_sale_ex。ODS 来源表:billiards_ods.store_goods_sales_records(对应 JSON:store_goods_sales_records.json;分析:store_goods_sales_records-Analysis.md)。装载/清洗逻辑参考:etl_billiards/tasks/dwd_load_task.py(DwdLoadTask)。';
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ task_codes AS (
|
|||||||
'ASSISTANTS',
|
'ASSISTANTS',
|
||||||
'COUPON_USAGE',
|
'COUPON_USAGE',
|
||||||
'CHECK_CUTOFF',
|
'CHECK_CUTOFF',
|
||||||
|
'DATA_INTEGRITY_CHECK',
|
||||||
'DWD_LOAD_FROM_ODS',
|
'DWD_LOAD_FROM_ODS',
|
||||||
'DWD_QUALITY_CHECK',
|
'DWD_QUALITY_CHECK',
|
||||||
'INIT_DWD_SCHEMA',
|
'INIT_DWD_SCHEMA',
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ python -m etl_billiards.cli.main ^
|
|||||||
--pg-dsn "%PG_DSN%" ^
|
--pg-dsn "%PG_DSN%" ^
|
||||||
--store-id %STORE_ID% ^
|
--store-id %STORE_ID% ^
|
||||||
--api-token "%API_TOKEN%"
|
--api-token "%API_TOKEN%"
|
||||||
(可选)指定落盘目录:加 --fetch-root "D:\etl\json";美化 JSON:--write-pretty-json
|
(可选)指定落盘目录:加 --fetch-root "export/JSON";美化 JSON:--write-pretty-json
|
||||||
|
|
||||||
2) DWD:ODS → DWD
|
2) DWD:ODS → DWD
|
||||||
python -m etl_billiards.cli.main ^
|
python -m etl_billiards.cli.main ^
|
||||||
|
|||||||
141
etl_billiards/gui/README.md
Normal file
141
etl_billiards/gui/README.md
Normal file
@@ -0,0 +1,141 @@
|
|||||||
|
# 飞球 ETL GUI 管理系统
|
||||||
|
|
||||||
|
一个基于 PySide6 的图形化 ETL 管理工具。
|
||||||
|
|
||||||
|
## 功能特性
|
||||||
|
|
||||||
|
- **任务配置**: 选择和配置 ETL 任务,支持参数设置和 CLI 命令预览
|
||||||
|
- **任务管理**: 任务队列管理、执行历史记录、自动执行
|
||||||
|
- **环境配置**: 图形化编辑 `.env` 配置文件
|
||||||
|
- **数据库查看**: 浏览表结构、执行 SQL 查询
|
||||||
|
- **ETL 状态**: 实时查看 ODS/DWD 数据状态和执行记录
|
||||||
|
- **日志查看**: 实时日志输出、过滤、导出
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
### 1. 安装依赖
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd etl_billiards
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. 运行 GUI
|
||||||
|
|
||||||
|
**方法一:使用启动脚本**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Windows 命令行
|
||||||
|
run_gui.bat
|
||||||
|
|
||||||
|
# 或 PowerShell
|
||||||
|
.\run_gui.ps1
|
||||||
|
```
|
||||||
|
|
||||||
|
**方法二:直接运行 Python**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd etl_billiards
|
||||||
|
python -m gui.main
|
||||||
|
```
|
||||||
|
|
||||||
|
## 打包为 EXE
|
||||||
|
|
||||||
|
### 安装打包工具
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install pyinstaller
|
||||||
|
```
|
||||||
|
|
||||||
|
### 执行打包
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 目录模式(推荐,启动更快)
|
||||||
|
python build_exe.py
|
||||||
|
|
||||||
|
# 单文件模式
|
||||||
|
python build_exe.py --onefile
|
||||||
|
|
||||||
|
# 显示控制台(调试用)
|
||||||
|
python build_exe.py --console
|
||||||
|
|
||||||
|
# 清理并重新打包
|
||||||
|
python build_exe.py --clean
|
||||||
|
```
|
||||||
|
|
||||||
|
打包完成后,EXE 文件位于 `dist/ETL管理系统/` 目录。
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
gui/
|
||||||
|
├── main.py # 应用入口
|
||||||
|
├── main_window.py # 主窗口
|
||||||
|
├── widgets/ # UI 组件
|
||||||
|
│ ├── task_panel.py # 任务配置面板
|
||||||
|
│ ├── task_manager.py # 任务管理器
|
||||||
|
│ ├── env_editor.py # 环境变量编辑器
|
||||||
|
│ ├── log_viewer.py # 日志查看器
|
||||||
|
│ ├── db_viewer.py # 数据库查看器
|
||||||
|
│ └── status_panel.py # ETL 状态面板
|
||||||
|
├── workers/ # 后台工作线程
|
||||||
|
│ ├── task_worker.py # 任务执行线程
|
||||||
|
│ └── db_worker.py # 数据库查询线程
|
||||||
|
├── models/ # 数据模型
|
||||||
|
│ └── task_model.py # 任务数据模型
|
||||||
|
├── utils/ # 工具模块
|
||||||
|
│ ├── cli_builder.py # CLI 命令构建器
|
||||||
|
│ └── config_helper.py # 配置辅助
|
||||||
|
└── resources/ # 资源文件
|
||||||
|
└── styles.qss # 样式表
|
||||||
|
```
|
||||||
|
|
||||||
|
## 使用说明
|
||||||
|
|
||||||
|
### 任务配置
|
||||||
|
|
||||||
|
1. 在左侧选择任务分类
|
||||||
|
2. 勾选要执行的任务
|
||||||
|
3. 配置运行参数(Pipeline 模式、时间窗口等)
|
||||||
|
4. 查看底部的 CLI 命令预览
|
||||||
|
5. 点击「立即执行」或「添加到队列」
|
||||||
|
|
||||||
|
### 环境配置
|
||||||
|
|
||||||
|
1. 打开「环境配置」面板
|
||||||
|
2. 编辑各项配置(数据库、API、路径等)
|
||||||
|
3. 点击「保存」
|
||||||
|
|
||||||
|
### 数据库查看
|
||||||
|
|
||||||
|
1. 打开「数据库」面板
|
||||||
|
2. 输入或使用 .env 中的 DSN
|
||||||
|
3. 点击「连接」
|
||||||
|
4. 浏览表结构或执行 SQL 查询
|
||||||
|
|
||||||
|
## 常见问题
|
||||||
|
|
||||||
|
### Q: 启动时提示缺少 PySide6
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install PySide6
|
||||||
|
```
|
||||||
|
|
||||||
|
### Q: 连接数据库失败
|
||||||
|
|
||||||
|
检查 `.env` 中的 `PG_DSN` 配置是否正确。
|
||||||
|
|
||||||
|
### Q: 打包后运行闪退
|
||||||
|
|
||||||
|
使用 `--console` 参数重新打包,查看错误信息:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python build_exe.py --console
|
||||||
|
```
|
||||||
|
|
||||||
|
## 技术栈
|
||||||
|
|
||||||
|
- Python 3.10+
|
||||||
|
- PySide6 (Qt for Python)
|
||||||
|
- psycopg2 (PostgreSQL)
|
||||||
|
- PyInstaller (打包)
|
||||||
5
etl_billiards/gui/__init__.py
Normal file
5
etl_billiards/gui/__init__.py
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""ETL GUI 客户端模块"""
|
||||||
|
|
||||||
|
__version__ = "1.0.0"
|
||||||
|
__author__ = "ETL Team"
|
||||||
46
etl_billiards/gui/main.py
Normal file
46
etl_billiards/gui/main.py
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""ETL GUI 应用入口"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# 确保项目根目录在 Python 路径中
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
if str(PROJECT_ROOT) not in sys.path:
|
||||||
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import QApplication
|
||||||
|
from PySide6.QtCore import Qt
|
||||||
|
from PySide6.QtGui import QFont
|
||||||
|
|
||||||
|
from gui.main_window import MainWindow
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""主函数"""
|
||||||
|
# 设置高 DPI 支持
|
||||||
|
QApplication.setHighDpiScaleFactorRoundingPolicy(
|
||||||
|
Qt.HighDpiScaleFactorRoundingPolicy.PassThrough
|
||||||
|
)
|
||||||
|
|
||||||
|
# 创建应用
|
||||||
|
app = QApplication(sys.argv)
|
||||||
|
app.setApplicationName("飞球 ETL 管理系统")
|
||||||
|
app.setApplicationVersion("1.0.0")
|
||||||
|
app.setOrganizationName("Billiards")
|
||||||
|
|
||||||
|
# 设置默认字体
|
||||||
|
font = QFont("Microsoft YaHei", 10)
|
||||||
|
app.setFont(font)
|
||||||
|
|
||||||
|
# 创建主窗口
|
||||||
|
window = MainWindow()
|
||||||
|
window.show()
|
||||||
|
|
||||||
|
# 运行应用
|
||||||
|
sys.exit(app.exec())
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
397
etl_billiards/gui/main_window.py
Normal file
397
etl_billiards/gui/main_window.py
Normal file
@@ -0,0 +1,397 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""主窗口"""
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
|
||||||
|
QStackedWidget, QListWidget, QListWidgetItem,
|
||||||
|
QStatusBar, QLabel, QMessageBox, QSplitter
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt, QSize, Signal
|
||||||
|
from PySide6.QtGui import QIcon, QAction
|
||||||
|
|
||||||
|
from .widgets.task_panel import TaskPanel
|
||||||
|
from .widgets.task_manager import TaskManager
|
||||||
|
from .widgets.env_editor import EnvEditor
|
||||||
|
from .widgets.log_viewer import LogViewer
|
||||||
|
from .widgets.db_viewer import DBViewer
|
||||||
|
from .widgets.status_panel import StatusPanel
|
||||||
|
from .resources import load_stylesheet
|
||||||
|
|
||||||
|
|
||||||
|
class MainWindow(QMainWindow):
|
||||||
|
"""ETL GUI 主窗口"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
status_message = Signal(str, int) # message, timeout_ms
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super().__init__()
|
||||||
|
self.setWindowTitle("飞球 ETL 管理系统")
|
||||||
|
self.setMinimumSize(1200, 800)
|
||||||
|
self.resize(1400, 900)
|
||||||
|
|
||||||
|
# 应用样式
|
||||||
|
self.setStyleSheet(load_stylesheet())
|
||||||
|
|
||||||
|
# 初始化 UI
|
||||||
|
self._init_ui()
|
||||||
|
self._init_menu()
|
||||||
|
self._init_status_bar()
|
||||||
|
self._connect_signals()
|
||||||
|
|
||||||
|
# 默认选中第一项
|
||||||
|
self.nav_list.setCurrentRow(0)
|
||||||
|
|
||||||
|
# 首次显示标记
|
||||||
|
self._first_show = True
|
||||||
|
|
||||||
|
def showEvent(self, event):
|
||||||
|
"""窗口显示事件"""
|
||||||
|
super().showEvent(event)
|
||||||
|
if self._first_show:
|
||||||
|
self._first_show = False
|
||||||
|
# 延迟检查配置,让窗口先显示
|
||||||
|
from PySide6.QtCore import QTimer
|
||||||
|
QTimer.singleShot(100, self._check_config_on_startup)
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
"""初始化界面"""
|
||||||
|
# 中央部件
|
||||||
|
central_widget = QWidget()
|
||||||
|
self.setCentralWidget(central_widget)
|
||||||
|
|
||||||
|
# 主布局
|
||||||
|
main_layout = QHBoxLayout(central_widget)
|
||||||
|
main_layout.setContentsMargins(0, 0, 0, 0)
|
||||||
|
main_layout.setSpacing(0)
|
||||||
|
|
||||||
|
# 创建分割器
|
||||||
|
splitter = QSplitter(Qt.Horizontal)
|
||||||
|
main_layout.addWidget(splitter)
|
||||||
|
|
||||||
|
# 左侧导航栏
|
||||||
|
nav_widget = self._create_nav_widget()
|
||||||
|
splitter.addWidget(nav_widget)
|
||||||
|
|
||||||
|
# 右侧内容区
|
||||||
|
self.content_stack = QStackedWidget()
|
||||||
|
splitter.addWidget(self.content_stack)
|
||||||
|
|
||||||
|
# 设置分割比例
|
||||||
|
splitter.setSizes([200, 1200])
|
||||||
|
splitter.setStretchFactor(0, 0)
|
||||||
|
splitter.setStretchFactor(1, 1)
|
||||||
|
|
||||||
|
# 创建各个面板
|
||||||
|
self._create_panels()
|
||||||
|
|
||||||
|
def _create_nav_widget(self) -> QWidget:
|
||||||
|
"""创建导航侧边栏"""
|
||||||
|
nav_widget = QWidget()
|
||||||
|
nav_widget.setMaximumWidth(220)
|
||||||
|
nav_widget.setMinimumWidth(180)
|
||||||
|
|
||||||
|
layout = QVBoxLayout(nav_widget)
|
||||||
|
layout.setContentsMargins(0, 0, 0, 0)
|
||||||
|
layout.setSpacing(0)
|
||||||
|
|
||||||
|
# 标题
|
||||||
|
title_label = QLabel(" ETL 控制台")
|
||||||
|
title_label.setProperty("heading", True)
|
||||||
|
title_label.setFixedHeight(60)
|
||||||
|
title_label.setAlignment(Qt.AlignVCenter)
|
||||||
|
layout.addWidget(title_label)
|
||||||
|
|
||||||
|
# 导航列表
|
||||||
|
self.nav_list = QListWidget()
|
||||||
|
self.nav_list.setObjectName("navList")
|
||||||
|
layout.addWidget(self.nav_list)
|
||||||
|
|
||||||
|
# 添加导航项
|
||||||
|
nav_items = [
|
||||||
|
("任务配置", "配置并执行 ETL 任务"),
|
||||||
|
("任务管理", "管理任务队列和历史记录"),
|
||||||
|
("环境配置", "编辑 .env 配置文件"),
|
||||||
|
("数据库", "查看数据库和执行查询"),
|
||||||
|
("ETL 状态", "查看 ETL 运行状态"),
|
||||||
|
("日志", "查看执行日志"),
|
||||||
|
]
|
||||||
|
|
||||||
|
for name, tooltip in nav_items:
|
||||||
|
item = QListWidgetItem(name)
|
||||||
|
item.setToolTip(tooltip)
|
||||||
|
item.setSizeHint(QSize(0, 44))
|
||||||
|
self.nav_list.addItem(item)
|
||||||
|
|
||||||
|
return nav_widget
|
||||||
|
|
||||||
|
def _create_panels(self):
|
||||||
|
"""创建各个功能面板"""
|
||||||
|
# 任务配置面板
|
||||||
|
self.task_panel = TaskPanel()
|
||||||
|
self.content_stack.addWidget(self.task_panel)
|
||||||
|
|
||||||
|
# 任务管理面板
|
||||||
|
self.task_manager = TaskManager()
|
||||||
|
self.content_stack.addWidget(self.task_manager)
|
||||||
|
|
||||||
|
# 环境配置面板
|
||||||
|
self.env_editor = EnvEditor()
|
||||||
|
self.content_stack.addWidget(self.env_editor)
|
||||||
|
|
||||||
|
# 数据库查看器
|
||||||
|
self.db_viewer = DBViewer()
|
||||||
|
self.content_stack.addWidget(self.db_viewer)
|
||||||
|
|
||||||
|
# ETL 状态面板
|
||||||
|
self.status_panel = StatusPanel()
|
||||||
|
self.content_stack.addWidget(self.status_panel)
|
||||||
|
|
||||||
|
# 日志面板
|
||||||
|
self.log_viewer = LogViewer()
|
||||||
|
self.content_stack.addWidget(self.log_viewer)
|
||||||
|
|
||||||
|
def _init_menu(self):
|
||||||
|
"""初始化菜单栏"""
|
||||||
|
menubar = self.menuBar()
|
||||||
|
|
||||||
|
# 文件菜单
|
||||||
|
file_menu = menubar.addMenu("文件(&F)")
|
||||||
|
|
||||||
|
refresh_action = QAction("刷新配置(&R)", self)
|
||||||
|
refresh_action.setShortcut("Ctrl+R")
|
||||||
|
refresh_action.triggered.connect(self._refresh_config)
|
||||||
|
file_menu.addAction(refresh_action)
|
||||||
|
|
||||||
|
settings_action = QAction("设置(&S)...", self)
|
||||||
|
settings_action.setShortcut("Ctrl+,")
|
||||||
|
settings_action.triggered.connect(self._show_settings)
|
||||||
|
file_menu.addAction(settings_action)
|
||||||
|
|
||||||
|
file_menu.addSeparator()
|
||||||
|
|
||||||
|
exit_action = QAction("退出(&X)", self)
|
||||||
|
exit_action.setShortcut("Ctrl+Q")
|
||||||
|
exit_action.triggered.connect(self.close)
|
||||||
|
file_menu.addAction(exit_action)
|
||||||
|
|
||||||
|
# 视图菜单
|
||||||
|
view_menu = menubar.addMenu("视图(&V)")
|
||||||
|
|
||||||
|
task_config_action = QAction("任务配置(&T)", self)
|
||||||
|
task_config_action.setShortcut("Ctrl+1")
|
||||||
|
task_config_action.triggered.connect(lambda: self._switch_panel(0))
|
||||||
|
view_menu.addAction(task_config_action)
|
||||||
|
|
||||||
|
task_manager_action = QAction("任务管理(&M)", self)
|
||||||
|
task_manager_action.setShortcut("Ctrl+2")
|
||||||
|
task_manager_action.triggered.connect(lambda: self._switch_panel(1))
|
||||||
|
view_menu.addAction(task_manager_action)
|
||||||
|
|
||||||
|
env_action = QAction("环境配置(&E)", self)
|
||||||
|
env_action.setShortcut("Ctrl+3")
|
||||||
|
env_action.triggered.connect(lambda: self._switch_panel(2))
|
||||||
|
view_menu.addAction(env_action)
|
||||||
|
|
||||||
|
db_action = QAction("数据库(&D)", self)
|
||||||
|
db_action.setShortcut("Ctrl+4")
|
||||||
|
db_action.triggered.connect(lambda: self._switch_panel(3))
|
||||||
|
view_menu.addAction(db_action)
|
||||||
|
|
||||||
|
status_action = QAction("ETL 状态(&S)", self)
|
||||||
|
status_action.setShortcut("Ctrl+5")
|
||||||
|
status_action.triggered.connect(lambda: self._switch_panel(4))
|
||||||
|
view_menu.addAction(status_action)
|
||||||
|
|
||||||
|
log_action = QAction("日志(&L)", self)
|
||||||
|
log_action.setShortcut("Ctrl+6")
|
||||||
|
log_action.triggered.connect(lambda: self._switch_panel(5))
|
||||||
|
view_menu.addAction(log_action)
|
||||||
|
|
||||||
|
# 帮助菜单
|
||||||
|
help_menu = menubar.addMenu("帮助(&H)")
|
||||||
|
|
||||||
|
about_action = QAction("关于(&A)", self)
|
||||||
|
about_action.triggered.connect(self._show_about)
|
||||||
|
help_menu.addAction(about_action)
|
||||||
|
|
||||||
|
def _init_status_bar(self):
|
||||||
|
"""初始化状态栏"""
|
||||||
|
self.status_bar = QStatusBar()
|
||||||
|
self.setStatusBar(self.status_bar)
|
||||||
|
|
||||||
|
# 连接状态
|
||||||
|
self.conn_status_label = QLabel("数据库: 未连接")
|
||||||
|
self.conn_status_label.setProperty("status", "warning")
|
||||||
|
self.status_bar.addPermanentWidget(self.conn_status_label)
|
||||||
|
|
||||||
|
# 任务状态
|
||||||
|
self.task_status_label = QLabel("任务: 空闲")
|
||||||
|
self.status_bar.addPermanentWidget(self.task_status_label)
|
||||||
|
|
||||||
|
# 默认消息
|
||||||
|
self.status_bar.showMessage("就绪", 3000)
|
||||||
|
|
||||||
|
def _connect_signals(self):
|
||||||
|
"""连接信号"""
|
||||||
|
# 导航切换
|
||||||
|
self.nav_list.currentRowChanged.connect(self._on_nav_changed)
|
||||||
|
|
||||||
|
# 任务面板信号
|
||||||
|
self.task_panel.task_started.connect(self._on_task_started)
|
||||||
|
self.task_panel.task_finished.connect(self._on_task_finished)
|
||||||
|
self.task_panel.log_message.connect(self.log_viewer.append_log)
|
||||||
|
self.task_panel.add_to_queue.connect(self._on_add_to_queue)
|
||||||
|
self.task_panel.create_schedule.connect(self._on_create_schedule)
|
||||||
|
|
||||||
|
# 任务管理器信号
|
||||||
|
self.task_manager.task_started.connect(self._on_task_started)
|
||||||
|
self.task_manager.task_finished.connect(self._on_task_finished)
|
||||||
|
self.task_manager.log_message.connect(self.log_viewer.append_log)
|
||||||
|
|
||||||
|
# 数据库连接状态
|
||||||
|
self.db_viewer.connection_changed.connect(self._on_db_connection_changed)
|
||||||
|
|
||||||
|
# 状态消息
|
||||||
|
self.status_message.connect(self._show_status_message)
|
||||||
|
|
||||||
|
def _on_nav_changed(self, index: int):
|
||||||
|
"""导航项切换"""
|
||||||
|
self.content_stack.setCurrentIndex(index)
|
||||||
|
|
||||||
|
def _switch_panel(self, index: int):
|
||||||
|
"""切换到指定面板"""
|
||||||
|
self.nav_list.setCurrentRow(index)
|
||||||
|
|
||||||
|
def _refresh_config(self):
|
||||||
|
"""刷新配置"""
|
||||||
|
self.env_editor.load_config()
|
||||||
|
self.task_panel.refresh_tasks()
|
||||||
|
self.status_bar.showMessage("配置已刷新", 3000)
|
||||||
|
|
||||||
|
def _on_task_started(self, task_info: str):
|
||||||
|
"""任务开始时"""
|
||||||
|
self.task_status_label.setText(f"任务: 执行中 - {task_info}")
|
||||||
|
self.task_status_label.setProperty("status", "info")
|
||||||
|
self.task_status_label.style().unpolish(self.task_status_label)
|
||||||
|
self.task_status_label.style().polish(self.task_status_label)
|
||||||
|
|
||||||
|
def _on_task_finished(self, success: bool, message: str):
|
||||||
|
"""任务完成时"""
|
||||||
|
if success:
|
||||||
|
self.task_status_label.setText("任务: 完成")
|
||||||
|
self.task_status_label.setProperty("status", "success")
|
||||||
|
else:
|
||||||
|
self.task_status_label.setText("任务: 失败")
|
||||||
|
self.task_status_label.setProperty("status", "error")
|
||||||
|
self.task_status_label.style().unpolish(self.task_status_label)
|
||||||
|
self.task_status_label.style().polish(self.task_status_label)
|
||||||
|
self.status_bar.showMessage(message, 5000)
|
||||||
|
|
||||||
|
def _on_db_connection_changed(self, connected: bool, message: str):
|
||||||
|
"""数据库连接状态变化"""
|
||||||
|
if connected:
|
||||||
|
self.conn_status_label.setText("数据库: 已连接")
|
||||||
|
self.conn_status_label.setProperty("status", "success")
|
||||||
|
else:
|
||||||
|
self.conn_status_label.setText("数据库: 未连接")
|
||||||
|
self.conn_status_label.setProperty("status", "warning")
|
||||||
|
self.conn_status_label.style().unpolish(self.conn_status_label)
|
||||||
|
self.conn_status_label.style().polish(self.conn_status_label)
|
||||||
|
if message:
|
||||||
|
self.status_bar.showMessage(message, 3000)
|
||||||
|
|
||||||
|
def _show_status_message(self, message: str, timeout: int):
|
||||||
|
"""显示状态栏消息"""
|
||||||
|
self.status_bar.showMessage(message, timeout)
|
||||||
|
|
||||||
|
def _on_add_to_queue(self, config):
|
||||||
|
"""添加任务到队列"""
|
||||||
|
task_id = self.task_manager.add_task(config)
|
||||||
|
self.status_bar.showMessage(f"任务已添加到队列 (ID: {task_id})", 3000)
|
||||||
|
|
||||||
|
def _on_create_schedule(self, name: str, task_codes: list, task_config: dict):
|
||||||
|
"""创建调度任务"""
|
||||||
|
# 打开调度编辑对话框
|
||||||
|
from .widgets.task_manager import ScheduleEditDialog
|
||||||
|
from .models.schedule_model import ScheduledTask, ScheduleConfig
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
# 创建一个预填充的调度任务
|
||||||
|
task = ScheduledTask(
|
||||||
|
id=str(uuid.uuid4())[:8],
|
||||||
|
name=name,
|
||||||
|
task_codes=task_codes,
|
||||||
|
schedule=ScheduleConfig(),
|
||||||
|
task_config=task_config,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 打开编辑对话框
|
||||||
|
dialog = ScheduleEditDialog(task=task, parent=self)
|
||||||
|
if dialog.exec():
|
||||||
|
updated_task = dialog.get_task()
|
||||||
|
if updated_task:
|
||||||
|
self.task_manager.schedule_store.add_task(updated_task)
|
||||||
|
self.task_manager._refresh_schedule_table()
|
||||||
|
self.status_bar.showMessage(f"调度任务已创建: {updated_task.name}", 3000)
|
||||||
|
# 切换到任务管理面板的调度选项卡
|
||||||
|
self._switch_panel(1)
|
||||||
|
|
||||||
|
def _show_settings(self):
|
||||||
|
"""显示设置对话框"""
|
||||||
|
from .widgets.settings_dialog import SettingsDialog
|
||||||
|
dialog = SettingsDialog(self)
|
||||||
|
if dialog.exec():
|
||||||
|
# 重新加载配置
|
||||||
|
self._refresh_config()
|
||||||
|
self.status_bar.showMessage("设置已保存", 3000)
|
||||||
|
|
||||||
|
def _check_config_on_startup(self):
|
||||||
|
"""启动时检查配置"""
|
||||||
|
from .utils.app_settings import app_settings
|
||||||
|
if not app_settings.is_configured():
|
||||||
|
QMessageBox.information(
|
||||||
|
self,
|
||||||
|
"首次配置",
|
||||||
|
"欢迎使用 ETL 管理系统!\n\n"
|
||||||
|
"请先配置 ETL 项目路径,否则无法执行任务。\n\n"
|
||||||
|
"点击 文件 → 设置 进行配置。"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _show_about(self):
|
||||||
|
"""显示关于对话框"""
|
||||||
|
QMessageBox.about(
|
||||||
|
self,
|
||||||
|
"关于 飞球 ETL 管理系统",
|
||||||
|
"<h3>飞球 ETL 管理系统</h3>"
|
||||||
|
"<p>版本: 1.0.0</p>"
|
||||||
|
"<p>一个用于管理台球场门店数据 ETL 的图形化工具。</p>"
|
||||||
|
"<p>功能包括:</p>"
|
||||||
|
"<ul>"
|
||||||
|
"<li>任务配置与执行</li>"
|
||||||
|
"<li>环境变量管理</li>"
|
||||||
|
"<li>数据库查询</li>"
|
||||||
|
"<li>ETL 状态监控</li>"
|
||||||
|
"</ul>"
|
||||||
|
)
|
||||||
|
|
||||||
|
def closeEvent(self, event):
|
||||||
|
"""关闭事件"""
|
||||||
|
# 检查是否有正在运行的任务
|
||||||
|
if hasattr(self, 'task_panel') and self.task_panel.is_running():
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"确认退出",
|
||||||
|
"当前有任务正在执行,确定要退出吗?",
|
||||||
|
QMessageBox.Yes | QMessageBox.No,
|
||||||
|
QMessageBox.No
|
||||||
|
)
|
||||||
|
if reply == QMessageBox.No:
|
||||||
|
event.ignore()
|
||||||
|
return
|
||||||
|
|
||||||
|
# 关闭数据库连接
|
||||||
|
if hasattr(self, 'db_viewer'):
|
||||||
|
self.db_viewer.close_connection()
|
||||||
|
|
||||||
|
event.accept()
|
||||||
20
etl_billiards/gui/models/__init__.py
Normal file
20
etl_billiards/gui/models/__init__.py
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""数据模型模块"""
|
||||||
|
|
||||||
|
from .task_model import TaskItem, TaskStatus, TaskHistory, TaskConfig, QueuedTask
|
||||||
|
from .schedule_model import (
|
||||||
|
ScheduledTask, ScheduleConfig, ScheduleType, IntervalUnit, ScheduleStore
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"TaskItem",
|
||||||
|
"TaskStatus",
|
||||||
|
"TaskHistory",
|
||||||
|
"TaskConfig",
|
||||||
|
"QueuedTask",
|
||||||
|
"ScheduledTask",
|
||||||
|
"ScheduleConfig",
|
||||||
|
"ScheduleType",
|
||||||
|
"IntervalUnit",
|
||||||
|
"ScheduleStore",
|
||||||
|
]
|
||||||
391
etl_billiards/gui/models/schedule_model.py
Normal file
391
etl_billiards/gui/models/schedule_model.py
Normal file
@@ -0,0 +1,391 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""调度任务数据模型"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from dataclasses import dataclass, field, asdict
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Optional, List, Dict, Any
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
class ScheduleType(Enum):
|
||||||
|
"""调度类型"""
|
||||||
|
ONCE = "once" # 一次性
|
||||||
|
INTERVAL = "interval" # 固定间隔
|
||||||
|
DAILY = "daily" # 每天
|
||||||
|
WEEKLY = "weekly" # 每周
|
||||||
|
CRON = "cron" # Cron 表达式
|
||||||
|
|
||||||
|
|
||||||
|
class IntervalUnit(Enum):
|
||||||
|
"""间隔单位"""
|
||||||
|
MINUTES = "minutes"
|
||||||
|
HOURS = "hours"
|
||||||
|
DAYS = "days"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScheduleConfig:
|
||||||
|
"""调度配置"""
|
||||||
|
schedule_type: ScheduleType = ScheduleType.ONCE
|
||||||
|
|
||||||
|
# 间隔调度
|
||||||
|
interval_value: int = 1
|
||||||
|
interval_unit: IntervalUnit = IntervalUnit.HOURS
|
||||||
|
|
||||||
|
# 每日调度
|
||||||
|
daily_time: str = "04:00" # HH:MM
|
||||||
|
|
||||||
|
# 每周调度
|
||||||
|
weekly_days: List[int] = field(default_factory=lambda: [1]) # 1-7, 1=周一
|
||||||
|
weekly_time: str = "04:00"
|
||||||
|
|
||||||
|
# Cron 表达式
|
||||||
|
cron_expression: str = "0 4 * * *"
|
||||||
|
|
||||||
|
# 通用设置
|
||||||
|
enabled: bool = True
|
||||||
|
start_date: Optional[str] = None # YYYY-MM-DD
|
||||||
|
end_date: Optional[str] = None # YYYY-MM-DD
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
"""转换为字典"""
|
||||||
|
return {
|
||||||
|
"schedule_type": self.schedule_type.value,
|
||||||
|
"interval_value": self.interval_value,
|
||||||
|
"interval_unit": self.interval_unit.value,
|
||||||
|
"daily_time": self.daily_time,
|
||||||
|
"weekly_days": self.weekly_days,
|
||||||
|
"weekly_time": self.weekly_time,
|
||||||
|
"cron_expression": self.cron_expression,
|
||||||
|
"enabled": self.enabled,
|
||||||
|
"start_date": self.start_date,
|
||||||
|
"end_date": self.end_date,
|
||||||
|
}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, data: dict) -> "ScheduleConfig":
|
||||||
|
"""从字典创建"""
|
||||||
|
return cls(
|
||||||
|
schedule_type=ScheduleType(data.get("schedule_type", "once")),
|
||||||
|
interval_value=data.get("interval_value", 1),
|
||||||
|
interval_unit=IntervalUnit(data.get("interval_unit", "hours")),
|
||||||
|
daily_time=data.get("daily_time", "04:00"),
|
||||||
|
weekly_days=data.get("weekly_days", [1]),
|
||||||
|
weekly_time=data.get("weekly_time", "04:00"),
|
||||||
|
cron_expression=data.get("cron_expression", "0 4 * * *"),
|
||||||
|
enabled=data.get("enabled", True),
|
||||||
|
start_date=data.get("start_date"),
|
||||||
|
end_date=data.get("end_date"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_description(self) -> str:
|
||||||
|
"""获取调度描述"""
|
||||||
|
if self.schedule_type == ScheduleType.ONCE:
|
||||||
|
return "一次性执行"
|
||||||
|
elif self.schedule_type == ScheduleType.INTERVAL:
|
||||||
|
unit_names = {"minutes": "分钟", "hours": "小时", "days": "天"}
|
||||||
|
return f"每 {self.interval_value} {unit_names[self.interval_unit.value]}"
|
||||||
|
elif self.schedule_type == ScheduleType.DAILY:
|
||||||
|
return f"每天 {self.daily_time}"
|
||||||
|
elif self.schedule_type == ScheduleType.WEEKLY:
|
||||||
|
day_names = {1: "一", 2: "二", 3: "三", 4: "四", 5: "五", 6: "六", 7: "日"}
|
||||||
|
days = "、".join(f"周{day_names[d]}" for d in sorted(self.weekly_days))
|
||||||
|
return f"每周 {days} {self.weekly_time}"
|
||||||
|
elif self.schedule_type == ScheduleType.CRON:
|
||||||
|
return f"Cron: {self.cron_expression}"
|
||||||
|
return "未知"
|
||||||
|
|
||||||
|
# 首次执行延迟秒数
|
||||||
|
FIRST_RUN_DELAY_SECONDS = 60
|
||||||
|
|
||||||
|
def get_next_run_time(self, last_run: Optional[datetime] = None) -> Optional[datetime]:
|
||||||
|
"""计算下次运行时间
|
||||||
|
|
||||||
|
注意:首次执行(last_run 为 None)时会延迟 60 秒,避免创建后立即执行
|
||||||
|
"""
|
||||||
|
now = datetime.now()
|
||||||
|
|
||||||
|
# 检查日期范围
|
||||||
|
if self.start_date:
|
||||||
|
start = datetime.strptime(self.start_date, "%Y-%m-%d")
|
||||||
|
if now < start:
|
||||||
|
now = start
|
||||||
|
|
||||||
|
if self.end_date:
|
||||||
|
end = datetime.strptime(self.end_date, "%Y-%m-%d") + timedelta(days=1)
|
||||||
|
if now >= end:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# 首次执行延迟 60 秒
|
||||||
|
first_run_time = now + timedelta(seconds=self.FIRST_RUN_DELAY_SECONDS)
|
||||||
|
|
||||||
|
if self.schedule_type == ScheduleType.ONCE:
|
||||||
|
return None if last_run else first_run_time
|
||||||
|
|
||||||
|
elif self.schedule_type == ScheduleType.INTERVAL:
|
||||||
|
if not last_run:
|
||||||
|
return first_run_time
|
||||||
|
if self.interval_unit == IntervalUnit.MINUTES:
|
||||||
|
delta = timedelta(minutes=self.interval_value)
|
||||||
|
elif self.interval_unit == IntervalUnit.HOURS:
|
||||||
|
delta = timedelta(hours=self.interval_value)
|
||||||
|
else:
|
||||||
|
delta = timedelta(days=self.interval_value)
|
||||||
|
return last_run + delta
|
||||||
|
|
||||||
|
elif self.schedule_type == ScheduleType.DAILY:
|
||||||
|
hour, minute = map(int, self.daily_time.split(":"))
|
||||||
|
next_run = now.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||||
|
if next_run <= now:
|
||||||
|
next_run += timedelta(days=1)
|
||||||
|
return next_run
|
||||||
|
|
||||||
|
elif self.schedule_type == ScheduleType.WEEKLY:
|
||||||
|
hour, minute = map(int, self.weekly_time.split(":"))
|
||||||
|
# 找到下一个匹配的日期
|
||||||
|
for i in range(8):
|
||||||
|
check_date = now + timedelta(days=i)
|
||||||
|
weekday = check_date.isoweekday() # 1-7
|
||||||
|
if weekday in self.weekly_days:
|
||||||
|
next_run = check_date.replace(hour=hour, minute=minute, second=0, microsecond=0)
|
||||||
|
if next_run > now:
|
||||||
|
return next_run
|
||||||
|
return None
|
||||||
|
|
||||||
|
elif self.schedule_type == ScheduleType.CRON:
|
||||||
|
# 简化版 Cron 解析(只支持基本格式)
|
||||||
|
try:
|
||||||
|
return self._parse_simple_cron(now)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
def _parse_simple_cron(self, now: datetime) -> Optional[datetime]:
|
||||||
|
"""简化版 Cron 解析"""
|
||||||
|
parts = self.cron_expression.split()
|
||||||
|
if len(parts) != 5:
|
||||||
|
return None
|
||||||
|
|
||||||
|
minute, hour, day, month, weekday = parts
|
||||||
|
|
||||||
|
# 只处理简单情况
|
||||||
|
if minute.isdigit() and hour.isdigit():
|
||||||
|
next_run = now.replace(
|
||||||
|
hour=int(hour),
|
||||||
|
minute=int(minute),
|
||||||
|
second=0,
|
||||||
|
microsecond=0
|
||||||
|
)
|
||||||
|
if next_run <= now:
|
||||||
|
next_run += timedelta(days=1)
|
||||||
|
return next_run
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScheduleExecutionRecord:
|
||||||
|
"""调度执行记录"""
|
||||||
|
task_id: str # 关联的 QueuedTask ID
|
||||||
|
executed_at: datetime # 执行时间
|
||||||
|
status: str = "" # 状态:success, failed, pending
|
||||||
|
exit_code: Optional[int] = None # 退出码
|
||||||
|
duration_seconds: float = 0.0 # 耗时(秒)
|
||||||
|
summary: str = "" # 执行摘要
|
||||||
|
output: str = "" # 完整执行日志
|
||||||
|
error: str = "" # 错误信息
|
||||||
|
|
||||||
|
# 日志最大长度限制(字符数)
|
||||||
|
MAX_OUTPUT_LENGTH: int = 100000 # 100KB
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
return {
|
||||||
|
"task_id": self.task_id,
|
||||||
|
"executed_at": self.executed_at.isoformat(),
|
||||||
|
"status": self.status,
|
||||||
|
"exit_code": self.exit_code,
|
||||||
|
"duration_seconds": self.duration_seconds,
|
||||||
|
"summary": self.summary,
|
||||||
|
"output": self.output[:self.MAX_OUTPUT_LENGTH] if self.output else "",
|
||||||
|
"error": self.error[:5000] if self.error else "",
|
||||||
|
}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, data: dict) -> "ScheduleExecutionRecord":
|
||||||
|
return cls(
|
||||||
|
task_id=data.get("task_id", ""),
|
||||||
|
executed_at=datetime.fromisoformat(data["executed_at"]) if data.get("executed_at") else datetime.now(),
|
||||||
|
status=data.get("status", ""),
|
||||||
|
exit_code=data.get("exit_code"),
|
||||||
|
duration_seconds=data.get("duration_seconds", 0.0),
|
||||||
|
summary=data.get("summary", ""),
|
||||||
|
output=data.get("output", ""),
|
||||||
|
error=data.get("error", ""),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScheduledTask:
|
||||||
|
"""调度任务"""
|
||||||
|
id: str
|
||||||
|
name: str
|
||||||
|
task_codes: List[str]
|
||||||
|
schedule: ScheduleConfig
|
||||||
|
task_config: Dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
# 运行状态
|
||||||
|
enabled: bool = True
|
||||||
|
last_run: Optional[datetime] = None
|
||||||
|
next_run: Optional[datetime] = None
|
||||||
|
run_count: int = 0
|
||||||
|
last_status: str = ""
|
||||||
|
|
||||||
|
# 执行历史(最近 N 次执行记录)
|
||||||
|
execution_history: List[ScheduleExecutionRecord] = field(default_factory=list)
|
||||||
|
MAX_HISTORY_SIZE: int = field(default=50, repr=False) # 保留最近50次执行记录
|
||||||
|
|
||||||
|
created_at: datetime = field(default_factory=datetime.now)
|
||||||
|
updated_at: datetime = field(default_factory=datetime.now)
|
||||||
|
|
||||||
|
def add_execution_record(self, record: ScheduleExecutionRecord):
|
||||||
|
"""添加执行记录"""
|
||||||
|
self.execution_history.insert(0, record)
|
||||||
|
# 限制历史记录数量
|
||||||
|
if len(self.execution_history) > self.MAX_HISTORY_SIZE:
|
||||||
|
self.execution_history = self.execution_history[:self.MAX_HISTORY_SIZE]
|
||||||
|
|
||||||
|
def update_execution_record(self, task_id: str, status: str, exit_code: int, duration: float,
|
||||||
|
summary: str, output: str = "", error: str = ""):
|
||||||
|
"""更新执行记录状态"""
|
||||||
|
for record in self.execution_history:
|
||||||
|
if record.task_id == task_id:
|
||||||
|
record.status = status
|
||||||
|
record.exit_code = exit_code
|
||||||
|
record.duration_seconds = duration
|
||||||
|
record.summary = summary
|
||||||
|
record.output = output
|
||||||
|
record.error = error
|
||||||
|
break
|
||||||
|
|
||||||
|
def to_dict(self) -> dict:
|
||||||
|
"""转换为字典"""
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"name": self.name,
|
||||||
|
"task_codes": self.task_codes,
|
||||||
|
"schedule": self.schedule.to_dict(),
|
||||||
|
"task_config": self.task_config,
|
||||||
|
"enabled": self.enabled,
|
||||||
|
"last_run": self.last_run.isoformat() if self.last_run else None,
|
||||||
|
"next_run": self.next_run.isoformat() if self.next_run else None,
|
||||||
|
"run_count": self.run_count,
|
||||||
|
"last_status": self.last_status,
|
||||||
|
"execution_history": [r.to_dict() for r in self.execution_history],
|
||||||
|
"created_at": self.created_at.isoformat(),
|
||||||
|
"updated_at": self.updated_at.isoformat(),
|
||||||
|
}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, data: dict) -> "ScheduledTask":
|
||||||
|
"""从字典创建"""
|
||||||
|
history_data = data.get("execution_history", [])
|
||||||
|
execution_history = [ScheduleExecutionRecord.from_dict(r) for r in history_data]
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
id=data["id"],
|
||||||
|
name=data["name"],
|
||||||
|
task_codes=data["task_codes"],
|
||||||
|
schedule=ScheduleConfig.from_dict(data.get("schedule", {})),
|
||||||
|
task_config=data.get("task_config", {}),
|
||||||
|
enabled=data.get("enabled", True),
|
||||||
|
last_run=datetime.fromisoformat(data["last_run"]) if data.get("last_run") else None,
|
||||||
|
next_run=datetime.fromisoformat(data["next_run"]) if data.get("next_run") else None,
|
||||||
|
run_count=data.get("run_count", 0),
|
||||||
|
last_status=data.get("last_status", ""),
|
||||||
|
execution_history=execution_history,
|
||||||
|
created_at=datetime.fromisoformat(data["created_at"]) if data.get("created_at") else datetime.now(),
|
||||||
|
updated_at=datetime.fromisoformat(data["updated_at"]) if data.get("updated_at") else datetime.now(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def update_next_run(self):
|
||||||
|
"""更新下次运行时间"""
|
||||||
|
if self.enabled and self.schedule.enabled:
|
||||||
|
self.next_run = self.schedule.get_next_run_time(self.last_run)
|
||||||
|
else:
|
||||||
|
self.next_run = None
|
||||||
|
self.updated_at = datetime.now()
|
||||||
|
|
||||||
|
|
||||||
|
class ScheduleStore:
|
||||||
|
"""调度任务存储"""
|
||||||
|
|
||||||
|
def __init__(self, storage_path: Optional[Path] = None):
|
||||||
|
if storage_path is None:
|
||||||
|
storage_path = Path(__file__).resolve().parents[2] / "scheduled_tasks.json"
|
||||||
|
self.storage_path = storage_path
|
||||||
|
self.tasks: Dict[str, ScheduledTask] = {}
|
||||||
|
self.load()
|
||||||
|
|
||||||
|
def load(self):
|
||||||
|
"""加载任务"""
|
||||||
|
if self.storage_path.exists():
|
||||||
|
try:
|
||||||
|
data = json.loads(self.storage_path.read_text(encoding="utf-8"))
|
||||||
|
self.tasks = {
|
||||||
|
task_id: ScheduledTask.from_dict(task_data)
|
||||||
|
for task_id, task_data in data.get("tasks", {}).items()
|
||||||
|
}
|
||||||
|
except Exception:
|
||||||
|
self.tasks = {}
|
||||||
|
|
||||||
|
def save(self):
|
||||||
|
"""保存任务"""
|
||||||
|
data = {
|
||||||
|
"tasks": {
|
||||||
|
task_id: task.to_dict()
|
||||||
|
for task_id, task in self.tasks.items()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.storage_path.write_text(
|
||||||
|
json.dumps(data, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
def add_task(self, task: ScheduledTask):
|
||||||
|
"""添加任务"""
|
||||||
|
task.update_next_run()
|
||||||
|
self.tasks[task.id] = task
|
||||||
|
self.save()
|
||||||
|
|
||||||
|
def remove_task(self, task_id: str):
|
||||||
|
"""移除任务"""
|
||||||
|
if task_id in self.tasks:
|
||||||
|
del self.tasks[task_id]
|
||||||
|
self.save()
|
||||||
|
|
||||||
|
def update_task(self, task: ScheduledTask):
|
||||||
|
"""更新任务"""
|
||||||
|
task.update_next_run()
|
||||||
|
task.updated_at = datetime.now()
|
||||||
|
self.tasks[task.id] = task
|
||||||
|
self.save()
|
||||||
|
|
||||||
|
def get_task(self, task_id: str) -> Optional[ScheduledTask]:
|
||||||
|
"""获取任务"""
|
||||||
|
return self.tasks.get(task_id)
|
||||||
|
|
||||||
|
def get_all_tasks(self) -> List[ScheduledTask]:
|
||||||
|
"""获取所有任务"""
|
||||||
|
return list(self.tasks.values())
|
||||||
|
|
||||||
|
def get_due_tasks(self) -> List[ScheduledTask]:
|
||||||
|
"""获取到期需要执行的任务"""
|
||||||
|
now = datetime.now()
|
||||||
|
due_tasks = []
|
||||||
|
for task in self.tasks.values():
|
||||||
|
if task.enabled and task.next_run and task.next_run <= now:
|
||||||
|
due_tasks.append(task)
|
||||||
|
return due_tasks
|
||||||
179
etl_billiards/gui/models/task_model.py
Normal file
179
etl_billiards/gui/models/task_model.py
Normal file
@@ -0,0 +1,179 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""任务数据模型"""
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Optional, List, Dict, Any
|
||||||
|
|
||||||
|
|
||||||
|
class TaskStatus(Enum):
|
||||||
|
"""任务状态枚举"""
|
||||||
|
PENDING = "pending" # 待执行
|
||||||
|
RUNNING = "running" # 执行中
|
||||||
|
SUCCESS = "success" # 成功
|
||||||
|
FAILED = "failed" # 失败
|
||||||
|
CANCELLED = "cancelled" # 已取消
|
||||||
|
|
||||||
|
|
||||||
|
class TaskCategory(Enum):
|
||||||
|
"""任务分类"""
|
||||||
|
ODS = "ODS" # ODS 数据抓取任务
|
||||||
|
DWD = "DWD" # DWD 装载任务
|
||||||
|
DWS = "DWS" # DWS 汇总任务
|
||||||
|
SCHEMA = "Schema" # Schema 初始化任务
|
||||||
|
QUALITY = "Quality" # 质量检查任务
|
||||||
|
OTHER = "Other" # 其他任务
|
||||||
|
|
||||||
|
|
||||||
|
# 任务分类映射
|
||||||
|
TASK_CATEGORIES: Dict[str, TaskCategory] = {
|
||||||
|
# ODS 任务
|
||||||
|
"ODS_PAYMENT": TaskCategory.ODS,
|
||||||
|
"ODS_MEMBER": TaskCategory.ODS,
|
||||||
|
"ODS_MEMBER_CARD": TaskCategory.ODS,
|
||||||
|
"ODS_MEMBER_BALANCE": TaskCategory.ODS,
|
||||||
|
"ODS_SETTLEMENT_RECORDS": TaskCategory.ODS,
|
||||||
|
"ODS_TABLE_USE": TaskCategory.ODS,
|
||||||
|
"ODS_ASSISTANT_ACCOUNT": TaskCategory.ODS,
|
||||||
|
"ODS_ASSISTANT_LEDGER": TaskCategory.ODS,
|
||||||
|
"ODS_ASSISTANT_ABOLISH": TaskCategory.ODS,
|
||||||
|
"ODS_REFUND": TaskCategory.ODS,
|
||||||
|
"ODS_PLATFORM_COUPON": TaskCategory.ODS,
|
||||||
|
"ODS_RECHARGE_SETTLE": TaskCategory.ODS,
|
||||||
|
"ODS_GROUP_PACKAGE": TaskCategory.ODS,
|
||||||
|
"ODS_GROUP_BUY_REDEMPTION": TaskCategory.ODS,
|
||||||
|
"ODS_INVENTORY_STOCK": TaskCategory.ODS,
|
||||||
|
"ODS_INVENTORY_CHANGE": TaskCategory.ODS,
|
||||||
|
"ODS_TABLES": TaskCategory.ODS,
|
||||||
|
"ODS_GOODS_CATEGORY": TaskCategory.ODS,
|
||||||
|
"ODS_STORE_GOODS": TaskCategory.ODS,
|
||||||
|
"ODS_STORE_GOODS_SALES": TaskCategory.ODS,
|
||||||
|
"ODS_TABLE_FEE_DISCOUNT": TaskCategory.ODS,
|
||||||
|
"ODS_TENANT_GOODS": TaskCategory.ODS,
|
||||||
|
"ODS_SETTLEMENT_TICKET": TaskCategory.ODS,
|
||||||
|
# DWD 任务
|
||||||
|
"DWD_LOAD_FROM_ODS": TaskCategory.DWD,
|
||||||
|
"DWD_QUALITY_CHECK": TaskCategory.QUALITY,
|
||||||
|
"PAYMENTS_DWD": TaskCategory.DWD,
|
||||||
|
"MEMBERS_DWD": TaskCategory.DWD,
|
||||||
|
"TICKET_DWD": TaskCategory.DWD,
|
||||||
|
# DWS 任务
|
||||||
|
"INIT_DWS_SCHEMA": TaskCategory.SCHEMA,
|
||||||
|
"DWS_BUILD_ORDER_SUMMARY": TaskCategory.DWS,
|
||||||
|
# Schema 任务
|
||||||
|
"INIT_ODS_SCHEMA": TaskCategory.SCHEMA,
|
||||||
|
"INIT_DWD_SCHEMA": TaskCategory.SCHEMA,
|
||||||
|
# 其他任务
|
||||||
|
"MANUAL_INGEST": TaskCategory.OTHER,
|
||||||
|
"CHECK_CUTOFF": TaskCategory.OTHER,
|
||||||
|
"DATA_INTEGRITY_CHECK": TaskCategory.QUALITY,
|
||||||
|
"ODS_JSON_ARCHIVE": TaskCategory.OTHER,
|
||||||
|
# 旧版任务(兼容)
|
||||||
|
"PRODUCTS": TaskCategory.ODS,
|
||||||
|
"TABLES": TaskCategory.ODS,
|
||||||
|
"MEMBERS": TaskCategory.ODS,
|
||||||
|
"ASSISTANTS": TaskCategory.ODS,
|
||||||
|
"PACKAGES_DEF": TaskCategory.ODS,
|
||||||
|
"ORDERS": TaskCategory.ODS,
|
||||||
|
"PAYMENTS": TaskCategory.ODS,
|
||||||
|
"REFUNDS": TaskCategory.ODS,
|
||||||
|
"COUPON_USAGE": TaskCategory.ODS,
|
||||||
|
"INVENTORY_CHANGE": TaskCategory.ODS,
|
||||||
|
"TOPUPS": TaskCategory.ODS,
|
||||||
|
"TABLE_DISCOUNT": TaskCategory.ODS,
|
||||||
|
"ASSISTANT_ABOLISH": TaskCategory.ODS,
|
||||||
|
"LEDGER": TaskCategory.ODS,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_task_category(task_code: str) -> TaskCategory:
|
||||||
|
"""获取任务分类"""
|
||||||
|
return TASK_CATEGORIES.get(task_code.upper(), TaskCategory.OTHER)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TaskItem:
|
||||||
|
"""任务项"""
|
||||||
|
task_code: str
|
||||||
|
name: str = ""
|
||||||
|
description: str = ""
|
||||||
|
category: TaskCategory = TaskCategory.OTHER
|
||||||
|
enabled: bool = True
|
||||||
|
|
||||||
|
def __post_init__(self):
|
||||||
|
if not self.name:
|
||||||
|
self.name = self.task_code
|
||||||
|
if not self.category or self.category == TaskCategory.OTHER:
|
||||||
|
self.category = get_task_category(self.task_code)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TaskConfig:
|
||||||
|
"""任务执行配置"""
|
||||||
|
tasks: List[str] = field(default_factory=list)
|
||||||
|
pipeline_flow: str = "FULL" # FULL, FETCH_ONLY, INGEST_ONLY
|
||||||
|
dry_run: bool = False
|
||||||
|
window_start: Optional[str] = None
|
||||||
|
window_end: Optional[str] = None
|
||||||
|
window_split: Optional[str] = None # none, month
|
||||||
|
window_compensation: int = 0 # 补偿小时数
|
||||||
|
ingest_source: Optional[str] = None
|
||||||
|
store_id: Optional[int] = None
|
||||||
|
pg_dsn: Optional[str] = None
|
||||||
|
api_token: Optional[str] = None
|
||||||
|
extra_args: Dict[str, Any] = field(default_factory=dict)
|
||||||
|
env_vars: Dict[str, str] = field(default_factory=dict) # 额外环境变量
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class TaskHistory:
|
||||||
|
"""任务执行历史"""
|
||||||
|
id: str
|
||||||
|
task_codes: List[str]
|
||||||
|
status: TaskStatus
|
||||||
|
start_time: datetime
|
||||||
|
end_time: Optional[datetime] = None
|
||||||
|
exit_code: Optional[int] = None
|
||||||
|
command: str = ""
|
||||||
|
output_log: str = ""
|
||||||
|
error_message: str = ""
|
||||||
|
summary: Dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def duration_seconds(self) -> Optional[float]:
|
||||||
|
"""执行时长(秒)"""
|
||||||
|
if self.end_time and self.start_time:
|
||||||
|
return (self.end_time - self.start_time).total_seconds()
|
||||||
|
return None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def duration_str(self) -> str:
|
||||||
|
"""格式化的执行时长"""
|
||||||
|
secs = self.duration_seconds
|
||||||
|
if secs is None:
|
||||||
|
return "-"
|
||||||
|
if secs < 60:
|
||||||
|
return f"{secs:.1f}秒"
|
||||||
|
elif secs < 3600:
|
||||||
|
mins = int(secs // 60)
|
||||||
|
secs = secs % 60
|
||||||
|
return f"{mins}分{secs:.0f}秒"
|
||||||
|
else:
|
||||||
|
hours = int(secs // 3600)
|
||||||
|
mins = int((secs % 3600) // 60)
|
||||||
|
return f"{hours}时{mins}分"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class QueuedTask:
|
||||||
|
"""队列中的任务"""
|
||||||
|
id: str
|
||||||
|
config: TaskConfig
|
||||||
|
status: TaskStatus = TaskStatus.PENDING
|
||||||
|
created_at: datetime = field(default_factory=datetime.now)
|
||||||
|
started_at: Optional[datetime] = None
|
||||||
|
finished_at: Optional[datetime] = None
|
||||||
|
output: str = ""
|
||||||
|
error: str = ""
|
||||||
|
exit_code: Optional[int] = None
|
||||||
14
etl_billiards/gui/resources/__init__.py
Normal file
14
etl_billiards/gui/resources/__init__.py
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""GUI 资源模块"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
RESOURCES_DIR = Path(__file__).parent
|
||||||
|
STYLES_PATH = RESOURCES_DIR / "styles.qss"
|
||||||
|
|
||||||
|
|
||||||
|
def load_stylesheet() -> str:
|
||||||
|
"""加载样式表"""
|
||||||
|
if STYLES_PATH.exists():
|
||||||
|
return STYLES_PATH.read_text(encoding="utf-8")
|
||||||
|
return ""
|
||||||
458
etl_billiards/gui/resources/styles.qss
Normal file
458
etl_billiards/gui/resources/styles.qss
Normal file
@@ -0,0 +1,458 @@
|
|||||||
|
/* ETL GUI 现代浅色主题样式表 */
|
||||||
|
|
||||||
|
/* ========== 全局样式 ========== */
|
||||||
|
QWidget {
|
||||||
|
font-family: "Microsoft YaHei", "Segoe UI", sans-serif;
|
||||||
|
font-size: 13px;
|
||||||
|
color: #333333;
|
||||||
|
background-color: #f5f5f5;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMainWindow {
|
||||||
|
background-color: #f5f5f5;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 菜单栏 ========== */
|
||||||
|
QMenuBar {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border-bottom: 1px solid #e0e0e0;
|
||||||
|
padding: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMenuBar::item {
|
||||||
|
padding: 6px 12px;
|
||||||
|
background-color: transparent;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMenuBar::item:selected {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMenu {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #e0e0e0;
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMenu::item {
|
||||||
|
padding: 8px 24px;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QMenu::item:selected {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 工具栏 ========== */
|
||||||
|
QToolBar {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border-bottom: 1px solid #e0e0e0;
|
||||||
|
padding: 4px;
|
||||||
|
spacing: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QToolButton {
|
||||||
|
background-color: transparent;
|
||||||
|
border: none;
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QToolButton:hover {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
QToolButton:pressed {
|
||||||
|
background-color: #d2e3fc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 按钮 ========== */
|
||||||
|
QPushButton {
|
||||||
|
background-color: #1a73e8;
|
||||||
|
color: white;
|
||||||
|
border: none;
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 8px 16px;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton:hover {
|
||||||
|
background-color: #1557b0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton:pressed {
|
||||||
|
background-color: #104080;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton:disabled {
|
||||||
|
background-color: #dadce0;
|
||||||
|
color: #9aa0a6;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton[secondary="true"] {
|
||||||
|
background-color: #ffffff;
|
||||||
|
color: #1a73e8;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton[secondary="true"]:hover {
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
border-color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton[danger="true"] {
|
||||||
|
background-color: #ea4335;
|
||||||
|
}
|
||||||
|
|
||||||
|
QPushButton[danger="true"]:hover {
|
||||||
|
background-color: #c5221f;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 输入框 ========== */
|
||||||
|
QLineEdit, QTextEdit, QPlainTextEdit {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 8px 12px;
|
||||||
|
selection-background-color: #d2e3fc;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLineEdit:focus, QTextEdit:focus, QPlainTextEdit:focus {
|
||||||
|
border-color: #1a73e8;
|
||||||
|
border-width: 2px;
|
||||||
|
padding: 7px 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLineEdit:disabled, QTextEdit:disabled, QPlainTextEdit:disabled {
|
||||||
|
background-color: #f1f3f4;
|
||||||
|
color: #9aa0a6;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 下拉框 ========== */
|
||||||
|
QComboBox {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 6px;
|
||||||
|
padding: 8px 12px;
|
||||||
|
padding-right: 30px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QComboBox:hover {
|
||||||
|
border-color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
QComboBox:focus {
|
||||||
|
border-color: #1a73e8;
|
||||||
|
border-width: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QComboBox::drop-down {
|
||||||
|
border: none;
|
||||||
|
width: 24px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QComboBox::down-arrow {
|
||||||
|
image: none;
|
||||||
|
border-left: 5px solid transparent;
|
||||||
|
border-right: 5px solid transparent;
|
||||||
|
border-top: 6px solid #5f6368;
|
||||||
|
margin-right: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QComboBox QAbstractItemView {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 8px;
|
||||||
|
selection-background-color: #e8f0fe;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 复选框 ========== */
|
||||||
|
QCheckBox {
|
||||||
|
spacing: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QCheckBox::indicator {
|
||||||
|
width: 18px;
|
||||||
|
height: 18px;
|
||||||
|
border-radius: 4px;
|
||||||
|
border: 2px solid #5f6368;
|
||||||
|
}
|
||||||
|
|
||||||
|
QCheckBox::indicator:checked {
|
||||||
|
background-color: #1a73e8;
|
||||||
|
border-color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
QCheckBox::indicator:hover {
|
||||||
|
border-color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 列表和树 ========== */
|
||||||
|
QListWidget, QTreeWidget, QTableWidget {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 8px;
|
||||||
|
outline: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
QListWidget::item, QTreeWidget::item {
|
||||||
|
padding: 8px;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QListWidget::item:selected, QTreeWidget::item:selected {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
QListWidget::item:hover, QTreeWidget::item:hover {
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
}
|
||||||
|
|
||||||
|
QHeaderView::section {
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
border: none;
|
||||||
|
border-bottom: 1px solid #dadce0;
|
||||||
|
padding: 10px 16px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTableWidget {
|
||||||
|
gridline-color: #e8eaed;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTableWidget::item {
|
||||||
|
padding: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTableWidget::item:selected {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 滚动条 ========== */
|
||||||
|
QScrollBar:vertical {
|
||||||
|
background-color: transparent;
|
||||||
|
width: 12px;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::handle:vertical {
|
||||||
|
background-color: #dadce0;
|
||||||
|
border-radius: 6px;
|
||||||
|
min-height: 30px;
|
||||||
|
margin: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::handle:vertical:hover {
|
||||||
|
background-color: #bdc1c6;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::add-line:vertical, QScrollBar::sub-line:vertical {
|
||||||
|
height: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar:horizontal {
|
||||||
|
background-color: transparent;
|
||||||
|
height: 12px;
|
||||||
|
margin: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::handle:horizontal {
|
||||||
|
background-color: #dadce0;
|
||||||
|
border-radius: 6px;
|
||||||
|
min-width: 30px;
|
||||||
|
margin: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::handle:horizontal:hover {
|
||||||
|
background-color: #bdc1c6;
|
||||||
|
}
|
||||||
|
|
||||||
|
QScrollBar::add-line:horizontal, QScrollBar::sub-line:horizontal {
|
||||||
|
width: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 选项卡 ========== */
|
||||||
|
QTabWidget::pane {
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 8px;
|
||||||
|
background-color: #ffffff;
|
||||||
|
margin-top: -1px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTabBar::tab {
|
||||||
|
background-color: transparent;
|
||||||
|
border: none;
|
||||||
|
padding: 10px 20px;
|
||||||
|
margin-right: 4px;
|
||||||
|
color: #5f6368;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTabBar::tab:selected {
|
||||||
|
color: #1a73e8;
|
||||||
|
border-bottom: 2px solid #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
QTabBar::tab:hover:!selected {
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
border-radius: 6px 6px 0 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 分组框 ========== */
|
||||||
|
QGroupBox {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 8px;
|
||||||
|
margin-top: 16px;
|
||||||
|
padding: 16px;
|
||||||
|
padding-top: 24px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QGroupBox::title {
|
||||||
|
subcontrol-origin: margin;
|
||||||
|
subcontrol-position: top left;
|
||||||
|
left: 16px;
|
||||||
|
padding: 0 8px;
|
||||||
|
background-color: #ffffff;
|
||||||
|
color: #5f6368;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 进度条 ========== */
|
||||||
|
QProgressBar {
|
||||||
|
background-color: #e8eaed;
|
||||||
|
border: none;
|
||||||
|
border-radius: 4px;
|
||||||
|
height: 8px;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
QProgressBar::chunk {
|
||||||
|
background-color: #1a73e8;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 分割器 ========== */
|
||||||
|
QSplitter::handle {
|
||||||
|
background-color: #e0e0e0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QSplitter::handle:horizontal {
|
||||||
|
width: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QSplitter::handle:vertical {
|
||||||
|
height: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QSplitter::handle:hover {
|
||||||
|
background-color: #1a73e8;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 状态栏 ========== */
|
||||||
|
QStatusBar {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border-top: 1px solid #e0e0e0;
|
||||||
|
padding: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QStatusBar::item {
|
||||||
|
border: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 提示框 ========== */
|
||||||
|
QToolTip {
|
||||||
|
background-color: #3c4043;
|
||||||
|
color: #ffffff;
|
||||||
|
border: none;
|
||||||
|
border-radius: 4px;
|
||||||
|
padding: 8px 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 消息框 ========== */
|
||||||
|
QMessageBox {
|
||||||
|
background-color: #ffffff;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 导航侧边栏 ========== */
|
||||||
|
QListWidget#navList {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: none;
|
||||||
|
border-right: 1px solid #e0e0e0;
|
||||||
|
padding: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QListWidget#navList::item {
|
||||||
|
padding: 12px 16px;
|
||||||
|
border-radius: 8px;
|
||||||
|
margin: 2px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
QListWidget#navList::item:selected {
|
||||||
|
background-color: #e8f0fe;
|
||||||
|
color: #1a73e8;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 日志查看器 ========== */
|
||||||
|
QPlainTextEdit#logViewer {
|
||||||
|
font-family: "Consolas", "Courier New", monospace;
|
||||||
|
font-size: 12px;
|
||||||
|
background-color: #fafafa;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== SQL 编辑器 ========== */
|
||||||
|
QPlainTextEdit#sqlEditor {
|
||||||
|
font-family: "Consolas", "Courier New", monospace;
|
||||||
|
font-size: 13px;
|
||||||
|
background-color: #ffffff;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 卡片样式 ========== */
|
||||||
|
QFrame[card="true"] {
|
||||||
|
background-color: #ffffff;
|
||||||
|
border: 1px solid #dadce0;
|
||||||
|
border-radius: 12px;
|
||||||
|
padding: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
QFrame[card="true"]:hover {
|
||||||
|
border-color: #1a73e8;
|
||||||
|
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ========== 标签 ========== */
|
||||||
|
QLabel[heading="true"] {
|
||||||
|
font-size: 18px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: #202124;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLabel[subheading="true"] {
|
||||||
|
font-size: 14px;
|
||||||
|
color: #5f6368;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLabel[status="success"] {
|
||||||
|
color: #1e8e3e;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLabel[status="error"] {
|
||||||
|
color: #d93025;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLabel[status="warning"] {
|
||||||
|
color: #f9ab00;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
QLabel[status="info"] {
|
||||||
|
color: #1a73e8;
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
8
etl_billiards/gui/utils/__init__.py
Normal file
8
etl_billiards/gui/utils/__init__.py
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""工具模块"""
|
||||||
|
|
||||||
|
from .cli_builder import CLIBuilder
|
||||||
|
from .config_helper import ConfigHelper
|
||||||
|
from .app_settings import app_settings, AppSettings
|
||||||
|
|
||||||
|
__all__ = ["CLIBuilder", "ConfigHelper", "app_settings", "AppSettings"]
|
||||||
434
etl_billiards/gui/utils/app_settings.py
Normal file
434
etl_billiards/gui/utils/app_settings.py
Normal file
@@ -0,0 +1,434 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""应用程序设置管理"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, Optional
|
||||||
|
|
||||||
|
|
||||||
|
class AppSettings:
|
||||||
|
"""应用程序设置单例"""
|
||||||
|
|
||||||
|
_instance: Optional["AppSettings"] = None
|
||||||
|
|
||||||
|
def __new__(cls):
|
||||||
|
if cls._instance is None:
|
||||||
|
cls._instance = super().__new__(cls)
|
||||||
|
cls._instance._initialized = False
|
||||||
|
return cls._instance
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
if self._initialized:
|
||||||
|
return
|
||||||
|
self._initialized = True
|
||||||
|
|
||||||
|
# 配置文件路径
|
||||||
|
self._settings_file = self._get_settings_path()
|
||||||
|
|
||||||
|
# 默认设置
|
||||||
|
self._settings = {
|
||||||
|
"etl_project_path": "", # ETL 项目路径
|
||||||
|
"env_file_path": "", # .env 文件路径
|
||||||
|
# 自动更新配置
|
||||||
|
"auto_update": {
|
||||||
|
"hours": 24,
|
||||||
|
"overlap_seconds": 3600,
|
||||||
|
"include_dwd": True,
|
||||||
|
"auto_verify": False,
|
||||||
|
"selected_tasks": [],
|
||||||
|
},
|
||||||
|
# 数据校验配置
|
||||||
|
"integrity_check": {
|
||||||
|
"mode": "history",
|
||||||
|
"history_start": "",
|
||||||
|
"history_end": "",
|
||||||
|
"lookback_hours": 24,
|
||||||
|
"include_dimensions": False,
|
||||||
|
"auto_backfill": False,
|
||||||
|
"ods_tasks": "",
|
||||||
|
},
|
||||||
|
# 高级配置
|
||||||
|
"advanced": {
|
||||||
|
"pipeline_flow": "FULL",
|
||||||
|
"dry_run": False,
|
||||||
|
"window_start": "",
|
||||||
|
"window_end": "",
|
||||||
|
"window_split": "none",
|
||||||
|
"window_compensation": 0,
|
||||||
|
"ingest_source": "",
|
||||||
|
"store_id": "",
|
||||||
|
"pg_dsn": "",
|
||||||
|
"api_token": "",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# 加载设置
|
||||||
|
self._load()
|
||||||
|
|
||||||
|
# 如果没有配置,尝试自动检测
|
||||||
|
if not self._settings["etl_project_path"]:
|
||||||
|
self._auto_detect_paths()
|
||||||
|
|
||||||
|
def _get_settings_path(self) -> Path:
|
||||||
|
"""获取设置文件路径"""
|
||||||
|
# 优先使用用户目录
|
||||||
|
if sys.platform == "win32":
|
||||||
|
app_data = os.environ.get("APPDATA", "")
|
||||||
|
if app_data:
|
||||||
|
settings_dir = Path(app_data) / "ETL管理系统"
|
||||||
|
else:
|
||||||
|
settings_dir = Path.home() / ".etl_gui"
|
||||||
|
else:
|
||||||
|
settings_dir = Path.home() / ".etl_gui"
|
||||||
|
|
||||||
|
settings_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
return settings_dir / "settings.json"
|
||||||
|
|
||||||
|
def _auto_detect_paths(self):
|
||||||
|
"""自动检测 ETL 项目路径"""
|
||||||
|
# 方法1: 检查是否从源码目录运行
|
||||||
|
try:
|
||||||
|
source_dir = Path(__file__).resolve().parents[2]
|
||||||
|
cli_main = source_dir / "cli" / "main.py"
|
||||||
|
if cli_main.exists():
|
||||||
|
rel_source = Path(os.path.relpath(source_dir, Path.cwd()))
|
||||||
|
self._settings["etl_project_path"] = str(rel_source)
|
||||||
|
env_file = rel_source / ".env"
|
||||||
|
if env_file.exists():
|
||||||
|
self._settings["env_file_path"] = str(env_file)
|
||||||
|
self._save()
|
||||||
|
return
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# 方法2: 检查常见位置
|
||||||
|
common_paths = [
|
||||||
|
Path("etl_billiards"),
|
||||||
|
Path("."),
|
||||||
|
]
|
||||||
|
|
||||||
|
for path in common_paths:
|
||||||
|
if path.exists() and (path / "cli" / "main.py").exists():
|
||||||
|
self._settings["etl_project_path"] = str(path)
|
||||||
|
env_file = path / ".env"
|
||||||
|
if env_file.exists():
|
||||||
|
self._settings["env_file_path"] = str(env_file)
|
||||||
|
self._save()
|
||||||
|
return
|
||||||
|
|
||||||
|
def _load(self):
|
||||||
|
"""加载设置"""
|
||||||
|
if self._settings_file.exists():
|
||||||
|
try:
|
||||||
|
data = json.loads(self._settings_file.read_text(encoding="utf-8"))
|
||||||
|
self._settings.update(data)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _save(self):
|
||||||
|
"""保存设置"""
|
||||||
|
try:
|
||||||
|
self._settings_file.write_text(
|
||||||
|
json.dumps(self._settings, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
@property
|
||||||
|
def etl_project_path(self) -> str:
|
||||||
|
"""获取 ETL 项目路径"""
|
||||||
|
return self._settings.get("etl_project_path", "")
|
||||||
|
|
||||||
|
@etl_project_path.setter
|
||||||
|
def etl_project_path(self, value: str):
|
||||||
|
"""设置 ETL 项目路径"""
|
||||||
|
self._settings["etl_project_path"] = value
|
||||||
|
# 同时更新 .env 路径
|
||||||
|
if value:
|
||||||
|
env_path = Path(value) / ".env"
|
||||||
|
if env_path.exists():
|
||||||
|
self._settings["env_file_path"] = str(env_path)
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def env_file_path(self) -> str:
|
||||||
|
"""获取 .env 文件路径"""
|
||||||
|
path = self._settings.get("env_file_path", "")
|
||||||
|
if not path and self.etl_project_path:
|
||||||
|
path = str(Path(self.etl_project_path) / ".env")
|
||||||
|
return path
|
||||||
|
|
||||||
|
@env_file_path.setter
|
||||||
|
def env_file_path(self, value: str):
|
||||||
|
"""设置 .env 文件路径"""
|
||||||
|
self._settings["env_file_path"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
def is_configured(self) -> bool:
|
||||||
|
"""检查是否已配置"""
|
||||||
|
path = self.etl_project_path
|
||||||
|
if not path:
|
||||||
|
return False
|
||||||
|
return Path(path).exists() and (Path(path) / "cli" / "main.py").exists()
|
||||||
|
|
||||||
|
def validate(self) -> tuple[bool, str]:
|
||||||
|
"""验证配置"""
|
||||||
|
path = self.etl_project_path
|
||||||
|
if not path:
|
||||||
|
return False, "未配置 ETL 项目路径"
|
||||||
|
|
||||||
|
project_path = Path(path)
|
||||||
|
if not project_path.exists():
|
||||||
|
return False, f"ETL 项目路径不存在: {path}"
|
||||||
|
|
||||||
|
cli_main = project_path / "cli" / "main.py"
|
||||||
|
if not cli_main.exists():
|
||||||
|
return False, f"找不到 CLI 入口: {cli_main}"
|
||||||
|
|
||||||
|
return True, "配置有效"
|
||||||
|
|
||||||
|
# ==================== 自动更新配置 ====================
|
||||||
|
|
||||||
|
@property
|
||||||
|
def auto_update_hours(self) -> int:
|
||||||
|
return self._settings.get("auto_update", {}).get("hours", 24)
|
||||||
|
|
||||||
|
@auto_update_hours.setter
|
||||||
|
def auto_update_hours(self, value: int):
|
||||||
|
self._settings.setdefault("auto_update", {})["hours"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def auto_update_overlap_seconds(self) -> int:
|
||||||
|
return self._settings.get("auto_update", {}).get("overlap_seconds", 3600)
|
||||||
|
|
||||||
|
@auto_update_overlap_seconds.setter
|
||||||
|
def auto_update_overlap_seconds(self, value: int):
|
||||||
|
self._settings.setdefault("auto_update", {})["overlap_seconds"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def auto_update_include_dwd(self) -> bool:
|
||||||
|
return self._settings.get("auto_update", {}).get("include_dwd", True)
|
||||||
|
|
||||||
|
@auto_update_include_dwd.setter
|
||||||
|
def auto_update_include_dwd(self, value: bool):
|
||||||
|
self._settings.setdefault("auto_update", {})["include_dwd"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def auto_update_auto_verify(self) -> bool:
|
||||||
|
return self._settings.get("auto_update", {}).get("auto_verify", False)
|
||||||
|
|
||||||
|
@auto_update_auto_verify.setter
|
||||||
|
def auto_update_auto_verify(self, value: bool):
|
||||||
|
self._settings.setdefault("auto_update", {})["auto_verify"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def auto_update_selected_tasks(self) -> list:
|
||||||
|
return self._settings.get("auto_update", {}).get("selected_tasks", [])
|
||||||
|
|
||||||
|
@auto_update_selected_tasks.setter
|
||||||
|
def auto_update_selected_tasks(self, value: list):
|
||||||
|
self._settings.setdefault("auto_update", {})["selected_tasks"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
# ==================== 数据校验配置 ====================
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_mode(self) -> str:
|
||||||
|
return self._settings.get("integrity_check", {}).get("mode", "history")
|
||||||
|
|
||||||
|
@integrity_mode.setter
|
||||||
|
def integrity_mode(self, value: str):
|
||||||
|
self._settings.setdefault("integrity_check", {})["mode"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_history_start(self) -> str:
|
||||||
|
return self._settings.get("integrity_check", {}).get("history_start", "")
|
||||||
|
|
||||||
|
@integrity_history_start.setter
|
||||||
|
def integrity_history_start(self, value: str):
|
||||||
|
self._settings.setdefault("integrity_check", {})["history_start"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_history_end(self) -> str:
|
||||||
|
return self._settings.get("integrity_check", {}).get("history_end", "")
|
||||||
|
|
||||||
|
@integrity_history_end.setter
|
||||||
|
def integrity_history_end(self, value: str):
|
||||||
|
self._settings.setdefault("integrity_check", {})["history_end"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_lookback_hours(self) -> int:
|
||||||
|
return self._settings.get("integrity_check", {}).get("lookback_hours", 24)
|
||||||
|
|
||||||
|
@integrity_lookback_hours.setter
|
||||||
|
def integrity_lookback_hours(self, value: int):
|
||||||
|
self._settings.setdefault("integrity_check", {})["lookback_hours"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_include_dimensions(self) -> bool:
|
||||||
|
return self._settings.get("integrity_check", {}).get("include_dimensions", False)
|
||||||
|
|
||||||
|
@integrity_include_dimensions.setter
|
||||||
|
def integrity_include_dimensions(self, value: bool):
|
||||||
|
self._settings.setdefault("integrity_check", {})["include_dimensions"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_auto_backfill(self) -> bool:
|
||||||
|
return self._settings.get("integrity_check", {}).get("auto_backfill", False)
|
||||||
|
|
||||||
|
@integrity_auto_backfill.setter
|
||||||
|
def integrity_auto_backfill(self, value: bool):
|
||||||
|
self._settings.setdefault("integrity_check", {})["auto_backfill"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def integrity_ods_tasks(self) -> str:
|
||||||
|
return self._settings.get("integrity_check", {}).get("ods_tasks", "")
|
||||||
|
|
||||||
|
@integrity_ods_tasks.setter
|
||||||
|
def integrity_ods_tasks(self, value: str):
|
||||||
|
self._settings.setdefault("integrity_check", {})["ods_tasks"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
# ==================== 高级配置 ====================
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_pipeline_flow(self) -> str:
|
||||||
|
return self._settings.get("advanced", {}).get("pipeline_flow", "FULL")
|
||||||
|
|
||||||
|
@advanced_pipeline_flow.setter
|
||||||
|
def advanced_pipeline_flow(self, value: str):
|
||||||
|
self._settings.setdefault("advanced", {})["pipeline_flow"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_dry_run(self) -> bool:
|
||||||
|
return self._settings.get("advanced", {}).get("dry_run", False)
|
||||||
|
|
||||||
|
@advanced_dry_run.setter
|
||||||
|
def advanced_dry_run(self, value: bool):
|
||||||
|
self._settings.setdefault("advanced", {})["dry_run"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_window_start(self) -> str:
|
||||||
|
return self._settings.get("advanced", {}).get("window_start", "")
|
||||||
|
|
||||||
|
@advanced_window_start.setter
|
||||||
|
def advanced_window_start(self, value: str):
|
||||||
|
self._settings.setdefault("advanced", {})["window_start"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_window_end(self) -> str:
|
||||||
|
return self._settings.get("advanced", {}).get("window_end", "")
|
||||||
|
|
||||||
|
@advanced_window_end.setter
|
||||||
|
def advanced_window_end(self, value: str):
|
||||||
|
self._settings.setdefault("advanced", {})["window_end"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_ingest_source(self) -> str:
|
||||||
|
return self._settings.get("advanced", {}).get("ingest_source", "")
|
||||||
|
|
||||||
|
@advanced_ingest_source.setter
|
||||||
|
def advanced_ingest_source(self, value: str):
|
||||||
|
self._settings.setdefault("advanced", {})["ingest_source"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_window_split(self) -> str:
|
||||||
|
return self._settings.get("advanced", {}).get("window_split", "none")
|
||||||
|
|
||||||
|
@advanced_window_split.setter
|
||||||
|
def advanced_window_split(self, value: str):
|
||||||
|
self._settings.setdefault("advanced", {})["window_split"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def advanced_window_compensation(self) -> int:
|
||||||
|
return self._settings.get("advanced", {}).get("window_compensation", 0)
|
||||||
|
|
||||||
|
@advanced_window_compensation.setter
|
||||||
|
def advanced_window_compensation(self, value: int):
|
||||||
|
self._settings.setdefault("advanced", {})["window_compensation"] = value
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
def get_all_settings(self) -> Dict[str, Any]:
|
||||||
|
"""获取所有设置(用于调试)"""
|
||||||
|
return self._settings.copy()
|
||||||
|
|
||||||
|
def save_all(self):
|
||||||
|
"""强制保存所有设置"""
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
# ==================== 任务历史存储 ====================
|
||||||
|
|
||||||
|
def _get_history_path(self) -> Path:
|
||||||
|
"""获取任务历史文件路径"""
|
||||||
|
return self._settings_file.parent / "task_history.json"
|
||||||
|
|
||||||
|
def save_task_history(self, history_list: list):
|
||||||
|
"""保存任务历史到文件"""
|
||||||
|
try:
|
||||||
|
history_path = self._get_history_path()
|
||||||
|
|
||||||
|
# 序列化任务历史
|
||||||
|
serialized = []
|
||||||
|
for task in history_list[:100]: # 最多保存100条
|
||||||
|
try:
|
||||||
|
task_data = {
|
||||||
|
"id": task.id,
|
||||||
|
"tasks": task.config.tasks if hasattr(task, 'config') else [],
|
||||||
|
"status": task.status.value if hasattr(task.status, 'value') else str(task.status),
|
||||||
|
"created_at": task.created_at.isoformat() if task.created_at else None,
|
||||||
|
"started_at": task.started_at.isoformat() if task.started_at else None,
|
||||||
|
"finished_at": task.finished_at.isoformat() if task.finished_at else None,
|
||||||
|
"exit_code": task.exit_code,
|
||||||
|
"error": task.error[:500] if task.error else "", # 限制长度
|
||||||
|
"output_preview": task.output[:1000] if task.output else "", # 输出预览
|
||||||
|
# 保存配置信息
|
||||||
|
"pipeline_flow": task.config.pipeline_flow if hasattr(task, 'config') else "FULL",
|
||||||
|
"window_start": task.config.window_start if hasattr(task, 'config') else None,
|
||||||
|
"window_end": task.config.window_end if hasattr(task, 'config') else None,
|
||||||
|
}
|
||||||
|
serialized.append(task_data)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
|
||||||
|
history_path.write_text(
|
||||||
|
json.dumps(serialized, ensure_ascii=False, indent=2),
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"保存任务历史失败: {e}")
|
||||||
|
|
||||||
|
def load_task_history(self) -> list:
|
||||||
|
"""从文件加载任务历史"""
|
||||||
|
try:
|
||||||
|
history_path = self._get_history_path()
|
||||||
|
if not history_path.exists():
|
||||||
|
return []
|
||||||
|
|
||||||
|
data = json.loads(history_path.read_text(encoding="utf-8"))
|
||||||
|
return data
|
||||||
|
except Exception as e:
|
||||||
|
print(f"加载任务历史失败: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
# 全局单例
|
||||||
|
app_settings = AppSettings()
|
||||||
131
etl_billiards/gui/utils/cli_builder.py
Normal file
131
etl_billiards/gui/utils/cli_builder.py
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""CLI 命令构建器"""
|
||||||
|
|
||||||
|
from typing import List, Dict, Any, Optional
|
||||||
|
from ..models.task_model import TaskConfig
|
||||||
|
|
||||||
|
|
||||||
|
# CLI 支持的命令行参数(来自 cli/main.py)
|
||||||
|
CLI_SUPPORTED_ARGS = {
|
||||||
|
# 值类型参数
|
||||||
|
"store_id", "tasks", "pg_dsn", "pg_host", "pg_port", "pg_name",
|
||||||
|
"pg_user", "pg_password", "api_base", "api_token", "api_timeout",
|
||||||
|
"api_page_size", "api_retry_max", "window_start", "window_end",
|
||||||
|
"export_root", "log_root", "pipeline_flow", "fetch_root",
|
||||||
|
"ingest_source", "idle_start", "idle_end",
|
||||||
|
# 布尔类型参数
|
||||||
|
"dry_run", "force_window_override", "write_pretty_json", "allow_empty_advance",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class CLIBuilder:
|
||||||
|
"""构建 CLI 命令行参数"""
|
||||||
|
|
||||||
|
def __init__(self, python_executable: str = "python"):
|
||||||
|
self.python_executable = python_executable
|
||||||
|
|
||||||
|
def build_command(self, config: TaskConfig) -> List[str]:
|
||||||
|
"""
|
||||||
|
根据任务配置构建命令行参数列表
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: 任务配置对象
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
命令行参数列表
|
||||||
|
"""
|
||||||
|
cmd = [self.python_executable, "-m", "cli.main"]
|
||||||
|
|
||||||
|
# 任务列表
|
||||||
|
if config.tasks:
|
||||||
|
cmd.extend(["--tasks", ",".join(config.tasks)])
|
||||||
|
|
||||||
|
# Pipeline 流程
|
||||||
|
if config.pipeline_flow:
|
||||||
|
cmd.extend(["--pipeline-flow", config.pipeline_flow])
|
||||||
|
|
||||||
|
# Dry-run 模式
|
||||||
|
if config.dry_run:
|
||||||
|
cmd.append("--dry-run")
|
||||||
|
|
||||||
|
# 时间窗口
|
||||||
|
if config.window_start:
|
||||||
|
cmd.extend(["--window-start", config.window_start])
|
||||||
|
if config.window_end:
|
||||||
|
cmd.extend(["--window-end", config.window_end])
|
||||||
|
|
||||||
|
# 数据源目录
|
||||||
|
if config.ingest_source:
|
||||||
|
cmd.extend(["--ingest-source", config.ingest_source])
|
||||||
|
|
||||||
|
# 门店 ID
|
||||||
|
if config.store_id is not None:
|
||||||
|
cmd.extend(["--store-id", str(config.store_id)])
|
||||||
|
|
||||||
|
# 数据库 DSN
|
||||||
|
if config.pg_dsn:
|
||||||
|
cmd.extend(["--pg-dsn", config.pg_dsn])
|
||||||
|
|
||||||
|
# API Token
|
||||||
|
if config.api_token:
|
||||||
|
cmd.extend(["--api-token", config.api_token])
|
||||||
|
|
||||||
|
# 额外参数(只传递 CLI 支持的参数)
|
||||||
|
for key, value in config.extra_args.items():
|
||||||
|
if value is not None and key in CLI_SUPPORTED_ARGS:
|
||||||
|
arg_name = f"--{key.replace('_', '-')}"
|
||||||
|
if isinstance(value, bool):
|
||||||
|
if value:
|
||||||
|
cmd.append(arg_name)
|
||||||
|
else:
|
||||||
|
cmd.extend([arg_name, str(value)])
|
||||||
|
|
||||||
|
return cmd
|
||||||
|
|
||||||
|
def build_command_string(self, config: TaskConfig) -> str:
|
||||||
|
"""
|
||||||
|
构建命令行字符串(用于显示)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: 任务配置对象
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
命令行字符串
|
||||||
|
"""
|
||||||
|
cmd = self.build_command(config)
|
||||||
|
# 对包含空格的参数添加引号
|
||||||
|
quoted_cmd = []
|
||||||
|
for arg in cmd:
|
||||||
|
if ' ' in arg or '"' in arg:
|
||||||
|
quoted_cmd.append(f'"{arg}"')
|
||||||
|
else:
|
||||||
|
quoted_cmd.append(arg)
|
||||||
|
return " ".join(quoted_cmd)
|
||||||
|
|
||||||
|
def build_from_dict(self, params: Dict[str, Any]) -> List[str]:
|
||||||
|
"""
|
||||||
|
从字典构建命令行参数
|
||||||
|
|
||||||
|
Args:
|
||||||
|
params: 参数字典
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
命令行参数列表
|
||||||
|
"""
|
||||||
|
config = TaskConfig(
|
||||||
|
tasks=params.get("tasks", []),
|
||||||
|
pipeline_flow=params.get("pipeline_flow", "FULL"),
|
||||||
|
dry_run=params.get("dry_run", False),
|
||||||
|
window_start=params.get("window_start"),
|
||||||
|
window_end=params.get("window_end"),
|
||||||
|
ingest_source=params.get("ingest_source"),
|
||||||
|
store_id=params.get("store_id"),
|
||||||
|
pg_dsn=params.get("pg_dsn"),
|
||||||
|
api_token=params.get("api_token"),
|
||||||
|
extra_args=params.get("extra_args", {}),
|
||||||
|
)
|
||||||
|
return self.build_command(config)
|
||||||
|
|
||||||
|
|
||||||
|
# 全局实例
|
||||||
|
cli_builder = CLIBuilder()
|
||||||
309
etl_billiards/gui/utils/config_helper.py
Normal file
309
etl_billiards/gui/utils/config_helper.py
Normal file
@@ -0,0 +1,309 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""配置辅助工具"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, List, Tuple, Optional, Any
|
||||||
|
|
||||||
|
|
||||||
|
# 环境变量分组
|
||||||
|
ENV_GROUPS = {
|
||||||
|
"database": {
|
||||||
|
"title": "数据库配置",
|
||||||
|
"keys": ["PG_DSN", "PG_HOST", "PG_PORT", "PG_NAME", "PG_USER", "PG_PASSWORD", "PG_CONNECT_TIMEOUT"],
|
||||||
|
"sensitive": ["PG_PASSWORD"],
|
||||||
|
},
|
||||||
|
"api": {
|
||||||
|
"title": "API 配置",
|
||||||
|
"keys": ["API_BASE", "API_TOKEN", "FICOO_TOKEN", "API_TIMEOUT", "API_PAGE_SIZE", "API_RETRY_MAX"],
|
||||||
|
"sensitive": ["API_TOKEN", "FICOO_TOKEN"],
|
||||||
|
},
|
||||||
|
"store": {
|
||||||
|
"title": "门店配置",
|
||||||
|
"keys": ["STORE_ID", "TIMEZONE", "SCHEMA_OLTP", "SCHEMA_ETL"],
|
||||||
|
"sensitive": [],
|
||||||
|
},
|
||||||
|
"paths": {
|
||||||
|
"title": "路径配置",
|
||||||
|
"keys": ["EXPORT_ROOT", "LOG_ROOT", "FETCH_ROOT", "INGEST_SOURCE_DIR", "JSON_FETCH_ROOT", "JSON_SOURCE_DIR"],
|
||||||
|
"sensitive": [],
|
||||||
|
},
|
||||||
|
"pipeline": {
|
||||||
|
"title": "流水线配置",
|
||||||
|
"keys": ["PIPELINE_FLOW", "RUN_TASKS", "OVERLAP_SECONDS"],
|
||||||
|
"sensitive": [],
|
||||||
|
},
|
||||||
|
"window": {
|
||||||
|
"title": "时间窗口配置",
|
||||||
|
"keys": ["WINDOW_START", "WINDOW_END", "WINDOW_BUSY_MIN", "WINDOW_IDLE_MIN", "IDLE_START", "IDLE_END"],
|
||||||
|
"sensitive": [],
|
||||||
|
},
|
||||||
|
"integrity": {
|
||||||
|
"title": "数据完整性配置",
|
||||||
|
"keys": ["INTEGRITY_MODE", "INTEGRITY_HISTORY_START", "INTEGRITY_HISTORY_END",
|
||||||
|
"INTEGRITY_INCLUDE_DIMENSIONS", "INTEGRITY_AUTO_CHECK", "INTEGRITY_ODS_TASK_CODES"],
|
||||||
|
"sensitive": [],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigHelper:
|
||||||
|
"""配置文件辅助类"""
|
||||||
|
|
||||||
|
def __init__(self, env_path: Optional[Path] = None):
|
||||||
|
"""
|
||||||
|
初始化配置辅助器
|
||||||
|
|
||||||
|
Args:
|
||||||
|
env_path: .env 文件路径,默认使用 AppSettings 中的路径
|
||||||
|
"""
|
||||||
|
if env_path is not None:
|
||||||
|
self.env_path = Path(env_path)
|
||||||
|
else:
|
||||||
|
# 从 AppSettings 获取路径
|
||||||
|
from .app_settings import app_settings
|
||||||
|
settings_path = app_settings.env_file_path
|
||||||
|
if settings_path:
|
||||||
|
self.env_path = Path(settings_path)
|
||||||
|
else:
|
||||||
|
# 回退到源码目录
|
||||||
|
self.env_path = Path(__file__).resolve().parents[2] / ".env"
|
||||||
|
|
||||||
|
def load_env(self) -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
加载 .env 文件内容
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
环境变量字典
|
||||||
|
"""
|
||||||
|
env_vars = {}
|
||||||
|
if not self.env_path.exists():
|
||||||
|
return env_vars
|
||||||
|
|
||||||
|
try:
|
||||||
|
content = self.env_path.read_text(encoding="utf-8", errors="ignore")
|
||||||
|
for line in content.splitlines():
|
||||||
|
parsed = self._parse_line(line)
|
||||||
|
if parsed:
|
||||||
|
key, value = parsed
|
||||||
|
env_vars[key] = value
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
return env_vars
|
||||||
|
|
||||||
|
def save_env(self, env_vars: Dict[str, str]) -> bool:
|
||||||
|
"""
|
||||||
|
保存环境变量到 .env 文件
|
||||||
|
|
||||||
|
Args:
|
||||||
|
env_vars: 环境变量字典
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
是否保存成功
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
lines = []
|
||||||
|
# 按分组输出
|
||||||
|
written_keys = set()
|
||||||
|
|
||||||
|
for group_id, group_info in ENV_GROUPS.items():
|
||||||
|
group_lines = []
|
||||||
|
for key in group_info["keys"]:
|
||||||
|
if key in env_vars:
|
||||||
|
value = env_vars[key]
|
||||||
|
group_lines.append(self._format_line(key, value))
|
||||||
|
written_keys.add(key)
|
||||||
|
|
||||||
|
if group_lines:
|
||||||
|
lines.append(f"\n# {group_info['title']}")
|
||||||
|
lines.extend(group_lines)
|
||||||
|
|
||||||
|
# 写入未分组的变量
|
||||||
|
other_lines = []
|
||||||
|
for key, value in env_vars.items():
|
||||||
|
if key not in written_keys:
|
||||||
|
other_lines.append(self._format_line(key, value))
|
||||||
|
|
||||||
|
if other_lines:
|
||||||
|
lines.append("\n# 其他配置")
|
||||||
|
lines.extend(other_lines)
|
||||||
|
|
||||||
|
content = "\n".join(lines).strip() + "\n"
|
||||||
|
self.env_path.write_text(content, encoding="utf-8")
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def get_grouped_env(self) -> Dict[str, List[Tuple[str, str, bool]]]:
|
||||||
|
"""
|
||||||
|
获取分组的环境变量
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
分组字典 {group_id: [(key, value, is_sensitive), ...]}
|
||||||
|
"""
|
||||||
|
env_vars = self.load_env()
|
||||||
|
result = {}
|
||||||
|
used_keys = set()
|
||||||
|
|
||||||
|
for group_id, group_info in ENV_GROUPS.items():
|
||||||
|
items = []
|
||||||
|
for key in group_info["keys"]:
|
||||||
|
value = env_vars.get(key, "")
|
||||||
|
is_sensitive = key in group_info.get("sensitive", [])
|
||||||
|
items.append((key, value, is_sensitive))
|
||||||
|
if key in env_vars:
|
||||||
|
used_keys.add(key)
|
||||||
|
result[group_id] = items
|
||||||
|
|
||||||
|
# 添加未分组的变量到 "other" 组
|
||||||
|
other_items = []
|
||||||
|
for key, value in env_vars.items():
|
||||||
|
if key not in used_keys:
|
||||||
|
other_items.append((key, value, False))
|
||||||
|
if other_items:
|
||||||
|
result["other"] = other_items
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
def validate_env(self, env_vars: Dict[str, str]) -> List[str]:
|
||||||
|
"""
|
||||||
|
验证环境变量
|
||||||
|
|
||||||
|
Args:
|
||||||
|
env_vars: 环境变量字典
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
错误消息列表
|
||||||
|
"""
|
||||||
|
errors = []
|
||||||
|
|
||||||
|
# 验证 PG_DSN 格式
|
||||||
|
pg_dsn = env_vars.get("PG_DSN", "")
|
||||||
|
if pg_dsn and not pg_dsn.startswith("postgresql://"):
|
||||||
|
errors.append("PG_DSN 应以 'postgresql://' 开头")
|
||||||
|
|
||||||
|
# 验证端口号
|
||||||
|
pg_port = env_vars.get("PG_PORT", "")
|
||||||
|
if pg_port:
|
||||||
|
try:
|
||||||
|
port = int(pg_port)
|
||||||
|
if port < 1 or port > 65535:
|
||||||
|
errors.append("PG_PORT 应在 1-65535 范围内")
|
||||||
|
except ValueError:
|
||||||
|
errors.append("PG_PORT 应为数字")
|
||||||
|
|
||||||
|
# 验证 STORE_ID
|
||||||
|
store_id = env_vars.get("STORE_ID", "")
|
||||||
|
if store_id:
|
||||||
|
try:
|
||||||
|
int(store_id)
|
||||||
|
except ValueError:
|
||||||
|
errors.append("STORE_ID 应为数字")
|
||||||
|
|
||||||
|
# 验证路径存在性(可选)
|
||||||
|
for key in ["EXPORT_ROOT", "LOG_ROOT", "FETCH_ROOT"]:
|
||||||
|
path = env_vars.get(key, "")
|
||||||
|
if path and not os.path.isabs(path):
|
||||||
|
errors.append(f"{key} 建议使用绝对路径")
|
||||||
|
|
||||||
|
return errors
|
||||||
|
|
||||||
|
def mask_sensitive(self, value: str, visible_chars: int = 4) -> str:
|
||||||
|
"""
|
||||||
|
脱敏敏感值
|
||||||
|
|
||||||
|
Args:
|
||||||
|
value: 原始值
|
||||||
|
visible_chars: 可见字符数
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
脱敏后的值
|
||||||
|
"""
|
||||||
|
if not value or len(value) <= visible_chars:
|
||||||
|
return "*" * len(value) if value else ""
|
||||||
|
return value[:visible_chars] + "*" * (len(value) - visible_chars)
|
||||||
|
|
||||||
|
def _parse_line(self, line: str) -> Optional[Tuple[str, str]]:
|
||||||
|
"""解析 .env 文件的一行"""
|
||||||
|
stripped = line.strip()
|
||||||
|
if not stripped or stripped.startswith("#"):
|
||||||
|
return None
|
||||||
|
if stripped.startswith("export "):
|
||||||
|
stripped = stripped[7:].strip()
|
||||||
|
if "=" not in stripped:
|
||||||
|
return None
|
||||||
|
|
||||||
|
key, value = stripped.split("=", 1)
|
||||||
|
key = key.strip()
|
||||||
|
value = self._unquote_value(value)
|
||||||
|
return key, value
|
||||||
|
|
||||||
|
def _unquote_value(self, value: str) -> str:
|
||||||
|
"""处理引号和注释"""
|
||||||
|
# 去除内联注释
|
||||||
|
value = self._strip_inline_comment(value)
|
||||||
|
value = value.rstrip(",").strip()
|
||||||
|
|
||||||
|
if not value:
|
||||||
|
return value
|
||||||
|
|
||||||
|
# 去除引号
|
||||||
|
if len(value) >= 2 and value[0] in ("'", '"') and value[-1] == value[0]:
|
||||||
|
return value[1:-1]
|
||||||
|
if len(value) >= 3 and value[0] in ("r", "R") and value[1] in ("'", '"') and value[-1] == value[1]:
|
||||||
|
return value[2:-1]
|
||||||
|
|
||||||
|
return value
|
||||||
|
|
||||||
|
def _strip_inline_comment(self, value: str) -> str:
|
||||||
|
"""去除内联注释"""
|
||||||
|
result = []
|
||||||
|
in_quote = False
|
||||||
|
quote_char = ""
|
||||||
|
escape = False
|
||||||
|
|
||||||
|
for ch in value:
|
||||||
|
if escape:
|
||||||
|
result.append(ch)
|
||||||
|
escape = False
|
||||||
|
continue
|
||||||
|
if ch == "\\":
|
||||||
|
escape = True
|
||||||
|
result.append(ch)
|
||||||
|
continue
|
||||||
|
if ch in ("'", '"'):
|
||||||
|
if not in_quote:
|
||||||
|
in_quote = True
|
||||||
|
quote_char = ch
|
||||||
|
elif quote_char == ch:
|
||||||
|
in_quote = False
|
||||||
|
quote_char = ""
|
||||||
|
result.append(ch)
|
||||||
|
continue
|
||||||
|
if ch == "#" and not in_quote:
|
||||||
|
break
|
||||||
|
result.append(ch)
|
||||||
|
|
||||||
|
return "".join(result).rstrip()
|
||||||
|
|
||||||
|
def _format_line(self, key: str, value: str) -> str:
|
||||||
|
"""格式化为 .env 行"""
|
||||||
|
# 如果值包含特殊字符,使用引号包裹
|
||||||
|
if any(c in value for c in [' ', '"', "'", '#', '\n', '\r']):
|
||||||
|
# 使用双引号,转义内部的双引号
|
||||||
|
escaped = value.replace('\\', '\\\\').replace('"', '\\"')
|
||||||
|
return f'{key}="{escaped}"'
|
||||||
|
return f"{key}={value}"
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def get_group_title(group_id: str) -> str:
|
||||||
|
"""获取分组标题"""
|
||||||
|
if group_id in ENV_GROUPS:
|
||||||
|
return ENV_GROUPS[group_id]["title"]
|
||||||
|
return "其他配置"
|
||||||
|
|
||||||
|
|
||||||
|
# 全局实例
|
||||||
|
config_helper = ConfigHelper()
|
||||||
18
etl_billiards/gui/widgets/__init__.py
Normal file
18
etl_billiards/gui/widgets/__init__.py
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""GUI 组件模块"""
|
||||||
|
|
||||||
|
from .task_panel import TaskPanel
|
||||||
|
from .env_editor import EnvEditor
|
||||||
|
from .log_viewer import LogViewer
|
||||||
|
from .db_viewer import DBViewer
|
||||||
|
from .status_panel import StatusPanel
|
||||||
|
from .task_manager import TaskManager
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"TaskPanel",
|
||||||
|
"EnvEditor",
|
||||||
|
"LogViewer",
|
||||||
|
"DBViewer",
|
||||||
|
"StatusPanel",
|
||||||
|
"TaskManager",
|
||||||
|
]
|
||||||
390
etl_billiards/gui/widgets/db_viewer.py
Normal file
390
etl_billiards/gui/widgets/db_viewer.py
Normal file
@@ -0,0 +1,390 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""数据库查看器"""
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QWidget, QVBoxLayout, QHBoxLayout, QSplitter,
|
||||||
|
QGroupBox, QLabel, QPushButton, QLineEdit, QPlainTextEdit,
|
||||||
|
QTableWidget, QTableWidgetItem, QTreeWidget, QTreeWidgetItem,
|
||||||
|
QHeaderView, QComboBox, QTabWidget, QMessageBox, QFrame
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt, Signal
|
||||||
|
from PySide6.QtGui import QFont
|
||||||
|
|
||||||
|
from ..workers.db_worker import DBWorker
|
||||||
|
from ..utils.config_helper import ConfigHelper
|
||||||
|
|
||||||
|
|
||||||
|
# 常用查询模板
|
||||||
|
QUERY_TEMPLATES = {
|
||||||
|
"ODS 行数统计": """
|
||||||
|
SELECT
|
||||||
|
table_name,
|
||||||
|
(xpath('/row/cnt/text()',
|
||||||
|
query_to_xml('SELECT COUNT(*) AS cnt FROM ' || table_schema || '.' || table_name, false, false, ''))
|
||||||
|
)[1]::text::bigint AS row_count
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_schema = 'billiards_ods'
|
||||||
|
ORDER BY table_name;
|
||||||
|
""",
|
||||||
|
"DWD 行数统计": """
|
||||||
|
SELECT
|
||||||
|
table_name,
|
||||||
|
(xpath('/row/cnt/text()',
|
||||||
|
query_to_xml('SELECT COUNT(*) AS cnt FROM ' || table_schema || '.' || table_name, false, false, ''))
|
||||||
|
)[1]::text::bigint AS row_count
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_schema = 'billiards_dwd'
|
||||||
|
ORDER BY table_name;
|
||||||
|
""",
|
||||||
|
"ETL 游标状态": """
|
||||||
|
SELECT
|
||||||
|
task_code,
|
||||||
|
last_start,
|
||||||
|
last_end,
|
||||||
|
last_run_id,
|
||||||
|
updated_at
|
||||||
|
FROM etl_admin.etl_cursor
|
||||||
|
ORDER BY task_code;
|
||||||
|
""",
|
||||||
|
"最近运行记录": """
|
||||||
|
SELECT
|
||||||
|
run_id,
|
||||||
|
task_code,
|
||||||
|
status,
|
||||||
|
started_at,
|
||||||
|
finished_at,
|
||||||
|
EXTRACT(EPOCH FROM (finished_at - started_at))::int AS duration_sec,
|
||||||
|
rows_affected
|
||||||
|
FROM etl_admin.run_tracker
|
||||||
|
ORDER BY started_at DESC
|
||||||
|
LIMIT 50;
|
||||||
|
""",
|
||||||
|
"ODS 最新入库时间": """
|
||||||
|
SELECT
|
||||||
|
'payment_transactions' AS table_name, MAX(fetched_at) AS max_fetched_at FROM billiards_ods.payment_transactions
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'member_profiles', MAX(fetched_at) FROM billiards_ods.member_profiles
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'settlement_records', MAX(fetched_at) FROM billiards_ods.settlement_records
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'recharge_settlements', MAX(fetched_at) FROM billiards_ods.recharge_settlements
|
||||||
|
ORDER BY table_name;
|
||||||
|
""",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class DBViewer(QWidget):
|
||||||
|
"""数据库查看器"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
connection_changed = Signal(bool, str) # 连接状态变化
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.config_helper = ConfigHelper()
|
||||||
|
self.db_worker = DBWorker(self)
|
||||||
|
self._connected = False
|
||||||
|
|
||||||
|
self._init_ui()
|
||||||
|
self._connect_signals()
|
||||||
|
self._load_dsn_from_env()
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
"""初始化界面"""
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setContentsMargins(16, 16, 16, 16)
|
||||||
|
layout.setSpacing(16)
|
||||||
|
|
||||||
|
# 标题
|
||||||
|
title = QLabel("数据库查看器")
|
||||||
|
title.setProperty("heading", True)
|
||||||
|
layout.addWidget(title)
|
||||||
|
|
||||||
|
# 连接配置
|
||||||
|
conn_group = QGroupBox("数据库连接")
|
||||||
|
conn_layout = QHBoxLayout(conn_group)
|
||||||
|
|
||||||
|
conn_layout.addWidget(QLabel("DSN:"))
|
||||||
|
self.dsn_edit = QLineEdit()
|
||||||
|
self.dsn_edit.setPlaceholderText("postgresql://user:password@host:5432/dbname")
|
||||||
|
self.dsn_edit.setEchoMode(QLineEdit.Password)
|
||||||
|
conn_layout.addWidget(self.dsn_edit, 1)
|
||||||
|
|
||||||
|
self.show_dsn_btn = QPushButton("显示")
|
||||||
|
self.show_dsn_btn.setProperty("secondary", True)
|
||||||
|
self.show_dsn_btn.setCheckable(True)
|
||||||
|
self.show_dsn_btn.setFixedWidth(60)
|
||||||
|
conn_layout.addWidget(self.show_dsn_btn)
|
||||||
|
|
||||||
|
self.connect_btn = QPushButton("连接")
|
||||||
|
self.connect_btn.setFixedWidth(80)
|
||||||
|
conn_layout.addWidget(self.connect_btn)
|
||||||
|
|
||||||
|
self.disconnect_btn = QPushButton("断开")
|
||||||
|
self.disconnect_btn.setProperty("secondary", True)
|
||||||
|
self.disconnect_btn.setFixedWidth(80)
|
||||||
|
self.disconnect_btn.setEnabled(False)
|
||||||
|
conn_layout.addWidget(self.disconnect_btn)
|
||||||
|
|
||||||
|
layout.addWidget(conn_group)
|
||||||
|
|
||||||
|
# 主分割器
|
||||||
|
main_splitter = QSplitter(Qt.Horizontal)
|
||||||
|
layout.addWidget(main_splitter, 1)
|
||||||
|
|
||||||
|
# 左侧:表浏览器
|
||||||
|
left_widget = self._create_table_browser()
|
||||||
|
main_splitter.addWidget(left_widget)
|
||||||
|
|
||||||
|
# 右侧:查询和结果
|
||||||
|
right_widget = self._create_query_area()
|
||||||
|
main_splitter.addWidget(right_widget)
|
||||||
|
|
||||||
|
# 设置分割比例
|
||||||
|
main_splitter.setSizes([300, 700])
|
||||||
|
|
||||||
|
def _create_table_browser(self) -> QWidget:
|
||||||
|
"""创建表浏览器"""
|
||||||
|
widget = QWidget()
|
||||||
|
layout = QVBoxLayout(widget)
|
||||||
|
layout.setContentsMargins(0, 0, 8, 0)
|
||||||
|
|
||||||
|
# 标题和刷新按钮
|
||||||
|
header_layout = QHBoxLayout()
|
||||||
|
header_layout.addWidget(QLabel("表结构"))
|
||||||
|
self.refresh_tables_btn = QPushButton("刷新")
|
||||||
|
self.refresh_tables_btn.setProperty("secondary", True)
|
||||||
|
self.refresh_tables_btn.setEnabled(False)
|
||||||
|
header_layout.addWidget(self.refresh_tables_btn)
|
||||||
|
layout.addLayout(header_layout)
|
||||||
|
|
||||||
|
# 表树形视图
|
||||||
|
self.table_tree = QTreeWidget()
|
||||||
|
self.table_tree.setHeaderLabels(["名称", "行数", "最后更新"])
|
||||||
|
self.table_tree.header().setSectionResizeMode(0, QHeaderView.Stretch)
|
||||||
|
self.table_tree.setColumnWidth(1, 80)
|
||||||
|
self.table_tree.setColumnWidth(2, 130)
|
||||||
|
layout.addWidget(self.table_tree, 1)
|
||||||
|
|
||||||
|
return widget
|
||||||
|
|
||||||
|
def _create_query_area(self) -> QWidget:
|
||||||
|
"""创建查询区域"""
|
||||||
|
widget = QWidget()
|
||||||
|
layout = QVBoxLayout(widget)
|
||||||
|
layout.setContentsMargins(8, 0, 0, 0)
|
||||||
|
|
||||||
|
# 查询输入区
|
||||||
|
query_group = QGroupBox("SQL 查询")
|
||||||
|
query_layout = QVBoxLayout(query_group)
|
||||||
|
|
||||||
|
# 模板选择
|
||||||
|
template_layout = QHBoxLayout()
|
||||||
|
template_layout.addWidget(QLabel("常用查询:"))
|
||||||
|
self.template_combo = QComboBox()
|
||||||
|
self.template_combo.addItem("-- 选择模板 --")
|
||||||
|
for name in QUERY_TEMPLATES.keys():
|
||||||
|
self.template_combo.addItem(name)
|
||||||
|
template_layout.addWidget(self.template_combo, 1)
|
||||||
|
query_layout.addLayout(template_layout)
|
||||||
|
|
||||||
|
# SQL 编辑器
|
||||||
|
self.sql_editor = QPlainTextEdit()
|
||||||
|
self.sql_editor.setObjectName("sqlEditor")
|
||||||
|
self.sql_editor.setPlaceholderText("输入 SQL 查询语句...")
|
||||||
|
self.sql_editor.setFont(QFont("Consolas", 11))
|
||||||
|
self.sql_editor.setMaximumHeight(150)
|
||||||
|
query_layout.addWidget(self.sql_editor)
|
||||||
|
|
||||||
|
# 执行按钮
|
||||||
|
exec_layout = QHBoxLayout()
|
||||||
|
exec_layout.addStretch()
|
||||||
|
|
||||||
|
self.exec_btn = QPushButton("执行查询 (Ctrl+Enter)")
|
||||||
|
self.exec_btn.setEnabled(False)
|
||||||
|
exec_layout.addWidget(self.exec_btn)
|
||||||
|
|
||||||
|
query_layout.addLayout(exec_layout)
|
||||||
|
layout.addWidget(query_group)
|
||||||
|
|
||||||
|
# 结果区域
|
||||||
|
result_group = QGroupBox("查询结果")
|
||||||
|
result_layout = QVBoxLayout(result_group)
|
||||||
|
|
||||||
|
# 结果表格
|
||||||
|
self.result_table = QTableWidget()
|
||||||
|
self.result_table.setAlternatingRowColors(True)
|
||||||
|
self.result_table.horizontalHeader().setStretchLastSection(True)
|
||||||
|
result_layout.addWidget(self.result_table, 1)
|
||||||
|
|
||||||
|
# 结果统计
|
||||||
|
self.result_label = QLabel("就绪")
|
||||||
|
self.result_label.setProperty("subheading", True)
|
||||||
|
result_layout.addWidget(self.result_label)
|
||||||
|
|
||||||
|
layout.addWidget(result_group, 1)
|
||||||
|
|
||||||
|
return widget
|
||||||
|
|
||||||
|
def _connect_signals(self):
|
||||||
|
"""连接信号"""
|
||||||
|
# 连接按钮
|
||||||
|
self.show_dsn_btn.toggled.connect(self._toggle_dsn_visibility)
|
||||||
|
self.connect_btn.clicked.connect(self._connect_db)
|
||||||
|
self.disconnect_btn.clicked.connect(self._disconnect_db)
|
||||||
|
self.refresh_tables_btn.clicked.connect(self._refresh_tables)
|
||||||
|
|
||||||
|
# 模板选择
|
||||||
|
self.template_combo.currentIndexChanged.connect(self._on_template_selected)
|
||||||
|
|
||||||
|
# 执行查询
|
||||||
|
self.exec_btn.clicked.connect(self._execute_query)
|
||||||
|
|
||||||
|
# 表双击
|
||||||
|
self.table_tree.itemDoubleClicked.connect(self._on_table_double_clicked)
|
||||||
|
|
||||||
|
# 工作线程信号
|
||||||
|
self.db_worker.connection_status.connect(self._on_connection_status)
|
||||||
|
self.db_worker.tables_loaded.connect(self._on_tables_loaded)
|
||||||
|
self.db_worker.query_finished.connect(self._on_query_finished)
|
||||||
|
self.db_worker.query_error.connect(self._on_query_error)
|
||||||
|
|
||||||
|
def _load_dsn_from_env(self):
|
||||||
|
"""从环境变量加载 DSN"""
|
||||||
|
env_vars = self.config_helper.load_env()
|
||||||
|
dsn = env_vars.get("PG_DSN", "")
|
||||||
|
if dsn:
|
||||||
|
self.dsn_edit.setText(dsn)
|
||||||
|
|
||||||
|
def _toggle_dsn_visibility(self, checked: bool):
|
||||||
|
"""切换 DSN 可见性"""
|
||||||
|
self.dsn_edit.setEchoMode(
|
||||||
|
QLineEdit.Normal if checked else QLineEdit.Password
|
||||||
|
)
|
||||||
|
self.show_dsn_btn.setText("隐藏" if checked else "显示")
|
||||||
|
|
||||||
|
def _connect_db(self):
|
||||||
|
"""连接数据库"""
|
||||||
|
dsn = self.dsn_edit.text().strip()
|
||||||
|
if not dsn:
|
||||||
|
QMessageBox.warning(self, "提示", "请输入数据库连接字符串")
|
||||||
|
return
|
||||||
|
|
||||||
|
self.connect_btn.setEnabled(False)
|
||||||
|
self.connect_btn.setText("连接中...")
|
||||||
|
self.db_worker.connect_db(dsn)
|
||||||
|
|
||||||
|
def _disconnect_db(self):
|
||||||
|
"""断开数据库连接"""
|
||||||
|
self.db_worker.disconnect_db()
|
||||||
|
|
||||||
|
def _refresh_tables(self):
|
||||||
|
"""刷新表列表"""
|
||||||
|
self.db_worker.load_tables()
|
||||||
|
|
||||||
|
def _on_connection_status(self, connected: bool, message: str):
|
||||||
|
"""处理连接状态变化"""
|
||||||
|
self._connected = connected
|
||||||
|
self.connect_btn.setEnabled(not connected)
|
||||||
|
self.connect_btn.setText("连接")
|
||||||
|
self.disconnect_btn.setEnabled(connected)
|
||||||
|
self.refresh_tables_btn.setEnabled(connected)
|
||||||
|
self.exec_btn.setEnabled(connected)
|
||||||
|
|
||||||
|
self.connection_changed.emit(connected, message)
|
||||||
|
|
||||||
|
if connected:
|
||||||
|
# 自动加载表列表
|
||||||
|
self._refresh_tables()
|
||||||
|
|
||||||
|
def _on_tables_loaded(self, tables_dict: dict):
|
||||||
|
"""处理表列表加载完成"""
|
||||||
|
self.table_tree.clear()
|
||||||
|
|
||||||
|
for schema, tables in tables_dict.items():
|
||||||
|
schema_item = QTreeWidgetItem([schema, "", ""])
|
||||||
|
schema_item.setExpanded(True)
|
||||||
|
|
||||||
|
for table_name, row_count, updated_at in tables:
|
||||||
|
table_item = QTreeWidgetItem([table_name, str(row_count), updated_at])
|
||||||
|
table_item.setData(0, Qt.UserRole, f"{schema}.{table_name}")
|
||||||
|
schema_item.addChild(table_item)
|
||||||
|
|
||||||
|
self.table_tree.addTopLevelItem(schema_item)
|
||||||
|
|
||||||
|
def _on_template_selected(self, index: int):
|
||||||
|
"""模板选择变化"""
|
||||||
|
if index <= 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
template_name = self.template_combo.currentText()
|
||||||
|
if template_name in QUERY_TEMPLATES:
|
||||||
|
self.sql_editor.setPlainText(QUERY_TEMPLATES[template_name].strip())
|
||||||
|
|
||||||
|
# 重置选择
|
||||||
|
self.template_combo.setCurrentIndex(0)
|
||||||
|
|
||||||
|
def _on_table_double_clicked(self, item: QTreeWidgetItem, column: int):
|
||||||
|
"""表双击事件"""
|
||||||
|
full_name = item.data(0, Qt.UserRole)
|
||||||
|
if full_name:
|
||||||
|
# 生成预览查询
|
||||||
|
sql = f"SELECT * FROM {full_name} LIMIT 100;"
|
||||||
|
self.sql_editor.setPlainText(sql)
|
||||||
|
self._execute_query()
|
||||||
|
|
||||||
|
def _execute_query(self):
|
||||||
|
"""执行查询"""
|
||||||
|
sql = self.sql_editor.toPlainText().strip()
|
||||||
|
if not sql:
|
||||||
|
QMessageBox.warning(self, "提示", "请输入 SQL 语句")
|
||||||
|
return
|
||||||
|
|
||||||
|
self.exec_btn.setEnabled(False)
|
||||||
|
self.exec_btn.setText("执行中...")
|
||||||
|
self.result_label.setText("正在查询...")
|
||||||
|
|
||||||
|
self.db_worker.execute_query(sql)
|
||||||
|
|
||||||
|
def _on_query_finished(self, columns: list, rows: list):
|
||||||
|
"""查询完成"""
|
||||||
|
self.exec_btn.setEnabled(True)
|
||||||
|
self.exec_btn.setText("执行查询 (Ctrl+Enter)")
|
||||||
|
|
||||||
|
# 更新结果表格
|
||||||
|
self.result_table.clear()
|
||||||
|
self.result_table.setColumnCount(len(columns))
|
||||||
|
self.result_table.setRowCount(len(rows))
|
||||||
|
self.result_table.setHorizontalHeaderLabels(columns)
|
||||||
|
|
||||||
|
for row_idx, row_data in enumerate(rows):
|
||||||
|
for col_idx, col_name in enumerate(columns):
|
||||||
|
value = row_data.get(col_name, "")
|
||||||
|
item = QTableWidgetItem(str(value) if value is not None else "NULL")
|
||||||
|
if value is None:
|
||||||
|
item.setForeground(Qt.gray)
|
||||||
|
self.result_table.setItem(row_idx, col_idx, item)
|
||||||
|
|
||||||
|
# 更新统计
|
||||||
|
self.result_label.setText(f"返回 {len(rows)} 行, {len(columns)} 列")
|
||||||
|
|
||||||
|
def _on_query_error(self, error: str):
|
||||||
|
"""查询错误"""
|
||||||
|
self.exec_btn.setEnabled(True)
|
||||||
|
self.exec_btn.setText("执行查询 (Ctrl+Enter)")
|
||||||
|
self.result_label.setText(f"错误: {error}")
|
||||||
|
QMessageBox.critical(self, "查询错误", error)
|
||||||
|
|
||||||
|
def close_connection(self):
|
||||||
|
"""关闭连接"""
|
||||||
|
if self._connected:
|
||||||
|
self.db_worker.disconnect_db()
|
||||||
|
|
||||||
|
def keyPressEvent(self, event):
|
||||||
|
"""键盘事件"""
|
||||||
|
# Ctrl+Enter 执行查询
|
||||||
|
if event.modifiers() == Qt.ControlModifier and event.key() == Qt.Key_Return:
|
||||||
|
if self._connected:
|
||||||
|
self._execute_query()
|
||||||
|
else:
|
||||||
|
super().keyPressEvent(event)
|
||||||
318
etl_billiards/gui/widgets/env_editor.py
Normal file
318
etl_billiards/gui/widgets/env_editor.py
Normal file
@@ -0,0 +1,318 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""环境变量编辑器"""
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QWidget, QVBoxLayout, QHBoxLayout, QGridLayout,
|
||||||
|
QGroupBox, QLabel, QLineEdit, QPushButton, QScrollArea,
|
||||||
|
QFrame, QMessageBox, QFileDialog, QCheckBox
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt, Signal
|
||||||
|
|
||||||
|
from ..utils.config_helper import ConfigHelper, ENV_GROUPS
|
||||||
|
|
||||||
|
|
||||||
|
class EnvEditor(QWidget):
|
||||||
|
"""环境变量编辑器"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
config_saved = Signal() # 配置保存成功
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.config_helper = ConfigHelper()
|
||||||
|
self.field_widgets = {} # 存储字段控件
|
||||||
|
self.show_sensitive = False
|
||||||
|
|
||||||
|
self._init_ui()
|
||||||
|
self.load_config()
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
"""初始化界面"""
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setContentsMargins(16, 16, 16, 16)
|
||||||
|
layout.setSpacing(16)
|
||||||
|
|
||||||
|
# 标题和按钮
|
||||||
|
header_layout = QHBoxLayout()
|
||||||
|
|
||||||
|
title = QLabel("环境配置")
|
||||||
|
title.setProperty("heading", True)
|
||||||
|
header_layout.addWidget(title)
|
||||||
|
|
||||||
|
header_layout.addStretch()
|
||||||
|
|
||||||
|
self.show_sensitive_check = QCheckBox("显示敏感信息")
|
||||||
|
self.show_sensitive_check.stateChanged.connect(self._toggle_sensitive)
|
||||||
|
header_layout.addWidget(self.show_sensitive_check)
|
||||||
|
|
||||||
|
self.import_btn = QPushButton("导入")
|
||||||
|
self.import_btn.setProperty("secondary", True)
|
||||||
|
self.import_btn.clicked.connect(self._import_config)
|
||||||
|
header_layout.addWidget(self.import_btn)
|
||||||
|
|
||||||
|
self.export_btn = QPushButton("导出")
|
||||||
|
self.export_btn.setProperty("secondary", True)
|
||||||
|
self.export_btn.clicked.connect(self._export_config)
|
||||||
|
header_layout.addWidget(self.export_btn)
|
||||||
|
|
||||||
|
self.reload_btn = QPushButton("重新加载")
|
||||||
|
self.reload_btn.setProperty("secondary", True)
|
||||||
|
self.reload_btn.clicked.connect(self.load_config)
|
||||||
|
header_layout.addWidget(self.reload_btn)
|
||||||
|
|
||||||
|
self.save_btn = QPushButton("保存")
|
||||||
|
self.save_btn.clicked.connect(self._save_config)
|
||||||
|
header_layout.addWidget(self.save_btn)
|
||||||
|
|
||||||
|
layout.addLayout(header_layout)
|
||||||
|
|
||||||
|
# 配置文件路径
|
||||||
|
path_layout = QHBoxLayout()
|
||||||
|
path_layout.addWidget(QLabel("配置文件:"))
|
||||||
|
self.path_label = QLabel(str(self.config_helper.env_path))
|
||||||
|
self.path_label.setProperty("subheading", True)
|
||||||
|
path_layout.addWidget(self.path_label, 1)
|
||||||
|
layout.addLayout(path_layout)
|
||||||
|
|
||||||
|
# 滚动区域
|
||||||
|
scroll_area = QScrollArea()
|
||||||
|
scroll_area.setWidgetResizable(True)
|
||||||
|
scroll_area.setFrameShape(QFrame.NoFrame)
|
||||||
|
layout.addWidget(scroll_area, 1)
|
||||||
|
|
||||||
|
# 配置组容器
|
||||||
|
config_widget = QWidget()
|
||||||
|
self.config_layout = QVBoxLayout(config_widget)
|
||||||
|
self.config_layout.setSpacing(16)
|
||||||
|
|
||||||
|
# 创建各配置组
|
||||||
|
self._create_config_groups()
|
||||||
|
|
||||||
|
# 弹性空间
|
||||||
|
self.config_layout.addStretch()
|
||||||
|
|
||||||
|
scroll_area.setWidget(config_widget)
|
||||||
|
|
||||||
|
# 验证结果
|
||||||
|
self.validation_label = QLabel()
|
||||||
|
self.validation_label.setWordWrap(True)
|
||||||
|
layout.addWidget(self.validation_label)
|
||||||
|
|
||||||
|
def _create_config_groups(self):
|
||||||
|
"""创建配置分组"""
|
||||||
|
for group_id, group_info in ENV_GROUPS.items():
|
||||||
|
group = QGroupBox(group_info["title"])
|
||||||
|
grid_layout = QGridLayout(group)
|
||||||
|
|
||||||
|
for row, key in enumerate(group_info["keys"]):
|
||||||
|
# 标签
|
||||||
|
label = QLabel(f"{key}:")
|
||||||
|
label.setMinimumWidth(180)
|
||||||
|
grid_layout.addWidget(label, row, 0)
|
||||||
|
|
||||||
|
# 输入框
|
||||||
|
edit = QLineEdit()
|
||||||
|
edit.setPlaceholderText(self._get_placeholder(key))
|
||||||
|
|
||||||
|
# 敏感字段处理
|
||||||
|
if key in group_info.get("sensitive", []):
|
||||||
|
edit.setEchoMode(QLineEdit.Password)
|
||||||
|
edit.setProperty("sensitive", True)
|
||||||
|
|
||||||
|
edit.textChanged.connect(self._on_value_changed)
|
||||||
|
grid_layout.addWidget(edit, row, 1)
|
||||||
|
|
||||||
|
# 存储控件引用
|
||||||
|
self.field_widgets[key] = edit
|
||||||
|
|
||||||
|
self.config_layout.addWidget(group)
|
||||||
|
|
||||||
|
# 其他配置组(动态添加)
|
||||||
|
self.other_group = QGroupBox("其他配置")
|
||||||
|
self.other_layout = QGridLayout(self.other_group)
|
||||||
|
self.other_group.setVisible(False)
|
||||||
|
self.config_layout.addWidget(self.other_group)
|
||||||
|
|
||||||
|
def load_config(self):
|
||||||
|
"""加载配置"""
|
||||||
|
env_vars = self.config_helper.load_env()
|
||||||
|
|
||||||
|
# 更新已知字段
|
||||||
|
for key, edit in self.field_widgets.items():
|
||||||
|
value = env_vars.get(key, "")
|
||||||
|
edit.blockSignals(True)
|
||||||
|
edit.setText(value)
|
||||||
|
edit.blockSignals(False)
|
||||||
|
|
||||||
|
# 处理其他字段
|
||||||
|
known_keys = set(self.field_widgets.keys())
|
||||||
|
other_keys = [k for k in env_vars.keys() if k not in known_keys]
|
||||||
|
|
||||||
|
# 清除旧的其他字段
|
||||||
|
while self.other_layout.count():
|
||||||
|
item = self.other_layout.takeAt(0)
|
||||||
|
if item.widget():
|
||||||
|
item.widget().deleteLater()
|
||||||
|
|
||||||
|
# 添加其他字段
|
||||||
|
if other_keys:
|
||||||
|
self.other_group.setVisible(True)
|
||||||
|
for row, key in enumerate(sorted(other_keys)):
|
||||||
|
label = QLabel(f"{key}:")
|
||||||
|
self.other_layout.addWidget(label, row, 0)
|
||||||
|
|
||||||
|
edit = QLineEdit(env_vars[key])
|
||||||
|
edit.textChanged.connect(self._on_value_changed)
|
||||||
|
self.other_layout.addWidget(edit, row, 1)
|
||||||
|
|
||||||
|
self.field_widgets[key] = edit
|
||||||
|
else:
|
||||||
|
self.other_group.setVisible(False)
|
||||||
|
|
||||||
|
self._validate()
|
||||||
|
|
||||||
|
def _save_config(self):
|
||||||
|
"""保存配置"""
|
||||||
|
# 收集所有值
|
||||||
|
env_vars = {}
|
||||||
|
for key, edit in self.field_widgets.items():
|
||||||
|
value = edit.text().strip()
|
||||||
|
if value:
|
||||||
|
env_vars[key] = value
|
||||||
|
|
||||||
|
# 验证
|
||||||
|
errors = self.config_helper.validate_env(env_vars)
|
||||||
|
if errors:
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"验证警告",
|
||||||
|
"配置存在以下问题:\n\n" + "\n".join(f"• {e}" for e in errors) + "\n\n是否仍要保存?",
|
||||||
|
QMessageBox.Yes | QMessageBox.No,
|
||||||
|
QMessageBox.No
|
||||||
|
)
|
||||||
|
if reply == QMessageBox.No:
|
||||||
|
return
|
||||||
|
|
||||||
|
# 保存
|
||||||
|
if self.config_helper.save_env(env_vars):
|
||||||
|
QMessageBox.information(self, "成功", "配置已保存")
|
||||||
|
self.config_saved.emit()
|
||||||
|
else:
|
||||||
|
QMessageBox.critical(self, "错误", "保存配置失败")
|
||||||
|
|
||||||
|
def _import_config(self):
|
||||||
|
"""导入配置"""
|
||||||
|
file_path, _ = QFileDialog.getOpenFileName(
|
||||||
|
self,
|
||||||
|
"导入配置文件",
|
||||||
|
"",
|
||||||
|
"环境文件 (*.env);;所有文件 (*.*)"
|
||||||
|
)
|
||||||
|
if not file_path:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
from pathlib import Path
|
||||||
|
temp_helper = ConfigHelper(Path(file_path))
|
||||||
|
env_vars = temp_helper.load_env()
|
||||||
|
|
||||||
|
# 更新字段
|
||||||
|
for key, value in env_vars.items():
|
||||||
|
if key in self.field_widgets:
|
||||||
|
self.field_widgets[key].setText(value)
|
||||||
|
|
||||||
|
QMessageBox.information(self, "成功", f"已导入 {len(env_vars)} 个配置项")
|
||||||
|
except Exception as e:
|
||||||
|
QMessageBox.critical(self, "错误", f"导入失败: {e}")
|
||||||
|
|
||||||
|
def _export_config(self):
|
||||||
|
"""导出配置"""
|
||||||
|
file_path, _ = QFileDialog.getSaveFileName(
|
||||||
|
self,
|
||||||
|
"导出配置文件",
|
||||||
|
".env.backup",
|
||||||
|
"环境文件 (*.env);;所有文件 (*.*)"
|
||||||
|
)
|
||||||
|
if not file_path:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
# 收集当前值
|
||||||
|
env_vars = {}
|
||||||
|
for key, edit in self.field_widgets.items():
|
||||||
|
value = edit.text().strip()
|
||||||
|
if value:
|
||||||
|
env_vars[key] = value
|
||||||
|
|
||||||
|
# 保存到指定路径
|
||||||
|
temp_helper = ConfigHelper(Path(file_path))
|
||||||
|
if temp_helper.save_env(env_vars):
|
||||||
|
QMessageBox.information(self, "成功", f"配置已导出到:\n{file_path}")
|
||||||
|
else:
|
||||||
|
QMessageBox.critical(self, "错误", "导出失败")
|
||||||
|
except Exception as e:
|
||||||
|
QMessageBox.critical(self, "错误", f"导出失败: {e}")
|
||||||
|
|
||||||
|
def _toggle_sensitive(self, state: int):
|
||||||
|
"""切换敏感信息显示"""
|
||||||
|
self.show_sensitive = state == Qt.Checked
|
||||||
|
|
||||||
|
for key, edit in self.field_widgets.items():
|
||||||
|
if edit.property("sensitive"):
|
||||||
|
edit.setEchoMode(
|
||||||
|
QLineEdit.Normal if self.show_sensitive else QLineEdit.Password
|
||||||
|
)
|
||||||
|
|
||||||
|
def _on_value_changed(self):
|
||||||
|
"""值变化时验证"""
|
||||||
|
self._validate()
|
||||||
|
|
||||||
|
def _validate(self):
|
||||||
|
"""验证配置"""
|
||||||
|
env_vars = {}
|
||||||
|
for key, edit in self.field_widgets.items():
|
||||||
|
value = edit.text().strip()
|
||||||
|
if value:
|
||||||
|
env_vars[key] = value
|
||||||
|
|
||||||
|
errors = self.config_helper.validate_env(env_vars)
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
self.validation_label.setText("⚠ " + "; ".join(errors))
|
||||||
|
self.validation_label.setProperty("status", "warning")
|
||||||
|
else:
|
||||||
|
self.validation_label.setText("✓ 配置验证通过")
|
||||||
|
self.validation_label.setProperty("status", "success")
|
||||||
|
|
||||||
|
self.validation_label.style().unpolish(self.validation_label)
|
||||||
|
self.validation_label.style().polish(self.validation_label)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _get_placeholder(key: str) -> str:
|
||||||
|
"""获取占位符提示"""
|
||||||
|
placeholders = {
|
||||||
|
"PG_DSN": "postgresql://user:password@host:5432/dbname",
|
||||||
|
"PG_HOST": "localhost",
|
||||||
|
"PG_PORT": "5432",
|
||||||
|
"PG_NAME": "billiards",
|
||||||
|
"PG_USER": "postgres",
|
||||||
|
"PG_PASSWORD": "密码",
|
||||||
|
"API_BASE": "https://pc.ficoo.vip/apiprod/admin/v1",
|
||||||
|
"API_TOKEN": "Bearer token",
|
||||||
|
"API_TIMEOUT": "20",
|
||||||
|
"API_PAGE_SIZE": "200",
|
||||||
|
"STORE_ID": "门店ID (数字)",
|
||||||
|
"TIMEZONE": "Asia/Taipei",
|
||||||
|
"EXPORT_ROOT": "export/JSON",
|
||||||
|
"LOG_ROOT": "export/LOG",
|
||||||
|
"FETCH_ROOT": "JSON 抓取输出目录",
|
||||||
|
"INGEST_SOURCE_DIR": "本地 JSON 输入目录",
|
||||||
|
"PIPELINE_FLOW": "FULL / FETCH_ONLY / INGEST_ONLY",
|
||||||
|
"RUN_TASKS": "任务列表,逗号分隔",
|
||||||
|
"OVERLAP_SECONDS": "3600",
|
||||||
|
"WINDOW_START": "2025-07-01 00:00:00",
|
||||||
|
"WINDOW_END": "2025-08-01 00:00:00",
|
||||||
|
}
|
||||||
|
return placeholders.get(key, "")
|
||||||
247
etl_billiards/gui/widgets/log_viewer.py
Normal file
247
etl_billiards/gui/widgets/log_viewer.py
Normal file
@@ -0,0 +1,247 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""日志查看器"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QWidget, QVBoxLayout, QHBoxLayout,
|
||||||
|
QPlainTextEdit, QPushButton, QLineEdit, QLabel,
|
||||||
|
QComboBox, QCheckBox, QFileDialog, QMessageBox
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt, Signal, Slot
|
||||||
|
from PySide6.QtGui import QTextCharFormat, QColor, QFont, QTextCursor
|
||||||
|
|
||||||
|
|
||||||
|
class LogViewer(QWidget):
|
||||||
|
"""日志查看器"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
log_cleared = Signal()
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.max_lines = 10000
|
||||||
|
self.auto_scroll = True
|
||||||
|
self.filter_text = ""
|
||||||
|
self.filter_level = "ALL"
|
||||||
|
self._all_logs = [] # 存储所有日志
|
||||||
|
|
||||||
|
self._init_ui()
|
||||||
|
self._connect_signals()
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
"""初始化界面"""
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setContentsMargins(16, 16, 16, 16)
|
||||||
|
layout.setSpacing(8)
|
||||||
|
|
||||||
|
# 标题和工具栏
|
||||||
|
header_layout = QHBoxLayout()
|
||||||
|
|
||||||
|
title = QLabel("执行日志")
|
||||||
|
title.setProperty("heading", True)
|
||||||
|
header_layout.addWidget(title)
|
||||||
|
|
||||||
|
header_layout.addStretch()
|
||||||
|
|
||||||
|
# 日志级别过滤
|
||||||
|
header_layout.addWidget(QLabel("级别:"))
|
||||||
|
self.level_combo = QComboBox()
|
||||||
|
self.level_combo.addItems(["ALL", "INFO", "WARNING", "ERROR", "DEBUG"])
|
||||||
|
self.level_combo.setFixedWidth(100)
|
||||||
|
header_layout.addWidget(self.level_combo)
|
||||||
|
|
||||||
|
# 搜索框
|
||||||
|
header_layout.addWidget(QLabel("搜索:"))
|
||||||
|
self.search_edit = QLineEdit()
|
||||||
|
self.search_edit.setPlaceholderText("输入关键字...")
|
||||||
|
self.search_edit.setFixedWidth(200)
|
||||||
|
header_layout.addWidget(self.search_edit)
|
||||||
|
|
||||||
|
# 自动滚动
|
||||||
|
self.auto_scroll_check = QCheckBox("自动滚动")
|
||||||
|
self.auto_scroll_check.setChecked(True)
|
||||||
|
header_layout.addWidget(self.auto_scroll_check)
|
||||||
|
|
||||||
|
layout.addLayout(header_layout)
|
||||||
|
|
||||||
|
# 日志文本区域
|
||||||
|
self.log_text = QPlainTextEdit()
|
||||||
|
self.log_text.setObjectName("logViewer")
|
||||||
|
self.log_text.setReadOnly(True)
|
||||||
|
self.log_text.setFont(QFont("Consolas", 10))
|
||||||
|
self.log_text.setLineWrapMode(QPlainTextEdit.NoWrap)
|
||||||
|
layout.addWidget(self.log_text, 1)
|
||||||
|
|
||||||
|
# 底部工具栏
|
||||||
|
footer_layout = QHBoxLayout()
|
||||||
|
|
||||||
|
self.line_count_label = QLabel("0 行")
|
||||||
|
self.line_count_label.setProperty("subheading", True)
|
||||||
|
footer_layout.addWidget(self.line_count_label)
|
||||||
|
|
||||||
|
footer_layout.addStretch()
|
||||||
|
|
||||||
|
self.copy_btn = QPushButton("复制全部")
|
||||||
|
self.copy_btn.setProperty("secondary", True)
|
||||||
|
footer_layout.addWidget(self.copy_btn)
|
||||||
|
|
||||||
|
self.export_btn = QPushButton("导出")
|
||||||
|
self.export_btn.setProperty("secondary", True)
|
||||||
|
footer_layout.addWidget(self.export_btn)
|
||||||
|
|
||||||
|
self.clear_btn = QPushButton("清空")
|
||||||
|
self.clear_btn.setProperty("secondary", True)
|
||||||
|
footer_layout.addWidget(self.clear_btn)
|
||||||
|
|
||||||
|
layout.addLayout(footer_layout)
|
||||||
|
|
||||||
|
def _connect_signals(self):
|
||||||
|
"""连接信号"""
|
||||||
|
self.level_combo.currentTextChanged.connect(self._apply_filter)
|
||||||
|
self.search_edit.textChanged.connect(self._apply_filter)
|
||||||
|
self.auto_scroll_check.stateChanged.connect(self._toggle_auto_scroll)
|
||||||
|
self.copy_btn.clicked.connect(self._copy_all)
|
||||||
|
self.export_btn.clicked.connect(self._export_log)
|
||||||
|
self.clear_btn.clicked.connect(self._clear_log)
|
||||||
|
|
||||||
|
@Slot(str)
|
||||||
|
def append_log(self, text: str):
|
||||||
|
"""追加日志"""
|
||||||
|
# 添加时间戳(如果没有)
|
||||||
|
if not re.match(r'^\d{4}-\d{2}-\d{2}', text) and not text.startswith('['):
|
||||||
|
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
text = f"[{timestamp}] {text}"
|
||||||
|
|
||||||
|
# 存储到全部日志
|
||||||
|
self._all_logs.append(text)
|
||||||
|
|
||||||
|
# 限制日志行数
|
||||||
|
if len(self._all_logs) > self.max_lines:
|
||||||
|
self._all_logs = self._all_logs[-self.max_lines:]
|
||||||
|
|
||||||
|
# 检查是否通过过滤器
|
||||||
|
if self._matches_filter(text):
|
||||||
|
self._append_formatted_line(text)
|
||||||
|
|
||||||
|
# 更新行数
|
||||||
|
self._update_line_count()
|
||||||
|
|
||||||
|
def _append_formatted_line(self, text: str):
|
||||||
|
"""追加格式化的行"""
|
||||||
|
cursor = self.log_text.textCursor()
|
||||||
|
cursor.movePosition(QTextCursor.End)
|
||||||
|
|
||||||
|
# 设置格式
|
||||||
|
fmt = QTextCharFormat()
|
||||||
|
|
||||||
|
text_lower = text.lower()
|
||||||
|
if "[error]" in text_lower or "错误" in text or "失败" in text:
|
||||||
|
fmt.setForeground(QColor("#d93025"))
|
||||||
|
fmt.setFontWeight(QFont.Bold)
|
||||||
|
elif "[warning]" in text_lower or "警告" in text or "warn" in text_lower:
|
||||||
|
fmt.setForeground(QColor("#f9ab00"))
|
||||||
|
elif "[info]" in text_lower:
|
||||||
|
fmt.setForeground(QColor("#1a73e8"))
|
||||||
|
elif "[debug]" in text_lower:
|
||||||
|
fmt.setForeground(QColor("#9aa0a6"))
|
||||||
|
elif "[gui]" in text_lower:
|
||||||
|
fmt.setForeground(QColor("#1e8e3e"))
|
||||||
|
else:
|
||||||
|
fmt.setForeground(QColor("#333333"))
|
||||||
|
|
||||||
|
cursor.insertText(text + "\n", fmt)
|
||||||
|
|
||||||
|
# 自动滚动
|
||||||
|
if self.auto_scroll:
|
||||||
|
self.log_text.verticalScrollBar().setValue(
|
||||||
|
self.log_text.verticalScrollBar().maximum()
|
||||||
|
)
|
||||||
|
|
||||||
|
def _matches_filter(self, text: str) -> bool:
|
||||||
|
"""检查是否匹配过滤器"""
|
||||||
|
# 级别过滤
|
||||||
|
if self.filter_level != "ALL":
|
||||||
|
level_marker = f"[{self.filter_level}]"
|
||||||
|
if level_marker.lower() not in text.lower():
|
||||||
|
return False
|
||||||
|
|
||||||
|
# 文本过滤
|
||||||
|
if self.filter_text:
|
||||||
|
if self.filter_text.lower() not in text.lower():
|
||||||
|
return False
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
def _apply_filter(self):
|
||||||
|
"""应用过滤器"""
|
||||||
|
self.filter_level = self.level_combo.currentText()
|
||||||
|
self.filter_text = self.search_edit.text().strip()
|
||||||
|
|
||||||
|
# 重新显示日志
|
||||||
|
self.log_text.clear()
|
||||||
|
for line in self._all_logs:
|
||||||
|
if self._matches_filter(line):
|
||||||
|
self._append_formatted_line(line)
|
||||||
|
|
||||||
|
self._update_line_count()
|
||||||
|
|
||||||
|
def _toggle_auto_scroll(self, state: int):
|
||||||
|
"""切换自动滚动"""
|
||||||
|
self.auto_scroll = state == Qt.Checked
|
||||||
|
|
||||||
|
def _copy_all(self):
|
||||||
|
"""复制全部日志"""
|
||||||
|
from PySide6.QtWidgets import QApplication
|
||||||
|
text = self.log_text.toPlainText()
|
||||||
|
QApplication.clipboard().setText(text)
|
||||||
|
QMessageBox.information(self, "提示", "日志已复制到剪贴板")
|
||||||
|
|
||||||
|
def _export_log(self):
|
||||||
|
"""导出日志"""
|
||||||
|
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
default_name = f"etl_log_{timestamp}.txt"
|
||||||
|
|
||||||
|
file_path, _ = QFileDialog.getSaveFileName(
|
||||||
|
self,
|
||||||
|
"导出日志",
|
||||||
|
default_name,
|
||||||
|
"文本文件 (*.txt);;日志文件 (*.log);;所有文件 (*.*)"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not file_path:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(file_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(self.log_text.toPlainText())
|
||||||
|
QMessageBox.information(self, "成功", f"日志已导出到:\n{file_path}")
|
||||||
|
except Exception as e:
|
||||||
|
QMessageBox.critical(self, "错误", f"导出失败: {e}")
|
||||||
|
|
||||||
|
def _clear_log(self):
|
||||||
|
"""清空日志"""
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self,
|
||||||
|
"确认",
|
||||||
|
"确定要清空所有日志吗?",
|
||||||
|
QMessageBox.Yes | QMessageBox.No,
|
||||||
|
QMessageBox.No
|
||||||
|
)
|
||||||
|
|
||||||
|
if reply == QMessageBox.Yes:
|
||||||
|
self._all_logs.clear()
|
||||||
|
self.log_text.clear()
|
||||||
|
self._update_line_count()
|
||||||
|
self.log_cleared.emit()
|
||||||
|
|
||||||
|
def _update_line_count(self):
|
||||||
|
"""更新行数显示"""
|
||||||
|
visible_count = self.log_text.document().blockCount() - 1
|
||||||
|
total_count = len(self._all_logs)
|
||||||
|
|
||||||
|
if visible_count < total_count:
|
||||||
|
self.line_count_label.setText(f"{visible_count} / {total_count} 行")
|
||||||
|
else:
|
||||||
|
self.line_count_label.setText(f"{total_count} 行")
|
||||||
166
etl_billiards/gui/widgets/settings_dialog.py
Normal file
166
etl_billiards/gui/widgets/settings_dialog.py
Normal file
@@ -0,0 +1,166 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""应用程序设置对话框"""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QDialog, QVBoxLayout, QHBoxLayout, QGridLayout,
|
||||||
|
QGroupBox, QLabel, QLineEdit, QPushButton,
|
||||||
|
QFileDialog, QMessageBox, QDialogButtonBox
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt
|
||||||
|
|
||||||
|
from ..utils.app_settings import app_settings
|
||||||
|
|
||||||
|
|
||||||
|
class SettingsDialog(QDialog):
|
||||||
|
"""设置对话框"""
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.setWindowTitle("应用程序设置")
|
||||||
|
self.setMinimumWidth(600)
|
||||||
|
self._init_ui()
|
||||||
|
self._load_settings()
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
|
||||||
|
# ETL 项目路径
|
||||||
|
project_group = QGroupBox("ETL 项目配置")
|
||||||
|
project_layout = QGridLayout(project_group)
|
||||||
|
|
||||||
|
project_layout.addWidget(QLabel("ETL 项目路径:"), 0, 0)
|
||||||
|
self.project_path_edit = QLineEdit()
|
||||||
|
self.project_path_edit.setPlaceholderText("例: etl_billiards")
|
||||||
|
project_layout.addWidget(self.project_path_edit, 0, 1)
|
||||||
|
|
||||||
|
browse_project_btn = QPushButton("浏览...")
|
||||||
|
browse_project_btn.clicked.connect(self._browse_project_path)
|
||||||
|
project_layout.addWidget(browse_project_btn, 0, 2)
|
||||||
|
|
||||||
|
project_layout.addWidget(QLabel(".env 文件路径:"), 1, 0)
|
||||||
|
self.env_path_edit = QLineEdit()
|
||||||
|
self.env_path_edit.setPlaceholderText("例: etl_billiards/.env")
|
||||||
|
project_layout.addWidget(self.env_path_edit, 1, 1)
|
||||||
|
|
||||||
|
browse_env_btn = QPushButton("浏览...")
|
||||||
|
browse_env_btn.clicked.connect(self._browse_env_path)
|
||||||
|
project_layout.addWidget(browse_env_btn, 1, 2)
|
||||||
|
|
||||||
|
# 验证按钮
|
||||||
|
validate_btn = QPushButton("验证配置")
|
||||||
|
validate_btn.clicked.connect(self._validate_config)
|
||||||
|
project_layout.addWidget(validate_btn, 2, 1)
|
||||||
|
|
||||||
|
# 验证结果
|
||||||
|
self.validation_label = QLabel()
|
||||||
|
self.validation_label.setWordWrap(True)
|
||||||
|
project_layout.addWidget(self.validation_label, 3, 0, 1, 3)
|
||||||
|
|
||||||
|
layout.addWidget(project_group)
|
||||||
|
|
||||||
|
# 说明
|
||||||
|
note = QLabel(
|
||||||
|
"说明:\n"
|
||||||
|
"• ETL 项目路径:包含 cli/main.py 的目录\n"
|
||||||
|
"• .env 文件路径:环境变量配置文件\n"
|
||||||
|
"• 配置后才能正常执行 ETL 任务"
|
||||||
|
)
|
||||||
|
note.setProperty("subheading", True)
|
||||||
|
note.setWordWrap(True)
|
||||||
|
layout.addWidget(note)
|
||||||
|
|
||||||
|
layout.addStretch()
|
||||||
|
|
||||||
|
# 按钮
|
||||||
|
btn_box = QDialogButtonBox(QDialogButtonBox.Ok | QDialogButtonBox.Cancel)
|
||||||
|
btn_box.accepted.connect(self._save_and_accept)
|
||||||
|
btn_box.rejected.connect(self.reject)
|
||||||
|
layout.addWidget(btn_box)
|
||||||
|
|
||||||
|
def _load_settings(self):
|
||||||
|
"""加载设置"""
|
||||||
|
self.project_path_edit.setText(app_settings.etl_project_path)
|
||||||
|
self.env_path_edit.setText(app_settings.env_file_path)
|
||||||
|
self._validate_config()
|
||||||
|
|
||||||
|
def _browse_project_path(self):
|
||||||
|
"""浏览项目路径"""
|
||||||
|
path = QFileDialog.getExistingDirectory(
|
||||||
|
self, "选择 ETL 项目目录",
|
||||||
|
self.project_path_edit.text() or str(Path.home())
|
||||||
|
)
|
||||||
|
if path:
|
||||||
|
self.project_path_edit.setText(path)
|
||||||
|
# 自动填充 .env 路径
|
||||||
|
env_path = Path(path) / ".env"
|
||||||
|
if env_path.exists():
|
||||||
|
self.env_path_edit.setText(str(env_path))
|
||||||
|
self._validate_config()
|
||||||
|
|
||||||
|
def _browse_env_path(self):
|
||||||
|
"""浏览 .env 文件"""
|
||||||
|
path, _ = QFileDialog.getOpenFileName(
|
||||||
|
self, "选择 .env 文件",
|
||||||
|
self.env_path_edit.text() or str(Path.home()),
|
||||||
|
"环境变量文件 (*.env);;所有文件 (*.*)"
|
||||||
|
)
|
||||||
|
if path:
|
||||||
|
self.env_path_edit.setText(path)
|
||||||
|
self._validate_config()
|
||||||
|
|
||||||
|
def _validate_config(self):
|
||||||
|
"""验证配置"""
|
||||||
|
project_path = self.project_path_edit.text().strip()
|
||||||
|
env_path = self.env_path_edit.text().strip()
|
||||||
|
|
||||||
|
issues = []
|
||||||
|
|
||||||
|
if not project_path:
|
||||||
|
issues.append("• 未设置 ETL 项目路径")
|
||||||
|
else:
|
||||||
|
p = Path(project_path)
|
||||||
|
if not p.exists():
|
||||||
|
issues.append(f"• ETL 项目路径不存在")
|
||||||
|
elif not (p / "cli" / "main.py").exists():
|
||||||
|
issues.append(f"• 找不到 cli/main.py")
|
||||||
|
|
||||||
|
if not env_path:
|
||||||
|
issues.append("• 未设置 .env 文件路径")
|
||||||
|
elif not Path(env_path).exists():
|
||||||
|
issues.append("• .env 文件不存在")
|
||||||
|
|
||||||
|
if issues:
|
||||||
|
self.validation_label.setText("❌ 配置问题:\n" + "\n".join(issues))
|
||||||
|
self.validation_label.setStyleSheet("color: #d93025;")
|
||||||
|
else:
|
||||||
|
self.validation_label.setText("✅ 配置有效")
|
||||||
|
self.validation_label.setStyleSheet("color: #1e8e3e;")
|
||||||
|
|
||||||
|
def _save_and_accept(self):
|
||||||
|
"""保存并关闭"""
|
||||||
|
project_path = self.project_path_edit.text().strip()
|
||||||
|
env_path = self.env_path_edit.text().strip()
|
||||||
|
|
||||||
|
# 简单验证
|
||||||
|
if project_path:
|
||||||
|
p = Path(project_path)
|
||||||
|
if not p.exists():
|
||||||
|
QMessageBox.warning(self, "警告", "ETL 项目路径不存在")
|
||||||
|
return
|
||||||
|
if not (p / "cli" / "main.py").exists():
|
||||||
|
reply = QMessageBox.question(
|
||||||
|
self, "确认",
|
||||||
|
"找不到 cli/main.py,确定要使用此路径吗?",
|
||||||
|
QMessageBox.Yes | QMessageBox.No
|
||||||
|
)
|
||||||
|
if reply == QMessageBox.No:
|
||||||
|
return
|
||||||
|
|
||||||
|
# 保存设置
|
||||||
|
app_settings.etl_project_path = project_path
|
||||||
|
if env_path:
|
||||||
|
app_settings.env_file_path = env_path
|
||||||
|
|
||||||
|
self.accept()
|
||||||
406
etl_billiards/gui/widgets/status_panel.py
Normal file
406
etl_billiards/gui/widgets/status_panel.py
Normal file
@@ -0,0 +1,406 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""ETL 状态面板"""
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Dict, List, Optional, Any
|
||||||
|
|
||||||
|
from PySide6.QtWidgets import (
|
||||||
|
QWidget, QVBoxLayout, QHBoxLayout, QGridLayout,
|
||||||
|
QGroupBox, QLabel, QPushButton, QTableWidget, QTableWidgetItem,
|
||||||
|
QHeaderView, QFrame, QScrollArea, QMessageBox
|
||||||
|
)
|
||||||
|
from PySide6.QtCore import Qt, Signal, QTimer
|
||||||
|
from PySide6.QtGui import QColor
|
||||||
|
|
||||||
|
from ..workers.db_worker import DBWorker
|
||||||
|
from ..utils.config_helper import ConfigHelper
|
||||||
|
|
||||||
|
|
||||||
|
class StatusCard(QFrame):
|
||||||
|
"""状态卡片"""
|
||||||
|
|
||||||
|
def __init__(self, title: str, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.setProperty("card", True)
|
||||||
|
self.setFrameShape(QFrame.StyledPanel)
|
||||||
|
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setContentsMargins(16, 12, 16, 12)
|
||||||
|
layout.setSpacing(8)
|
||||||
|
|
||||||
|
# 标题
|
||||||
|
self.title_label = QLabel(title)
|
||||||
|
self.title_label.setProperty("subheading", True)
|
||||||
|
layout.addWidget(self.title_label)
|
||||||
|
|
||||||
|
# 值
|
||||||
|
self.value_label = QLabel("-")
|
||||||
|
self.value_label.setStyleSheet("font-size: 24px; font-weight: bold;")
|
||||||
|
layout.addWidget(self.value_label)
|
||||||
|
|
||||||
|
# 描述
|
||||||
|
self.desc_label = QLabel("")
|
||||||
|
self.desc_label.setProperty("subheading", True)
|
||||||
|
layout.addWidget(self.desc_label)
|
||||||
|
|
||||||
|
def set_value(self, value: str, description: str = "", status: str = ""):
|
||||||
|
"""设置值"""
|
||||||
|
self.value_label.setText(value)
|
||||||
|
self.desc_label.setText(description)
|
||||||
|
|
||||||
|
if status:
|
||||||
|
self.value_label.setProperty("status", status)
|
||||||
|
self.value_label.style().unpolish(self.value_label)
|
||||||
|
self.value_label.style().polish(self.value_label)
|
||||||
|
|
||||||
|
|
||||||
|
class StatusPanel(QWidget):
|
||||||
|
"""ETL 状态面板"""
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.config_helper = ConfigHelper()
|
||||||
|
self.db_worker = DBWorker(self)
|
||||||
|
self._connected = False
|
||||||
|
|
||||||
|
self._init_ui()
|
||||||
|
self._connect_signals()
|
||||||
|
|
||||||
|
# 定时刷新
|
||||||
|
self.refresh_timer = QTimer(self)
|
||||||
|
self.refresh_timer.timeout.connect(self._auto_refresh)
|
||||||
|
|
||||||
|
def _init_ui(self):
|
||||||
|
"""初始化界面"""
|
||||||
|
layout = QVBoxLayout(self)
|
||||||
|
layout.setContentsMargins(16, 16, 16, 16)
|
||||||
|
layout.setSpacing(16)
|
||||||
|
|
||||||
|
# 标题和按钮
|
||||||
|
header_layout = QHBoxLayout()
|
||||||
|
|
||||||
|
title = QLabel("ETL 状态")
|
||||||
|
title.setProperty("heading", True)
|
||||||
|
header_layout.addWidget(title)
|
||||||
|
|
||||||
|
header_layout.addStretch()
|
||||||
|
|
||||||
|
self.auto_refresh_btn = QPushButton("自动刷新: 关")
|
||||||
|
self.auto_refresh_btn.setProperty("secondary", True)
|
||||||
|
self.auto_refresh_btn.setCheckable(True)
|
||||||
|
header_layout.addWidget(self.auto_refresh_btn)
|
||||||
|
|
||||||
|
self.refresh_btn = QPushButton("刷新")
|
||||||
|
self.refresh_btn.clicked.connect(self._refresh_all)
|
||||||
|
header_layout.addWidget(self.refresh_btn)
|
||||||
|
|
||||||
|
layout.addLayout(header_layout)
|
||||||
|
|
||||||
|
# 连接状态
|
||||||
|
self.conn_status_label = QLabel("数据库: 未连接")
|
||||||
|
self.conn_status_label.setProperty("status", "warning")
|
||||||
|
layout.addWidget(self.conn_status_label)
|
||||||
|
|
||||||
|
# 滚动区域
|
||||||
|
scroll_area = QScrollArea()
|
||||||
|
scroll_area.setWidgetResizable(True)
|
||||||
|
scroll_area.setFrameShape(QFrame.NoFrame)
|
||||||
|
layout.addWidget(scroll_area, 1)
|
||||||
|
|
||||||
|
# 内容容器
|
||||||
|
content_widget = QWidget()
|
||||||
|
content_layout = QVBoxLayout(content_widget)
|
||||||
|
content_layout.setSpacing(16)
|
||||||
|
|
||||||
|
# 概览卡片
|
||||||
|
cards_layout = QHBoxLayout()
|
||||||
|
|
||||||
|
self.ods_card = StatusCard("ODS 表数量")
|
||||||
|
cards_layout.addWidget(self.ods_card)
|
||||||
|
|
||||||
|
self.dwd_card = StatusCard("DWD 表数量")
|
||||||
|
cards_layout.addWidget(self.dwd_card)
|
||||||
|
|
||||||
|
self.last_update_card = StatusCard("最后更新")
|
||||||
|
cards_layout.addWidget(self.last_update_card)
|
||||||
|
|
||||||
|
self.task_count_card = StatusCard("今日任务")
|
||||||
|
cards_layout.addWidget(self.task_count_card)
|
||||||
|
|
||||||
|
content_layout.addLayout(cards_layout)
|
||||||
|
|
||||||
|
# ODS Cutoff 状态
|
||||||
|
cutoff_group = QGroupBox("ODS Cutoff 状态")
|
||||||
|
cutoff_layout = QVBoxLayout(cutoff_group)
|
||||||
|
|
||||||
|
self.cutoff_table = QTableWidget()
|
||||||
|
self.cutoff_table.setColumnCount(4)
|
||||||
|
self.cutoff_table.setHorizontalHeaderLabels(["表名", "最新 fetched_at", "行数", "状态"])
|
||||||
|
self.cutoff_table.horizontalHeader().setSectionResizeMode(0, QHeaderView.Stretch)
|
||||||
|
self.cutoff_table.setMaximumHeight(250)
|
||||||
|
cutoff_layout.addWidget(self.cutoff_table)
|
||||||
|
|
||||||
|
content_layout.addWidget(cutoff_group)
|
||||||
|
|
||||||
|
# 最近运行记录
|
||||||
|
history_group = QGroupBox("最近运行记录")
|
||||||
|
history_layout = QVBoxLayout(history_group)
|
||||||
|
|
||||||
|
self.history_table = QTableWidget()
|
||||||
|
self.history_table.setColumnCount(6)
|
||||||
|
self.history_table.setHorizontalHeaderLabels(["运行ID", "任务", "状态", "开始时间", "耗时", "影响行数"])
|
||||||
|
self.history_table.horizontalHeader().setSectionResizeMode(1, QHeaderView.Stretch)
|
||||||
|
self.history_table.setMaximumHeight(250)
|
||||||
|
history_layout.addWidget(self.history_table)
|
||||||
|
|
||||||
|
content_layout.addWidget(history_group)
|
||||||
|
|
||||||
|
# 弹性空间
|
||||||
|
content_layout.addStretch()
|
||||||
|
|
||||||
|
scroll_area.setWidget(content_widget)
|
||||||
|
|
||||||
|
def _connect_signals(self):
|
||||||
|
"""连接信号"""
|
||||||
|
self.auto_refresh_btn.toggled.connect(self._toggle_auto_refresh)
|
||||||
|
self.db_worker.connection_status.connect(self._on_connection_status)
|
||||||
|
self.db_worker.query_finished.connect(self._on_query_finished)
|
||||||
|
self.db_worker.query_error.connect(self._on_query_error)
|
||||||
|
|
||||||
|
def _toggle_auto_refresh(self, checked: bool):
|
||||||
|
"""切换自动刷新"""
|
||||||
|
if checked:
|
||||||
|
self.auto_refresh_btn.setText("自动刷新: 开")
|
||||||
|
self.refresh_timer.start(30000) # 30秒刷新一次
|
||||||
|
self._refresh_all()
|
||||||
|
else:
|
||||||
|
self.auto_refresh_btn.setText("自动刷新: 关")
|
||||||
|
self.refresh_timer.stop()
|
||||||
|
|
||||||
|
def _auto_refresh(self):
|
||||||
|
"""自动刷新"""
|
||||||
|
if self._connected:
|
||||||
|
self._refresh_all()
|
||||||
|
|
||||||
|
def _refresh_all(self):
|
||||||
|
"""刷新所有数据"""
|
||||||
|
# 尝试连接数据库
|
||||||
|
if not self._connected:
|
||||||
|
env_vars = self.config_helper.load_env()
|
||||||
|
dsn = env_vars.get("PG_DSN", "")
|
||||||
|
if dsn:
|
||||||
|
self.db_worker.connect_db(dsn)
|
||||||
|
else:
|
||||||
|
self.conn_status_label.setText("数据库: 未配置 DSN")
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
self._load_status_data()
|
||||||
|
|
||||||
|
def _on_connection_status(self, connected: bool, message: str):
|
||||||
|
"""处理连接状态"""
|
||||||
|
self._connected = connected
|
||||||
|
|
||||||
|
if connected:
|
||||||
|
self.conn_status_label.setText(f"数据库: 已连接")
|
||||||
|
self.conn_status_label.setProperty("status", "success")
|
||||||
|
self._load_status_data()
|
||||||
|
else:
|
||||||
|
self.conn_status_label.setText(f"数据库: {message}")
|
||||||
|
self.conn_status_label.setProperty("status", "error")
|
||||||
|
|
||||||
|
self.conn_status_label.style().unpolish(self.conn_status_label)
|
||||||
|
self.conn_status_label.style().polish(self.conn_status_label)
|
||||||
|
|
||||||
|
def _load_status_data(self):
|
||||||
|
"""加载状态数据"""
|
||||||
|
# 加载表统计
|
||||||
|
self._current_query = "table_count"
|
||||||
|
self.db_worker.execute_query("""
|
||||||
|
SELECT
|
||||||
|
table_schema,
|
||||||
|
COUNT(*) as table_count
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_schema IN ('billiards_ods', 'billiards_dwd', 'billiards_dws')
|
||||||
|
GROUP BY table_schema
|
||||||
|
""")
|
||||||
|
|
||||||
|
def _on_query_finished(self, columns: list, rows: list):
|
||||||
|
"""处理查询结果"""
|
||||||
|
query_type = getattr(self, '_current_query', '')
|
||||||
|
|
||||||
|
if query_type == "table_count":
|
||||||
|
self._process_table_count(rows)
|
||||||
|
# 继续加载 cutoff 数据
|
||||||
|
self._current_query = "cutoff"
|
||||||
|
self.db_worker.execute_query("""
|
||||||
|
SELECT
|
||||||
|
'payment_transactions' AS table_name,
|
||||||
|
MAX(fetched_at) AS max_fetched_at,
|
||||||
|
COUNT(*) AS row_count
|
||||||
|
FROM billiards_ods.payment_transactions
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'member_profiles', MAX(fetched_at), COUNT(*)
|
||||||
|
FROM billiards_ods.member_profiles
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'settlement_records', MAX(fetched_at), COUNT(*)
|
||||||
|
FROM billiards_ods.settlement_records
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'recharge_settlements', MAX(fetched_at), COUNT(*)
|
||||||
|
FROM billiards_ods.recharge_settlements
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'assistant_service_records', MAX(fetched_at), COUNT(*)
|
||||||
|
FROM billiards_ods.assistant_service_records
|
||||||
|
ORDER BY table_name
|
||||||
|
""")
|
||||||
|
elif query_type == "cutoff":
|
||||||
|
self._process_cutoff_data(rows)
|
||||||
|
# 继续加载运行历史
|
||||||
|
self._current_query = "history"
|
||||||
|
self.db_worker.execute_query("""
|
||||||
|
SELECT
|
||||||
|
run_id,
|
||||||
|
task_code,
|
||||||
|
status,
|
||||||
|
started_at,
|
||||||
|
finished_at,
|
||||||
|
rows_affected
|
||||||
|
FROM etl_admin.run_tracker
|
||||||
|
ORDER BY started_at DESC
|
||||||
|
LIMIT 20
|
||||||
|
""")
|
||||||
|
elif query_type == "history":
|
||||||
|
self._process_history_data(rows)
|
||||||
|
self._current_query = ""
|
||||||
|
|
||||||
|
def _process_table_count(self, rows: list):
|
||||||
|
"""处理表数量数据"""
|
||||||
|
ods_count = 0
|
||||||
|
dwd_count = 0
|
||||||
|
|
||||||
|
for row in rows:
|
||||||
|
schema = row.get("table_schema", "")
|
||||||
|
count = row.get("table_count", 0)
|
||||||
|
|
||||||
|
if schema == "billiards_ods":
|
||||||
|
ods_count = count
|
||||||
|
elif schema == "billiards_dwd":
|
||||||
|
dwd_count = count
|
||||||
|
|
||||||
|
self.ods_card.set_value(str(ods_count), "个表")
|
||||||
|
self.dwd_card.set_value(str(dwd_count), "个表")
|
||||||
|
|
||||||
|
def _process_cutoff_data(self, rows: list):
|
||||||
|
"""处理 Cutoff 数据"""
|
||||||
|
self.cutoff_table.setRowCount(len(rows))
|
||||||
|
|
||||||
|
latest_time = None
|
||||||
|
now = datetime.now()
|
||||||
|
|
||||||
|
for row_idx, row in enumerate(rows):
|
||||||
|
table_name = row.get("table_name", "")
|
||||||
|
max_fetched = row.get("max_fetched_at")
|
||||||
|
row_count = row.get("row_count", 0)
|
||||||
|
|
||||||
|
self.cutoff_table.setItem(row_idx, 0, QTableWidgetItem(table_name))
|
||||||
|
|
||||||
|
if max_fetched:
|
||||||
|
time_str = str(max_fetched)[:19]
|
||||||
|
self.cutoff_table.setItem(row_idx, 1, QTableWidgetItem(time_str))
|
||||||
|
|
||||||
|
# 更新最新时间
|
||||||
|
if latest_time is None or max_fetched > latest_time:
|
||||||
|
latest_time = max_fetched
|
||||||
|
|
||||||
|
# 计算状态
|
||||||
|
if isinstance(max_fetched, datetime):
|
||||||
|
hours_ago = (now - max_fetched).total_seconds() / 3600
|
||||||
|
if hours_ago < 1:
|
||||||
|
status = "正常"
|
||||||
|
status_color = QColor("#1e8e3e")
|
||||||
|
elif hours_ago < 24:
|
||||||
|
status = "较新"
|
||||||
|
status_color = QColor("#1a73e8")
|
||||||
|
else:
|
||||||
|
status = f"落后 {int(hours_ago)}h"
|
||||||
|
status_color = QColor("#f9ab00")
|
||||||
|
else:
|
||||||
|
status = "-"
|
||||||
|
status_color = QColor("#9aa0a6")
|
||||||
|
else:
|
||||||
|
self.cutoff_table.setItem(row_idx, 1, QTableWidgetItem("-"))
|
||||||
|
status = "无数据"
|
||||||
|
status_color = QColor("#d93025")
|
||||||
|
|
||||||
|
self.cutoff_table.setItem(row_idx, 2, QTableWidgetItem(str(row_count)))
|
||||||
|
|
||||||
|
status_item = QTableWidgetItem(status)
|
||||||
|
status_item.setForeground(status_color)
|
||||||
|
self.cutoff_table.setItem(row_idx, 3, status_item)
|
||||||
|
|
||||||
|
# 更新最后更新时间卡片
|
||||||
|
if latest_time:
|
||||||
|
time_str = str(latest_time)[:16]
|
||||||
|
self.last_update_card.set_value(time_str, "")
|
||||||
|
else:
|
||||||
|
self.last_update_card.set_value("-", "无数据")
|
||||||
|
|
||||||
|
def _process_history_data(self, rows: list):
|
||||||
|
"""处理运行历史数据"""
|
||||||
|
self.history_table.setRowCount(len(rows))
|
||||||
|
|
||||||
|
today_count = 0
|
||||||
|
today = datetime.now().date()
|
||||||
|
|
||||||
|
for row_idx, row in enumerate(rows):
|
||||||
|
run_id = row.get("run_id", "")
|
||||||
|
task_code = row.get("task_code", "")
|
||||||
|
status = row.get("status", "")
|
||||||
|
started_at = row.get("started_at")
|
||||||
|
finished_at = row.get("finished_at")
|
||||||
|
rows_affected = row.get("rows_affected", 0)
|
||||||
|
|
||||||
|
# 统计今日任务
|
||||||
|
if started_at and isinstance(started_at, datetime):
|
||||||
|
if started_at.date() == today:
|
||||||
|
today_count += 1
|
||||||
|
|
||||||
|
self.history_table.setItem(row_idx, 0, QTableWidgetItem(str(run_id)[:8] if run_id else "-"))
|
||||||
|
self.history_table.setItem(row_idx, 1, QTableWidgetItem(task_code))
|
||||||
|
|
||||||
|
# 状态
|
||||||
|
status_item = QTableWidgetItem(status)
|
||||||
|
if status and "success" in status.lower():
|
||||||
|
status_item.setForeground(QColor("#1e8e3e"))
|
||||||
|
elif status and ("fail" in status.lower() or "error" in status.lower()):
|
||||||
|
status_item.setForeground(QColor("#d93025"))
|
||||||
|
self.history_table.setItem(row_idx, 2, status_item)
|
||||||
|
|
||||||
|
# 开始时间
|
||||||
|
time_str = str(started_at)[:19] if started_at else "-"
|
||||||
|
self.history_table.setItem(row_idx, 3, QTableWidgetItem(time_str))
|
||||||
|
|
||||||
|
# 耗时
|
||||||
|
if started_at and finished_at:
|
||||||
|
try:
|
||||||
|
duration = (finished_at - started_at).total_seconds()
|
||||||
|
if duration < 60:
|
||||||
|
duration_str = f"{duration:.1f}秒"
|
||||||
|
else:
|
||||||
|
duration_str = f"{int(duration // 60)}分{int(duration % 60)}秒"
|
||||||
|
except:
|
||||||
|
duration_str = "-"
|
||||||
|
else:
|
||||||
|
duration_str = "-"
|
||||||
|
self.history_table.setItem(row_idx, 4, QTableWidgetItem(duration_str))
|
||||||
|
|
||||||
|
# 影响行数
|
||||||
|
self.history_table.setItem(row_idx, 5, QTableWidgetItem(str(rows_affected or 0)))
|
||||||
|
|
||||||
|
# 更新今日任务卡片
|
||||||
|
self.task_count_card.set_value(str(today_count), "次执行")
|
||||||
|
|
||||||
|
def _on_query_error(self, error: str):
|
||||||
|
"""处理查询错误"""
|
||||||
|
self._current_query = ""
|
||||||
|
# 可能是表不存在,忽略错误继续
|
||||||
|
pass
|
||||||
1948
etl_billiards/gui/widgets/task_manager.py
Normal file
1948
etl_billiards/gui/widgets/task_manager.py
Normal file
File diff suppressed because it is too large
Load Diff
1061
etl_billiards/gui/widgets/task_panel.py
Normal file
1061
etl_billiards/gui/widgets/task_panel.py
Normal file
File diff suppressed because it is too large
Load Diff
7
etl_billiards/gui/workers/__init__.py
Normal file
7
etl_billiards/gui/workers/__init__.py
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""后台工作线程模块"""
|
||||||
|
|
||||||
|
from .task_worker import TaskWorker
|
||||||
|
from .db_worker import DBWorker
|
||||||
|
|
||||||
|
__all__ = ["TaskWorker", "DBWorker"]
|
||||||
192
etl_billiards/gui/workers/db_worker.py
Normal file
192
etl_billiards/gui/workers/db_worker.py
Normal file
@@ -0,0 +1,192 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""数据库查询工作线程"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Dict, Any, Optional, Tuple
|
||||||
|
|
||||||
|
from PySide6.QtCore import QThread, Signal
|
||||||
|
|
||||||
|
# 添加项目路径
|
||||||
|
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
if str(PROJECT_ROOT) not in sys.path:
|
||||||
|
sys.path.insert(0, str(PROJECT_ROOT))
|
||||||
|
|
||||||
|
|
||||||
|
class DBWorker(QThread):
|
||||||
|
"""数据库查询工作线程"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
query_finished = Signal(list, list) # 查询完成 (columns, rows)
|
||||||
|
query_error = Signal(str) # 查询错误
|
||||||
|
connection_status = Signal(bool, str) # 连接状态 (connected, message)
|
||||||
|
tables_loaded = Signal(dict) # 表列表加载完成 {schema: [(table, rows, updated_at), ...]}
|
||||||
|
|
||||||
|
def __init__(self, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.conn = None
|
||||||
|
self._task = None
|
||||||
|
self._task_args = None
|
||||||
|
|
||||||
|
def connect_db(self, dsn: str):
|
||||||
|
"""连接数据库"""
|
||||||
|
self._task = "connect"
|
||||||
|
self._task_args = (dsn,)
|
||||||
|
self.start()
|
||||||
|
|
||||||
|
def disconnect_db(self):
|
||||||
|
"""断开数据库连接"""
|
||||||
|
self._task = "disconnect"
|
||||||
|
self._task_args = None
|
||||||
|
self.start()
|
||||||
|
|
||||||
|
def execute_query(self, sql: str, params: Optional[tuple] = None):
|
||||||
|
"""执行查询"""
|
||||||
|
self._task = "query"
|
||||||
|
self._task_args = (sql, params)
|
||||||
|
self.start()
|
||||||
|
|
||||||
|
def load_tables(self, schemas: Optional[List[str]] = None):
|
||||||
|
"""加载表列表"""
|
||||||
|
self._task = "load_tables"
|
||||||
|
self._task_args = (schemas,)
|
||||||
|
self.start()
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""执行任务"""
|
||||||
|
if self._task == "connect":
|
||||||
|
self._do_connect(*self._task_args)
|
||||||
|
elif self._task == "disconnect":
|
||||||
|
self._do_disconnect()
|
||||||
|
elif self._task == "query":
|
||||||
|
self._do_query(*self._task_args)
|
||||||
|
elif self._task == "load_tables":
|
||||||
|
self._do_load_tables(*self._task_args)
|
||||||
|
|
||||||
|
def _do_connect(self, dsn: str):
|
||||||
|
"""执行连接"""
|
||||||
|
try:
|
||||||
|
import psycopg2
|
||||||
|
from psycopg2.extras import RealDictCursor
|
||||||
|
|
||||||
|
self.conn = psycopg2.connect(dsn, connect_timeout=10)
|
||||||
|
self.conn.set_session(autocommit=True)
|
||||||
|
|
||||||
|
# 测试连接
|
||||||
|
with self.conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT version()")
|
||||||
|
version = cur.fetchone()[0]
|
||||||
|
|
||||||
|
self.connection_status.emit(True, f"已连接: {version[:50]}...")
|
||||||
|
except ImportError:
|
||||||
|
self.connection_status.emit(False, "缺少 psycopg2 模块,请安装: pip install psycopg2-binary")
|
||||||
|
except Exception as e:
|
||||||
|
self.conn = None
|
||||||
|
self.connection_status.emit(False, f"连接失败: {e}")
|
||||||
|
|
||||||
|
def _do_disconnect(self):
|
||||||
|
"""执行断开连接"""
|
||||||
|
if self.conn:
|
||||||
|
try:
|
||||||
|
self.conn.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
self.conn = None
|
||||||
|
self.connection_status.emit(False, "已断开连接")
|
||||||
|
|
||||||
|
def _do_query(self, sql: str, params: Optional[tuple]):
|
||||||
|
"""执行查询"""
|
||||||
|
if not self.conn:
|
||||||
|
self.query_error.emit("未连接到数据库")
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
from psycopg2.extras import RealDictCursor
|
||||||
|
|
||||||
|
with self.conn.cursor(cursor_factory=RealDictCursor) as cur:
|
||||||
|
cur.execute(sql, params)
|
||||||
|
|
||||||
|
# 检查是否有结果
|
||||||
|
if cur.description:
|
||||||
|
columns = [desc[0] for desc in cur.description]
|
||||||
|
rows = [dict(row) for row in cur.fetchall()]
|
||||||
|
self.query_finished.emit(columns, rows)
|
||||||
|
else:
|
||||||
|
self.query_finished.emit([], [])
|
||||||
|
except Exception as e:
|
||||||
|
self.query_error.emit(f"查询失败: {e}")
|
||||||
|
|
||||||
|
def _do_load_tables(self, schemas: Optional[List[str]]):
|
||||||
|
"""加载表列表"""
|
||||||
|
if not self.conn:
|
||||||
|
self.query_error.emit("未连接到数据库")
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
if schemas is None:
|
||||||
|
schemas = ["billiards_ods", "billiards_dwd", "billiards_dws", "etl_admin"]
|
||||||
|
|
||||||
|
result = {}
|
||||||
|
|
||||||
|
for schema in schemas:
|
||||||
|
tables = []
|
||||||
|
|
||||||
|
# 获取表列表
|
||||||
|
sql = """
|
||||||
|
SELECT
|
||||||
|
t.table_name,
|
||||||
|
COALESCE(s.n_live_tup, 0) as row_count
|
||||||
|
FROM information_schema.tables t
|
||||||
|
LEFT JOIN pg_stat_user_tables s
|
||||||
|
ON t.table_name = s.relname
|
||||||
|
AND t.table_schema = s.schemaname
|
||||||
|
WHERE t.table_schema = %s
|
||||||
|
AND t.table_type = 'BASE TABLE'
|
||||||
|
ORDER BY t.table_name
|
||||||
|
"""
|
||||||
|
|
||||||
|
with self.conn.cursor() as cur:
|
||||||
|
cur.execute(sql, (schema,))
|
||||||
|
for row in cur.fetchall():
|
||||||
|
table_name = row[0]
|
||||||
|
row_count = row[1] or 0
|
||||||
|
|
||||||
|
# 尝试获取最新更新时间
|
||||||
|
updated_at = None
|
||||||
|
try:
|
||||||
|
# 尝试 fetched_at 字段
|
||||||
|
cur.execute(f'SELECT MAX(fetched_at) FROM "{schema}"."{table_name}"')
|
||||||
|
result_row = cur.fetchone()
|
||||||
|
if result_row and result_row[0]:
|
||||||
|
updated_at = str(result_row[0])[:19]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not updated_at:
|
||||||
|
try:
|
||||||
|
# 尝试 updated_at 字段
|
||||||
|
cur.execute(f'SELECT MAX(updated_at) FROM "{schema}"."{table_name}"')
|
||||||
|
result_row = cur.fetchone()
|
||||||
|
if result_row and result_row[0]:
|
||||||
|
updated_at = str(result_row[0])[:19]
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
tables.append((table_name, row_count, updated_at or "-"))
|
||||||
|
|
||||||
|
result[schema] = tables
|
||||||
|
|
||||||
|
self.tables_loaded.emit(result)
|
||||||
|
except Exception as e:
|
||||||
|
self.query_error.emit(f"加载表列表失败: {e}")
|
||||||
|
|
||||||
|
def is_connected(self) -> bool:
|
||||||
|
"""检查是否已连接"""
|
||||||
|
if not self.conn:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
with self.conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT 1")
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
317
etl_billiards/gui/workers/task_worker.py
Normal file
317
etl_billiards/gui/workers/task_worker.py
Normal file
@@ -0,0 +1,317 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""任务执行工作线程"""
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import List, Optional, Dict
|
||||||
|
|
||||||
|
from PySide6.QtCore import QThread, Signal
|
||||||
|
|
||||||
|
from ..utils.app_settings import app_settings
|
||||||
|
|
||||||
|
|
||||||
|
class TaskWorker(QThread):
|
||||||
|
"""任务执行工作线程"""
|
||||||
|
|
||||||
|
# 信号
|
||||||
|
output_received = Signal(str) # 收到输出行
|
||||||
|
task_finished = Signal(int, str) # 任务完成 (exit_code, summary)
|
||||||
|
error_occurred = Signal(str) # 发生错误
|
||||||
|
progress_updated = Signal(int, int) # 进度更新 (current, total)
|
||||||
|
|
||||||
|
def __init__(self, command: List[str], working_dir: Optional[str] = None,
|
||||||
|
extra_env: Optional[Dict[str, str]] = None, parent=None):
|
||||||
|
super().__init__(parent)
|
||||||
|
self.command = command
|
||||||
|
self.extra_env = extra_env or {}
|
||||||
|
|
||||||
|
# 工作目录优先级: 参数 > 应用设置 > 自动检测
|
||||||
|
if working_dir is not None:
|
||||||
|
self.working_dir = working_dir
|
||||||
|
elif app_settings.etl_project_path:
|
||||||
|
self.working_dir = app_settings.etl_project_path
|
||||||
|
else:
|
||||||
|
# 回退到源码目录
|
||||||
|
self.working_dir = str(Path(__file__).resolve().parents[2])
|
||||||
|
|
||||||
|
self.process: Optional[subprocess.Popen] = None
|
||||||
|
self._stop_requested = False
|
||||||
|
self._exit_code: Optional[int] = None
|
||||||
|
self._output_lines: List[str] = []
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""执行任务"""
|
||||||
|
try:
|
||||||
|
self._stop_requested = False
|
||||||
|
self._output_lines = []
|
||||||
|
|
||||||
|
# 设置环境变量
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PYTHONIOENCODING"] = "utf-8"
|
||||||
|
env["PYTHONUNBUFFERED"] = "1"
|
||||||
|
|
||||||
|
# 添加项目根目录到 PYTHONPATH
|
||||||
|
project_root = self.working_dir
|
||||||
|
existing_path = env.get("PYTHONPATH", "")
|
||||||
|
if existing_path:
|
||||||
|
env["PYTHONPATH"] = f"{project_root}{os.pathsep}{existing_path}"
|
||||||
|
else:
|
||||||
|
env["PYTHONPATH"] = project_root
|
||||||
|
|
||||||
|
# 添加额外的环境变量
|
||||||
|
if self.extra_env:
|
||||||
|
for key, value in self.extra_env.items():
|
||||||
|
env[key] = str(value)
|
||||||
|
self.output_received.emit(f"[环境变量] {key}={value}")
|
||||||
|
|
||||||
|
self.output_received.emit(f"[工作目录] {self.working_dir}")
|
||||||
|
self.output_received.emit(f"[执行命令] {' '.join(self.command)}")
|
||||||
|
|
||||||
|
# 启动进程
|
||||||
|
self.process = subprocess.Popen(
|
||||||
|
self.command,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
text=True,
|
||||||
|
encoding="utf-8",
|
||||||
|
errors="replace",
|
||||||
|
cwd=self.working_dir,
|
||||||
|
env=env,
|
||||||
|
creationflags=subprocess.CREATE_NO_WINDOW if sys.platform == "win32" else 0,
|
||||||
|
)
|
||||||
|
|
||||||
|
# 读取输出
|
||||||
|
if self.process.stdout:
|
||||||
|
for line in iter(self.process.stdout.readline, ""):
|
||||||
|
if self._stop_requested:
|
||||||
|
break
|
||||||
|
|
||||||
|
line = line.rstrip("\n\r")
|
||||||
|
if line:
|
||||||
|
self._output_lines.append(line)
|
||||||
|
self.output_received.emit(line)
|
||||||
|
|
||||||
|
# 解析进度信息(如果有)
|
||||||
|
self._parse_progress(line)
|
||||||
|
|
||||||
|
# 等待进程结束
|
||||||
|
if self.process:
|
||||||
|
self.process.wait()
|
||||||
|
self._exit_code = self.process.returncode
|
||||||
|
|
||||||
|
# 生成摘要
|
||||||
|
summary = self._generate_summary()
|
||||||
|
self.task_finished.emit(self._exit_code or 0, summary)
|
||||||
|
|
||||||
|
except FileNotFoundError as e:
|
||||||
|
self.error_occurred.emit(f"找不到 Python 解释器: {e}")
|
||||||
|
self.task_finished.emit(-1, f"执行失败: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
self.error_occurred.emit(f"执行出错: {e}")
|
||||||
|
self.task_finished.emit(-1, f"执行失败: {e}")
|
||||||
|
finally:
|
||||||
|
self.process = None
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
"""停止任务"""
|
||||||
|
self._stop_requested = True
|
||||||
|
if self.process:
|
||||||
|
try:
|
||||||
|
self.process.terminate()
|
||||||
|
# 给进程一些时间来终止
|
||||||
|
try:
|
||||||
|
self.process.wait(timeout=5)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
self.process.kill()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def _parse_progress(self, line: str):
|
||||||
|
"""解析进度信息"""
|
||||||
|
# 尝试从日志中解析进度
|
||||||
|
# 示例: "[INFO] 处理进度: 50/100"
|
||||||
|
import re
|
||||||
|
match = re.search(r'进度[:\s]*(\d+)/(\d+)', line)
|
||||||
|
if match:
|
||||||
|
current = int(match.group(1))
|
||||||
|
total = int(match.group(2))
|
||||||
|
self.progress_updated.emit(current, total)
|
||||||
|
|
||||||
|
def _generate_summary(self) -> str:
|
||||||
|
"""生成执行摘要"""
|
||||||
|
if not self._output_lines:
|
||||||
|
return "无输出"
|
||||||
|
|
||||||
|
return self._parse_detailed_summary()
|
||||||
|
|
||||||
|
def _parse_detailed_summary(self) -> str:
|
||||||
|
"""解析详细的执行摘要"""
|
||||||
|
import re
|
||||||
|
import json
|
||||||
|
|
||||||
|
summary_parts = []
|
||||||
|
|
||||||
|
# 统计各类信息
|
||||||
|
ods_stats = [] # ODS 抓取统计
|
||||||
|
dwd_stats = [] # DWD 装载统计
|
||||||
|
integrity_stats = {} # 数据校验统计
|
||||||
|
errors = [] # 错误信息
|
||||||
|
task_results = [] # 任务结果
|
||||||
|
|
||||||
|
for line in self._output_lines:
|
||||||
|
# 1. 解析 ODS 抓取完成信息
|
||||||
|
# 格式: "xxx: 抓取完成,文件=xxx,记录数=123"
|
||||||
|
match = re.search(r'(\w+): 抓取完成.*记录数[=:]\s*(\d+)', line)
|
||||||
|
if match:
|
||||||
|
task_name = match.group(1)
|
||||||
|
record_count = int(match.group(2))
|
||||||
|
if record_count > 0:
|
||||||
|
ods_stats.append(f"{task_name}: {record_count}条")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 2. 解析 DWD 装载完成信息
|
||||||
|
# 格式: "DWD 装载完成:xxx,用时 1.02s"
|
||||||
|
match = re.search(r'DWD 装载完成[::]\s*(\S+).*用时\s*([\d.]+)s', line)
|
||||||
|
if match:
|
||||||
|
table_name = match.group(1).replace('billiards_dwd.', '')
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 3. 解析任务完成统计 (JSON格式)
|
||||||
|
# 格式: "xxx: 完成,统计={'tables': [...]}"
|
||||||
|
if "完成,统计=" in line or "完成,统计=" in line:
|
||||||
|
try:
|
||||||
|
match = re.search(r"统计=(\{.+\})", line)
|
||||||
|
if match:
|
||||||
|
stats_str = match.group(1).replace("'", '"')
|
||||||
|
stats = json.loads(stats_str)
|
||||||
|
|
||||||
|
# 解析 DWD 装载统计
|
||||||
|
if 'tables' in stats:
|
||||||
|
total_processed = 0
|
||||||
|
total_inserted = 0
|
||||||
|
tables_with_data = []
|
||||||
|
|
||||||
|
for tbl in stats['tables']:
|
||||||
|
table_name = tbl.get('table', '').replace('billiards_dwd.', '')
|
||||||
|
processed = tbl.get('processed', 0)
|
||||||
|
inserted = tbl.get('inserted', 0)
|
||||||
|
|
||||||
|
if processed > 0:
|
||||||
|
total_processed += processed
|
||||||
|
tables_with_data.append(f"{table_name}({processed})")
|
||||||
|
elif inserted > 0:
|
||||||
|
total_inserted += inserted
|
||||||
|
tables_with_data.append(f"{table_name}(+{inserted})")
|
||||||
|
|
||||||
|
if total_processed > 0 or total_inserted > 0:
|
||||||
|
dwd_stats.append(f"处理维度: {total_processed}条, 新增事实: {total_inserted}条")
|
||||||
|
if len(tables_with_data) <= 5:
|
||||||
|
dwd_stats.append(f"涉及表: {', '.join(tables_with_data)}")
|
||||||
|
else:
|
||||||
|
dwd_stats.append(f"涉及 {len(tables_with_data)} 张表")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 4. 解析数据校验结果
|
||||||
|
# 格式: "CHECK_DONE task=xxx missing=1 records=136 errors=0"
|
||||||
|
match = re.search(r'CHECK_DONE task=(\w+) missing=(\d+) records=(\d+)', line)
|
||||||
|
if match:
|
||||||
|
task_name = match.group(1)
|
||||||
|
missing = int(match.group(2))
|
||||||
|
records = int(match.group(3))
|
||||||
|
if missing > 0:
|
||||||
|
if 'missing_tasks' not in integrity_stats:
|
||||||
|
integrity_stats['missing_tasks'] = []
|
||||||
|
integrity_stats['missing_tasks'].append(f"{task_name}: 缺失{missing}/{records}")
|
||||||
|
integrity_stats['total_records'] = integrity_stats.get('total_records', 0) + records
|
||||||
|
integrity_stats['total_missing'] = integrity_stats.get('total_missing', 0) + missing
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 5. 解析数据校验最终结果
|
||||||
|
# 格式: "结果统计: {'missing': 463, 'errors': 0, 'backfilled': 0}"
|
||||||
|
if "结果统计:" in line or "结果统计:" in line:
|
||||||
|
try:
|
||||||
|
match = re.search(r"\{.+\}", line)
|
||||||
|
if match:
|
||||||
|
stats_str = match.group(0).replace("'", '"')
|
||||||
|
stats = json.loads(stats_str)
|
||||||
|
integrity_stats['final_missing'] = stats.get('missing', 0)
|
||||||
|
integrity_stats['final_errors'] = stats.get('errors', 0)
|
||||||
|
integrity_stats['backfilled'] = stats.get('backfilled', 0)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 6. 解析错误信息
|
||||||
|
if "[ERROR]" in line or "错误" in line.lower() or "error" in line.lower():
|
||||||
|
if "Traceback" not in line and "File " not in line:
|
||||||
|
errors.append(line.strip()[:100])
|
||||||
|
|
||||||
|
# 7. 解析任务完成信息
|
||||||
|
if "任务执行成功" in line or "ETL运行完成" in line:
|
||||||
|
task_results.append("✓ " + line.split("]")[-1].strip() if "]" in line else line.strip())
|
||||||
|
elif "任务执行失败" in line:
|
||||||
|
task_results.append("✗ " + line.split("]")[-1].strip() if "]" in line else line.strip())
|
||||||
|
|
||||||
|
# 构建摘要
|
||||||
|
if ods_stats:
|
||||||
|
summary_parts.append("【ODS 抓取】" + ", ".join(ods_stats[:5]))
|
||||||
|
if len(ods_stats) > 5:
|
||||||
|
summary_parts[-1] += f" 等{len(ods_stats)}项"
|
||||||
|
|
||||||
|
if dwd_stats:
|
||||||
|
summary_parts.append("【DWD 装载】" + "; ".join(dwd_stats))
|
||||||
|
|
||||||
|
if integrity_stats:
|
||||||
|
total_missing = integrity_stats.get('final_missing', integrity_stats.get('total_missing', 0))
|
||||||
|
total_records = integrity_stats.get('total_records', 0)
|
||||||
|
backfilled = integrity_stats.get('backfilled', 0)
|
||||||
|
|
||||||
|
int_summary = f"【数据校验】检查 {total_records} 条记录"
|
||||||
|
if total_missing > 0:
|
||||||
|
int_summary += f", 发现 {total_missing} 条缺失"
|
||||||
|
if backfilled > 0:
|
||||||
|
int_summary += f", 已补全 {backfilled} 条"
|
||||||
|
else:
|
||||||
|
int_summary += ", 数据完整"
|
||||||
|
summary_parts.append(int_summary)
|
||||||
|
|
||||||
|
# 显示缺失详情
|
||||||
|
if integrity_stats.get('missing_tasks'):
|
||||||
|
missing_detail = integrity_stats['missing_tasks'][:3]
|
||||||
|
summary_parts.append(" 缺失: " + "; ".join(missing_detail))
|
||||||
|
if len(integrity_stats['missing_tasks']) > 3:
|
||||||
|
summary_parts[-1] += f" 等{len(integrity_stats['missing_tasks'])}项"
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
summary_parts.append("【错误】" + "; ".join(errors[:3]))
|
||||||
|
|
||||||
|
if task_results:
|
||||||
|
summary_parts.append("【结果】" + " | ".join(task_results))
|
||||||
|
|
||||||
|
if summary_parts:
|
||||||
|
return "\n".join(summary_parts)
|
||||||
|
|
||||||
|
# 如果没有解析到任何信息,返回最后几行关键信息
|
||||||
|
key_lines = []
|
||||||
|
for line in self._output_lines[-10:]:
|
||||||
|
if "完成" in line or "成功" in line or "失败" in line:
|
||||||
|
key_lines.append(line.strip()[:80])
|
||||||
|
|
||||||
|
if key_lines:
|
||||||
|
return "\n".join(key_lines[-3:])
|
||||||
|
|
||||||
|
return self._output_lines[-1] if self._output_lines else "执行完成"
|
||||||
|
|
||||||
|
@property
|
||||||
|
def exit_code(self) -> Optional[int]:
|
||||||
|
"""获取退出码"""
|
||||||
|
return self._exit_code
|
||||||
|
|
||||||
|
@property
|
||||||
|
def output(self) -> str:
|
||||||
|
"""获取完整输出"""
|
||||||
|
return "\n".join(self._output_lines)
|
||||||
@@ -129,6 +129,13 @@ class ETLScheduler:
|
|||||||
# ------------------------------------------------------------------ internals
|
# ------------------------------------------------------------------ internals
|
||||||
def _run_single_task(self, task_code: str, run_uuid: str, store_id: int):
|
def _run_single_task(self, task_code: str, run_uuid: str, store_id: int):
|
||||||
"""单个任务的抓取/清洗编排。"""
|
"""单个任务的抓取/清洗编排。"""
|
||||||
|
task_code_upper = task_code.upper()
|
||||||
|
|
||||||
|
# 工具类任务:直接执行,不记录 cursor/run
|
||||||
|
if task_code_upper in self.NO_DB_CONFIG_TASKS:
|
||||||
|
self._run_utility_task(task_code_upper, store_id)
|
||||||
|
return
|
||||||
|
|
||||||
task_cfg = self._load_task_config(task_code, store_id)
|
task_cfg = self._load_task_config(task_code, store_id)
|
||||||
if not task_cfg:
|
if not task_cfg:
|
||||||
self.logger.warning("任务 %s 未启用或不存在", task_code)
|
self.logger.warning("任务 %s 未启用或不存在", task_code)
|
||||||
@@ -183,6 +190,7 @@ class ETLScheduler:
|
|||||||
window_end=window.get("end"),
|
window_end=window.get("end"),
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
)
|
)
|
||||||
|
self._maybe_run_integrity_check(task_code, window)
|
||||||
return
|
return
|
||||||
|
|
||||||
if self._flow_includes_fetch():
|
if self._flow_includes_fetch():
|
||||||
@@ -221,6 +229,7 @@ class ETLScheduler:
|
|||||||
window_end=window.get("end"),
|
window_end=window.get("end"),
|
||||||
run_id=run_id,
|
run_id=run_id,
|
||||||
)
|
)
|
||||||
|
self._maybe_run_integrity_check(task_code, window)
|
||||||
|
|
||||||
except Exception as exc: # noqa: BLE001
|
except Exception as exc: # noqa: BLE001
|
||||||
self.run_tracker.update_run(
|
self.run_tracker.update_run(
|
||||||
@@ -322,6 +331,53 @@ class ETLScheduler:
|
|||||||
def _flow_includes_ingest(self) -> bool:
|
def _flow_includes_ingest(self) -> bool:
|
||||||
return self.pipeline_flow in {"INGEST_ONLY", "FULL"}
|
return self.pipeline_flow in {"INGEST_ONLY", "FULL"}
|
||||||
|
|
||||||
|
# 不需要数据库配置即可运行的任务(工具类/初始化类)
|
||||||
|
NO_DB_CONFIG_TASKS = {
|
||||||
|
# Schema 初始化任务
|
||||||
|
"INIT_ODS_SCHEMA",
|
||||||
|
"INIT_DWD_SCHEMA",
|
||||||
|
"INIT_DWS_SCHEMA",
|
||||||
|
# 质量检查任务
|
||||||
|
"DATA_INTEGRITY_CHECK",
|
||||||
|
"DWD_QUALITY_CHECK",
|
||||||
|
# 工具任务
|
||||||
|
"CHECK_CUTOFF",
|
||||||
|
"MANUAL_INGEST",
|
||||||
|
"ODS_JSON_ARCHIVE",
|
||||||
|
# DWS 汇总任务
|
||||||
|
"DWS_BUILD_ORDER_SUMMARY",
|
||||||
|
}
|
||||||
|
|
||||||
|
def _run_utility_task(self, task_code: str, store_id: int):
|
||||||
|
"""
|
||||||
|
执行工具类任务(不记录 cursor/run,直接执行)。
|
||||||
|
这些任务不需要游标管理和运行跟踪。
|
||||||
|
"""
|
||||||
|
self.logger.info("%s: 开始执行工具类任务", task_code)
|
||||||
|
|
||||||
|
try:
|
||||||
|
# 创建任务实例(不需要 API client,使用 None)
|
||||||
|
task = self.task_registry.create_task(
|
||||||
|
task_code, self.config, self.db_ops, None, self.logger
|
||||||
|
)
|
||||||
|
|
||||||
|
# 执行任务(工具类任务通常不需要 cursor_data)
|
||||||
|
result = task.execute(None)
|
||||||
|
|
||||||
|
status = (result.get("status") or "").upper() if isinstance(result, dict) else "SUCCESS"
|
||||||
|
if status == "SUCCESS":
|
||||||
|
self.logger.info("%s: 工具类任务执行成功", task_code)
|
||||||
|
if isinstance(result, dict):
|
||||||
|
counts = result.get("counts", {})
|
||||||
|
if counts:
|
||||||
|
self.logger.info("%s: 结果统计: %s", task_code, counts)
|
||||||
|
else:
|
||||||
|
self.logger.warning("%s: 工具类任务执行结果: %s", task_code, status)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
self.logger.error("%s: 工具类任务执行失败: %s", task_code, exc, exc_info=True)
|
||||||
|
raise
|
||||||
|
|
||||||
def _load_task_config(self, task_code: str, store_id: int) -> dict | None:
|
def _load_task_config(self, task_code: str, store_id: int) -> dict | None:
|
||||||
"""从数据库加载任务配置。"""
|
"""从数据库加载任务配置。"""
|
||||||
sql = """
|
sql = """
|
||||||
@@ -334,6 +390,45 @@ class ETLScheduler:
|
|||||||
rows = self.db_conn.query(sql, (store_id, task_code))
|
rows = self.db_conn.query(sql, (store_id, task_code))
|
||||||
return rows[0] if rows else None
|
return rows[0] if rows else None
|
||||||
|
|
||||||
|
def _maybe_run_integrity_check(self, task_code: str, window: dict | None) -> None:
|
||||||
|
if not self.config.get("integrity.auto_check", False):
|
||||||
|
return
|
||||||
|
if str(task_code or "").upper() != "DWD_LOAD_FROM_ODS":
|
||||||
|
return
|
||||||
|
if not isinstance(window, dict):
|
||||||
|
return
|
||||||
|
window_start = window.get("start")
|
||||||
|
window_end = window.get("end")
|
||||||
|
if not window_start or not window_end:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
from quality.integrity_checker import IntegrityWindow, run_integrity_window
|
||||||
|
|
||||||
|
include_dimensions = bool(self.config.get("integrity.include_dimensions", False))
|
||||||
|
task_codes = str(self.config.get("integrity.ods_task_codes", "") or "").strip()
|
||||||
|
report = run_integrity_window(
|
||||||
|
cfg=self.config,
|
||||||
|
window=IntegrityWindow(
|
||||||
|
start=window_start,
|
||||||
|
end=window_end,
|
||||||
|
label="etl_window",
|
||||||
|
granularity="window",
|
||||||
|
),
|
||||||
|
include_dimensions=include_dimensions,
|
||||||
|
task_codes=task_codes,
|
||||||
|
logger=self.logger,
|
||||||
|
write_report=True,
|
||||||
|
)
|
||||||
|
self.logger.info(
|
||||||
|
"Integrity check done: report=%s missing=%s errors=%s",
|
||||||
|
report.get("report_path"),
|
||||||
|
report.get("api_to_ods", {}).get("total_missing"),
|
||||||
|
report.get("api_to_ods", {}).get("total_errors"),
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
self.logger.warning("Integrity check failed: %s", exc, exc_info=True)
|
||||||
|
|
||||||
def close(self):
|
def close(self):
|
||||||
"""关闭连接。"""
|
"""关闭连接。"""
|
||||||
self.db_conn.close()
|
self.db_conn.close()
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ from tasks.ods_json_archive_task import OdsJsonArchiveTask
|
|||||||
from tasks.check_cutoff_task import CheckCutoffTask
|
from tasks.check_cutoff_task import CheckCutoffTask
|
||||||
from tasks.init_dws_schema_task import InitDwsSchemaTask
|
from tasks.init_dws_schema_task import InitDwsSchemaTask
|
||||||
from tasks.dws_build_order_summary_task import DwsBuildOrderSummaryTask
|
from tasks.dws_build_order_summary_task import DwsBuildOrderSummaryTask
|
||||||
|
from tasks.data_integrity_task import DataIntegrityTask
|
||||||
|
|
||||||
class TaskRegistry:
|
class TaskRegistry:
|
||||||
"""任务注册和工厂"""
|
"""任务注册和工厂"""
|
||||||
@@ -78,6 +79,7 @@ default_registry.register("DWD_LOAD_FROM_ODS", DwdLoadTask)
|
|||||||
default_registry.register("DWD_QUALITY_CHECK", DwdQualityTask)
|
default_registry.register("DWD_QUALITY_CHECK", DwdQualityTask)
|
||||||
default_registry.register("ODS_JSON_ARCHIVE", OdsJsonArchiveTask)
|
default_registry.register("ODS_JSON_ARCHIVE", OdsJsonArchiveTask)
|
||||||
default_registry.register("CHECK_CUTOFF", CheckCutoffTask)
|
default_registry.register("CHECK_CUTOFF", CheckCutoffTask)
|
||||||
|
default_registry.register("DATA_INTEGRITY_CHECK", DataIntegrityTask)
|
||||||
default_registry.register("INIT_DWS_SCHEMA", InitDwsSchemaTask)
|
default_registry.register("INIT_DWS_SCHEMA", InitDwsSchemaTask)
|
||||||
default_registry.register("DWS_BUILD_ORDER_SUMMARY", DwsBuildOrderSummaryTask)
|
default_registry.register("DWS_BUILD_ORDER_SUMMARY", DwsBuildOrderSummaryTask)
|
||||||
for code, task_cls in ODS_TASK_CLASSES.items():
|
for code, task_cls in ODS_TASK_CLASSES.items():
|
||||||
|
|||||||
2
etl_billiards/pytest.ini
Normal file
2
etl_billiards/pytest.ini
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
[pytest]
|
||||||
|
pythonpath = .
|
||||||
390
etl_billiards/quality/integrity_checker.py
Normal file
390
etl_billiards/quality/integrity_checker.py
Normal file
@@ -0,0 +1,390 @@
|
|||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
"""Integrity checks across API -> ODS -> DWD."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import date, datetime, time, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, Iterable, List, Tuple
|
||||||
|
from zoneinfo import ZoneInfo
|
||||||
|
|
||||||
|
import json
|
||||||
|
|
||||||
|
from config.settings import AppConfig
|
||||||
|
from database.connection import DatabaseConnection
|
||||||
|
from tasks.dwd_load_task import DwdLoadTask
|
||||||
|
from scripts.check_ods_gaps import run_gap_check
|
||||||
|
|
||||||
|
AMOUNT_KEYWORDS = ("amount", "money", "fee", "balance")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class IntegrityWindow:
|
||||||
|
start: datetime
|
||||||
|
end: datetime
|
||||||
|
label: str
|
||||||
|
granularity: str
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_tz(dt: datetime, tz: ZoneInfo) -> datetime:
|
||||||
|
if dt.tzinfo is None:
|
||||||
|
return dt.replace(tzinfo=tz)
|
||||||
|
return dt.astimezone(tz)
|
||||||
|
|
||||||
|
|
||||||
|
def _month_start(day: date) -> date:
|
||||||
|
return date(day.year, day.month, 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _next_month(day: date) -> date:
|
||||||
|
if day.month == 12:
|
||||||
|
return date(day.year + 1, 1, 1)
|
||||||
|
return date(day.year, day.month + 1, 1)
|
||||||
|
|
||||||
|
|
||||||
|
def _date_to_start(dt: date, tz: ZoneInfo) -> datetime:
|
||||||
|
return datetime.combine(dt, time.min).replace(tzinfo=tz)
|
||||||
|
|
||||||
|
|
||||||
|
def _date_to_end_exclusive(dt: date, tz: ZoneInfo) -> datetime:
|
||||||
|
return datetime.combine(dt, time.min).replace(tzinfo=tz) + timedelta(days=1)
|
||||||
|
|
||||||
|
|
||||||
|
def build_history_windows(start_dt: datetime, end_dt: datetime, tz: ZoneInfo) -> List[IntegrityWindow]:
|
||||||
|
"""Build weekly windows for current month, monthly windows for earlier months."""
|
||||||
|
start_dt = _ensure_tz(start_dt, tz)
|
||||||
|
end_dt = _ensure_tz(end_dt, tz)
|
||||||
|
if end_dt <= start_dt:
|
||||||
|
return []
|
||||||
|
|
||||||
|
start_date = start_dt.date()
|
||||||
|
end_date = end_dt.date()
|
||||||
|
current_month_start = _month_start(end_date)
|
||||||
|
|
||||||
|
windows: List[IntegrityWindow] = []
|
||||||
|
cur = start_date
|
||||||
|
while cur <= end_date:
|
||||||
|
month_start = _month_start(cur)
|
||||||
|
month_end_exclusive = _next_month(cur)
|
||||||
|
range_start = max(cur, month_start)
|
||||||
|
range_end = min(end_date, month_end_exclusive - timedelta(days=1))
|
||||||
|
|
||||||
|
if month_start == current_month_start:
|
||||||
|
week_start = range_start
|
||||||
|
while week_start <= range_end:
|
||||||
|
week_end = min(week_start + timedelta(days=6), range_end)
|
||||||
|
w_start_dt = _date_to_start(week_start, tz)
|
||||||
|
w_end_dt = _date_to_end_exclusive(week_end, tz)
|
||||||
|
if w_start_dt < end_dt and w_end_dt > start_dt:
|
||||||
|
windows.append(
|
||||||
|
IntegrityWindow(
|
||||||
|
start=max(w_start_dt, start_dt),
|
||||||
|
end=min(w_end_dt, end_dt),
|
||||||
|
label=f"week_{week_start.isoformat()}",
|
||||||
|
granularity="week",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
week_start = week_end + timedelta(days=1)
|
||||||
|
else:
|
||||||
|
m_start_dt = _date_to_start(range_start, tz)
|
||||||
|
m_end_dt = _date_to_end_exclusive(range_end, tz)
|
||||||
|
if m_start_dt < end_dt and m_end_dt > start_dt:
|
||||||
|
windows.append(
|
||||||
|
IntegrityWindow(
|
||||||
|
start=max(m_start_dt, start_dt),
|
||||||
|
end=min(m_end_dt, end_dt),
|
||||||
|
label=f"month_{month_start.isoformat()}",
|
||||||
|
granularity="month",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
cur = month_end_exclusive
|
||||||
|
|
||||||
|
return windows
|
||||||
|
|
||||||
|
|
||||||
|
def _split_table(name: str, default_schema: str) -> Tuple[str, str]:
|
||||||
|
if "." in name:
|
||||||
|
schema, table = name.split(".", 1)
|
||||||
|
return schema, table
|
||||||
|
return default_schema, name
|
||||||
|
|
||||||
|
|
||||||
|
def _pick_time_column(dwd_cols: Iterable[str], ods_cols: Iterable[str]) -> str | None:
|
||||||
|
lower_cols = {c.lower() for c in dwd_cols} & {c.lower() for c in ods_cols}
|
||||||
|
for candidate in DwdLoadTask.FACT_ORDER_CANDIDATES:
|
||||||
|
if candidate.lower() in lower_cols:
|
||||||
|
return candidate.lower()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_columns(cur, schema: str, table: str) -> Tuple[List[str], Dict[str, str]]:
|
||||||
|
cur.execute(
|
||||||
|
"""
|
||||||
|
SELECT column_name, data_type
|
||||||
|
FROM information_schema.columns
|
||||||
|
WHERE table_schema = %s AND table_name = %s
|
||||||
|
ORDER BY ordinal_position
|
||||||
|
""",
|
||||||
|
(schema, table),
|
||||||
|
)
|
||||||
|
cols = []
|
||||||
|
types: Dict[str, str] = {}
|
||||||
|
for name, data_type in cur.fetchall():
|
||||||
|
cols.append(name)
|
||||||
|
types[name.lower()] = (data_type or "").lower()
|
||||||
|
return cols, types
|
||||||
|
|
||||||
|
|
||||||
|
def _amount_columns(cols: List[str], types: Dict[str, str]) -> List[str]:
|
||||||
|
numeric_types = {"numeric", "double precision", "integer", "bigint", "smallint", "real", "decimal"}
|
||||||
|
out = []
|
||||||
|
for col in cols:
|
||||||
|
lc = col.lower()
|
||||||
|
if types.get(lc) not in numeric_types:
|
||||||
|
continue
|
||||||
|
if any(key in lc for key in AMOUNT_KEYWORDS):
|
||||||
|
out.append(lc)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _count_table(cur, schema: str, table: str, time_col: str | None, window: IntegrityWindow | None) -> int:
|
||||||
|
where = ""
|
||||||
|
params: List[Any] = []
|
||||||
|
if time_col and window:
|
||||||
|
where = f'WHERE "{time_col}" >= %s AND "{time_col}" < %s'
|
||||||
|
params = [window.start, window.end]
|
||||||
|
sql = f'SELECT COUNT(1) FROM "{schema}"."{table}" {where}'
|
||||||
|
cur.execute(sql, params)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return int(row[0] if row else 0)
|
||||||
|
|
||||||
|
|
||||||
|
def _sum_column(cur, schema: str, table: str, col: str, time_col: str | None, window: IntegrityWindow | None) -> float:
|
||||||
|
where = ""
|
||||||
|
params: List[Any] = []
|
||||||
|
if time_col and window:
|
||||||
|
where = f'WHERE "{time_col}" >= %s AND "{time_col}" < %s'
|
||||||
|
params = [window.start, window.end]
|
||||||
|
sql = f'SELECT COALESCE(SUM("{col}"), 0) FROM "{schema}"."{table}" {where}'
|
||||||
|
cur.execute(sql, params)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return float(row[0] if row else 0)
|
||||||
|
|
||||||
|
|
||||||
|
def run_dwd_vs_ods_check(
|
||||||
|
*,
|
||||||
|
cfg: AppConfig,
|
||||||
|
window: IntegrityWindow | None,
|
||||||
|
include_dimensions: bool,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
dsn = cfg["db"]["dsn"]
|
||||||
|
session = cfg["db"].get("session")
|
||||||
|
db_conn = DatabaseConnection(dsn=dsn, session=session)
|
||||||
|
try:
|
||||||
|
with db_conn.conn.cursor() as cur:
|
||||||
|
results: List[Dict[str, Any]] = []
|
||||||
|
table_map = DwdLoadTask.TABLE_MAP
|
||||||
|
for dwd_table, ods_table in table_map.items():
|
||||||
|
if not include_dimensions and ".dim_" in dwd_table:
|
||||||
|
continue
|
||||||
|
schema_dwd, name_dwd = _split_table(dwd_table, "billiards_dwd")
|
||||||
|
schema_ods, name_ods = _split_table(ods_table, "billiards_ods")
|
||||||
|
try:
|
||||||
|
dwd_cols, dwd_types = _fetch_columns(cur, schema_dwd, name_dwd)
|
||||||
|
ods_cols, ods_types = _fetch_columns(cur, schema_ods, name_ods)
|
||||||
|
time_col = _pick_time_column(dwd_cols, ods_cols)
|
||||||
|
count_dwd = _count_table(cur, schema_dwd, name_dwd, time_col, window)
|
||||||
|
count_ods = _count_table(cur, schema_ods, name_ods, time_col, window)
|
||||||
|
|
||||||
|
dwd_amount_cols = _amount_columns(dwd_cols, dwd_types)
|
||||||
|
ods_amount_cols = _amount_columns(ods_cols, ods_types)
|
||||||
|
common_amount_cols = sorted(set(dwd_amount_cols) & set(ods_amount_cols))
|
||||||
|
amounts: List[Dict[str, Any]] = []
|
||||||
|
for col in common_amount_cols:
|
||||||
|
dwd_sum = _sum_column(cur, schema_dwd, name_dwd, col, time_col, window)
|
||||||
|
ods_sum = _sum_column(cur, schema_ods, name_ods, col, time_col, window)
|
||||||
|
amounts.append(
|
||||||
|
{
|
||||||
|
"column": col,
|
||||||
|
"dwd_sum": dwd_sum,
|
||||||
|
"ods_sum": ods_sum,
|
||||||
|
"diff": dwd_sum - ods_sum,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"dwd_table": dwd_table,
|
||||||
|
"ods_table": ods_table,
|
||||||
|
"windowed": bool(time_col and window),
|
||||||
|
"window_col": time_col,
|
||||||
|
"count": {"dwd": count_dwd, "ods": count_ods, "diff": count_dwd - count_ods},
|
||||||
|
"amounts": amounts,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
results.append(
|
||||||
|
{
|
||||||
|
"dwd_table": dwd_table,
|
||||||
|
"ods_table": ods_table,
|
||||||
|
"windowed": bool(window),
|
||||||
|
"window_col": None,
|
||||||
|
"count": {"dwd": None, "ods": None, "diff": None},
|
||||||
|
"amounts": [],
|
||||||
|
"error": f"{type(exc).__name__}: {exc}",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
total_count_diff = sum(
|
||||||
|
int(item.get("count", {}).get("diff") or 0)
|
||||||
|
for item in results
|
||||||
|
if isinstance(item.get("count", {}).get("diff"), (int, float))
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"tables": results,
|
||||||
|
"total_count_diff": total_count_diff,
|
||||||
|
}
|
||||||
|
finally:
|
||||||
|
db_conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _default_report_path(prefix: str) -> Path:
|
||||||
|
root = Path(__file__).resolve().parents[1]
|
||||||
|
stamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
return root / "reports" / f"{prefix}_{stamp}.json"
|
||||||
|
|
||||||
|
|
||||||
|
def run_integrity_window(
|
||||||
|
*,
|
||||||
|
cfg: AppConfig,
|
||||||
|
window: IntegrityWindow,
|
||||||
|
include_dimensions: bool,
|
||||||
|
task_codes: str,
|
||||||
|
logger,
|
||||||
|
write_report: bool,
|
||||||
|
report_path: Path | None = None,
|
||||||
|
window_split_unit: str | None = None,
|
||||||
|
window_compensation_hours: int | None = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
total_seconds = max(0, int((window.end - window.start).total_seconds()))
|
||||||
|
if total_seconds >= 86400:
|
||||||
|
window_days = max(1, total_seconds // 86400)
|
||||||
|
window_hours = 0
|
||||||
|
else:
|
||||||
|
window_days = 0
|
||||||
|
window_hours = max(1, total_seconds // 3600 or 1)
|
||||||
|
|
||||||
|
ods_payload = run_gap_check(
|
||||||
|
cfg=cfg,
|
||||||
|
start=window.start,
|
||||||
|
end=window.end,
|
||||||
|
window_days=window_days,
|
||||||
|
window_hours=window_hours,
|
||||||
|
page_size=int(cfg.get("api.page_size") or 200),
|
||||||
|
chunk_size=500,
|
||||||
|
sample_limit=50,
|
||||||
|
sleep_per_window=0,
|
||||||
|
sleep_per_page=0,
|
||||||
|
task_codes=task_codes,
|
||||||
|
from_cutoff=False,
|
||||||
|
cutoff_overlap_hours=24,
|
||||||
|
allow_small_window=True,
|
||||||
|
logger=logger,
|
||||||
|
window_split_unit=window_split_unit,
|
||||||
|
window_compensation_hours=window_compensation_hours,
|
||||||
|
)
|
||||||
|
|
||||||
|
dwd_payload = run_dwd_vs_ods_check(
|
||||||
|
cfg=cfg,
|
||||||
|
window=window,
|
||||||
|
include_dimensions=include_dimensions,
|
||||||
|
)
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": window.start.isoformat(),
|
||||||
|
"end": window.end.isoformat(),
|
||||||
|
"label": window.label,
|
||||||
|
"granularity": window.granularity,
|
||||||
|
},
|
||||||
|
"api_to_ods": ods_payload,
|
||||||
|
"ods_to_dwd": dwd_payload,
|
||||||
|
"generated_at": datetime.now(ZoneInfo(cfg.get("app.timezone", "Asia/Taipei"))).isoformat(),
|
||||||
|
}
|
||||||
|
|
||||||
|
if write_report:
|
||||||
|
path = report_path or _default_report_path("data_integrity_window")
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
report["report_path"] = str(path)
|
||||||
|
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def run_integrity_history(
|
||||||
|
*,
|
||||||
|
cfg: AppConfig,
|
||||||
|
start_dt: datetime,
|
||||||
|
end_dt: datetime,
|
||||||
|
include_dimensions: bool,
|
||||||
|
task_codes: str,
|
||||||
|
logger,
|
||||||
|
write_report: bool,
|
||||||
|
report_path: Path | None = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
tz = ZoneInfo(cfg.get("app.timezone", "Asia/Taipei"))
|
||||||
|
windows = build_history_windows(start_dt, end_dt, tz)
|
||||||
|
results: List[Dict[str, Any]] = []
|
||||||
|
total_missing = 0
|
||||||
|
total_errors = 0
|
||||||
|
|
||||||
|
for window in windows:
|
||||||
|
logger.info("校验窗口 起始=%s 结束=%s", window.start, window.end)
|
||||||
|
payload = run_integrity_window(
|
||||||
|
cfg=cfg,
|
||||||
|
window=window,
|
||||||
|
include_dimensions=include_dimensions,
|
||||||
|
task_codes=task_codes,
|
||||||
|
logger=logger,
|
||||||
|
write_report=False,
|
||||||
|
)
|
||||||
|
results.append(payload)
|
||||||
|
total_missing += int(payload.get("api_to_ods", {}).get("total_missing") or 0)
|
||||||
|
total_errors += int(payload.get("api_to_ods", {}).get("total_errors") or 0)
|
||||||
|
|
||||||
|
report = {
|
||||||
|
"mode": "history",
|
||||||
|
"start": _ensure_tz(start_dt, tz).isoformat(),
|
||||||
|
"end": _ensure_tz(end_dt, tz).isoformat(),
|
||||||
|
"windows": results,
|
||||||
|
"total_missing": total_missing,
|
||||||
|
"total_errors": total_errors,
|
||||||
|
"generated_at": datetime.now(tz).isoformat(),
|
||||||
|
}
|
||||||
|
|
||||||
|
if write_report:
|
||||||
|
path = report_path or _default_report_path("data_integrity_history")
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||||
|
report["report_path"] = str(path)
|
||||||
|
|
||||||
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
def compute_last_etl_end(cfg: AppConfig) -> datetime | None:
|
||||||
|
dsn = cfg["db"]["dsn"]
|
||||||
|
session = cfg["db"].get("session")
|
||||||
|
db_conn = DatabaseConnection(dsn=dsn, session=session)
|
||||||
|
try:
|
||||||
|
rows = db_conn.query(
|
||||||
|
"SELECT MAX(window_end) AS mx FROM etl_admin.etl_run WHERE store_id = %s",
|
||||||
|
(cfg.get("app.store_id"),),
|
||||||
|
)
|
||||||
|
mx = rows[0]["mx"] if rows else None
|
||||||
|
if isinstance(mx, datetime):
|
||||||
|
tz = ZoneInfo(cfg.get("app.timezone", "Asia/Taipei"))
|
||||||
|
return _ensure_tz(mx, tz)
|
||||||
|
finally:
|
||||||
|
db_conn.close()
|
||||||
|
return None
|
||||||
19596
etl_billiards/reports/data_integrity_history_20260119_045009.json
Normal file
19596
etl_billiards/reports/data_integrity_history_20260119_045009.json
Normal file
File diff suppressed because it is too large
Load Diff
45066
etl_billiards/reports/data_integrity_history_20260119_060453.json
Normal file
45066
etl_billiards/reports/data_integrity_history_20260119_060453.json
Normal file
File diff suppressed because it is too large
Load Diff
45060
etl_billiards/reports/data_integrity_history_20260119_070903.json
Normal file
45060
etl_billiards/reports/data_integrity_history_20260119_070903.json
Normal file
File diff suppressed because it is too large
Load Diff
24517
etl_billiards/reports/data_integrity_history_20260119_082915.json
Normal file
24517
etl_billiards/reports/data_integrity_history_20260119_082915.json
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"mode": "history",
|
||||||
|
"start": "2026-01-19T00:00:00+08:00",
|
||||||
|
"end": "2026-01-19T00:00:00+08:00",
|
||||||
|
"windows": [],
|
||||||
|
"total_missing": 0,
|
||||||
|
"total_errors": 0,
|
||||||
|
"generated_at": "2026-01-19T17:07:01.451073+08:00"
|
||||||
|
}
|
||||||
10555
etl_billiards/reports/data_integrity_history_20260120_041412.json
Normal file
10555
etl_billiards/reports/data_integrity_history_20260120_041412.json
Normal file
File diff suppressed because it is too large
Load Diff
19348
etl_billiards/reports/data_integrity_history_20260126_061648.json
Normal file
19348
etl_billiards/reports/data_integrity_history_20260126_061648.json
Normal file
File diff suppressed because it is too large
Load Diff
18946
etl_billiards/reports/data_integrity_history_20260126_192038.json
Normal file
18946
etl_billiards/reports/data_integrity_history_20260126_192038.json
Normal file
File diff suppressed because it is too large
Load Diff
11273
etl_billiards/reports/data_integrity_history_20260126_205212.json
Normal file
11273
etl_billiards/reports/data_integrity_history_20260126_205212.json
Normal file
File diff suppressed because it is too large
Load Diff
14940
etl_billiards/reports/data_integrity_history_20260126_213058.json
Normal file
14940
etl_billiards/reports/data_integrity_history_20260126_213058.json
Normal file
File diff suppressed because it is too large
Load Diff
12264
etl_billiards/reports/data_integrity_history_20260126_220521.json
Normal file
12264
etl_billiards/reports/data_integrity_history_20260126_220521.json
Normal file
File diff suppressed because it is too large
Load Diff
12704
etl_billiards/reports/data_integrity_history_20260126_225143.json
Normal file
12704
etl_billiards/reports/data_integrity_history_20260126_225143.json
Normal file
File diff suppressed because it is too large
Load Diff
12989
etl_billiards/reports/data_integrity_history_20260126_234946.json
Normal file
12989
etl_billiards/reports/data_integrity_history_20260126_234946.json
Normal file
File diff suppressed because it is too large
Load Diff
1690
etl_billiards/reports/data_integrity_history_20260127_024335.json
Normal file
1690
etl_billiards/reports/data_integrity_history_20260127_024335.json
Normal file
File diff suppressed because it is too large
Load Diff
1275
etl_billiards/reports/data_integrity_history_20260127_031951.json
Normal file
1275
etl_billiards/reports/data_integrity_history_20260127_031951.json
Normal file
File diff suppressed because it is too large
Load Diff
1980
etl_billiards/reports/data_integrity_window_20260119_222704.json
Normal file
1980
etl_billiards/reports/data_integrity_window_20260119_222704.json
Normal file
File diff suppressed because it is too large
Load Diff
1311
etl_billiards/reports/data_integrity_window_20260119_233704.json
Normal file
1311
etl_billiards/reports/data_integrity_window_20260119_233704.json
Normal file
File diff suppressed because it is too large
Load Diff
1096
etl_billiards/reports/data_integrity_window_20260120_003547.json
Normal file
1096
etl_billiards/reports/data_integrity_window_20260120_003547.json
Normal file
File diff suppressed because it is too large
Load Diff
1041
etl_billiards/reports/data_integrity_window_20260120_023005.json
Normal file
1041
etl_billiards/reports/data_integrity_window_20260120_023005.json
Normal file
File diff suppressed because it is too large
Load Diff
972
etl_billiards/reports/data_integrity_window_20260120_034306.json
Normal file
972
etl_billiards/reports/data_integrity_window_20260120_034306.json
Normal file
@@ -0,0 +1,972 @@
|
|||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-10T00:37:58+08:00",
|
||||||
|
"end": "2026-01-20T05:37:58+08:00",
|
||||||
|
"segments": 1
|
||||||
|
},
|
||||||
|
"windows": [
|
||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-10T00:37:58+08:00",
|
||||||
|
"end": "2026-01-20T05:37:58+08:00",
|
||||||
|
"label": "segment_1",
|
||||||
|
"granularity": "window"
|
||||||
|
},
|
||||||
|
"api_to_ods": {
|
||||||
|
"window_split_unit": "none",
|
||||||
|
"window_compensation_hours": 0,
|
||||||
|
"start": "2026-01-10T00:37:58+08:00",
|
||||||
|
"end": "2026-01-20T05:37:58+08:00",
|
||||||
|
"cutoff": null,
|
||||||
|
"window_days": 10,
|
||||||
|
"window_hours": 0,
|
||||||
|
"page_size": 200,
|
||||||
|
"chunk_size": 500,
|
||||||
|
"sample_limit": 50,
|
||||||
|
"store_id": 2790685415443269,
|
||||||
|
"base_url": "https://pc.ficoo.vip/apiprod/admin/v1/",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ACCOUNT",
|
||||||
|
"table": "billiards_ods.assistant_accounts_master",
|
||||||
|
"endpoint": "/PersonnelManagement/SearchAssistantInfo",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 136,
|
||||||
|
"records_with_pk": 136,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_RECORDS",
|
||||||
|
"table": "billiards_ods.settlement_records",
|
||||||
|
"endpoint": "/Site/GetAllOrderSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 1234,
|
||||||
|
"records_with_pk": 1234,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 8,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_USE",
|
||||||
|
"table": "billiards_ods.table_fee_transactions",
|
||||||
|
"endpoint": "/Site/GetSiteTableOrderDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 9940,
|
||||||
|
"records_with_pk": 9940,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 50,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_LEDGER",
|
||||||
|
"table": "billiards_ods.assistant_service_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetOrderAssistantDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 299,
|
||||||
|
"records_with_pk": 299,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 3,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ABOLISH",
|
||||||
|
"table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetAbolitionAssistant",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 9,
|
||||||
|
"records_with_pk": 9,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS_SALES",
|
||||||
|
"table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsSalesList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PAYMENT",
|
||||||
|
"table": "billiards_ods.payment_transactions",
|
||||||
|
"endpoint": "/PayLog/GetPayLogListPage",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 11894,
|
||||||
|
"records_with_pk": 11894,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 60,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_REFUND",
|
||||||
|
"table": "billiards_ods.refund_transactions",
|
||||||
|
"endpoint": "/Order/GetRefundPayLogList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 37,
|
||||||
|
"records_with_pk": 37,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PLATFORM_COUPON",
|
||||||
|
"table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"endpoint": "/Promotion/GetOfflineCouponConsumePageList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 16480,
|
||||||
|
"records_with_pk": 16480,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 83,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER",
|
||||||
|
"table": "billiards_ods.member_profiles",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 554,
|
||||||
|
"records_with_pk": 554,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 3,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_CARD",
|
||||||
|
"table": "billiards_ods.member_stored_value_cards",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberCardList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 943,
|
||||||
|
"records_with_pk": 943,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 5,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_BALANCE",
|
||||||
|
"table": "billiards_ods.member_balance_changes",
|
||||||
|
"endpoint": "/MemberProfile/GetMemberCardBalanceChange",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 2474,
|
||||||
|
"records_with_pk": 2474,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 13,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_RECHARGE_SETTLE",
|
||||||
|
"table": "billiards_ods.recharge_settlements",
|
||||||
|
"endpoint": "/Site/GetRechargeSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 28,
|
||||||
|
"records_with_pk": 28,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_PACKAGE",
|
||||||
|
"table": "billiards_ods.group_buy_packages",
|
||||||
|
"endpoint": "/PackageCoupon/QueryPackageCouponList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 18,
|
||||||
|
"records_with_pk": 18,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_BUY_REDEMPTION",
|
||||||
|
"table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"endpoint": "/Site/GetSiteTableUseDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 8282,
|
||||||
|
"records_with_pk": 8282,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 42,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_STOCK",
|
||||||
|
"table": "billiards_ods.goods_stock_summary",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsStockReport",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsid"
|
||||||
|
],
|
||||||
|
"records": 170,
|
||||||
|
"records_with_pk": 170,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_CHANGE",
|
||||||
|
"table": "billiards_ods.goods_stock_movements",
|
||||||
|
"endpoint": "/GoodsStockManage/QueryGoodsOutboundReceipt",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsstockid"
|
||||||
|
],
|
||||||
|
"records": 1689,
|
||||||
|
"records_with_pk": 1689,
|
||||||
|
"missing": 1,
|
||||||
|
"missing_samples": [
|
||||||
|
{
|
||||||
|
"sitegoodsstockid": 3058615392274245
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"pages": 10,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLES",
|
||||||
|
"table": "billiards_ods.site_tables_master",
|
||||||
|
"endpoint": "/Table/GetSiteTables",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 74,
|
||||||
|
"records_with_pk": 74,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GOODS_CATEGORY",
|
||||||
|
"table": "billiards_ods.stock_goods_category_tree",
|
||||||
|
"endpoint": "/TenantGoodsCategory/QueryPrimarySecondaryCategory",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 9,
|
||||||
|
"records_with_pk": 9,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS",
|
||||||
|
"table": "billiards_ods.store_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsInventoryList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 170,
|
||||||
|
"records_with_pk": 170,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_FEE_DISCOUNT",
|
||||||
|
"table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"endpoint": "/Site/GetTaiFeeAdjustList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 1690,
|
||||||
|
"records_with_pk": 1690,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 9,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TENANT_GOODS",
|
||||||
|
"table": "billiards_ods.tenant_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/QueryTenantGoods",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 171,
|
||||||
|
"records_with_pk": 171,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_TICKET",
|
||||||
|
"table": "billiards_ods.settlement_ticket_details",
|
||||||
|
"endpoint": "/Order/GetOrderSettleTicketNew",
|
||||||
|
"pk_columns": [
|
||||||
|
"ordersettleid"
|
||||||
|
],
|
||||||
|
"records": 1279,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 8,
|
||||||
|
"skipped_missing_pk": 1279,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null,
|
||||||
|
"source_endpoint": "/PayLog/GetPayLogListPage"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0,
|
||||||
|
"generated_at": "2026-01-20T03:43:01.888371+08:00"
|
||||||
|
},
|
||||||
|
"ods_to_dwd": {
|
||||||
|
"tables": [
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 22622,
|
||||||
|
"ods": 22622,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head_ex",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 22622,
|
||||||
|
"ods": 22622,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 1091,
|
||||||
|
"ods": 1091,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "adjust_amount",
|
||||||
|
"dwd_sum": 26391.76,
|
||||||
|
"ods_sum": 26391.76,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "coupon_promotion_amount",
|
||||||
|
"dwd_sum": 56895.59,
|
||||||
|
"ods_sum": 56895.59,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 133626.16,
|
||||||
|
"ods_sum": 133626.16,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 7778.22,
|
||||||
|
"ods_sum": 7778.22,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_table_charge_money",
|
||||||
|
"dwd_sum": 53929.55,
|
||||||
|
"ods_sum": 53929.55,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 17711,
|
||||||
|
"ods": 17711,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "fee_total",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "mgmt_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "used_card_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2736,
|
||||||
|
"ods": 2736,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 315562.38,
|
||||||
|
"ods_sum": 315562.38,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2736,
|
||||||
|
"ods": 2736,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 0,
|
||||||
|
"ods": 0,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "cost_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_goods_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale_ex",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 607092,
|
||||||
|
"ods": 17563,
|
||||||
|
"diff": 589529
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "discount_money",
|
||||||
|
"dwd_sum": 745318.1,
|
||||||
|
"ods_sum": 21673.2,
|
||||||
|
"diff": 723644.9
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_member_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money_cost",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "push_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 303,
|
||||||
|
"ods": 303,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 190.72,
|
||||||
|
"ods_sum": 190.72,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 93547.06,
|
||||||
|
"ods_sum": 93547.06,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4811,
|
||||||
|
"ods": 4811,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "manual_discount_amount",
|
||||||
|
"dwd_sum": 414.17,
|
||||||
|
"ods_sum": 414.17,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 92,
|
||||||
|
"ods": 92,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 92,
|
||||||
|
"ods": 92,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4615,
|
||||||
|
"ods": 4615,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change_ex",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4615,
|
||||||
|
"ods": 4615,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 910,
|
||||||
|
"ods": 910,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 59094.0,
|
||||||
|
"ods_sum": 59094.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 56895.59,
|
||||||
|
"ods_sum": 56895.59,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 10938,
|
||||||
|
"ods": 10938,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "assistant_promotion_money",
|
||||||
|
"dwd_sum": 7353.59,
|
||||||
|
"ods_sum": 7353.59,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "assistant_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "goods_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "recharge_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "reward_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "table_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 905,
|
||||||
|
"ods": 905,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 58612.0,
|
||||||
|
"ods_sum": 58612.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 16480,
|
||||||
|
"ods": 16480,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 444,
|
||||||
|
"ods": 444,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order_ex",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 444,
|
||||||
|
"ods": 444,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_payment",
|
||||||
|
"ods_table": "billiards_ods.payment_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 1279,
|
||||||
|
"ods": 1279,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": 153388.0,
|
||||||
|
"ods_sum": 153388.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 3,
|
||||||
|
"ods": 3,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "channel_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": -43.0,
|
||||||
|
"ods_sum": -43.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund_ex",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 43,
|
||||||
|
"ods": 43,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "balance_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "card_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "round_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_count_diff": 589529
|
||||||
|
},
|
||||||
|
"generated_at": "2026-01-20T03:43:06.774768+08:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"api_to_ods": {
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0
|
||||||
|
},
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0,
|
||||||
|
"generated_at": "2026-01-20T03:43:06.774790+08:00"
|
||||||
|
}
|
||||||
1048
etl_billiards/reports/data_integrity_window_20260120_162840.json
Normal file
1048
etl_billiards/reports/data_integrity_window_20260120_162840.json
Normal file
File diff suppressed because it is too large
Load Diff
1412
etl_billiards/reports/data_integrity_window_20260122_235540.json
Normal file
1412
etl_billiards/reports/data_integrity_window_20260122_235540.json
Normal file
File diff suppressed because it is too large
Load Diff
968
etl_billiards/reports/data_integrity_window_20260124_221448.json
Normal file
968
etl_billiards/reports/data_integrity_window_20260124_221448.json
Normal file
@@ -0,0 +1,968 @@
|
|||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-19T19:14:15+08:00",
|
||||||
|
"end": "2026-01-25T00:14:15+08:00",
|
||||||
|
"segments": 1
|
||||||
|
},
|
||||||
|
"windows": [
|
||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-19T19:14:15+08:00",
|
||||||
|
"end": "2026-01-25T00:14:15+08:00",
|
||||||
|
"label": "segment_1",
|
||||||
|
"granularity": "window"
|
||||||
|
},
|
||||||
|
"api_to_ods": {
|
||||||
|
"window_split_unit": "none",
|
||||||
|
"window_compensation_hours": 0,
|
||||||
|
"start": "2026-01-19T19:14:15+08:00",
|
||||||
|
"end": "2026-01-25T00:14:15+08:00",
|
||||||
|
"cutoff": null,
|
||||||
|
"window_days": 5,
|
||||||
|
"window_hours": 0,
|
||||||
|
"page_size": 200,
|
||||||
|
"chunk_size": 500,
|
||||||
|
"sample_limit": 50,
|
||||||
|
"store_id": 2790685415443269,
|
||||||
|
"base_url": "https://pc.ficoo.vip/apiprod/admin/v1/",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ACCOUNT",
|
||||||
|
"table": "billiards_ods.assistant_accounts_master",
|
||||||
|
"endpoint": "/PersonnelManagement/SearchAssistantInfo",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/PersonnelManagement/SearchAssistantInfo"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_RECORDS",
|
||||||
|
"table": "billiards_ods.settlement_records",
|
||||||
|
"endpoint": "/Site/GetAllOrderSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Site/GetAllOrderSettleList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_USE",
|
||||||
|
"table": "billiards_ods.table_fee_transactions",
|
||||||
|
"endpoint": "/Site/GetSiteTableOrderDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Site/GetSiteTableOrderDetails"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_LEDGER",
|
||||||
|
"table": "billiards_ods.assistant_service_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetOrderAssistantDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/AssistantPerformance/GetOrderAssistantDetails"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ABOLISH",
|
||||||
|
"table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetAbolitionAssistant",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/AssistantPerformance/GetAbolitionAssistant"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS_SALES",
|
||||||
|
"table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsSalesList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/TenantGoods/GetGoodsSalesList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PAYMENT",
|
||||||
|
"table": "billiards_ods.payment_transactions",
|
||||||
|
"endpoint": "/PayLog/GetPayLogListPage",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/PayLog/GetPayLogListPage"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_REFUND",
|
||||||
|
"table": "billiards_ods.refund_transactions",
|
||||||
|
"endpoint": "/Order/GetRefundPayLogList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Order/GetRefundPayLogList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PLATFORM_COUPON",
|
||||||
|
"table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"endpoint": "/Promotion/GetOfflineCouponConsumePageList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Promotion/GetOfflineCouponConsumePageList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER",
|
||||||
|
"table": "billiards_ods.member_profiles",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/MemberProfile/GetTenantMemberList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_CARD",
|
||||||
|
"table": "billiards_ods.member_stored_value_cards",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberCardList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/MemberProfile/GetTenantMemberCardList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_BALANCE",
|
||||||
|
"table": "billiards_ods.member_balance_changes",
|
||||||
|
"endpoint": "/MemberProfile/GetMemberCardBalanceChange",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/MemberProfile/GetMemberCardBalanceChange"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_RECHARGE_SETTLE",
|
||||||
|
"table": "billiards_ods.recharge_settlements",
|
||||||
|
"endpoint": "/Site/GetRechargeSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Site/GetRechargeSettleList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_PACKAGE",
|
||||||
|
"table": "billiards_ods.group_buy_packages",
|
||||||
|
"endpoint": "/PackageCoupon/QueryPackageCouponList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/PackageCoupon/QueryPackageCouponList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_BUY_REDEMPTION",
|
||||||
|
"table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"endpoint": "/Site/GetSiteTableUseDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Site/GetSiteTableUseDetails"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_STOCK",
|
||||||
|
"table": "billiards_ods.goods_stock_summary",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsStockReport",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsid"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/TenantGoods/GetGoodsStockReport"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_CHANGE",
|
||||||
|
"table": "billiards_ods.goods_stock_movements",
|
||||||
|
"endpoint": "/GoodsStockManage/QueryGoodsOutboundReceipt",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsstockid"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/GoodsStockManage/QueryGoodsOutboundReceipt"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLES",
|
||||||
|
"table": "billiards_ods.site_tables_master",
|
||||||
|
"endpoint": "/Table/GetSiteTables",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Table/GetSiteTables"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GOODS_CATEGORY",
|
||||||
|
"table": "billiards_ods.stock_goods_category_tree",
|
||||||
|
"endpoint": "/TenantGoodsCategory/QueryPrimarySecondaryCategory",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/TenantGoodsCategory/QueryPrimarySecondaryCategory"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS",
|
||||||
|
"table": "billiards_ods.store_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsInventoryList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/TenantGoods/GetGoodsInventoryList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_FEE_DISCOUNT",
|
||||||
|
"table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"endpoint": "/Site/GetTaiFeeAdjustList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/Site/GetTaiFeeAdjustList"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TENANT_GOODS",
|
||||||
|
"table": "billiards_ods.tenant_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/QueryTenantGoods",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/TenantGoods/QueryTenantGoods"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_TICKET",
|
||||||
|
"table": "billiards_ods.settlement_ticket_details",
|
||||||
|
"endpoint": "/Order/GetOrderSettleTicketNew",
|
||||||
|
"pk_columns": [
|
||||||
|
"ordersettleid"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 0,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 1,
|
||||||
|
"error_detail": "HTTPError: 401 Client Error: Unauthorized for url: https://pc.ficoo.vip/apiprod/admin/v1/PayLog/GetPayLogListPage",
|
||||||
|
"source_endpoint": "/PayLog/GetPayLogListPage"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_missing": 0,
|
||||||
|
"total_errors": 23,
|
||||||
|
"generated_at": "2026-01-24T22:14:42.999090+08:00"
|
||||||
|
},
|
||||||
|
"ods_to_dwd": {
|
||||||
|
"tables": [
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 22929,
|
||||||
|
"ods": 22929,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head_ex",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 22929,
|
||||||
|
"ods": 22929,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 358,
|
||||||
|
"ods": 358,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "adjust_amount",
|
||||||
|
"dwd_sum": 10139.37,
|
||||||
|
"ods_sum": 10139.37,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "coupon_promotion_amount",
|
||||||
|
"dwd_sum": 17057.33,
|
||||||
|
"ods_sum": 17057.33,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 52686.58,
|
||||||
|
"ods_sum": 52686.58,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 1171.63,
|
||||||
|
"ods_sum": 1171.63,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_table_charge_money",
|
||||||
|
"dwd_sum": 24821.37,
|
||||||
|
"ods_sum": 24821.37,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 17992,
|
||||||
|
"ods": 17992,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "fee_total",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "mgmt_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "used_card_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2779,
|
||||||
|
"ods": 2779,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 323227.69,
|
||||||
|
"ods_sum": 323227.69,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2779,
|
||||||
|
"ods": 2779,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 0,
|
||||||
|
"ods": 0,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "cost_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_goods_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale_ex",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 17563,
|
||||||
|
"ods": 17563,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "discount_money",
|
||||||
|
"dwd_sum": 21673.2,
|
||||||
|
"ods_sum": 21673.2,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_member_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money_cost",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "push_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 37,
|
||||||
|
"ods": 106,
|
||||||
|
"diff": -69
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 191.21,
|
||||||
|
"diff": -191.21
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 12505.5,
|
||||||
|
"ods_sum": 37535.53,
|
||||||
|
"diff": -25030.03
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4883,
|
||||||
|
"ods": 4883,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "manual_discount_amount",
|
||||||
|
"dwd_sum": 414.17,
|
||||||
|
"ods_sum": 414.17,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 97,
|
||||||
|
"ods": 97,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 97,
|
||||||
|
"ods": 97,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4677,
|
||||||
|
"ods": 4677,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change_ex",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4677,
|
||||||
|
"ods": 4677,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 266,
|
||||||
|
"ods": 266,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 18240.0,
|
||||||
|
"ods_sum": 18240.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 17057.33,
|
||||||
|
"ods_sum": 17057.33,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 11157,
|
||||||
|
"ods": 11157,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "assistant_promotion_money",
|
||||||
|
"dwd_sum": 7544.8,
|
||||||
|
"ods_sum": 7544.8,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "assistant_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "goods_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "recharge_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "reward_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "table_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 259,
|
||||||
|
"ods": 259,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 17394.0,
|
||||||
|
"ods_sum": 17394.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 16707,
|
||||||
|
"ods": 16707,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 453,
|
||||||
|
"ods": 453,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order_ex",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 453,
|
||||||
|
"ods": 453,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_payment",
|
||||||
|
"ods_table": "billiards_ods.payment_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 407,
|
||||||
|
"ods": 407,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": 80965.0,
|
||||||
|
"ods_sum": 80965.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 1,
|
||||||
|
"ods": 1,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "channel_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": -2.0,
|
||||||
|
"ods_sum": -2.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund_ex",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 44,
|
||||||
|
"ods": 44,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "balance_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "card_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "round_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_count_diff": -69
|
||||||
|
},
|
||||||
|
"generated_at": "2026-01-24T22:14:48.193821+08:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"api_to_ods": {
|
||||||
|
"total_missing": 0,
|
||||||
|
"total_errors": 23
|
||||||
|
},
|
||||||
|
"total_missing": 0,
|
||||||
|
"total_errors": 23,
|
||||||
|
"generated_at": "2026-01-24T22:14:48.193853+08:00"
|
||||||
|
}
|
||||||
1400
etl_billiards/reports/data_integrity_window_20260124_222607.json
Normal file
1400
etl_billiards/reports/data_integrity_window_20260124_222607.json
Normal file
File diff suppressed because it is too large
Load Diff
1340
etl_billiards/reports/data_integrity_window_20260125_205910.json
Normal file
1340
etl_billiards/reports/data_integrity_window_20260125_205910.json
Normal file
File diff suppressed because it is too large
Load Diff
1012
etl_billiards/reports/data_integrity_window_20260125_213632.json
Normal file
1012
etl_billiards/reports/data_integrity_window_20260125_213632.json
Normal file
File diff suppressed because it is too large
Load Diff
1066
etl_billiards/reports/data_integrity_window_20260125_224002.json
Normal file
1066
etl_billiards/reports/data_integrity_window_20260125_224002.json
Normal file
File diff suppressed because it is too large
Load Diff
1196
etl_billiards/reports/data_integrity_window_20260126_033329.json
Normal file
1196
etl_billiards/reports/data_integrity_window_20260126_033329.json
Normal file
File diff suppressed because it is too large
Load Diff
972
etl_billiards/reports/data_integrity_window_20260126_035534.json
Normal file
972
etl_billiards/reports/data_integrity_window_20260126_035534.json
Normal file
@@ -0,0 +1,972 @@
|
|||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-24T00:45:10+08:00",
|
||||||
|
"end": "2026-01-26T05:45:10+08:00",
|
||||||
|
"segments": 1
|
||||||
|
},
|
||||||
|
"windows": [
|
||||||
|
{
|
||||||
|
"mode": "window",
|
||||||
|
"window": {
|
||||||
|
"start": "2026-01-24T00:45:10+08:00",
|
||||||
|
"end": "2026-01-26T05:45:10+08:00",
|
||||||
|
"label": "segment_1",
|
||||||
|
"granularity": "window"
|
||||||
|
},
|
||||||
|
"api_to_ods": {
|
||||||
|
"window_split_unit": "none",
|
||||||
|
"window_compensation_hours": 0,
|
||||||
|
"start": "2026-01-24T00:45:10+08:00",
|
||||||
|
"end": "2026-01-26T05:45:10+08:00",
|
||||||
|
"cutoff": null,
|
||||||
|
"window_days": 2,
|
||||||
|
"window_hours": 0,
|
||||||
|
"page_size": 200,
|
||||||
|
"chunk_size": 500,
|
||||||
|
"sample_limit": 50,
|
||||||
|
"store_id": 2790685415443269,
|
||||||
|
"base_url": "https://pc.ficoo.vip/apiprod/admin/v1/",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ACCOUNT",
|
||||||
|
"table": "billiards_ods.assistant_accounts_master",
|
||||||
|
"endpoint": "/PersonnelManagement/SearchAssistantInfo",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 136,
|
||||||
|
"records_with_pk": 136,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_RECORDS",
|
||||||
|
"table": "billiards_ods.settlement_records",
|
||||||
|
"endpoint": "/Site/GetAllOrderSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 284,
|
||||||
|
"records_with_pk": 284,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 3,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_USE",
|
||||||
|
"table": "billiards_ods.table_fee_transactions",
|
||||||
|
"endpoint": "/Site/GetSiteTableOrderDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 10070,
|
||||||
|
"records_with_pk": 10070,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 51,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_LEDGER",
|
||||||
|
"table": "billiards_ods.assistant_service_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetOrderAssistantDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 83,
|
||||||
|
"records_with_pk": 83,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_ASSISTANT_ABOLISH",
|
||||||
|
"table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"endpoint": "/AssistantPerformance/GetAbolitionAssistant",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 1,
|
||||||
|
"records_with_pk": 1,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS_SALES",
|
||||||
|
"table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsSalesList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 0,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PAYMENT",
|
||||||
|
"table": "billiards_ods.payment_transactions",
|
||||||
|
"endpoint": "/PayLog/GetPayLogListPage",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 11926,
|
||||||
|
"records_with_pk": 11926,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 60,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_REFUND",
|
||||||
|
"table": "billiards_ods.refund_transactions",
|
||||||
|
"endpoint": "/Order/GetRefundPayLogList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 38,
|
||||||
|
"records_with_pk": 38,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_PLATFORM_COUPON",
|
||||||
|
"table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"endpoint": "/Promotion/GetOfflineCouponConsumePageList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 16953,
|
||||||
|
"records_with_pk": 16953,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 85,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER",
|
||||||
|
"table": "billiards_ods.member_profiles",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 556,
|
||||||
|
"records_with_pk": 556,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 3,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_CARD",
|
||||||
|
"table": "billiards_ods.member_stored_value_cards",
|
||||||
|
"endpoint": "/MemberProfile/GetTenantMemberCardList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 945,
|
||||||
|
"records_with_pk": 945,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 5,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_MEMBER_BALANCE",
|
||||||
|
"table": "billiards_ods.member_balance_changes",
|
||||||
|
"endpoint": "/MemberProfile/GetMemberCardBalanceChange",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 2472,
|
||||||
|
"records_with_pk": 2472,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 13,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_RECHARGE_SETTLE",
|
||||||
|
"table": "billiards_ods.recharge_settlements",
|
||||||
|
"endpoint": "/Site/GetRechargeSettleList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 1,
|
||||||
|
"records_with_pk": 1,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 2,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_PACKAGE",
|
||||||
|
"table": "billiards_ods.group_buy_packages",
|
||||||
|
"endpoint": "/PackageCoupon/QueryPackageCouponList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 18,
|
||||||
|
"records_with_pk": 18,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GROUP_BUY_REDEMPTION",
|
||||||
|
"table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"endpoint": "/Site/GetSiteTableUseDetails",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 8266,
|
||||||
|
"records_with_pk": 8266,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 42,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_STOCK",
|
||||||
|
"table": "billiards_ods.goods_stock_summary",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsStockReport",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsid"
|
||||||
|
],
|
||||||
|
"records": 170,
|
||||||
|
"records_with_pk": 170,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_INVENTORY_CHANGE",
|
||||||
|
"table": "billiards_ods.goods_stock_movements",
|
||||||
|
"endpoint": "/GoodsStockManage/QueryGoodsOutboundReceipt",
|
||||||
|
"pk_columns": [
|
||||||
|
"sitegoodsstockid"
|
||||||
|
],
|
||||||
|
"records": 548,
|
||||||
|
"records_with_pk": 548,
|
||||||
|
"missing": 1,
|
||||||
|
"missing_samples": [
|
||||||
|
{
|
||||||
|
"sitegoodsstockid": 3067169196329861
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"pages": 4,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLES",
|
||||||
|
"table": "billiards_ods.site_tables_master",
|
||||||
|
"endpoint": "/Table/GetSiteTables",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 74,
|
||||||
|
"records_with_pk": 74,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_GOODS_CATEGORY",
|
||||||
|
"table": "billiards_ods.stock_goods_category_tree",
|
||||||
|
"endpoint": "/TenantGoodsCategory/QueryPrimarySecondaryCategory",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 9,
|
||||||
|
"records_with_pk": 9,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_STORE_GOODS",
|
||||||
|
"table": "billiards_ods.store_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/GetGoodsInventoryList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 170,
|
||||||
|
"records_with_pk": 170,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TABLE_FEE_DISCOUNT",
|
||||||
|
"table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"endpoint": "/Site/GetTaiFeeAdjustList",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 1737,
|
||||||
|
"records_with_pk": 1737,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 9,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_TENANT_GOODS",
|
||||||
|
"table": "billiards_ods.tenant_goods_master",
|
||||||
|
"endpoint": "/TenantGoods/QueryTenantGoods",
|
||||||
|
"pk_columns": [
|
||||||
|
"id"
|
||||||
|
],
|
||||||
|
"records": 171,
|
||||||
|
"records_with_pk": 171,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 1,
|
||||||
|
"skipped_missing_pk": 0,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"task_code": "ODS_SETTLEMENT_TICKET",
|
||||||
|
"table": "billiards_ods.settlement_ticket_details",
|
||||||
|
"endpoint": "/Order/GetOrderSettleTicketNew",
|
||||||
|
"pk_columns": [
|
||||||
|
"ordersettleid"
|
||||||
|
],
|
||||||
|
"records": 286,
|
||||||
|
"records_with_pk": 0,
|
||||||
|
"missing": 0,
|
||||||
|
"missing_samples": [],
|
||||||
|
"pages": 3,
|
||||||
|
"skipped_missing_pk": 286,
|
||||||
|
"errors": 0,
|
||||||
|
"error_detail": null,
|
||||||
|
"source_endpoint": "/PayLog/GetPayLogListPage"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0,
|
||||||
|
"generated_at": "2026-01-26T03:55:28.356697+08:00"
|
||||||
|
},
|
||||||
|
"ods_to_dwd": {
|
||||||
|
"tables": [
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 23339,
|
||||||
|
"ods": 23339,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_settlement_head_ex",
|
||||||
|
"ods_table": "billiards_ods.settlement_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 23339,
|
||||||
|
"ods": 23339,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 254,
|
||||||
|
"ods": 254,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "adjust_amount",
|
||||||
|
"dwd_sum": 4468.13,
|
||||||
|
"ods_sum": 4468.13,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "coupon_promotion_amount",
|
||||||
|
"dwd_sum": 11343.14,
|
||||||
|
"ods_sum": 11343.14,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 29695.36,
|
||||||
|
"ods_sum": 29695.36,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 819.01,
|
||||||
|
"ods_sum": 819.01,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_table_charge_money",
|
||||||
|
"dwd_sum": 13065.08,
|
||||||
|
"ods_sum": 13065.08,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_log_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 18361,
|
||||||
|
"ods": 18361,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "fee_total",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "mgmt_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "used_card_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2837,
|
||||||
|
"ods": 2845,
|
||||||
|
"diff": -8
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 328358.71,
|
||||||
|
"ods_sum": 328901.21,
|
||||||
|
"diff": -542.5
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_table_fee_adjust_ex",
|
||||||
|
"ods_table": "billiards_ods.table_fee_discount_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 2837,
|
||||||
|
"ods": 2845,
|
||||||
|
"diff": -8
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 0,
|
||||||
|
"ods": 0,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "cost_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "real_goods_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_store_goods_sale_ex",
|
||||||
|
"ods_table": "billiards_ods.store_goods_sales_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 17563,
|
||||||
|
"ods": 17563,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "discount_money",
|
||||||
|
"dwd_sum": 21673.2,
|
||||||
|
"ods_sum": 21673.2,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "option_member_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "point_discount_money_cost",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "push_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 83,
|
||||||
|
"ods": 83,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_deduct_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 21036.95,
|
||||||
|
"ods_sum": 21036.95,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_service_log_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_service_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4998,
|
||||||
|
"ods": 4998,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "manual_discount_amount",
|
||||||
|
"dwd_sum": 414.17,
|
||||||
|
"ods_sum": 414.17,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "member_discount_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "service_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 98,
|
||||||
|
"ods": 98,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_assistant_trash_event_ex",
|
||||||
|
"ods_table": "billiards_ods.assistant_cancellation_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 98,
|
||||||
|
"ods": 98,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4735,
|
||||||
|
"ods": 4735,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_member_balance_change_ex",
|
||||||
|
"ods_table": "billiards_ods.member_balance_changes",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 4735,
|
||||||
|
"ods": 4735,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 146,
|
||||||
|
"ods": 169,
|
||||||
|
"diff": -23
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 10116.0,
|
||||||
|
"ods_sum": 11552.0,
|
||||||
|
"diff": -1436.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "ledger_amount",
|
||||||
|
"dwd_sum": 9907.14,
|
||||||
|
"ods_sum": 11343.14,
|
||||||
|
"diff": -1436.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_groupbuy_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.group_buy_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 11388,
|
||||||
|
"ods": 11411,
|
||||||
|
"diff": -23
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "assistant_promotion_money",
|
||||||
|
"dwd_sum": 7544.8,
|
||||||
|
"ods_sum": 7544.8,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "assistant_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "goods_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "recharge_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "reward_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "table_service_promotion_money",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "create_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 165,
|
||||||
|
"ods": 165,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "coupon_money",
|
||||||
|
"dwd_sum": 11216.0,
|
||||||
|
"ods_sum": 11216.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_platform_coupon_redemption_ex",
|
||||||
|
"ods_table": "billiards_ods.platform_coupon_redemption_records",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 16953,
|
||||||
|
"ods": 16953,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 454,
|
||||||
|
"ods": 454,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_recharge_order_ex",
|
||||||
|
"ods_table": "billiards_ods.recharge_settlements",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 454,
|
||||||
|
"ods": 454,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_payment",
|
||||||
|
"ods_table": "billiards_ods.payment_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 286,
|
||||||
|
"ods": 286,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": 25812.0,
|
||||||
|
"ods_sum": 25812.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": true,
|
||||||
|
"window_col": "pay_time",
|
||||||
|
"count": {
|
||||||
|
"dwd": 0,
|
||||||
|
"ods": 0,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "channel_fee",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "pay_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"dwd_table": "billiards_dwd.dwd_refund_ex",
|
||||||
|
"ods_table": "billiards_ods.refund_transactions",
|
||||||
|
"windowed": false,
|
||||||
|
"window_col": null,
|
||||||
|
"count": {
|
||||||
|
"dwd": 45,
|
||||||
|
"ods": 45,
|
||||||
|
"diff": 0
|
||||||
|
},
|
||||||
|
"amounts": [
|
||||||
|
{
|
||||||
|
"column": "balance_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "card_frozen_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "refund_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"column": "round_amount",
|
||||||
|
"dwd_sum": 0.0,
|
||||||
|
"ods_sum": 0.0,
|
||||||
|
"diff": 0.0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"total_count_diff": -62
|
||||||
|
},
|
||||||
|
"generated_at": "2026-01-26T03:55:34.698078+08:00"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"api_to_ods": {
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0
|
||||||
|
},
|
||||||
|
"total_missing": 1,
|
||||||
|
"total_errors": 0,
|
||||||
|
"generated_at": "2026-01-26T03:55:34.698101+08:00"
|
||||||
|
}
|
||||||
1151
etl_billiards/reports/data_integrity_window_20260126_182902.json
Normal file
1151
etl_billiards/reports/data_integrity_window_20260126_182902.json
Normal file
File diff suppressed because it is too large
Load Diff
1309
etl_billiards/reports/data_integrity_window_20260127_025004.json
Normal file
1309
etl_billiards/reports/data_integrity_window_20260127_025004.json
Normal file
File diff suppressed because it is too large
Load Diff
1322
etl_billiards/reports/data_integrity_window_20260127_034634.json
Normal file
1322
etl_billiards/reports/data_integrity_window_20260127_034634.json
Normal file
File diff suppressed because it is too large
Load Diff
1232
etl_billiards/reports/data_integrity_window_20260127_044633.json
Normal file
1232
etl_billiards/reports/data_integrity_window_20260127_044633.json
Normal file
File diff suppressed because it is too large
Load Diff
1206
etl_billiards/reports/data_integrity_window_20260127_054632.json
Normal file
1206
etl_billiards/reports/data_integrity_window_20260127_054632.json
Normal file
File diff suppressed because it is too large
Load Diff
1213
etl_billiards/reports/data_integrity_window_20260127_064633.json
Normal file
1213
etl_billiards/reports/data_integrity_window_20260127_064633.json
Normal file
File diff suppressed because it is too large
Load Diff
1198
etl_billiards/reports/data_integrity_window_20260127_074634.json
Normal file
1198
etl_billiards/reports/data_integrity_window_20260127_074634.json
Normal file
File diff suppressed because it is too large
Load Diff
1186
etl_billiards/reports/data_integrity_window_20260127_084637.json
Normal file
1186
etl_billiards/reports/data_integrity_window_20260127_084637.json
Normal file
File diff suppressed because it is too large
Load Diff
1186
etl_billiards/reports/data_integrity_window_20260127_094626.json
Normal file
1186
etl_billiards/reports/data_integrity_window_20260127_094626.json
Normal file
File diff suppressed because it is too large
Load Diff
1214
etl_billiards/reports/data_integrity_window_20260127_104627.json
Normal file
1214
etl_billiards/reports/data_integrity_window_20260127_104627.json
Normal file
File diff suppressed because it is too large
Load Diff
1186
etl_billiards/reports/data_integrity_window_20260127_114628.json
Normal file
1186
etl_billiards/reports/data_integrity_window_20260127_114628.json
Normal file
File diff suppressed because it is too large
Load Diff
1217
etl_billiards/reports/data_integrity_window_20260127_124632.json
Normal file
1217
etl_billiards/reports/data_integrity_window_20260127_124632.json
Normal file
File diff suppressed because it is too large
Load Diff
1190
etl_billiards/reports/data_integrity_window_20260127_134628.json
Normal file
1190
etl_billiards/reports/data_integrity_window_20260127_134628.json
Normal file
File diff suppressed because it is too large
Load Diff
1231
etl_billiards/reports/data_integrity_window_20260127_144641.json
Normal file
1231
etl_billiards/reports/data_integrity_window_20260127_144641.json
Normal file
File diff suppressed because it is too large
Load Diff
1250
etl_billiards/reports/data_integrity_window_20260127_154658.json
Normal file
1250
etl_billiards/reports/data_integrity_window_20260127_154658.json
Normal file
File diff suppressed because it is too large
Load Diff
1279
etl_billiards/reports/data_integrity_window_20260127_164657.json
Normal file
1279
etl_billiards/reports/data_integrity_window_20260127_164657.json
Normal file
File diff suppressed because it is too large
Load Diff
1311
etl_billiards/reports/data_integrity_window_20260127_174656.json
Normal file
1311
etl_billiards/reports/data_integrity_window_20260127_174656.json
Normal file
File diff suppressed because it is too large
Load Diff
1306
etl_billiards/reports/data_integrity_window_20260127_184644.json
Normal file
1306
etl_billiards/reports/data_integrity_window_20260127_184644.json
Normal file
File diff suppressed because it is too large
Load Diff
1291
etl_billiards/reports/data_integrity_window_20260127_194657.json
Normal file
1291
etl_billiards/reports/data_integrity_window_20260127_194657.json
Normal file
File diff suppressed because it is too large
Load Diff
1426
etl_billiards/reports/data_integrity_window_20260127_204701.json
Normal file
1426
etl_billiards/reports/data_integrity_window_20260127_204701.json
Normal file
File diff suppressed because it is too large
Load Diff
1494
etl_billiards/reports/data_integrity_window_20260127_214717.json
Normal file
1494
etl_billiards/reports/data_integrity_window_20260127_214717.json
Normal file
File diff suppressed because it is too large
Load Diff
1447
etl_billiards/reports/data_integrity_window_20260127_221542.json
Normal file
1447
etl_billiards/reports/data_integrity_window_20260127_221542.json
Normal file
File diff suppressed because it is too large
Load Diff
@@ -3,3 +3,10 @@ psycopg2-binary>=2.9.0
|
|||||||
requests>=2.28.0
|
requests>=2.28.0
|
||||||
python-dateutil>=2.8.0
|
python-dateutil>=2.8.0
|
||||||
tzdata>=2023.0
|
tzdata>=2023.0
|
||||||
|
flask>=2.3
|
||||||
|
|
||||||
|
# GUI 依赖
|
||||||
|
PySide6>=6.5.0
|
||||||
|
|
||||||
|
# 打包工具 (可选,仅打包 EXE 时需要)
|
||||||
|
# pyinstaller>=6.0.0
|
||||||
|
|||||||
27
etl_billiards/run_gui.bat
Normal file
27
etl_billiards/run_gui.bat
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
@echo off
|
||||||
|
chcp 65001 >nul
|
||||||
|
cd /d "%~dp0"
|
||||||
|
|
||||||
|
echo ====================================
|
||||||
|
echo 飞球 ETL 管理系统
|
||||||
|
echo ====================================
|
||||||
|
echo.
|
||||||
|
|
||||||
|
REM 检查 Python
|
||||||
|
python --version >nul 2>&1
|
||||||
|
if errorlevel 1 (
|
||||||
|
echo [错误] 未找到 Python,请先安装 Python 3.10+
|
||||||
|
pause
|
||||||
|
exit /b 1
|
||||||
|
)
|
||||||
|
|
||||||
|
REM 启动 GUI
|
||||||
|
echo 正在启动 GUI...
|
||||||
|
python -m gui.main
|
||||||
|
|
||||||
|
if errorlevel 1 (
|
||||||
|
echo.
|
||||||
|
echo [错误] 启动失败,请检查依赖是否已安装
|
||||||
|
echo 运行: pip install -r requirements.txt
|
||||||
|
pause
|
||||||
|
)
|
||||||
42
etl_billiards/run_gui.ps1
Normal file
42
etl_billiards/run_gui.ps1
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# PowerShell 启动脚本
|
||||||
|
# 飞球 ETL 管理系统
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$OutputEncoding = [System.Text.Encoding]::UTF8
|
||||||
|
|
||||||
|
Write-Host "===================================="
|
||||||
|
Write-Host " 飞球 ETL 管理系统"
|
||||||
|
Write-Host "===================================="
|
||||||
|
Write-Host ""
|
||||||
|
|
||||||
|
# 切换到脚本目录
|
||||||
|
Set-Location $PSScriptRoot
|
||||||
|
|
||||||
|
# 检查 Python
|
||||||
|
try {
|
||||||
|
$pythonVersion = python --version 2>&1
|
||||||
|
Write-Host "Python 版本: $pythonVersion"
|
||||||
|
} catch {
|
||||||
|
Write-Host "[错误] 未找到 Python,请先安装 Python 3.10+" -ForegroundColor Red
|
||||||
|
Read-Host "按回车键退出"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# 检查 PySide6
|
||||||
|
$hasPySide6 = python -c "import PySide6" 2>&1
|
||||||
|
if ($LASTEXITCODE -ne 0) {
|
||||||
|
Write-Host "[提示] 正在安装 GUI 依赖..." -ForegroundColor Yellow
|
||||||
|
pip install PySide6
|
||||||
|
}
|
||||||
|
|
||||||
|
# 启动 GUI
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "正在启动 GUI..." -ForegroundColor Cyan
|
||||||
|
python -m gui.main
|
||||||
|
|
||||||
|
if ($LASTEXITCODE -ne 0) {
|
||||||
|
Write-Host ""
|
||||||
|
Write-Host "[错误] 启动失败" -ForegroundColor Red
|
||||||
|
Write-Host "请运行: pip install -r requirements.txt"
|
||||||
|
Read-Host "按回车键退出"
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@ setlocal
|
|||||||
cd /d %~dp0
|
cd /d %~dp0
|
||||||
|
|
||||||
REM 如果需要覆盖示例目录,可修改下面的 INGEST_DIR
|
REM 如果需要覆盖示例目录,可修改下面的 INGEST_DIR
|
||||||
set "INGEST_DIR=C:\dev\LLTQ\export\test-json-doc"
|
set "INGEST_DIR=export\\test-json-doc"
|
||||||
|
|
||||||
echo [INIT_ODS_SCHEMA] 准备执行,源目录=%INGEST_DIR%
|
echo [INIT_ODS_SCHEMA] 准备执行,源目录=%INGEST_DIR%
|
||||||
python -m cli.main --tasks INIT_ODS_SCHEMA --pipeline-flow INGEST_ONLY --ingest-source "%INGEST_DIR%"
|
python -m cli.main --tasks INIT_ODS_SCHEMA --pipeline-flow INGEST_ONLY --ingest-source "%INGEST_DIR%"
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user