ETL 完成

This commit is contained in:
Neo
2026-01-18 22:37:38 +08:00
parent 8da6cb6563
commit 7ca19a4a2c
159 changed files with 31225 additions and 467 deletions

View File

@@ -0,0 +1,34 @@
在线抓取更新ODS 然后将更新的ODS内容对应到DWD的更新。
可以按“两段定时”跑:先在线抓取+入库更新 ODS再跑 DWD_LOAD_FROM_ODS 把新增/变更同步到 DWD。CLI 用 python -m etl_billiards.cli.main。
1) ODS在线抓取 + 入库FULL
python -m etl_billiards.cli.main ^
--pipeline-flow FULL ^
--tasks PRODUCTS,TABLES,MEMBERS,ASSISTANTS,PACKAGES_DEF,ORDERS,PAYMENTS,REFUNDS,COUPON_USAGE,INVENTORY_CHANGE,TOPUPS,TABLE_DISCOUNT,ASSISTANT_ABOLISH,LEDGER ^
--pg-dsn "%PG_DSN%" ^
--store-id %STORE_ID% ^
--api-token "%API_TOKEN%"
(可选)指定落盘目录:加 --fetch-root "D:\etl\json";美化 JSON--write-pretty-json
2) DWDODS → DWD
python -m etl_billiards.cli.main ^
--pipeline-flow INGEST_ONLY ^
--tasks DWD_LOAD_FROM_ODS ^
--pg-dsn "%PG_DSN%" ^
--store-id %STORE_ID%
推荐的环境变量
PG_DSN=postgresql://user:pwd@host:5432/db
STORE_ID=...
API_TOKEN=...
可选JSON_FETCH_ROOT=... / FETCH_ROOT=...LOG_ROOT=...
如果你希望“一条命令顺序跑完 ODS+DWD”也可以直接
python -m etl_billiards.cli.main ^
--pipeline-flow FULL ^
--tasks PRODUCTS,TABLES,MEMBERS,ASSISTANTS,PACKAGES_DEF,ORDERS,PAYMENTS,REFUNDS,COUPON_USAGE,INVENTORY_CHANGE,TOPUPS,TABLE_DISCOUNT,ASSISTANT_ABOLISH,LEDGER,DWD_LOAD_FROM_ODS ^
--pg-dsn "%PG_DSN%" ^
--store-id %STORE_ID% ^
--api-token "%API_TOKEN%"
(这会对前半段任务走在线抓取+入库,对 DWD_LOAD_FROM_ODS 只做入库阶段,因为它没有抓取逻辑。)