193 lines
8.2 KiB
Markdown
193 lines
8.2 KiB
Markdown
# etl_feiqiu Schema 迁移文档
|
||
|
||
---
|
||
|
||
## 迁移 2：ODS "取最新版本"复合索引（2026-02-17）
|
||
|
||
### 迁移文件
|
||
|
||
`db/etl_feiqiu/migrations/2026-02-17__add_ods_latest_version_indexes.sql`
|
||
|
||
### 变更说明
|
||
|
||
为全部 23 张 ODS 表添加 `(业务主键, fetched_at DESC)` 复合索引，支持 `DISTINCT ON (pk) ORDER BY pk, fetched_at DESC` 查询模式高效取每条业务记录的最新版本。
|
||
|
||
索引命名规范：`idx_ods_{table_name}_latest`
|
||
|
||
| # | 表名 | 业务主键列 | 索引名 |
|
||
|---|------|-----------|--------|
|
||
| 1 | assistant_accounts_master | id | idx_ods_assistant_accounts_master_latest |
|
||
| 2 | settlement_records | id | idx_ods_settlement_records_latest |
|
||
| 3 | table_fee_transactions | id | idx_ods_table_fee_transactions_latest |
|
||
| 4 | assistant_service_records | id | idx_ods_assistant_service_records_latest |
|
||
| 5 | assistant_cancellation_records | id | idx_ods_assistant_cancellation_records_latest |
|
||
| 6 | store_goods_sales_records | id | idx_ods_store_goods_sales_records_latest |
|
||
| 7 | payment_transactions | id | idx_ods_payment_transactions_latest |
|
||
| 8 | refund_transactions | id | idx_ods_refund_transactions_latest |
|
||
| 9 | platform_coupon_redemption_records | id | idx_ods_platform_coupon_redemption_records_latest |
|
||
| 10 | member_profiles | id | idx_ods_member_profiles_latest |
|
||
| 11 | member_stored_value_cards | id | idx_ods_member_stored_value_cards_latest |
|
||
| 12 | member_balance_changes | id | idx_ods_member_balance_changes_latest |
|
||
| 13 | recharge_settlements | id | idx_ods_recharge_settlements_latest |
|
||
| 14 | group_buy_packages | id | idx_ods_group_buy_packages_latest |
|
||
| 15 | group_buy_redemption_records | id | idx_ods_group_buy_redemption_records_latest |
|
||
| 16 | goods_stock_summary | siteGoodsId | idx_ods_goods_stock_summary_latest |
|
||
| 17 | goods_stock_movements | siteGoodsStockId | idx_ods_goods_stock_movements_latest |
|
||
| 18 | site_tables_master | id | idx_ods_site_tables_master_latest |
|
||
| 19 | stock_goods_category_tree | id | idx_ods_stock_goods_category_tree_latest |
|
||
| 20 | store_goods_master | id | idx_ods_store_goods_master_latest |
|
||
| 21 | table_fee_discount_records | id | idx_ods_table_fee_discount_records_latest |
|
||
| 22 | tenant_goods_master | id | idx_ods_tenant_goods_master_latest |
|
||
| 23 | settlement_ticket_details | orderSettleId | idx_ods_settlement_ticket_details_latest |
|
||
|
||
### 关联需求
|
||
|
||
`ods-dedup-standardize` Requirements 6.1, 6.2, 6.3
|
||
|
||
### 兼容性
|
||
|
||
- **非破坏性变更**：仅新增索引，不修改表结构、不影响现有数据和写入逻辑
|
||
- **ETL Connector**：无需改动；索引自动加速 `skip_unchanged` 去重查询和 `_mark_missing_as_deleted` 快照对比查询
|
||
- **后端 API / 小程序**：不受影响（不直接查询 ODS 层）
|
||
- **`CREATE INDEX CONCURRENTLY`**：在线创建，不阻塞表的读写操作；但不能在事务块内执行，需逐条运行或使用支持单语句模式的工具
|
||
- **DDL 源文件**：索引定义已同步写入 `db/etl_feiqiu/schemas/ods.sql`，新环境初始化时自动创建
|
||
|
||
### 回滚策略
|
||
|
||
逐条删除索引即可，不影响数据：
|
||
|
||
```sql
|
||
DROP INDEX IF EXISTS ods.idx_ods_assistant_accounts_master_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_settlement_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_transactions_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_assistant_service_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_assistant_cancellation_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_sales_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_payment_transactions_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_refund_transactions_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_platform_coupon_redemption_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_member_profiles_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_member_stored_value_cards_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_member_balance_changes_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_recharge_settlements_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_packages_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_redemption_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_summary_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_movements_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_site_tables_master_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_stock_goods_category_tree_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_master_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_discount_records_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_tenant_goods_master_latest;
|
||
DROP INDEX IF EXISTS ods.idx_ods_settlement_ticket_details_latest;
|
||
```
|
||
|
||
### 验证 SQL
|
||
|
||
```sql
|
||
-- 1. 验证 23 个索引均已创建
|
||
SELECT indexname, tablename
|
||
FROM pg_indexes
|
||
WHERE schemaname = 'ods'
|
||
AND indexname LIKE 'idx_ods_%_latest'
|
||
ORDER BY indexname;
|
||
|
||
-- 2. 验证索引数量为 23
|
||
SELECT COUNT(*) AS index_count
|
||
FROM pg_indexes
|
||
WHERE schemaname = 'ods'
|
||
AND indexname LIKE 'idx_ods_%_latest';
|
||
|
||
-- 3. 验证索引列定义正确（以 member_profiles 为例）
|
||
SELECT indexdef
|
||
FROM pg_indexes
|
||
WHERE schemaname = 'ods'
|
||
AND indexname = 'idx_ods_member_profiles_latest';
|
||
|
||
-- 4. 验证索引可用（非 INVALID 状态）
|
||
SELECT c.relname AS index_name, i.indisvalid
|
||
FROM pg_index i
|
||
JOIN pg_class c ON c.oid = i.indexrelid
|
||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||
WHERE n.nspname = 'ods'
|
||
AND c.relname LIKE 'idx_ods_%_latest'
|
||
AND i.indisvalid = false;
|
||
-- 预期结果：0 行（无无效索引）
|
||
```
|
||
|
||
---
|
||
|
||
## 迁移 1：六层 Schema 架构重组
|
||
|
||
### 变更说明
|
||
|
||
将现有 4 个 schema 重组为 6 层 schema 架构：
|
||
|
||
| 原 Schema | 新 Schema | 文件 | 说明 |
|
||
|-----------|-----------|------|------|
|
||
| etl_admin | meta | meta.sql | 调度、游标、运行记录（3 表） |
|
||
| billiards_ods | ods | ods.sql | ODS 原始数据（23 表） |
|
||
| billiards_dwd | dwd | dwd.sql | DWD 明细，保留 main+EX 拆分（40 表） |
|
||
| （新增） | core | core.sql | 统一维度/事实最小字段集（7 表） |
|
||
| billiards_dws | dws | dws.sql | DWS 汇总（29 表） |
|
||
| （新增） | app | app.sql | 视图+RLS（7 视图，6 策略） |
|
||
|
||
### 新增表（core schema）
|
||
- core.dim_site — 门店维度核心字段
|
||
- core.dim_member — 会员维度核心字段
|
||
- core.dim_assistant — 助教维度核心字段
|
||
- core.dim_table — 台桌维度核心字段
|
||
- core.dim_goods_category — 商品分类维度核心字段
|
||
- core.fact_settlement — 结算事实核心字段
|
||
- core.fact_payment — 支付事实核心字段
|
||
|
||
### 新增视图（app schema）
|
||
- pp.v_site — 门店视图
|
||
- pp.v_member — 会员视图
|
||
- pp.v_assistant — 助教视图
|
||
- pp.v_assistant_daily — 助教日明细视图
|
||
- pp.v_finance_daily — 财务日报视图
|
||
- pp.v_member_consumption — 会员消费汇总视图
|
||
- pp.v_order_summary — 订单汇总视图
|
||
|
||
### RLS 策略
|
||
- 所有 core 表启用 ROW LEVEL SECURITY
|
||
- 策略基于 current_setting('app.current_site_id')::bigint 过滤
|
||
- 角色 pp_reader 仅有 SELECT 权限
|
||
|
||
## 兼容性
|
||
|
||
- **ETL Connector**：所有代码中的 schema 引用已更新完成（etl_admin → meta, billiards_ods → ods, billiards_dwd → dwd, billiards_dws → dws）
|
||
- **后端 API**：etl_status 路由已更新为 meta.etl_cursor；通过 app schema 视图访问，无需直接引用底层表
|
||
- **管理后台**：已由 `apps/admin-web/` Web 管理后台完全替代原 PySide6 桌面 GUI，通过后端 API 访问数据
|
||
- **小程序**：通过 FDW 映射 app schema，不受影响
|
||
|
||
## 回滚策略
|
||
|
||
1. 删除新 schema：DROP SCHEMA IF EXISTS meta, ods, dwd, core, dws, app CASCADE;
|
||
2. 重建原 schema：执行原始 schema_etl_admin.sql、schema_ODS_doc.sql、schema_dwd_doc.sql、schema_dws.sql
|
||
3. 原始 DDL 文件保留在 db/etl_feiqiu/schemas/schema_*.sql 作为参考
|
||
|
||
## 验证 SQL
|
||
|
||
`sql
|
||
-- 1. 验证六个 schema 均已创建
|
||
SELECT schema_name FROM information_schema.schemata
|
||
WHERE schema_name IN ('meta', 'ods', 'dwd', 'core', 'dws', 'app')
|
||
ORDER BY schema_name;
|
||
|
||
-- 2. 验证各 schema 表数量
|
||
SELECT table_schema, COUNT(*) AS table_count
|
||
FROM information_schema.tables
|
||
WHERE table_schema IN ('meta', 'ods', 'dwd', 'core', 'dws', 'app')
|
||
GROUP BY table_schema ORDER BY table_schema;
|
||
|
||
-- 3. 验证 RLS 策略已启用
|
||
SELECT schemaname, tablename, rowsecurity
|
||
FROM pg_tables
|
||
WHERE schemaname = 'core' AND rowsecurity = true;
|
||
|
||
-- 4. 验证 app_reader 角色存在
|
||
SELECT rolname FROM pg_roles WHERE rolname = 'app_reader';
|
||
`
|