在准备环境前提交次全部更改。
This commit is contained in:
@@ -1,6 +1,125 @@
|
||||
# etl_feiqiu 六层 Schema 迁移文档
|
||||
# etl_feiqiu Schema 迁移文档
|
||||
|
||||
## 变更说明
|
||||
---
|
||||
|
||||
## 迁移 2:ODS "取最新版本"复合索引(2026-02-17)
|
||||
|
||||
### 迁移文件
|
||||
|
||||
`db/etl_feiqiu/migrations/2026-02-17__add_ods_latest_version_indexes.sql`
|
||||
|
||||
### 变更说明
|
||||
|
||||
为全部 23 张 ODS 表添加 `(业务主键, fetched_at DESC)` 复合索引,支持 `DISTINCT ON (pk) ORDER BY pk, fetched_at DESC` 查询模式高效取每条业务记录的最新版本。
|
||||
|
||||
索引命名规范:`idx_ods_{table_name}_latest`
|
||||
|
||||
| # | 表名 | 业务主键列 | 索引名 |
|
||||
|---|------|-----------|--------|
|
||||
| 1 | assistant_accounts_master | id | idx_ods_assistant_accounts_master_latest |
|
||||
| 2 | settlement_records | id | idx_ods_settlement_records_latest |
|
||||
| 3 | table_fee_transactions | id | idx_ods_table_fee_transactions_latest |
|
||||
| 4 | assistant_service_records | id | idx_ods_assistant_service_records_latest |
|
||||
| 5 | assistant_cancellation_records | id | idx_ods_assistant_cancellation_records_latest |
|
||||
| 6 | store_goods_sales_records | id | idx_ods_store_goods_sales_records_latest |
|
||||
| 7 | payment_transactions | id | idx_ods_payment_transactions_latest |
|
||||
| 8 | refund_transactions | id | idx_ods_refund_transactions_latest |
|
||||
| 9 | platform_coupon_redemption_records | id | idx_ods_platform_coupon_redemption_records_latest |
|
||||
| 10 | member_profiles | id | idx_ods_member_profiles_latest |
|
||||
| 11 | member_stored_value_cards | id | idx_ods_member_stored_value_cards_latest |
|
||||
| 12 | member_balance_changes | id | idx_ods_member_balance_changes_latest |
|
||||
| 13 | recharge_settlements | id | idx_ods_recharge_settlements_latest |
|
||||
| 14 | group_buy_packages | id | idx_ods_group_buy_packages_latest |
|
||||
| 15 | group_buy_redemption_records | id | idx_ods_group_buy_redemption_records_latest |
|
||||
| 16 | goods_stock_summary | siteGoodsId | idx_ods_goods_stock_summary_latest |
|
||||
| 17 | goods_stock_movements | siteGoodsStockId | idx_ods_goods_stock_movements_latest |
|
||||
| 18 | site_tables_master | id | idx_ods_site_tables_master_latest |
|
||||
| 19 | stock_goods_category_tree | id | idx_ods_stock_goods_category_tree_latest |
|
||||
| 20 | store_goods_master | id | idx_ods_store_goods_master_latest |
|
||||
| 21 | table_fee_discount_records | id | idx_ods_table_fee_discount_records_latest |
|
||||
| 22 | tenant_goods_master | id | idx_ods_tenant_goods_master_latest |
|
||||
| 23 | settlement_ticket_details | orderSettleId | idx_ods_settlement_ticket_details_latest |
|
||||
|
||||
### 关联需求
|
||||
|
||||
`ods-dedup-standardize` Requirements 6.1, 6.2, 6.3
|
||||
|
||||
### 兼容性
|
||||
|
||||
- **非破坏性变更**:仅新增索引,不修改表结构、不影响现有数据和写入逻辑
|
||||
- **ETL Connector**:无需改动;索引自动加速 `skip_unchanged` 去重查询和 `_mark_missing_as_deleted` 快照对比查询
|
||||
- **后端 API / 小程序**:不受影响(不直接查询 ODS 层)
|
||||
- **`CREATE INDEX CONCURRENTLY`**:在线创建,不阻塞表的读写操作;但不能在事务块内执行,需逐条运行或使用支持单语句模式的工具
|
||||
- **DDL 源文件**:索引定义已同步写入 `db/etl_feiqiu/schemas/ods.sql`,新环境初始化时自动创建
|
||||
|
||||
### 回滚策略
|
||||
|
||||
逐条删除索引即可,不影响数据:
|
||||
|
||||
```sql
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_accounts_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_settlement_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_service_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_cancellation_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_sales_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_payment_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_refund_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_platform_coupon_redemption_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_profiles_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_stored_value_cards_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_balance_changes_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_recharge_settlements_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_packages_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_redemption_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_summary_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_movements_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_site_tables_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_stock_goods_category_tree_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_discount_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_tenant_goods_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_settlement_ticket_details_latest;
|
||||
```
|
||||
|
||||
### 验证 SQL
|
||||
|
||||
```sql
|
||||
-- 1. 验证 23 个索引均已创建
|
||||
SELECT indexname, tablename
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname LIKE 'idx_ods_%_latest'
|
||||
ORDER BY indexname;
|
||||
|
||||
-- 2. 验证索引数量为 23
|
||||
SELECT COUNT(*) AS index_count
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname LIKE 'idx_ods_%_latest';
|
||||
|
||||
-- 3. 验证索引列定义正确(以 member_profiles 为例)
|
||||
SELECT indexdef
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname = 'idx_ods_member_profiles_latest';
|
||||
|
||||
-- 4. 验证索引可用(非 INVALID 状态)
|
||||
SELECT c.relname AS index_name, i.indisvalid
|
||||
FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'ods'
|
||||
AND c.relname LIKE 'idx_ods_%_latest'
|
||||
AND i.indisvalid = false;
|
||||
-- 预期结果:0 行(无无效索引)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 迁移 1:六层 Schema 架构重组
|
||||
|
||||
### 变更说明
|
||||
|
||||
将现有 4 个 schema 重组为 6 层 schema 架构:
|
||||
|
||||
@@ -38,8 +157,9 @@
|
||||
|
||||
## 兼容性
|
||||
|
||||
- **ETL 管线**:需更新 ETL 代码中的 schema 引用(etl_admin -> meta, billiards_ods -> ods 等)
|
||||
- **后端 API**:通过 app schema 视图访问,无需直接引用底层表
|
||||
- **ETL Connector**:所有代码中的 schema 引用已更新完成(etl_admin → meta, billiards_ods → ods, billiards_dwd → dwd, billiards_dws → dws)
|
||||
- **后端 API**:etl_status 路由已更新为 meta.etl_cursor;通过 app schema 视图访问,无需直接引用底层表
|
||||
- **管理后台**:已由 `apps/admin-web/` Web 管理后台完全替代原 PySide6 桌面 GUI,通过后端 API 访问数据
|
||||
- **小程序**:通过 FDW 映射 app schema,不受影响
|
||||
|
||||
## 回滚策略
|
||||
|
||||
Reference in New Issue
Block a user