在准备环境前提交次全部更改。

This commit is contained in:
Neo
2026-02-19 08:35:13 +08:00
parent ded6dfb9d8
commit 4eac07da47
1387 changed files with 6107191 additions and 33002 deletions

View File

@@ -1,6 +1,125 @@
# etl_feiqiu 六层 Schema 迁移文档
# etl_feiqiu Schema 迁移文档
## 变更说明
---
## 迁移 2ODS "取最新版本"复合索引2026-02-17
### 迁移文件
`db/etl_feiqiu/migrations/2026-02-17__add_ods_latest_version_indexes.sql`
### 变更说明
为全部 23 张 ODS 表添加 `(业务主键, fetched_at DESC)` 复合索引,支持 `DISTINCT ON (pk) ORDER BY pk, fetched_at DESC` 查询模式高效取每条业务记录的最新版本。
索引命名规范:`idx_ods_{table_name}_latest`
| # | 表名 | 业务主键列 | 索引名 |
|---|------|-----------|--------|
| 1 | assistant_accounts_master | id | idx_ods_assistant_accounts_master_latest |
| 2 | settlement_records | id | idx_ods_settlement_records_latest |
| 3 | table_fee_transactions | id | idx_ods_table_fee_transactions_latest |
| 4 | assistant_service_records | id | idx_ods_assistant_service_records_latest |
| 5 | assistant_cancellation_records | id | idx_ods_assistant_cancellation_records_latest |
| 6 | store_goods_sales_records | id | idx_ods_store_goods_sales_records_latest |
| 7 | payment_transactions | id | idx_ods_payment_transactions_latest |
| 8 | refund_transactions | id | idx_ods_refund_transactions_latest |
| 9 | platform_coupon_redemption_records | id | idx_ods_platform_coupon_redemption_records_latest |
| 10 | member_profiles | id | idx_ods_member_profiles_latest |
| 11 | member_stored_value_cards | id | idx_ods_member_stored_value_cards_latest |
| 12 | member_balance_changes | id | idx_ods_member_balance_changes_latest |
| 13 | recharge_settlements | id | idx_ods_recharge_settlements_latest |
| 14 | group_buy_packages | id | idx_ods_group_buy_packages_latest |
| 15 | group_buy_redemption_records | id | idx_ods_group_buy_redemption_records_latest |
| 16 | goods_stock_summary | siteGoodsId | idx_ods_goods_stock_summary_latest |
| 17 | goods_stock_movements | siteGoodsStockId | idx_ods_goods_stock_movements_latest |
| 18 | site_tables_master | id | idx_ods_site_tables_master_latest |
| 19 | stock_goods_category_tree | id | idx_ods_stock_goods_category_tree_latest |
| 20 | store_goods_master | id | idx_ods_store_goods_master_latest |
| 21 | table_fee_discount_records | id | idx_ods_table_fee_discount_records_latest |
| 22 | tenant_goods_master | id | idx_ods_tenant_goods_master_latest |
| 23 | settlement_ticket_details | orderSettleId | idx_ods_settlement_ticket_details_latest |
### 关联需求
`ods-dedup-standardize` Requirements 6.1, 6.2, 6.3
### 兼容性
- **非破坏性变更**:仅新增索引,不修改表结构、不影响现有数据和写入逻辑
- **ETL Connector**:无需改动;索引自动加速 `skip_unchanged` 去重查询和 `_mark_missing_as_deleted` 快照对比查询
- **后端 API / 小程序**:不受影响(不直接查询 ODS 层)
- **`CREATE INDEX CONCURRENTLY`**:在线创建,不阻塞表的读写操作;但不能在事务块内执行,需逐条运行或使用支持单语句模式的工具
- **DDL 源文件**:索引定义已同步写入 `db/etl_feiqiu/schemas/ods.sql`,新环境初始化时自动创建
### 回滚策略
逐条删除索引即可,不影响数据:
```sql
DROP INDEX IF EXISTS ods.idx_ods_assistant_accounts_master_latest;
DROP INDEX IF EXISTS ods.idx_ods_settlement_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_table_fee_transactions_latest;
DROP INDEX IF EXISTS ods.idx_ods_assistant_service_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_assistant_cancellation_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_store_goods_sales_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_payment_transactions_latest;
DROP INDEX IF EXISTS ods.idx_ods_refund_transactions_latest;
DROP INDEX IF EXISTS ods.idx_ods_platform_coupon_redemption_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_member_profiles_latest;
DROP INDEX IF EXISTS ods.idx_ods_member_stored_value_cards_latest;
DROP INDEX IF EXISTS ods.idx_ods_member_balance_changes_latest;
DROP INDEX IF EXISTS ods.idx_ods_recharge_settlements_latest;
DROP INDEX IF EXISTS ods.idx_ods_group_buy_packages_latest;
DROP INDEX IF EXISTS ods.idx_ods_group_buy_redemption_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_summary_latest;
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_movements_latest;
DROP INDEX IF EXISTS ods.idx_ods_site_tables_master_latest;
DROP INDEX IF EXISTS ods.idx_ods_stock_goods_category_tree_latest;
DROP INDEX IF EXISTS ods.idx_ods_store_goods_master_latest;
DROP INDEX IF EXISTS ods.idx_ods_table_fee_discount_records_latest;
DROP INDEX IF EXISTS ods.idx_ods_tenant_goods_master_latest;
DROP INDEX IF EXISTS ods.idx_ods_settlement_ticket_details_latest;
```
### 验证 SQL
```sql
-- 1. 验证 23 个索引均已创建
SELECT indexname, tablename
FROM pg_indexes
WHERE schemaname = 'ods'
AND indexname LIKE 'idx_ods_%_latest'
ORDER BY indexname;
-- 2. 验证索引数量为 23
SELECT COUNT(*) AS index_count
FROM pg_indexes
WHERE schemaname = 'ods'
AND indexname LIKE 'idx_ods_%_latest';
-- 3. 验证索引列定义正确(以 member_profiles 为例)
SELECT indexdef
FROM pg_indexes
WHERE schemaname = 'ods'
AND indexname = 'idx_ods_member_profiles_latest';
-- 4. 验证索引可用(非 INVALID 状态)
SELECT c.relname AS index_name, i.indisvalid
FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'ods'
AND c.relname LIKE 'idx_ods_%_latest'
AND i.indisvalid = false;
-- 预期结果0 行(无无效索引)
```
---
## 迁移 1六层 Schema 架构重组
### 变更说明
将现有 4 个 schema 重组为 6 层 schema 架构:
@@ -38,8 +157,9 @@
## 兼容性
- **ETL 管线**:需更新 ETL 代码中的 schema 引用etl_admin -> meta, billiards_ods -> ods
- **后端 API**:通过 app schema 视图访问,无需直接引用底层表
- **ETL Connector**:所有代码中的 schema 引用已更新完成etl_admin meta, billiards_ods ods, billiards_dwd → dwd, billiards_dws → dws
- **后端 API**etl_status 路由已更新为 meta.etl_cursor通过 app schema 视图访问,无需直接引用底层表
- **管理后台**:已由 `apps/admin-web/` Web 管理后台完全替代原 PySide6 桌面 GUI通过后端 API 访问数据
- **小程序**:通过 FDW 映射 app schema不受影响
## 回滚策略