在准备环境前提交次全部更改。
This commit is contained in:
@@ -1,6 +1,125 @@
|
||||
# etl_feiqiu 六层 Schema 迁移文档
|
||||
# etl_feiqiu Schema 迁移文档
|
||||
|
||||
## 变更说明
|
||||
---
|
||||
|
||||
## 迁移 2：ODS "取最新版本"复合索引（2026-02-17）
|
||||
|
||||
### 迁移文件
|
||||
|
||||
`db/etl_feiqiu/migrations/2026-02-17__add_ods_latest_version_indexes.sql`
|
||||
|
||||
### 变更说明
|
||||
|
||||
为全部 23 张 ODS 表添加 `(业务主键, fetched_at DESC)` 复合索引，支持 `DISTINCT ON (pk) ORDER BY pk, fetched_at DESC` 查询模式高效取每条业务记录的最新版本。
|
||||
|
||||
索引命名规范：`idx_ods_{table_name}_latest`
|
||||
|
||||
| # | 表名 | 业务主键列 | 索引名 |
|
||||
|---|------|-----------|--------|
|
||||
| 1 | assistant_accounts_master | id | idx_ods_assistant_accounts_master_latest |
|
||||
| 2 | settlement_records | id | idx_ods_settlement_records_latest |
|
||||
| 3 | table_fee_transactions | id | idx_ods_table_fee_transactions_latest |
|
||||
| 4 | assistant_service_records | id | idx_ods_assistant_service_records_latest |
|
||||
| 5 | assistant_cancellation_records | id | idx_ods_assistant_cancellation_records_latest |
|
||||
| 6 | store_goods_sales_records | id | idx_ods_store_goods_sales_records_latest |
|
||||
| 7 | payment_transactions | id | idx_ods_payment_transactions_latest |
|
||||
| 8 | refund_transactions | id | idx_ods_refund_transactions_latest |
|
||||
| 9 | platform_coupon_redemption_records | id | idx_ods_platform_coupon_redemption_records_latest |
|
||||
| 10 | member_profiles | id | idx_ods_member_profiles_latest |
|
||||
| 11 | member_stored_value_cards | id | idx_ods_member_stored_value_cards_latest |
|
||||
| 12 | member_balance_changes | id | idx_ods_member_balance_changes_latest |
|
||||
| 13 | recharge_settlements | id | idx_ods_recharge_settlements_latest |
|
||||
| 14 | group_buy_packages | id | idx_ods_group_buy_packages_latest |
|
||||
| 15 | group_buy_redemption_records | id | idx_ods_group_buy_redemption_records_latest |
|
||||
| 16 | goods_stock_summary | siteGoodsId | idx_ods_goods_stock_summary_latest |
|
||||
| 17 | goods_stock_movements | siteGoodsStockId | idx_ods_goods_stock_movements_latest |
|
||||
| 18 | site_tables_master | id | idx_ods_site_tables_master_latest |
|
||||
| 19 | stock_goods_category_tree | id | idx_ods_stock_goods_category_tree_latest |
|
||||
| 20 | store_goods_master | id | idx_ods_store_goods_master_latest |
|
||||
| 21 | table_fee_discount_records | id | idx_ods_table_fee_discount_records_latest |
|
||||
| 22 | tenant_goods_master | id | idx_ods_tenant_goods_master_latest |
|
||||
| 23 | settlement_ticket_details | orderSettleId | idx_ods_settlement_ticket_details_latest |
|
||||
|
||||
### 关联需求
|
||||
|
||||
`ods-dedup-standardize` Requirements 6.1, 6.2, 6.3
|
||||
|
||||
### 兼容性
|
||||
|
||||
- **非破坏性变更**：仅新增索引，不修改表结构、不影响现有数据和写入逻辑
|
||||
- **ETL Connector**：无需改动；索引自动加速 `skip_unchanged` 去重查询和 `_mark_missing_as_deleted` 快照对比查询
|
||||
- **后端 API / 小程序**：不受影响（不直接查询 ODS 层）
|
||||
- **`CREATE INDEX CONCURRENTLY`**：在线创建，不阻塞表的读写操作；但不能在事务块内执行，需逐条运行或使用支持单语句模式的工具
|
||||
- **DDL 源文件**：索引定义已同步写入 `db/etl_feiqiu/schemas/ods.sql`，新环境初始化时自动创建
|
||||
|
||||
### 回滚策略
|
||||
|
||||
逐条删除索引即可，不影响数据：
|
||||
|
||||
```sql
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_accounts_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_settlement_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_service_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_assistant_cancellation_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_sales_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_payment_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_refund_transactions_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_platform_coupon_redemption_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_profiles_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_stored_value_cards_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_member_balance_changes_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_recharge_settlements_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_packages_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_group_buy_redemption_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_summary_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_goods_stock_movements_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_site_tables_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_stock_goods_category_tree_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_store_goods_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_table_fee_discount_records_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_tenant_goods_master_latest;
|
||||
DROP INDEX IF EXISTS ods.idx_ods_settlement_ticket_details_latest;
|
||||
```
|
||||
|
||||
### 验证 SQL
|
||||
|
||||
```sql
|
||||
-- 1. 验证 23 个索引均已创建
|
||||
SELECT indexname, tablename
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname LIKE 'idx_ods_%_latest'
|
||||
ORDER BY indexname;
|
||||
|
||||
-- 2. 验证索引数量为 23
|
||||
SELECT COUNT(*) AS index_count
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname LIKE 'idx_ods_%_latest';
|
||||
|
||||
-- 3. 验证索引列定义正确（以 member_profiles 为例）
|
||||
SELECT indexdef
|
||||
FROM pg_indexes
|
||||
WHERE schemaname = 'ods'
|
||||
AND indexname = 'idx_ods_member_profiles_latest';
|
||||
|
||||
-- 4. 验证索引可用（非 INVALID 状态）
|
||||
SELECT c.relname AS index_name, i.indisvalid
|
||||
FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indexrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'ods'
|
||||
AND c.relname LIKE 'idx_ods_%_latest'
|
||||
AND i.indisvalid = false;
|
||||
-- 预期结果：0 行（无无效索引）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 迁移 1：六层 Schema 架构重组
|
||||
|
||||
### 变更说明
|
||||
|
||||
将现有 4 个 schema 重组为 6 层 schema 架构：
|
||||
|
||||
@@ -38,8 +157,9 @@
|
||||
|
||||
## 兼容性
|
||||
|
||||
- **ETL 管线**：需更新 ETL 代码中的 schema 引用（etl_admin -> meta, billiards_ods -> ods 等）
|
||||
- **后端 API**：通过 app schema 视图访问，无需直接引用底层表
|
||||
- **ETL Connector**：所有代码中的 schema 引用已更新完成（etl_admin → meta, billiards_ods → ods, billiards_dwd → dwd, billiards_dws → dws）
|
||||
- **后端 API**：etl_status 路由已更新为 meta.etl_cursor；通过 app schema 视图访问，无需直接引用底层表
|
||||
- **管理后台**：已由 `apps/admin-web/` Web 管理后台完全替代原 PySide6 桌面 GUI，通过后端 API 访问数据
|
||||
- **小程序**：通过 FDW 映射 app schema，不受影响
|
||||
|
||||
## 回滚策略
|
||||
|
||||
144
docs/database/zqyy_app_admin_web_tables.md
Normal file
144
docs/database/zqyy_app_admin_web_tables.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# zqyy_app — Web 管理后台表结构文档
|
||||
|
||||
## 变更说明
|
||||
|
||||
在 `zqyy_app` 数据库中新增 4 张表，支撑 Web 管理后台的用户认证、任务队列、执行日志和定时调度功能。
|
||||
|
||||
| 操作 | 表名 | 说明 |
|
||||
|------|------|------|
|
||||
| 新增 | admin_users | 管理后台操作员账户，绑定门店 site_id |
|
||||
| 新增 | task_queue | ETL 任务执行队列，支持排序和状态流转 |
|
||||
| 新增 | task_execution_log | 任务执行历史记录，含日志和摘要 |
|
||||
| 新增 | scheduled_tasks | 定时调度任务，支持 5 种调度类型 |
|
||||
|
||||
### 新增字段概览
|
||||
|
||||
#### admin_users
|
||||
| 字段 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| id | SERIAL | PK | 自增主键 |
|
||||
| username | VARCHAR(64) | UNIQUE NOT NULL | 登录用户名 |
|
||||
| password_hash | VARCHAR(256) | NOT NULL | bcrypt 密码哈希 |
|
||||
| display_name | VARCHAR(128) | | 显示名称 |
|
||||
| site_id | BIGINT | NOT NULL | 绑定的门店 ID |
|
||||
| is_active | BOOLEAN | DEFAULT TRUE | 是否启用 |
|
||||
| created_at | TIMESTAMPTZ | DEFAULT NOW() | 创建时间 |
|
||||
| updated_at | TIMESTAMPTZ | DEFAULT NOW() | 更新时间 |
|
||||
|
||||
#### task_queue
|
||||
| 字段 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| id | UUID | PK, DEFAULT gen_random_uuid() | 队列任务 ID |
|
||||
| site_id | BIGINT | NOT NULL | 门店隔离 |
|
||||
| config | JSONB | NOT NULL | 序列化的 TaskConfig |
|
||||
| status | VARCHAR(20) | NOT NULL, DEFAULT 'pending' | pending/running/success/failed/cancelled |
|
||||
| position | INTEGER | NOT NULL, DEFAULT 0 | 队列排序位置 |
|
||||
| created_at | TIMESTAMPTZ | DEFAULT NOW() | 创建时间 |
|
||||
| started_at | TIMESTAMPTZ | | 开始执行时间 |
|
||||
| finished_at | TIMESTAMPTZ | | 执行完成时间 |
|
||||
| exit_code | INTEGER | | 子进程退出码 |
|
||||
| error_message | TEXT | | 错误信息 |
|
||||
|
||||
#### task_execution_log
|
||||
| 字段 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| id | UUID | PK, DEFAULT gen_random_uuid() | 日志 ID |
|
||||
| queue_id | UUID | FK → task_queue(id) | 关联的队列任务（可空） |
|
||||
| site_id | BIGINT | NOT NULL | 门店隔离 |
|
||||
| task_codes | TEXT[] | NOT NULL | 执行的任务编码列表 |
|
||||
| status | VARCHAR(20) | NOT NULL | 执行状态 |
|
||||
| started_at | TIMESTAMPTZ | NOT NULL | 开始时间 |
|
||||
| finished_at | TIMESTAMPTZ | | 结束时间 |
|
||||
| exit_code | INTEGER | | 退出码 |
|
||||
| duration_ms | INTEGER | | 执行时长（毫秒） |
|
||||
| command | TEXT | | 实际执行的 CLI 命令 |
|
||||
| output_log | TEXT | | stdout 完整日志 |
|
||||
| error_log | TEXT | | stderr 日志 |
|
||||
| summary | JSONB | | 执行摘要 |
|
||||
| created_at | TIMESTAMPTZ | DEFAULT NOW() | 记录创建时间 |
|
||||
|
||||
#### scheduled_tasks
|
||||
| 字段 | 类型 | 约束 | 说明 |
|
||||
|------|------|------|------|
|
||||
| id | UUID | PK, DEFAULT gen_random_uuid() | 调度任务 ID |
|
||||
| site_id | BIGINT | NOT NULL | 门店隔离 |
|
||||
| name | VARCHAR(256) | NOT NULL | 调度任务名称 |
|
||||
| task_codes | TEXT[] | NOT NULL | 任务编码列表 |
|
||||
| task_config | JSONB | NOT NULL | 序列化的 TaskConfig |
|
||||
| schedule_config | JSONB | NOT NULL | 序列化的 ScheduleConfig |
|
||||
| enabled | BOOLEAN | DEFAULT TRUE | 是否启用 |
|
||||
| last_run_at | TIMESTAMPTZ | | 上次执行时间 |
|
||||
| next_run_at | TIMESTAMPTZ | | 下次执行时间 |
|
||||
| run_count | INTEGER | DEFAULT 0 | 累计执行次数 |
|
||||
| last_status | VARCHAR(20) | | 上次执行状态 |
|
||||
| created_at | TIMESTAMPTZ | DEFAULT NOW() | 创建时间 |
|
||||
| updated_at | TIMESTAMPTZ | DEFAULT NOW() | 更新时间 |
|
||||
|
||||
### 索引
|
||||
|
||||
| 索引名 | 表 | 列 | 类型 | 说明 |
|
||||
|--------|------|------|------|------|
|
||||
| idx_admin_users_site | admin_users | site_id | B-tree | 按门店查询用户 |
|
||||
| idx_task_queue_status | task_queue | status | B-tree | 按状态查询队列 |
|
||||
| idx_task_queue_site_position | task_queue | site_id, position | 部分索引 (WHERE status='pending') | 取出待执行任务 |
|
||||
| idx_execution_log_site_started | task_execution_log | site_id, started_at DESC | B-tree | 执行历史列表 |
|
||||
| idx_scheduled_tasks_site | scheduled_tasks | site_id | B-tree | 按门店查询调度 |
|
||||
| idx_scheduled_tasks_next_run | scheduled_tasks | next_run_at | 部分索引 (WHERE enabled=TRUE) | 查询到期调度 |
|
||||
|
||||
### 种子数据
|
||||
|
||||
- 默认管理员：`admin` / `admin123`，site_id=1
|
||||
- 种子脚本：`db/zqyy_app/seeds/admin_web_seed.sql`
|
||||
|
||||
## 兼容性
|
||||
|
||||
- **ETL Connector**：无影响。新表位于 `zqyy_app` 库，ETL 数据仍在 `etl_feiqiu` 库
|
||||
- **后端 API**：新增的 FastAPI 路由将读写这 4 张表，需配置 `zqyy_app` 数据库连接
|
||||
- **小程序**：无影响。小程序通过 FDW 访问 ETL 数据，不涉及管理后台表
|
||||
- **现有 zqyy_app 表**：无影响。新表与现有 users/roles/tasks 等表无外键关联
|
||||
|
||||
## 回滚策略
|
||||
|
||||
```sql
|
||||
-- 按依赖顺序删除（task_execution_log 引用 task_queue）
|
||||
BEGIN;
|
||||
DROP TABLE IF EXISTS task_execution_log CASCADE;
|
||||
DROP TABLE IF EXISTS task_queue CASCADE;
|
||||
DROP TABLE IF EXISTS scheduled_tasks CASCADE;
|
||||
DROP TABLE IF EXISTS admin_users CASCADE;
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
## 验证 SQL
|
||||
|
||||
```sql
|
||||
-- 1. 验证 4 张表均已创建
|
||||
SELECT table_name
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name IN ('admin_users', 'task_queue', 'task_execution_log', 'scheduled_tasks')
|
||||
ORDER BY table_name;
|
||||
|
||||
-- 2. 验证索引已创建（应返回 6 条）
|
||||
SELECT indexname, tablename
|
||||
FROM pg_indexes
|
||||
WHERE tablename IN ('admin_users', 'task_queue', 'task_execution_log', 'scheduled_tasks')
|
||||
AND indexname LIKE 'idx_%'
|
||||
ORDER BY tablename, indexname;
|
||||
|
||||
-- 3. 验证默认管理员已插入
|
||||
SELECT id, username, display_name, site_id, is_active
|
||||
FROM admin_users
|
||||
WHERE username = 'admin';
|
||||
|
||||
-- 4. 验证 task_queue 的部分索引条件
|
||||
SELECT indexname, indexdef
|
||||
FROM pg_indexes
|
||||
WHERE indexname = 'idx_task_queue_site_position';
|
||||
|
||||
-- 5. 验证 task_execution_log 的外键约束
|
||||
SELECT conname, conrelid::regclass, confrelid::regclass
|
||||
FROM pg_constraint
|
||||
WHERE conrelid = 'task_execution_log'::regclass
|
||||
AND contype = 'f';
|
||||
```
|
||||
Reference in New Issue
Block a user