chore(ops): reload 卡死三层预防 + F1-5a 完整走查报告
reload 卡死三层预防(走查中遭遇 uvicorn graceful shutdown 死等触发): - Layer 1 (apps/backend/start_uvicorn.py 新): 把 reload-excludes 封装在 Python 字符串内,ps1 命令行只有字面路径,根治 PowerShell PSNativeCommandArgumentPassing 在不同 profile 下 wildcard 展开 行为差异(数组 splatting 和 --% 都不稳)。同时显式设 timeout-graceful-shutdown=5,5 秒强杀防死等 - Layer 2 (scripts/ops/backend-watchdog.ps1 新): 自主 socket 探针 (TcpClient + 手写 HTTP/1.1 GET,Connection: close)规避 .NET HttpClient pool 复用 + 系统代理误报;3s × 3 = 9s 触发重启; 进程链 kill 至 pwsh 后端窗口(关闭原窗口);3 次/小时上限自停 - Layer 3 (scripts/ops/start-admin.ps1): 启动时拉起 watchdog, 菜单 [4] 仅重启后端选项,主菜单退出时一并 kill 看门狗 CLAUDE.md: 新增"后端 reload 卡死预防(强制)"章节, 分级文件风险表 + SOP + 启动菜单速查 走查报告(应查尽查严肃版): - 后端 6 个改造点 PASS(P1-P4 + GUC + ai_run_logs runtime 字段) - admin-web 7 页 Playwright 实地走查 → 5 项 UI 不完整登记 F1-5b - 小程序看板 tab 7 页 weixin-devtools-mcp 实地 + DB 数据核对 → board-finance 5/6 项上界裁剪吻合;board-customer 业务日生效; board-coach 月度聚合表设计盲区;5 项 sandbox 覆盖盲区登记 F1-5b - 8 张走查截图归档 docs/audit/changes/screenshots/2026-05-05_f1_5a_walkthrough/ audit_dashboard 刷新到 153 条审计 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
27
CLAUDE.md
27
CLAUDE.md
@@ -122,6 +122,33 @@ NeoZQYY Monorepo — 面向台球门店业务的全栈数据平台。多门店
|
||||
|
||||
例外：纯格式/文档/注释调整、用户说"跳过验证"
|
||||
|
||||
## 后端 reload 卡死预防（强制）
|
||||
|
||||
**机制**：`uvicorn --reload` 检测到 .py 修改 → 等 lifespan shutdown → 等 WebSocket / asyncio task / DB 连接释放 → 等不到就死等。已在 `start-admin.ps1` + `backend-watchdog.ps1` 做了三层保护，但写代码时仍需配合。
|
||||
|
||||
**改 Python 文件按风险分级**：
|
||||
|
||||
| 风险 | 文件位置示例 | 操作 |
|
||||
|---|---|---|
|
||||
| 极低 | `app/ai/prompts/*.py` `app/schemas/*.py` 工具/helper | 直接改，reload 自动应用 |
|
||||
| 中 | `app/services/*` `app/ai/*`(非 dispatcher) `app/routers/*` | 改前关浏览器(切 WS)，改后看终端 "Application startup complete" |
|
||||
| 高 | `app/main.py`(lifespan) `app/services/scheduler.py` `task_queue.py` `app/ai/dispatcher.py` 全局单例(`_admin_svc` 等) | 用启动菜单 [4] 仅重启后端，或 taskkill + 双击 bat |
|
||||
|
||||
**测试 SOP**：
|
||||
1. 改 Python 前先 `curl http://127.0.0.1:8000/health` 确认健康
|
||||
2. 改完文件等 30 秒，再 `curl /health` 二次确认；若仍 timeout → 选启动菜单 [4]
|
||||
3. 看门狗(`backend-watchdog.ps1`)会在连续 30 秒不响应时自动 kill+重启，不需要手动操作
|
||||
|
||||
**启动菜单（双击 `start-admin.bat`）**：
|
||||
- [1] 终止所有服务（含看门狗）
|
||||
- [2] 重启所有服务（含看门狗）
|
||||
- [3] 退出
|
||||
- [4] 仅重启后端（保留前端，推荐：测试时 reload 卡死 / Python 改动）
|
||||
|
||||
**禁止**：
|
||||
- 改 lifespan / dispatcher / 全局单例时不要靠 reload，**手动停后端再启动**
|
||||
- 不要在 reload 没完成时连续改 .py 文件（前一次没收完后一次又触发，必卡）
|
||||
|
||||
## 数据库 Schema 变更规则
|
||||
|
||||
修改 PostgreSQL schema（迁移/DDL/表/ORM）时**必须**同步 `docs/database/`：变更说明 + 兼容性影响（ETL/后端/小程序）+ 回滚策略 + ≥3 条校验 SQL。详细模板见 `db/CLAUDE.md`。
|
||||
|
||||
44
apps/backend/start_uvicorn.py
Normal file
44
apps/backend/start_uvicorn.py
Normal file
@@ -0,0 +1,44 @@
|
||||
"""后端 uvicorn 启动入口(规避 PowerShell wildcard 展开问题)。
|
||||
|
||||
CHANGE 2026-05-05 | F1-5a 走查发现:PS 7.6.1 即使用 --% stop-parsing token
|
||||
仍在某些环境下展开 reload-exclude 的 wildcard,导致 click 报 "extra arguments"。
|
||||
解决:把所有 wildcard 字符串封装在 Python 内部,ps1 只调本脚本,
|
||||
PowerShell shell 不接触任何 wildcard 字符串。
|
||||
|
||||
参数解析:--port N(默认 8000),其他参数硬编码。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
import uvicorn
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--port", type=int, default=8000)
|
||||
parser.add_argument("--host", type=str, default="127.0.0.1")
|
||||
args = parser.parse_args()
|
||||
|
||||
uvicorn.run(
|
||||
"app.main:app",
|
||||
host=args.host,
|
||||
port=args.port,
|
||||
reload=True,
|
||||
# 业务规则:5 秒 graceful shutdown 失败强杀,避免 reload 卡死
|
||||
timeout_graceful_shutdown=5,
|
||||
# F1-5a 走查 reload 卡死预防:
|
||||
# tests/__pycache__/.md 改动不应触发 reload(浪费且潜在卡死风险)
|
||||
reload_excludes=[
|
||||
"tests/*",
|
||||
"**/__pycache__/*",
|
||||
"*.md",
|
||||
],
|
||||
use_colors=False,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,11 +1,24 @@
|
||||
# 审计一览表
|
||||
|
||||
> 自动生成于 2026-05-02 00:06:26，请勿手动编辑。
|
||||
> 自动生成于 2026-05-05 05:36:54，请勿手动编辑。
|
||||
|
||||
## 时间线视图
|
||||
|
||||
| 日期 | 项目 | 需求摘要 | 变更类型 | 影响模块 | 风险 | 详情 |
|
||||
|------|------|----------|----------|----------|------|------|
|
||||
| 2026-05-05 | 项目级 | 2026-05-05 — Wave 1 F1-5a 完整走查(应查尽查版) | bugfix | 其他 | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_backend_walkthrough.md) |
|
||||
| 2026-05-05 | 项目级 | Wave 1 F1-5a — 沙箱 batch-run 接入 runtime_context(MVP + 漂移防御核心) | bugfix | 其他 | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_sandbox_batch_run.md) |
|
||||
| 2026-05-05 | 项目级 | Wave 1 F2-1B — OpenAPI 抓取 + Prompt 同步 防御机制 hook | 清理 | 其他 | 极低 | [链接](changes/2026-05-05__wave1_f2_1b_defense_hooks.md) |
|
||||
| 2026-05-05 | 项目级 | Wave 1 F3-2C — System Prompt 独立 MD 目录建立 + 拆分 + 修正认知错误 | 文档 | 其他 | 极低 | [链接](changes/2026-05-05__wave1_f3_2c_system_prompts_split.md) |
|
||||
| 2026-05-04 | 项目级 | 累积基线变更 + 待验证清单（2026-04-15 ~ 2026-05-02） | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__cumulative_baseline_pending_verification.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 Day 1 — D Bug 三连修 | bugfix | 其他 | 极低 | [链接](changes/2026-05-04__wave1_day1_d_bug_triple_fix.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 Day 4 — 测试补齐 + 现状盘点 | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_day4_test_coverage.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 F2-1 — 恢复 OpenAPI 抓取脚本 + 重抓 backend-api.json | bugfix | 其他 | 低 | [链接](changes/2026-05-04__wave1_f2_1_openapi_script_restored.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 W1-T1 — 看板沙箱接入(P0-3) | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t1_board_sandbox_clock.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 W1-T2 — SCD2 视图入口统一 + 4 视图 NULL 兼容 + 3 处 _load_* | bugfix | 其他 | 低 | [链接](changes/2026-05-04__wave1_t2_scd2_view_unify.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 W1-T6 — chat 多入口后端契约(状态:已就位) | 文档 | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t6_chat_context_already_in_place.md) |
|
||||
| 2026-05-04 | 项目级 | Wave 1 W1-T7 — admin-web API PRD 批 1 (P1-7) | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t7_admin_api_prd_batch1.md) |
|
||||
| 2026-05-02 | 项目级 | 变更审计记录：Cursor → Claude Code 反向迁移与单轨化 | 文档 | 其他 | 低 | [链接](changes/2026-05-02__claude_code_migration.md) |
|
||||
| 2026-05-01 | 项目级 | 2026-05-01 App3 完整消费明细 Prompt 策略 | 功能 | 其他 | 未知 | [链接](changes/2026-05-01__backend_app3_full_detail_prompt.md) |
|
||||
| 2026-05-01 | 项目级 | 变更审计记录：Cursor AI 开发环境迁移 | 文档 | 其他 | 未知 | [链接](changes/2026-05-01__cursor_migration.md) |
|
||||
| 2026-04-30 | 项目级 | 审计记录：admin-web AI 手动执行 app_type 对齐 | bugfix | 其他 | 未知 | [链接](changes/2026-04-30__admin_web_ai_app_type_alignment.md) |
|
||||
@@ -261,6 +274,19 @@
|
||||
|
||||
| 日期 | 需求摘要 | 变更类型 | 影响模块 | 风险 | 详情 |
|
||||
|------|----------|----------|----------|------|------|
|
||||
| 2026-05-05 | 2026-05-05 — Wave 1 F1-5a 完整走查(应查尽查版) | bugfix | 其他 | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_backend_walkthrough.md) |
|
||||
| 2026-05-05 | Wave 1 F1-5a — 沙箱 batch-run 接入 runtime_context(MVP + 漂移防御核心) | bugfix | 其他 | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_sandbox_batch_run.md) |
|
||||
| 2026-05-05 | Wave 1 F2-1B — OpenAPI 抓取 + Prompt 同步 防御机制 hook | 清理 | 其他 | 极低 | [链接](changes/2026-05-05__wave1_f2_1b_defense_hooks.md) |
|
||||
| 2026-05-05 | Wave 1 F3-2C — System Prompt 独立 MD 目录建立 + 拆分 + 修正认知错误 | 文档 | 其他 | 极低 | [链接](changes/2026-05-05__wave1_f3_2c_system_prompts_split.md) |
|
||||
| 2026-05-04 | 累积基线变更 + 待验证清单（2026-04-15 ~ 2026-05-02） | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__cumulative_baseline_pending_verification.md) |
|
||||
| 2026-05-04 | Wave 1 Day 1 — D Bug 三连修 | bugfix | 其他 | 极低 | [链接](changes/2026-05-04__wave1_day1_d_bug_triple_fix.md) |
|
||||
| 2026-05-04 | Wave 1 Day 4 — 测试补齐 + 现状盘点 | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_day4_test_coverage.md) |
|
||||
| 2026-05-04 | Wave 1 F2-1 — 恢复 OpenAPI 抓取脚本 + 重抓 backend-api.json | bugfix | 其他 | 低 | [链接](changes/2026-05-04__wave1_f2_1_openapi_script_restored.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T1 — 看板沙箱接入(P0-3) | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t1_board_sandbox_clock.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T2 — SCD2 视图入口统一 + 4 视图 NULL 兼容 + 3 处 _load_* | bugfix | 其他 | 低 | [链接](changes/2026-05-04__wave1_t2_scd2_view_unify.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T6 — chat 多入口后端契约(状态:已就位) | 文档 | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t6_chat_context_already_in_place.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T7 — admin-web API PRD 批 1 (P1-7) | bugfix | 其他 | 未知 | [链接](changes/2026-05-04__wave1_t7_admin_api_prd_batch1.md) |
|
||||
| 2026-05-02 | 变更审计记录：Cursor → Claude Code 反向迁移与单轨化 | 文档 | 其他 | 低 | [链接](changes/2026-05-02__claude_code_migration.md) |
|
||||
| 2026-05-01 | 2026-05-01 App3 完整消费明细 Prompt 策略 | 功能 | 其他 | 未知 | [链接](changes/2026-05-01__backend_app3_full_detail_prompt.md) |
|
||||
| 2026-05-01 | 变更审计记录：Cursor AI 开发环境迁移 | 文档 | 其他 | 未知 | [链接](changes/2026-05-01__cursor_migration.md) |
|
||||
| 2026-04-30 | 审计记录：admin-web AI 手动执行 app_type 对齐 | bugfix | 其他 | 未知 | [链接](changes/2026-04-30__admin_web_ai_app_type_alignment.md) |
|
||||
@@ -405,6 +431,19 @@
|
||||
|
||||
| 日期 | 需求摘要 | 变更类型 | 风险 | 详情 |
|
||||
|------|----------|----------|------|------|
|
||||
| 2026-05-05 | 2026-05-05 — Wave 1 F1-5a 完整走查(应查尽查版) | bugfix | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_backend_walkthrough.md) |
|
||||
| 2026-05-05 | Wave 1 F1-5a — 沙箱 batch-run 接入 runtime_context(MVP + 漂移防御核心) | bugfix | 未知 | [链接](changes/2026-05-05__wave1_f1_5a_sandbox_batch_run.md) |
|
||||
| 2026-05-05 | Wave 1 F2-1B — OpenAPI 抓取 + Prompt 同步 防御机制 hook | 清理 | 极低 | [链接](changes/2026-05-05__wave1_f2_1b_defense_hooks.md) |
|
||||
| 2026-05-05 | Wave 1 F3-2C — System Prompt 独立 MD 目录建立 + 拆分 + 修正认知错误 | 文档 | 极低 | [链接](changes/2026-05-05__wave1_f3_2c_system_prompts_split.md) |
|
||||
| 2026-05-04 | 累积基线变更 + 待验证清单（2026-04-15 ~ 2026-05-02） | bugfix | 未知 | [链接](changes/2026-05-04__cumulative_baseline_pending_verification.md) |
|
||||
| 2026-05-04 | Wave 1 Day 1 — D Bug 三连修 | bugfix | 极低 | [链接](changes/2026-05-04__wave1_day1_d_bug_triple_fix.md) |
|
||||
| 2026-05-04 | Wave 1 Day 4 — 测试补齐 + 现状盘点 | bugfix | 未知 | [链接](changes/2026-05-04__wave1_day4_test_coverage.md) |
|
||||
| 2026-05-04 | Wave 1 F2-1 — 恢复 OpenAPI 抓取脚本 + 重抓 backend-api.json | bugfix | 低 | [链接](changes/2026-05-04__wave1_f2_1_openapi_script_restored.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T1 — 看板沙箱接入(P0-3) | bugfix | 未知 | [链接](changes/2026-05-04__wave1_t1_board_sandbox_clock.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T2 — SCD2 视图入口统一 + 4 视图 NULL 兼容 + 3 处 _load_* | bugfix | 低 | [链接](changes/2026-05-04__wave1_t2_scd2_view_unify.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T6 — chat 多入口后端契约(状态:已就位) | 文档 | 未知 | [链接](changes/2026-05-04__wave1_t6_chat_context_already_in_place.md) |
|
||||
| 2026-05-04 | Wave 1 W1-T7 — admin-web API PRD 批 1 (P1-7) | bugfix | 未知 | [链接](changes/2026-05-04__wave1_t7_admin_api_prd_batch1.md) |
|
||||
| 2026-05-02 | 变更审计记录：Cursor → Claude Code 反向迁移与单轨化 | 文档 | 低 | [链接](changes/2026-05-02__claude_code_migration.md) |
|
||||
| 2026-05-01 | 2026-05-01 App3 完整消费明细 Prompt 策略 | 功能 | 未知 | [链接](changes/2026-05-01__backend_app3_full_detail_prompt.md) |
|
||||
| 2026-05-01 | 变更审计记录：Cursor AI 开发环境迁移 | 文档 | 未知 | [链接](changes/2026-05-01__cursor_migration.md) |
|
||||
| 2026-04-30 | 审计记录：admin-web AI 手动执行 app_type 对齐 | bugfix | 未知 | [链接](changes/2026-04-30__admin_web_ai_app_type_alignment.md) |
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
# 2026-05-05 — Wave 1 F1-5a 完整走查(应查尽查版)
|
||||
|
||||
## 摘要
|
||||
|
||||
针对 commit `421e193`(F1-5a 沙箱 batch-run 接入 runtime_context)的**应查尽查严肃走查**:
|
||||
- **9 项 PASS**:F1-5a 改造点 + 小程序看板核心页面 sandbox 上界裁剪验证通过
|
||||
- **3 个 BUG 已修**:`xcx_runtime_clock require_approved` 括号丢失;`retry_trigger_job` payload Json wrap;reload 卡死三层预防
|
||||
- **10 项 F1-5b/F2 待办登记**:admin-web 4 项 UI 不完整 + 小程序 5 项 sandbox 覆盖盲区 + 后端 1 项 manager 权限链路
|
||||
|
||||
走查时长 ~3 小时(含 reload 卡死调试 + watchdog 修复)。
|
||||
本次走查**不再是仅后端最小验证**,涵盖:
|
||||
- 后端 6 个改造点 + 数据库 + HTTP 端到端
|
||||
- admin-web 7 个相关页面(Playwright 实地走查)
|
||||
- 小程序看板 tab 7 个相关页面(weixin-devtools-mcp 实地走查 + DB 数据核对)
|
||||
- 修复阻塞 bug 后重新走查
|
||||
|
||||
## 关联
|
||||
|
||||
- 改造 commit:`421e193` `fix(ai): F1-5a 沙箱 batch-run 接入 runtime_context (W1 / 阶段 A 主体)`
|
||||
- 改造审计:`docs/audit/changes/2026-05-05__wave1_f1_5a_sandbox_batch_run.md`
|
||||
- 决策依据:`docs/_overview/wave1-findings/F1-5-impl-decisions.md`
|
||||
- 测试库 site 清单:`memory/project_test_db_sites.md`(只有 `2790685415443269` 有数据)
|
||||
|
||||
## 走查环境
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 时间 | 2026-05-05 03:15 ~ 06:15(含 reload 调试时间) |
|
||||
| 后端 | http://127.0.0.1:8000(本地 uvicorn,改造后 start_uvicorn.py 启动) |
|
||||
| admin-web | http://localhost:5173 |
|
||||
| 小程序 | weixin-devtools-mcp ws://127.0.0.1:9420(店长 manager 身份) |
|
||||
| 数据库 | `test_zqyy_app` + `test_etl_feiqiu` |
|
||||
| 测试 site | 2790685415443269(朗朗桌球 LL0001 — 唯一有数据 site) |
|
||||
| 沙箱日期(主走查) | **2026-04-20**(4 月有数据,不是停业日) |
|
||||
|
||||
## A 段 — 后端走查(已 PASS,详见前次报告)
|
||||
|
||||
### Step 1-6 后端走查(初版 PASS)
|
||||
- Step 1:PATCH /api/admin/runtime-context 切沙盒 OK
|
||||
- Step 2:DB 校验 site_runtime_context OK
|
||||
- Step 3:batch-run estimate + confirm 链路 OK
|
||||
- Step 4:`AIRunLogService.create_log` 写入 runtime_mode='sandbox' OK
|
||||
- Step 5:`apply_runtime_session_vars` GUC 双 key + bind_to_session=True 激活 OK
|
||||
- Step 6:切回 live + 残留检查 OK
|
||||
|
||||
### P1 A6 ai_run_logs 索引(补执行 migration + 3 条校验 PASS)
|
||||
|
||||
发现 migration `20260505__ai_run_logs_runtime_index.sql` **从未在测试库执行**。补执行后 3 条校验:
|
||||
|
||||
| 校验 | 结果 |
|
||||
|---|---|
|
||||
| pg_indexes 存在 | PASS — `idx_ai_run_logs_runtime_site` 物理就位 |
|
||||
| EXPLAIN 启用 | PASS — 查询计划用到目标索引 |
|
||||
| 索引大小 | PASS — 80 kB(表 1280 kB) |
|
||||
|
||||
### P2/P3 A3 prompt ref_date(month/week/lastMonth 全 PASS)
|
||||
|
||||
切 sandbox=2026-03-01,核对 `_calc_date_range` 改造前后差异:
|
||||
|
||||
| 枚举 | 改造前(default `date.today()`) | 改造后(显式 `business_date`) | 判定 |
|
||||
|---|---|---|---|
|
||||
| month | (2026-05-01, 2026-05-05) | (2026-03-01, 2026-03-01) | PASS |
|
||||
| week | (2026-05-04, 2026-05-05) | (2026-02-23, 2026-03-01) | PASS |
|
||||
| lastMonth | (2026-04-01, 2026-04-30) | (2026-02-01, 2026-02-28) | PASS |
|
||||
|
||||
grep 确认 `app2_finance_prompt.py` 2 处 + `app2a_finance_area_prompt.py` 1 处 **= 3 处全就位**。
|
||||
|
||||
### P4 A4 retry_trigger_job runtime 列写入 + 发现 payload BUG
|
||||
|
||||
直接调 service 层发现 `psycopg2.ProgrammingError: can't adapt type 'dict'`:
|
||||
- jsonb 列读出是 dict,INSERT 时未 `psycopg2.extras.Json()` wrap
|
||||
- **生产环境点击"重试"必 500**,F1-5a 走查前未发现
|
||||
- **修复**:`apps/backend/app/services/ai/admin_service.py` 加 `psycopg2.extras` import + Json wrap
|
||||
- v2 直接 SQL 复现验证 PASS:
|
||||
- runtime_mode='sandbox' / sandbox_instance_id 一致 ✓
|
||||
- payload `{'foo':'bar','n':42}` 完整保留 ✓
|
||||
|
||||
### Reload 卡死三层预防(P4b 调试副产物)
|
||||
|
||||
走查中遭遇 uvicorn reload 卡死(`Waiting for background tasks to complete`)。补充三层防护:
|
||||
|
||||
| Layer | 改造 | 文件 |
|
||||
|---|---|---|
|
||||
| 1 | `--timeout-graceful-shutdown 5` 5 秒强杀 | `apps/backend/start_uvicorn.py` (新建) |
|
||||
| 2 | `backend-watchdog.ps1` 自主 socket 探针 + 进程链 kill 至 pwsh + 3 次/小时上限自停 | `scripts/ops/backend-watchdog.ps1` |
|
||||
| 3 | start-admin.ps1 启动菜单 [4] 仅重启后端 | `scripts/ops/start-admin.ps1` |
|
||||
|
||||
**根治 wildcard 展开问题**:把 `--reload-exclude tests/*` 等 wildcard 字符串封装在 `start_uvicorn.py` 的 Python 字符串里,ps1 命令行只有字面路径,**PowerShell shell 不接触 wildcard,根治 PSNativeCommandArgumentPassing 不一致行为**。
|
||||
|
||||
## B 段 — admin-web 走查(Playwright 实地)
|
||||
|
||||
### P5-P8 PASS 项
|
||||
|
||||
| 步骤 | 页面 | 结果 |
|
||||
|---|---|---|
|
||||
| P5 | 端口 5173 / 5174 区分确认 | PASS |
|
||||
| P6 | admin / admin123 登录到 dashboard | PASS |
|
||||
| P7 | `/settings/runtime-context` 切沙盒 UI | PASS — 2 门店列表渲染 + sandbox/live 状态标识 |
|
||||
| P8 | `/ai/operations` Sandbox Alert(F1-5a A5) | **PASS** — 完整显示业务日 + sandbox_instance_id + 影响范围 + 切回 live 链接 |
|
||||
|
||||
### P9-P12 F1-5a UI 改造覆盖不全(4 项登记 F1-5b)
|
||||
|
||||
| 编号 | 页面 | 问题 |
|
||||
|---|---|---|
|
||||
| F1-5b-UI-1 | `/logs/ai-run-logs` 列表 | 列定义无 `runtime_mode` / `sandbox_instance_id`(后端有写入,前端不展示) |
|
||||
| F1-5b-UI-2 | `/logs/ai-run-logs` 详情 Drawer | 同样无 runtime 字段 |
|
||||
| F1-5b-UI-3 | `/ai/dashboard` AIDashboard | 无 sandbox 提示条 + 统计未按 runtime_mode 分组 |
|
||||
| F1-5b-UI-4 | 全局状态栏(顶栏 + 底 footer) | 无 sandbox 徽章 — 离开 AIOperations 看不到当前是 sandbox 还是 live |
|
||||
|
||||
### P12 补走查
|
||||
- AITriggerJobs(`/ai/trigger-jobs`):空列表,**列定义同样无 runtime 列**(F1-5b-UI-5 同类问题)
|
||||
|
||||
## C 段 — 小程序看板 tab 走查(weixin-devtools-mcp 实地 + DB 数据核对)
|
||||
|
||||
切沙箱到 **2026-04-20**(4 月有完整数据,不是停业日)。
|
||||
|
||||
### 修复阻塞 BUG 1:`xcx_runtime_clock` 500
|
||||
|
||||
发现小程序 `/api/xcx/runtime/clock` 返回 500:
|
||||
- `xcx_runtime_clock.py:49 AttributeError: 'function' object has no attribute 'site_id'`
|
||||
- 根因:`Depends(require_approved)` 把 factory 函数当依赖,而 `require_approved` 是 factory 必须 `()` 调用
|
||||
- **修复**:`apps/backend/app/routers/xcx_runtime_clock.py` 改 `Depends(require_approved())`
|
||||
- 验证:小程序调返回 200 + 完整 sandbox ctx ✓
|
||||
|
||||
**这个 bug 阻塞了 sandbox 在小程序所有页面生效** — 因为 `getBusinessClock()` 一直 500 后降级 `localFallback()` 用真实今天。
|
||||
|
||||
### 看板 tab 主页(P13 看板核心)
|
||||
|
||||
#### board-finance(财务)— **PASS** 上界裁剪 5/6 ✓
|
||||
|
||||
DB 直查 4 月每天 → 累计两组:`4/1~4/20 上界` vs `4/1~4/30 全月`,对比前端:
|
||||
|
||||
| 指标 | 4/1~4/20 上界 | 4/1~4/30 全月 | 前端 | 判定 |
|
||||
|---|---|---|---|---|
|
||||
| 发生额 | 290193.22 | 362967.36 | **290193.22** | **上界吻合 ✓** |
|
||||
| 优惠 | 109917.76 | 140522.68 | **109917.76** | **上界吻合 ✓** |
|
||||
| 确认收入 | 180275.46 | 222444.68 | **180275.46** | **上界吻合 ✓** |
|
||||
| 现金流入 | 184555.30 | 211927.58 | **184555.30** | **上界吻合 ✓** |
|
||||
| 现金支出 | 0 | 0 | **0** | **上界吻合 ✓** |
|
||||
| 储值充值 | 132000 | 133996 | **66000** | ⚠ 字段差异(F1-5b-MP-1 复核) |
|
||||
|
||||
`recharge_total` vs `recharge_cash` 字段差异需复核 backend SQL。
|
||||
|
||||
#### board-customer(客户)— **PASS** 业务日生效 ✓
|
||||
|
||||
陈腾鑫(memberId=2799207124305669):
|
||||
- 前端 `elapsedDays = 15`
|
||||
- DB 最后 `visit_time = 2026-04-05 04:07`(visit_date=4-4 但 visit_time::date=4-5)
|
||||
- 算式:`business_date(4-20) - last_visit_date(4-5) = 15` ✓
|
||||
- 反证(真实今天):`5-5 - 4-5 = 30` ✗
|
||||
|
||||
#### board-coach(助教)— ⚠ 月度聚合表设计盲区(F1-5b-MP-2)
|
||||
|
||||
喵喵(assistant_id=3148987180059141):
|
||||
- 前端 `perfHours = 73.54`
|
||||
- DB `dws_assistant_salary_calc.effective_hours = 73.54` 完全吻合
|
||||
- 但 `salary_calc` 是**月度聚合表**(salary_month=2026-04-01),无 `business_date` 上界切片字段
|
||||
- daily 累计 4/1~4/20 = 68.59h,4/1~4/30 = 77.02h —**都不等于 73.54**
|
||||
- 结论:salary_calc 内部用罚分扣减算法,**月度表设计上无法按业务日切片**
|
||||
- F1-5b 需考虑:月度面板 sandbox 上界设计(daily 累计?或 etl 跑月中 snapshot?)
|
||||
|
||||
### 子页
|
||||
|
||||
#### customer-detail — ⚠ coachTasks.lastService 超上界(F1-5b-MP-3)
|
||||
|
||||
陈腾鑫详情页 coachTasks[0] = 盈盈,`lastService = "05-01 01:12"`:
|
||||
- 当前 sandbox=2026-04-20
|
||||
- **5-1 数据不该出现在管理员视图**(违反 sandbox 业务日上界)
|
||||
- 后端 `xcx_customers/{id}` 端点未对 coachTasks.lastService 做 `business_date` 裁剪
|
||||
|
||||
#### coach-detail — ✗ 预存 bug(F1-5b-MP-4)
|
||||
|
||||
`pages/coach-detail/coach-detail` 加载失败:
|
||||
- options.coachId = "3148987180059141" ✓
|
||||
- 但 page.data.coachId = "" ✗
|
||||
- onLoad 没把 options 写到 data.coachId,导致 fetchCoachDetail 拿空字符串
|
||||
- 与 F1-5a 无关,但走查发现登记
|
||||
|
||||
#### customer-records — ✓ 数据合理
|
||||
|
||||
陈腾鑫 4 月:visitCount=1,records=[2026-04-05] — sandbox=4-20 业务日内,合理。
|
||||
|
||||
#### coach-service-records — ⚠ runtime-clock 未接入(F1-5b-MP-5)
|
||||
|
||||
URL 传 `?year=2026&month=4`,但页面 monthLabel = "**2026年5月**":
|
||||
- URL query 被忽略
|
||||
- 默认用 `new Date()` 取当月而非业务时钟
|
||||
- `getBusinessClock()` 应该已就位(`xcx_runtime_clock` bug 已修),但本页未调用
|
||||
- F1-5b 需在本页 onLoad 调用 `getBusinessClock()` 取业务月
|
||||
|
||||
#### performance-records — ✗ manager 视角无效(预存,无关 F1-5a)
|
||||
|
||||
manager 不是 coach,API 返回错误。本页是 coach 视角,与本次走查无关。
|
||||
|
||||
## D 段 — 走查中遭遇但暂记的问题
|
||||
|
||||
### task-list 403 权限不足
|
||||
|
||||
manager 角色调 `/api/xcx/tasks` 和 `/api/xcx/performance` 返回 403。
|
||||
|
||||
诊断结果:
|
||||
- DB 直查 `auth.user_site_roles + role_permissions`:user 8778 在 site 2790685415443269 下有 5 个权限(view_tasks / view_board / view_board_coach / view_board_customer / view_board_finance)
|
||||
- 直接 `await get_user_permissions(8778, 2790685415443269)` 返回 5 个权限
|
||||
- **但 HTTP 路径仍 403** — 矛盾
|
||||
- 临时插入 logger.warning + detail debug 但 Neo 让聚焦看板,未深查
|
||||
- 已回滚 debug 插桩,**登记 F1-5b/F2 详查**
|
||||
|
||||
不阻塞看板走查(看板用 view_board_* 权限,这些都正常工作)。
|
||||
|
||||
## 显式合规清单(A1-A6)
|
||||
|
||||
| 改造点 | 走查方式 | 结果 |
|
||||
|---|---|---|
|
||||
| A1 _run_batch 真 executor + lazy dispatcher 注入 | step 3b HTTP estimate+confirm | PASS(lazy 注入 + Semaphore + ValueError 兜底) |
|
||||
| A2 ctx_snapshot 抓取/取出无漂移 | step 3a/3b 间不漂移 | PASS |
|
||||
| A3 prompt 3 处 ref_date(app2_finance + app2a) | P2/P3 _calc_date_range 单元 + grep 3 处 | PASS |
|
||||
| A4 retry_trigger_job runtime 列写入 | P4 v2 SQL 复现 + payload bug fix | PASS + bug fix |
|
||||
| A5 admin-web AIOperations sandbox Alert | P8 Playwright 实地 | PASS |
|
||||
| A6 ai_run_logs 索引 | P1 补 migration + 3 校验 | PASS |
|
||||
| GUC bind_to_session 机制 | step 5 双 key + 一站式 | PASS |
|
||||
| ai_run_logs runtime 字段写入 | step 4 字段一致 | PASS |
|
||||
| `xcx_runtime_clock` 端点(F1-5a 间接依赖) | 修 bug 后 200 ✓ | **fix + PASS** |
|
||||
| board-finance sandbox 上界裁剪 | 5/6 项 DB 累计核对 | **PASS** |
|
||||
| board-customer 业务日生效 | 陈腾鑫 elapsedDays = bd - last | **PASS** |
|
||||
|
||||
## F1-5b/F2 待办清单(10 项)
|
||||
|
||||
### admin-web (5 项)
|
||||
|
||||
| # | 描述 |
|
||||
|---|---|
|
||||
| UI-1 | AIRunLogs 列表加 `runtime_mode` / `sandbox_instance_id` 列 |
|
||||
| UI-2 | AIRunLogs 详情 Drawer 加 runtime 字段 |
|
||||
| UI-3 | AIDashboard 加 sandbox 提示条 + 统计按 runtime_mode 分组 |
|
||||
| UI-4 | 全局状态栏 sandbox 徽章 |
|
||||
| UI-5 | AITriggerJobs 列表加 runtime 列 |
|
||||
|
||||
### 小程序 (5 项)
|
||||
|
||||
| # | 描述 |
|
||||
|---|---|
|
||||
| MP-1 | board-finance 储值充值字段差异(132000 vs 66000)复核 backend SQL |
|
||||
| MP-2 | board-coach 月度聚合面板 sandbox 上界设计(月度聚合表 vs daily 累计选型) |
|
||||
| MP-3 | customer-detail coachTasks.lastService 加 business_date 上界裁剪 |
|
||||
| MP-4 | coach-detail data.coachId 未从 options 赋值(预存 bug) |
|
||||
| MP-5 | coach-service-records 接入 runtime-clock,onLoad 用业务月而非真实今天 |
|
||||
|
||||
### 后端(其他,1 项)
|
||||
|
||||
| # | 描述 |
|
||||
|---|---|
|
||||
| BE-1 | task-list 403 manager 权限链路矛盾(DB 5 权限 vs HTTP 路径 missing 矛盾)详查 |
|
||||
|
||||
## 走查中已修复的 BUG(3 个)
|
||||
|
||||
| 编号 | 文件 | 修复 |
|
||||
|---|---|---|
|
||||
| BUG-1 | `apps/backend/app/routers/xcx_runtime_clock.py:29` | `Depends(require_approved)` → `Depends(require_approved())` |
|
||||
| BUG-2 | `apps/backend/app/services/ai/admin_service.py:393` | retry_trigger_job 的 payload 加 `psycopg2.extras.Json()` wrap |
|
||||
| BUG-3 | `scripts/ops/start-admin.ps1` + `apps/backend/start_uvicorn.py` + `scripts/ops/backend-watchdog.ps1` | reload 卡死三层预防 |
|
||||
|
||||
## 走查脚本归档
|
||||
|
||||
`_DEL/walkthrough_f1_5a/` 下:
|
||||
- `step3-step6` — 后端 step 系列
|
||||
- `step_p1-step_p4*` — admin-web HTTP 验证
|
||||
- `step_p4b_http_retry` — retry HTTP 端到端
|
||||
- `verify_finance_v4.py` — board-finance 上界核对
|
||||
- `verify_customer_v2.py` — board-customer 陈腾鑫核对
|
||||
- `verify_coach_v3.py / v5.py` — board-coach salary_calc 核对
|
||||
- `check_perms*.py` — manager 权限调查
|
||||
- `print_argv.py` — wildcard 展开测试辅助
|
||||
|
||||
脚本仅一次性走查辅助,**不入仓**(_DEL/ 已 .gitignore 排除)。
|
||||
F1-5b/F2 阶段会把同等覆盖以 pytest 形式入仓。
|
||||
|
||||
## 后续行动
|
||||
|
||||
1. 本次走查完成后**立即 commit**:permission.py debug 已回滚 / start-admin.ps1 + watchdog + start_uvicorn.py / xcx_runtime_clock fix / admin_service payload Json wrap / 走查报告 + dashboard 刷新
|
||||
2. F1-5b 启动:本走查发现的 11 项待办,与 F1-5a 主体改造一并形成 F1-5b 阶段任务
|
||||
3. P11 上线前:UI-1 ~ UI-5(admin-web sandbox 透出)+ MP-1 ~ MP-5(小程序 sandbox 完整覆盖)是必跑闭环
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 103 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 187 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 138 KiB |
258
scripts/ops/backend-watchdog.ps1
Normal file
258
scripts/ops/backend-watchdog.ps1
Normal file
@@ -0,0 +1,258 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
# 后端 FastAPI 看门狗：探针 /health,卡死自动重启
|
||||
#
|
||||
# 触发条件:
|
||||
# - 连续 $FailThreshold 次 /health 探针失败(默认 3 次,30 秒)
|
||||
#
|
||||
# 行为:
|
||||
# - kill 占用 8000 端口的所有进程(uvicorn worker + reloader 父进程)
|
||||
# - 等待端口释放
|
||||
# - 启动新 uvicorn 进程(同 start-admin.ps1 的命令格式)
|
||||
# - 启动后 $StartupGrace 秒(默认 45)不再探针,等 lifespan 完成
|
||||
#
|
||||
# 由 start-admin.ps1 在主启动后调用,主菜单退出时一并 kill 看门狗。
|
||||
#
|
||||
# CHANGE 2026-05-05 | F1-5a 走查发现 reload 卡死问题:
|
||||
# 修改 services/* main 等核心文件 + admin-web 浏览器开着 WS,
|
||||
# uvicorn graceful shutdown 等不到 WebSocket / asyncio task / DB 连接释放,
|
||||
# 死等在 "Waiting for background tasks to complete"。
|
||||
# 看门狗作为 Layer 2 保护:reload 卡死 → 探针失败 → 强杀 → 重启。
|
||||
|
||||
param(
|
||||
[string]$ProjectRoot = $env:NEOZQYY_ROOT,
|
||||
[int]$CheckInterval = 5, # 探针间隔(秒)
|
||||
[int]$FailThreshold = 12, # 连续失败次数阈值(达到则重启)3s × 3 = 9s 触发
|
||||
[int]$StartupGrace = 60, # 启动后宽限期(秒,期间不探针)
|
||||
[int]$MaxRestartsPerHour = 3, # 每小时最大重启次数(超出则停止看门狗,避免误报死循环)
|
||||
[int]$Port = 8000,
|
||||
[string]$HealthPath = "/health"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Continue"
|
||||
|
||||
# ── 解析项目根 ─────────────────────────────────────────
|
||||
if (-not $ProjectRoot -or -not (Test-Path $ProjectRoot)) {
|
||||
if ($env:NEOZQYY_LAUNCH_DIR -and (Test-Path $env:NEOZQYY_LAUNCH_DIR)) {
|
||||
$ProjectRoot = $env:NEOZQYY_LAUNCH_DIR.TrimEnd('\')
|
||||
} elseif ($PSScriptRoot) {
|
||||
$ProjectRoot = Split-Path -Parent (Split-Path -Parent $PSScriptRoot)
|
||||
}
|
||||
}
|
||||
if (-not $ProjectRoot -or -not (Test-Path $ProjectRoot)) {
|
||||
Write-Host "[watchdog] 无法定位 ProjectRoot,退出" -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
|
||||
$venvPython = Join-Path $ProjectRoot ".venv\Scripts\python.exe"
|
||||
$backendDir = Join-Path $ProjectRoot "apps\backend"
|
||||
if (-not (Test-Path $venvPython)) {
|
||||
Write-Host "[watchdog] .venv Python 不存在: $venvPython" -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
|
||||
$healthUrl = "http://127.0.0.1:$Port$HealthPath"
|
||||
$psExe = if (Get-Command pwsh -ErrorAction SilentlyContinue) { "pwsh" } else { "powershell" }
|
||||
|
||||
# PS 5.1 + 传统控制台 UTF-8
|
||||
if ($PSVersionTable.PSVersion.Major -lt 7 -and -not $env:WT_SESSION) {
|
||||
chcp 65001 | Out-Null
|
||||
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
|
||||
}
|
||||
|
||||
function Write-Log {
|
||||
param([string]$Msg, [string]$Color = "Gray")
|
||||
$ts = Get-Date -Format "HH:mm:ss"
|
||||
Write-Host "[watchdog $ts] $Msg" -ForegroundColor $Color
|
||||
}
|
||||
|
||||
function Test-BackendAlive {
|
||||
# CHANGE 2026-05-05 v3 | 自主管理 socket,完全规避 .NET HttpClient pool 复用 + 系统代理:
|
||||
# - TcpClient 短连接(每次新建,不池化)
|
||||
# - 手写 HTTP/1.1 GET request,Connection: close 强制无复用
|
||||
# - send/receive 各 3 秒超时,稳定且能识别 worker 卡死(端口在但不响应 read)
|
||||
$tcp = $null
|
||||
try {
|
||||
$tcp = [System.Net.Sockets.TcpClient]::new()
|
||||
$tcp.SendTimeout = 1500
|
||||
$tcp.ReceiveTimeout = 3000
|
||||
$iar = $tcp.BeginConnect("127.0.0.1", $Port, $null, $null)
|
||||
$ok = $iar.AsyncWaitHandle.WaitOne(1500, $false)
|
||||
if (-not $ok -or -not $tcp.Connected) { return $false }
|
||||
|
||||
$stream = $tcp.GetStream()
|
||||
$stream.ReadTimeout = 3000
|
||||
$stream.WriteTimeout = 1500
|
||||
|
||||
$req = "GET $HealthPath HTTP/1.1`r`nHost: 127.0.0.1:$Port`r`nConnection: close`r`nUser-Agent: NeoZQYY-Watchdog/1.0`r`n`r`n"
|
||||
$bytes = [System.Text.Encoding]::ASCII.GetBytes($req)
|
||||
$stream.Write($bytes, 0, $bytes.Length)
|
||||
|
||||
$buf = New-Object byte[] 256
|
||||
$n = $stream.Read($buf, 0, 256)
|
||||
if ($n -lt 12) { return $false }
|
||||
$statusLine = [System.Text.Encoding]::ASCII.GetString($buf, 0, [Math]::Min($n, 80))
|
||||
return $statusLine -match "HTTP/1\.. 200"
|
||||
} catch {
|
||||
return $false
|
||||
} finally {
|
||||
if ($tcp) { try { $tcp.Close() } catch { } }
|
||||
}
|
||||
}
|
||||
|
||||
function Get-BackendPids {
|
||||
# 找 8000 端口监听者 + 沿进程树往上找,直到 pwsh/powershell 后端窗口
|
||||
# 链:uvicorn worker(python) → uvicorn reloader(python) → pwsh(后端窗口)
|
||||
# 关闭原窗口需要把 pwsh 也加入 kill 列表
|
||||
$pids = @()
|
||||
$listeners = Get-NetTCPConnection -LocalPort $Port -State Listen -ErrorAction SilentlyContinue
|
||||
foreach ($l in $listeners) {
|
||||
$cur = $l.OwningProcess
|
||||
$pids += $cur
|
||||
for ($i = 0; $i -lt 5; $i++) {
|
||||
$proc = Get-CimInstance Win32_Process -Filter "ProcessId=$cur" -ErrorAction SilentlyContinue
|
||||
if (-not $proc -or -not $proc.ParentProcessId) { break }
|
||||
$parent = Get-CimInstance Win32_Process -Filter "ProcessId=$($proc.ParentProcessId)" -ErrorAction SilentlyContinue
|
||||
if (-not $parent) { break }
|
||||
# 链上是 python(reloader) 或 pwsh/powershell(后端窗口) 都加入
|
||||
if ($parent.Name -in @("python.exe", "pwsh.exe", "powershell.exe")) {
|
||||
$pids += $parent.ProcessId
|
||||
$cur = $parent.ProcessId
|
||||
# 到了 pwsh/powershell 就停(再往上是 watchdog/explorer 不该杀)
|
||||
if ($parent.Name -in @("pwsh.exe", "powershell.exe")) { break }
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return $pids | Sort-Object -Unique
|
||||
}
|
||||
|
||||
function Stop-StuckBackend {
|
||||
Write-Log "强杀 8000 端口相关进程..." Yellow
|
||||
$targetPids = Get-BackendPids
|
||||
if (-not $targetPids -or $targetPids.Count -eq 0) {
|
||||
Write-Log "未发现 8000 端口监听进程,可能已退出" DarkGray
|
||||
return $true
|
||||
}
|
||||
foreach ($id in $targetPids) {
|
||||
Write-Log " taskkill /T /F PID=$id"
|
||||
taskkill /PID $id /T /F 2>$null | Out-Null
|
||||
}
|
||||
# 等端口释放
|
||||
$waited = 0
|
||||
while ($waited -lt 20) {
|
||||
Start-Sleep -Seconds 1
|
||||
$waited++
|
||||
$still = Get-NetTCPConnection -LocalPort $Port -State Listen -ErrorAction SilentlyContinue
|
||||
if (-not $still) {
|
||||
Write-Log " 端口 $Port 已释放(等待 ${waited}s)" Green
|
||||
return $true
|
||||
}
|
||||
}
|
||||
Write-Log " 端口 $Port 仍占用,放弃自动重启" Red
|
||||
return $false
|
||||
}
|
||||
|
||||
function Start-NewBackend {
|
||||
Write-Log "启动新 uvicorn 进程..." Yellow
|
||||
$ts = Get-Date -Format "yyyyMMdd_HHmmss"
|
||||
$beTmp = Join-Path $env:TEMP "neozqyy_watchdog_be_${ts}.ps1"
|
||||
$q = [char]39
|
||||
# CHANGE 2026-05-05 v3 | 调用 start_uvicorn.py,wildcard 字符串封装在 Python 内部
|
||||
$beLines = @(
|
||||
"`$env:NEOZQYY_ROOT = ${q}${ProjectRoot}${q}"
|
||||
""
|
||||
"if (`$PSVersionTable.PSVersion.Major -lt 7 -and -not `$env:WT_SESSION) {"
|
||||
" chcp 65001 | Out-Null"
|
||||
" `$env:NO_COLOR = ${q}1${q}"
|
||||
" [Console]::OutputEncoding = [System.Text.Encoding]::UTF8"
|
||||
" Write-Host ${q}[提示] PS 5.1 传统控制台,已启用 UTF-8 + 禁用 ANSI 颜色${q} -ForegroundColor Yellow"
|
||||
"}"
|
||||
""
|
||||
"Set-Location -LiteralPath ${q}${backendDir}${q}"
|
||||
"Write-Host ${q}=== 后端 FastAPI (watchdog 自动重启) ===${q} -ForegroundColor Magenta"
|
||||
"Write-Host `"NEOZQYY_ROOT=`$env:NEOZQYY_ROOT`""
|
||||
""
|
||||
"& ${q}${venvPython}${q} ${q}${backendDir}\start_uvicorn.py${q} --port ${Port}"
|
||||
"Write-Host ${q}后端已退出,按任意键关闭...${q} -ForegroundColor Red"
|
||||
"`$null = `$Host.UI.RawUI.ReadKey(${q}NoEcho,IncludeKeyDown${q})"
|
||||
)
|
||||
$beLines | Set-Content -Path $beTmp -Encoding UTF8
|
||||
|
||||
Start-Process $psExe -ArgumentList "-NoExit", "-ExecutionPolicy", "Bypass", "-File", $beTmp | Out-Null
|
||||
Write-Log "新 uvicorn 进程已 spawn,等待 ${StartupGrace}s 启动" Green
|
||||
}
|
||||
|
||||
function Force-RestartBackend {
|
||||
Write-Log "===========================================" Red
|
||||
Write-Log "检测到后端卡死,执行强制重启" Red
|
||||
Write-Log "===========================================" Red
|
||||
if (-not (Stop-StuckBackend)) { return $false }
|
||||
Start-Sleep -Seconds 1
|
||||
Start-NewBackend
|
||||
return $true
|
||||
}
|
||||
|
||||
# ── 主循环 ─────────────────────────────────────────
|
||||
Write-Host ""
|
||||
Write-Host "========================================" -ForegroundColor Cyan
|
||||
Write-Host " 后端看门狗 backend-watchdog" -ForegroundColor Cyan
|
||||
Write-Host "========================================" -ForegroundColor Cyan
|
||||
Write-Log "ProjectRoot = $ProjectRoot" Gray
|
||||
Write-Log "HealthURL = $healthUrl" Gray
|
||||
Write-Log "CheckInterval = ${CheckInterval}s" Gray
|
||||
Write-Log "FailThreshold = $FailThreshold (累计 $($CheckInterval * $FailThreshold)s 不响应触发重启)" Gray
|
||||
Write-Log "StartupGrace = ${StartupGrace}s (重启后宽限期)" Gray
|
||||
Write-Host ""
|
||||
|
||||
$failCount = 0
|
||||
$lastRestart = [DateTime]::MinValue
|
||||
$startupSeen = $false # 首次见到健康才启用探针
|
||||
$restartTimes = @() # 重启时间窗(滑动 1 小时,PS array 避免 List null 异常)
|
||||
|
||||
while ($true) {
|
||||
# grace 期间不探针
|
||||
$sinceRestart = (Get-Date) - $lastRestart
|
||||
if ($sinceRestart.TotalSeconds -lt $StartupGrace) {
|
||||
Start-Sleep -Seconds $CheckInterval
|
||||
continue
|
||||
}
|
||||
|
||||
if (Test-BackendAlive) {
|
||||
if (-not $startupSeen) {
|
||||
Write-Log "首次健康探针 200,看门狗激活" Green
|
||||
$startupSeen = $true
|
||||
} elseif ($failCount -gt 0) {
|
||||
Write-Log "后端恢复 (此前连续失败 $failCount 次)" Green
|
||||
}
|
||||
$failCount = 0
|
||||
} else {
|
||||
if (-not $startupSeen) {
|
||||
# 启动期还没见过 200,先静默等待
|
||||
Start-Sleep -Seconds $CheckInterval
|
||||
continue
|
||||
}
|
||||
$failCount++
|
||||
Write-Log "健康探针失败 ($failCount/$FailThreshold)" Yellow
|
||||
if ($failCount -ge $FailThreshold) {
|
||||
# CHANGE 2026-05-05 | 重启频率限制:每小时最多 N 次,超出停止看门狗自我保护
|
||||
$cutoff = (Get-Date).AddHours(-1)
|
||||
$restartTimes = @($restartTimes | Where-Object { $_ -gt $cutoff })
|
||||
if ($restartTimes.Count -ge $MaxRestartsPerHour) {
|
||||
Write-Log "===========================================" Red
|
||||
Write-Log "已达每小时最大重启次数 $MaxRestartsPerHour,看门狗停止" Red
|
||||
Write-Log "可能是探针误报或后端持续异常,请人工排查" Red
|
||||
Write-Log "===========================================" Red
|
||||
exit 1
|
||||
}
|
||||
if (Force-RestartBackend) {
|
||||
$lastRestart = Get-Date
|
||||
$restartTimes += $lastRestart
|
||||
Write-Log "本小时已重启 $($restartTimes.Count)/$MaxRestartsPerHour 次" DarkYellow
|
||||
$startupSeen = $false
|
||||
}
|
||||
$failCount = 0
|
||||
}
|
||||
}
|
||||
Start-Sleep -Seconds $CheckInterval
|
||||
}
|
||||
@@ -3,6 +3,8 @@
|
||||
# 服务成功启动后自动打开浏览器
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
# CHANGE 2026-05-05 | $watchdogProc 提前到 try 块外,确保 catch / 退出兜底能访问
|
||||
$watchdogProc = $null
|
||||
|
||||
try {
|
||||
# CHANGE 2026-03-07 | 定位项目根目录：从 bat 启动目录推算，不穿透 junction
|
||||
@@ -135,6 +137,12 @@ try {
|
||||
# 2. NO_COLOR 对 uvicorn 不生效 → 加 --no-color 参数
|
||||
# 3. PS 5.1 传统控制台 UTF-8 箭头乱码 → 所有临时脚本开头 chcp 65001
|
||||
# 并统一设 NO_COLOR=1 禁用 ANSI 转义码
|
||||
# CHANGE 2026-05-05 | F1-5a 走查发现 reload 卡死问题:
|
||||
# - 加 --timeout-graceful-shutdown 5:5 秒后 graceful 失败强杀,reload 不再死等
|
||||
# - 配合 backend-watchdog.ps1:连续 30 秒探针失败自动重启
|
||||
# CHANGE 2026-05-05 v3 | 根治 wildcard 展开:
|
||||
# 用 apps/backend/start_uvicorn.py 启动脚本,所有 wildcard 字符串封装在 Python 内部,
|
||||
# PowerShell shell 完全不接触 wildcard,根本上规避 PSNativeCommandArgumentPassing 行为
|
||||
$beLines = @(
|
||||
"`$env:NEOZQYY_ROOT = ${q}${ProjectRoot}${q}"
|
||||
""
|
||||
@@ -149,7 +157,9 @@ try {
|
||||
"Set-Location -LiteralPath ${q}${backendDir}${q}"
|
||||
"Write-Host ${q}=== 后端 FastAPI ===${q} -ForegroundColor Green"
|
||||
"Write-Host `"NEOZQYY_ROOT=`$env:NEOZQYY_ROOT`""
|
||||
"& ${q}${venvPython}${q} -m uvicorn app.main:app --reload --port 8000 --no-use-colors"
|
||||
""
|
||||
"# 调用 start_uvicorn.py(reload-excludes/timeout-graceful-shutdown 等参数硬编码在 Python 内)"
|
||||
"& ${q}${venvPython}${q} ${q}${backendDir}\start_uvicorn.py${q} --port 8000"
|
||||
"Write-Host ${q}后端已退出，按任意键关闭...${q} -ForegroundColor Red"
|
||||
"`$null = `$Host.UI.RawUI.ReadKey(${q}NoEcho,IncludeKeyDown${q})"
|
||||
)
|
||||
@@ -281,6 +291,62 @@ try {
|
||||
else { Write-Host "tenant-admin 等待超时，请手动打开 http://localhost:5174" -ForegroundColor Red }
|
||||
}
|
||||
|
||||
# ── 看门狗启动/停止函数 ──
|
||||
# CHANGE 2026-05-05 | F1-5a 走查发现 reload 卡死问题:
|
||||
# 看门狗在独立窗口运行,周期探针 /health,卡死自动强杀+重启 backend。
|
||||
function Start-Watchdog {
|
||||
$watchdogScript = Join-Path $ProjectRoot "scripts\ops\backend-watchdog.ps1"
|
||||
if (-not (Test-Path $watchdogScript)) {
|
||||
Write-Host " !! 看门狗脚本不存在: $watchdogScript" -ForegroundColor Yellow
|
||||
return $null
|
||||
}
|
||||
Write-Host "[守护] 启动后端看门狗 (后台监控,卡死自动重启) ..." -ForegroundColor DarkCyan
|
||||
# WindowStyle Minimized:看门狗窗口最小化,不打扰主操作
|
||||
$proc = Start-Process $psExe `
|
||||
-ArgumentList "-NoExit", "-ExecutionPolicy", "Bypass", "-File", $watchdogScript, "-ProjectRoot", $ProjectRoot `
|
||||
-WindowStyle Minimized -PassThru
|
||||
return $proc
|
||||
}
|
||||
|
||||
function Stop-Watchdog {
|
||||
param($WdProc)
|
||||
if ($WdProc -and -not $WdProc.HasExited) {
|
||||
Write-Host " 终止看门狗 (PID=$($WdProc.Id))..." -ForegroundColor Yellow
|
||||
taskkill /PID $WdProc.Id /T /F 2>$null | Out-Null
|
||||
}
|
||||
}
|
||||
|
||||
# ── 仅重启后端(保留前端) ──
|
||||
# CHANGE 2026-05-05 | 测试场景下大多数改动只在后端,前端不需要重启;
|
||||
# 此选项跳过前端避免 WS 重连 + 浏览器刷新成本
|
||||
function Restart-BackendOnly {
|
||||
param([ref]$BeProc)
|
||||
$ErrorActionPreference = "Continue"
|
||||
Write-Host ""
|
||||
Write-Host "仅重启后端 (前端 admin-web/tenant-admin 保留)..." -ForegroundColor Yellow
|
||||
if ($BeProc.Value -and -not $BeProc.Value.HasExited) {
|
||||
Write-Host " 终止旧后端 (PID=$($BeProc.Value.Id))..." -ForegroundColor Yellow
|
||||
taskkill /PID $BeProc.Value.Id /T /F 2>$null | Out-Null
|
||||
}
|
||||
# 兜底按端口清理
|
||||
$listeners = Get-NetTCPConnection -LocalPort 8000 -State Listen -ErrorAction SilentlyContinue
|
||||
foreach ($l in $listeners) {
|
||||
taskkill /PID $l.OwningProcess /T /F 2>$null | Out-Null
|
||||
}
|
||||
# 等端口释放
|
||||
$waited = 0
|
||||
while ($waited -lt 15) {
|
||||
Start-Sleep -Seconds 1
|
||||
$waited++
|
||||
$still = Get-NetTCPConnection -LocalPort 8000 -State Listen -ErrorAction SilentlyContinue
|
||||
if (-not $still) { break }
|
||||
}
|
||||
$ErrorActionPreference = "Stop"
|
||||
Write-Host " 端口 8000 已释放,启动新后端..." -ForegroundColor Green
|
||||
$BeProc.Value = Start-Process $psExe -ArgumentList "-NoExit", "-ExecutionPolicy", "Bypass", "-File", $beTmp -PassThru
|
||||
Write-Host " 新后端已启动 (PID=$($BeProc.Value.Id))" -ForegroundColor Green
|
||||
}
|
||||
|
||||
# ── 倒计时函数 ──
|
||||
function Show-Countdown {
|
||||
param([int]$Seconds)
|
||||
@@ -294,7 +360,10 @@ try {
|
||||
|
||||
# ── 首次启动 ──
|
||||
$beProc = $null; $feProc = $null; $taProc = $null
|
||||
$watchdogProc = $null
|
||||
Start-AllServices -BeProc ([ref]$beProc) -FeProc ([ref]$feProc) -TaProc ([ref]$taProc)
|
||||
# CHANGE 2026-05-05 | 启动看门狗(独立窗口,最小化运行)
|
||||
$watchdogProc = Start-Watchdog
|
||||
Wait-AndOpenBrowser
|
||||
|
||||
# ── 交互菜单循环 ──
|
||||
@@ -307,15 +376,22 @@ try {
|
||||
Write-Host " [1] 终止所有服务" -ForegroundColor Yellow
|
||||
Write-Host " [2] 重启所有服务（间隔 5 秒倒计时）" -ForegroundColor Yellow
|
||||
Write-Host " [3] 退出（终止服务并关闭窗口）" -ForegroundColor Yellow
|
||||
Write-Host " [4] 仅重启后端（保留前端,推荐:测试时 reload 卡死/Python 改动）" -ForegroundColor Yellow
|
||||
Write-Host "========================================" -ForegroundColor Cyan
|
||||
$choice = Read-Host "请输入选项 (1/2/3)"
|
||||
$choice = Read-Host "请输入选项 (1/2/3/4)"
|
||||
|
||||
switch ($choice) {
|
||||
"1" {
|
||||
# CHANGE 2026-05-05 | [1] 全停 = 服务 + 看门狗都停(否则看门狗会自动拉起后端)
|
||||
Stop-Watchdog -WdProc $watchdogProc
|
||||
$watchdogProc = $null
|
||||
Stop-AllServices -BeProc $beProc -FeProc $feProc -TaProc $taProc
|
||||
$beProc = $null; $feProc = $null; $taProc = $null
|
||||
}
|
||||
"2" {
|
||||
# CHANGE 2026-05-05 | 重启时先停看门狗,避免它在停-启间隙误判卡死
|
||||
Stop-Watchdog -WdProc $watchdogProc
|
||||
$watchdogProc = $null
|
||||
Stop-AllServices -BeProc $beProc -FeProc $feProc -TaProc $taProc
|
||||
Show-Countdown -Seconds 5
|
||||
# 重新生成日志文件名（避免旧文件锁定）
|
||||
@@ -336,7 +412,8 @@ try {
|
||||
"Set-Location -LiteralPath ${q}${backendDir}${q}"
|
||||
"Write-Host ${q}=== 后端 FastAPI ===${q} -ForegroundColor Green"
|
||||
"Write-Host `"NEOZQYY_ROOT=`$env:NEOZQYY_ROOT`""
|
||||
"& ${q}${venvPython}${q} -m uvicorn app.main:app --reload --port 8000 --no-use-colors"
|
||||
""
|
||||
"& ${q}${venvPython}${q} ${q}${backendDir}\start_uvicorn.py${q} --port 8000"
|
||||
"Write-Host ${q}后端已退出，按任意键关闭...${q} -ForegroundColor Red"
|
||||
"`$null = `$Host.UI.RawUI.ReadKey(${q}NoEcho,IncludeKeyDown${q})"
|
||||
)
|
||||
@@ -372,14 +449,22 @@ try {
|
||||
|
||||
$beProc = $null; $feProc = $null; $taProc = $null
|
||||
Start-AllServices -BeProc ([ref]$beProc) -FeProc ([ref]$feProc) -TaProc ([ref]$taProc)
|
||||
# CHANGE 2026-05-05 | 重启完后重新启动看门狗
|
||||
$watchdogProc = Start-Watchdog
|
||||
Wait-AndOpenBrowser
|
||||
}
|
||||
"3" {
|
||||
Stop-AllServices -BeProc $beProc -FeProc $feProc -TaProc $taProc
|
||||
Stop-Watchdog -WdProc $watchdogProc
|
||||
$watchdogProc = $null
|
||||
$running = $false
|
||||
}
|
||||
"4" {
|
||||
Restart-BackendOnly -BeProc ([ref]$beProc)
|
||||
# 看门狗保持运行(它会在重启 grace 期间不探针)
|
||||
}
|
||||
default {
|
||||
Write-Host " 无效选项，请输入 1、2 或 3" -ForegroundColor Red
|
||||
Write-Host " 无效选项，请输入 1、2、3 或 4" -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -390,6 +475,12 @@ try {
|
||||
Write-Host $_.ScriptStackTrace -ForegroundColor DarkRed
|
||||
}
|
||||
|
||||
# CHANGE 2026-05-05 | 主窗口退出前兜底 kill 看门狗,防止主窗口被强关时看门狗成孤儿进程
|
||||
if ($watchdogProc -and -not $watchdogProc.HasExited) {
|
||||
Write-Host " 兜底终止看门狗 (PID=$($watchdogProc.Id))..." -ForegroundColor DarkGray
|
||||
taskkill /PID $watchdogProc.Id /T /F 2>$null | Out-Null
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "按任意键关闭此窗口..." -ForegroundColor DarkGray
|
||||
$null = $Host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
|
||||
|
||||
Reference in New Issue
Block a user