- [P20260220-111837] 2026-02-20 11:18:37 +0800
  - summary: CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary: --- ## TASK 1: Full Prod…
  - prompt:
```text
CONTEXT TRANSFER: We are continuing a conversation that had gotten too long. Here is a summary:

---
## TASK 1: Full Production Integration Test (api_full, verify_only, custom window)

**STATUS**: done

**DETAILS**: ETL run `896b9647` completed successfully. 22 ODS tasks: 21 SUCC, 1 FAIL (`ODS_ASSISTANT_LEDGER` — known JSONB bug, fix already in code). `verify_only` mode only executes ODS fetch + verification, skips DWD/DWS/INDEX.

**FILEPATHS**:
- `export/ETL-Connectors/feiqiu/REPORTS/etl_timing_20260220_091414.md`
- `export/ETL-Connectors/feiqiu/REPORTS/consistency_report_20260220_091414.md`

---

## TASK 2: Fix consistency report issues (missing DWD fields + whitelist)

**STATUS**: done

**DETAILS**:
- `principal_change_amount` fix: Added FACT_MAPPINGS expression
- `update_time` fix: Confirmed upstream API has NO update_time field. Added `KNOWN_NO_SOURCE` whitelist in `consistency_checker.py`
- All 735 unit tests passing at time of completion

**FILEPATHS**:
- `apps/etl/connectors/feiqiu/tasks/dwd/dwd_load_task.py`
- `apps/etl/connectors/feiqiu/quality/consistency_checker.py`

---

## TASK 3: ODS_ASSISTANT_LEDGER JSONB fix

**STATUS**: done (code fix applied, awaiting next ETL run to verify)

**DETAILS**: `_mark_missing_as_deleted` in `ods_tasks.py` now detects ALL JSONB columns via `cols_info` udt_name and wraps dict/list values with `Json()`.

**FILEPATHS**: `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py`

---

## TASK 4: Explain increment_only vs increment_verify vs verify_only modes

**STATUS**: done

---

## TASK 5: Explain full increment_verify data pipeline (API→ODS→DWD→DWS→INDEX)

**STATUS**: done

---

## TASK 6: Remove `pipeline` parameter, rename to `flow` everywhere

**STATUS**: in-progress

**DETAILS**:
User wants to completely remove `pipeline` parameter/field name across the entire codebase, replacing with `flow`. No backward compatibility needed — clean break.

**What's been done (actually implemented):**

1. `apps/etl/connectors/feiqiu/orchestration/flow_runner.py` — DONE:
   - `run()` parameter `pipeline` → `flow`
   - Return dict key `"pipeline"` → `"flow"`
   - Docstring comment updated

2. `apps/etl/connectors/feiqiu/orchestration/scheduler.py` — DONE:
   - `PIPELINE_LAYERS` → `FLOW_LAYERS`

3. `apps/etl/connectors/feiqiu/cli/main.py` — DONE:
   - Removed `--pipeline` argument definition entirely
   - Removed `if args.pipeline_deprecated:` block
   - Changed `runner.run(pipeline=args.flow, ...)` → `runner.run(flow=args.flow, ...)`
   - Changed `runner.run(pipeline=None, layers=layers, ...)` → `runner.run(flow=None, layers=layers, ...)`
   - Removed `--pipeline` from module docstring
   - NOTE: `--pipeline-flow` (data_source deprecated param) is intentionally KEPT — it's a separate concept

4. `apps/backend/app/schemas/tasks.py` — DONE:
   - `pipeline: str = "api_ods_dwd"` → `flow: str = "api_ods_dwd"`
   - Docstring updated

5. `apps/backend/app/services/cli_builder.py` — DONE:
   - `config.pipeline` → `config.flow`

6. `apps/backend/app/routers/tasks.py` — DONE:
   - `config.pipeline` → `config.flow`

7. Frontend files — ALL DONE:
   - `apps/admin-web/src/types/index.ts`: `pipeline: string` → `flow: string`, `PipelineDefinition` → `FlowDefinition`
   - `apps/admin-web/src/pages/TaskConfig.tsx`: `pipeline: flow` → `flow: flow`
   - `apps/admin-web/src/App.tsx`: `runningTask.config.pipeline` → `runningTask.config.flow`
   - `apps/admin-web/src/pages/TaskManager.tsx`: `config.pipeline` → `config.flow`
   - `apps/admin-web/src/components/ScheduleTab.tsx`: `pipeline: 'api_full'` → `flow: 'api_full'`

8. Test files — BULK REPLACED via Python scripts (3 rounds):
   - Round 1 (`_rename_pipeline_to_flow.py`): Regex-based replacement of `pipeline=` → `flow=`, `"pipeline"` → `"flow"`, `.pipeline` → `.flow`, `--pipeline` → `--flow` across 15 test files
   - Round 2 (`_rename_pipeline_vars.py`): Cleaned up local variable names (`pipeline` → `flow_name`/`flow_id`), docstrings, comments across 7 files
   - Round 3 (`_rename_pipeline_feature_tags.py`): `etl-pipeline-debug` → `etl-flow-debug` in 3 files

9. `apps/etl/connectors/feiqiu/tests/unit/test_layers_cli.py` — PARTIALLY DONE:
   - `TestFlowParameter` class rewritten to only test `--flow` (removed all `--pipeline` deprecated tests)
   - `TestLayersPipelineMutualExclusion` → `TestLayersFlowMutualExclusion`
   - `TestLayersArgParsing.test_pipeline_still_works` removed
   - Section comment `# 3. --layers 与 --pipeline 互斥` still present (cosmetic)
   - Section comment `# 5. --flow / --pipeline 弃用别名测试` still present (cosmetic)

**What still needs to be done (10 test failures remain):**

The last test run showed 10 failures, 674 passed. The failures are:

A. `test_cli_args.py::TestPipelineAndTasks::test_pipeline_and_tasks_both_parsed` — This entire class needs to be DELETED. It tests `--pipeline` + `--tasks` which no longer exists. The class is at the bottom of the file (lines ~126-141). Also need to remove `pipeline_deprecated=None` from `_make_args` defaults in `TestBuildCliOverrides` (line ~84).

B. `test_e2e_flow.py` — 2 failures. Needs:
   - `runner.run(pipeline="api_ods", ...)` → `runner.run(flow="api_ods", ...)`  (2 occurrences)
   - `result["pipeline"]` → `result["flow"]`  (1 occurrence)
   - `scheduler.run_flow_with_verification(pipeline="api_ods")` → check if this method exists and fix
   - NOTE: `"pipeline.fetch_root"` and `"pipeline.ingest_source_dir"` are AppConfig keys, NOT the pipeline parameter — leave them alone

C. `test_layers_cli.py` — 7 failures listed but the file has already been cleaned. These failures are likely from **pytest cache** (`.pytest_cache/` or `__pycache__`). Need to clear cache and re-run: `pytest tests/unit --cache-clear` or delete `__pycache__` dirs.

**After fixing tests, need to run:**
1. ETL unit tests: `cd apps/etl/connectors/feiqiu && pytest tests/unit -q --tb=short --ignore=tests/unit/test_audit_doc_alignment.py`
2. Backend tests: `cd apps/backend && pytest tests/ -q --tb=short`
3. Monorepo property tests: `pytest tests/ -v` (from root)

**FILEPATHS** (files that were modified):
- `apps/etl/connectors/feiqiu/orchestration/flow_runner.py`
- `apps/etl/connectors/feiqiu/orchestration/scheduler.py`
- `apps/etl/connectors/feiqiu/cli/main.py`
- `apps/backend/app/schemas/tasks.py`
- `apps/backend/app/services/cli_builder.py`
- `apps/backend/app/routers/tasks.py`
- `apps/admin-web/src/types/index.ts`
- `apps/admin-web/src/pages/TaskConfig.tsx`
- `apps/admin-web/src/pages/TaskManager.tsx`
- `apps/admin-web/src/App.tsx`
- `apps/admin-web/src/components/ScheduleTab.tsx`
- `apps/etl/connectors/feiqiu/tests/unit/test_pipeline_runner_properties.py`
- `apps/etl/connectors/feiqiu/tests/unit/test_debug_orchestration_properties.py`
- `apps/etl/connectors/feiqiu/tests/unit/test_layers_cli.py`
- `apps/etl/connectors/feiqiu/scripts/debug/debug_orchestration.py`
- `tests/test_property_5_etl_param_parsing.py`
- All 11 backend test files listed in context transfer

**FILEPATHS** (files that still need changes):
- `apps/etl/connectors/feiqiu/tests/unit/test_cli_args.py` — delete `TestPipelineAndTasks` class, remove `pipeline_deprecated=None` from `_make_args`
- `apps/etl/connectors/feiqiu/tests/unit/test_e2e_flow.py` — fix `pipeline=` → `flow=` and `result["pipeline"]` → `result["flow"]`

**One-time scripts created (can be deleted after task completion):**
- `scripts/ops/_rename_pipeline_to_flow.py`
- `scripts/ops/_rename_pipeline_vars.py`
- `scripts/ops/_rename_pipeline_feature_tags.py`

---

## TASK 7: New `full_window` processing mode

**STATUS**: not-started

**DETAILS**:
- User wants a new `processing_mode="full_window"` distinct from `increment_verify`
- ODS tasks should NOT use cursor to calculate time window; instead use the actual time bounds from the returned JSON data
- No `_run_verification` needed (API data is the source of truth, no cursor drift risk)
- ODS entry still uses `content_hash` dedup + idempotent upsert (unchanged)
- DWD/DWS processing unchanged

**FILEPATHS**:
- `apps/etl/connectors/feiqiu/orchestration/flow_runner.py`
- `apps/etl/connectors/feiqiu/tasks/ods/ods_tasks.py`

---

## KEY ARCHITECTURE:
- Frontend: React + Vite + Ant Design at `apps/admin-web/`
- Backend: FastAPI at `apps/backend/`
- ETL: `apps/etl/connectors/feiqiu/`
- ODS PK = `(id, content_hash)` — snapshot mode, content change = new row
- DWD has no cursor; dim tables use SCD2, fact tables use `fetched_at` window upsert
- DWS has no cursor; delete-before-insert by date range
- Four DB connections: `mcp_pg_etl`, `mcp_pg_etl_test`, `mcp_pg_app`, `mcp_pg_app_test`
- `pipeline_flow` / `--pipeline-flow` is a SEPARATE concept (data_source deprecated param) — NOT part of the pipeline→flow rename
- `"pipeline.fetch_root"` and `"pipeline.ingest_source_dir"` are AppConfig configuration keys — NOT part of the pipeline→flow rename

## USER CORRECTIONS AND INSTRUCTIONS:
- All output text must be in simplified Chinese
- `fetched_at` is ETL ingestion timestamp, NOT business update time
- ODS content_hash change = NEW ROW (snapshot), not in-place update
- `pipeline` is NOT a legacy/deprecated alias for `flow` — user wants it completely removed, not kept for compatibility
- `--pipeline-flow` is a separate deprecated param (for data_source), should NOT be renamed
- Audit reminder pending for high-risk path changes
- Python scripts preferred over PowerShell for complex operations
- `test_audit_doc_alignment.py` has a pre-existing deadline flaky test — ignore it (not related to this task)

## Files to read
- `apps/etl/connectors/feiqiu/tests/unit/test_cli_args.py` (CRITICAL — needs `TestPipelineAndTasks` class deleted and `pipeline_deprecated` removed from `_make_args`)
- `apps/etl/connectors/feiqiu/tests/unit/test_e2e_flow.py` (CRITICAL — needs `pipeline=` → `flow=` and `result["pipeline"]` → `result["flow"]`)
- `apps/etl/connectors/feiqiu/tests/unit/test_layers_cli.py` (verify clean — may just need pytest cache clear)

USER QUERIES(most recent first):
1. Output:
Command executed successfully with no output.

Exit Code: 0
2. Output:
Command executed successfully with no output.

Exit Code: 0
---

METADATA:
The previous conversation had 2 messages.

INSTRUCTIONS:
Continue working until the user query has been fully addressed. Do not ask for clarification - proceed with the work based on the context provided.
IMPORTANT: you need to read from the files to Read section
```
