shy3130
a9af2bce1c
fix(pipeline): enriched 全量重建流式化, 峰值内存与历史长度解耦 (#208/#174)
小内存机器 (2C2G) 延长历史后全量重建必然 OOM 的两个根因:
1. 全量模式把所有批次结果累积在内存 date_buffers 直到统一写盘,
5 年 x 5500 只 (~800 万行) 全表驻留 0.5-1GB;
2. 批次大小按「每只 ~244 行」的年代设定, 历史延长后单批宽表
(指标全列) 瞬时 ~600MB。
修复 (双核模拟实测: 275 万行峰值 RSS 958MB -> 415MB, 耗时 +37%):
- 流式暂存发布: 每批结果立即写暂存文件 (enriched 树外隐藏目录,
不被 **/*.parquet 业务 glob 扫到; polars/duckdb 的 ** 均匹配
点目录), 最后按日期分块流式合并、逐分区原子替换; 日期覆盖
校验与 EnrichedPublication 崩溃语义保持不变, finally 保证
暂存清理, 历史残留按 mtime (24h) 启动清扫;
- 自适应批次: 检测内存 <8GB 时按单批目标行数 15 万收缩,
>=8GB 保持用户设置 (大机器行为不变, 20 核实测耗时持平);
- 刷新路径同修: 启动/盘后的 300 天窗口计算 (repository) 改走
compute_enriched_history_window 分批执行, 与整帧顺序等价
(各步骤均 over(symbol) 分组), 峰值与标的总量解耦。
验证: 新增 5 项回归测试 (流式输出与整帧直算逐列相等/暂存清理/
残留清扫/自适应批次/窗口分批等价), 全套 backend 1498 项通过,
ruff 基线零新增。
2026-09-04 12:40:04 +08:00
..
2026-09-01 22:20:01 +08:00
2026-08-30 19:05:13 +08:00
2026-08-30 19:05:32 +08:00
2026-09-03 13:01:43 +08:00
2026-07-18 18:31:02 +08:00
2026-08-24 17:14:39 +08:00
2026-08-13 20:16:02 +08:00
2026-08-30 19:05:27 +08:00
2026-08-30 19:05:31 +08:00
2026-08-05 17:56:27 +08:00
2026-07-12 13:52:49 +08:00
2026-09-03 13:26:42 +08:00
2026-08-31 22:53:24 +08:00
2026-08-31 22:53:24 +08:00
2026-08-02 10:56:03 +08:00
2026-08-02 10:56:03 +08:00
2026-08-02 10:56:03 +08:00
2026-08-05 17:56:27 +08:00
2026-09-03 13:20:59 +08:00
2026-08-23 19:36:00 +08:00
2026-07-25 22:51:27 +08:00
2026-08-30 19:05:20 +08:00
2026-09-03 14:42:32 +08:00
2026-08-24 14:21:55 +08:00
2026-07-26 18:35:31 +08:00
2026-08-23 19:13:03 +08:00
2026-09-01 22:33:28 +08:00
2026-08-16 23:39:07 +08:00
2026-09-02 20:20:38 +08:00
2026-07-08 13:42:53 +08:00
2026-08-30 19:05:32 +08:00
2026-08-30 19:05:29 +08:00
2026-09-04 12:40:04 +08:00
2026-08-23 19:13:03 +08:00
2026-09-03 13:01:43 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-07-18 00:49:28 +08:00
2026-07-16 14:34:07 +08:00
2026-09-04 11:53:15 +08:00
2026-09-03 13:45:12 +08:00
2026-07-31 19:56:33 +08:00
2026-08-30 18:51:24 +09:00
2026-08-26 00:54:39 +08:00
2026-08-16 23:39:07 +08:00
2026-07-19 16:42:03 +08:00
2026-08-30 19:05:25 +08:00
2026-08-16 23:39:07 +08:00
2026-08-30 19:05:26 +08:00
2026-09-03 14:42:32 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-07-16 12:17:27 +08:00
2026-08-16 23:39:07 +08:00
2026-07-31 19:38:36 +08:00
2026-08-30 22:29:33 +08:00
2026-08-30 22:29:33 +08:00
2026-08-30 19:05:27 +08:00
2026-09-01 22:42:59 +08:00
2026-09-03 17:35:16 +08:00
2026-08-16 23:39:07 +08:00
2026-08-30 19:05:14 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-07-18 00:49:28 +08:00
2026-07-28 09:46:51 +08:00
2026-09-02 20:20:38 +08:00
2026-08-16 23:39:07 +08:00
2026-09-04 11:59:00 +08:00
2026-08-16 23:39:07 +08:00
2026-09-03 13:01:43 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-26 13:19:16 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-08-30 19:05:25 +08:00
2026-08-16 23:39:07 +08:00
2026-08-31 22:53:24 +08:00
2026-09-01 13:04:00 +08:00
2026-08-30 19:05:11 +08:00
2026-08-30 19:05:25 +08:00
2026-07-08 12:12:29 +08:00
2026-08-22 20:26:34 +08:00
2026-07-28 21:13:10 +08:00
2026-08-16 23:39:07 +08:00
2026-07-09 17:38:44 +08:00
2026-08-05 17:57:05 +08:00
2026-08-30 18:38:24 +09:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-07-19 09:36:55 +08:00
2026-07-19 10:06:26 +08:00
2026-07-28 21:13:10 +08:00
2026-08-30 19:05:21 +08:00
2026-08-16 23:39:07 +08:00
2026-07-19 10:06:26 +08:00
2026-07-30 13:09:07 +08:00
2026-08-30 19:05:27 +08:00
2026-08-16 23:39:07 +08:00
2026-08-17 16:49:28 +08:00
2026-08-16 23:39:07 +08:00
2026-07-26 18:35:31 +08:00
2026-09-04 11:59:00 +08:00
2026-08-16 23:39:07 +08:00
2026-07-20 13:01:03 +08:00
2026-08-31 22:53:12 +08:00
2026-09-03 13:26:42 +08:00
2026-09-03 13:45:12 +08:00
2026-08-31 16:50:21 +08:00
2026-08-16 23:39:07 +08:00
2026-08-16 23:39:07 +08:00
2026-07-26 18:35:31 +08:00
2026-09-02 10:53:08 +08:00
2026-08-13 20:16:02 +08:00
2026-07-20 13:01:03 +08:00
2026-08-23 19:29:44 +08:00
2026-07-22 11:21:13 +08:00
2026-08-13 20:16:02 +08:00
2026-08-26 09:36:06 +08:00
2026-08-13 20:16:02 +08:00
2026-08-13 20:16:02 +08:00
2026-07-16 12:17:27 +08:00
2026-08-13 20:16:02 +08:00
2026-08-23 19:36:00 +08:00
2026-08-13 20:16:02 +08:00
2026-08-13 20:16:02 +08:00
2026-08-31 22:53:24 +08:00
2026-08-25 14:07:16 +08:00
2026-09-02 14:48:32 +08:00
2026-07-26 18:35:31 +08:00
2026-08-20 16:15:13 +08:00
2026-07-26 12:19:55 +08:00
2026-09-02 13:36:06 +08:00