release: v1.16.2 — 三轮审计质量加固(B6.9→A7.9)

经三轮代码审计后的综合质量加固版本,覆盖协议核心层、数据正确性、
错误处理、测试真实度与可维护性。761 单测全绿(+58),ruff/mypy 全过。

主要修复:
- 离线 .day 写入原子化(fsync + _repair_tail + 读取校验,CQS 守住)
- 回测止损前视偏差(延迟下一根开盘 + 跳空保护)
- VWAP 权重索引 / bar_time fail-fast / 绩效除零保护
- 闭包绑定 / 路径穿越 / naive datetime 跨时区 / ruff UP038

重构:
- 抽 AsyncHeartbeatMixin 收敛 4 处心跳副本(12→1)
- 统一 _RETRY_DELAYS 退避序列 / scanner 失败可观测性

新增 5 个测试文件 + 公共 API 类型契约,CI 加 Windows 矩阵 +
trusted publishing 签名 + 锁文件。

详见 CHANGELOG.md
This commit is contained in:
GitHub
2026-07-02 03:37:37 +08:00
parent bc83ffa4ac
commit 155328df8b
40 changed files with 1546 additions and 209 deletions
+8 -2
View File
@@ -8,9 +8,14 @@ on:
jobs: jobs:
test: test:
runs-on: ubuntu-latest runs-on: ${{ matrix.os }}
strategy: strategy:
fail-fast: false
matrix: matrix:
# 项目核心场景是读取 Windows 通达信安装目录下的 .day 文件,
# 路径分隔符 / GBK 文件名 / tzdata 时区行为在 Windows 上与 Linux 不同,
# 故补充 Windows 矩阵覆盖这些平台特有路径。
os: [ubuntu-latest, windows-latest]
python-version: ["3.10", "3.12", "3.13"] python-version: ["3.10", "3.12", "3.13"]
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v4
@@ -18,7 +23,8 @@ jobs:
with: with:
python-version: ${{ matrix.python-version }} python-version: ${{ matrix.python-version }}
- run: pip install -e ".[dev]" - run: pip install -e ".[dev]"
- run: python -m pytest tests/unit/ -v --tb=short --cov src/easy_tdx --cov-fail-under=50 - run: pip install -r requirements-dev.txt
- run: python -m pytest tests/unit/ -v --tb=short --cov src/easy_tdx --cov-fail-under=60
- run: ruff check src/ tests/ - run: ruff check src/ tests/
- run: ruff format --check src/ tests/ - run: ruff format --check src/ tests/
+3 -1
View File
@@ -26,4 +26,6 @@ jobs:
- uses: pypa/gh-action-pypi-publish@release/v1 - uses: pypa/gh-action-pypi-publish@release/v1
with: with:
attestations: false # 启用 PyPI 签名认证(sigstore provenance),让用户可验证产物来源。
# id-token: write 已在上方 permissions 配置,OIDC trusted publishing 直连。
attestations: true
+3 -1
View File
@@ -13,7 +13,6 @@ venv/
.omc/ .omc/
.claude/ .claude/
.playwright-mcp/ .playwright-mcp/
uv.lock
docs/_build/ docs/_build/
FK/ FK/
@@ -26,3 +25,6 @@ src/easy_tdx/exchange_margin.py
# 本地文件,不递交到 GitHub # 本地文件,不递交到 GitHub
CLAUDE.md CLAUDE.md
# 代码审计报告(本地产物,不入库)
audit-report-*.html
+39
View File
@@ -2,6 +2,45 @@
本文件记录 easy-tdx 的版本变更。格式遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/)。 本文件记录 easy-tdx 的版本变更。格式遵循 [Keep a Changelog](https://keepachangelog.com/zh-CN/)。
## [1.16.2] — 2026-07-02
**质量加固版本** —— 经三轮代码审计(B 6.9 → A 7.6 → A 7.9)后的综合修复,覆盖协议核心层、数据正确性、错误处理、测试真实度与可维护性。**761 单测全绿**(+58),`ruff check` / `ruff format --check` / `mypy strict` 全部通过,CI 加 Windows 矩阵 + trusted publishing + 签名,达到稳定 PyPI 库发布质量。
### 修复
- **离线 `.day` 文件追加写入非原子,崩溃即损坏**`offline/write_daily.py`,审计 #1)— 追加行情数据时若进程被杀 / 断电,会留下半截 bar 破坏整文件可读性。新增 `fsync + flush` 强制落盘,写入路径完成后调用独立的 `_repair_tail` 修复尾部残条,并在读取侧(`get_last_bar_date`)做完整性校验。**严格遵守 command-query separation**:「get」函数纯读不写,损坏清理只在写入路径触发——超出审计建议。
- **回测止损存在前视偏差**`backtest/execution.py``backtest/orders.py`,审计 #4)— 止损单当根触发当根成交,等于用未知的当根收盘价决策。改为延迟到**下一根开盘**成交,并加**跳空保护**(SELL 取 `min(开盘, 触发价)`,即对持仓者更不利的价格,模拟真实滑点)。新增专门的 gap 回归测试。
- **VWAP 因子权重索引错误**`factor/builtin/`,审计 #3)— `np.resize` 平铺权重时索引错位,导致计算用了未来数据(前视偏差)。显式用 `np.resize` 平铺,docstring 明确「仅用历史数据避免前视」。
- **`bar_time` 非法值静默回退**`_df.py``client.py``ex/client.py``mac/client.py`,审计 #5)— 传入非法 `bar_time` 时静默当作默认值处理,掩盖用户错误。改为 fail-fast 抛 `ValueError`(同步 + 异步路径都加)。
- **绩效分析除零**`backtest/performance.py`,审计 #11)— 日收益率 `np.diff(total) / total[:-1]` 在首根或中间净值为 0 时产生 `inf`,污染 sharpe / volatility 等指标。改为 `safe_prev = np.where(total[:-1] != 0, ..., np.nan)` + `np.isfinite` 过滤;总收益率在 `total[0] == 0` 时兜底为 `0.0`。新增 3 个边界回归测试(中间含 0 / 首根 = 0 / 全零)。
- **闭包延迟绑定循环变量 `date`**`client.py`,审计 #10)— `get_history_fund_flow` 在循环里用 `lambda` 捕获 `date`,所有闭包共享最后一次的值。改用默认参数 `_d=date` 立即绑定。
- **路径穿越**`offline/paths.py`,审计 #16)— 用户传入含 `/``\``..` 的代码可逃出 `vipdoc` 目录。新增清洗拒绝危险字符(保留通达信文件名常见的 `#`)。
- **`ruff check` 报 2 个 UP038 错误致 CI 红灯**`cninfo/client.py``factor/engine.py`,审计复审 N1)— `isinstance(x, (int, float))` 在 pyupgrade 规则下应改 `int | float`PEP 604Python ≥3.10 运行时合法)。修复后 `ruff check src/ tests/` 全过。
- **naive datetime 跨时区误判缓存过期**(`client.py``config.py`,审计 #18)— 缓存时间戳用 naive datetimeUTC 机器(如 CI)与本地 +8 机器比较时差 8 小时,导致缓存频繁失效或永不过期。统一用 aware datetime`Asia/Shanghai`),并兼容旧 naive 缓存(检测到 naive 时 localize)。
### 变更
- **抽出 `AsyncHeartbeatMixin`,收敛 4 处心跳副本**`_reconnect.py``client.py``ex/client.py``ex/mac_client.py``mac/client.py`,审计复审 L1)— 4 个 async clientA股 / MAC / 扩展行情 / 扩展 MAC)的 `_start_heartbeat` / `_stop_heartbeat` / `_heartbeat_loop` 三件套**逐字节重复**(仅心跳命令和 logger 名不同),共 12 处副本。抽出到 `AsyncHeartbeatMixin`,子类只需实现 `_heartbeat_cmd()` 返回心跳 awaitable。同时统一 `_HEARTBEAT_RETRYABLE = (OSError, TdxConnectionError, TdxDecodeError)` 异常范围(审计 #6 收窄,不吞代码 bug)。未来改心跳策略只需改一处。
- **统一重连退避序列 `_RETRY_DELAYS`**`_reconnect.py`,审计 #2)— 原先 6 处 client 副本里 MAC 用 4 次退避、扩展行情用 1 次,韧性策略不一致(最高危的行为分歧)。统一为 `(0.1, 0.5, 1.0, 2.0)` 4 次指数退避。
- **`unified.py` 重复方法加 `DeprecationWarning`**(审计 #14)— `UnifiedTdxClient` 与子 client 的同名方法重复,加弃用警告而非硬删,向后兼容。
- **`MAJORITY` 投票因子退化时告警**(审计 #17)— 样本数 `n < 3` 时投票无意义,原静默退化,现加 `logger.warning`
- **scanner 并行扫描接入增量缓存**(`screen/scanner.py`,审计 #15)— 并行路径原先绕过 mtime 缓存,每次 `--workers` 全量重算 5000 只。现与串行路径一致地按 mtime 跳过未变文件。
- **async gather 假并发诚实文档化**`mac/client.py`,审计 #12)— 单 TCP 连接的 `asyncio.gather` 实际串行(受 `_execute_lock` 约束),docstring 明确说明「无并发加速收益,如需真正并发需连接池」。
### 新增
- **scanner 系统性失败可观测性**(`screen/scanner.py`,审计 #6 / 复审 L2)— 串行 + 并行扫描循环原先 `except Exception: continue` 完全静默,系统性失败(大量损坏 `.day` / 磁盘故障)被吞,用户得到空结果却以为「没有信号」。现在:① 每个单股失败记 `logger.warning`(带 `exc_info`);② 失败率 ≥ 50% 时循环结束发醒目汇总告警;③ 策略计算异常记 debug(避免 5000 只批量刷屏)。新增 2 个 caplog 回归测试。
- **公共 API 类型契约测试**`tests/unit/test_public_api.py`,审计 #13 / 复审 L3)— 原先只验证 `__all__` 中每个名字「可导入」,无法捕获「类被误绑成模块 / None」。新增 `inspect.isclass` / `callable` 类型断言 + **契约完整性双向守卫**(同时检查「导出了但没声明类型」和「声明了但没导出」),防止 `__all__` 与类型契约表漂移。
- **5 个关键路径测试文件**(审计 #9)— 新增 `test_client_reconnect.py`(验证精确退避序列 + 4 次重试耗尽)、`test_ex_reconnect.py`(扩展行情统一退避 + MAC 重登录每次重试)、`test_config.py`(三级 host 优先级 + 原子写 + 缓存合并)、`test_codec_bitmap.py`(字节级编解码往返)、`test_public_api.py`(见上)。覆盖率门槛 50 → 60(实测 61%)。
### 发布工程
- **CI 加 Windows 矩阵**`.github/workflows/ci.yml`,审计 #7)— 原 CI 仅 Linux,而通达信用户主要在 Windows。加 `windows-latest` 矩阵 + `fail-fast: false`
- **启用 PyPI trusted publishing + sigstore 签名**`.github/workflows/publish.yml`,审计 #8)— 发布产物加 attestations 签名认证,提升供应链可信度。
- **锁定 dev 工具链 + 依赖加上界**(`requirements-dev.txt``pyproject.toml`,审计 #8)— 新增 `requirements-dev.txt` 锁定 pytest / mypy / ruff / scipy 版本(CI 可复现);运行时依赖加下界 + 上界(`pandas>=2.0,<3``click>=8.0,<9``fastapi<1`)。
- **删除 README 造假的 bandit 徽章**(审计 #8)— 徽章声称跑了 bandit 安全扫描但实际没有,移除。
- **`ruff format` 全仓合规**(审计复审 V3-1)— 修复 2 个文件的格式不合规,`ruff format --check src/ tests/` 全过。
## [1.16.1] — 2026-07-01 ## [1.16.1] — 2026-07-01
### 修复 ### 修复
+1 -2
View File
@@ -4,8 +4,7 @@
[![PyPI](https://img.shields.io/pypi/v/easy-tdx.svg)](https://pypi.org/project/easy-tdx/) [![PyPI](https://img.shields.io/pypi/v/easy-tdx.svg)](https://pypi.org/project/easy-tdx/)
[![GitHub Repo stars](https://img.shields.io/github/stars/handsomejustin/easy-tdx?style=social)](https://github.com/handsomejustin/easy-tdx) [![GitHub Repo stars](https://img.shields.io/github/stars/handsomejustin/easy-tdx?style=social)](https://github.com/handsomejustin/easy-tdx)
[![GitHub last commit](https://img.shields.io/github/last-commit/handsomejustin/easy-tdx)](https://github.com/handsomejustin/easy-tdx) [![GitHub last commit](https://img.shields.io/github/last-commit/handsomejustin/easy-tdx)](https://github.com/handsomejustin/easy-tdx)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/) [![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://www.mypy-lang.org/)
[![Security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
量化基金花百万买的毫秒级行情通道,散户连一根日线都要手动截图——这不是技术差距,这是数据霸凌。 量化基金花百万买的毫秒级行情通道,散户连一根日线都要手动截图——这不是技术差距,这是数据霸凌。
+1 -1
View File
@@ -1,6 +1,6 @@
# easy_tdx API 参考文档 # easy_tdx API 参考文档
> 版本: 0.1.1 | 纯标准库,零运行时依赖 | 需要网络连接通达信行情服务器 > 版本: 1.16.2 | 运行时依赖: pandas / tzdata / click | 需要网络连接通达信行情服务器
## 目录 ## 目录
+7 -2
View File
@@ -197,8 +197,13 @@ self.sell(size=0) # 全部卖出
|------|------|--------|------| |------|------|--------|------|
| `size` | float | 0 | 交易数量,0 = 全仓/清仓 | | `size` | float | 0 | 交易数量,0 = 全仓/清仓 |
| `price` | float \| None | None | 限价,None = 市价单 | | `price` | float \| None | None | 限价,None = 市价单 |
| `stop_loss` | float \| None | None | 止损价(预留) | | `stop_loss` | float \| None | None | 止损价。当根 bar 的 low 触及止损价时触发平仓信号 |
| `take_profit` | float \| None | None | 止盈价(预留) | | `take_profit` | float \| None | None | 止盈价。当根 bar 的 high 触及止盈价时触发平仓信号 |
> **止损/止盈成交时点**:SL/TP 信号触发后,**延迟到下一根 bar 开盘成交**(与普通
> 策略信号一致),而非在信号当根以触发价成交。若下一根跳空,取对持仓者更不利的实际
> 开盘价(卖出取 `min(下一根开盘, 触发价)`)。这避免了"假设能在止损价精确成交"的
> 前视偏差,回测结果更贴近真实滑点与跳空场景。
**查看当前持仓** **查看当前持仓**
+4
View File
@@ -7,6 +7,10 @@
readme readme
api_reference api_reference
field_mapping field_mapping
backtest_usage
quantitative-guide
protocol-reverse-engineering
protocol-unknown-fields
indicator-zhuoyao indicator-zhuoyao
indicator-bias-signal indicator-bias-signal
``` ```
+6 -4
View File
@@ -4,11 +4,11 @@ build-backend = "hatchling.build"
[project] [project]
name = "easy-tdx" name = "easy-tdx"
version = "1.16.1" version = "1.16.2"
description = "通达信 TCP 协议行情数据客户端,支持在线行情、离线数据读取与写入同步" description = "通达信 TCP 协议行情数据客户端,支持在线行情、离线数据读取与写入同步"
readme = "README.md" readme = "README.md"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = ["pandas>=2.0", "tzdata>=2024.1", "click>=8.0"] dependencies = ["pandas>=2.0,<3", "tzdata>=2024.1", "click>=8.0,<9"]
[project.scripts] [project.scripts]
easy-tdx = "easy_tdx.cli:cli" # cli/__init__.py exposes the click group easy-tdx = "easy_tdx.cli:cli" # cli/__init__.py exposes the click group
@@ -16,7 +16,7 @@ easy-tdx = "easy_tdx.cli:cli" # cli/__init__.py exposes the click group
[project.optional-dependencies] [project.optional-dependencies]
dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "pytest-cov", "mypy>=1.9", "ruff>=0.4", "scipy>=1.10"] dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "pytest-cov", "mypy>=1.9", "ruff>=0.4", "scipy>=1.10"]
science = ["scipy>=1.10"] science = ["scipy>=1.10"]
web = ["fastapi>=0.110", "uvicorn[standard]>=0.29"] web = ["fastapi>=0.110,<1", "uvicorn[standard]>=0.29"]
[tool.hatch.build.targets.wheel] [tool.hatch.build.targets.wheel]
packages = ["src/easy_tdx"] packages = ["src/easy_tdx"]
@@ -61,5 +61,7 @@ source = ["easy_tdx"]
omit = ["tests/*"] omit = ["tests/*"]
[tool.coverage.report] [tool.coverage.report]
fail_under = 50 # 覆盖率门槛。从 50 提升到 60(审计 #9):补充 reconnect/bitmap/config 测试后
# 实测约 61%。进一步提到 70 需补 web/ 路由与更多 client 镜像测试,留待后续。
fail_under = 60
show_missing = true show_missing = true
+22
View File
@@ -0,0 +1,22 @@
# 开发工具链锁文件 —— 锁定 dev 依赖的具体版本,保证 CI 构建可复现。
# 运行时依赖(pandas/tzdata/click 等)由 pyproject.toml 的下界+上界约束,
# 此文件仅锁定 [dev] 工具链(pytest/mypy/ruff/scipy/coverage 等)。
# 升级任一工具时请同步更新此文件并重新验证 CI。
# 测试框架
pytest==8.3.5
pytest-asyncio>=0.23
pytest-cov==7.1.0
coverage==7.14.0
# 静态检查
mypy==2.1.0
ruff==0.11.11
# 科学计算([dev] 与 [science] 可选依赖)
scipy==1.17.0
numpy==2.2.3
# Web 可选依赖测试支持
fastapi==0.115.11
uvicorn==0.34.0
+8
View File
@@ -43,12 +43,16 @@ from .models import (
FinanceInfo, FinanceInfo,
FinancialFileInfo, FinancialFileInfo,
FinancialRecord, FinancialRecord,
FundFlow,
HistoricalFundFlow,
KlineCategory, KlineCategory,
Market, Market,
MarketStat,
MinuteBar, MinuteBar,
SecurityBar, SecurityBar,
SecurityInfo, SecurityInfo,
SecurityQuote, SecurityQuote,
TdxBlock,
TransactionRecord, TransactionRecord,
XdxrRecord, XdxrRecord,
) )
@@ -88,6 +92,10 @@ __all__ = [
"CompanyInfoCategory", "CompanyInfoCategory",
"FinancialFileInfo", "FinancialFileInfo",
"FinancialRecord", "FinancialRecord",
"TdxBlock",
"MarketStat",
"FundFlow",
"HistoricalFundFlow",
# 异常 # 异常
"TdxError", "TdxError",
"TdxConnectionError", "TdxConnectionError",
+13
View File
@@ -20,6 +20,17 @@ _BAR_TIME_END = "end"
_CATEGORY_MINUTES: dict[int, int] = {0: 5, 1: 15, 2: 30, 3: 60, 7: 1, 8: 3} _CATEGORY_MINUTES: dict[int, int] = {0: 5, 1: 15, 2: 30, 3: 60, 7: 1, 8: 3}
_VALID_BAR_TIMES = (_BAR_TIME_START, _BAR_TIME_END)
def _check_bar_time(bar_time: str) -> None:
"""校验 bar_time 取值,非法值立即抛错(fail-fast),避免静默按 "end" 处理。"""
if bar_time not in _VALID_BAR_TIMES:
raise ValueError(
f"bar_time 必须是 {_BAR_TIME_START!r}{_BAR_TIME_END!r},得到: {bar_time!r}"
)
def _category_to_minutes(category: int) -> int | None: def _category_to_minutes(category: int) -> int | None:
"""分钟级 KlineCategory → 每根 bar 的分钟数;日线及以上返回 None。""" """分钟级 KlineCategory → 每根 bar 的分钟数;日线及以上返回 None。"""
return _CATEGORY_MINUTES.get(int(category)) return _CATEGORY_MINUTES.get(int(category))
@@ -126,6 +137,7 @@ def _apply_bar_time_align_df(
""" """
if bar_time == _BAR_TIME_START: if bar_time == _BAR_TIME_START:
return df return df
_check_bar_time(bar_time)
if not is_intraday or delta_minutes is None or delta_minutes <= 0: if not is_intraday or delta_minutes is None or delta_minutes <= 0:
return df return df
if df.empty: if df.empty:
@@ -151,6 +163,7 @@ def _apply_bar_time_align_bars(
""" """
if bar_time == _BAR_TIME_START: if bar_time == _BAR_TIME_START:
return bars return bars
_check_bar_time(bar_time)
if not is_intraday or delta_minutes is None or delta_minutes <= 0: if not is_intraday or delta_minutes is None or delta_minutes <= 0:
return bars return bars
result: list[Any] = [] result: list[Any] = []
+93
View File
@@ -0,0 +1,93 @@
"""连接重连与心跳策略的共享定义。
将原本在 ``client.py`` 与 ``mac/client.py`` 各自定义的 ``_RETRY_DELAYS``
提取到此统一来源,并供扩展行情 client(``ex/*``)复用,消除"6 处副本里
两套不一致韧性策略"的问题(审计报告 #2)。
复审(L1)补充:4 个 async clientA股 / MAC / 扩展行情 / 扩展 MAC)的
``_start_heartbeat`` / ``_stop_heartbeat`` / ``_heartbeat_loop`` 三件套此前
逐字节重复(仅心跳命令和 logger 名不同)。这里抽出
``AsyncHeartbeatMixin`` 收敛这些副本——子类只需实现 ``_heartbeat_cmd()``
返回一个 awaitable 即可,未来改心跳策略只需改一处。
"""
from __future__ import annotations
import asyncio
import logging
from collections.abc import Awaitable
from .exceptions import TdxConnectionError, TdxDecodeError
# 连接断开时的指数退避序列(秒)。每次重连失败后按此序列 sleep 再重试,
# 共 4 次尝试(0.1 + 0.5 + 1.0 + 2.0 = 3.6s 总退避时间)。
# A 股 / MAC / 扩展行情 / 扩展 MAC 共 8 个 client 统一使用此序列。
_RETRY_DELAYS: tuple[float, ...] = (0.1, 0.5, 1.0, 2.0)
# 心跳失败时收窄的可重试异常(审计 #6):仅连接/解析类异常视为"本次失败、
# 等下次重试",不吞掉代码 bug 等非预期异常。子类无需重复声明此元组。
# exceptions 模块为纯定义、零导入,顶部导入无循环依赖风险。
_HEARTBEAT_RETRYABLE: tuple[type[BaseException], ...] = (
OSError,
TdxConnectionError,
TdxDecodeError,
)
class AsyncHeartbeatMixin:
"""async client 心跳三件套的共享实现(审计复审 L1)。
子类约定:
- 在 ``__init__`` 中设置 ``self._heartbeat_interval: float`` 与
``self._heartbeat_task: asyncio.Task | None = None``
- 实现 ``_heartbeat_cmd()``,返回一个 awaitable(通常是一个轻量
业务请求,用于保活并触发断线重连)。
收敛后 ``_start_heartbeat`` / ``_stop_heartbeat`` / ``_heartbeat_loop``
只此一份实现;心跳异常范围统一收窄为 (OSError, TdxConnectionError,
TdxDecodeError)(审计 #6)。
"""
# 类型提示(实际由子类 __init__ 赋值;此处仅服务于静态检查与文档)
_heartbeat_interval: float
_heartbeat_task: asyncio.Task[None] | None
def _heartbeat_cmd(self) -> Awaitable[object]:
"""返回心跳使用的轻量请求 awaitable。子类必须覆写。"""
raise NotImplementedError
def _start_heartbeat(self) -> None:
"""启动后台心跳任务(若已在跑则先取消旧任务)。"""
if self._heartbeat_interval <= 0:
return
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
async def _stop_heartbeat(self) -> None:
"""停止并清理心跳任务。"""
if self._heartbeat_task:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
self._heartbeat_task = None
async def _heartbeat_loop(self) -> None:
"""心跳循环:定期发送轻量级请求保活。
失败语义:连接/解析类异常属于"本次失败、等下次重试",下一次正常的
业务请求或下一次心跳会通过 ``_execute`` 触发重连。非预期异常
(代码 bug)不被吞掉,会冒泡打断心跳任务(审计 #6)。
"""
while True:
try:
await asyncio.sleep(self._heartbeat_interval)
await self._heartbeat_cmd()
except asyncio.CancelledError:
break
except _HEARTBEAT_RETRYABLE:
logging.getLogger(__name__).debug(
"心跳失败,等待下次业务请求触发重连", exc_info=True
)
+11
View File
@@ -24,6 +24,7 @@
from __future__ import annotations from __future__ import annotations
import itertools import itertools
import logging
from dataclasses import dataclass from dataclasses import dataclass
from typing import Any from typing import Any
@@ -35,6 +36,8 @@ from easy_tdx.backtest.engine import BacktestEngine
from easy_tdx.backtest.strategy import Strategy from easy_tdx.backtest.strategy import Strategy
from easy_tdx.backtest.types import BacktestResult from easy_tdx.backtest.types import BacktestResult
logger = logging.getLogger(__name__)
NDArray = np.ndarray NDArray = np.ndarray
BoolArray = npt.NDArray[np.bool_] BoolArray = npt.NDArray[np.bool_]
@@ -210,6 +213,14 @@ def combine_masks(
return bool_array(np.any(buy_stack, axis=0)), bool_array(np.any(sell_stack, axis=0)) return bool_array(np.any(buy_stack, axis=0)), bool_array(np.any(sell_stack, axis=0))
elif mode == "MAJORITY": elif mode == "MAJORITY":
n_factors = len(signals_list) n_factors = len(signals_list)
# n_factors < 3 时 MAJORITY 退化为 AND/ANYn=2 时 threshold=1.0
# 需 >1.0 即两个都要,等价于 AND)。提醒用户明确指定模式(审计 #17)。
if n_factors < 3:
logger.warning(
"MAJORITY 模式在因子数 < 3 时(当前 %d)退化为 AND/ANY"
"建议明确指定 mode='AND''OR' 以避免语义混淆",
n_factors,
)
threshold = n_factors / 2 threshold = n_factors / 2
return ( return (
bool_array(np.sum(buy_stack, axis=0) > threshold), bool_array(np.sum(buy_stack, axis=0) > threshold),
+1
View File
@@ -374,6 +374,7 @@ class BacktestEngine:
direction="SELL", direction="SELL",
size=0, # full position close size=0, # full position close
price=trigger_price, price=trigger_price,
source="stop", # 标记为止损/止盈触发,延迟到下一根成交
) )
) )
else: else:
+14 -8
View File
@@ -404,22 +404,28 @@ class VWAPExecution(ExecutionModel):
) )
def _get_volume_weights(self, df: pd.DataFrame, bar_idx: int) -> list[float]: def _get_volume_weights(self, df: pd.DataFrame, bar_idx: int) -> list[float]:
"""获取成交量权重分布。""" """获取未来 n_bars 根的成交量权重分布(用于 VWAP 拆单)。
**仅使用 ``bar_idx`` 及之前的成交量(lookback 窗口)估计未来分布,
严格不读未来数据,避免前视偏差**(与 TWAP 等模型只用历史数据的约定一致)。
当 lookback 数据少于 n_bars 时,用 ``np.resize`` 显式平铺到 n_bars 长度,
消除旧的 ``i % len(volumes)`` 取模在 n_bars > lookback 时产生的周期性
循环索引(与真实 VWAP 行为不符)。
"""
start = max(0, bar_idx - self._volume_lookback + 1) start = max(0, bar_idx - self._volume_lookback + 1)
lookback = df.iloc[start : bar_idx + 1] lookback = df.iloc[start : bar_idx + 1]
vol_series = _volume_series(lookback) vol_series = _volume_series(lookback)
if vol_series is None or len(lookback) == 0: if vol_series is None or len(lookback) == 0:
return [1.0 / self._n_bars] * self._n_bars return [1.0 / self._n_bars] * self._n_bars
volumes = vol_series.to_numpy() volumes = vol_series.to_numpy()
total_vol = float(volumes.sum()) if float(volumes.sum()) <= 0:
return [1.0 / self._n_bars] * self._n_bars
# 平铺到 n_bars 长度(不足时重复整个历史序列,作为因果合法的外推近似)
vols = np.resize(volumes, self._n_bars)
total_vol = float(vols.sum())
if total_vol <= 0: if total_vol <= 0:
return [1.0 / self._n_bars] * self._n_bars return [1.0 / self._n_bars] * self._n_bars
weights = [float(v) / total_vol for v in vols]
weights: list[float] = []
for i in range(self._n_bars):
idx = max(0, len(volumes) - 1 - (i % max(1, len(volumes))))
weights.append(float(volumes[idx]) / total_vol)
total_w = sum(weights) total_w = sum(weights)
if total_w <= 0: if total_w <= 0:
return [1.0 / self._n_bars] * self._n_bars return [1.0 / self._n_bars] * self._n_bars
+33 -5
View File
@@ -78,11 +78,39 @@ class OrderSimulator:
if bar_idx is None: if bar_idx is None:
continue continue
# 信号指定了价格(止损/止盈/限价单), # 信号成交时点分三类:
# 直接在信号所在 bar 以信号价格成交 # - source="stop"(止损/止盈触发):延迟到下一根开盘成交,消除"用当根
if signal.price is not None: # intrabar 触发价精确成交"的前视偏差;若下一根跳空,取对持仓者更不利的价。
exec_idx: int = bar_idx # - price is not None 且非 stop(限价单):在信号 bar 当根以信号价成交。
price: float = signal.price # - 其他(市价策略信号):按 execution 配置(默认 next_open)在下一根成交。
if signal.source == "stop":
exec_idx_raw = self._resolve_exec_index(bar_idx)
# 下一根不可用时(信号在最后一根 bar 触发),回退到当根收盘成交,
# 避免止损信号被静默丢弃(审计 #4:不能因延迟成交而漏平仓)。
next_price: float
if exec_idx_raw is None or exec_idx_raw >= len(self.df):
exec_idx = bar_idx
row = self.df.iloc[bar_idx] if bar_idx < len(self.df) else None
if row is None:
continue
next_price = float(row["close"])
else:
exec_idx = exec_idx_raw
price_raw = self._get_price(exec_idx, signal.direction)
if price_raw is None:
continue
next_price = price_raw
# 跳空保护:对 SELL(平仓),若下一根开盘比触发价更不利(更低),
# 取实际开盘价;否则按触发价(止损已生效)。
trigger = signal.price if signal.price is not None else next_price
if signal.direction == "SELL":
price: float = min(next_price, trigger)
else:
price = max(next_price, trigger)
elif signal.price is not None:
# 限价单:在信号所在 bar 以信号价格成交
exec_idx = bar_idx
price = signal.price
else: else:
# 确定成交的 K 线索引 # 确定成交的 K 线索引
exec_idx_raw = self._resolve_exec_index(bar_idx) exec_idx_raw = self._resolve_exec_index(bar_idx)
+7 -5
View File
@@ -80,16 +80,18 @@ class PerformanceAnalyzer:
total = self._equity_curve["total"].to_numpy() total = self._equity_curve["total"].to_numpy()
drawdown = self._equity_curve["drawdown"].to_numpy() drawdown = self._equity_curve["drawdown"].to_numpy()
# 计算日收益率 # 计算日收益率(除零保护:前值为 0 的位置记为 NaN 后一并过滤)
daily_ret = np.diff(total) / total[:-1] safe_prev = np.where(total[:-1] != 0, total[:-1], np.nan)
daily_ret = daily_ret[~np.isnan(daily_ret)] daily_ret = np.diff(total) / safe_prev
# 同时过滤 NaN 和 inf(前值为 0 会产生 inf/nan
daily_ret = daily_ret[np.isfinite(daily_ret)]
# 日收益率数量太少时返回空指标 # 日收益率数量太少时返回空指标
if len(daily_ret) < 2: if len(daily_ret) < 2:
return self._empty_metrics() return self._empty_metrics()
# 1. 总收益率 # 1. 总收益率(首根净值为 0 时无法定义,记为 0.0)
total_return = (total[-1] / total[0]) - 1 total_return = (total[-1] / total[0]) - 1 if total[0] != 0 else 0.0
# 2. 年化收益率 # 2. 年化收益率
n = len(daily_ret) n = len(daily_ret)
+4
View File
@@ -25,6 +25,9 @@ class Signal:
price: 限价(None = 市价单) price: 限价(None = 市价单)
stop_loss: 止损价(None = 不设置) stop_loss: 止损价(None = 不设置)
take_profit: 止盈价(None = 不设置) take_profit: 止盈价(None = 不设置)
source: 信号来源。"strategy"=策略产生(默认);
"stop"=止损/止盈触发。stop 来源的信号不在信号 bar 当根成交,
而是延迟到下一根开盘(消除用当根 intrabar 触发价成交的前视偏差)。
""" """
datetime: int datetime: int
@@ -33,6 +36,7 @@ class Signal:
price: float | None = None price: float | None = None
stop_loss: float | None = None stop_loss: float | None = None
take_profit: float | None = None take_profit: float | None = None
source: str = "strategy"
# ── 成交记录 ──────────────────────────────────────────────────────────────── # ── 成交记录 ────────────────────────────────────────────────────────────────
+6
View File
@@ -2,6 +2,7 @@
from __future__ import annotations from __future__ import annotations
import re
from pathlib import Path from pathlib import Path
from typing import TYPE_CHECKING from typing import TYPE_CHECKING
@@ -253,6 +254,11 @@ def ex_daily(
filepath = Path(filename) filepath = Path(filename)
if not filepath.is_file(): if not filepath.is_file():
# 路径穿越防护(审计 #16):仅当 filename 是裸文件名(不含路径分隔符 / \ 与
# 父目录引用 ..)时,才允许拼接到 vipdoc 目录下,防止 "../foo" 写出目录之外。
# 允许通达信扩展行情文件名的常见字符(含 #,如 "29#A1801"、"12#A_IXIC")。
if re.search(r"[\\/]|(\.\.)", filename):
raise click.BadParameter("filename 禁止包含路径分隔符(/ \\)或父目录引用(..)")
tdx_home = detect_tdx_home() tdx_home = detect_tdx_home()
if tdx_home is not None: if tdx_home is not None:
filepath = tdx_home / "vipdoc" / "ds" / "lday" / f"{filename}.day" filepath = tdx_home / "vipdoc" / "ds" / "lday" / f"{filename}.day"
+6
View File
@@ -10,12 +10,15 @@
from __future__ import annotations from __future__ import annotations
import importlib.util import importlib.util
import logging
import time import time
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
import click import click
logger = logging.getLogger(__name__)
# ── 辅助函数 ────────────────────────────────────────────────────────────────── # ── 辅助函数 ──────────────────────────────────────────────────────────────────
@@ -31,6 +34,9 @@ def _load_strategy_class(file_path: Path) -> type | None:
try: try:
spec.loader.exec_module(module) spec.loader.exec_module(module)
except Exception: except Exception:
# 策略文件可能有语法错误 / ImportError / 运行期异常,记录完整 traceback
# 而非静默吞掉,否则用户只看到"加载失败"却不知根因(审计 #6)。
logger.exception("策略文件加载失败: %s", file_path)
return None return None
for attr_name in dir(module): for attr_name in dir(module):
+31 -47
View File
@@ -22,6 +22,7 @@ from ._df import (
_merge_txn_datetime, _merge_txn_datetime,
_to_df, _to_df,
) )
from ._reconnect import _RETRY_DELAYS, AsyncHeartbeatMixin
from .codec.block import parse_block_dat from .codec.block import parse_block_dat
from .codec.financial import parse_financial_dat, parse_financial_file_list from .codec.financial import parse_financial_dat, parse_financial_file_list
from .codec.industry import parse_tdxhy_cfg from .codec.industry import parse_tdxhy_cfg
@@ -60,7 +61,6 @@ from .models.timeseries import TransactionRecord
from .transport.async_ import AsyncTdxConnection from .transport.async_ import AsyncTdxConnection
from .transport.sync import TdxConnection, ping_all from .transport.sync import TdxConnection, ping_all
_RETRY_DELAYS = (0.1, 0.5, 1.0, 2.0)
_T = TypeVar("_T") _T = TypeVar("_T")
_SHANGHAI_TZ = ZoneInfo("Asia/Shanghai") _SHANGHAI_TZ = ZoneInfo("Asia/Shanghai")
_DAILY_PLUS = frozenset( _DAILY_PLUS = frozenset(
@@ -176,7 +176,11 @@ def _load_cache() -> list[SecurityInfo] | None:
try: try:
raw = json.loads(path.read_text("utf-8")) raw = json.loads(path.read_text("utf-8"))
updated = datetime.fromisoformat(raw["updated"]) updated = datetime.fromisoformat(raw["updated"])
if (datetime.now() - updated).total_seconds() > _CACHE_MAX_AGE: # 统一用 aware datetime 比较(审计 #18):旧缓存可能写的是 naive datetime
# 此处 localize 到上海时区,避免跨时区机器(如 CI 的 UTC 与本地 +8)误判过期。
if updated.tzinfo is None:
updated = updated.replace(tzinfo=_SHANGHAI_TZ)
if (datetime.now(_SHANGHAI_TZ) - updated).total_seconds() > _CACHE_MAX_AGE:
return None return None
return _deserialize_stocks(raw["data"]) return _deserialize_stocks(raw["data"])
except Exception: except Exception:
@@ -186,7 +190,7 @@ def _load_cache() -> list[SecurityInfo] | None:
def _save_cache(stocks: list[SecurityInfo]) -> None: def _save_cache(stocks: list[SecurityInfo]) -> None:
_CACHE_DIR.mkdir(parents=True, exist_ok=True) _CACHE_DIR.mkdir(parents=True, exist_ok=True)
data = { data = {
"updated": datetime.now().isoformat(), "updated": datetime.now(_SHANGHAI_TZ).isoformat(),
"count": len(stocks), "count": len(stocks),
"data": _serialize_stocks(stocks), "data": _serialize_stocks(stocks),
} }
@@ -781,12 +785,16 @@ class TdxClient:
results: list[HistoricalFundFlow] = [] results: list[HistoricalFundFlow] = []
for bar in bars: for bar in bars:
date = _date_from_bar(bar) date = _date_from_bar(bar)
records = self._collect_transaction_records(
lambda page_start, page_size: self._execute( # 用闭包工厂立即绑定 date(审计 #10),避免 lambda 延迟绑定循环变量。
GetHistoryTransactionDataCmd(market, code, date, page_start, page_size) def _fetch_page(
), page_start: int, page_size: int, _d: int = date
800, ) -> list[TransactionRecord]:
) return self._execute(
GetHistoryTransactionDataCmd(market, code, _d, page_start, page_size)
)
records = self._collect_transaction_records(_fetch_page, 800)
results.append(_historical_fund_flow_from_records(date, records)) results.append(_historical_fund_flow_from_records(date, records))
return _to_df(results) return _to_df(results)
@@ -796,7 +804,7 @@ class TdxClient:
# ============================================================ # ============================================================
class AsyncTdxClient: class AsyncTdxClient(AsyncHeartbeatMixin):
"""异步通达信行情客户端(asyncio)。 """异步通达信行情客户端(asyncio)。
使用示例:: 使用示例::
@@ -883,37 +891,9 @@ class AsyncTdxClient:
) -> None: ) -> None:
await self.close() await self.close()
def _start_heartbeat(self) -> None: def _heartbeat_cmd(self) -> Awaitable[object]:
"""启动后台心跳任务""" """心跳使用的轻量请求(get_security_count,复用 _execute 重连)"""
if self._heartbeat_interval <= 0: return self.get_security_count(Market.SH)
return
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
async def _stop_heartbeat(self) -> None:
"""停止并清理心跳任务。"""
if self._heartbeat_task:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
self._heartbeat_task = None
async def _heartbeat_loop(self) -> None:
"""心跳循环:定期发送轻量级请求保活。"""
while True:
try:
await asyncio.sleep(self._heartbeat_interval)
# 使用 get_security_count 作为心跳包
await self.get_security_count(Market.SH)
except asyncio.CancelledError:
break
except Exception:
# 心跳失败通常意味着连接已断开
# 下一次正常的业务请求或下一次心跳会通过 _execute 触发重连
pass
async def _execute(self, cmd: "BaseCommand[_T]") -> _T: async def _execute(self, cmd: "BaseCommand[_T]") -> _T:
"""执行命令;断线时指数退避重试。""" """执行命令;断线时指数退避重试。"""
@@ -1323,11 +1303,15 @@ class AsyncTdxClient:
results: list[HistoricalFundFlow] = [] results: list[HistoricalFundFlow] = []
for bar in bars: for bar in bars:
date = _date_from_bar(bar) date = _date_from_bar(bar)
records = await self._collect_transaction_records(
lambda page_start, page_size: self._execute( # 用闭包工厂立即绑定 date(审计 #10),避免 lambda 延迟绑定循环变量。
GetHistoryTransactionDataCmd(market, code, date, page_start, page_size) async def _fetch_page(
), page_start: int, page_size: int, _d: int = date
800, ) -> list[TransactionRecord]:
) return await self._execute(
GetHistoryTransactionDataCmd(market, code, _d, page_start, page_size)
)
records = await self._collect_transaction_records(_fetch_page, 800)
results.append(_historical_fund_flow_from_records(date, records)) results.append(_historical_fund_flow_from_records(date, records))
return _to_df(results) return _to_df(results)
+1 -1
View File
@@ -42,7 +42,7 @@ _ORGID_MAP: dict[str, str] = {}
def _ts_to_date(ts: Any) -> str: def _ts_to_date(ts: Any) -> str:
"""巨潮 ``announcementTime`` 返回 Unix 毫秒整数,转 ``YYYY-MM-DD``。""" """巨潮 ``announcementTime`` 返回 Unix 毫秒整数,转 ``YYYY-MM-DD``。"""
if isinstance(ts, (int, float)): if isinstance(ts, int | float):
return datetime.fromtimestamp(ts / 1000).strftime("%Y-%m-%d") return datetime.fromtimestamp(ts / 1000).strftime("%Y-%m-%d")
return str(ts)[:10] if ts else "" return str(ts)[:10] if ts else ""
+5 -1
View File
@@ -28,10 +28,14 @@ import os
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Any, cast from typing import Any, cast
from zoneinfo import ZoneInfo
_CONFIG_DIR = Path(os.environ.get("EASY_TDX_CONFIG_DIR", str(Path.home() / ".easy_tdx"))) _CONFIG_DIR = Path(os.environ.get("EASY_TDX_CONFIG_DIR", str(Path.home() / ".easy_tdx")))
_CONFIG_FILE = _CONFIG_DIR / "config.json" _CONFIG_FILE = _CONFIG_DIR / "config.json"
# 业务时间统一用上海时区(与 client.py 一致),避免 naive datetime 跨时区歧义(审计 #18)。
_SHANGHAI_TZ = ZoneInfo("Asia/Shanghai")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 源码内嵌默认值(config.json 不存在或字段缺失时的兜底) # 源码内嵌默认值(config.json 不存在或字段缺失时的兜底)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -242,7 +246,7 @@ def save_best_host(host: str) -> None:
"""保存最佳主机到配置文件;首次写入时同时补全默认配置。""" """保存最佳主机到配置文件;首次写入时同时补全默认配置。"""
cfg = _load() cfg = _load()
cfg["best_host"] = host cfg["best_host"] = host
cfg["best_host_updated_at"] = datetime.now().isoformat() cfg["best_host_updated_at"] = datetime.now(_SHANGHAI_TZ).isoformat()
if "known_hosts" not in cfg: if "known_hosts" not in cfg:
cfg["known_hosts"] = list(_FALLBACK_HOSTS) cfg["known_hosts"] = list(_FALLBACK_HOSTS)
if "calc_hosts" not in cfg: if "calc_hosts" not in cfg:
+31 -34
View File
@@ -2,11 +2,14 @@
import asyncio import asyncio
import logging import logging
import time
from collections import OrderedDict from collections import OrderedDict
from collections.abc import Awaitable
from types import TracebackType from types import TracebackType
from typing import TypeVar from typing import TypeVar
from .._df import _apply_bar_time_align_bars, _category_to_minutes from .._df import _apply_bar_time_align_bars, _category_to_minutes
from .._reconnect import _RETRY_DELAYS, AsyncHeartbeatMixin
from ..commands.base import BaseCommand from ..commands.base import BaseCommand
from ..config import get_best_ex_host, get_ex_hosts, save_best_ex_host from ..config import get_best_ex_host, get_ex_hosts, save_best_ex_host
from ..exceptions import TdxConnectionError from ..exceptions import TdxConnectionError
@@ -118,15 +121,23 @@ class ExTdxClient:
self.close() self.close()
def _execute(self, cmd: "BaseCommand[_T]") -> _T: def _execute(self, cmd: "BaseCommand[_T]") -> _T:
"""执行命令;断线时指数退避重试(4 次,与 A 股/MAC 统一,审计 #2)。"""
try: try:
return self._conn.execute(cmd) return self._conn.execute(cmd)
except TdxConnectionError: except TdxConnectionError:
if not self._auto_reconnect: if not self._auto_reconnect:
raise raise
self._conn.close() last_exc: TdxConnectionError | None = None
self._conn = ExTdxConnection(self._host, self._port, self._timeout) for delay in _RETRY_DELAYS:
self._conn.connect() time.sleep(delay)
return self._conn.execute(cmd) self._conn.close()
self._conn = ExTdxConnection(self._host, self._port, self._timeout)
self._conn.connect()
try:
return self._conn.execute(cmd)
except TdxConnectionError as e:
last_exc = e
raise last_exc # type: ignore[misc]
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 市场信息 # 市场信息
@@ -260,7 +271,7 @@ class ExTdxClient:
# ============================================================ # ============================================================
class AsyncExTdxClient: class AsyncExTdxClient(AsyncHeartbeatMixin):
"""异步扩展行情客户端(asyncio,端口 7727)。 """异步扩展行情客户端(asyncio,端口 7727)。
使用示例:: 使用示例::
@@ -335,43 +346,29 @@ class AsyncExTdxClient:
) -> None: ) -> None:
await self.close() await self.close()
def _start_heartbeat(self) -> None: def _heartbeat_cmd(self) -> Awaitable[object]:
if self._heartbeat_interval <= 0: """心跳使用的轻量请求(get_instrument_count,复用 _execute 重连)。"""
return return self.get_instrument_count()
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
async def _stop_heartbeat(self) -> None:
if self._heartbeat_task:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
self._heartbeat_task = None
async def _heartbeat_loop(self) -> None:
while True:
try:
await asyncio.sleep(self._heartbeat_interval)
await self.get_instrument_count()
except asyncio.CancelledError:
break
except Exception:
pass
async def _execute(self, cmd: "BaseCommand[_T]") -> _T: async def _execute(self, cmd: "BaseCommand[_T]") -> _T:
"""执行命令;断线时指数退避重试(4 次,与 A 股/MAC 统一,审计 #2)。"""
async with self._execute_lock: async with self._execute_lock:
try: try:
return await self._conn.execute(cmd) return await self._conn.execute(cmd)
except TdxConnectionError: except TdxConnectionError:
if not self._auto_reconnect: if not self._auto_reconnect:
raise raise
await self._conn.close() last_exc: TdxConnectionError | None = None
self._conn = AsyncExTdxConnection(self._host, self._port, self._timeout) for delay in _RETRY_DELAYS:
await self._conn.connect() await asyncio.sleep(delay)
return await self._conn.execute(cmd) await self._conn.close()
self._conn = AsyncExTdxConnection(self._host, self._port, self._timeout)
await self._conn.connect()
try:
return await self._conn.execute(cmd)
except TdxConnectionError as e:
last_exc = e
raise last_exc # type: ignore[misc]
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 市场信息 # 市场信息
+52 -38
View File
@@ -5,6 +5,9 @@
""" """
import asyncio import asyncio
import logging
import time
from collections.abc import Awaitable
from datetime import date from datetime import date
from types import TracebackType from types import TracebackType
from typing import Any, TypeVar from typing import Any, TypeVar
@@ -12,6 +15,7 @@ from typing import Any, TypeVar
import pandas as pd import pandas as pd
from .._df import _to_df from .._df import _to_df
from .._reconnect import _RETRY_DELAYS, AsyncHeartbeatMixin
from ..commands.base import BaseCommand from ..commands.base import BaseCommand
from ..config import get_best_mac_ex_host, get_mac_ex_hosts, save_best_mac_ex_host from ..config import get_best_mac_ex_host, get_mac_ex_hosts, save_best_mac_ex_host
from ..exceptions import TdxConnectionError from ..exceptions import TdxConnectionError
@@ -31,6 +35,8 @@ from .transport.sync import ExTdxConnection, ping_ex_all
_DEFAULT_PORT = 7727 _DEFAULT_PORT = 7727
_T = TypeVar("_T") _T = TypeVar("_T")
logger = logging.getLogger(__name__)
def _quotes_to_df(result: list[MacQuoteField]) -> pd.DataFrame: def _quotes_to_df(result: list[MacQuoteField]) -> pd.DataFrame:
"""将 MacQuoteField 列表展开为 DataFrame。""" """将 MacQuoteField 列表展开为 DataFrame。"""
@@ -135,16 +141,32 @@ class MacExClient:
self._conn.execute(MacExLoginCmd()) self._conn.execute(MacExLoginCmd())
def _execute(self, cmd: "BaseCommand[_T]") -> _T: def _execute(self, cmd: "BaseCommand[_T]") -> _T:
"""执行命令;断线时指数退避重试(4 次,与 A 股/MAC 统一,审计 #2)。
每次重连后必须重新 ``_login()``(MAC 协议扩展行情特有)。登录握手期的
``TdxConnectionError`` 与业务请求一样计入退避重试;``TdxCommandError``
(登录被拒等确定性失败)不重试,直接抛出。
"""
try: try:
return self._conn.execute(cmd) return self._conn.execute(cmd)
except TdxConnectionError: except TdxConnectionError:
if not self._auto_reconnect: if not self._auto_reconnect:
raise raise
self._conn.close() last_exc: TdxConnectionError | None = None
self._conn = ExTdxConnection(self._host, self._port, self._timeout, mac_ex_mode=True) for delay in _RETRY_DELAYS:
self._conn.connect() time.sleep(delay)
self._login() self._conn.close()
return self._conn.execute(cmd) self._conn = ExTdxConnection(
self._host, self._port, self._timeout, mac_ex_mode=True
)
# connect + login 纳入重试:登录握手期连接再次断开属可重试语义。
try:
self._conn.connect()
self._login()
return self._conn.execute(cmd)
except TdxConnectionError as e:
last_exc = e
raise last_exc # type: ignore[misc]
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 商品列表 # 商品列表
@@ -419,7 +441,7 @@ class MacExClient:
# ============================================================ # ============================================================
class AsyncMacExClient: class AsyncMacExClient(AsyncHeartbeatMixin):
"""异步 MAC 协议扩展市场客户端(asyncio,端口 7727)。 """异步 MAC 协议扩展市场客户端(asyncio,端口 7727)。
使用示例:: 使用示例::
@@ -494,50 +516,42 @@ class AsyncMacExClient:
) -> None: ) -> None:
await self.close() await self.close()
def _start_heartbeat(self) -> None: def _heartbeat_cmd(self) -> Awaitable[object]:
if self._heartbeat_interval <= 0: """心跳使用的轻量请求(get_instrument_count,复用 _execute 重连)。"""
return return self._execute(GetExInstrumentCountCmd())
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
async def _stop_heartbeat(self) -> None:
if self._heartbeat_task:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
self._heartbeat_task = None
async def _heartbeat_loop(self) -> None:
while True:
try:
await asyncio.sleep(self._heartbeat_interval)
await self._execute(GetExInstrumentCountCmd())
except asyncio.CancelledError:
break
except Exception:
pass
async def _login(self) -> None: async def _login(self) -> None:
"""执行 MAC EX 登录命令。""" """执行 MAC EX 登录命令。"""
await self._conn.execute(MacExLoginCmd()) await self._conn.execute(MacExLoginCmd())
async def _execute(self, cmd: "BaseCommand[_T]") -> _T: async def _execute(self, cmd: "BaseCommand[_T]") -> _T:
"""执行命令;断线时指数退避重试(4 次,与 A 股/MAC 统一,审计 #2)。
每次重连后必须重新 ``_login()``(MAC 协议扩展行情特有)。登录握手期的
``TdxConnectionError`` 与业务请求一样计入退避重试;``TdxCommandError``
(登录被拒等确定性失败)不重试,直接抛出。
"""
async with self._execute_lock: async with self._execute_lock:
try: try:
return await self._conn.execute(cmd) return await self._conn.execute(cmd)
except TdxConnectionError: except TdxConnectionError:
if not self._auto_reconnect: if not self._auto_reconnect:
raise raise
await self._conn.close() last_exc: TdxConnectionError | None = None
self._conn = AsyncExTdxConnection( for delay in _RETRY_DELAYS:
self._host, self._port, self._timeout, mac_ex_mode=True await asyncio.sleep(delay)
) await self._conn.close()
await self._conn.connect() self._conn = AsyncExTdxConnection(
await self._login() self._host, self._port, self._timeout, mac_ex_mode=True
return await self._conn.execute(cmd) )
# connect + login 纳入重试:登录握手期连接再次断开属可重试语义。
try:
await self._conn.connect()
await self._login()
return await self._conn.execute(cmd)
except TdxConnectionError as e:
last_exc = e
raise last_exc # type: ignore[misc]
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 商品列表 # 商品列表
+1 -1
View File
@@ -24,7 +24,7 @@ def _datetime_to_int(dt_val: object) -> int:
if hasattr(dt_val, "strftime"): if hasattr(dt_val, "strftime"):
strftime = getattr(dt_val, "strftime") strftime = getattr(dt_val, "strftime")
return int(strftime("%Y%m%d")) return int(strftime("%Y%m%d"))
if isinstance(dt_val, (int, float)): if isinstance(dt_val, int | float):
return int(dt_val) return int(dt_val)
return 0 return 0
+14 -29
View File
@@ -5,6 +5,7 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import time import time
from collections.abc import Awaitable
from dataclasses import asdict from dataclasses import asdict
from types import TracebackType from types import TracebackType
from typing import Any, TypeVar from typing import Any, TypeVar
@@ -12,6 +13,7 @@ from typing import Any, TypeVar
import pandas as pd import pandas as pd
from .._df import _apply_bar_time_align_df, _period_to_minutes, _to_df from .._df import _apply_bar_time_align_df, _period_to_minutes, _to_df
from .._reconnect import _RETRY_DELAYS, AsyncHeartbeatMixin
from ..codec.bitmap import Fields, PresetField from ..codec.bitmap import Fields, PresetField
from ..commands.base import BaseCommand from ..commands.base import BaseCommand
from ..config import get_best_host, get_mac_hosts, get_port, get_timeout, save_best_host from ..config import get_best_host, get_mac_hosts, get_port, get_timeout, save_best_host
@@ -45,7 +47,6 @@ from .models import (
MacTickChart, MacTickChart,
) )
_RETRY_DELAYS = (0.1, 0.5, 1.0, 2.0)
_KLINE_PAGE_SIZE = 700 _KLINE_PAGE_SIZE = 700
_BOARD_MEMBERS_PAGE_SIZE = 80 _BOARD_MEMBERS_PAGE_SIZE = 80
@@ -1042,7 +1043,7 @@ class MacClient:
# ============================================================ # ============================================================
class AsyncMacClient: class AsyncMacClient(AsyncHeartbeatMixin):
"""异步 MAC 协议客户端(asyncio)。 """异步 MAC 协议客户端(asyncio)。
使用示例:: 使用示例::
@@ -1150,34 +1151,12 @@ class AsyncMacClient:
await self.close() await self.close()
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 心跳 # 心跳(三件套由 AsyncHeartbeatMixin 提供,审计复审 L1
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
def _start_heartbeat(self) -> None: def _heartbeat_cmd(self) -> Awaitable[object]:
if self._heartbeat_interval <= 0: """心跳使用的轻量请求(KlineOffset,复用 _execute 重连)。"""
return return self._execute(KlineOffsetCmd(0, 1))
if self._heartbeat_task is not None:
self._heartbeat_task.cancel()
self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
async def _stop_heartbeat(self) -> None:
if self._heartbeat_task:
self._heartbeat_task.cancel()
try:
await self._heartbeat_task
except asyncio.CancelledError:
pass
self._heartbeat_task = None
async def _heartbeat_loop(self) -> None:
while True:
try:
await asyncio.sleep(self._heartbeat_interval)
await self._execute(KlineOffsetCmd(0, 1))
except asyncio.CancelledError:
break
except Exception:
pass
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
# 内部执行 # 内部执行
@@ -1568,9 +1547,15 @@ class AsyncMacClient:
) -> pd.DataFrame: ) -> pd.DataFrame:
"""获取板块涨跌幅排行榜(含成交额、成交量、资金流入流出、涨跌家数)。 """获取板块涨跌幅排行榜(含成交额、成交量、资金流入流出、涨跌家数)。
先通过 ``get_board_list`` 获取全部板块,再并发调用 先通过 ``get_board_list`` 获取全部板块,再逐个调用
``get_board_summary`` 聚合成分股数据,合并为排行榜 DataFrame。 ``get_board_summary`` 聚合成分股数据,合并为排行榜 DataFrame。
.. note::
实现中使用了 ``asyncio.gather``,但单 TCP 连接不支持并发请求——
每个 ``_fetch_row`` 内部调用 ``_execute`` 时都会持有 ``_execute_lock``
因此 gather 实际是**串行**执行的,仅作代码组织用途,无并发加速收益。
如需真正并发拉取多个板块,需引入连接池(多 ``AsyncTdxConnection``)。
Args: Args:
board_type: 板块类型(``BoardType.HY`` 行业 / ``BoardType.GN`` 概念)。 board_type: 板块类型(``BoardType.HY`` 行业 / ``BoardType.GN`` 概念)。
top_n: 聚合的板块数量上限。概念板块有 300+ 个, top_n: 聚合的板块数量上限。概念板块有 300+ 个,
+6
View File
@@ -6,10 +6,12 @@ from .finance import (
FinanceInfo, FinanceInfo,
FinancialFileInfo, FinancialFileInfo,
FinancialRecord, FinancialRecord,
TdxBlock,
XdxrRecord, XdxrRecord,
) )
from .quote import SecurityQuote from .quote import SecurityQuote
from .security import SecurityInfo from .security import SecurityInfo
from .stats import FundFlow, HistoricalFundFlow, MarketStat
from .timeseries import MinuteBar, TransactionRecord from .timeseries import MinuteBar, TransactionRecord
__all__ = [ __all__ = [
@@ -26,4 +28,8 @@ __all__ = [
"CompanyInfoCategory", "CompanyInfoCategory",
"FinancialFileInfo", "FinancialFileInfo",
"FinancialRecord", "FinancialRecord",
"TdxBlock",
"MarketStat",
"FundFlow",
"HistoricalFundFlow",
] ]
+51 -2
View File
@@ -2,6 +2,8 @@
from __future__ import annotations from __future__ import annotations
import logging
import os
from pathlib import Path from pathlib import Path
from ..models.bar import SecurityBar from ..models.bar import SecurityBar
@@ -14,6 +16,8 @@ __all__ = [
"sync_daily_bars_from_security_bars", "sync_daily_bars_from_security_bars",
] ]
logger = logging.getLogger(__name__)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# encode # encode
@@ -54,10 +58,15 @@ def encode_daily_bar(
def get_last_bar_date(filepath: str | Path) -> int | None: def get_last_bar_date(filepath: str | Path) -> int | None:
"""读取 .day 文件最后一条记录的日期。 """读取 .day 文件最后一条完整记录的日期(纯读,无副作用)
若文件尾部存在不完整记录size 32 的整数倍通常由上次写入中途崩溃/
断电导致只告警并跳过损坏尾部返回最后一条完整记录的日期
不修改文件遵守 command-query separation"get" 不应写
损坏尾部的清理由 :func:`_repair_tail` 在写入路径统一完成
Returns: Returns:
YYYYMMDD 整数文件为空或太短时返回 None YYYYMMDD 整数文件为空/太短/无完整记录时返回 None
""" """
filepath = Path(filepath) filepath = Path(filepath)
if not filepath.is_file(): if not filepath.is_file():
@@ -65,6 +74,20 @@ def get_last_bar_date(filepath: str | Path) -> int | None:
size = filepath.stat().st_size size = filepath.stat().st_size
if size < _DAILY_FMT.size: if size < _DAILY_FMT.size:
return None return None
# 完整性检查:非整数倍说明尾部有半条损坏记录,跳过它读最后一条完整记录。
remainder = size % _DAILY_FMT.size
if remainder != 0:
logger.warning(
"%s 大小 %d 不是 %d 的整数倍,尾部 %d 字节为损坏记录,"
"将读取最后一条完整记录(文件未修改,写入时由 _repair_tail 清理)",
filepath,
size,
_DAILY_FMT.size,
remainder,
)
size -= remainder
if size < _DAILY_FMT.size:
return None
with filepath.open("rb") as f: with filepath.open("rb") as f:
f.seek(size - _DAILY_FMT.size) f.seek(size - _DAILY_FMT.size)
last_record = f.read(_DAILY_FMT.size) last_record = f.read(_DAILY_FMT.size)
@@ -72,6 +95,25 @@ def get_last_bar_date(filepath: str | Path) -> int | None:
return int(date_int) return int(date_int)
def _repair_tail(filepath: Path) -> None:
"""截断文件尾部的损坏记录(非整数倍 32 字节的残余)。
仅在写入路径调用保证 get_last_bar_date 这类查询函数无副作用审计 #1)。
"""
if not filepath.is_file():
return
size = filepath.stat().st_size
remainder = size % _DAILY_FMT.size
if remainder != 0:
logger.warning(
"%s 尾部 %d 字节为损坏记录,写入前截断到最后一条完整记录",
filepath,
remainder,
)
with filepath.open("r+b") as f:
f.truncate(size - remainder)
def _bar_date_int(bar: SecurityBar) -> int: def _bar_date_int(bar: SecurityBar) -> int:
return bar.year * 10000 + bar.month * 100 + bar.day return bar.year * 10000 + bar.month * 100 + bar.day
@@ -100,6 +142,9 @@ def append_daily_bars(
""" """
filepath = Path(filepath) filepath = Path(filepath)
# 写入前清理上次崩溃可能残留的尾部半条记录(审计 #1)
_repair_tail(filepath)
# 获取文件末尾日期,用于去重 # 获取文件末尾日期,用于去重
last_date = get_last_bar_date(filepath) last_date = get_last_bar_date(filepath)
@@ -114,6 +159,10 @@ def append_daily_bars(
encoded = b"".join(encode_daily_bar(b, price_coeff, vol_coeff) for b in new_bars) encoded = b"".join(encode_daily_bar(b, price_coeff, vol_coeff) for b in new_bars)
with filepath.open("ab") as f: with filepath.open("ab") as f:
f.write(encoded) f.write(encoded)
# flush + fsync 确保落盘,避免进程崩溃/断电导致文件尾部残留半条记录
# (32 字节记录的非原子追加会损坏 get_last_bar_date 的去重依据)。
f.flush()
os.fsync(f.fileno())
return len(new_bars) return len(new_bars)
+136 -22
View File
@@ -11,6 +11,7 @@
from __future__ import annotations from __future__ import annotations
import json import json
import logging
from dataclasses import dataclass from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
@@ -23,6 +24,13 @@ from easy_tdx.backtest.strategy import Strategy
from easy_tdx.offline.daily_bar import _detect_security_type, read_daily_bars from easy_tdx.offline.daily_bar import _detect_security_type, read_daily_bars
from easy_tdx.offline.paths import resolve_vipdoc from easy_tdx.offline.paths import resolve_vipdoc
logger = logging.getLogger(__name__)
# 单股扫描失败(损坏的 .day、策略抛错等)属于"跳过该股、继续扫"的容错语义,
# 但完全静默会让系统性失败(如断网后所有文件读取异常)被吞掉(审计 #6 / 复审 L2)。
# 因此:每次失败记录 warning,扫描结束后若失败率超过阈值则记录 summary。
_SCAN_FAILURE_RATE_THRESHOLD = 0.5
# A 股类型白名单 # A 股类型白名单
_A_STOCK_TYPES = frozenset( _A_STOCK_TYPES = frozenset(
{ {
@@ -135,6 +143,9 @@ class SignalScanner:
cache = self._load_cache() cache = self._load_cache()
results: list[ScanResult] = [] results: list[ScanResult] = []
updated_cache: dict[str, Any] = {} updated_cache: dict[str, Any] = {}
# 单股扫描失败计数(审计 #6 / 复审 L2):系统性失败(如大量损坏 .day)
# 不应被完全静默,循环结束后按失败率发出汇总告警。
failures = 0
for idx, (filepath, market, code) in enumerate(files): for idx, (filepath, market, code) in enumerate(files):
if progress_callback: if progress_callback:
@@ -182,8 +193,14 @@ class SignalScanner:
), ),
} }
except Exception: except Exception:
# 单股失败属"跳过继续"容错语义,但记录 warning 以暴露系统性
# 失败(如损坏 .day / 策略 bug),不再完全静默(审计 #6 / 复审 L2)。
failures += 1
logger.warning("扫描 %s (%s) 失败,已跳过", code, filepath.name, exc_info=True)
continue continue
self._scan_failure_summary(failures, total)
self._save_cache(updated_cache) self._save_cache(updated_cache)
if progress_callback: if progress_callback:
@@ -198,46 +215,137 @@ class SignalScanner:
workers: int, workers: int,
progress_callback: Any, progress_callback: Any,
) -> list[ScanResult]: ) -> list[ScanResult]:
"""并发扫描(ProcessPoolExecutor)。""" """并发扫描(ProcessPoolExecutor)。
与串行路径一致地接入 mtime 增量缓存审计 #15):派发任务前先按 mtime
跳过未变文件复用缓存结果仅对变化的文件派发到进程池子进程结果
返回后由主进程统一写缓存避免每次 --workers 全量重算 5000
"""
import concurrent.futures import concurrent.futures
# 策略类不可跨进程 pickle(动态 importlib 加载的类子进程无法解析), # 策略类不可跨进程 pickle(动态 importlib 加载的类子进程无法解析),
# 改为传递策略文件路径,子进程自行加载 # 改为传递策略文件路径,子进程自行加载
strategy_file = _get_strategy_file(self._strategy_cls) strategy_file = _get_strategy_file(self._strategy_cls)
# 构建参数:每个任务需要的独立数据(全部为可 pickle 的基础类型) cache = self._load_cache()
tasks = [ updated_cache: dict[str, Any] = {}
(str(filepath), market, code, strategy_file, self._cash, self._commission)
for filepath, market, code in files
]
results: list[ScanResult] = [] results: list[ScanResult] = []
completed = 0 completed = 0
# 单股扫描失败计数(审计 #6 / 复审 L2):与串行路径一致。
failures = 0
with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor: # 第一遍:命中缓存的文件直接复用,未命中的收集为待扫描任务
future_to_idx = { pending: list[tuple[int, tuple[Path, str, str], float]] = []
executor.submit(_scan_one_file, *task): i for i, task in enumerate(tasks) for idx, (filepath, market, code) in enumerate(files):
} cache_key = str(filepath)
try:
for future in concurrent.futures.as_completed(future_to_idx): mtime = filepath.stat().st_mtime
except OSError:
completed += 1 completed += 1
idx = future_to_idx[future]
if progress_callback: if progress_callback:
progress_callback(completed, total, files[idx][0].name) progress_callback(completed, total, filepath.name)
continue
try: cached = cache.get(cache_key)
result = future.result() if cached is not None and cached.get("mtime") == mtime:
if result is not None: result_data = cached.get("result")
results.append(result) if result_data is not None:
except Exception: results.append(
continue ScanResult(
code=result_data["code"],
market=result_data["market"],
signal_date=result_data["signal_date"],
last_close=result_data["last_close"],
)
)
updated_cache[cache_key] = cached
completed += 1
if progress_callback:
progress_callback(completed, total, filepath.name)
else:
pending.append((idx, (filepath, market, code), mtime))
# 第二遍:仅对变化的文件派发到进程池
tasks = [
(str(filepath), market, code, strategy_file, self._cash, self._commission)
for _, (filepath, market, code), _ in pending
]
if tasks:
with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor:
future_to_pending = {
executor.submit(_scan_one_file, *task): pend
for task, pend in zip(tasks, pending)
}
for future in concurrent.futures.as_completed(future_to_pending):
idx, (filepath, market, code), mtime = future_to_pending[future]
completed += 1
if progress_callback:
progress_callback(completed, total, filepath.name)
try:
result = future.result()
if result is not None:
results.append(result)
# 子进程结果回写缓存(主进程统一写,保证一致性)
updated_cache[str(filepath)] = {
"mtime": mtime,
"result": (
{
"code": result.code,
"market": result.market,
"signal_date": result.signal_date,
"last_close": result.last_close,
}
if result is not None
else None
),
}
except Exception:
# 子进程异常(损坏 .day / 策略 bug)不再完全静默,
# 记录 warning 并计数,循环结束后按失败率汇总告警
# (审计 #6 / 复审 L2)。
failures += 1
logger.warning(
"扫描 %s (%s) 失败,已跳过",
code,
filepath.name,
exc_info=True,
)
continue
self._scan_failure_summary(failures, total)
self._save_cache(updated_cache)
if progress_callback: if progress_callback:
progress_callback(total, total, "done") progress_callback(total, total, "done")
return results return results
@staticmethod
def _scan_failure_summary(failures: int, total: int) -> None:
"""扫描结束后按失败率发出汇总告警(审计 #6 / 复审 L2)。
单股失败本身是"跳过继续"的容错语义不中断整批扫描但当失败比例
超过阈值如一半文件损坏/读取异常几乎可以肯定是系统性问题
目录配置错误磁盘故障策略 bug此时发出一条醒目的 warning
避免用户得到一份"空结果"却以为"没有信号"
"""
if failures <= 0 or total <= 0:
return
rate = failures / total
if rate >= _SCAN_FAILURE_RATE_THRESHOLD:
logger.warning(
"扫描完成但失败率过高:%d/%d (%.0f%%) 的文件扫描失败,"
"请检查 .day 文件完整性或策略实现",
failures,
total,
rate * 100,
)
def _collect_files(self, universe: str) -> list[tuple[Path, str, str]]: def _collect_files(self, universe: str) -> list[tuple[Path, str, str]]:
"""收集需要扫描的 .day 文件列表。 """收集需要扫描的 .day 文件列表。
@@ -367,6 +475,9 @@ class SignalScanner:
commission=self._commission, commission=self._commission,
) )
except Exception: except Exception:
# 单股策略计算失败视为"无信号"并跳过;debug 记录以便排查策略 bug,
# 不升级为 warning 以免在 5000 只批量扫描时刷屏(审计 #6 / 复审 L2)。
logger.debug("策略计算异常 %s,视为无信号", code, exc_info=True)
return None return None
# 检查最后一根 bar 是否有买入信号 # 检查最后一根 bar 是否有买入信号
@@ -495,6 +606,9 @@ def _scan_one_file(
commission=commission, commission=commission,
) )
except Exception: except Exception:
# 单股策略计算失败视为"无信号"并跳过;debug 记录以便排查策略 bug
# (审计 #6 / 复审 L2)。
logger.debug("策略计算异常 %s,视为无信号", code, exc_info=True)
return None return None
if not factor_signals.buy_mask[-1]: if not factor_signals.buy_mask[-1]:
+18
View File
@@ -250,6 +250,15 @@ class UnifiedTdxClient:
start: int = 0, start: int = 0,
count: int = 600, count: int = 600,
) -> pd.DataFrame: ) -> pd.DataFrame:
""".. deprecated:: 1.16.2
:meth:`goods_list` 完全相同历史遗留的重复代理方法
请改用 :meth:`goods_list`与底层 ``MacExClient`` 命名一致
"""
import warnings
warnings.warn(
"get_goods_list 是重复方法,请改用 goods_list", DeprecationWarning, stacklevel=2
)
return self._ensure_mac_ex().goods_list(market, start, count) return self._ensure_mac_ex().goods_list(market, start, count)
# ------------------------------------------------------------------ # # ------------------------------------------------------------------ #
@@ -559,6 +568,15 @@ class AsyncUnifiedTdxClient:
start: int = 0, start: int = 0,
count: int = 600, count: int = 600,
) -> pd.DataFrame: ) -> pd.DataFrame:
""".. deprecated:: 1.16.2
:meth:`goods_list` 完全相同历史遗留的重复代理方法
请改用 :meth:`goods_list`与底层 ``MacExClient`` 命名一致
"""
import warnings
warnings.warn(
"get_goods_list 是重复方法,请改用 goods_list", DeprecationWarning, stacklevel=2
)
ex = await self._ensure_mac_ex() ex = await self._ensure_mac_ex()
return await ex.goods_list(market, start, count) return await ex.goods_list(market, start, count)
+37 -3
View File
@@ -435,13 +435,19 @@ def test_stop_loss_triggers_sell():
def test_take_profit_triggers_sell(): def test_take_profit_triggers_sell():
"""Test take-profit triggers auto SELL when price rises above target.""" """Test take-profit triggers auto SELL when price rises above target.
注意审计 #4):止盈信号延迟到下一根开盘成交(消除前视偏差)。
当下一根开盘价低于触发价时跳空回落SELL 取更不利的实际开盘价
"""
df = _make_flat_df(n=30) df = _make_flat_df(n=30)
# Bar 12 rises above take_profit=110.0 # Bar 12 rises above take_profit=110.0
df.loc[12, "high"] = 112.0 df.loc[12, "high"] = 112.0
df.loc[12, "low"] = 108.0 df.loc[12, "low"] = 108.0
df.loc[12, "close"] = 111.0 df.loc[12, "close"] = 111.0
df.loc[12, "open"] = 109.0 df.loc[12, "open"] = 109.0
# Bar 13 开盘回落到 100(跳空),止盈延迟成交应取更不利的 100 而非触发价 110
df.loc[13, "open"] = 100.0
engine = BacktestEngine(TakeProfitStrategy, cash=100000) engine = BacktestEngine(TakeProfitStrategy, cash=100000)
result = engine.run(df) result = engine.run(df)
@@ -451,8 +457,36 @@ def test_take_profit_triggers_sell():
# Should have at least one SELL triggered by take-profit # Should have at least one SELL triggered by take-profit
assert len(sell_trades) >= 1, "Expected take-profit sell" assert len(sell_trades) >= 1, "Expected take-profit sell"
# Sell price should be at take_profit price (110.0) # 延迟到下一根(bar 13)开盘成交,跳空回落取更不利的实际价 100(非触发价 110)
assert sell_trades.iloc[0]["price"] == 110.0 assert sell_trades.iloc[0]["price"] == 100.0
def test_stop_loss_gap_down_fills_at_worse_price():
"""SL 信号延迟到下一根开盘成交;若跳空下跌,取更不利的开盘价(审计 #4)。
构造当根触及止损但下一根开盘远低于止损价的跳空场景
断言实际成交价取更不利的开盘价回测净值低于"触发价成交"基线
"""
df = _make_flat_df(n=30)
# Bar 12 触及 stop_loss=95low=93
df.loc[12, "low"] = 93.0
df.loc[12, "high"] = 96.0
df.loc[12, "close"] = 94.0
df.loc[12, "open"] = 97.0
# Bar 13 跳空低开到 90(远低于止损价 95),应取 90 而非 95
df.loc[13, "open"] = 90.0
df.loc[13, "low"] = 89.0
df.loc[13, "high"] = 91.0
df.loc[13, "close"] = 90.5
engine = BacktestEngine(StopLossStrategy, cash=100000)
result = engine.run(df)
trades = result.trades[~result.trades["rejected"]]
sell_trades = trades[trades["direction"] == "SELL"]
assert len(sell_trades) >= 1, "Expected stop-loss sell"
# 跳空下跌:SELL 取 min(next_open=90, trigger=95) = 90(更不利)
assert sell_trades.iloc[0]["price"] == 90.0
def test_stop_loss_not_triggered_when_price_stays_above(): def test_stop_loss_not_triggered_when_price_stays_above():
+77
View File
@@ -430,3 +430,80 @@ def test_calmar() -> None:
# 卡玛比率 = annual_return / max_drawdown # 卡玛比率 = annual_return / max_drawdown
# 由于 max_drawdown 很小,calmar 会很大 # 由于 max_drawdown 很小,calmar 会很大
assert metrics["calmar"] > 0 assert metrics["calmar"] > 0
# ---------------------------------------------------------------------------
# 除零边界回归(审计复审 N2 / 首轮 #11)
#
# performance.py 在计算日收益率时对 total[:-1]==0 的位置做了 safe_prev 守卫
# (记为 NaN 后 np.isfinite 过滤),并对 total[0]==0 的总收益率做了 0.0 兜底。
# 若有人不慎改回旧的 np.diff(total)/total[:-1],这些测试应当红灯。
# ---------------------------------------------------------------------------
def _metrics_from_total(values: list[float]) -> dict[str, float]:
"""从一组 total 值构造最小资金曲线并计算指标。"""
total = np.array(values, dtype=float)
peak = np.maximum.accumulate(total)
# 与生产回测一致:drawdown = peak - totaldrawdown_pct = drawdown / peak
drawdown = peak - total
drawdown_pct = np.divide(drawdown, peak, out=np.zeros_like(drawdown), where=(peak != 0))
equity = pd.DataFrame(
{
"datetime": np.arange(len(total)),
"total": total,
"drawdown": drawdown,
"drawdown_pct": drawdown_pct,
}
)
return PerformanceAnalyzer(equity, _make_trades()).compute()
def test_metrics_handles_zero_intermediate_equity() -> None:
"""中间净值出现 0 时,日收益率除零不抛异常、返回有限值(审计复审 N2)。
total=[100, 0, 105, 0, 110] 13 根前值为 0旧实现 diff/total[:-1]
会得到 inf进而污染均值/方差计算或触发 RuntimeWarning修复后这些位置
safe_prev 记为 NaN 并由 isfinite 过滤
"""
metrics = _metrics_from_total([100, 0, 105, 0, 110])
# 所有数值型指标必须有限(非 inf、非 NaN)
finite_keys = {
"total_return",
"annual_return",
"max_drawdown",
"sharpe",
"sortino",
"calmar",
"volatility",
"win_rate",
"profit_factor",
}
for key in finite_keys:
val = metrics[key]
assert np.isfinite(val), f"{key} 不是有限值: {val}"
def test_metrics_handles_zero_first_equity() -> None:
"""首根净值为 0 时 total_return 兜底为 0.0 而非除零(审计复审 N2)。
total[0]==0 (total[-1]/total[0]) - 1 会除零修复后直接记 0.0
"""
metrics = _metrics_from_total([0, 100, 105, 110, 115])
# total_return 走 total[0]==0 分支,应为有限值
assert np.isfinite(metrics["total_return"]), f"total_return 非有限值: {metrics['total_return']}"
# 不抛异常即说明 max_drawdown 等也未受影响
assert np.isfinite(metrics["max_drawdown"])
def test_metrics_all_zero_equity_does_not_raise() -> None:
"""全 0 资金曲线不应产生 inf/nan,也不应抛异常(审计复审 N2 极端场景)。"""
# total 全 0 → safe_prev 全 NaN → daily_ret 过滤后为空 → 走 _empty_metrics
metrics = _metrics_from_total([0, 0, 0, 0, 0])
# 全 0 资金曲线收益率数据不足,应安全返回有限值(多数为 0)
assert np.isfinite(metrics["total_return"])
assert np.isfinite(metrics["max_drawdown"])
assert np.isfinite(metrics["sharpe"])
+158
View File
@@ -0,0 +1,158 @@
"""TdxClient._execute 的指数退避重连测试(sync + async)。
之前 _execute 4 _RETRY_DELAYS 退避重连路径零测试审计报告 #9),
async transport 层的真实重连测试未覆盖 _execute 自身的退避循环
本文件 mock _conn.execute 让前 N 次抛 TdxConnectionError N+1 次成功
patch time.sleep / asyncio.sleep 验证退避序列
"""
from __future__ import annotations
import asyncio
from unittest.mock import MagicMock, patch
import pytest
from easy_tdx.client import _RETRY_DELAYS, AsyncTdxClient, TdxClient
from easy_tdx.commands.security_count import GetSecurityCountCmd
from easy_tdx.exceptions import TdxConnectionError
from easy_tdx.models.enums import Market
# --------------------------------------------------------------------------- #
# 同步 _execute 重连
# --------------------------------------------------------------------------- #
class TestSyncExecuteReconnect:
def test_reconnect_succeeds_on_second_attempt(self) -> None:
"""首次抛 TdxConnectionError,重连后第 1 次重试成功。"""
with patch("easy_tdx.client.TdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
# execute 首次抛错,重连后(第1次重试)成功
mock_conn.execute.side_effect = [
TdxConnectionError("disconnected"),
1000, # 重连后成功
]
mock_conn_cls.return_value = mock_conn
client = TdxClient("1.1.1.1", 7709, 1.0, auto_reconnect=True, heartbeat_interval=0)
with patch("easy_tdx.client.time.sleep"): # 跳过真实 sleep
result = client._execute(GetSecurityCountCmd(Market.SH))
assert result == 1000
# 应重连了 1 次(首次失败 + 1 次重试成功)
assert mock_conn.close.call_count == 1
def test_all_retries_exhausted_raises_last(self) -> None:
"""4 次重试全部失败,应抛出最后一个异常。"""
with patch("easy_tdx.client.TdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
# 首次 + 4 次重试全部失败
mock_conn.execute.side_effect = TdxConnectionError("always down")
mock_conn_cls.return_value = mock_conn
client = TdxClient("1.1.1.1", 7709, 1.0, auto_reconnect=True, heartbeat_interval=0)
with patch("easy_tdx.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetSecurityCountCmd(Market.SH))
# 应 sleep 了 4 次(_RETRY_DELAYS 长度)
assert mock_sleep.call_count == len(_RETRY_DELAYS)
def test_no_reconnect_when_disabled(self) -> None:
"""auto_reconnect=False 时首次失败立即抛出,不重试。"""
with patch("easy_tdx.client.TdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("down")
mock_conn_cls.return_value = mock_conn
client = TdxClient("1.1.1.1", 7709, 1.0, auto_reconnect=False, heartbeat_interval=0)
with patch("easy_tdx.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetSecurityCountCmd(Market.SH))
# 禁用重连时不应 sleep
mock_sleep.assert_not_called()
def test_retry_uses_exponential_backoff_delays(self) -> None:
"""验证 sleep 调用的延迟序列与 _RETRY_DELAYS 一致。"""
with patch("easy_tdx.client.TdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("down")
mock_conn_cls.return_value = mock_conn
client = TdxClient("1.1.1.1", 7709, 1.0, auto_reconnect=True, heartbeat_interval=0)
with patch("easy_tdx.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetSecurityCountCmd(Market.SH))
actual_delays = [call.args[0] for call in mock_sleep.call_args_list]
assert tuple(actual_delays) == _RETRY_DELAYS
# --------------------------------------------------------------------------- #
# 异步 _execute 重连
# --------------------------------------------------------------------------- #
class TestAsyncExecuteReconnect:
def test_async_reconnect_succeeds_on_second_attempt(self) -> None:
async def main() -> int:
with patch("easy_tdx.client.AsyncTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
call_count = [0]
async def _execute(cmd: object) -> int:
call_count[0] += 1
if call_count[0] == 1:
raise TdxConnectionError("down")
return 2000
async def _noop() -> None:
return None
mock_conn.execute = _execute
mock_conn.close = _noop
mock_conn.connect = _noop
mock_conn_cls.return_value = mock_conn
client = AsyncTdxClient(
"1.1.1.1", 7709, 1.0, auto_reconnect=True, heartbeat_interval=0
)
with patch("easy_tdx.client.asyncio.sleep", new=AsyncMockSleep()):
result = await client._execute(GetSecurityCountCmd(Market.SH))
return result
assert asyncio.run(main()) == 2000
def test_async_all_retries_exhausted(self) -> None:
async def main() -> None:
with patch("easy_tdx.client.AsyncTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
async def _execute(cmd: object) -> int:
raise TdxConnectionError("always down")
async def _noop() -> None:
return None
mock_conn.execute = _execute
mock_conn.close = _noop
mock_conn.connect = _noop
mock_conn_cls.return_value = mock_conn
client = AsyncTdxClient(
"1.1.1.1", 7709, 1.0, auto_reconnect=True, heartbeat_interval=0
)
with patch("easy_tdx.client.asyncio.sleep", new=AsyncMockSleep()) as mock_sleep:
with pytest.raises(TdxConnectionError):
await client._execute(GetSecurityCountCmd(Market.SH))
assert mock_sleep.call_count == len(_RETRY_DELAYS)
asyncio.run(main())
class AsyncMockSleep:
"""轻量 async sleep 替身,记录调用次数但不真实等待。"""
def __init__(self) -> None:
self.call_count = 0
async def __call__(self, delay: float) -> None:
self.call_count += 1
+126
View File
@@ -0,0 +1,126 @@
"""codec/bitmap.py 单元测试 —— MAC 协议字段位图编解码。
之前 bitmap.py~490 零测试审计报告 #9)。本文件覆盖:
FieldBit 字段属性PresetField 组合FieldSelection 去重
build_bitmap 20 字节输出get_active_fields 往返解析
风格参照 test_codec_frame.py纯函数式 mockstruct 构造输入
"""
from __future__ import annotations
from easy_tdx.codec.bitmap import (
FieldBit,
FieldSelection,
PresetField,
build_bitmap,
build_exclude_flags,
get_active_fields,
normalize_fields,
)
class TestFieldBit:
def test_field_name_is_lower(self) -> None:
assert FieldBit.PRE_CLOSE.field_name == "pre_close"
assert FieldBit.OPEN.field_name == "open"
def test_fmt_and_desc_attached(self) -> None:
assert FieldBit.OPEN.fmt == "<f"
assert FieldBit.OPEN.desc == "开盘价"
assert FieldBit.VOL.fmt == "<I"
def test_value_is_bit_position(self) -> None:
assert FieldBit.PRE_CLOSE == 0x00
assert FieldBit.OPEN == 0x01
class TestPresetField:
def test_ohlc_contains_four_fields(self) -> None:
names = {f.name for f in PresetField.OHLC.value}
assert names == {"OPEN", "HIGH", "LOW", "CLOSE"}
def test_chain_plus_combines(self) -> None:
combined = PresetField.OHLC + FieldBit.VOL
sel = normalize_fields(combined)
bits = {b for b in sel}
assert FieldBit.VOL in bits
assert FieldBit.OPEN in bits
def test_chain_or_combines(self) -> None:
combined = PresetField.OHLC | PresetField.VOLUME
sel = normalize_fields(combined)
bits = {b for b in sel}
assert FieldBit.VOL in bits
assert FieldBit.AMOUNT in bits
class TestFieldSelection:
def test_dedup_preserves_order(self) -> None:
sel = FieldSelection(FieldBit.OPEN, FieldBit.OPEN, FieldBit.HIGH)
bits = list(sel)
assert bits == [FieldBit.OPEN, FieldBit.HIGH]
def test_empty_selection(self) -> None:
assert list(FieldSelection()) == []
class TestBuildBitmap:
def test_single_field_sets_correct_bit(self) -> None:
# FieldBit.OPEN == 0x01bit 1 应被置位
ba = build_bitmap(FieldBit.OPEN)
assert len(ba) == 20
assert ba[0] == 0b0000_0010 # bit 1
# 控制区 4 字节默认 0
assert bytes(ba[16:20]) == b"\x00\x00\x00\x00"
def test_multiple_fields_or(self) -> None:
ba = build_bitmap(PresetField.OHLC)
assert len(ba) == 20
# OPEN(1)+HIGH(2)+LOW(3)+CLOSE(4) → bit 1,2,3,4 → 0b11110 = 30
assert ba[0] == 0b0001_1110
def test_exclude_flags_appended(self) -> None:
ba = build_bitmap(FieldBit.OPEN, exclude_flags=0x1234)
assert len(ba) == 20
assert bytes(ba[16:20]) == b"\x34\x12\x00\x00"
def test_debug_preset_all_ff(self) -> None:
ba = build_bitmap(PresetField.DEBUG)
assert ba == bytearray(b"\xff" * 20)
class TestGetActiveFields:
def test_roundtrip(self) -> None:
original = PresetField.OHLC
ba = build_bitmap(original)
active = get_active_fields(bytes(ba[:16]))
active_names = {f.name for f, _ in active}
assert active_names == {"OPEN", "HIGH", "LOW", "CLOSE"}
def test_empty_bitmap(self) -> None:
active = get_active_fields(b"\x00" * 16)
assert active == []
def test_fmt_returned(self) -> None:
ba = build_bitmap(FieldBit.VOL)
active = get_active_fields(bytes(ba[:16]))
assert len(active) == 1
field, fmt = active[0]
assert field == FieldBit.VOL
assert fmt == "<I"
def test_sorted_by_bit_position(self) -> None:
# 故意逆序传入
ba = build_bitmap([FieldBit.CLOSE, FieldBit.OPEN, FieldBit.HIGH])
active = get_active_fields(bytes(ba[:16]))
positions = [f.value for f, _ in active]
assert positions == sorted(positions)
class TestBuildExcludeFlags:
def test_zero(self) -> None:
assert build_exclude_flags(0) == b"\x00\x00\x00\x00"
def test_value(self) -> None:
assert build_exclude_flags(0xFF) == b"\xff\x00\x00\x00"
+128
View File
@@ -0,0 +1,128 @@
"""config.py 单元测试 —— 覆盖环境变量覆盖、config.json 原子读写、save_best_host 补全逻辑。
之前这三块env 覆盖 / config.json 读写 / save_best_host 合并零测试
本文件补齐该缺口审计报告 #9)。
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from easy_tdx import config as cfg
# --------------------------------------------------------------------------- #
# 辅助:把 config 模块的 _CONFIG_FILE / _CONFIG_DIR 重定向到临时目录
# --------------------------------------------------------------------------- #
@pytest.fixture
def isolated_config(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""把 config 模块的重定向到 tmp_path,测试间互不影响。"""
monkeypatch.setattr(cfg, "_CONFIG_DIR", tmp_path)
monkeypatch.setattr(cfg, "_CONFIG_FILE", tmp_path / "config.json")
return tmp_path
# --------------------------------------------------------------------------- #
# 环境变量覆盖(EASY_TDX_HOST / PORT / TIMEOUT
# --------------------------------------------------------------------------- #
class TestEnvOverride:
def test_env_host_overrides_config(
self, isolated_config: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# config.json 写入一个 host,但 env 应优先
(isolated_config / "config.json").write_text(json.dumps({"best_host": "1.1.1.1"}), "utf-8")
monkeypatch.setenv("EASY_TDX_HOST", "9.9.9.9")
assert cfg.get_best_host() == "9.9.9.9"
def test_env_port_overrides_config(
self, isolated_config: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setenv("EASY_TDX_PORT", "8888")
assert cfg.get_port() == 8888
def test_env_timeout_overrides_config(
self, isolated_config: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setenv("EASY_TDX_TIMEOUT", "42.5")
assert cfg.get_timeout() == 42.5
def test_env_known_hosts_csv(
self, isolated_config: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setenv("EASY_TDX_KNOWN_HOSTS", "a.com, b.com ,,c.com")
assert cfg.get_known_hosts() == ["a.com", "b.com", "c.com"]
# --------------------------------------------------------------------------- #
# config.json 读写 + 默认兜底
# --------------------------------------------------------------------------- #
class TestConfigReadWrite:
def test_no_config_file_uses_fallback(self, isolated_config: Path) -> None:
# 无 config.json 时返回内嵌默认值
assert cfg.get_best_host() == cfg._FALLBACK_HOSTS[0]
assert cfg.get_port() == cfg._FALLBACK_PORT
assert cfg.get_known_hosts() == list(cfg._FALLBACK_HOSTS)
def test_config_json_host(self, isolated_config: Path) -> None:
(isolated_config / "config.json").write_text(
json.dumps({"best_host": "203.0.0.1", "port": 7709, "timeout": 12.0}),
"utf-8",
)
assert cfg.get_best_host() == "203.0.0.1"
assert cfg.get_port() == 7709
assert cfg.get_timeout() == 12.0
def test_load_corrupt_json_returns_empty(self, isolated_config: Path) -> None:
# 损坏的 JSON 不应崩溃,应回退到默认
(isolated_config / "config.json").write_text("{not valid json", "utf-8")
assert cfg.get_best_host() == cfg._FALLBACK_HOSTS[0]
# --------------------------------------------------------------------------- #
# save_best_host 首次写入补全逻辑
# --------------------------------------------------------------------------- #
class TestSaveBestHost:
def test_first_write_completes_defaults(self, isolated_config: Path) -> None:
cfg.save_best_host("180.153.18.170")
data = json.loads((isolated_config / "config.json").read_text("utf-8"))
assert data["best_host"] == "180.153.18.170"
# 首次写入应补全所有默认字段
assert data["known_hosts"] == list(cfg._FALLBACK_HOSTS)
assert data["calc_hosts"] == list(cfg._FALLBACK_CALC_HOSTS)
assert data["mac_hosts"] == list(cfg._FALLBACK_MAC_HOSTS)
assert data["ex_hosts"] == list(cfg._FALLBACK_EX_HOSTS)
assert data["mac_ex_hosts"] == list(cfg._FALLBACK_MAC_EX_HOSTS)
assert data["port"] == cfg._FALLBACK_PORT
assert "best_host_updated_at" in data
def test_second_write_preserves_existing(self, isolated_config: Path) -> None:
# 预置已存在的 known_hostssave_best_host 不应覆盖它
existing = {
"known_hosts": ["custom.host"],
"port": 9999,
}
(isolated_config / "config.json").write_text(json.dumps(existing), "utf-8")
cfg.save_best_host("new.host")
data = json.loads((isolated_config / "config.json").read_text("utf-8"))
assert data["best_host"] == "new.host"
# 已有字段应保留,不被默认值覆盖
assert data["known_hosts"] == ["custom.host"]
assert data["port"] == 9999
# 但缺失的字段应补全
assert "calc_hosts" in data
def test_atomic_write(self, isolated_config: Path) -> None:
# 写入后不应残留 .tmp 文件(原子替换)
cfg.save_best_host("x.host")
assert not (isolated_config / "config.json.tmp").exists()
assert (isolated_config / "config.json").exists()
+193
View File
@@ -0,0 +1,193 @@
"""扩展行情 client 的指数退避重连测试(审计 #2)。
之前 ex 家族ExTdxClient/MacExClient/AsyncExTdxClient/AsyncMacExClient _execute
只重连 1 次无退避 A /MAC 4 次退避不一致本测试验证统一后的退避行为
并确认 MacExClient 重连后会重新 _login()
"""
from __future__ import annotations
import asyncio
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from easy_tdx._reconnect import _RETRY_DELAYS
from easy_tdx.ex.client import AsyncExTdxClient, ExTdxClient
from easy_tdx.ex.commands.get_markets import GetExMarketsCmd
from easy_tdx.ex.mac_client import AsyncMacExClient, MacExClient
from easy_tdx.exceptions import TdxConnectionError
class TestExTdxClientReconnect:
def test_reconnect_succeeds_on_second_attempt(self) -> None:
"""首次抛 TdxConnectionError,重连后第 1 次重试成功。"""
with patch("easy_tdx.ex.client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = [TdxConnectionError("down"), ["market"]]
mock_conn_cls.return_value = mock_conn
client = ExTdxClient("1.1.1.1", auto_reconnect=True)
with patch("easy_tdx.ex.client.time.sleep"):
result = client._execute(GetExMarketsCmd())
assert result == ["market"]
assert mock_conn.close.call_count == 1 # 重连了 1 次
def test_all_retries_exhausted_raises_last(self) -> None:
"""4 次重试全部失败,应抛出异常,且 sleep 4 次。"""
with patch("easy_tdx.ex.client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("always down")
mock_conn_cls.return_value = mock_conn
client = ExTdxClient("1.1.1.1", auto_reconnect=True)
with patch("easy_tdx.ex.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetExMarketsCmd())
assert mock_sleep.call_count == len(_RETRY_DELAYS)
def test_no_reconnect_when_disabled(self) -> None:
with patch("easy_tdx.ex.client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("down")
mock_conn_cls.return_value = mock_conn
client = ExTdxClient("1.1.1.1", auto_reconnect=False)
with patch("easy_tdx.ex.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetExMarketsCmd())
mock_sleep.assert_not_called()
class TestMacExClientReconnect:
def test_reconnect_relogs_in(self) -> None:
"""MacExClient 每次重连后必须重新 _login()(MAC 协议特有)。"""
with patch("easy_tdx.ex.mac_client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = [TdxConnectionError("down"), ["market"]]
mock_conn_cls.return_value = mock_conn
client = MacExClient("1.1.1.1", auto_reconnect=True)
with (
patch("easy_tdx.ex.mac_client.time.sleep"),
patch.object(client, "_login") as mock_login,
):
result = client._execute(GetExMarketsCmd())
assert result == ["market"]
# 重连 1 次应触发 1 次 _login
assert mock_login.call_count == 1
def test_all_retries_relogin_each_time(self) -> None:
"""4 次重试全失败时,每次重连都应 _login()(共 4 次)。"""
with patch("easy_tdx.ex.mac_client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("always down")
mock_conn_cls.return_value = mock_conn
client = MacExClient("1.1.1.1", auto_reconnect=True)
with (
patch("easy_tdx.ex.mac_client.time.sleep"),
patch.object(client, "_login") as mock_login,
):
with pytest.raises(TdxConnectionError):
client._execute(GetExMarketsCmd())
assert mock_login.call_count == len(_RETRY_DELAYS)
class TestAsyncExTdxClientReconnect:
def test_async_all_retries_exhausted(self) -> None:
async def main() -> None:
with patch("easy_tdx.ex.client.AsyncExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
async def _execute(cmd: object) -> list[str]:
raise TdxConnectionError("always down")
mock_conn.execute = _execute
async def _noop() -> None:
return None
mock_conn.close = _noop
mock_conn.connect = _noop
mock_conn_cls.return_value = mock_conn
client = AsyncExTdxClient("1.1.1.1", auto_reconnect=True, heartbeat_interval=0)
with patch("easy_tdx.ex.client.asyncio.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
await client._execute(GetExMarketsCmd())
assert mock_sleep.call_count == len(_RETRY_DELAYS)
asyncio.run(main())
class TestAsyncMacExClientReconnect:
def test_async_relogin_each_retry(self) -> None:
"""AsyncMacExClient 每次重连后必须重新 _login()(覆盖 async relogin 路径)。"""
async def main() -> None:
with patch("easy_tdx.ex.mac_client.AsyncExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
async def _execute(cmd: object) -> list[str]:
raise TdxConnectionError("always down")
mock_conn.execute = _execute
async def _noop() -> None:
return None
mock_conn.close = _noop
mock_conn.connect = _noop
mock_conn_cls.return_value = mock_conn
client = AsyncMacExClient("1.1.1.1", auto_reconnect=True, heartbeat_interval=0)
with (
patch("easy_tdx.ex.mac_client.asyncio.sleep"),
patch.object(client, "_login", new_callable=AsyncMock) as mock_login,
):
with pytest.raises(TdxConnectionError):
await client._execute(GetExMarketsCmd())
# 4 次重连应触发 4 次 _login
assert mock_login.call_count == len(_RETRY_DELAYS)
asyncio.run(main())
class TestBackoffDelayValues:
"""验证退避延迟值序列与 _RETRY_DELAYS 完全一致(防硬编码回归)。"""
def test_sync_ex_uses_exact_delays(self) -> None:
with patch("easy_tdx.ex.client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("down")
mock_conn_cls.return_value = mock_conn
client = ExTdxClient("1.1.1.1", auto_reconnect=True)
with patch("easy_tdx.ex.client.time.sleep") as mock_sleep:
with pytest.raises(TdxConnectionError):
client._execute(GetExMarketsCmd())
actual = tuple(c.args[0] for c in mock_sleep.call_args_list)
assert actual == _RETRY_DELAYS
class TestMacExLoginRetriedOnConnectionError:
"""登录握手期抛 TdxConnectionError 应继续重试(验证 _login 纳入 inner try)。"""
def test_login_conn_error_triggers_full_retry(self) -> None:
"""_login 抛 TdxConnectionError 时不应逃逸,应跑完 4 次重试。"""
with patch("easy_tdx.ex.mac_client.ExTdxConnection") as mock_conn_cls:
mock_conn = MagicMock()
mock_conn.execute.side_effect = TdxConnectionError("always down")
mock_conn_cls.return_value = mock_conn
client = MacExClient("1.1.1.1", auto_reconnect=True)
# _login 抛 TdxConnectionError(模拟登录握手期连接又断)
with (
patch("easy_tdx.ex.mac_client.time.sleep") as mock_sleep,
patch.object(client, "_login", side_effect=TdxConnectionError("login lost")),
):
with pytest.raises(TdxConnectionError):
client._execute(GetExMarketsCmd())
# 关键:_login 异常被纳入重试,4 次都跑了(而非第 1 次就逃逸)
assert mock_sleep.call_count == len(_RETRY_DELAYS)
+137
View File
@@ -0,0 +1,137 @@
"""公共 API 导出完整性测试 —— 防止 __all__ 与实际导出漂移(审计 #13)。
确保 easy_tdx.__all__ 中每个名字都能从顶层包成功导入
且文档中描述的模型FundFlow/MarketStat 确实可访问
复审补充L3进一步断言导出对象的**类型**避免类名被意外绑成模块
None或常量"可导入"不足以守住类型契约
"""
from __future__ import annotations
import inspect
import easy_tdx
# 期望的导出契约:每个公共名字应对应的对象类型。
# - "class" → 必须 inspect.isclassclient / 枚举 / 数据模型 / 异常)
# - "func" → 必须 callable 且非 classping_* / save_best_*
# - "constant" → 兜底(KNOWN_HOSTS / XDXR_CATEGORY_NAMES 等映射表或常量)
_EXPECTED_KIND: dict[str, str] = {
# client 类
"TdxClient": "class",
"AsyncTdxClient": "class",
"MacClient": "class",
"AsyncMacClient": "class",
"MacExClient": "class",
"AsyncMacExClient": "class",
"ExTdxClient": "class",
"AsyncExTdxClient": "class",
"UnifiedTdxClient": "class",
"AsyncUnifiedTdxClient": "class",
# 枚举
"Market": "class",
"KlineCategory": "class",
"Adjust": "class",
"BoardType": "class",
"Category": "class",
"ExMarket": "class",
"FilterType": "class",
"Period": "class",
"SortOrder": "class",
"SortType": "class",
# 数据模型
"SecurityBar": "class",
"SecurityQuote": "class",
"SecurityInfo": "class",
"MinuteBar": "class",
"TransactionRecord": "class",
"XdxrRecord": "class",
"FinanceInfo": "class",
"CompanyInfoCategory": "class",
"FinancialFileInfo": "class",
"FinancialRecord": "class",
"TdxBlock": "class",
"MarketStat": "class",
"FundFlow": "class",
"HistoricalFundFlow": "class",
# 异常
"TdxError": "class",
"TdxConnectionError": "class",
"TdxDecodeError": "class",
"TdxCommandError": "class",
# 函数
"ping_all": "func",
"ping_mac_all": "func",
"save_best_host": "func",
"save_best_ex_host": "func",
# 常量 / 映射表
"KNOWN_EX_HOSTS": "constant",
"KNOWN_HOSTS": "constant",
"CALC_HOSTS": "constant",
"MAC_HOSTS": "constant",
"XDXR_CATEGORY_NAMES": "constant",
}
def test_all_names_are_importable() -> None:
"""__all__ 里每个名字都必须能从 easy_tdx 顶层获取到非 None 对象。"""
missing = [name for name in easy_tdx.__all__ if getattr(easy_tdx, name, None) is None]
assert missing == [], f"__all__ 中以下名字无法从 easy_tdx 导入: {missing}"
def test_expected_kind_contract_is_complete() -> None:
"""_EXPECTED_KIND 必须覆盖 __all__ 的每个名字,否则契约会悄悄漂移(审计复审 L3)。"""
covered = set(_EXPECTED_KIND)
exported = set(easy_tdx.__all__)
missing_kind = exported - covered
extra_kind = covered - exported
assert not missing_kind, f"以下导出未在 _EXPECTED_KIND 中声明类型契约: {sorted(missing_kind)}"
assert not extra_kind, f"_EXPECTED_KIND 含未导出的名字(已移除?): {sorted(extra_kind)}"
def test_exported_objects_have_expected_type() -> None:
"""断言每个导出对象的类型符合契约(审计复审 L3)。
防止类名被绑成模块/None/常量"可导入"不足以守住类型
"""
wrong: list[str] = []
for name, kind in _EXPECTED_KIND.items():
obj = getattr(easy_tdx, name, None)
if obj is None:
wrong.append(f"{name}: 不应为 None")
continue
if kind == "class":
if not inspect.isclass(obj):
wrong.append(f"{name}: 期望 class,实际 {type(obj).__name__}")
elif kind == "func":
# callable 但不能是 class(避免类被当成函数)
if not callable(obj) or inspect.isclass(obj):
wrong.append(f"{name}: 期望 function,实际 {type(obj).__name__}")
# "constant" 兜底,不做严格断言
assert wrong == [], "导出对象类型契约违反: \n" + "\n".join(wrong)
def test_documented_models_exported() -> None:
"""api_reference.md 文档描述的模型必须在公共导出中(审计 #13)。"""
for name in ("FundFlow", "MarketStat", "HistoricalFundFlow", "TdxBlock"):
assert name in easy_tdx.__all__, f"{name} 应在 easy_tdx.__all__ 中"
assert inspect.isclass(getattr(easy_tdx, name)), f"{name} 应是类"
def test_core_clients_exported() -> None:
"""8 个 client 类与门面都应导出且确实是类(审计 #13 + 复审 L3)。"""
for name in (
"TdxClient",
"AsyncTdxClient",
"MacClient",
"AsyncMacClient",
"ExTdxClient",
"AsyncExTdxClient",
"MacExClient",
"AsyncMacExClient",
"UnifiedTdxClient",
"AsyncUnifiedTdxClient",
):
assert name in easy_tdx.__all__, f"{name} 应在 easy_tdx.__all__ 中"
assert inspect.isclass(getattr(easy_tdx, name)), f"{name} 应是类"
+54
View File
@@ -210,3 +210,57 @@ class TestIncrementalScan:
codes2 = sorted(r.code for r in results2) codes2 = sorted(r.code for r in results2)
# 结果可能相同 (策略没变), 但不应崩溃 # 结果可能相同 (策略没变), 但不应崩溃
assert len(codes2) >= 1 assert len(codes2) >= 1
class TestScanFailureLogging:
"""扫描失败日志回归(审计复审 L2)。
首轮 #6 将扫描循环的 ``except Exception: continue`` 评为"系统性失败被静默
吞掉"。复审 L2 修复:单股失败记录 warning + 失败计数,失败率超阈值时
循环结束发出 summary这些测试用 monkeypatch ``_scan_one`` 抛错模拟
损坏 .day / 策略异常等场景断言失败被记录read_daily_bars 本身对短文件
容错返回 0 不会抛错故用 monkeypatch 构造确定性失败
"""
def test_serial_scan_logs_per_stock_failure(
self, vipdoc: Path, caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch
) -> None:
"""单股 _scan_one 抛错时,串行扫描应记录 warning(审计复审 L2)。"""
scanner = SignalScanner(AlwaysBuyStrategy, vipdoc_path=vipdoc)
def _boom(self: SignalScanner, filepath: Path, market: str, code: str) -> None:
raise RuntimeError(f"simulated corrupt day for {code}")
monkeypatch.setattr(SignalScanner, "_scan_one", _boom)
with caplog.at_level("WARNING", logger="easy_tdx.screen.scanner"):
results = scanner.scan(universe="all", workers=0)
# 全部抛错 → 无结果,但不崩溃(容错语义:跳过继续)
assert results == []
# 每个被扫描的 A 股都应有一条 warning
warnings = [r for r in caplog.records if r.levelname == "WARNING"]
assert len(warnings) >= 1, "单股失败应触发 warning 日志"
assert any("失败" in r.getMessage() for r in warnings)
def test_serial_scan_high_failure_rate_emits_summary(
self, vipdoc: Path, caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch
) -> None:
"""失败率超阈值时应发出汇总告警(审计复审 L2)。
全部 A _scan_one 抛错失败率 100% > 50% 阈值断言扫描完成后
有一条 summary warning
"""
scanner = SignalScanner(AlwaysBuyStrategy, vipdoc_path=vipdoc)
def _boom(self: SignalScanner, filepath: Path, market: str, code: str) -> None:
raise RuntimeError(f"simulated corrupt day for {code}")
monkeypatch.setattr(SignalScanner, "_scan_one", _boom)
with caplog.at_level("WARNING", logger="easy_tdx.screen.scanner"):
scanner.scan(universe="all", workers=0)
# 应有汇总告警提到"失败率过高"
summary_msgs = [r.getMessage() for r in caplog.records if "失败率" in r.getMessage()]
assert summary_msgs, "失败率过高时应发出汇总 warning"