mirror of
https://ghfast.top/https://github.com/aeroxw/tick-stock-panel.git
synced 2026-09-12 17:54:15 +08:00
feat: 自定义数据源扩展 + 限频集中化 (v0.1.80)
This commit is contained in:
shy3130
@@ -0,0 +1,205 @@
|
||||
# 自定义数据源接入
|
||||
|
||||
本项目默认使用 TickFlow。自定义数据源是一个可选扩展: 外部 HTTP 服务负责取数和整理, 本项目只把返回结果映射成内部标准字段, 然后复用现有存储、指标、enriched、策略和前端展示逻辑。
|
||||
|
||||
## 支持范围
|
||||
|
||||
当前自定义源支持三类数据:
|
||||
|
||||
| 数据集 | 配置名 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 日K | `daily` | 批量返回一组股票在指定区间内的日K |
|
||||
| 除权因子 | `adj_factor` | 批量返回一组股票的复权因子 |
|
||||
| 实时行情 | `realtime` | 返回全市场快照,用于盘中 enriched 增量计算 |
|
||||
|
||||
分钟K、财务、深度盘口暂时仍走 TickFlow。
|
||||
|
||||
## 配置位置
|
||||
|
||||
把 YAML 放到运行数据目录下:
|
||||
|
||||
```text
|
||||
data/data_sources/*.yaml
|
||||
```
|
||||
|
||||
在桌面版中,`data/` 位于程序目录旁;在开发环境中,通常是项目根目录的 `data/`。
|
||||
|
||||
修改 YAML 后可在「设置 -> 数据源」点击「重新加载」,或调用:
|
||||
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:3018/api/settings/data-sources/reload
|
||||
```
|
||||
|
||||
## 最小 YAML
|
||||
|
||||
```yaml
|
||||
name: mock_source
|
||||
display_name: "Mock 自定义数据源"
|
||||
auth:
|
||||
type: none
|
||||
|
||||
datasets:
|
||||
daily:
|
||||
url: http://127.0.0.1:3021/daily
|
||||
method: POST
|
||||
batch: 100
|
||||
rpm: 200
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
trade_date: date
|
||||
open: open
|
||||
high: high
|
||||
low: low
|
||||
close: close
|
||||
vol: volume
|
||||
amt: amount
|
||||
transforms:
|
||||
date: "parse_date(value, '%Y-%m-%d')"
|
||||
|
||||
adj_factor:
|
||||
url: http://127.0.0.1:3021/adj_factor
|
||||
method: POST
|
||||
batch: 100
|
||||
rpm: 200
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
trade_date: trade_date
|
||||
factor: ex_factor
|
||||
transforms:
|
||||
trade_date: "parse_date(value, '%Y-%m-%d')"
|
||||
|
||||
realtime:
|
||||
url: http://127.0.0.1:3021/realtime
|
||||
method: GET
|
||||
rpm: 60
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
name: name
|
||||
last: last_price
|
||||
pre_close: prev_close
|
||||
open: open
|
||||
high: high
|
||||
low: low
|
||||
vol: volume
|
||||
amt: amount
|
||||
pct: change_pct
|
||||
amount_change: change_amount
|
||||
amplitude: amplitude
|
||||
turnover: turnover_rate
|
||||
```
|
||||
|
||||
## 字段契约
|
||||
|
||||
### daily 必填
|
||||
|
||||
| 内部字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `symbol` | 标准代码,如 `000001.SZ` |
|
||||
| `date` | 交易日 |
|
||||
| `open` / `high` / `low` / `close` | 不复权 OHLC |
|
||||
| `volume` | 成交量 |
|
||||
| `amount` | 成交额 |
|
||||
|
||||
### adj_factor 必填
|
||||
|
||||
| 内部字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `symbol` | 标准代码 |
|
||||
| `trade_date` | 除权日期 |
|
||||
| `ex_factor` | 复权因子 |
|
||||
|
||||
### realtime 必填
|
||||
|
||||
| 内部字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `symbol` | 标准代码 |
|
||||
| `last_price` | 最新价 |
|
||||
| `prev_close` | 昨收 |
|
||||
| `open` / `high` / `low` | 当日 OHLC |
|
||||
| `volume` | 成交量 |
|
||||
|
||||
建议实时接口额外提供 `amount`、`change_pct`、`change_amount`、`amplitude`、`turnover_rate`、`name`。缺失时部分字段会由 pipeline 回算,但精度取决于可用输入。
|
||||
|
||||
`change_pct` 和 `amplitude` 使用小数制,例如 `0.0366` 表示 `3.66%`。
|
||||
|
||||
## 请求约定
|
||||
|
||||
- `daily` / `adj_factor` 会按 `batch` 切分 symbols。
|
||||
- POST 请求会发送 JSON body: `symbols`、`start_time`、`end_time`。
|
||||
- GET 请求会发送 query 参数: `symbols=000001.SZ,600000.SH`。
|
||||
- `realtime` 必须是全市场快照接口,不支持逐个 symbol 拉实时行情。
|
||||
|
||||
可通过这些字段改参数名:
|
||||
|
||||
```yaml
|
||||
symbols_param: symbols
|
||||
start_param: start_time
|
||||
end_param: end_time
|
||||
```
|
||||
|
||||
## 鉴权
|
||||
|
||||
支持三种简单鉴权:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
type: bearer
|
||||
token_env: MY_DATA_TOKEN
|
||||
```
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
type: header
|
||||
header: X-Token
|
||||
token_env: MY_DATA_TOKEN
|
||||
```
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
type: query
|
||||
param: token
|
||||
token_env: MY_DATA_TOKEN
|
||||
```
|
||||
|
||||
Token 可以放在系统环境变量或项目 `.env` 中。
|
||||
|
||||
## 联调流程
|
||||
|
||||
1. 启动 mock 数据源:
|
||||
|
||||
```bash
|
||||
cd docs/examples/custom-data-source
|
||||
python mock_server.py
|
||||
```
|
||||
|
||||
2. 复制示例配置:
|
||||
|
||||
```bash
|
||||
mkdir -p data/data_sources
|
||||
cp docs/examples/custom-data-source/mock_source.yaml data/data_sources/mock_source.yaml
|
||||
```
|
||||
|
||||
3. 在「设置 -> 数据源」点击「重新加载」。
|
||||
|
||||
4. 使用「试拉测试」选择 `mock_source` 和 `daily` / `adj_factor` / `realtime`。
|
||||
|
||||
5. 保存数据源选择:
|
||||
|
||||
- 日K: `mock_source`
|
||||
- 除权因子: `same_as_daily` 或 `mock_source`
|
||||
- 实时行情: `mock_source`
|
||||
|
||||
6. 触发同步或开启实时行情。
|
||||
|
||||
## 常见错误
|
||||
|
||||
| 现象 | 处理 |
|
||||
| --- | --- |
|
||||
| 列表里没有 custom 源 | 检查 YAML 是否放在 `data/data_sources/` 并点击重新加载 |
|
||||
| errors 提示 missing mapped fields | `field_map` 没映射到必填内部字段 |
|
||||
| 试拉 rows 为 0 | 检查 `response_path` 是否指向数组 |
|
||||
| 日期列全为空 | 检查 `parse_date` 的格式是否和返回值一致 |
|
||||
| 实时行情没刷新 | 确认实时数据源已保存为 custom,且返回全市场快照 |
|
||||
@@ -0,0 +1,43 @@
|
||||
# 自定义数据源 mock 联调示例
|
||||
|
||||
这个目录提供一个本地 mock HTTP 数据源,用于验证项目的自定义数据源接入链路。
|
||||
|
||||
## 运行 mock 服务
|
||||
|
||||
```bash
|
||||
cd docs/examples/custom-data-source
|
||||
python mock_server.py
|
||||
```
|
||||
|
||||
服务默认监听:
|
||||
|
||||
```text
|
||||
http://127.0.0.1:3021
|
||||
```
|
||||
|
||||
端点:
|
||||
|
||||
| 端点 | 数据 |
|
||||
| --- | --- |
|
||||
| `/daily` | 日K |
|
||||
| `/adj_factor` | 除权因子 |
|
||||
| `/realtime` | 全市场实时快照 |
|
||||
|
||||
## 接入项目
|
||||
|
||||
复制示例 YAML 到运行数据目录:
|
||||
|
||||
```bash
|
||||
mkdir -p data/data_sources
|
||||
cp docs/examples/custom-data-source/mock_source.yaml data/data_sources/mock_source.yaml
|
||||
```
|
||||
|
||||
然后在项目里打开:
|
||||
|
||||
```text
|
||||
设置 -> 数据源 -> 重新加载
|
||||
```
|
||||
|
||||
选择 `mock_source` 后,可用「试拉测试」验证 `daily`、`adj_factor` 和 `realtime`。
|
||||
|
||||
完整说明见 [../../custom-data-source.md](../../custom-data-source.md)。
|
||||
@@ -0,0 +1,129 @@
|
||||
"""Mock custom market data source for local integration tests.
|
||||
|
||||
Run:
|
||||
python mock_server.py
|
||||
|
||||
Then copy mock_source.yaml to data/data_sources/mock_source.yaml and reload data
|
||||
sources in the app settings page.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
import uvicorn
|
||||
from fastapi import FastAPI, Request
|
||||
|
||||
app = FastAPI(title="Mock Custom Market Data Source")
|
||||
|
||||
SYMBOLS = {
|
||||
"000001.SZ": "平安银行",
|
||||
"600000.SH": "浦发银行",
|
||||
"300750.SZ": "宁德时代",
|
||||
}
|
||||
|
||||
BASE = {
|
||||
"000001.SZ": 10.20,
|
||||
"600000.SH": 8.60,
|
||||
"300750.SZ": 186.00,
|
||||
}
|
||||
|
||||
|
||||
def _parse_symbols(value: Any) -> list[str]:
|
||||
if isinstance(value, list):
|
||||
return [str(v) for v in value if str(v) in SYMBOLS]
|
||||
if isinstance(value, str) and value:
|
||||
return [s.strip() for s in value.split(",") if s.strip() in SYMBOLS]
|
||||
return list(SYMBOLS)
|
||||
|
||||
|
||||
async def _payload(request: Request) -> dict:
|
||||
if request.method == "POST":
|
||||
try:
|
||||
return await request.json()
|
||||
except Exception:
|
||||
return {}
|
||||
return dict(request.query_params)
|
||||
|
||||
|
||||
def _parse_date(value: Any, fallback: date) -> date:
|
||||
if not value:
|
||||
return fallback
|
||||
text = str(value)[:10]
|
||||
try:
|
||||
return date.fromisoformat(text)
|
||||
except ValueError:
|
||||
return fallback
|
||||
|
||||
|
||||
@app.api_route("/daily", methods=["GET", "POST"])
|
||||
async def daily(request: Request):
|
||||
body = await _payload(request)
|
||||
symbols = _parse_symbols(body.get("symbols"))
|
||||
end = _parse_date(body.get("end_time"), date.today())
|
||||
start = _parse_date(body.get("start_time"), end - timedelta(days=5))
|
||||
rows = []
|
||||
cur = start
|
||||
while cur <= end:
|
||||
if cur.weekday() < 5:
|
||||
offset = (cur - start).days
|
||||
for sym in symbols:
|
||||
base = BASE[sym] + offset * 0.03
|
||||
rows.append({
|
||||
"ts_code": sym,
|
||||
"trade_date": cur.isoformat(),
|
||||
"open": round(base, 2),
|
||||
"high": round(base * 1.015, 2),
|
||||
"low": round(base * 0.985, 2),
|
||||
"close": round(base * 1.004, 2),
|
||||
"vol": 120000 + offset * 1000,
|
||||
"amt": round((120000 + offset * 1000) * base, 2),
|
||||
})
|
||||
cur += timedelta(days=1)
|
||||
return {"code": 0, "data": rows}
|
||||
|
||||
|
||||
@app.api_route("/adj_factor", methods=["GET", "POST"])
|
||||
async def adj_factor(request: Request):
|
||||
body = await _payload(request)
|
||||
symbols = _parse_symbols(body.get("symbols"))
|
||||
today = date.today()
|
||||
return {
|
||||
"code": 0,
|
||||
"data": [
|
||||
{"ts_code": sym, "trade_date": today.isoformat(), "factor": 1.0}
|
||||
for sym in symbols
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@app.api_route("/realtime", methods=["GET", "POST"])
|
||||
async def realtime():
|
||||
now = datetime.now().isoformat(timespec="seconds")
|
||||
rows = []
|
||||
for i, (sym, name) in enumerate(SYMBOLS.items()):
|
||||
prev = BASE[sym]
|
||||
last = round(prev * (1 + (i + 1) * 0.006), 2)
|
||||
change = round(last - prev, 2)
|
||||
rows.append({
|
||||
"ts_code": sym,
|
||||
"name": name,
|
||||
"last": last,
|
||||
"pre_close": prev,
|
||||
"open": round(prev * 1.002, 2),
|
||||
"high": round(last * 1.01, 2),
|
||||
"low": round(prev * 0.99, 2),
|
||||
"vol": 150000 + i * 20000,
|
||||
"amt": round((150000 + i * 20000) * last, 2),
|
||||
"pct": change / prev,
|
||||
"amount_change": change,
|
||||
"amplitude": 0.025,
|
||||
"turnover": 0.012 + i * 0.001,
|
||||
"timestamp": now,
|
||||
"session": "regular",
|
||||
})
|
||||
return {"code": 0, "data": rows}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
uvicorn.run(app, host="127.0.0.1", port=3021)
|
||||
@@ -0,0 +1,58 @@
|
||||
name: mock_source
|
||||
display_name: "Mock 自定义数据源"
|
||||
auth:
|
||||
type: none
|
||||
|
||||
datasets:
|
||||
daily:
|
||||
url: http://127.0.0.1:3021/daily
|
||||
method: POST
|
||||
batch: 100
|
||||
rpm: 200
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
trade_date: date
|
||||
open: open
|
||||
high: high
|
||||
low: low
|
||||
close: close
|
||||
vol: volume
|
||||
amt: amount
|
||||
transforms:
|
||||
date: "parse_date(value, '%Y-%m-%d')"
|
||||
|
||||
adj_factor:
|
||||
url: http://127.0.0.1:3021/adj_factor
|
||||
method: POST
|
||||
batch: 100
|
||||
rpm: 200
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
trade_date: trade_date
|
||||
factor: ex_factor
|
||||
transforms:
|
||||
trade_date: "parse_date(value, '%Y-%m-%d')"
|
||||
|
||||
realtime:
|
||||
url: http://127.0.0.1:3021/realtime
|
||||
method: GET
|
||||
rpm: 60
|
||||
response_path: data
|
||||
field_map:
|
||||
ts_code: symbol
|
||||
name: name
|
||||
last: last_price
|
||||
pre_close: prev_close
|
||||
open: open
|
||||
high: high
|
||||
low: low
|
||||
vol: volume
|
||||
amt: amount
|
||||
pct: change_pct
|
||||
amount_change: change_amount
|
||||
amplitude: amplitude
|
||||
turnover: turnover_rate
|
||||
timestamp: timestamp
|
||||
session: session
|
||||
Reference in New Issue
Block a user