<!DOCTYPE html><html><head>
|
<title>system-architecture-summary</title>
|
<meta charset="utf-8">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<link rel="stylesheet" href="file:////Users/work/.vscode/extensions/shd101wyy.markdown-preview-enhanced-0.8.20/crossnote/dependencies/katex/katex.min.css">
|
|
|
|
|
|
<style>
|
code[class*=language-],pre[class*=language-]{color:#333;background:0 0;font-family:Consolas,"Liberation Mono",Menlo,Courier,monospace;text-align:left;white-space:pre;word-spacing:normal;word-break:normal;word-wrap:normal;line-height:1.4;-moz-tab-size:8;-o-tab-size:8;tab-size:8;-webkit-hyphens:none;-moz-hyphens:none;-ms-hyphens:none;hyphens:none}pre[class*=language-]{padding:.8em;overflow:auto;border-radius:3px;background:#f5f5f5}:not(pre)>code[class*=language-]{padding:.1em;border-radius:.3em;white-space:normal;background:#f5f5f5}.token.blockquote,.token.comment{color:#969896}.token.cdata{color:#183691}.token.doctype,.token.macro.property,.token.punctuation,.token.variable{color:#333}.token.builtin,.token.important,.token.keyword,.token.operator,.token.rule{color:#a71d5d}.token.attr-value,.token.regex,.token.string,.token.url{color:#183691}.token.atrule,.token.boolean,.token.code,.token.command,.token.constant,.token.entity,.token.number,.token.property,.token.symbol{color:#0086b3}.token.prolog,.token.selector,.token.tag{color:#63a35c}.token.attr-name,.token.class,.token.class-name,.token.function,.token.id,.token.namespace,.token.pseudo-class,.token.pseudo-element,.token.url-reference .token.variable{color:#795da3}.token.entity{cursor:help}.token.title,.token.title .token.punctuation{font-weight:700;color:#1d3e81}.token.list{color:#ed6a43}.token.inserted{background-color:#eaffea;color:#55a532}.token.deleted{background-color:#ffecec;color:#bd2c00}.token.bold{font-weight:700}.token.italic{font-style:italic}.language-json .token.property{color:#183691}.language-markup .token.tag .token.punctuation{color:#333}.language-css .token.function,code.language-css{color:#0086b3}.language-yaml .token.atrule{color:#63a35c}code.language-yaml{color:#183691}.language-ruby .token.function{color:#333}.language-markdown .token.url{color:#795da3}.language-makefile .token.symbol{color:#795da3}.language-makefile .token.variable{color:#183691}.language-makefile .token.builtin{color:#0086b3}.language-bash .token.keyword{color:#0086b3}pre[data-line]{position:relative;padding:1em 0 1em 3em}pre[data-line] .line-highlight-wrapper{position:absolute;top:0;left:0;background-color:transparent;display:block;width:100%}pre[data-line] .line-highlight{position:absolute;left:0;right:0;padding:inherit 0;margin-top:1em;background:hsla(24,20%,50%,.08);background:linear-gradient(to right,hsla(24,20%,50%,.1) 70%,hsla(24,20%,50%,0));pointer-events:none;line-height:inherit;white-space:pre}pre[data-line] .line-highlight:before,pre[data-line] .line-highlight[data-end]:after{content:attr(data-start);position:absolute;top:.4em;left:.6em;min-width:1em;padding:0 .5em;background-color:hsla(24,20%,50%,.4);color:#f4f1ef;font:bold 65%/1.5 sans-serif;text-align:center;vertical-align:.3em;border-radius:999px;text-shadow:none;box-shadow:0 1px #fff}pre[data-line] .line-highlight[data-end]:after{content:attr(data-end);top:auto;bottom:.4em}html body{font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif;font-size:16px;line-height:1.6;color:#333;background-color:#fff;overflow:initial;box-sizing:border-box;word-wrap:break-word}html body>:first-child{margin-top:0}html body h1,html body h2,html body h3,html body h4,html body h5,html body h6{line-height:1.2;margin-top:1em;margin-bottom:16px;color:#000}html body h1{font-size:2.25em;font-weight:300;padding-bottom:.3em}html body h2{font-size:1.75em;font-weight:400;padding-bottom:.3em}html body h3{font-size:1.5em;font-weight:500}html body h4{font-size:1.25em;font-weight:600}html body h5{font-size:1.1em;font-weight:600}html body h6{font-size:1em;font-weight:600}html body h1,html body h2,html body h3,html body h4,html body h5{font-weight:600}html body h5{font-size:1em}html body h6{color:#5c5c5c}html body strong{color:#000}html body del{color:#5c5c5c}html body a:not([href]){color:inherit;text-decoration:none}html body a{color:#08c;text-decoration:none}html body a:hover{color:#00a3f5;text-decoration:none}html body img{max-width:100%}html body>p{margin-top:0;margin-bottom:16px;word-wrap:break-word}html body>ol,html body>ul{margin-bottom:16px}html body ol,html body ul{padding-left:2em}html body ol.no-list,html body ul.no-list{padding:0;list-style-type:none}html body ol ol,html body ol ul,html body ul ol,html body ul ul{margin-top:0;margin-bottom:0}html body li{margin-bottom:0}html body li.task-list-item{list-style:none}html body li>p{margin-top:0;margin-bottom:0}html body .task-list-item-checkbox{margin:0 .2em .25em -1.8em;vertical-align:middle}html body .task-list-item-checkbox:hover{cursor:pointer}html body blockquote{margin:16px 0;font-size:inherit;padding:0 15px;color:#5c5c5c;background-color:#f0f0f0;border-left:4px solid #d6d6d6}html body blockquote>:first-child{margin-top:0}html body blockquote>:last-child{margin-bottom:0}html body hr{height:4px;margin:32px 0;background-color:#d6d6d6;border:0 none}html body table{margin:10px 0 15px 0;border-collapse:collapse;border-spacing:0;display:block;width:100%;overflow:auto;word-break:normal;word-break:keep-all}html body table th{font-weight:700;color:#000}html body table td,html body table th{border:1px solid #d6d6d6;padding:6px 13px}html body dl{padding:0}html body dl dt{padding:0;margin-top:16px;font-size:1em;font-style:italic;font-weight:700}html body dl dd{padding:0 16px;margin-bottom:16px}html body code{font-family:Menlo,Monaco,Consolas,'Courier New',monospace;font-size:.85em;color:#000;background-color:#f0f0f0;border-radius:3px;padding:.2em 0}html body code::after,html body code::before{letter-spacing:-.2em;content:'\00a0'}html body pre>code{padding:0;margin:0;word-break:normal;white-space:pre;background:0 0;border:0}html body .highlight{margin-bottom:16px}html body .highlight pre,html body pre{padding:1em;overflow:auto;line-height:1.45;border:#d6d6d6;border-radius:3px}html body .highlight pre{margin-bottom:0;word-break:normal}html body pre code,html body pre tt{display:inline;max-width:initial;padding:0;margin:0;overflow:initial;line-height:inherit;word-wrap:normal;background-color:transparent;border:0}html body pre code:after,html body pre code:before,html body pre tt:after,html body pre tt:before{content:normal}html body blockquote,html body dl,html body ol,html body p,html body pre,html body ul{margin-top:0;margin-bottom:16px}html body kbd{color:#000;border:1px solid #d6d6d6;border-bottom:2px solid #c7c7c7;padding:2px 4px;background-color:#f0f0f0;border-radius:3px}@media print{html body{background-color:#fff}html body h1,html body h2,html body h3,html body h4,html body h5,html body h6{color:#000;page-break-after:avoid}html body blockquote{color:#5c5c5c}html body pre{page-break-inside:avoid}html body table{display:table}html body img{display:block;max-width:100%;max-height:100%}html body code,html body pre{word-wrap:break-word;white-space:pre}}.markdown-preview{width:100%;height:100%;box-sizing:border-box}.markdown-preview ul{list-style:disc}.markdown-preview ul ul{list-style:circle}.markdown-preview ul ul ul{list-style:square}.markdown-preview ol{list-style:decimal}.markdown-preview ol ol,.markdown-preview ul ol{list-style-type:lower-roman}.markdown-preview ol ol ol,.markdown-preview ol ul ol,.markdown-preview ul ol ol,.markdown-preview ul ul ol{list-style-type:lower-alpha}.markdown-preview .newpage,.markdown-preview .pagebreak{page-break-before:always}.markdown-preview pre.line-numbers{position:relative;padding-left:3.8em;counter-reset:linenumber}.markdown-preview pre.line-numbers>code{position:relative}.markdown-preview pre.line-numbers .line-numbers-rows{position:absolute;pointer-events:none;top:1em;font-size:100%;left:0;width:3em;letter-spacing:-1px;border-right:1px solid #999;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}.markdown-preview pre.line-numbers .line-numbers-rows>span{pointer-events:none;display:block;counter-increment:linenumber}.markdown-preview pre.line-numbers .line-numbers-rows>span:before{content:counter(linenumber);color:#999;display:block;padding-right:.8em;text-align:right}.markdown-preview .mathjax-exps .MathJax_Display{text-align:center!important}.markdown-preview:not([data-for=preview]) .code-chunk .code-chunk-btn-group{display:none}.markdown-preview:not([data-for=preview]) .code-chunk .status{display:none}.markdown-preview:not([data-for=preview]) .code-chunk .output-div{margin-bottom:16px}.markdown-preview .md-toc{padding:0}.markdown-preview .md-toc .md-toc-link-wrapper .md-toc-link{display:inline;padding:.25rem 0}.markdown-preview .md-toc .md-toc-link-wrapper .md-toc-link div,.markdown-preview .md-toc .md-toc-link-wrapper .md-toc-link p{display:inline}.markdown-preview .md-toc .md-toc-link-wrapper.highlighted .md-toc-link{font-weight:800}.scrollbar-style::-webkit-scrollbar{width:8px}.scrollbar-style::-webkit-scrollbar-track{border-radius:10px;background-color:transparent}.scrollbar-style::-webkit-scrollbar-thumb{border-radius:5px;background-color:rgba(150,150,150,.66);border:4px solid rgba(150,150,150,.66);background-clip:content-box}html body[for=html-export]:not([data-presentation-mode]){position:relative;width:100%;height:100%;top:0;left:0;margin:0;padding:0;overflow:auto}html body[for=html-export]:not([data-presentation-mode]) .markdown-preview{position:relative;top:0;min-height:100vh}@media screen and (min-width:914px){html body[for=html-export]:not([data-presentation-mode]) .markdown-preview{padding:2em calc(50% - 457px + 2em)}}@media screen and (max-width:914px){html body[for=html-export]:not([data-presentation-mode]) .markdown-preview{padding:2em}}@media screen and (max-width:450px){html body[for=html-export]:not([data-presentation-mode]) .markdown-preview{font-size:14px!important;padding:1em}}@media print{html body[for=html-export]:not([data-presentation-mode]) #sidebar-toc-btn{display:none}}html body[for=html-export]:not([data-presentation-mode]) #sidebar-toc-btn{position:fixed;bottom:8px;left:8px;font-size:28px;cursor:pointer;color:inherit;z-index:99;width:32px;text-align:center;opacity:.4}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] #sidebar-toc-btn{opacity:1}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc{position:fixed;top:0;left:0;width:300px;height:100%;padding:32px 0 48px 0;font-size:14px;box-shadow:0 0 4px rgba(150,150,150,.33);box-sizing:border-box;overflow:auto;background-color:inherit}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar{width:8px}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar-track{border-radius:10px;background-color:transparent}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc::-webkit-scrollbar-thumb{border-radius:5px;background-color:rgba(150,150,150,.66);border:4px solid rgba(150,150,150,.66);background-clip:content-box}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc a{text-decoration:none}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc .md-toc{padding:0 16px}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc .md-toc .md-toc-link-wrapper .md-toc-link{display:inline;padding:.25rem 0}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc .md-toc .md-toc-link-wrapper .md-toc-link div,html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc .md-toc .md-toc-link-wrapper .md-toc-link p{display:inline}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .md-sidebar-toc .md-toc .md-toc-link-wrapper.highlighted .md-toc-link{font-weight:800}html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{left:300px;width:calc(100% - 300px);padding:2em calc(50% - 457px - 300px / 2);margin:0;box-sizing:border-box}@media screen and (max-width:1274px){html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{padding:2em}}@media screen and (max-width:450px){html body[for=html-export]:not([data-presentation-mode])[html-show-sidebar-toc] .markdown-preview{width:100%}}html body[for=html-export]:not([data-presentation-mode]):not([html-show-sidebar-toc]) .markdown-preview{left:50%;transform:translateX(-50%)}html body[for=html-export]:not([data-presentation-mode]):not([html-show-sidebar-toc]) .md-sidebar-toc{display:none}
|
/* Please visit the URL below for more information: */
|
/* https://shd101wyy.github.io/markdown-preview-enhanced/#/customize-css */
|
|
</style>
|
<!-- The content below will be included at the end of the <head> element. --><script type="text/javascript">
|
document.addEventListener("DOMContentLoaded", function () {
|
// your code here
|
});
|
</script></head><body for="html-export">
|
|
|
<div class="crossnote markdown-preview ">
|
|
<h1 id="深度学习平台迭代方案概述">深度学习平台迭代方案概述 </h1>
|
<h2 id="1-背景与目标">1. 背景与目标 </h2>
|
<ul>
|
<li>前端:Electron + Vue3(Pinia、Vue Router、Vite),目前在 Windows 侧完成项目管理、标注、训练触发,数据多保存在本地 JSON/文件夹。</li>
|
<li>后端:<code>deep-learning-platform-backend</code> 以 FastAPI 单体运行,SQLite 持久化;训练直接在服务进程中以线程池 + Python 进程执行。</li>
|
<li>目标:<strong>Windows 端只负责操作与展示,训练/推理/测试在服务器执行;新增 C++ 推理接口;将 JSON 数据迁移至后端数据库;训练/推理结果按阶段顺序输出并可终止。</strong></li>
|
</ul>
|
<h2 id="2-现有体系梳理">2. 现有体系梳理 </h2>
|
<table>
|
<thead>
|
<tr>
|
<th>区域</th>
|
<th>现状</th>
|
<th>关键文件/模块</th>
|
</tr>
|
</thead>
|
<tbody>
|
<tr>
|
<td>前端</td>
|
<td>Electron shell + Vue;通过 <code>window.electronAPI</code> 访问本地磁盘;API 调 FastAPI REST;标注区域如 <code>ImageArea.vue</code> 直接读本地图片</td>
|
<td><code>src/renderer</code> 目录、<code>api/index.js</code>、<code>store/trainingAndEvaluateData.js</code></td>
|
</tr>
|
<tr>
|
<td>后端</td>
|
<td>FastAPI 应用(<code>app.py</code>)挂载 models/training/labeling/dataset/evaluator 路由;训练接口 <code>src/api/training.py</code> 用线程池调用 <code>src/core/trainer.py</code>;状态写 SQLite</td>
|
<td><code>src/core</code>, <code>src/api</code>, <code>src/database</code></td>
|
</tr>
|
<tr>
|
<td>数据</td>
|
<td>项目、方法、模型配置多为 JSON;SQLite 中只有 <code>model_info</code>、<code>model_record</code>、<code>task_records</code></td>
|
<td><code>config/*.json</code>, <code>src/database/app_database.db</code></td>
|
</tr>
|
<tr>
|
<td>任务控制</td>
|
<td><code>task_manager.py</code> 为进程内单例,仅能管理当前进程任务;无法穿透多台服务器</td>
|
<td><code>src/core/task_manager.py</code></td>
|
</tr>
|
</tbody>
|
</table>
|
<h2 id="3-核心痛点">3. 核心痛点 </h2>
|
<ol>
|
<li><strong>计算位置</strong>:训练/推理与前端同机,难以托管服务器 GPU;线程池方式不可扩展。</li>
|
<li><strong>C++ 推理缺失</strong>:只能用 Python 模型,无法嵌入高性能推理接口。</li>
|
<li><strong>数据孤岛</strong>:JSON + 本地文件导致 Windows/服务器数据不一致,难以协同。</li>
|
<li><strong>任务状态与结果</strong>:目前结果一次性写 <code>model_info</code>,前端轮询 <code>/models/latest</code>;不可分阶段推送,也无法可靠终止。</li>
|
<li><strong>可运维性</strong>:单体 FastAPI + SQLite 缺乏分层与监控,日志/模型/数据与任务耦合。</li>
|
</ol>
|
<h2 id="4-目标能力拆解">4. 目标能力拆解 </h2>
|
<ol>
|
<li>Windows 前端仅负责交互,所有计算任务调度到服务器。</li>
|
<li>C++ 推理接口纳入统一调度,兼容 Python 训练。</li>
|
<li>统一数据库存数据(落地 Python 技术栈,与现有后端一致)。</li>
|
<li>训练/推理流程按阶段输出结果(顺序展示),支持随时暂停/恢复/终止。</li>
|
<li>支持扩容(多 GPU/多节点)、任务排队、资源隔离。</li>
|
</ol>
|
<h2 id="5-推荐总体架构">5. 推荐总体架构 </h2>
|
<pre data-role="codeBlock" data-info="" class="language-text"><code>Windows UI (Electron + Vue) ──REST/WebSocket──> API Gateway (FastAPI)
|
│
|
├─任务编排 (异步队列/调度器: Celery/Ray)
|
│
|
├─Python 训练 Worker (GPU)
|
├─C++ 推理服务 (gRPC/pybind11)
|
│
|
└─数据服务 (PostgreSQL/MySQL + 对象存储)
|
</code></pre><p>要点:</p>
|
<ul>
|
<li><strong>API Gateway</strong>:负责认证、项目信息、任务提交、状态查询、消息推送。</li>
|
<li><strong>任务编排</strong>:将 <code>task_manager</code> 升级为分布式任务状态机(Redis/DB 驱动),支持阶段管理。</li>
|
<li><strong>计算平面</strong>:Python 训练 Worker 运行 <code>Trainer</code>;C++ 推理可以嵌入或独立服务。</li>
|
<li><strong>数据平面</strong>:结构化数据进关系型数据库;大文件(数据集、模型、日志)进对象存储/NAS。</li>
|
<li><strong>事件流</strong>:WebSocket/SSE 将阶段事件回推前端,实现顺序展示与终止控制。</li>
|
</ul>
|
<h2 id="6-关键模块设计">6. 关键模块设计 </h2>
|
<h3 id="61-控制平面api-gateway">6.1 控制平面(API Gateway) </h3>
|
<ul>
|
<li>FastAPI 继续作为入口,但拆分出 <code>gateway</code> 层,专注 REST/WebSocket。</li>
|
<li>新增 <code>/auth</code>, <code>/projects</code>, <code>/tasks</code>, <code>/events</code> API,所有前端行为走远程接口。</li>
|
<li>WebSocket 渠道向前端推送任务阶段、日志、指标,减少轮询。</li>
|
</ul>
|
<h3 id="62-任务编排层">6.2 任务编排层 </h3>
|
<ul>
|
<li>引入 Celery/RQ/Ray/Airflow 之一:前端 <code>start training</code> → Gateway → 队列 → Worker。</li>
|
<li><code>task_manager</code> 改为 Redis/DB 持久化状态,包含 <code>PENDING/READY/RUNNING/PAUSED/STOPPED/FAILED/SUCCEEDED</code> 以及阶段信息。</li>
|
<li>统一处理暂停/恢复/终止指令:命令写入状态表或 Redis channel,Worker 轮询/订阅后执行。</li>
|
</ul>
|
<h3 id="63-计算平面">6.3 计算平面 </h3>
|
<ol>
|
<li><strong>Python 训练服务</strong>
|
<ul>
|
<li>复用 <code>src/core/trainer.py</code>,封装为 Worker;支持多 GPU(DDP),按任务配置加载数据。</li>
|
<li>将指标写入 <code>task_stage_events</code>,在每个 epoch/step 检查暂停/停止标志。</li>
|
</ul>
|
</li>
|
<li><strong>C++ 推理服务</strong>
|
<ul>
|
<li>定义统一接口:<code>loadModel</code>, <code>warmup</code>, <code>infer</code>, <code>cleanup</code>。</li>
|
<li>方案 A:pybind11 嵌入 Python Worker,复用相同任务调度。</li>
|
<li>方案 B:独立 C++ gRPC 服务,Gateway/Worker 通过 RPC 调用。</li>
|
<li>输出结构化结果(scores、bboxes、heatmaps)并存入数据库/对象存储。</li>
|
</ul>
|
</li>
|
</ol>
|
<h3 id="64-数据平面">6.4 数据平面 </h3>
|
<ul>
|
<li>关系型数据库:模型、任务、项目、标注、拆分、配置、用户等实体统一管理。</li>
|
<li>对象存储:训练日志、模型权重、推理输出、可视化图像。</li>
|
<li>提供导入导出工具,将旧 JSON 项目迁移到新库。</li>
|
</ul>
|
<h3 id="65-事件与结果顺序">6.5 事件与结果顺序 </h3>
|
<ul>
|
<li>设计 <code>task_stage</code>(stage_id, name, order, status, started_at, finished_at)与 <code>task_stage_events</code>(日志、指标、文件引用)。</li>
|
<li>Worker 在阶段进入/退出时写记录,并通过消息队列推送。</li>
|
<li>前端按阶段顺序渲染时间线,可显示进行中/成功/失败,每条阶段可单独终止。</li>
|
</ul>
|
<h2 id="7-数据库迁移与实现方案python">7. 数据库迁移与实现方案(Python) </h2>
|
<h3 id="71-技术栈选择">7.1 技术栈选择 </h3>
|
<ul>
|
<li><strong>数据库</strong>:PostgreSQL(推荐,支持 JSONB / 并发)或 MySQL 8。</li>
|
<li><strong>ORM & 迁移</strong>:SQLAlchemy 2.x + Alembic,统一管理 schema。</li>
|
<li><strong>连接池</strong>:SQLAlchemy AsyncEngine + asyncpg(或 psycopg3),配合 PgBouncer 做长连复用。</li>
|
<li><strong>部署</strong>:与 FastAPI Gateway 同语言,可直接在现有后端进程中启动 Session 工厂,避免跨语言 RPC。</li>
|
</ul>
|
<h3 id="72-数据模型">7.2 数据模型 </h3>
|
<p>核心实体:Project、Dataset、Split、Annotation、Model、ModelVersion、Task、TaskStage、TaskEvent、User、AccessToken。<br>
|
示例(伪代码):</p>
|
<pre data-role="codeBlock" data-info="python" class="language-python python"><code><span class="token keyword keyword-class">class</span> <span class="token class-name">Task</span><span class="token punctuation">(</span>Base<span class="token punctuation">)</span><span class="token punctuation">:</span>
|
__tablename__ <span class="token operator">=</span> <span class="token string">"tasks"</span>
|
<span class="token builtin">id</span> <span class="token operator">=</span> Column<span class="token punctuation">(</span>UUID<span class="token punctuation">,</span> primary_key<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span>
|
project_id <span class="token operator">=</span> Column<span class="token punctuation">(</span>UUID<span class="token punctuation">,</span> ForeignKey<span class="token punctuation">(</span><span class="token string">"projects.id"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
|
<span class="token builtin">type</span> <span class="token operator">=</span> Column<span class="token punctuation">(</span>Enum<span class="token punctuation">(</span>TaskType<span class="token punctuation">)</span><span class="token punctuation">)</span>
|
status <span class="token operator">=</span> Column<span class="token punctuation">(</span>Enum<span class="token punctuation">(</span>TaskStatus<span class="token punctuation">)</span><span class="token punctuation">)</span>
|
priority <span class="token operator">=</span> Column<span class="token punctuation">(</span>Integer<span class="token punctuation">,</span> default<span class="token operator">=</span><span class="token number">5</span><span class="token punctuation">)</span>
|
submitter <span class="token operator">=</span> Column<span class="token punctuation">(</span>UUID<span class="token punctuation">,</span> ForeignKey<span class="token punctuation">(</span><span class="token string">"users.id"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
|
created_at <span class="token operator">=</span> Column<span class="token punctuation">(</span>DateTime<span class="token punctuation">,</span> server_default<span class="token operator">=</span>func<span class="token punctuation">.</span>now<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
|
updated_at <span class="token operator">=</span> Column<span class="token punctuation">(</span>DateTime<span class="token punctuation">,</span> onupdate<span class="token operator">=</span>func<span class="token punctuation">.</span>now<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
|
</code></pre><p>每个 Stage、Event 表均包含 <code>task_id</code> 外键与 JSONB 字段 <code>metrics</code> / <code>artifacts</code> 以便扩展。</p>
|
<h3 id="73-演进步骤">7.3 演进步骤 </h3>
|
<ol>
|
<li><strong>Schema 初始化</strong>:使用 Alembic 生成迁移脚本,创建全量表结构与索引(task 状态+project_id 复合索引、event 时间索引等)。</li>
|
<li><strong>数据迁移工具</strong>:编写 <code>python scripts/migrate_from_json.py</code>,流程为
|
<ul>
|
<li>扫描 <code>config/*.json</code>, <code>project/**/project.json</code> 等;</li>
|
<li>解析出项目/拆分/标注/模型文件,映射到 ORM 对象;</li>
|
<li>批量写入新库,并将原始文件路径写入对象存储后回填 URI。</li>
|
</ul>
|
</li>
|
<li><strong>双写阶段</strong>:前端改造后,同时写入旧 JSON 与新 DB,验证一致性;稳定后切换为数据库唯一事实源。</li>
|
<li><strong>读写分离与备份</strong>:
|
<ul>
|
<li>接入 PgBouncer/ProxySQL,Gateway 与 Worker 走读写分离;</li>
|
<li>使用 WAL 归档或 pgBackRest 做自动备份与 PITR(Point-in-time Recovery)。</li>
|
</ul>
|
</li>
|
</ol>
|
<h3 id="74-api服务改动">7.4 API/服务改动 </h3>
|
<ul>
|
<li><code>db_manager</code> 升级为 ORM Session 工厂(async with SessionLocal as session)。</li>
|
<li>所有 <code>/models/*</code>, <code>/training/*</code>, <code>/dataset/*</code> API 改为访问数据库,而非本地 JSON/SQLite。</li>
|
<li>新增 <code>/admin/migrations</code> 或 CLI 命令,支持查看当前 schema 版本、回滚、重新迁移。</li>
|
</ul>
|
<h3 id="75-验证与回滚">7.5 验证与回滚 </h3>
|
<ul>
|
<li>数据校验脚本:对比 JSON 与数据库记录数量、哈希、样本抽检。</li>
|
<li>回滚策略:迁移失败时,删除新库数据/回滚事务,保留旧 JSON;生产环境启用只读窗口。</li>
|
<li>监控:对 PostgreSQL 连接数、慢查询、复制延迟进行告警,确保高并发场景稳定。</li>
|
</ul>
|
<h2 id="8-c-推理接口策略">8. C++ 推理接口策略 </h2>
|
<ul>
|
<li><strong>接口契约</strong>:<code>InferenceRequest</code>(模型/版本/输入/设备/批量参数)→ <code>InferenceResult</code>(状态、耗时、输出 URI、可视化路径、错误信息)。</li>
|
<li><strong>部署形态</strong>:
|
<ul>
|
<li>嵌入式:pybind11/ctypes 将 C++ 动态库暴露给 Python Worker,低延迟。</li>
|
<li>微服务:C++ gRPC 服务与 Python Worker 解耦,利于独立扩缩容。</li>
|
</ul>
|
</li>
|
<li><strong>资源管理</strong>:通过任务编排器向 C++ 服务申请 GPU/CPU 配额;可引入 Kubernetes/NVIDIA DCGM 做 GPU 调度。</li>
|
</ul>
|
<h2 id="9-windows-前端改造">9. Windows 前端改造 </h2>
|
<ol>
|
<li><code>window.electronAPI</code> 仅保留文件选择/本地预览;项目数据改为调用 <code>/projects</code> API。</li>
|
<li>标注提交、拆分配置、训练配置等都由后端持久化;本地仅做缓存。</li>
|
<li>新增 WebSocket 客户端,接收任务阶段事件,驱动 Train/Evaluate 页面。</li>
|
<li>提供项目导入/导出功能:导入旧 JSON → 上传到服务器 → 服务器写入数据库。</li>
|
</ol>
|
<h2 id="10-建议实施步骤">10. 建议实施步骤 </h2>
|
<ol>
|
<li><strong>基础设施</strong>:搭建 API Gateway + 新数据库 + 对象存储;完成鉴权与基础 CRUD。</li>
|
<li><strong>任务队列</strong>:引入队列/调度器,重构 <code>task_manager</code>;Python 训练 Worker 接入。</li>
|
<li><strong>C++ 推理 MVP</strong>:确定接口方式,完成一个模型的推理链路。</li>
|
<li><strong>前端重构</strong>:改用远程 API + WebSocket;Train/Evaluate 页面切换到事件驱动。</li>
|
<li><strong>数据迁移</strong>:编写脚本迁移 JSON/SQLite → 新库;验证一致性。</li>
|
<li><strong>逐步上线</strong>:先将训练迁移到服务器,再切换推理/评估;最后关闭本地 JSON 写入。</li>
|
</ol>
|
<p>此方案在保留现有代码基础上,提供清晰的演进路径,满足集中计算、C++ 推理、统一数据库以及任务可控可视的需求。</p>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
</body></html>
|