/** * 数据拆分后的训练数据准备工具 * 用于在拆分创建后进行实际的数据组织和准备工作 */ import { performDatasetSplit } from '../../api/index.js' /** * 准备拆分数据集 * @param {Object} project - 项目信息 * @param {Object} splitData - 拆分配置数据 * @param {Object} fullSplitData - 完整拆分数据 * @param {boolean} useBackendSplit - 是否使用后端拆分逻辑(默认true) * @returns {Promise} 准备结果 */ export async function prepareSplitDataset(project, splitData, fullSplitData, useBackendSplit = true) { console.log('[SplitDataPreparation] 开始准备拆分数据集:', splitData.name, '使用后端拆分:', useBackendSplit) try { // 1. 创建拆分目录 const splitDir = `${project.path}/splits/${splitData.id}` const metadataPath = `${splitDir}/metadata.json` const splitConfigPath = `${splitDir}/split-config.json` // 确保拆分目录存在 if (window.electronAPI && window.electronAPI.ensureDirectory) { const result = await window.electronAPI.ensureDirectory(splitDir) if (!result.success) { throw new Error(`创建拆分目录失败: ${splitDir} - ${result.error}`) } console.log('[SplitDataPreparation] 拆分目录创建成功:', splitDir) } // 2. 生成训练元数据文件(根据配置选择使用后端或前端逻辑) let metadataResult if (useBackendSplit) { // 使用后端拆分逻辑 metadataResult = await generateTrainingMetadataWithBackend(project, splitData, fullSplitData, splitDir, metadataPath) } else { // 使用前端拆分逻辑(备用) metadataResult = await generateTrainingMetadata(project, splitData, fullSplitData, splitDir, metadataPath) } // 3. 生成拆分配置文件(包含项目和拆分的关联信息) const splitConfigResult = await generateSplitConfig(project, splitData, fullSplitData, splitDir, splitConfigPath) console.log('[SplitDataPreparation] 数据集准备完成:', { splitDir, metadata: metadataResult, splitConfig: splitConfigResult }) return { success: true, splitDir, metadataPath: metadataResult.path, splitConfigPath: splitConfigResult.path, totalImages: metadataResult.images } } catch (error) { console.error('[SplitDataPreparation] 准备数据集失败:', error) throw error } } /** * 使用后端拆分逻辑生成训练元数据 * @param {Object} project - 项目信息 * @param {Object} splitData - 拆分配置数据 * @param {Object} fullSplitData - 完整拆分数据 * @param {string} splitDir - 拆分目录路径 * @param {string} metadataPath - 元数据文件路径 * @returns {Promise} 元数据生成结果 */ export async function generateTrainingMetadataWithBackend(project, splitData, fullSplitData, splitDir, metadataPath) { console.log('[SplitDataPreparation] 使用后端拆分逻辑生成训练元数据...') try { // 1. 准备符合后端格式的数据 const categories = [] const categoryMap = new Map() const categorySet = new Map() // 从项目收集类别信息 if (project.types && Array.isArray(project.types)) { for (const category of project.types) { if (!categorySet.has(category.key)) { categorySet.set(category.key, { key: category.key, title: category.title, color: category.color }) } } } // 如果没有从图片收集到类别,使用项目默认类别 if (categorySet.size === 0 && project.types && Array.isArray(project.types)) { for (const projectCategory of project.types) { categorySet.set(projectCategory.key, { key: projectCategory.key, title: projectCategory.title, color: projectCategory.color }) } } // 特殊处理:异常检测和缺陷生成项目确保有"good"类别 if ((project.instance?.id === 2 || project.instance?.id === 5) && !categorySet.has(0)) { categorySet.set(0, { key: 0, title: 'good', color: '#00B578' }) } // 转换为后端格式的类别数组 let categoryId = 0 for (const [key, category] of categorySet) { categories.push({ id: categoryId, name: category.title, color: category.color || null }) categoryMap.set(key, categoryId) categoryId++ } // 2. 准备图片数据(转换为后端格式) const images = {} for (const image of project.images || []) { const annotations = [] if (image.annotations && Array.isArray(image.annotations)) { for (const annotation of image.annotations) { let categoryId = null if (annotation.categoryId !== null && annotation.categoryId !== undefined) { categoryId = categoryMap.get(annotation.categoryId) } // 转换标注类型和points格式以符合后端要求 let convertedType = annotation.type let convertedPoints = null switch (annotation.type) { case 'image-level': convertedType = 'classify' convertedPoints = null break case 'line-polygon': case 'ai-polygon': convertedType = 'polygon' // 转换points格式:从{x,y}对象数组转换为坐标数组 if (annotation.points && Array.isArray(annotation.points)) { const flatCoords = [] for (const point of annotation.points) { if (point && typeof point === 'object' && 'x' in point && 'y' in point) { flatCoords.push(point.x, point.y) } } convertedPoints = flatCoords.length > 0 ? [flatCoords] : null } break case 'classify': convertedType = 'classify' convertedPoints = null break case 'rectangle': convertedType = 'rectangle' // rectangle类型:转换为左上角和右下角坐标 if (annotation.points && Array.isArray(annotation.points)) { convertedPoints = [annotation.points] } break default: convertedType = annotation.type convertedPoints = annotation.points break } annotations.push({ id: annotation.id, type: convertedType, categoryId: categoryId, points: convertedPoints, zIndex: annotation.zIndex || 0, thumbnailData: null }) } } // 构建图片信息 let imageWidth = image.width || 0 let imageHeight = image.height || 0 if ((imageWidth === 0 || imageHeight === 0) && image.size && typeof image.size === 'string') { const sizeMatch = image.size.match(/(\d+)\*(\d+)/) if (sizeMatch) { imageWidth = parseInt(sizeMatch[1], 10) imageHeight = parseInt(sizeMatch[2], 10) } } const absoluteImagePath = `${project.path}/${image.path}` images[image.id] = { id: image.id, name: image.name, path: absoluteImagePath, height: imageHeight, width: imageWidth, dataset: 'unlabeled', // 初始状态,将由后端拆分逻辑决定 annotations } } // 3. 准备任务类型映射 const taskMapping = { '分类': '分类', '目标检测': '目标检测', '实例分割': '实例分割', '异常检测': '异常检测', '缺陷生成': '缺陷生成', 'Classification': '分类', 'ObjectDetection': '目标检测', 'InstanceSegmentation': '实例分割', 'AnomalyDetection': '异常检测', 'DefectGeneration': '缺陷生成' } const task = taskMapping[project.method] || project.method || '分类' // 判断是否勾选了无监督任务 (project.instance?.id === 2 用于判断是否显示无监督选项) const is_unsupervised = splitData.isUnsupervised || false // 4. 调用后端API进行拆分 // 计算实际的比例 const totalCount = splitData.trainCount + splitData.valCount + splitData.testCount const ratios = { train: totalCount > 0 ? splitData.trainCount / totalCount : 0.7, val: totalCount > 0 ? splitData.valCount / totalCount : 0.2, test: totalCount > 0 ? splitData.testCount / totalCount : 0.1 } console.log('[SplitDataPreparation] 计算拆分比例:', { counts: { train: splitData.trainCount, val: splitData.valCount, test: splitData.testCount }, ratios, totalImages: Object.keys(images).length }) let splitResult try { splitResult = await performDatasetSplit({ metadata: { categories, images, task }, split_id: splitData.id, ratios, task, is_unsupervised, seed: splitData.seed || splitData.randomSeed || 0 }) if (!splitResult.success) { throw new Error(splitResult.error || '后端拆分失败') } } catch (apiError) { // API调用失败,清理已创建的目录 console.error('[SplitDataPreparation] 后端API调用失败,开始清理本地文件:', apiError) // 删除拆分目录 if (window.electronAPI && window.electronAPI.deleteDirectory) { const deleteResult = await window.electronAPI.deleteDirectory(splitDir) if (deleteResult.success) { console.log('[SplitDataPreparation] 已清理拆分目录:', splitDir) } else { console.error('[SplitDataPreparation] 清理拆分目录失败:', deleteResult.error) } } // 重新抛出错误 throw new Error(`网络请求失败: ${apiError.message || '无法连接到后端服务'}`) } // 5. 更新images的dataset字段和assignments const splitInfo = splitResult.data.splits[splitData.id] // 创建新的assignments对象,基于后端返回的实际分配结果 const updatedAssignments = {} // 根据后端返回的拆分结果更新图片的dataset字段和assignments for (const imageId of splitInfo.train || []) { if (images[imageId]) { images[imageId].dataset = 'train' updatedAssignments[imageId] = 'train' } } for (const imageId of splitInfo.val || []) { if (images[imageId]) { images[imageId].dataset = 'val' updatedAssignments[imageId] = 'val' } } for (const imageId of splitInfo.test || []) { if (images[imageId]) { images[imageId].dataset = 'test' updatedAssignments[imageId] = 'test' } } for (const imageId of splitInfo.unselected || []) { if (images[imageId]) { images[imageId].dataset = 'unselected' updatedAssignments[imageId] = 'unselected' } } for (const imageId of splitInfo.unlabeled || []) { if (images[imageId]) { images[imageId].dataset = 'unlabeled' updatedAssignments[imageId] = 'unlabeled' } } // 使用后端返回的实际分配结果更新splitData splitData.assignments = updatedAssignments // 6. 构建最终的元数据对象 const metadata = { categories, images, splits: splitResult.data.splits, task } // 7. 保存元数据文件 const metadataContent = JSON.stringify(metadata, null, 2) if (window.electronAPI && window.electronAPI.saveFile) { const saveResult = await window.electronAPI.saveFile(metadataPath, metadataContent) if (!saveResult.success) { throw new Error(`保存元数据文件失败: ${saveResult.error}`) } } console.log('[SplitDataPreparation] 使用后端拆分逻辑成功生成元数据:', metadataPath) // 统计信息 const trainingStats = { totalImages: Object.keys(images).length, totalCategories: categories.length, trainImages: (splitInfo.train || []).length, validImages: (splitInfo.val || []).length, testImages: (splitInfo.test || []).length, unselectedImages: (splitInfo.unselected || []).length, unlabelImages: (splitInfo.unlabeled || []).length } return { path: metadataPath, statistics: trainingStats, categories: categories.length, images: Object.keys(images).length } } catch (error) { console.error('[SplitDataPreparation] 使用后端拆分逻辑失败:', error) throw error } } /** * 生成训练元数据文件(原有的拆分逻辑,备用) * @param {Object} project - 项目信息 * @param {Object} splitData - 拆分配置数据 * @param {Object} fullSplitData - 完整拆分数据 * @param {string} splitDir - 拆分目录路径 * @param {string} metadataPath - 元数据文件路径 * @returns {Promise} 元数据生成结果 */ export async function generateTrainingMetadata(project, splitData, fullSplitData, splitDir, metadataPath) { console.log('[SplitDataPreparation] 生成训练元数据...') try { // 构建类别信息 - 从所有图片收集类别 const categories = [] const categoryMap = new Map() const categorySet = new Map() // 用于去重,key是category.key,value是category对象 // 从项目收集类别信息 if (project.types && Array.isArray(project.types)) { for (const category of project.types) { if (!categorySet.has(category.key)) { categorySet.set(category.key, { key: category.key, title: category.title, color: category.color }) } } } // 如果没有从图片收集到类别,尝试使用项目默认类别 if (categorySet.size === 0 && project.types && Array.isArray(project.types)) { console.log('[SplitDataPreparation] 图片无类别,使用项目默认类别:', project.types) for (const projectCategory of project.types) { categorySet.set(projectCategory.key, { key: projectCategory.key, title: projectCategory.title, color: projectCategory.color }) } } // 特殊处理:异常检测和缺陷生成项目确保有"good"类别 if ((project.instance?.id === 2 || project.instance?.id === 5) && !categorySet.has(0)) { console.log('[SplitDataPreparation] 为异常检测/缺陷生成项目添加默认 "good" 类别') categorySet.set(0, { key: 0, title: 'good', color: '#00B578' }) } // 转换为数组并创建映射 let categoryId = 0 for (const [key, category] of categorySet) { categories.push({ id: categoryId, name: category.title, color: category.color || null }) categoryMap.set(key, categoryId) categoryId++ } console.log('[SplitDataPreparation] 收集到的类别:', categories) // 构建符合系统要求格式的图片信息对象 const images = {} const assignments = new Map(Object.entries(splitData.assignments || {})) const datasetStats = { train: 0, val: 0, test: 0, unselected: 0, unlabel: 0 } const categoryStats = { train: {}, val: {}, test: {}, unselected: {}, unlabel: {} } // 初始化类别统计 for (const category of categories) { categoryStats.train[category.id] = 0 categoryStats.val[category.id] = 0 categoryStats.test[category.id] = 0 categoryStats.unselected[category.id] = 0 categoryStats.unlabel[category.id] = 0 } // 处理所有图片,直接使用拆分时已经确定的分配结果 for (const image of project.images || []) { const assignment = assignments.get(image.id) || 'unlabel' // 统计各数据集的图片数量 if (datasetStats.hasOwnProperty(assignment)) { datasetStats[assignment]++ } // 转换标注信息 const annotations = [] if (image.annotations && Array.isArray(image.annotations)) { for (const annotation of image.annotations) { // 查找对应的类别ID let categoryId = null if (annotation.categoryId !== null && annotation.categoryId !== undefined) { categoryId = categoryMap.get(annotation.categoryId) // 统计类别分布 if (categoryId !== null && categoryStats[assignment]) { categoryStats[assignment][categoryId]++ } // 如果仍然找不到类别,记录警告 if (categoryId === null || categoryId === undefined) { console.warn(`[SplitDataPreparation] 无法找到categoryId ${annotation.categoryId} 对应的类别映射`) } } else { console.warn(`[SplitDataPreparation] 标注 ${annotation.id} 的categoryId为null或undefined`) } // 转换标注类型和points格式 let convertedType = annotation.type let convertedPoints = null // 标注类型转换规则 switch (annotation.type) { case 'image-level': convertedType = 'classify' convertedPoints = null break case 'line-polygon': case 'ai-polygon': convertedType = 'polygon' // 转换points格式:从{x,y}对象数组转换为坐标数组 if (annotation.points && Array.isArray(annotation.points)) { const flatCoords = [] for (const point of annotation.points) { if (point && typeof point === 'object' && 'x' in point && 'y' in point) { flatCoords.push(point.x, point.y) } } convertedPoints = flatCoords.length > 0 ? [flatCoords] : null } break case 'classify': convertedType = 'classify' convertedPoints = null break case 'rectangle': convertedType = 'rectangle' // rectangle类型保持原始points格式 convertedPoints = annotation.points break default: // 其他类型保持不变 convertedType = annotation.type convertedPoints = annotation.points break } annotations.push({ id: annotation.id, type: convertedType, categoryId: categoryId, points: convertedPoints, zIndex: annotation.zIndex || 0, thumbnailData: null }) } } // 构建符合系统格式的图片信息 let imageWidth = image.width || 0 let imageHeight = image.height || 0 // 如果没有独立的width/height字段,尝试从size字段解析 if ((imageWidth === 0 || imageHeight === 0) && image.size && typeof image.size === 'string') { const sizeMatch = image.size.match(/(\d+)\*(\d+)/) if (sizeMatch) { imageWidth = parseInt(sizeMatch[1], 10) imageHeight = parseInt(sizeMatch[2], 10) } } // 构建绝对路径 const absoluteImagePath = `${project.path}/${image.path}` images[image.id] = { id: image.id, name: image.name, path: absoluteImagePath, height: imageHeight, width: imageWidth, dataset: assignment, annotations } } // 构建拆分信息 - 直接从assignments读取分配结果 const splitInfo = { train: [], val: [], test: [], unselected: [], unlabel: [], trainDist: categoryStats.train, valDist: categoryStats.val, testDist: categoryStats.test, unselectedDist: categoryStats.unselected, unlabelDist: categoryStats.unlabel } // 根据assignments填充拆分列表 for (const [imageId, assignment] of assignments) { if (splitInfo[assignment]) { splitInfo[assignment].push(imageId) } } // 处理未分配的图片(全部归类为unlabel) for (const image of project.images || []) { if (!assignments.has(image.id)) { splitInfo.unlabel.push(image.id) } } // 构建符合系统要求格式的完整元数据对象 const metadata = { categories, images, splits: { [splitData.id]: splitInfo }, task: project.method || "分类" // 根据项目方法确定任务类型 } // 保留训练统计信息用于返回 const trainingStats = { totalImages: Object.keys(images).length, totalCategories: categories.length, trainImages: datasetStats.train, validImages: datasetStats.val, testImages: datasetStats.test, unselectedImages: datasetStats.unselected, unlabelImages: datasetStats.unlabel } // 保存符合系统要求格式的元数据文件 const metadataContent = JSON.stringify(metadata, null, 2) if (window.electronAPI && window.electronAPI.saveFile) { const saveResult = await window.electronAPI.saveFile(metadataPath, metadataContent) if (!saveResult.success) { throw new Error(`保存元数据文件失败: ${saveResult.error}`) } } console.log('[SplitDataPreparation] 元数据文件生成成功:', metadataPath) console.log('[SplitDataPreparation] 数据集统计:', trainingStats) console.log('[SplitDataPreparation] 拆分信息:', { train: splitInfo.train.length, val: splitInfo.val.length, test: splitInfo.test.length, unselected: splitInfo.unselected.length, unlabel: splitInfo.unlabel.length }) return { path: metadataPath, statistics: trainingStats, categories: categories.length, images: Object.keys(images).length } } catch (error) { console.error('[SplitDataPreparation] 生成元数据失败:', error) throw error } } /** * 生成拆分配置文件 * @param {Object} project - 项目信息 * @param {Object} splitData - 拆分配置数据 * @param {Object} fullSplitData - 完整拆分数据 * @param {string} splitDir - 拆分目录路径 * @param {string} splitConfigPath - 拆分配置文件路径 * @returns {Promise} 拆分配置生成结果 */ export async function generateSplitConfig(project, splitData, fullSplitData, splitDir, splitConfigPath) { console.log('[SplitDataPreparation] 生成拆分配置文件...') try { // 构建拆分配置信息 const splitConfig = { // 拆分基本信息 split: { id: splitData.id, name: splitData.name, title: splitData.title, description: splitData.description || '', createdAt: splitData.createdAt || new Date().toISOString(), updatedAt: new Date().toISOString(), strategy: splitData.strategy || 'random', ratios: splitData.ratios || { train: 0.7, valid: 0.2, test: 0.1 }, assignments: splitData.assignments || {} }, // 项目关联信息 project: { id: project.id, name: project.name, path: project.path, method: project.method, instance: project.instance }, // 统计信息 statistics: { totalImages: Object.keys(splitData.assignments || {}).length, datasets: {} }, // 版本和时间戳 version: "1.0", createdAt: new Date().toISOString() } // 计算各数据集的统计信息 const datasetCounts = { train: 0, valid: 0, test: 0 } if (splitData.assignments) { for (const assignment of Object.values(splitData.assignments)) { if (datasetCounts.hasOwnProperty(assignment)) { datasetCounts[assignment]++ } } } splitConfig.statistics.datasets = datasetCounts // 保存拆分配置文件 const configContent = JSON.stringify(splitConfig, null, 2) if (window.electronAPI && window.electronAPI.saveFile) { const saveResult = await window.electronAPI.saveFile(splitConfigPath, configContent) if (!saveResult.success) { throw new Error(`保存拆分配置文件失败: ${saveResult.error}`) } } console.log('[SplitDataPreparation] 拆分配置文件生成成功:', splitConfigPath) console.log('[SplitDataPreparation] 拆分统计:', splitConfig.statistics) return { path: splitConfigPath, statistics: splitConfig.statistics } } catch (error) { console.error('[SplitDataPreparation] 生成拆分配置失败:', error) throw error } } /** * 验证拆分数据集完整性 * @param {string} splitDir - 拆分目录路径 * @returns {Promise} 验证结果 */ export async function validateDataset(splitDir) { console.log('[SplitDataPreparation] 验证拆分数据集完整性...') const validation = { metadataExists: false, splitConfigExists: false, errors: [] } try { const metadataPath = `${splitDir}/metadata.json` const splitConfigPath = `${splitDir}/split-config.json` // 检查训练元数据文件 if (window.electronAPI && window.electronAPI.pathExists) { validation.metadataExists = await window.electronAPI.pathExists(metadataPath) validation.splitConfigExists = await window.electronAPI.pathExists(splitConfigPath) } // 记录缺失的组件 if (!validation.metadataExists) { validation.errors.push('训练元数据文件 (metadata.json) 不存在') } if (!validation.splitConfigExists) { validation.errors.push('拆分配置文件 (split-config.json) 不存在') } validation.isValid = validation.errors.length === 0 console.log('[SplitDataPreparation] 拆分数据集验证结果:', validation) return validation } catch (error) { console.error('[SplitDataPreparation] 验证拆分数据集失败:', error) validation.errors.push(`验证失败: ${error.message}`) validation.isValid = false return validation } }