diff --git a/cv/README.md b/cv/README.md index d8edae0..223bb8c 100644 --- a/cv/README.md +++ b/cv/README.md @@ -9,6 +9,7 @@ This directory contains ready-to-use Computer Vision application notebooks built | 1 | [ResNet](./resnet/) | Includes notebooks for ResNet finetuning on tasks such as chinese herbal classification | | 2 | [U-Net](./unet/) | Includes notebooks for U-Net training on tasks such as segmentation | | 3 | [SAM](./sam/) | Includes notebooks for using SAM to inference | +| 4 | [Swin Transformer](./swin/) | Includes notebooks for Swin Transformer finetuning on tasks such as image classification | ## Contributing New CV Applications diff --git a/cv/swin/finetune_swin_tiny_image_classification.ipynb b/cv/swin/finetune_swin_tiny_image_classification.ipynb new file mode 100644 index 0000000..6cc65ff --- /dev/null +++ b/cv/swin/finetune_swin_tiny_image_classification.ipynb @@ -0,0 +1,462 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# 基于 Swin-Tiny 的图像分类微调\n", + "\n", + "## 1. 案例简介\n", + "\n", + "本案例展示了如何使用 MindSpore 和 MindSpore NLP 库,在自定义数据集(EuroSAT)上微调预训练的 Swin-Tiny 视觉 Transformer 模型。\n", + "\n", + "**Swin Transformer** 是一种分层的 Vision Transformer,其通过移动窗口计算自注意力(Shifted Windows),在图像分类、目标检测等任务上取得了优异的性能。Swin-Tiny 是该系列中轻量级的版本,适合在算力有限或需要快速迭代的场景下使用。\n", + "\n", + "### 任务流程\n", + "1. **环境准备**:安装并导入必要的库。\n", + "2. **数据加载**:加载 EuroSAT 卫星图像数据集。\n", + "3. **数据预处理**:使用 MindSpore Dataset 对图像进行 Resize、归一化等操作。\n", + "4. **模型构建**:加载预训练的 `swin-tiny-patch4-window7-224` 模型,并修改分类头。\n", + "5. **模型微调**:配置训练参数,使用 `Trainer` 接口进行训练。\n", + "6. **推理演示**:加载微调后的模型对新图像进行预测。\n", + "\n", + "### 运行环境\n", + "\n", + "| Python | MindSpore | MindSpore NLP |\n", + "| :----- | :-------- | :------------ |\n", + "| 3.10 | 2.7.0 | 0.5.1 |" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1", + "metadata": {}, + "outputs": [], + "source": [ + "#若在https://internstudio-ascend.intern-ai.org.cn/console/instance进行开发时,使用notebook会出现无法正常使用NPU,可进行以下步骤:\n", + "# 进入开发机的命令窗口\n", + "# 1. 激活你的环境\n", + "# conda activate mind_py310\n", + "# pip install ipykernel\n", + "# python -m ipykernel install --user --name=mind_py310 --display-name=\"Python (mind_py310)\"\n", + "# 2. 加载系统基础驱动配置\n", + "# source /usr/local/Ascend/ascend-toolkit/set_env.sh\n", + "# 3.【核心步骤】手动补全深层驱动路径 (修复 libascend_hal.so 报错)\n", + "# export LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64:$LD_LIBRARY_PATH\n", + "# 4. 启动 Jupyter Lab\n", + "#jupyter lab --allow-root" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "# 安装依赖\n", + "# !pip install mindnlp==0.5.1\n", + "# !pip install scikit-learn" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## 2. 环境配置\n", + "首先,我们需要应用一个补丁来修复 mindtorch 的版本兼容性问题,并配置 MindSpore 的运行环境。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "# --------兼容性补丁---------\n", + "import mindtorch.autograd.function\n", + "if not hasattr(mindtorch.autograd.function, 'FunctionCtx'):\n", + " class FunctionCtx:\n", + " def __init__(self):\n", + " self.saved_tensors = ()\n", + " def save_for_backward(self, *tensors):\n", + " self.saved_tensors = tensors\n", + " mindtorch.autograd.function.FunctionCtx = FunctionCtx\n", + " print(\"已应用 mindtorch 兼容性修复\")\n", + "\n", + "# ----基础配置与环境 ----\n", + "import mindnlp\n", + "import mindspore\n", + "import os\n", + "\n", + "# 设置 Token\n", + "os.environ[\"HF_TOKEN\"] = \"hf_**\" \n", + "\n", + "# 清理离线环境变量,确保联网\n", + "if 'HF_DATASETS_OFFLINE' in os.environ: del os.environ['HF_DATASETS_OFFLINE']\n", + "if 'TRANSFORMERS_OFFLINE' in os.environ: del os.environ['TRANSFORMERS_OFFLINE']\n", + "\n", + "# 设置随机种子\n", + "mindspore.set_seed(42)\n", + "print(\"环境配置完成\")" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## 3. 数据集加载与预处理\n", + "\n", + "我们将使用 **EuroSAT** 数据集,这是一个基于 Sentinel-2 卫星图像的土地利用和土地覆盖分类数据集,包含 10 个类别(如森林、河流、高速公路等)。由于 EuroSAT 只有 'train' 分割,我们需要将其划分为训练集和验证集。同时,我们需要构建标签 ID 到标签名称的映射字典。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "from mindnlp.dataset import load_dataset\n", + "# 加载 EuroSAT 数据\n", + "dataset = load_dataset(\"jonathan-roberts1/EuroSAT\", split=\"train\")\n", + "\n", + "# 定义标签映射\n", + "labels = [\"AnnualCrop\", \"Forest\", \"HerbaceousVegetation\", \"Highway\", \"Industrial\", \n", + " \"Pasture\", \"PermanentCrop\", \"Residential\", \"River\", \"SeaLake\"]\n", + "\n", + "label2id = {label: i for i, label in enumerate(labels)}\n", + "id2label = {i: label for i, label in enumerate(labels)}\n", + "\n", + "# 切分数据集\n", + "train_dataset, eval_dataset = dataset.split([0.9, 0.1])" + ] + }, + { + "cell_type": "markdown", + "id": "7", + "metadata": {}, + "source": [ + "### 图像预处理\n", + "\n", + "为了适配 Swin Transformer 模型的输入要求并提升模型泛化能力,我们需要对数据进行特定的预处理。\n", + "\n", + "我们使用 `AutoImageProcessor` 自动加载模型对应的配置(如均值、方差、输入尺寸),并定义了两套不同的处理管道:\n", + "\n", + "* **训练集 (Training)**: 引入随机增强策略(随机裁剪、水平翻转),增加数据多样性,防止过拟合。\n", + "* **验证集 (Evaluation)**: 仅进行确定性的调整大小和中心裁剪,确保评估结果的一致性。\n", + "* **通用处理**: 包括像素归一化 (Normalize) 和维度转换 (HWC 转 CHW)。\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "import mindspore.dataset.vision as vision\n", + "import mindspore.dataset.transforms as transforms\n", + "from mindnlp.transformers import AutoImageProcessor\n", + "\n", + "model_checkpoint = \"microsoft/swin-tiny-patch4-window7-224\"\n", + "image_processor = AutoImageProcessor.from_pretrained(model_checkpoint)\n", + "\n", + "\n", + "# 自动获取模型所需的输入尺寸 (适配不同模型配置)\n", + "if \"shortest_edge\" in image_processor.size:\n", + " size = image_processor.size[\"shortest_edge\"]\n", + " crop_size = (size, size)\n", + "elif \"height\" in image_processor.size:\n", + " size = (image_processor.size[\"height\"], image_processor.size[\"width\"])\n", + " crop_size = size\n", + " \n", + "print(f\"Image Processor Size Configuration: Resize={size}, Crop={crop_size}\")\n", + "\n", + "# 训练集增强策略\n", + "def get_train_transforms():\n", + " \"\"\"\n", + " 1. RandomResizedCrop: 随机裁剪并缩放 (让模型学会看局部)\n", + " 2. RandomHorizontalFlip: 随机水平翻转 (让模型学会左右不变性)\n", + " 3. Rescale + Normalize: 归一化 (对应 ToTensor + Normalize)\n", + " \"\"\"\n", + " return transforms.Compose([\n", + " vision.RandomResizedCrop(crop_size, scale=(0.08, 1.0), ratio=(0.75, 1.333)),\n", + " vision.RandomHorizontalFlip(prob=0.5),\n", + " vision.Rescale(1.0 / 255.0, 0.0), \n", + " vision.Normalize(mean=image_processor.image_mean, std=image_processor.image_std),\n", + " vision.HWC2CHW()\n", + " ])\n", + " \n", + "# 验证集增强策略\n", + "def get_eval_transforms():\n", + " \"\"\"\n", + " 1. Resize: 调整大小\n", + " 2. CenterCrop: 中心裁剪 (保证每次评估输入一致)\n", + " 3. Rescale + Normalize: 归一化\n", + " \"\"\"\n", + " return transforms.Compose([\n", + " vision.Resize(size),\n", + " vision.CenterCrop(crop_size),\n", + " vision.Rescale(1.0 / 255.0, 0.0),\n", + " vision.Normalize(mean=image_processor.image_mean, std=image_processor.image_std),\n", + " vision.HWC2CHW()\n", + " ])\n", + "\n", + "# 定义两个处理函数\n", + "def train_transform_func(image, label):\n", + " trans = get_train_transforms()\n", + " return trans(image), label\n", + "\n", + "def eval_transform_func(image, label):\n", + " trans = get_eval_transforms()\n", + " return trans(image), label\n", + "\n", + "# 训练集应用增强策略\n", + "train_dataset = train_dataset.map(\n", + " operations=train_transform_func, \n", + " input_columns=[\"image\", \"label\"], \n", + " output_columns=[\"pixel_values\", \"labels\"]\n", + ")\n", + "\n", + "# 验证集应用确定性策略\n", + "eval_dataset = eval_dataset.map(\n", + " operations=eval_transform_func, \n", + " input_columns=[\"image\", \"label\"], \n", + " output_columns=[\"pixel_values\", \"labels\"]\n", + ") " + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "### 数据集内存化适配\n", + "\n", + "为了解决 `Trainer` 在处理流式数据集(IterableDataset)时可能出现的兼容性问题(如 `not subscriptable` 或 `datapipes` 缺失报错),同时充分利用服务器的大内存优势(240GB RAM),我们将处理后的数据集一次性加载到内存列表中。\n", + "\n", + "**核心优势:**\n", + "* **兼容性**:将 MindSpore 数据集转换为 Python `List`,使其天然支持索引访问(`__getitem__`),完美适配 Trainer 的数据分发逻辑。\n", + "* **训练加速**:避免了训练过程中频繁的磁盘 I/O 读取和实时预处理,数据直接从内存获取,极大提升训练吞吐量。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Converting datasets to memory lists (Compatibility Bridge)...\")\n", + "\n", + "def ms_dataset_to_list(ms_dataset):\n", + " data_list = []\n", + " iterator = ms_dataset.create_dict_iterator(num_epochs=1, output_numpy=True)\n", + " for item in iterator:\n", + " data_list.append(item)\n", + " return data_list\n", + "\n", + "# 执行转换\n", + "train_dataset_list = ms_dataset_to_list(train_dataset)\n", + "eval_dataset_list = ms_dataset_to_list(eval_dataset)\n", + "print(f\"Converted Train Size: {len(train_dataset_list)}\")\n", + "print(f\"Converted Eval Size: {len(eval_dataset_list)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "11", + "metadata": {}, + "source": [ + "## 4. 模型构建\n", + "\n", + "使用 `AutoModelForImageClassification` 加载预训练的 Swin-Tiny 模型。\n", + "- `num_labels`: 设置为 10(EuroSAT 的类别数)。\n", + "- `ignore_mismatched_sizes=True`: 允许加载时忽略分类头尺寸不匹配的问题(因为预训练模型是 1000 类 ImageNet,我们需要替换为 10 类),使用随机初始化的分类头。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12", + "metadata": {}, + "outputs": [], + "source": [ + "from mindnlp.transformers import AutoModelForImageClassification\n", + "\n", + "model = AutoModelForImageClassification.from_pretrained(\n", + " model_checkpoint,\n", + " num_labels=len(labels),\n", + " label2id=label2id,\n", + " id2label=id2label,\n", + " ignore_mismatched_sizes=True\n", + ")\n", + "\n", + "print(\"Model loaded successfully.\")" + ] + }, + { + "cell_type": "markdown", + "id": "13", + "metadata": {}, + "source": [ + "## 5. 模型训练\n", + "\n", + "使用 MindSpore NLP 提供的 `Trainer` 接口进行训练。我们需要定义 `TrainingArguments` 来配置训练参数(学习率、Epoch 数、保存策略等)。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "14", + "metadata": {}, + "outputs": [], + "source": [ + "from mindnlp.transformers import Trainer, TrainingArguments\n", + "import evaluate\n", + "import numpy as np\n", + "\n", + "# 加载评估指标\n", + "metric = evaluate.load(\"accuracy\")\n", + "\n", + "# 定义评估函数\n", + "def compute_metrics(eval_pred):\n", + " logits, labels = eval_pred\n", + " predictions = np.argmax(logits, axis=-1)\n", + " return metric.compute(predictions=predictions, references=labels)\n", + "\n", + "BATCH_SIZE = 32\n", + "# 配置训练参数\n", + "training_args = TrainingArguments(\n", + " output_dir=\"swin_tiny_eurosat_finetune\", # 输出目录\n", + " eval_strategy=\"epoch\", # 每个 epoch 评估一次\n", + " save_strategy=\"epoch\", # 每个 epoch 保存一次\n", + " learning_rate=5e-5, # 学习率\n", + " per_device_train_batch_size=BATCH_SIZE, # 训练 Batch Size\n", + " per_device_eval_batch_size=BATCH_SIZE, # 评估 Batch Size\n", + " num_train_epochs=1, # 训练轮次\n", + " weight_decay=0.01, # 权重衰减\n", + " load_best_model_at_end=True, # 训练结束后加载最佳模型\n", + " metric_for_best_model=\"accuracy\", # 以准确率作为最佳模型指标\n", + " logging_steps=10, # 每 10 步打印日志\n", + " save_total_limit=2 # 最多保存 2 个 Checkpoint\n", + ")\n", + "\n", + "# 初始化 Trainer\n", + "trainer = Trainer(\n", + " model=model,\n", + " args=training_args,\n", + " train_dataset=train_dataset_list,\n", + " eval_dataset=eval_dataset_list,\n", + " compute_metrics=compute_metrics,\n", + ")\n", + "\n", + "# 开始训练\n", + "trainer.train()" + ] + }, + { + "cell_type": "markdown", + "id": "15", + "metadata": {}, + "source": [ + "## 6. 模型评估与保存\n", + "\n", + "训练完成后,我们在验证集上进行最终评估,并保存微调后的模型。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "16", + "metadata": {}, + "outputs": [], + "source": [ + "# 在验证集上评估\n", + "eval_metrics = trainer.evaluate()\n", + "print(f\"Evaluation Metrics: {eval_metrics}\")\n", + "\n", + "# 保存最终模型\n", + "trainer.save_model(\"output/swin-tiny-eurosat\")\n", + "print(\"Model saved to output/swin-tiny-eurosat\")" + ] + }, + { + "cell_type": "markdown", + "id": "17", + "metadata": {}, + "source": [ + "## 7. 推理演示\n", + "\n", + "使用微调后的模型对一张新的网络图片进行分类预测。我们将展示如何使用 Pipeline API 快速进行推理。" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "18", + "metadata": {}, + "outputs": [], + "source": [ + "from PIL import Image\n", + "import requests\n", + "from mindnlp.transformers import pipeline\n", + "\n", + "\n", + "# 下载一张测试图片(不在训练集中)\n", + "url = 'https://huggingface.co/nielsr/convnext-tiny-finetuned-eurostat/resolve/main/forest.png'\n", + "try:\n", + " image = Image.open(requests.get(url, stream=True).raw)\n", + "except:\n", + " # 如果网络无法访问,创建一个随机图片用于演示代码跑通\n", + " print(\"Network error, using dummy image.\")\n", + " image = Image.new('RGB', (224, 224), color='green')\n", + "\n", + "print(f\"Test Image Info: {image.format}, Size: {image.size}\")\n", + "\n", + "# 创建推理 Pipeline\n", + "# task: 图像分类\n", + "# model: 刚才训练好的模型对象\n", + "# image_processor: 预处理配置\n", + "pipe = pipeline(\"image-classification\", model=model, image_processor=image_processor)\n", + "\n", + "# 执行预测\n", + "result = pipe(image)\n", + "\n", + "# 打印结果\n", + "print(\"\\nPrediction Result:\")\n", + "for res in result:\n", + " print(f\"Label: {res['label']}, Score: {res['score']:.4f}\")\n", + "\n", + "print(f\"Test Image Info: {image.format}, Size: {image.size}\") " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python (Ascend-Final)", + "language": "python", + "name": "my_analysis" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.19" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}