.. _sec_semantic_segmentation:
语义分割和数据集
================
在 :numref:`sec_bbox`—
:numref:`sec_rcnn`\ 中讨论的目标检测问题中，我们一直使用方形边界框来标注和预测图像中的目标。
本节将探讨\ *语义分割*\ （semantic
segmentation）问题，它重点关注于如何将图像分割成属于不同语义类别的区域。
与目标检测不同，语义分割可以识别并理解图像中每一个像素的内容：其语义区域的标注和预测是像素级的。
:numref:`fig_segmentation`\ 展示了语义分割中图像有关狗、猫和背景的标签。
与目标检测相比，语义分割标注的像素级的边框显然更加精细。
.. _fig_segmentation:
.. figure:: ../img/segmentation.svg
语义分割中图像有关狗、猫和背景的标签
图像分割和实例分割
------------------
计算机视觉领域还有2个与语义分割相似的重要问题，即\ *图像分割*\ （image
segmentation）和\ *实例分割*\ （instance segmentation）。
我们在这里将它们同语义分割简单区分一下。
- *图像分割*\ 将图像划分为若干组成区域，这类问题的方法通常利用图像中像素之间的相关性。它在训练时不需要有关图像像素的标签信息，在预测时也无法保证分割出的区域具有我们希望得到的语义。以
:numref:`fig_segmentation`\ 中的图像作为输入，图像分割可能会将狗分为两个区域：一个覆盖以黑色为主的嘴和眼睛，另一个覆盖以黄色为主的其余部分身体。
- *实例分割*\ 也叫\ *同时检测并分割*\ （simultaneous detection and
segmentation），它研究如何识别图像中各个目标实例的像素级区域。与语义分割不同，实例分割不仅需要区分语义，还要区分不同的目标实例。例如，如果图像中有两条狗，则实例分割需要区分像素属于的两条狗中的哪一条。
Pascal VOC2012 语义分割数据集
-----------------------------
最重要的语义分割数据集之一是\ `Pascal
VOC2012 `__\ 。
下面我们深入了解一下这个数据集。
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
%matplotlib inline
import os
import mindspore
import numpy as np
from PIL import Image
from d2l import mindspore as d2l
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
%matplotlib inline
import os
import torch
import torchvision
from d2l import torch as d2l
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def read_voc_images(voc_dir, is_train=True):
"""读取VOC数据集图像和标注的文件名列表"""
txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',
'train.txt' if is_train else 'val.txt')
with open(txt_fname, 'r') as f:
names = f.read().split()
image_names = names
label_names = names
return image_names, label_names
train_features, train_labels = read_voc_images(voc_dir, True)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def read_voc_images(voc_dir, is_train=True):
"""读取所有VOC图像并标注"""
txt_fname = os.path.join(voc_dir, 'ImageSets', 'Segmentation',
'train.txt' if is_train else 'val.txt')
mode = torchvision.io.image.ImageReadMode.RGB
with open(txt_fname, 'r') as f:
images = f.read().split()
features, labels = [], []
for i, fname in enumerate(images):
features.append(torchvision.io.read_image(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg')))
labels.append(torchvision.io.read_image(os.path.join(
voc_dir, 'SegmentationClass' ,f'{fname}.png'), mode))
return features, labels
train_features, train_labels = read_voc_images(voc_dir, True)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
n = 5
features, labels = [], []
for fname in train_features[:n]:
features.append(mindspore.dataset.vision.read_image(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg')))
labels.append(mindspore.dataset.vision.read_image(os.path.join(
voc_dir, 'SegmentationClass', f'{fname}.png')))
imgs = features + labels
d2l.show_images(imgs, 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_23_0.png
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
n = 5
imgs = train_features[0:n] + train_labels[0:n]
imgs = [img.permute(1,2,0) for img in imgs]
d2l.show_images(imgs, 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_26_0.png
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def voc_colormap2label():
"""构建从RGB到VOC类别索引的映射"""
colormap2label = np.zeros(256 ** 3, dtype=np.int64)
for i, colormap in enumerate(VOC_COLORMAP):
colormap2label[
(colormap[0] * 256 + colormap[1]) * 256 + colormap[2]] = i
return colormap2label
#@save
def voc_label_indices(colormap, colormap2label):
"""将VOC标签中的RGB值映射到它们的类别索引"""
colormap = colormap.astype('int32')
if colormap.ndim == 3:
idx = ((colormap[:, :, 0] * 256 + colormap[:, :, 1]) * 256
+ colormap[:, :, 2])
result = colormap2label[idx]
# 处理 void 像素：RGB 值为 255 的情况
void_mask = (colormap[:, :, 0] == 255) & (colormap[:, :, 1] == 255) & (colormap[:, :, 2] == 255)
result[void_mask] = 0
return result
else:
# 单通道索引图，void(255) 映射为 0（background）
result = colormap.copy()
result[result == 255] = 0
return result
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def voc_colormap2label():
"""构建从RGB到VOC类别索引的映射"""
colormap2label = torch.zeros(256 ** 3, dtype=torch.long)
for i, colormap in enumerate(VOC_COLORMAP):
colormap2label[
(colormap[0] * 256 + colormap[1]) * 256 + colormap[2]] = i
return colormap2label
#@save
def voc_label_indices(colormap, colormap2label):
"""将VOC标签中的RGB值映射到它们的类别索引"""
colormap = colormap.permute(1, 2, 0).numpy().astype('int32')
idx = ((colormap[:, :, 0] * 256 + colormap[:, :, 1]) * 256
+ colormap[:, :, 2])
return colormap2label[idx]
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
test_label = mindspore.dataset.vision.read_image(os.path.join(
voc_dir, 'SegmentationClass', f'{train_labels[0]}.png'))
y = voc_label_indices(test_label, voc_colormap2label())
y[105:115, 130:140], VOC_CLASSES[1]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 1]]),
'aeroplane')
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
y = voc_label_indices(train_labels[0], voc_colormap2label())
y[105:115, 130:140], VOC_CLASSES[1]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 0, 0, 1, 1]]),
'aeroplane')
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def voc_rand_crop(feature, label, height, width):
"""随机裁剪特征和标签图像"""
h, w = feature.shape[:2]
top = np.random.randint(0, h - height + 1)
left = np.random.randint(0, w - width + 1)
feature = feature[top:top+height, left:left+width, :]
label = (label[top:top+height, left:left+width] if label.ndim == 2
else label[top:top+height, left:left+width, :])
return feature, label
imgs = []
for _ in range(n):
fname = train_features[0]
feature = mindspore.dataset.vision.read_image(os.path.join(
voc_dir, 'JPEGImages', f'{fname}.jpg'))
label = mindspore.dataset.vision.read_image(os.path.join(
voc_dir, 'SegmentationClass', f'{fname}.png'))
f, l = voc_rand_crop(feature, label, 200, 300)
imgs.extend([f, l])
d2l.show_images(imgs[::2] + imgs[1::2], 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_52_0.png
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def voc_rand_crop(feature, label, height, width):
"""随机裁剪特征和标签图像"""
rect = torchvision.transforms.RandomCrop.get_params(
feature, (height, width))
feature = torchvision.transforms.functional.crop(feature, *rect)
label = torchvision.transforms.functional.crop(label, *rect)
return feature, label
imgs = []
for _ in range(n):
imgs += voc_rand_crop(train_features[0], train_labels[0], 200, 300)
imgs = [img.permute(1, 2, 0) for img in imgs]
d2l.show_images(imgs[::2] + imgs[1::2], 2, n);
.. figure:: output_semantic-segmentation-and-dataset_23ff18_55_0.png
.. raw:: html
.. raw:: html
.. raw:: html
接下来，我们自定义了一个语义分割数据集类\ ``VOCSegDataset``\ 。
.. raw:: html
.. raw:: html
我们通过继承高级API提供的\ ``Dataset``\ 类，自定义了一个语义分割数据集类\ ``VOCSegDataset``\ 。
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
class VOCSegDataset():
"""延迟加载版VOC数据集"""
def __init__(self, is_train, crop_size, voc_dir):
self.crop_size = crop_size
self.voc_dir = voc_dir
self.transform = mindspore.dataset.vision.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
self.colormap2label = voc_colormap2label()
names, _ = read_voc_images(voc_dir, is_train)
self.fnames = self._filter(names)
print('read ' + str(len(self.fnames)) + ' examples')
def _filter(self, names):
valid = []
for fname in names:
img_path = os.path.join(self.voc_dir, 'JPEGImages', f'{fname}.jpg')
with Image.open(img_path) as img:
w, h = img.size
if h >= self.crop_size[0] and w >= self.crop_size[1]:
valid.append(fname)
return valid
def _load_image(self, fname):
return mindspore.dataset.vision.read_image(
os.path.join(self.voc_dir, 'JPEGImages', f'{fname}.jpg'))
def _load_label(self, fname):
lbl = Image.open(os.path.join(
self.voc_dir, 'SegmentationClass', f'{fname}.png'))
return np.array(lbl, dtype=np.uint8)
def normalize_image(self, img):
return self.transform(img.astype('float32') / 255)
def __getitem__(self, idx):
fname = self.fnames[idx]
feature = self._load_image(fname)
label = self._load_label(fname)
feature, label = voc_rand_crop(feature, label, *self.crop_size)
return (self.normalize_image(feature),
voc_label_indices(label, self.colormap2label))
def __len__(self):
return len(self.fnames)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
class VOCSegDataset(torch.utils.data.Dataset):
"""一个用于加载VOC数据集的自定义数据集"""
def __init__(self, is_train, crop_size, voc_dir):
self.transform = torchvision.transforms.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
self.crop_size = crop_size
features, labels = read_voc_images(voc_dir, is_train=is_train)
self.features = [self.normalize_image(feature)
for feature in self.filter(features)]
self.labels = self.filter(labels)
self.colormap2label = voc_colormap2label()
print('read ' + str(len(self.features)) + ' examples')
def normalize_image(self, img):
return self.transform(img.float() / 255)
def filter(self, imgs):
return [img for img in imgs if (
img.shape[1] >= self.crop_size[0] and
img.shape[2] >= self.crop_size[1])]
def __getitem__(self, idx):
feature, label = voc_rand_crop(self.features[idx], self.labels[idx],
*self.crop_size)
return (feature, voc_label_indices(label, self.colormap2label))
def __len__(self):
return len(self.features)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
crop_size = (320, 480)
voc_train = VOCSegDataset(True, crop_size, voc_dir)
voc_test = VOCSegDataset(False, crop_size, voc_dir)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
read 1114 examples
read 1078 examples
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
crop_size = (320, 480)
voc_train = VOCSegDataset(True, crop_size, voc_dir)
voc_test = VOCSegDataset(False, crop_size, voc_dir)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
read 1114 examples
read 1078 examples
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
batch_size = 64
train_iter = mindspore.dataset.GeneratorDataset(voc_train, column_names=["data", "label"], shuffle=True)
train_iter = train_iter.map(mindspore.dataset.vision.HWC2CHW())
train_iter = train_iter.batch(batch_size=batch_size, drop_remainder=True)
for X, Y in train_iter:
print(X.shape)
print(Y.shape)
break
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(64, 3, 320, 480)
(64, 320, 480)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
batch_size = 64
train_iter = torch.utils.data.DataLoader(voc_train, batch_size, shuffle=True,
drop_last=True,
num_workers=d2l.get_dataloader_workers())
for X, Y in train_iter:
print(X.shape)
print(Y.shape)
break
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
torch.Size([64, 3, 320, 480])
torch.Size([64, 320, 480])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def load_data_voc(batch_size, crop_size):
"""加载VOC语义分割数据集"""
voc_dir = d2l.download_extract('voc2012', os.path.join(
'VOCdevkit', 'VOC2012'))
train_iter = mindspore.dataset.GeneratorDataset(
VOCSegDataset(True, crop_size, voc_dir),
column_names=['data', 'label'],
shuffle=True)
train_iter = train_iter.map(mindspore.dataset.vision.HWC2CHW())
train_iter = train_iter.batch(batch_size=batch_size, drop_remainder=True)
test_iter = mindspore.dataset.GeneratorDataset(
VOCSegDataset(False, crop_size, voc_dir),
column_names=['data', 'label'],
shuffle=False)
test_iter = test_iter.map(mindspore.dataset.vision.HWC2CHW())
test_iter = test_iter.batch(batch_size=batch_size, drop_remainder=True)
return train_iter, test_iter
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
#@save
def load_data_voc(batch_size, crop_size):
"""加载VOC语义分割数据集"""
voc_dir = d2l.download_extract('voc2012', os.path.join(
'VOCdevkit', 'VOC2012'))
num_workers = d2l.get_dataloader_workers()
train_iter = torch.utils.data.DataLoader(
VOCSegDataset(True, crop_size, voc_dir), batch_size,
shuffle=True, drop_last=True, num_workers=num_workers)
test_iter = torch.utils.data.DataLoader(
VOCSegDataset(False, crop_size, voc_dir), batch_size,
drop_last=True, num_workers=num_workers)
return train_iter, test_iter
.. raw:: html
.. raw:: html
.. raw:: html
`讨论 `__
.. raw:: html
.. raw:: html
`讨论 `__
.. raw:: html
.. raw:: html