Little bear paddle exercise book-02 eye disease identification

Time:2022-5-23

Little bear paddle exercise book-02 eye disease identification

brief introduction

The development and testing of this project are carried out under Ubuntu 20.04 system.
View the latest code of the project home page:Little bear paddle exercise book
Home page of Baidu PaddlePaddle AI Studio:Little bear paddle exercise book-02 eye disease identification
CUDA reference for Ubuntu system installation:Installation of Ubuntu Baidu PaddlePaddle and CUDA

Document description

file explain
train.py Training procedure
test.py Test procedure
test-gtk.py Test program GTK interface
report.py Report program
get-data.sh Get data to dataset directory
check-data.sh Check whether the data in the dataset directory exists
mod/alexnet.py Alexnet network model
mod/dataset.py Analysis of imageCLASS image classification data set
mod/utils.py miscellaneous
mod/config.py to configure
mod/report.py Result report
dataset Dataset directory
params Model parameter saving directory
log Visualdl log save directory

data set

The data set comes from the public data set of Baidu PaddlePaddle:Eye disease identification data set ichallenge – finishing version

get data

If it is running on the local computer, after downloading the data, the file will be placed in thedatasetUnder the project directory, run the following script under the project directory.
If running in BaiduAI StudioEnvironment, viewingdataWhether the directory has data, run the following script under the project directory.

bash get-data.sh

Classification label

  • Non eye disease 0
  • Eye disease 1

Check data

After obtaining the data, run the following script in the project directory to check whether the data in the dataset directory exists.

bash check-data.sh

network model

Network model usageAlexnet network modelSource: Baidu PaddlePaddle tutorial and network.
Alexnet network modelreference resources:Baidu PaddlePaddle tutorial

import paddle
import paddle.nn as nn
import paddle.nn.functional as F


#Alexnet network model
class AlexNet(nn.Layer):
    """
    Alexnet network model

    The input image size is 224 x 224
    Pool layer kernel_ Size = 2, the first layer of convolution filling padding = 2
    """
    def __init__(self, num_classes=10, pool_kernel_size=2, conv1_paddling=2, fc1_in_features=9216):
        """
        Alexnet network model

        Args:
            num_ Classes (int, optional): number of classes. The default is 10
            pool_ kernel_ Size (int, optional): pool layer core size, default 2
            conv1_ Padding (int, optional): the first layer is filled by convolution layer. The default is 2,
                The input image size is 224 x 224 填充 2
            fc1_ in_ Features (int, optional): input the number of features in the first full connection layer. The default is 9216, 
                According to Max_ Pool3 output results, calculated 256 * 6 * 6 = 9216

        Raises:
            Exception: classification quantity num_ Classes must be greater than or equal to 2
        """        
        super(AlexNet, self).__init__()
        if num_classes < 2:
            Raise exception ("the number of categories num_classes must be greater than or equal to 2: {}". Format (num_classes))
        self.num_classes = num_classes
        self.pool_kernel_size = pool_kernel_size
        self.fc1_in_features = fc1_in_features
        self.conv1 = nn.Conv2D(
            in_channels=3, out_channels=96, kernel_size=11, stride=4, padding=conv1_paddling)
        self.max_pool1 = nn.MaxPool2D(kernel_size=pool_kernel_size, stride=2)
        self.conv2 = nn.Conv2D(
            in_channels=96, out_channels=256, kernel_size=5, stride=1, padding=2)
        self.max_pool2 = nn.MaxPool2D(kernel_size=pool_kernel_size, stride=2)
        self.conv3 = nn.Conv2D(
            in_channels=256, out_channels=384, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2D(
            in_channels=384, out_channels=384, kernel_size=3, stride=1, padding=1)
        self.conv5 = nn.Conv2D(
            in_channels=384, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.max_pool3 = nn.MaxPool2D(kernel_size=pool_kernel_size, stride=2)
        # in_ features 9216 = max_ Pool3 output 256 * 6 * 6
        self.fc1 = nn.Linear(in_features=fc1_in_features, out_features=4096)
        self.drop_ratio1 = 0.5
        self.drop1 = nn.Dropout(self.drop_ratio1)
        self.fc2 = nn.Linear(in_features=4096, out_features=4096)
        self.drop_ratio2 = 0.5
        self.drop2 = nn.Dropout(self.drop_ratio2)
        self.fc3 = nn.Linear(in_features=4096, out_features=num_classes)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.max_pool1(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.max_pool2(x)
        x = self.conv3(x)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.relu(x)
        x = self.conv5(x)
        x = F.relu(x)
        x = self.max_pool3(x)
        #Flatten according to the given start_ Axis and stop_ Axis flattens successive dimensions
        x = paddle.flatten(x, start_axis=1, stop_axis=-1)
        x = self.fc1(x)
        x = F.relu(x)
        #Use dropout to suppress overfitting after full connection
        x = self.drop1(x)
        x = self.fc2(x)
        x = F.relu(x)
        #Use dropout to suppress overfitting after full connection
        x = self.drop2(x)
        x = self.fc3(x)
        return x

Data set analysis

Data set analysis, mainly analysisImage path and label textThen read the image and label according to the image path.

import paddle
import os
import random
import numpy as np
from PIL import Image
import paddle.vision as ppvs


class ImageClass(paddle.io.Dataset):
    """
    ImageCLASS image classification data set analysis, inheriting padding io. Dataset class
    """

    def __init__(self,
                 dataset_path: str,
                 images_labels_txt_path: str,
                 transform=None,
                 shuffle=True
                 ):
        """
        Constructor, defining data sets

        Args:
            dataset_ Path (STR): data set path
            images_ labels_ txt_ Path (STR): text path of images and labels
            Transform (compose, optional): operation combination for converting data. The default is none
            Shuffle (bool, true): randomly disrupt data. The default is true
        """

        super(ImageClass, self).__init__()
        self.dataset_path = dataset_path
        self.images_labels_txt_path = images_labels_txt_path
        self._ check_ Path (dataset_path, "dataset path error")
        self._ check_ Path (images_labels_txt_path, "wrong text path for images and labels")
        self.transform = transform
        self.image_paths, self.labels = self.parse_dataset(
            dataset_path, images_labels_txt_path, shuffle)

    def __getitem__(self, idx):
        """
        Get individual data and labels

        Args:
            IDX (any): index

        Returns:
            Image (float32): image
            Label (int): Label
        """
        image_path, label = self.image_paths[idx], self.labels[idx]
        return self.get_item(image_path, label, self.transform)

    @staticmethod
    def get_item(image_path: str, label: int, transform=None):
        """
        Get individual data and labels

        Args:
            image_ Path (STR): image path
            Label (int): Label
            Transform (compose, optional): operation combination for converting data. The default is none

        Returns:
            Image (float32): image
            Label (int): Label
        """
        ppvs.set_image_backend("pil")
        image = Image.open(image_path)
        if transform is not None:
            image = transform(image)
        #Convert image HWC to CHW
        image = np.transpose(image, (2, 0, 1))
        return image.astype("float32"), label

    def __len__(self):
        """
        Data quantity

        Returns:
            int: Data quantity
        """
        return len(self.labels)

    def _check_path(self, path: str, msg: str):
        """
        Check whether the path exists

        Args:
            Path (STR): Path
            MSG (STR, optional): exception message

        Raises:
            Exception: path error, exception
        """
        if not os.path.exists(path):
            raise Exception("{}: {}".format(msg, path))

    @staticmethod
    def parse_dataset(dataset_path: str, images_labels_txt_path: str, shuffle: bool):
        """
        Data set analysis

        Args:
            dataset_ Path (STR): data set path
            images_ labels_ txt_ Path (STR): text path of images and labels

        Returns:
            image_ Paths: image path set
            Labels: classification label set
        """
        lines = []
        image_paths = []
        labels = []
        with open(images_labels_txt_path, "r") as f:
            lines = f.readlines()
        #Scramble random data
        if (shuffle):
            random.shuffle(lines)
        for i in lines:
            data = i.split(" ")
            image_paths.append(os.path.join(dataset_path, data[0]))
            labels.append(int(data[1]))
        return image_paths, labels

Configuration module

You can view and modifymod/config.pyDocument with detailed description

Start training

functiontrain.pyFile, view the command line parameter plus – H

python3 train.py
--Whether CPU uses CPU for calculation. CUDA is used by default
  --Learning rate, default 0.001
  --Epochs training rounds, 2 rounds by default
  --Batch size batch quantity, default 2
  --Default number of threads, num-2
  --No save whether to save the model parameters. It is saved by default. After selection, the model parameters are not saved
  --Load dir reads model parameters and subfolders under params directory. It does not read by default
  --Log whether to output visualdl log. It is not output by default
  --Summary outputs network model information, which is not output by default. After selection, only information is output and training will not be started

test model

functiontest.pyFile, view the command line parameter plus – H

python3 test.py
--Whether CPU uses CPU for calculation. CUDA is used by default
  --Batch size batch quantity, default 2
  --Number of num workers threads, default 2
  --Load dir reads model parameters, reads subfolders under params directory, and defaults to best directory

Test model GTK interface

functiontest-gtk.pyFile, this program relies on GTK library and can only run on the local computer.

python3 test-gtk.py

GTK library installation

python3 -m pip install pygobject

User manual

  • 1. ClickSelect modelButton.
  • 2. In the pop-up file dialog box, select the model. The model is displayed in theparamsOf subdirectories under directorymodel.pdparamsDocuments.
  • 3. ClickRandom testButton, you can see the test image, prediction results and actual results.

View results report

functionreport.pyFile, can displayparamsOf all subdirectories under the directoryreport.json
Add parameter–bestaccording tolossThe smallest model parameters are saved inbestUnder the subdirectory.

python3 report.py

report. JSON description

Key name explain
id String ID generated by time
loss Loss value of this training
acc ACC value of this training
epochs Epochs value of this training
batch_size Batch of this training_ Size value
learning_rate Learning in this training_ Rate value

Visualdl visual analysis tool

  • Installation and use instructions refer to:VisualDL
  • Add parameters during training–log
  • If it isAI StudioEnvironmental traininglogDownload the directory, unzip it and put it under the local project directorylogcatalogue
  • Run the following command in the project directory
  • Then, according to the prompted URL, open the browser to access the prompted URL
visualdl --logdir ./log

Recommended Today

Ansible combat MySQL installation

Do the preparatory work first, prepare a client, and then equip a master machine. Check connectivity: [[email protected] ~]# ansible mytest -m ping 10.128.25.175| SUCCESS => { “ansible_facts”: { “discovered_interpreter_python”: “/usr/bin/python” }, “changed”: false, “ping”: “pong” } [[email protected] ~]# 1. Ansible module search method 1. Ansible’s yum module search yum module online documentation: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yum_module.html Example: – […]