Learn modelarts from me explore the personalized federated learning API of modelarts platform

Time:2021-4-8

Abstract:Modelarts provides an API for personalized federated learning — python_ fedamp_ emnist_ Classification is a horizontal federal learning framework for customers with similar data distribution to cooperate more. Let’s learn and explore it.

This article is shared from Huawei cloud communityModelarts “pytorch”, one of the federal learning algorithms_ fedamp_ emnist_ “Classification” learning (1),Author: darkpard.

With the development of digital technology and the continuous attention of the whole society to digitization, the resource attribute of data has been highlighted unprecedentedly. Accordingly, data privacy and data security have attracted more and more attention. Federated learning emerges as the times require and becomes a popular AI algorithm development direction.

What is federal learning:

Federated learning is an emerging basic technology of artificial intelligence, which was introduced in 2016 First proposed by Google in, it was originally used to solve the problem of Android mobile terminal users updating the model locally. Its design goal is to carry out efficient machine learning among multi participants or multi computing nodes on the premise of ensuring the information security of big data exchange, protecting the privacy of terminal data and personal data, and ensuring legal compliance.

Prospect of financial application field:

At present, in the financial field, each financial institution will build its own risk control model based on its own business scenarios. When federal learning is used, a joint model can be established based on its own risk control model, which can more accurately identify credit risk and financial fraud. At the same time, the federal learning model can solve the problems of less samples and low quality of the original data.

How to combat Federation learning:

Modelarts provides an API for personalized federated learning — python_ fedamp_ emnist_ Classification is a horizontal federal learning framework for customers with similar data distribution to cooperate more. Let’s learn and explore it.

1. Environmental preparation


1.1. Import file operation module and output cleaning module

import os
import shutil
from IPython.display import clear_output

1.2. Download federal learning package and clear output

!wget https://obsfs-notebook.obs.cn-north-4.myhuaweicloud.com/FedAMP.zip 
clear_output(wait=False)

1.3. If there is a fedamp folder, delete it completely, and then recreate the fedamp folder

if os.path.exists('FedAMP'):
    shutil.rmtree('FedAMP')
!mkdir FedAMP

1.4. Unzip the downloaded federal learning package to this folder, delete the compressed package, and clean up the output

!unzip FedAMP.zip -d FedAMP
!rm FedAMP.zip
clear_output(wait=False)

1.5. Install torch vision, an image processing module based on python, and clean up the output

!pip install torchvision==0.5.0
clear_output(wait=False)

1.6. Install torch framework and clean up output

!pip install torch==1.4.0
clear_output(wait=False)

1.7. Install the federal learning package, delete the original file, and clean up the output file

!pip install FedAMP/package/moxing_pytorch-1.17.3.fed-cp36-cp36m-linux_x86_64.whl
!rm -r FedAMP/package
clear_output(wait=False)

1.8. Import torch framework, numpy, random matplotlib.pyplot Huawei Mobile

import torch, random
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import moxing as mox

1.9. Importing federated algorithms, support and services under Huawei Moxing framework

from moxing.framework.federated import fed_algorithm
from moxing.framework.federated import fed_backend
from moxing.framework.federated import fed_server

1.10. Import the load, save, hook and service of Federated learning under Huawei mobile torch framework

from moxing.pytorch.executor.federated.util import torch_load
from moxing.pytorch.executor.federated.util import torch_save
from moxing.pytorch.executor.federated.util import TorchFedHook
from moxing.pytorch.executor.federated import client

1.11. Import fedamp and torch.nn Function package under

from moxing.framework.federated.fed_algorithm import FedAMP
import torch.nn.functional as F

1.12. Prepare file path

if mox.file.is_directory('/tmp/fed_workspace/'):
    mox.file.remove('/tmp/fed_workspace/', recursive=True)

2. Create data read class and data structure class (the specific content will be explained later)

class DataFileLoaderHorizontal():
    def __init__(self, data=None, label=None):
        if data is None:
            self.data = data
        if label is None:
            self.label = label
    def getDataToTorch(self):
        return torch.FloatTensor(self.data), torch.FloatTensor(self.label)
    def load_file_binary(self, data_filename=None, label_filename=None):
        assert data_filename is not None
        assert label_filename is not None
        self.data = np.load(data_filename, allow_pickle=True)
        self.label = np.load(label_filename, allow_pickle=True)
        self.data, self.label = self.data.astype(float), self.label.astype(float)
class m_Data():
    def __init__(self):
        self.train_samples          = None
        self.train_labels           = None
        self.test_samples           = None
        self.train_samples          = None

3. Read the data into the corresponding virtual tenant

3.1. Setting the number of virtual tenants

num_clients = 62

3.2. Create a data reading class

df1 = DataFileLoaderHorizontal()

3.3. Initialize training set, test set file name and file extension

rain_sample_filename = 'FedAMP/EMNIST/client_train_samples_'
train_label_filename = 'FedAMP/EMNIST/client_train_labels_'
val_sample_filename = 'FedAMP/EMNIST/client_test_samples_'
val_label_filename = 'FedAMP/EMNIST/client_test_labels_'
filename_sx = '.npy'

3.4. Let’s explore the training data set

3.4.1. Import a sample set first
df1.load_file_binary(data_filename=train_sample_filename + str(1) + filename_sx,
                         label_filename=train_label_filename + str(1) + filename_sx)

The load of datafileloaderhorizontal class in “2.” is used here_ file_ Binary method, which first confirms that the incoming file name is not empty, then numpy’s load method loads the. NPY file, and finally uses the astype method to convert it to the float type

3.4.2. Let’s take a look at the independent variable
df1.data

array([[[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

…,

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.]],

…,

[[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

…,

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.]],

[[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

…,

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.],

[0., 0., 0., …, 0., 0., 0.]]])

df1.data[0]

array([[0. , 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. ],

[0. , 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. ],

………………

[0. , 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. ],

[0. , 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. , 0. , 0. ,

, 0. , 0. ]])

len(df1.data)

1000

As you can see, it is a relatively sparse three-dimensional array, composed of 1000 two-dimensional arrays

3.4.3. Let’s take a look at the tag set
df1.label

array([40., 7., 5., 4., 8., 8., 0., 9., 3., 34., 2., 2., 8.,

6., 5., 3., 9., 9., 6., 8., 5., 6., 6., 6., 6., 5.,

5., 8., 5., 6., 7., 5., 8., 59., 2., 9., 7., 6., 3.,

4., 57., 7., 9., 49., 52., 25., 4., 2., 43., 6., 9., 5.,

3., 5., 7., 7., 3., 0., 6., 7., 5., 27., 9., 24., 2.,

2., 7., 6., 1., 9., 45., 7., 0., 14., 9., 9., 0., 2.,

6., 5., 1., 4., 6., 36., 8., 0., 34., 0., 0., 53., 5.,

0., 2., 7., 52., 32., 2., 4., 35., 49., 15., 2., 60., 8.,

0., 7., 51., 19., 3., 1., 24., 9., 2., 2., 4., 8., 8.,

4., 3., 0., 9., 7., 6., 26., 7., 4., 7., 7., 2., 8.,

9., 4., 1., 4., 9., 8., 3., 16., 0., 5., 3., 16., 5.,

3., 1., 7., 19., 53., 4., 0., 2., 5., 23., 19., 46., 5.,

2., 7., 4., 51., 57., 7., 16., 2., 1., 0., 2., 4., 0.,

41., 21., 8., 1., 2., 39., 3., 1., 6., 58., 32., 3., 9.,

6., 4., 3., 54., 7., 3., 60., 7., 8., 3., 3., 2., 2.,

5., 60., 5., 5., 6., 7., 9., 9., 2., 8., 3., 43., 5.,

1., 9., 5., 9., 13., 6., 7., 6., 6., 59., 0., 8., 7.,

7., 2., 57., 4., 8., 3., 4., 6., 4., 3., 9., 8., 0.,

6., 48., 0., 4., 2., 3., 4., 8., 18., 2., 2., 4., 30.,

7., 2., 9., 7., 1., 1., 2., 20., 36., 9., 5., 32., 3.,

3., 3., 3., 3., 20., 37., 1., 25., 1., 0., 57., 2., 2.,

0., 3., 9., 2., 18., 2., 3., 40., 28., 1., 4., 2., 8.,

4., 8., 5., 0., 18., 0., 1., 2., 7., 8., 6., 0., 2.,

5., 35., 0., 1., 53., 2., 3., 3., 2., 8., 32., 3., 5.,

6., 8., 2., 7., 40., 8., 5., 6., 8., 4., 9., 1., 13.,

6., 3., 3., 5., 3., 51., 60., 2., 3., 40., 1., 0., 47.,

59., 9., 6., 1., 2., 1., 9., 8., 0., 3., 8., 53., 61.,

8., 5., 18., 7., 0., 4., 1., 1., 51., 0., 9., 43., 6.,

51., 5., 7., 22., 24., 42., 3., 47., 0., 59., 7., 42., 7.,

58., 7., 1., 0., 4., 8., 8., 8., 20., 1., 16., 9., 0.,

3., 23., 6., 4., 45., 5., 0., 1., 2., 9., 1., 27., 9.,

5., 4., 7., 7., 0., 15., 3., 9., 36., 9., 47., 3., 29.,

56., 42., 2., 7., 42., 4., 1., 9., 0., 34., 3., 5., 0.,

15., 0., 6., 4., 7., 4., 5., 0., 15., 9., 8., 43., 7.,

7., 6., 42., 6., 8., 7., 61., 2., 8., 1., 5., 7., 57.,

2., 23., 9., 4., 1., 59., 3., 1., 9., 9., 15., 5., 47.,

27., 6., 6., 0., 4., 2., 3., 2., 22., 3., 6., 2., 6.,

5., 8., 7., 9., 7., 3., 49., 5., 5., 1., 6., 8., 0.,

6., 7., 45., 4., 6., 3., 9., 5., 0., 12., 18., 8., 4.,

3., 4., 6., 6., 4., 5., 3., 29., 7., 7., 5., 9., 7.,

4., 0., 6., 8., 5., 2., 8., 1., 9., 8., 7., 25., 1.,

6., 8., 4., 9., 3., 1., 2., 9., 2., 5., 1., 9., 5.,

1., 2., 1., 5., 24., 45., 7., 0., 4., 8., 49., 9., 6.,

4., 2., 35., 4., 9., 8., 7., 8., 1., 6., 1., 7., 9.,

1., 8., 1., 1., 3., 0., 17., 47., 6., 0., 3., 2., 5.,

5., 55., 28., 9., 56., 7., 8., 2., 2., 50., 8., 4., 9.,

4., 3., 1., 1., 0., 5., 38., 8., 9., 0., 1., 5., 2.,

25., 5., 0., 4., 7., 9., 7., 61., 4., 4., 2., 2., 6.,

41., 45., 20., 5., 8., 5., 8., 7., 9., 4., 3., 1., 7.,

19., 3., 8., 1., 9., 7., 27., 3., 0., 4., 8., 8., 2.,

46., 6., 6., 5., 1., 8., 6., 8., 2., 4., 5., 33., 5.,

5., 5., 8., 0., 2., 31., 5., 1., 7., 1., 5., 48., 41.,

9., 4., 61., 9., 9., 34., 16., 7., 5., 0., 5., 32., 0.,

52., 3., 1., 4., 6., 29., 4., 2., 0., 4., 0., 1., 48.,

3., 9., 5., 1., 7., 6., 4., 4., 5., 8., 8., 9., 1.,

46., 0., 29., 0., 5., 4., 4., 48., 56., 9., 3., 1., 3.,

1., 5., 7., 9., 8., 8., 6., 6., 0., 8., 0., 53., 1.,

6., 1., 4., 4., 8., 11., 9., 8., 1., 44., 4., 2., 1.,

3., 7., 6., 2., 39., 8., 9., 4., 6., 4., 1., 2., 7.,

33., 4., 36., 3., 40., 1., 8., 5., 3., 3., 3., 28., 13.,

9., 1., 46., 1., 5., 22., 0., 9., 0., 0., 2., 1., 2.,

43., 7., 4., 0., 2., 28., 39., 48., 4., 0., 5., 3., 6.,

6., 7., 19., 6., 4., 0., 35., 13., 3., 28., 2., 6., 23.,

2., 5., 1., 0., 8., 8., 2., 10., 27., 0., 49., 58., 23.,

9., 2., 7., 7., 2., 9., 5., 4., 9., 22., 5., 8., 6.,

4., 58., 6., 5., 4., 9., 1., 7., 0., 3., 33., 3., 7.,

9., 6., 3., 1., 1., 6., 2., 1., 2., 7., 3., 7., 8.,

6., 0., 4., 34., 41., 8., 3., 6., 8., 6., 1., 6., 3.,

56., 24., 0., 0., 1., 58., 0., 1., 9., 29., 8., 9., 6.,

6., 8., 9., 1., 39., 3., 0., 4., 25., 8., 33., 0., 2.,

3., 7., 5., 0., 7., 7., 6., 46., 7., 8., 6., 2., 0.,

8., 7., 5., 20., 56., 9., 4., 41., 9., 8., 4., 13., 5.,

3., 61., 4., 5., 1., 33., 0., 1., 7., 1., 0., 6., 3.,

6., 2., 6., 4., 22., 5., 4., 36., 0., 9., 2., 9., 3.,

2., 0., 0., 7., 2., 35., 5., 9., 4., 4., 0., 6., 6.,

9., 5., 5., 39., 3., 1., 60., 4., 52., 6., 4., 0., 1.,

6., 9., 8., 52., 3., 1., 7., 3., 3., 9., 7., 8.])

It can be inferred that each training set consists of 1000 samples, and the independent variables are two-dimensional arrays

3.4.4. Transform sample set and tag set into torch.FloatTensor Type
samples, labels = df1.getDataToTorch()

3.5. Let’s create an m in “2.0”_ The training sample set and label set are imported into M_ Data¶

3.5.1. First create an M_ Data¶
m_data = m_Data()
3.5.2. Initialization input format
input_dim = (-1, 1, 28, 28)
3.5.3. Create M_ Data’s training set
m_data.train_samples = samples.reshape(input_dim)
m_data.train_labels = labels.squeeze()
3.5.4. Create M_ Test set of data
df1.load_file_binary(data_filename=val_sample_filename + str(1) + filename_sx,
                     label_filename=val_label_filename + str(1) + filename_sx)
samples, labels = df1.getDataToTorch()
m_data.val_samples = samples.reshape(input_dim)
m_data.val_labels = labels.squeeze()

Here, we compare M_ Data data structure, we can find that, M_ There seems to be a small bug in the data structure of data, although it does not affect the use of class M_ Data(): definit(self): self.train_ samples = None self.train_ labels = None self.test_ samples = None self.train_ Samples = none_ Samples should be val_ Samples, the last train_ Samples should be val_ labels

pytorch_ fedamp_ emnist_ Let’s learn here for the first time.

We have implemented pytorch on modelats_ fedamp_ emnist_ The environment configuration of classification, the sample data structure and the python_ fedamp_ emnist_ The data structure needed by classification is simply explored.

Click follow to learn about Huawei’s new cloud technology for the first time~

Recommended Today

Third party calls wechat payment interface

Step one: preparation 1. Wechat payment interface can only be called if the developer qualification has been authenticated on wechat open platform, so the first thing is to authenticate. It’s very simple, but wechat will charge 300 yuan for audit 2. Set payment directory Login wechat payment merchant platform( pay.weixin.qq . com) — > Product […]