Intel SGX 运行 resnet

2022-04-02 | 0 浏览

Intel SGX 运行 resnet

1.BIOS配置

1.Total Memory Encryption
Enable
2.Disable UMA-Based Clustering:
Disable
3.SW Guard Extensions (SGX)
Enable
4.SGX PRMRR Size
64GB
5.SGX Auto MP Registration Agent
Enable
6.Patrol Scrub
Disable
7.Mirror Mode
Disable
8.Secure Boot
Disable

2.安装系统更新内核

安装centos8.2
更新内核版本支持sgx

yum更新

#cd /etc/yum.repos.d/
#wget https://download.01.org/intelsgxstack/2021-12-08/rhel/intelsgxstack.repo
#dnf install -y intel-mvp-mainline-kernel-sgx-kvm
创建文件/etc/udev/rules.d/10-sgx.rules 添加如下内容：
SUBSYSTEM=="misc",KERNEL=="enclave",MODE="0666"
SUBSYSTEM=="misc",KERNEL=="provision",GROUP="sgx_prv",MODE="0660"
SUBSYSTEM=="misc",KERNEL=="sgx_enclave",MODE="0666",SYMLINK+="sgx/enclave"
SUBSYSTEM=="misc",KERNEL=="sgx_provision",GROUP="sgx_prv",MODE="0660",SYMLINK+="sgx/provision"

#groupadd sgx_prv
#udevadm trigger
#reboot
#uname -r
5.15.5-mvp2.el8.x86_64

手工编译

.config文件里记得要设置CONFIG_X86_SGX=y

参考建议

如果要在centos上跑demo的话建议把ubuntu上的avalon相关的镜像直接拷贝过来，centos上编译avalon镜像需要改些配置，为了方便可以直接拷过来导入

检查SGX驱动安装情况

ls -l /dev/{sgx_enclave,sgx_provision}

下载软件包

cd /opt/intel
wget https://download.01.org/intel-sgx/latest/linux-latest/distro/rhel8.2-server/sgx_rpm_local_repo.tgz

3.机器注册

MP注册

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install libsgx-ra-network libsgx-ra-uefi sgx-ra-service

正式机注册

#/opt/intel/sgx-ra-service/mpa_registration
#/opt/intel/sgx-ra-service/mpa_manage -h
Tools options:

-h               Shows usage instructions.
-v               Verbose logs.
-get_platform_manifest <file_name>       Copies Platform Manifest into a file. Sets registration status to completed.
               The users responsibility is to pass Platform Manifest to registration server.
-get_key_blobs <key_blobs_file_name>       Copies Package Info Key Blobs into a file. Sets package info status to completed.
-set_server_info <server_id_file_name> <hex_flags> <URL> Sets registration server configurations.
-get_registration_status         Prints and returns registration status.
-get_last_registration_error_code      Prints and returns last registration error code.
-get_sgx_status            Prints SGX status.

In case of a tool error a negative number will be returned.
A positive return value defines as MpResult.
#/opt/intel/sgx-ra-service/mpa_manage -get_last_registration_error_code
Last reported registration error code: 0

4.安装SGX DCAP

安装依赖软件

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install libsgx-urts libsgx-launch libsgx-epid libsgx-quote-ex libsgx-dcap-ql libsgx-uae-service
dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install libsgx-aesm-ecdsa-plugin-debuginfo libsgx-aesm-epid-plugin-debuginfo libsgx-aesm-launch-plugin-debuginfo libsgx-aesm-pce-plugin-debuginfo libsgx-aesm-quote-ex-plugin-debuginfo libsgx-dcap-default-qpl-debuginfo libsgx-dcap-ql-debuginfo libsgx-dcap-quote-verify-debuginfo libsgx-enclave-common-debuginfo libsgx-epid-debuginfo libsgx-launch-debuginfo libsgx-pce-logic-debuginfo libsgx-qe3-logic-debuginfo libsgx-quote-ex-debuginfo libsgx-ra-network-debuginfo libsgx-ra-uefi-debuginfo libsgx-uae-service-debuginfo libsgx-urts-debuginfo sgx-aesm-service-debuginfo sgx-pck-id-retrieval-tool-debuginfo sgx-ra-service-debuginfo

安装DCAP QPL

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install libsgx-dcap-default-qpl

安装nodejs

curl -sL https://rpm.nodesource.com/setup_14.x | sudo bash -
sudo yum install -y nodejs

查看nodejs版本

node --version
v10.24.0

安装PCCS DCAP

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install sgx-dcap-pccs

设置PCCS

sudo -u pccs /opt/intel/sgx-dcap-pccs/install.sh
Do you want to install PCCS now? (Y/N) :Y
Enter your http proxy server address, e.g. http://proxy-server:port (Press ENTER if there is no proxy server) : Enter
Enter your https proxy server address, e.g. http://proxy-server:port (Press ENTER if there is no proxy server) : Enter
Do you want to configure PCCS now? (Y/N) Y 
Set HTTPS listening port [8081] (1024-65535) Enter 
Set the PCCS service to accept local connections only? [Y] N 
Set your Intel PCS API key (Press ENTER to skip) Enter 
Choose caching fill method : [LAZY] (LAZY/OFFLINE/REQ) LAZY 
Set PCCS server administrator password: 1qaz@WSX3edc
之后一直回车直到安装完成。
密码：1qaz@WSX3edc

修改PCCS配置文件

vi /opt/intel/sgx-dcap-pccs/config/default.json
如果是production机器 修改对应项为以下值 
"uri": "https://api.trustedservices.intel.com/sgx/certification/v3/", 
"ApiKey" : "d9cafa222a414522a9c4169c07815910", 
"proxy" :
如果是非production机器 修改对应项为以下值 
"uri": "https://sbx.api.trustedservices.intel.com/sgx/certification/v3/", 
"ApiKey" : "5f13908fb4414e868644f9ef48e50b3b", 
"proxy" :

启动PCCS

systemctl start pccs
systemctl enable pccs

运行以下命令以确保PCCS服务正常工作

curl -k -G "https://localhost:8081/sgx/certification/v3/rootcacrl"

5.安装SGX SDK

安装依赖

yum groupinstall 'Development Tools'

下载

wget - https://download.01.org/intel-sgx/latest/linux-latest/distro/ <distro>/sgx_linux_x64_sdk_2.15.101.1.bin
chmod +x sgx_linux_x64_sdk_2.15.101.1.bin

安装

#./sgx_linux_x64_sdk_2.15.101.1.bin
Do you want to install in current directory? [yes/no] : no
Please input the directory which you want to install in : /opt/intel

加载环境变量

source /opt/intel/sgxsdk/environment

安装开发包

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install libsgx-enclave-common-devel libsgx-dcap-ql-devel libsgx-dcap-default-qpl-devel libsgx-quote-ex-devel libsgx-launch-devel libsgx-epid-devel libsgx-dcap-ql-devel libsgx-dcap-quote-verify-devel

QPL配置文件设置了要连接的PCCS的URL

vi /etc/sgx_default_qcnl.conf
# PCCS server address 
PCCS_URL=https://localhost:8081/sgx/certification/v3/ 
# To accept insecure HTTPS cert, set this option to FALSE 
USE_SECURE_CERT=FALSE

安装PCKIDRetrievalTool

dnf --nogpgcheck --repofrompath=SGX,/opt/intel/sgx_rpm_local_repo install sgx-pck-id-retrieval-tool
# 修改配置文件
vi /opt/intel/sgx-pck-id-retrieval-tool/network_setting.conf
USE_SECURE_CERT=FALSE
user_token = 1qaz@WSX3edc (设置user_token为安装pccs时设置的user password)

连接PCS获取PCKCerts

[root@dellemc-sgx /opt/intel]#PCKIDRetrievalTool
Intel(R) Software Guard Extensions PCK Cert ID Retrieval Tool Version 1.12.101.1

Registration status has been set to completed status.
pckid_retrieval.csv has been generated successfully!

6.准备demo环境

安装docker

# docker
yum install https://download.docker.com/linux/fedora/30/x86_64/stable/Packages/containerd.io-1.2.6-3.3.fc30.x86_64.rpm
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf -y  install docker-ce --nobest
systemctl start docker && sudo systemctl enable docker
# docker-compose
curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

镜像编译结果:

Measurement:
 578b18e8ca61526c25d4ddb5b1e4c84108663a041d619bba66af2e5914fd1c46
graphene-sgx-get-token --output python.token --sig python.sig
Attributes:
 mr_enclave:  578b18e8ca61526c25d4ddb5b1e4c84108663a041d619bba66af2e5914fd1c46
 mr_signer:   8ed9118e6052573edaf9d68ced4225a89307edf0b241b92857fe1463be4e00c1
 isv_prod_id: 0
 isv_svn:     0
 attr.flags:  0600000000000000
 attr.xfrm:   e702000000000000
 misc_select: 00000000
 misc_mask:   00000000
 modulus:     554190a13067a3443fe83ebcda28368f...
 exponent:    3
 signature:   9fbd7ff980b7a6509e0144ab7488acf5...
 date:        2022-03-29
Removing intermediate container dc49f9cebf7a
---> 62ff71a50650

Successfully built 62ff71a50650
Successfully tagged graphene-cfl:v1.2

7.运行sgx demo验证

名称	ICX地址	组织	操作者
dellemc-sgx	192.168.4.xxx	Party A	requester
lenovo-sgx	192.168.4.xxx	Party B	data owner
dell750-sgx	192.168.4.xxx	Party C	data owner

1.gRPC通信配置/etc/environment文件

cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
no_proxy=localhost,127.0.0.1,192.168.4.227,192.168.4.180

2.配置操作者的秘钥

在requester机器上把第1组秘 分配给操作 requester. 
cd $GRAPHENE_DOCKER_DIR/volumes/config 
rm *.pem 
cp ../../data_owner_keys/DO-A/* .
在data-owner机器上把第2组秘 分配给操作 data owner. 
cd $GRAPHENE_DOCKER_DIR/volumes/config 
rm *.pem 
cp ../../data_owner_keys/DO-B/* .

3.启动Avalon服务

在所有机器上分别执行如下操作：

两台机器置不同的OrganizationId 例如只修改末位

[root@dellemc-sgx ~/Desktop/cfl-demo/avalon_docker]#grep OrganizationId config/singleton_enclave_config.toml
# Replace ApplicationTypeId and OrganizationId with real hexadecimal values, 64 characters
OrganizationId = "aabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeef2"

[root@lenovo-sgx ~/Desktop/cfl-demo/avalon_docker]#grep OrganizationId config/singleton_enclave_config.toml
# Replace ApplicationTypeId and OrganizationId with real hexadecimal values, 64 characters
OrganizationId = "aabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeef1"

启动Avalon worker：

cd $AVALON_DOCKER_DIR 
./avalon-worker.sh start 
#之后会输出提示，需要用户输入worker_id，两台机器上worker_id不同即可，例如id1和id2： 
Input worker_id:id1

启动Web Service

在所有机器上修改 群配置文件  先修改$GRAPHENE_DOCKER_DIR/volumes/config/cfl.toml

然后修改$GRAPHENE_DOCKER_DIR/volumes/config/tcs_config.toml，将OrganizationId设置为本机OrganizationId，即修改本机Avalon配置文件时设置的值，例如：

[root@dellemc-sgx ~/Desktop/cfl-demo/graphene_docker]#grep OrganizationId volumes/config/tcs_config.toml
# Replace ApplicationTypeId and OrganizationId with real hexadecimal values
OrganizationId = "aabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeef2"

[root@lenovo-sgx ~/Desktop/cfl-demo/graphene_docker]#grep OrganizationId volumes/config/tcs_config.toml
# Replace ApplicationTypeId and OrganizationId with real hexadecimal values
OrganizationId = "aabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeeffaabbcc1234ddeef1"

所有机器上启动Web Service：

cd $GRAPHENE_DOCKER_DIR 
./web_service.sh start

8.web控制台配置

配置Requester Avalon服务
分别打开requester和data owner的管理页面http://192.168.4.180:5000/和http://192.168.x4.227:5000/  
点击下图中的set up as a requester按钮，选择192.168.4.180机器为requester角色
添加worker measurement
1.在192.168.4.180机器编译graphene-cfl镜像，编译结果输出mr_enclave和mr_signer信息
2.导出graphene-cfl镜像
# docker  save -o graphene-cfl-image.tar  graphene-cfl:v1.2
3.将文件传到192.168.4.227机器上
# scp graphene-cfl-image.tar 192.168.4.227:/root/Desktop/cfl-demo
4.登录192.168.4.227机器，删除旧镜像，导入新镜像
# docker images |  grep graphene-cfl
# docker rmi $id
# cd /root/Desktop/cfl-demo
# docker load -i graphene-cfl-image.tar
创建Data Owner Graphene worker ，在data owner机器192.168.4.227上，启动data owner Graphene worker：
# 登录192.168.4.227
cd /root/Desktop/cfl-demo/graphene_docker
./graphene_worker.sh start dataowner |  tee /tmp/dataowner.log
# 记录终端输出的checksum
例如：graphene_worker | 54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd dataset/cifar-10-batches-py/data_batch_1
在地址为192.168.4.227的页面data owner web进行添加数据集
点击Graphene worker列表中的worker id进入worker管理页面
点击Add dataset添加数据
Path:
/tcf/dataset/cifar-10-batches-py/data_batch_1
Checksum:
54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd
创建Requester Graphene worker

在requester机器192.168.4.180上，启动requester Graphene worker：
cd /root/Desktop/cfl-demo/graphene_docker
./graphene_worker.sh start requester
创建Job

在192.168.4.180（Requester）的requester avalon管理页面，点击Add Job按钮，即可创建Job

加入Job

分别刷新192.168.4.180和192.168.4.227的requester avalon管理页面，在两个管理页面点击Job id，进入

Job管理页面。

分别进入192.168.4.180和192.168.4.227的Job管理页面，分别在Graphene id选项框中各自选择一个Graphene worker ID，点击Submit加入workflow。

启动Job

在192.168.4.180（Requester）的workflow管理页面，点击start按钮，即可启动workflow。此时可以看到两个Graphene worker 都进入工作状态。

算法