diff --git a/Tools/gsc/Examples/tensorflow/README.md b/Tools/gsc/Examples/tensorflow/README.md new file mode 100644 index 0000000000..48ac97b24d --- /dev/null +++ b/Tools/gsc/Examples/tensorflow/README.md @@ -0,0 +1,92 @@ +# Inference on TensorFlow BERT and ResNet50 models: +For additional information on how to install, run and optimize TensorFlow, please see +https://github.com/Satya1493/graphene/blob/tensorflow/Examples/tensorflow/README.md. + +## Build graphenize Docker image and run BERT inference: +1. Build docker image: +``` +docker build --rm -t ubuntu18.04-tensorflow-bert -f ubuntu18.04-tensorflow-bert.dockerfile . +``` + +2. Graphenize the docker image using gsc build: +``` +cd ../.. +./gsc build --insecure-args ubuntu18.04-tensorflow-bert Examples/tensorflow/ubuntu18.04-tensorflow.manifest +``` + +3. Sign the graphenized Docker image using gsc sign-image: +``` +./gsc sign-image ubuntu18.04-tensorflow-bert enclave-key.pem +``` + +4. To run fp32 inference on GSC: +``` +docker run --device=/dev/sgx_enclave --cpuset-cpus="0-35" --env OMP_NUM_THREADS=36 \ +--env KMP_AFFINITY=granularity=fine,noverbose,compact,1,0 \ +gsc-ubuntu18.04-tensorflow-bert \ +models/models/language_modeling/tensorflow/bert_large/inference/run_squad.py \ +--init_checkpoint=data/bert_large_checkpoints/model.ckpt-3649 \ +--vocab_file=data/wwm_uncased_L-24_H-1024_A-16/vocab.txt \ +--bert_config_file=data/wwm_uncased_L-24_H-1024_A-16/bert_config.json \ +--predict_file=data/wwm_uncased_L-24_H-1024_A-16/dev-v1.1.json \ +--precision=fp32 \ +--predict_batch_size=32 \ +--experimental_gelu=True \ +--optimized_softmax=True \ +--input_graph=data/fp32_bert_squad.pb \ +--do_predict=True \ +--mode=benchmark \ +--inter_op_parallelism_threads=1 \ +--intra_op_parallelism_threads=36 \ +--output_dir=output/bert-squad-output +``` + +5. To run fp32 inference on native container (outside Graphene), remove +``--device=/dev/sgx_enclave`` and replace ``gsc-ubuntu18.04-tensorflow-bert`` with +``ubuntu18.04-tensorflow-bert`` in the above command. + +6. Above commands are for a 36 core system. Please check +https://github.com/Satya1493/graphene/blob/tensorflow/Examples/tensorflow/README.md for setting +different options for optimal performance. + +## Build graphenize Docker image and run ResNet50 inference: +1. Build docker image: +``` +docker build --rm -t ubuntu18.04-tensorflow-resnet50 -f ubuntu18.04-tensorflow-resnet50.dockerfile . +``` + +2. Graphenize the docker image using gsc build: +``` +cd ../.. +./gsc build --insecure-args ubuntu18.04-tensorflow-resnet50 Example/tensorflow/ubuntu18.04-tensorflow.manifest +``` + +3. Sign the graphenized Docker image using gsc sign-image: +``` +./gsc sign-image ubuntu18.04-tensorflow-resnet50 enclave-key.pem +``` + +4. To run int8 inference on GSC: +``` +docker run --device=/dev/sgx_enclave --cpuset-cpus="0-35" --env OMP_NUM_THREADS=36 \ +--env KMP_AFFINITY=granularity=fine,noverbose,compact,1,0 \ +gsc-ubuntu18.04-tensorflow-resnet50 \ +models/models/image_recognition/tensorflow/resnet50v1_5/inference/eval_image_classifier_inference.py \ +--input-graph=resnet50v1_5_int8_pretrained_model.pb \ +--num-inter-threads=1 \ +--num-intra-threads=36 \ +--batch-size=32 \ +--warmup-steps=50 \ +--steps=500 +``` +**NOTE**: When OOM happens user can set environment varibale ``TF_MKL_ALLOC_MAX_BYTES`` to upper +bound on memory allocation. As an example in a machine with 32 GB memory pass option +``--env TF_MKL_ALLOC_MAX_BYTES=17179869184`` to docker run command when OOM happens. + +5. To run int8 inference on native container (outside Graphene), remove +``--device=/dev/sgx_enclave`` and replace ``gsc-ubuntu18.04-tensorflow-resnet50`` with +``ubuntu18.04-tensorflow-resnet50`` in the above command. + +6. Above commands are for a 36 core system. Please check +https://github.com/Satya1493/graphene/blob/tensorflow/Examples/tensorflow/README.md for setting +different options for optimal performance. diff --git a/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-bert.dockerfile b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-bert.dockerfile new file mode 100644 index 0000000000..d4c5e097c5 --- /dev/null +++ b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-bert.dockerfile @@ -0,0 +1,24 @@ +From ubuntu:18.04 + +# Install prerequisites +RUN apt-get update \ + && apt-get install -y git wget python3 python3-pip unzip \ + && pip3 install --upgrade pip + +# Install tensorflow +RUN pip3 install intel-tensorflow-avx512==2.4.0 + +# Download models +RUN git clone https://github.com/IntelAI/models.git /models/ + +# Download data +RUN mkdir -p data \ + && cd data \ + && wget https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip \ + && unzip wwm_uncased_L-24_H-1024_A-16.zip \ + && wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json -P wwm_uncased_L-24_H-1024_A-16 \ + && wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/bert_large_checkpoints.zip \ + && unzip bert_large_checkpoints.zip \ + && wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_4_0/fp32_bert_squad.pb + +ENTRYPOINT ["python3"] diff --git a/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-resnet50.dockerfile b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-resnet50.dockerfile new file mode 100644 index 0000000000..0345987a29 --- /dev/null +++ b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow-resnet50.dockerfile @@ -0,0 +1,17 @@ +From ubuntu:18.04 + +# Install prerequisites +RUN apt-get update \ + && apt-get install -y git wget python3 python3-pip \ + && pip3 install --upgrade pip + +# Install tensorflow +RUN pip3 install intel-tensorflow-avx512==2.4.0 + +# Download input graph file +RUN wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50v1_5_int8_pretrained_model.pb + +# Download model +RUN git clone https://github.com/IntelAI/models.git /models/ + +ENTRYPOINT ["python3"] diff --git a/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow.manifest b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow.manifest new file mode 100644 index 0000000000..ee93a2000d --- /dev/null +++ b/Tools/gsc/Examples/tensorflow/ubuntu18.04-tensorflow.manifest @@ -0,0 +1,6 @@ +sgx.enclave_size = "32G" +sgx.thread_num = 300 +loader.pal_internal_mem_size = "64M" +loader.insecure__use_host_env = 1 +sgx.allowed_files.tmp = "file:/tmp" +sgx.preheat_enclave = 1