Skip to content
This repository has been archived by the owner on Oct 28, 2024. It is now read-only.

Cannot compile model into target device; error: Resource temporarily unavailable; model: controlled-access-model; version: 1; device: CPU #10

Open
CatClimb opened this issue Jul 25, 2024 · 0 comments

Comments

@CatClimb
Copy link

There is a bug r execrun docker iamge for gsc-openvino/model_server-ovsa_sgx-nginx-mtls:latest:
run command:
/opt/ovsa/gramine/example_runtime# sudo ./start_secure_ovsa_sgx_model_server.sh

sample.json file configuration:

{ "custom_loader_config_list":[ { "config":{ "loader_name":"ovsa", "library_path": "/ovsa-runtime/lib/libovsaruntime.so" } } ], "model_config_list":[ { "config":{ "name":"controlled-access-model", "base_path":"/opt/ovsa/gramine/artefacts/fd", "custom_loader_options": {"loader_name": "ovsa", "keystore": "/opt/ovsa/gramine/keystore/custkeystore", "controlled_access_file": "face_detection_model"} } } ] }

The error information is as follows:
`Parsing TOML manifest file, this may take some time...
[E:] [BSL] found 0 ioexpander device
E: [xLinkUsb] [ 769987] [] usb_find_device_with_bcd:266 Library has not been initialized when loaded
[2024-07-25 05:02:49.989][129][modelmanager][info][modelmanager.cpp:110] Available devices for Open VINO: CPU, GNA
[2024-07-25 05:02:49.990][129][serving][info][modelmanager.cpp:534] Reading Custom Loader: ovsa configuration
OvsaCustomLoader: Instance of Custom SampleLoader created
OvsaCustomLoader: Custom loaderInit
[2024-07-25 05:02:50.084][129][serving][info][customloaders.cpp:80] Clearing the list
[2024-07-25 05:02:50.084][129][serving][info][customloaders.cpp:83] Adding new list to the old list
[2024-07-25 05:02:50.084][129][serving][info][model.cpp:38] Getting model from /opt/ovsa/gramine/artefacts/fd
[2024-07-25 05:02:50.084][129][serving][info][model.cpp:45] Model downloaded to /opt/ovsa/gramine/artefacts/fd
[2024-07-25 05:02:50.084][129][serving][info][model.cpp:145] Will add model: controlled-access-model; version: 1 ...
[2024-07-25 05:02:50.084][129][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model controlled-access-model status change. New status: ( "state": "START", "error_code": "OK" )
[2024-07-25 05:02:50.084][129][serving][info][modelinstance.cpp:870] Loading model: controlled-access-model, version: 1, from path: /opt/ovsa/gramine/artefacts/fd/1, with target device: CPU ...
[2024-07-25 05:02:50.084][129][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model controlled-access-model status change. New status: ( "state": "START", "error_code": "OK" )
[2024-07-25 05:02:50.084][129][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model controlled-access-model status change. New status: ( "state": "LOADING", "error_code": "OK" )
[2024-07-25 05:02:50.084][129][serving][info][modelinstance.cpp:594] loading ov::Model for model: controlled-access-model basepath: /opt/ovsa/gramine/artefacts/fd/1 <> /opt/ovsa/gramine/artefacts/fd version: 1
OvsaCustomLoader: Custom loadModel
Type of member loader_name is ovsa
Type of member keystore is /opt/ovsa/gramine/keystore/custkeystore
Type of member controlled_access_file is face_detection_model
OvsaCustomLoader:
loader_name:/opt/ovsa/gramine/artefacts/fd/1/ovsa
datFile:/opt/ovsa/gramine/artefacts/fd/1/face_detection_model.dat
licFile:/opt/ovsa/gramine/artefacts/fd/1/face_detection_model.lic
keystore:/opt/ovsa/gramine/keystore/custkeystore
OVSA: Load Asymmetric Key
OVSA: Verify customer certificate
LibOVSA: Certificate signature verified OK
LibOVSA: Certificate verified OK
OVSA: Validate customer license
OVSA: Verify PEER certificate
LibOVSA: Certificate signature verified OK
LibOVSA: Certificate verified OK
OVSA: Verify customer license signature
LibOVSA: HMAC Verified OK
OVSA: Validate controlled access model
OVSA: Verify controlled access model signature
LibOVSA: Verified OK
OVSA: Generate HASH For Controlled Access Model

OVSA: Controlled_access model_hash HASH verified OK.
OVSA: Verify PEER certificate
LibOVSA: Certificate signature verified OK
LibOVSA: Certificate verified OK
OVSA: Verify customer license signature
LibOVSA: HMAC Verified OK
OVSA: Perform Platform Validation using TLS
OVSA:Attempting to connect license server url ........

OVSA:License_serv_url_0: 'localhost:4451' localhost:4451
OVSA:Entering ovsa_license_service_start
OVSA:Entering ovsa_validate_peer_cert_hash

OVSA: Peer certificate HASH verified OK.
LibOVSA: Certificate signature verified OK
LibOVSA: Certificate verified OK
OVSA:ovsa_license_service_start Exit
OVSA:ovsa_license_service_start() connected to license server 'localhost:4451'

OVSA:Connect to license server url status true

OVSA: Platform Validation completed successfully
OVSA: Received payload from server
'{
"command": "OVSA_SEND_NONCE",
"payload": "0AcK+USNSWQKNNU4YRkzBXN4bZQk9a4r22DIErkSJ9s=\n"
}'
LibOVSA: Signing Done
OVSA:Send signed nonce to server
OVSA:Send signed nonce to server
00000200{
"command": "OVSA_SEND_SIGN_NONCE",
"payload": "MGUCMQCnOJTE0pHMe4Zcg/7T3n9vM+lYlMflfVzW76WbT9XQZhqBSQwivmeXBKmT\nQ6jVTGACMBTb14hQ6eMxxVnZsMHG4htNsIO+qhRMW6YhxTdflGzx0MaQHTOiUM1P\nPAynHWUQyA==\n"
}
OVSA:Send OVSA_SEND_CUST_LICENSE to server
00003157{
"command": "OVSA_SEND_CUST_LICENSE",
"payload": "{\n\t"creation_date":\t"Thu Jul 25 03:06:29 2024",\n\t"isv_certificate":\t"-----BEGIN CERTIFICATE-----\nMIIBejCCAQACFCn1eRMZixHzRnt0MTf5IO7wn52FMAoGCCqGSM49BAMEMCExCzAJ\nBgNVBAYTAklOMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjQwNzI1MDMwNTQ3WhcN\nMjYwMTI1MDMwNTQ3WjAhMQswCQYDVQQGEwJJTjESMBAGA1UEAwwJbG9jYWxob3N0\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEaJ/WVdqLeatMsC9SHT7rRMPtQxgL+Z/E\nauwikHMzqPfF6ydNpMAh9fmxmKk9eb0vtr4Vh0gsv4N7VzLp8PlcGQIikwFK3HP9\ntK7j310TYpoJ9yXvi/vAMsoSpAuPKco+MAoGCCqGSM49BAMEA2gAMGUCMAbXB9Je\nsAz2gsJCeZlVDzHWVU3OGufk9CNrUTHxuQdc1jqXe+k8mEmpfk4Ttwk5wgIxAKVt\nx4mqUzNtfdCzfsFurIfdhSVslsfwKlqCnlFZH6Q9p3GURDI7Z+p4jBf7U/HVPg==\n-----END CERTIFICATE-----\n",\n\t"model_hash":\t"ZDYzMWYwNDMzOThiMDkzNDgwZmRmZmIwMGMxNjE2NjkwZDU3NzFmNDNhYWE2ZGM2\nY2UwNjdlNTYxNTI0NmI0ODVlNzlkMjBjNjI3YWY3MmIxMTRjNzI3MTkzNmM5OTZi\nODJhYmY5OTBiYjNhZWVhMDAwODU4NmRmYzYwYzcxNDE=\n",\n\t"license_name":\t"Time Limit License Config",\n\t"license_version":\t"1.0",\n\t"license_guid":\t"40ce4505-b332-4fee-a04e-3008a33a43b4",\n\t"model_guid":\t"bb23aee8-15b6-4e2c-939c-d60b39591a4e",\n\t"encryption_key":\t"U2FsdGVkX1+m4tRYmb74YLLaxUw8sJD52RShAas5GGWlTqnk4i0tSH1d6zfJM9H4eN/G6ebTM1pyd2hEnVg9RgwyJJC9p04pMUHOtvEYC7Ha",\n\t"license_serv_url":\t{\n\t\t"url_0":\t{\n\t\t\t"url":\t"localhost:4451",\n\t\t\t"cur_cert_hash":\t"MmM0ZWE0NDdlY2FiNWYxMDA1OGZiOTBhYzI1MjI3MTU3MjZhYTZjMjQ5ZTc5ZWEx\nYzdkMDljMTQ3Njk2OTU2YTI4YmVmZTc1ZGE3ZTJlMTQ1YzRmYWNhNzEwZWQ3Yzdh\n",\n\t\t\t"fut_cert_hash":\t""\n\t\t}\n\t},\n\t"license_type":\t"TimeLimit",\n\t"usage_count":\t0,\n\t"time_limit":\t30,\n\t"tcb_signature":\t{\n\t\t"sig_0":\t"{\n\t\"name\":\t\"Face Detect @ Runtime\",\n\t\"version\":\t\"1.0\",\n\t\"TCB Signature\":\t{\n\t\t\"SGX_ENCLAVE_Quote\":\t{\n\t\t\t\"mrenclave\":\t\"b907978745bbe7109bbb8001875d10708c323bf027cb09fef263d44d94714798\",\n\t\t\t\"mrsigner\":\t\"e1dfcf9fef7a54790100c50e78f7bb4e19342f5fc06660f97d45eeffafbf40ff\",\n\t\t\t\"isv_svn\":\t0,\n\t\t\t\"isv_prod_id\":\t0\n\t\t}\n\t},\n\t\"isv_certificate\":\t\"-----BEGIN CERTIFICATE-----\\nMIIBejCCAQACFCn1eRMZixHzRnt0MTf5IO7wn52FMAoGCCqGSM49BAMEMCExCzAJ\\nBgNVBAYTAklOMRIwEAYDVQQDDAlsb2NhbGhvc3QwHhcNMjQwNzI1MDMwNTQ3WhcN\\nMjYwMTI1MDMwNTQ3WjAhMQswCQYDVQQGEwJJTjESMBAGA1UEAwwJbG9jYWxob3N0\\nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEaJ/WVdqLeatMsC9SHT7rRMPtQxgL+Z/E\\nauwikHMzqPfF6ydNpMAh9fmxmKk9eb0vtr4Vh0gsv4N7VzLp8PlcGQIikwFK3HP9\\ntK7j310TYpoJ9yXvi/vAMsoSpAuPKco+MAoGCCqGSM49BAMEA2gAMGUCMAbXB9Je\\nsAz2gsJCeZlVDzHWVU3OGufk9CNrUTHxuQdc1jqXe+k8mEmpfk4Ttwk5wgIxAKVt\\nx4mqUzNtfdCzfsFurIfdhSVslsfwKlqCnlFZH6Q9p3GURDI7Z+p4jBf7U/HVPg==\\n-----END CERTIFICATE-----\\n\",\n\t\"signature\":\t\"MGUCMD1u7NiUyNJETdRT6V598KXa+v8dpE6UhRiqrhxCrCjvMaDpEbF9pVkHb1DB\\n4Ze9NwIxAM6g/+CrYqiFcb1RxhjUtOR3imn/12INAJ0u8k/s86KSmMIDTo9kTfds\\nNZNAdShuSQ==\\n\"\n}"\n\t},\n\t"signature":\t"bZaRjEWYVD9euXoBC4SSYV5YzVkYZqtVc4UZ7z/yO9r+53HxtJxfQUnf51eR5cTH\n614L7oFi1hpDNYKf+isNMQ==\n"\n}"
}OVSA: Received payload from server
'{
"command": "OVSA_SEND_LICENSE_CHECK_RESP",
"payload": "PASS"
}'
OVSA:Received license check result from Server: 'PASS'
OVSA: Platform and License Validation completed successfully
OVSA: Invoking model loader
OVSA: Unwrap model encryption key Successful
OVSA: Decrypt model file : face-detection-retail-0004.xml Successful
OVSA: Decrypt model file : face-detection-retail-0004.bin Successful
OvsaCustomLoader: face-detection-retail-0004.xml
OvsaCustomLoader: face-detection-retail-0004.bin

  • echo 'Starting Nginx...'
    Starting Nginx...
    Parsing TOML manifest file, this may take some time...
    OvsaModelInstance: Instance of Custom OvsaModelInstance created for model
    OvsaModelInstance: Thread Start for model controlled-access-model
    OvsaModelInstance: StartWatcher exit for model controlled-access-model
    OvsaModelInstance: Doing Some Work 1 -> for model controlled-access-model : version 1
    [2024-07-25 05:02:51.665][129][modelmanager][info][modelinstance.cpp:414] Input name: data; mapping_name: data; shape: (1,3,300,300); precision: FP32; layout: N...
    [2024-07-25 05:02:51.665][129][modelmanager][info][modelinstance.cpp:465] Output name: detection_out; mapping_name: detection_out; shape: (1,1,200,7); precision: FP32; layout: N...
    [2024-07-25 05:02:51.691][129][modelmanager][error][modelinstance.cpp:671] Cannot compile model into target device; error: Resource temporarily unavailable; model: controlled-access-model; version: 1; device: CPU
    [2024-07-25 05:02:51.691][129][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model controlled-access-model status change. New status: ( "state": "LOADING", "error_code": "UNKNOWN" )
    [2024-07-25 05:02:51.691][129][serving][error][model.cpp:150] Error occurred while loading model: controlled-access-model; version: 1; error: Cannot compile model into target device
    [2024-07-25 05:02:51.691][129][modelmanager][error][modelmanager.cpp:1106] Error occurred while loading model: controlled-access-model versions; error: Cannot compile model into target device
    [2024-07-25 05:02:51.691][129][serving][info][model.cpp:187] Will clean up model: controlled-access-model; version: 1 ...[2024-07-25 05:02:51.691][129][serving][info][model.cpp:85] Updating default version for model: controlled-access-model, from: 0
    [2024-07-25 05:02:51.691][129][serving][info][model.cpp:97] Model: controlled-access-model will not have default version since no version is available.
    [2024-07-25 05:02:51.691][129][serving][info][modelversionstatus.cpp:109] STATUS CHANGE: Version 1 of model controlled-access-model status change. New status: ( "state": "LOADING", "error_code": "UNKNOWN" )
    OvsaCustomLoader: Custom unloadModel
  • (( 1 ))
  • (( 1 ))
  • '[' -f /tmp/job_finished ']'
  • sleep 1
    Parsing TOML manifest file, this may take some time...
  • nginx -g 'daemon off;'
  • NGINX_PID=39
  • echo 'Nginx PID: 39'
    Nginx PID: 39
  • echo 39
  • wait 39
    nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/certs/server.pem"
    nginx: [warn] "ssl_stapling" ignored, issuer certificate not found for certificate "/certs/server.pem"
    Parsing TOML manifest file, this may take some time...
  • (( 1 ))
  • (( 1 ))
  • '[' -f /tmp/job_finished ']'
  • sleep 1
    Parsing TOML manifest file, this may take some time...
    Parsing TOML manifest file, this may take some time...`
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant