Cannot copy files to volume: error writing files to device

[girlpunk]

Summary

When trying to copy files to a volume attached to a container, the following error is returned:
Error response from daemon: Server error from portlayer: error writing files to device <container>

Environment information

Newly created volume and container on VIC 1.5.4-7589-e70af4fe and VCH v1.5.4, 21221, b3d3b06

vSphere and vCenter Server version

vCenter & ESXi 6.7.0

VIC version

v1.5.4-7589-e70af4fe

VCH configuration

Details

Steps to reproduce

docker cp example.txt container:/path/to/volume/destination.txt

Actual behavior

The following error is recieved
Error response from daemon: Server error from portlayer: error writing files to device <volume name>

Expected behavior

File is copied to the volume

Logs

N.B. link to the log bundle instructions does not work (404)
logs.zip

Found some relevent parts in the port layer logs:

Mar 31 2020 16:54:11.128Z INFO  op=247.6380.8826:   Attach/Mount [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk
Mar 31 2020 16:54:11.128Z INFO  op=247.6380.8826:   Create/attach vmdk [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk from parent <nil>
Mar 31 2020 16:54:11.132Z INFO  op=247.6380.8826:   No disks match name and persistence: [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk, true
Mar 31 2020 16:54:12.837Z ERROR op=247.6380.8826:   unexpected fault on task retry: &types.GenericVmConfigFault{VmConfigFault:types.VmConfigFault{VimFault:types.VimFault{MethodFault:types.MethodFault{FaultCause:(*types.LocalizedMethodFault)(nil), FaultMessage:[]types.LocalizableMessage{types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.hotadd.Failed", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"scsi0:3"}}, Message:"Failed to add disk 'scsi0:3'."}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.hotadd.poweron.failed", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"scsi0:3"}}, Message:"Failed to power on 'scsi0:3'. "}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.noBackEnd", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"/vmfs/volumes/vsan:52db2c501cf2cec0-6e715b769b08a3da/2527815e-8f23-59c1-db91-7446a095d606/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk"}}, Message:"Cannot open the disk '/vmfs/volumes/vsan:52db2c501cf2cec0-6e715b769b08a3da/2527815e-8f23-59c1-db91-7446a095d606/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk' or one of the snapshot disks it depends on. "}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.fileio.lock", Arg:[]types.KeyAnyValue(nil), Message:"Failed to lock the file"}}}}}, Reason:"Failed to add disk 'scsi0:3'."}
Mar 31 2020 16:54:12.839Z ERROR op=247.6380.8826:   vmdk storage driver failed to attach disk: Failed to add disk 'scsi0:3'.
Mar 31 2020 16:54:12.878Z INFO  op=247.6380.8826:   No online owners were found for letsencrypt_config-3ecb047
Mar 31 2020 16:54:12.879Z ERROR op=247.6380.8826:   Error getting data source: unable to create offline data source and no online owners found
Mar 31 2020 16:54:12.882Z INFO  op=247.6381.8827:   Attach/Mount [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk
Mar 31 2020 16:54:12.882Z INFO  op=247.6381.8827:   Create/attach vmdk [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk from parent <nil>
Mar 31 2020 16:54:12.886Z INFO  op=247.6381.8827:   No disks match name and persistence: [vsanDatastore] VIC/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk, true
Mar 31 2020 16:54:14.561Z ERROR op=247.6381.8827:   unexpected fault on task retry: &types.GenericVmConfigFault{VmConfigFault:types.VmConfigFault{VimFault:types.VimFault{MethodFault:types.MethodFault{FaultCause:(*types.LocalizedMethodFault)(nil), FaultMessage:[]types.LocalizableMessage{types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.hotadd.Failed", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"scsi0:3"}}, Message:"Failed to add disk 'scsi0:3'."}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.hotadd.poweron.failed", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"scsi0:3"}}, Message:"Failed to power on 'scsi0:3'. "}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.disk.noBackEnd", Arg:[]types.KeyAnyValue{types.KeyAnyValue{DynamicData:types.DynamicData{}, Key:"1", Value:"/vmfs/volumes/vsan:52db2c501cf2cec0-6e715b769b08a3da/2527815e-8f23-59c1-db91-7446a095d606/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk"}}, Message:"Cannot open the disk '/vmfs/volumes/vsan:52db2c501cf2cec0-6e715b769b08a3da/2527815e-8f23-59c1-db91-7446a095d606/volumes/letsencrypt_config-3ecb047/letsencrypt_config-3ecb047.vmdk' or one of the snapshot disks it depends on. "}, types.LocalizableMessage{DynamicData:types.DynamicData{}, Key:"msg.fileio.lock", Arg:[]types.KeyAnyValue(nil), Message:"Failed to lock the file"}}}}}, Reason:"Failed to add disk 'scsi0:3'."}
Mar 31 2020 16:54:14.563Z ERROR op=247.6381.8827:   vmdk storage driver failed to attach disk: Failed to add disk 'scsi0:3'.
Mar 31 2020 16:54:14.600Z INFO  op=247.6381.8827:   No online owners were found for letsencrypt_config-3ecb047
Mar 31 2020 16:54:14.600Z ERROR op=247.6381.8827:   import failed: unable to create offline data sink and no online owners found

See also

Troubleshooting attempted

• Searched GitHub for existing issues. (Mention any similar issues under "See also", above.)
• Searched the documentation for relevant troubleshooting guidance.
• Searched for a relevant VMware KB article.

[samized]

Same here.