Fix parallel downloads of the same image

[nirs]

When starting multiple vms in parallel using the same uncached yet image, we downloaded the data to the same temporary file (data.tmp) and wrote metadata to the same files (url, time, type, {digest}.digest). We could fail in many ways:

• One process removes the cache directory when another tries to rename a temporary file to the destination file.
• One process fails to remove the cache directory since another process started to download and the directory is not empty.
• Corrupting data when writing to the same file from multiple processes. Mostly likely when writing downloaded data.

Fixes:

• Do not remove the cache directory before the download. This avoids the failing to remove non-empty directory, and failure to rename a temporary file.
• Write all data to temporary files, replacing the destination file when the write is complete. This avoid corrupted data.
• Do not remove destination file when creating a new temporary file or renaming since creating a file truncate existing file and renaming replace it.

Issues:

• On windows os.Rename is not atomic, but it should work when using NTFS.
• On failures we may have lefover temporary files ({name}.tmp.{pid}).

Fixes #2722

Logs from creating 10 vms in parallel:
test-parallel-download.tar.gz

[AkihiroSuda]

Thanks, can we have unit tests?

[nirs]

Thanks, can we have unit tests?

I did not check the current tests yet, but I don't wan to have a test doing parallel downloads from a real server that will run on very pr.

We can do something like this:

1. start in process http server serving a directory inside the test temporary directory
2. add tiny image with random data to the http server root directory
3. run downloading the tiny image in parallel
4. verify that all downloads succeeded and contents of cached directory is correct

With a tiny image this can run very quickly.

@AkihiroSuda what do you think?

[AkihiroSuda]

SGTM

[nirs]

Thanks, can we have unit tests?

Here: #2728