port to new azure SDK #532
Conversation
krnowak
force-pushed
the
krnowak/azure-new-stuff
branch
3 times, most recently
from
66d4786
to
ad42e53
Compare
krnowak
force-pushed
the
krnowak/azure-new-stuff
branch
3 times, most recently
from
cb8a773
to
9f85c83
Compare
|
Github really fails to handle this many changes :/ I wanted to leave a comment: can you modify the storage account creation |
krnowak
force-pushed
the
krnowak/azure-new-stuff
branch
from
9f85c83
to
481ec57
Compare
This commit replaces old resource clients with their counterparts from the new SDK. Most of the changes made were rather straightforward. The larger changes were made in storage account handling and blob uploading. We stopped generating shared keys for storage accounts, so for uploading we expect the authenticated user to have an RBAC permission that allows creating containers and upload blobs already assigned. This can be done by, for example, assigning "Storage Blob Data Owner" role at a subscription level. Uploading the blobs is now delegated to the azure-vhd-utils package. Another thing that has changed is authentication to Azure - the whole thing is replaced with a default authentication method provided by the azidentity package. So support for azure profile and azure credentials JSON files is dropped. More about the default authentication method can be read on azidentity documentation website: https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#section-readme Some functionality that was unused and was difficult to port (image sharing, replication and unreplication, some OS image stuff) was dropped.
The obsolete options were related to authentication.
The obsolete options were mostly about authentication. The obsolete commands were mostly about dropped functionality from Azure API (image sharing, replication and unreplication). Commands for creating images and uploading blobs are deduplicated. The code needed also some adaptation to the API changes.
Obsolete options were mostly about authentication. The adaptations were mostly about dropping support for storage keys and some of the API being changed.
krnowak
force-pushed
the
krnowak/azure-new-stuff
branch
from
481ec57
to
0bc5aaf
Compare
I have split the work into separate commits, maybe will be a tad easier to review and for github to display stuff. The whole vendor stuff is now in a separate commit too. I also dropped fiddling with RBAC, so that shrunk the vendored code too (not pulling the msgraph stuff any more).
Done. |
|
Tests:
Will need small changes in scripts on all branches (to drop obsolete |
|
|
||
| return false, nil | ||
| // log-level=NONE only affects the log file - stdout is unaffected | ||
| cmd := exec.Command(azcopy, "cp", "--blob-type=PageBlob", "--log-level=NONE", sourceURL, dstSAS) |
There was a problem hiding this comment.
why are we doing this? There is no lib for azcopy?
There was a problem hiding this comment.
there is no lib for azcopy, and the the SDK CopyBlob function copies all 30GB of a sparse VHD instead of just the populated 1GB. But I think this code path is unused since we switched to the container pipeline (we don't copyblobs anymore, we always upload a vhd).
There was a problem hiding this comment.
azcopy is doing similar thing to the tryCopyPageBlob function, but with goroutines and throttling and stuff. What I wasn't sure about is whether to keep the code that invokes azcopy.
|
It's a bit dense to review but if the CI is green it's already a good sign.
We don't need to forget about backporting this change to maintenance branches. The dropped commands do not seem to be used in the Flatcar org. |
Yeah, sorry about that. It's hard to split it into more commits, though.
I have a PR (flatcar/scripts#2063) for it. I think I'll just amend the PRs that will be created in scripts repo after merging this PR.
Right, I was asking about them on matrix some time ago. |
|
can this be merged? |
|
@jepio yeah, as discussed I'm fine with merging in this state |
--
changelog/directory (user-facing change, bug fix, security fix, update)