Concerns over sonicstorage.blob.core.windows.net downtime

[wdoekes]

Over the weekend, sonicstorage.blob.core.windows.net had been in private mode (as far as I can tell).
That meant that everyone who tried to build a SONiC image from friday until now, couldn't.

These tickets all concern those build failures:
#20749
#20737
#20685

In one of them, @liushilongbuaa writes "Fixed" without any explanation. In another, @Gfrom2016 says that "[public] access has been re-enabled".

As a community member, I'd very much like to know a few things:

• How did the access get toggled to private?
• What are the chances that this ever happens again?
• How can we guard against this? Both through technical means (mirrors, cache, fallback location at upstream) and through policy means (who gets to toggle this)?

We'd love to help test and develop SONiC as SDN core component, but we will be hesitant to invest time and money if we cannot depend on the build consistenly working. (In this case: depend on dependencies being available.)

For the debian components, we know that they will be available. And github objects we can certainly source from somewhere if needed. But for instance the broadcom files, like libsaibcm_11.2.13.1_amd64.deb, seem to be available only on the sonicstorage. (Where is its upstream?) This makes us fear that someone on the internet can just pull the plug and disable the builds without any moments notice.

Please take a moment and address our concerns.

Thank you!
Walter Doekes
OSSO B.V.

[prgeor]

@wangxin can you provide explanation on how this fixed?

[liushilongbuaa]

Some packages are stored on Azure(sonicstorage controlled by microsoft). Some components build refers these packages.
It depends on each component. Maybe you can download these files to local place for backup.

[wdoekes]

@liushilongbuaa I appreciate you thinking along, but downloading the necessary files separately is obviously something that has occurred to me already. Right now I'm rigidly holding on to all build artifacts/cache/downloads in case something goes wrong again.

For example, here's a Makefile that collects all libsaibcm*deb files I could find, just in case I need a different version:

.PHONY: all
all: \
        11.2.13.1 \
        11.2.12.1 \
...
libsaibcm_dnx-dev_$(BCMVER)_amd64.deb:
        wget https://sonicstorage.blob.core.windows.net/public/sai/sai-broadcom/$(BCMBRANCH)/$(BCMVER)/dnx/$@
endif

Makefile.txt <- download

But right now these are just band-aids. If things are down again we'll have to start a big digging operation to find/fix the build.

I can backup a Debian repo. I can backup Git projects. I cannot backup the sonicstorage, because I have no file listing. (The counter argument that I've downloaded the files already earlier doesn't hold true. If I change a build setting, a new download might just pop up.)

What we're still looking for is:

• A simple explanation of what went wrong. Did the intern press the is_private checkbox? Can they just do that?
• A procedural explanation of what is stored where/why. A file listing of the entire storage bucket helps. We're particularly interested in the origin of parts that are not source code: like libbcm. Where would we source updates if someone decides that the sonicstorage gets no updates?
• A technical solution, like a single HTTP_PROXY setting so we can use a local proxy to cache everything and not depend on sonicstorage being up. (This is something we can contribute to.)

Thanks! If you have the answer to either of these three questions, please share. (Don't wait to get them all.)

[liushilongbuaa]

@liushilongbuaa I appreciate you thinking along, but downloading the necessary files separately is obviously something that has occurred to me already. Right now I'm rigidly holding on to all build artifacts/cache/downloads in case something goes wrong again.

For example, here's a Makefile that collects all libsaibcm*deb files I could find, just in case I need a different version:

.PHONY: all
all: \
        11.2.13.1 \
        11.2.12.1 \
...
libsaibcm_dnx-dev_$(BCMVER)_amd64.deb:
        wget https://sonicstorage.blob.core.windows.net/public/sai/sai-broadcom/$(BCMBRANCH)/$(BCMVER)/dnx/$@
endif

Makefile.txt <- download

But right now these are just band-aids. If things are down again we'll have to start a big digging operation to find/fix the build.

I can backup a Debian repo. I can backup Git projects. I cannot backup the sonicstorage, because I have no file listing. (The counter argument that I've downloaded the files already earlier doesn't hold true. If I change a build setting, a new download might just pop up.)

What we're still looking for is:

• A simple explanation of what went wrong. Did the intern press the is_private checkbox? Can they just do that?
• A procedural explanation of what is stored where/why. A file listing of the entire storage bucket helps. We're particularly interested in the origin of parts that are not source code: like libbcm. Where would we source updates if someone decides that the sonicstorage gets no updates?
• A technical solution, like a single HTTP_PROXY setting so we can use a local proxy to cache everything and not depend on sonicstorage being up. (This is something we can contribute to.)

Thanks! If you have the answer to either of these three questions, please share. (Don't wait to get them all.)

1. We have internal security requirement on storage account settings. Change on security settings caused access issues.
2. You can file list here https://github.com/sonic-net/sonic-buildimage/blob/master/files/build/versions/default/versions-web
3. I don't know he reason to use sonictorage. Maybe you can search each url in Makefile and find the original PR to find the reason.

[wdoekes]

1. We have internal security requirement on storage account settings. Change on security settings caused access issues.

Not an entirely satisfactory answer, but I'll take it 😺

3. I don't know [the] reason to use [sonicstorage]. Maybe you can search each url in Makefile and find the original PR to find the reason.

Fair enough. I think I'm particularly interested in the broadcom binaries. I'll try and find someone who knows through other channels.

2. You can [find the] file list here https://github.com/sonic-net/sonic-buildimage/blob/master/files/build/versions/default/versions-web

Nice. That is definitely something.

I do wonder if that:

(a) is up to date -- seeing that ipmitool has versions that have changed several times, see #20195 #20836 ;

(b) includes everything -- seeing that the libbcm* I mentioned above (Makefile.txt) is not in there.

Can you elaborate on how this list is maintained/updated?