[download] Migrate to Pulsar's Rolling Release Repo, off of CirrusCI

[confused-Techie]

This PR removes our download microservices interaction with CirrusCI, instead now collecting rolling releases from pulsar-edit/pulsar-rolling-releases repository.

These changes relate to the necessary changes needed to be made, detailed in pulsar-edit/pulsar#685.

And should resolve any and all issues related to the download microservice, to ensure uninterrupted functionality of our download microservice.

[DeeDeeG]

In my ideal design in my head, the download microservice would speak both Cirrus CI and GitHub Releases. It could branch its logic based on "ARM Linux/Apple Silicon macOS? --> get from Cirrus", "other stuff? --> get from GitHub Releases rolling".

So that the download microservice still works for resolving Apple Silicon and ARM Linux payload URLs. Otherwise the service is dropping our ARM builds.

(EDIT: And I surely don't want to try and yeet the Cirrus Rolling binaries onto the GitHub Releases Rolling, that's too much shenanigans for me. Although arguably on the same level and degree of shenanigans as having the download microservice speak to two binary hosts with their own JSON APIs, but eh...)

As for the PR as-is, I haven't taken a close look, but it's nice if this is working already for GitHub Releases.

[confused-Techie]

@DeeDeeG You do have a really good point there.

Although what if we take a slightly different approach and instead add scripts to the end of builds on cirrus to upload those to the github releases? Since you can see in the PR the GitHub API is much simpler to work with, and would be nice to have a proper visual archive of all our rolling releases.

Although I do get if that's difficult to do and we have to add back in the capabilities to work with Cirrus. Since that wasn't something fully considered during the drafting of this PR. Sure this PR will still look for ARM and Silicon binaries, but I had forgotten that we had no plans to make them available here

[DeeDeeG]

🤷 It's a little bit down to what we can dream up, design and implement in the next few days. Not trivial, theoretically doable. I'm sure we can add back more stuff if some of it gets left on the cutting room floor in order to get something out the door in time. Something is better than nothing, to be sure.

Edit: Not to downplay efforts, I think we're doing pretty darn well in response to this situation, as always being volunteers in our spare time. Including this PR seems to have taken some real work to it already. Kudos to all that's working already in this and thanks for this and all the other efforts!! We're gettin' through this.

[confused-Techie]

Appreciate you! But very true, we are limited by time.

Although over on the GitHub Actions PR on the main repo, I put all the logic of uploading artifacts into it's own NodeJS script, not relying on much else, so it may be totally possible to get Cirrus to run that same script after the fact, I'll actually take a try at implementing that soon, since if so it'd be rad to get this all functioning without to many more logic changes lol

[DeeDeeG]

Some stuff I noticed, most of it is either really minor or just optional nitpicks.

There's one && check that I think is meant to be ||, but other than that, I think this looks pretty good! Worked in my testing already, as-is!

Thank you for this!!!!

[DeeDeeG]

Posting my test results, as always for this microservice's PR's:

My old test-find-link.js from before (see code snippet below) totally works with the new download microservice code. So, this seems to be a functional drop-in replacement for the old code. (Just using GitHub Releases instead of Cirrus's API.)

// This is test-find-link.js
const utils = require("./utils.js");

(async () => {
  console.log(await utils.findLink("arm_linux", "linux_deb"));
})();

Test output when modifying and running node ./test-find-link to test all possible binary assets (click to expand):

% time node ./test-find-link.js 
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/Pulsar-1.108.2023082516.AppImage'
}
node ./test-find-link.js  0.12s user 0.04s system 37% cpu 0.410 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/pulsar-1.108.2023082516.tar.gz'
}
node ./test-find-link.js  0.12s user 0.03s system 54% cpu 0.271 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/pulsar-1.108.2023082516.x86_64.rpm'
}
node ./test-find-link.js  0.12s user 0.03s system 56% cpu 0.269 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/pulsar_1.108.2023082516_amd64.deb'
}
node ./test-find-link.js  0.12s user 0.04s system 37% cpu 0.413 total
% time node ./test-find-link.js
{
  ok: false,
  code: 404,
  msg: 'Unable to find any assets matching the provided parameters: os=linux;type=windows_setup'
}
node ./test-find-link.js  0.12s user 0.03s system 53% cpu 0.274 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082517/Pulsar.Setup.1.108.2023082517.exe'
}
node ./test-find-link.js  0.12s user 0.03s system 35% cpu 0.406 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082517/Pulsar-1.108.2023082517-win.zip'
}
node ./test-find-link.js  0.12s user 0.03s system 56% cpu 0.264 total
% time node ./test-find-link.js
{
  ok: false,
  code: 404,
  msg: 'Unable to find any assets matching the provided parameters: os=windows;type=windows_blockmap'
}
node ./test-find-link.js  0.12s user 0.03s system 59% cpu 0.250 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/Pulsar-1.108.2023082516-mac.zip'
}
node ./test-find-link.js  0.12s user 0.03s system 38% cpu 0.378 total
% time node ./test-find-link.js
{
  ok: false,
  code: 404,
  msg: 'Unable to find any assets matching the provided parameters: os=intel_mac;type=mac_zip_blockmap'
}
node ./test-find-link.js  0.12s user 0.04s system 56% cpu 0.277 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082516/Pulsar-1.108.2023082516.dmg'
}
node ./test-find-link.js  0.12s user 0.03s system 53% cpu 0.270 total
% time node ./test-find-link.js
{
  ok: false,
  code: 404,
  msg: 'Unable to find any assets matching the provided parameters: os=intel_mac;type=mac_dmg_blockmap'
}
node ./test-find-link.js  0.12s user 0.03s system 52% cpu 0.275 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/Pulsar-1.108.2023082808-arm64-mac.zip'
}
node ./test-find-link.js  0.12s user 0.04s system 37% cpu 0.416 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/Pulsar-1.108.2023082808-arm64-mac.zip.blockmap'
}
node ./test-find-link.js  0.11s user 0.03s system 58% cpu 0.246 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/Pulsar-1.108.2023082808-arm64.dmg'
}
node ./test-find-link.js  0.12s user 0.03s system 37% cpu 0.389 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/Pulsar-1.108.2023082808-arm64.dmg.blockmap'
}
node ./test-find-link.js  0.12s user 0.03s system 55% cpu 0.271 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/Pulsar-1.108.2023082808-arm64.AppImage'
}
node ./test-find-link.js  0.12s user 0.03s system 35% cpu 0.433 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/pulsar-1.108.2023082808-arm64.tar.gz'
}
node ./test-find-link.js  0.12s user 0.03s system 59% cpu 0.260 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/pulsar-1.108.2023082808.aarch64.rpm'
}
node ./test-find-link.js  0.11s user 0.03s system 34% cpu 0.414 total
% time node ./test-find-link.js
{
  ok: true,
  content: 'https://github.com/pulsar-edit/pulsar-rolling-releases/releases/download/v1.108.2023082808/pulsar_1.108.2023082808_arm64.deb'
}
node ./test-find-link.js  0.12s user 0.03s system 39% cpu 0.379 total

Note that there are some ok: false results where I gave incorrect parameter combinations, or where there was a genuine and legitimate reason to return a 404, which worked as intended and as written, even without doing anything with the stuff I mentioned in my review comments.

In summary: I didn't find any real-world failures of the microservice logic with this testing. Handling bad input parameters and handling missing assets were also good and both working as intended in these tests.

[confused-Techie]

@DeeDeeG Addressed everything you commented on, if you'd like to do one more review so we can push this. Apparently our changes to cirrus have in fact already broken the download microservice

[DeeDeeG]

Suggested change

	let name = asset.name;
	let name = asset?.name;

If you want to put optional chaining back, I might have been misinformed about just how "impossible" it is for asset to be undefined here (I have found out it is indeed possible, if you loop through an array with literal undefined in it).

We're pulling data from JSON where undefined isn't allowed, so it should be safe, but in general I think you had the motivation/implementation probably right before, sorry about that.

[DeeDeeG]

Approve with or without the optional chaining, feedback was addressed and implementation works in my testing, should un-break the download microservice in production.

(EDIT to add: Note on the source of the microservice breakage happening that this PR should fix... We have completed builds with no tasks now, given the filtering we're doing for what runs can/can't go through on Cirrus... Which the old logic didn't account for! That older stuff was complex and turned out to be a bit brittle. A simpler API to interface is certainly a blessing for this microservice.)

Another good PR, lots of these flying fast around here, kudos for all the hard work.

[confused-Techie]

Thanks a ton @DeeDeeG, appreciate you and all your work on this PR as well.

Gonna go ahead and merge, then push to production!