Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ood-portal broken in httpd 2.4.62 #3728

Closed
johrstrom opened this issue Aug 13, 2024 · 10 comments · Fixed by #3730
Closed

ood-portal broken in httpd 2.4.62 #3728

johrstrom opened this issue Aug 13, 2024 · 10 comments · Fixed by #3730
Labels
bug Existing functionality not working as expected
Milestone

Comments

@johrstrom
Copy link
Contributor

Amazon linux has updated to httpd 2.4.62 and it appears the ood portal is broken and so our CI is broken, at least for Amazon linux 2023.

This is the only indication of any failure. The server simply returns 500s.

[Mon Aug 12 20:55:06.455849 2024] [proxy:warn] [pid 9512:tid 9599] [client 172.17.0.1:51742] AH01144: No protocol handler was valid for the URL /pun/sys/file-editor (scheme 'unix'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
@johrstrom johrstrom added the bug Existing functionality not working as expected label Aug 13, 2024
@johrstrom johrstrom added this to the 4.0 milestone Aug 13, 2024
@johrstrom
Copy link
Contributor Author

I believe this is an actual apache httpd bug. I've file https://bz.apache.org/bugzilla/show_bug.cgi?id=69261 with a repository that can demonstrate and reproduce the bug.

@johrstrom
Copy link
Contributor Author

Which is to say, we'll have to start skipping the amazon linux builds in the github CI. Don't know what that means for nightlies or an actual release when it comes to it.

@treydock
Copy link
Contributor

Which is to say, we'll have to start skipping the amazon linux builds in the github CI. Don't know what that means for nightlies or an actual release when it comes to it.

Can skip for Github Actions but packages and nightlies are all part of Gitlab CI and they don't run any of the E2E tests so I don't think a buggy Apache should prevent the package process.

@johrstrom
Copy link
Contributor Author

I don't think a buggy Apache should prevent the package process.

That's good news, but even so, we can't actually distribute Amazon 2023 packages unless we require a specific httpd version (or < 2.6.62)?

@treydock
Copy link
Contributor

we can't actually distribute Amazon 2023 packages unless we require a specific httpd version (or < 2.6.62)?

We just depend on any version of Apache. We only depend on >= 2.4 of < 2.5:

https://github.com/OSC/ondemand-packaging/blob/b160fcef081c7397c1be15b95e4edcc083307012/packages/ondemand-runtime/rpm/ondemand-runtime.spec#L114

It would be up to the Amazon Linux admins to avoid the broken version, which is likely just a documentation mention. I'm not familiar with how Amazon does their repos if they keep older versions of packages so it's possible to downgrade.

@johrstrom
Copy link
Contributor Author

We only depend on >= 2.4 of < 2.5:

I wonder if we need to tighten that to >= 2.4 of < 2.4.62 because 2.4.62 on any OS isn't likely to work. I filed the bug upstream, but who knows how long it'll take to fix and get distributed.

johrstrom added a commit that referenced this issue Aug 22, 2024
* fixes #3728 with solution provided by httpd devs

* use base uri and let httpd set the query parameters
@CSC-swesters
Copy link
Contributor

Hi. Interesting bug, good that it was resolved!

Would this be possible to backport to 3.1.x as well? I suppose stable deployments of OOD might hit this as well if other distros than Amazon Linux decide to upgrade httpd with the new changes.

@johrstrom
Copy link
Contributor Author

Yea we'll need to backport this.

@giuliolibrando
Copy link
Contributor

Please update the package version now. I lost 4 days debugging this issue on Apache.

@giuliolibrando
Copy link
Contributor

You alse need to update node_proxy.lua file

conn.uri = uri and (r.args and (uri .. "?" .. r.args) or uri) or r.unparsed_uri

must be changed to
conn.uri = uri and (r.args and (uri .. "?" .. r.args) or uri) or r.uri

johrstrom added a commit that referenced this issue Sep 5, 2024
* fixes #3728 with solution provided by httpd devs

* use base uri and let httpd set the query parameters
johrstrom added a commit that referenced this issue Sep 5, 2024
* fixes #3728 with solution provided by httpd devs

* use base uri and let httpd set the query parameters
DininduSenanayake pushed a commit to nesi/ondemand that referenced this issue Nov 29, 2024
* fixes OSC#3728 with solution provided by httpd devs

* use base uri and let httpd set the query parameters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Existing functionality not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants