[osquery 5.13.1] osqueryd crashing on Fedora 40 host with distributed queries

[dherder]

Fleet version: 4.53.1

---

💥  Actual behavior

When running fleetd on a Fedora 40 host, continued osqueryd crashing is observed.

🧑‍💻  Steps to reproduce

Crash logs:
https://drive.google.com/file/d/1oazsLdWMmoMLRMYPbINV409n199Fj_sj/view?usp=drive_link

[dherder]

For whoever looks at this issue, we have repro'd in our cloud eval environment. Please reach out to me for the creds to access.

[sharon-fdm]

Reproduced by @dherder removing the reproduce label.

[dherder]

@sharon-fdm this is a high priority issue for prospect-redwine. Should we add the "P2" label?

[lucasmrod]

Seems related to kolide/launcher#1773

And see public discussion here.

[sharon-fdm]

@dherder, makes sense to me to add P2.
We can swap it with other items in the sprint (@noahtalerman maybe swap with #19561).
@lukeheath, please approve.

[lukeheath]

@dherder @sharon-fdm Agreed, upgrading to P2.

[sharon-fdm]

Thanks @lukeheath

[lucasmrod]

UPDATE:

How to reproduce

We've reproduced the issue in Fedora 38 and 40 (most likely it's on Fedora 39 too). You can reproduce this by running the following query: select 1 from rpm_packages rp, os_version ov where rp.name = "foo-fedora-playbooks" AND ov.name = "Fedora Linux"; on such systems (it will crash osquery 3 out 5 times or so).

What's happening

I've opened an osquery issue that describes the findings so far: osquery/osquery#8384.

Next steps

Stefano from the osquery team will try to upgrade librpm in osquery from 4.18.0 to 4.18.2. 4.18.2 has the following fix related to the segfault: https://patchwork.yoctoproject.org/project/oe-core/patch/[email protected]. We believe that upgrading may fix the crash.

[JoStableford]

Related to a Slack conversation

[JoStableford]

Related to a Slack conversation

[lucasmrod]

Removing Fleet's milestone as this is an osquery core bug (being fixed in 5.13.1 or 5.14.0, TBD)

[lucasmrod]

I can confirm that the update of librpm from 4.18.0 to 4.18.2 fixed the issue (no more crashes when querying rpm_packages). (Verified by downloading artifacts from today's master builds which contain the bug fix.)

[xpkoala]

Confirmed no crashes on Fedora 38.

[fleet-release]

Fedora host finds calm,
Crashing clouds part with each query,
Fleet sails smooth, no harm.