Executables from k3s maybe be incorrectly flagged as malware

[benklett]

Environmental Info:
K3s Version:

$ k3s -v
k3s version v1.29.2+k3s1 (86f10213)
go version go1.21.7

affected are the channels from v1.23 until v1.29

Node(s) CPU architecture, OS, and Version:

$ uname -a
Linux ben-Virtual-Machine 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

single node cluster

Describe the bug:
A lot of files in /var/lib/rancher/k3s/data/current/bin/ get flagged by various Antvirus Software, e. g. ipset:

https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

SHA256: b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

This is the same sha for every current version in the channels v1.23 until latest.

The ipset file from version v1.22.17+k3s1:

https://www.virustotal.com/gui/file/618d9c1952ecfc13ce4358afec73e9283b7c24f968208656ad6fe2b3c7b3f8ef
SHA256: 618d9c1952ecfc13ce4358afec73e9283b7c24f968208656ad6fe2b3c7b3f8ef

List of files matched by F-Secure:

blkid
coreutils
aux/ebtablesd
aux/ebtables-legacy
find
fuse-overlayfs
ip
ipset
losetup
nsenter
pigz
slirp4netns
aux/xtables-legacy-multi
aux/xtables-nft-multi

Steps To Reproduce:

• Installed K3s: sudo INSTALL_K3S_CHANNEL= bash install.sh
• Check sha256: sha256sum /var/lib/rancher/k3s/data/current/bin/ipset
• Copy the sha into https://www.virustotal.com/gui/home/search

Expected behavior:
No results on virustotal.com

Actual behavior:
Some Hits on virustotal.com

Additional context / logs:

[thejan2009]

These packages are sourced from k3s-root, the same ipset SHA can be found there in the last two releases https://github.com/k3s-io/k3s-root/releases/download/v0.12.2/k3s-root-amd64.tar and https://github.com/k3s-io/k3s-root/releases/download/v0.13.0/k3s-root-amd64.tar

[benklett]

These packages are sourced from k3s-root, the same ipset SHA can be found there in the last two releases https://github.com/k3s-io/k3s-root/releases/download/v0.12.2/k3s-root-amd64.tar and https://github.com/k3s-io/k3s-root/releases/download/v0.13.0/k3s-root-amd64.tar

Thank you for the hint. I have tried to build my own binaries with this repo, but the resulting ipset< binary has the exact same SHA hash.

k3s-root seems to use https://github.com/buildroot/buildroot to build the required binaries.

[cwayne18]

This is a false positive, for some reason it happens sometimes on the ML-driven VirusTotal scanners. We will ensure they are properly marked as safe, thank you for the heads up!

[cwayne18]

It seems the false positives have been cleared: https://www.virustotal.com/gui/file/618d9c1952ecfc13ce4358afec73e9283b7c24f968208656ad6fe2b3c7b3f8ef

[benklett]

That is the older file version, which was not affected.

The current version is still affected:
https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd

[benklett]

blkid
coreutils
aux/ebtablesd
aux/ebtables-legacy
find
fuse-overlayfs
ip
ipset
losetup
nsenter
pigz
slirp4netns
aux/xtables-legacy-multi
aux/xtables-nft-multi

@cwayne18 Can you tell me if you already submitted the files as false positive or do I have to do that?

[cwayne18]

Our security team has already submitted them, thank you for offering!

[cippaciong]

Hello @cwayne18, sorry for asking the same question as @benklett, it's just to be 100% sure I understood correctly.

We have a similar report for the pigz file included in K3s (https://www.virustotal.com/gui/file/d7ac92d1df715d2cb5ffa7a8a49b12be738e6e5808e2d8bcadadd799aaad2d8e/detection), do you confirm that this and all other files affected have been submitted as false positives to VirusTotal?

Thanks

[mjrist]

@cwayne18 we are also investigating this. You claim that the security team has submitted these as false positives to VirusTotal. Any idea on the timeline here? When will these no longer be flagged?

[rjshrjndrn]

Can we keep this issue open till the scanners clear the flag? And, anybody who stumbles upon this issue will have visibility.

[brandond]

The false positives have been reported. There is no additional work to be done on our side at this time, and no timeline for when these low-quality scanners will address their issues.

[mjrist]

The false positives have been reported. There is no additional work to be done on our side at this time, and no timeline for when these low-quality scanners will address their issues.

Hey @brandond, many of us have customers who are inquiring into this. So having some visibility on what is happening here would be very helpful.

By the way, if I compile these executables (e.g. pigz) from source the scanners don't complain. My assumption is that they don't like something about the way they are being sourced from buildroot.

Is there an FAQ/known issues documentation that notes on this could be added to? Open to other ideas. In essence just looking for something I can point customers to: "This is not an issue. Here is why."

Thanks.

[brandond]

Unfortunately we don't have any visibility into these ML-based scanners either. We report issues when they pop up, they go away after a period of time. These vendors have no SLA on their free services, and offer no assistance other than some vague warnings about false positives in their ToS. It seems to happen about once a year or so they decide some random thing (usually busybox) is a virus or hacking tool, and we have to go through it again.

Feel free to point your users at this issue for context.

[benklett]

I have just read, that somehow signing could help with the issue. Would it be possible to always sign the released binaries to not let this happen again?

[cwayne18]

Signing the binaries wouldn't likely solve this issue, it's a bit out of our hands at the moment but we have reached out to Virustotal

[maxksec]

xtables-nft-multi is still flaged by virus total :(

k3s version v1.29.3+k3s1 (8aecc26b)

/var/lib/rancher/k3s/data/3fcd4fcf3ae2ba4d577d4ee08ad7092538cd7a7f0da701efa2a8807d44a25f66/bin/aux$ sha256sum xtables-nft-multi
60efc5a86f68103c5cf263ba61bdd2a468f93685a947e987f018cac8b95497f2  xtables-nft-multi

https://www.virustotal.com/gui/file/60efc5a86f68103c5cf263ba61bdd2a468f93685a947e987f018cac8b95497f2?nocache=1

[uniit]

Hi there,

k3s --version
k3s version v1.27.5+k3s1 (8d074ec)
go version go1.20.7

We received a notification from rapid7 about xtables-legacy-multi:

  "process": {
...
    "name": "xtables-legacy-multi",
...
    "exe_path": "/var/lib/rancher/k3s/data/b2c9b831d16a6f5e7d0224148df7be9b67491fc0f36e7a9a58e1e5b0c42b7258/bin/aux/xtables-legacy-multi",
    "cmd_line": "/var/lib/rancher/k3s/data/b2c9b831d16a6f5e7d0224148df7be9b67491fc0f36e7a9a58e1e5b0c42b7258/bin/aux/xtables-legacy-multi --help",
...
    "hash_reputation": {
      "reputation": "Malicious",
      "threat_level": "Critical",
      "reliability": "Low",
...
      "classification": {
        "family_name": "LuckyMouse",
        "type": "Trojan",
        "platform": "Linux"
      }
...

https://www.virustotal.com/gui/file/8cc834717af150b879f9cb9892236fd0fd4695d54bc50fc82a8d31f66a1bfa84

[brandond]

Still a false positive. Report it to your AV vendor and get them to fix it.

[burlyunixguy]

Regarding: https://www.virustotal.com/gui/file/b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd/detection

We finally got a response from our vendor that confirmed ipset (b52d5de7999a5b5b08ecc0bad539f99263b00bc6129aeb7327ead0e8f8b98bcd) is a false positive. VirusTotal still has non-zero community score, but at least we can move on.