Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parse kthreads during /proc scanning #2089

Merged
merged 3 commits into from
Feb 20, 2024
Merged

Parse kthreads during /proc scanning #2089

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
692e9ef
Add kthreads during /proc scanning
tpapagian Feb 12, 2024
1f1934c
Revert "ebpf: Ignore kernel threads during clone events"
tpapagian Feb 12, 2024
0ce1177
Update gitignore
tpapagian Feb 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions bpf/process/bpf_fork.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,16 +23,11 @@ BPF_KPROBE(event_wake_up_new_task, struct task_struct *task)
struct execve_map_value *curr, *parent;
struct msg_clone_event msg;
u64 msg_size = sizeof(struct msg_clone_event);
u32 flags, tgid = 0;
u32 tgid = 0;

if (!task)
return 0;

/* We do not care about kernel threads. */
flags = BPF_CORE_READ(task, flags);
if (flags & PF_KTHREAD)
return 0;

tgid = BPF_CORE_READ(task, tgid);

/* Do not try to create any msg or calling execve_map_get
Expand Down
1 change: 1 addition & 0 deletions contrib/tester-progs/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,4 @@ killer-tester-32
/getcpu
drop-privileges
change-capabilities
direct-write-tester
tixxdz marked this conversation as resolved.
Show resolved Hide resolved
45 changes: 45 additions & 0 deletions pkg/grpc/exec/kthread_init.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// SPDX-License-Identifier: Apache-2.0
// Copyright Authors of Tetragon

package exec

import (
"github.com/cilium/tetragon/api/v1/tetragon"
"github.com/cilium/tetragon/pkg/api/processapi"
"github.com/cilium/tetragon/pkg/logger"
"github.com/cilium/tetragon/pkg/process"
"github.com/cilium/tetragon/pkg/reader/notify"
)

// This is used only when we add kernel threads during /proc scanning.
// We use the same interface with all the other events to make the handling
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we also add kernel threads from the clone right? your PR does that? I'm just thinking on how to improve that comment so it reads we handle /proc scanning here and clone part of the usuall clone.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we also add kernel threads from the clone right? your PR does that?

Yes, the next commit reverts a commit that didn't do that.

This is only used for kernel threads during proc scanning. This should not be used during normal operation. The reason that I added that is that this is somewhere in the middle compared to what the exec and clone handler do.

In that case, we don't need to generate events compared to exec. In comparison to clones, we do not have a way to provide a process name as we always inherit that from the parent.

But this is also much more simple. There is no need to do anything related to the eventcache. I preferred to create a new (simple) handler compared to changing the existing handlers (and making them more complex).

// of them easier.
type MsgKThreadInitUnix struct {
Unix *processapi.MsgExecveEventUnix
}

func (msg *MsgKThreadInitUnix) HandleMessage() *tetragon.GetEventsResponse {
proc := process.AddExecEvent(msg.Unix)
parent, err := process.Get(proc.UnsafeGetProcess().ParentExecId)
if err != nil {
logger.GetLogger().Warnf("Failed to find parent for kernel thread %d", msg.Unix.Msg.Parent.Pid)
}
parent.RefInc()
return nil
}

func (msg *MsgKThreadInitUnix) RetryInternal(_ notify.Event, _ uint64) (*process.ProcessInternal, error) {
return nil, nil
}

func (msg *MsgKThreadInitUnix) Retry(_ *process.ProcessInternal, _ notify.Event) error {
return nil
}

func (msg *MsgKThreadInitUnix) Notify() bool {
return false
}

func (msg *MsgKThreadInitUnix) Cast(_ interface{}) notify.Message {
return &MsgKThreadInitUnix{}
}
188 changes: 130 additions & 58 deletions pkg/sensors/exec/procevents/proc_reader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ import (
"fmt"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"time"
"unicode/utf8"

Expand Down Expand Up @@ -80,6 +82,7 @@ type procs struct {
time_for_children_ns uint32
cgroup_ns uint32
user_ns uint32
kernel_thread bool
}

func (p procs) args() []byte {
Expand Down Expand Up @@ -186,58 +189,95 @@ func pushExecveEvents(p procs) {
args = args + " " + cwd
}

m := exec.MsgExecveEventUnix{}
m.Unix = &processapi.MsgExecveEventUnix{}
m.Unix.Msg = &processapi.MsgExecveEvent{}
m.Unix.Msg.Common.Op = ops.MSG_OP_EXECVE
m.Unix.Msg.Common.Size = processapi.MsgUnixSize + p.psize + p.size

m.Unix.Msg.Kube.NetNS = 0
m.Unix.Msg.Kube.Cid = 0
m.Unix.Msg.Kube.Cgrpid = 0
if p.pid > 0 {
m.Unix.Kube.Docker, err = procsDockerId(p.pid)
if err != nil {
logger.GetLogger().WithError(err).Warn("Procfs execve event pods/ identifier error")
}
// If this is a kernel thread, we use its filename as process name
// similarly to what ps reports.
if p.kernel_thread {
filename = fmt.Sprintf("[%s]", filename)
args = ""
}

m.Unix.Msg.Parent.Pid = p.ppid
m.Unix.Msg.Parent.Ktime = p.pktime

m.Unix.Msg.Capabilities.Permitted = p.permitted
m.Unix.Msg.Capabilities.Effective = p.effective
m.Unix.Msg.Capabilities.Inheritable = p.inheritable

m.Unix.Msg.Namespaces.UtsInum = p.uts_ns
m.Unix.Msg.Namespaces.IpcInum = p.ipc_ns
m.Unix.Msg.Namespaces.MntInum = p.mnt_ns
m.Unix.Msg.Namespaces.PidInum = p.pid_ns
m.Unix.Msg.Namespaces.PidChildInum = p.pid_for_children_ns
m.Unix.Msg.Namespaces.NetInum = p.net_ns
m.Unix.Msg.Namespaces.TimeInum = p.time_ns
m.Unix.Msg.Namespaces.TimeChildInum = p.time_for_children_ns
m.Unix.Msg.Namespaces.CgroupInum = p.cgroup_ns
m.Unix.Msg.Namespaces.UserInum = p.user_ns

m.Unix.Process.Size = p.size
m.Unix.Process.PID = p.pid
m.Unix.Process.TID = p.tid
m.Unix.Process.NSPID = p.nspid
// use euid to be compatible with ps
m.Unix.Process.UID = p.uids[1]
m.Unix.Process.AUID = p.auid
m.Unix.Msg.Creds = processapi.MsgGenericCredMinimal{
Uid: p.uids[0], Euid: p.uids[1], Suid: p.uids[2], FSuid: p.uids[3],
Gid: p.gids[0], Egid: p.gids[1], Sgid: p.gids[2], FSgid: p.gids[3],
}
m.Unix.Process.Flags = p.flags | flags
m.Unix.Process.Ktime = p.ktime
m.Unix.Msg.Common.Ktime = p.ktime
m.Unix.Process.Filename = filename
m.Unix.Process.Args = args
if p.kernel_thread {
m := exec.MsgKThreadInitUnix{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't we use MsgExecveEventUnix for kernel threads as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reasons that I didn't use the MsgExecveEventUnix are:

  1. I didn't want to have process_exec events for kernel threads (similar to what we do now)
  2. We didn't want to put any of these events in the eventcache as if we cannot find its parent at that time we will never find that. I have added a log.Fatalf in that case, but maybe it is a good idea to just print a warning instead.

I could have modified MsgExecveEventUnix to handle these, but I believe that it is a cleaner approach to have a separate message type.

m.Unix = &processapi.MsgExecveEventUnix{}
m.Unix.Msg = &processapi.MsgExecveEvent{}

m.Unix.Msg.Common = processapi.MsgCommon{}
m.Unix.Msg.Kube = processapi.MsgK8s{}
m.Unix.Msg.CleanupProcess = processapi.MsgExecveKey{}

m.Unix.Msg.Parent.Pid = p.ppid
m.Unix.Msg.Parent.Ktime = p.pktime

m.Unix.Msg.Capabilities = processapi.MsgCapabilities{}
m.Unix.Msg.Namespaces = processapi.MsgNamespaces{}

m.Unix.Process.Size = p.size
m.Unix.Process.PID = p.pid
m.Unix.Process.TID = p.pid
m.Unix.Process.NSPID = p.nspid
m.Unix.Process.UID = 0
m.Unix.Process.AUID = proc.InvalidUid

m.Unix.Process.Flags = api.EventProcFS
m.Unix.Process.Ktime = p.ktime
m.Unix.Process.Filename = filename
m.Unix.Process.Args = ""

observer.AllListeners(&m)
} else {
m := exec.MsgExecveEventUnix{}
m.Unix = &processapi.MsgExecveEventUnix{}
m.Unix.Msg = &processapi.MsgExecveEvent{}
m.Unix.Msg.Common.Op = ops.MSG_OP_EXECVE
m.Unix.Msg.Common.Size = processapi.MsgUnixSize + p.psize + p.size

m.Unix.Msg.Kube.NetNS = 0
m.Unix.Msg.Kube.Cid = 0
m.Unix.Msg.Kube.Cgrpid = 0
if p.pid > 0 {
m.Unix.Kube.Docker, err = procsDockerId(p.pid)
if err != nil {
logger.GetLogger().WithError(err).Warn("Procfs execve event pods/ identifier error")
}
}

m.Unix.Msg.Parent.Pid = p.ppid
m.Unix.Msg.Parent.Ktime = p.pktime

m.Unix.Msg.Capabilities.Permitted = p.permitted
m.Unix.Msg.Capabilities.Effective = p.effective
m.Unix.Msg.Capabilities.Inheritable = p.inheritable

m.Unix.Msg.Namespaces.UtsInum = p.uts_ns
m.Unix.Msg.Namespaces.IpcInum = p.ipc_ns
m.Unix.Msg.Namespaces.MntInum = p.mnt_ns
m.Unix.Msg.Namespaces.PidInum = p.pid_ns
m.Unix.Msg.Namespaces.PidChildInum = p.pid_for_children_ns
m.Unix.Msg.Namespaces.NetInum = p.net_ns
m.Unix.Msg.Namespaces.TimeInum = p.time_ns
m.Unix.Msg.Namespaces.TimeChildInum = p.time_for_children_ns
m.Unix.Msg.Namespaces.CgroupInum = p.cgroup_ns
m.Unix.Msg.Namespaces.UserInum = p.user_ns

m.Unix.Process.Size = p.size
m.Unix.Process.PID = p.pid
m.Unix.Process.TID = p.tid
m.Unix.Process.NSPID = p.nspid
// use euid to be compatible with ps
m.Unix.Process.UID = p.uids[1]
m.Unix.Process.AUID = p.auid
m.Unix.Msg.Creds = processapi.MsgGenericCredMinimal{
Uid: p.uids[0], Euid: p.uids[1], Suid: p.uids[2], FSuid: p.uids[3],
Gid: p.gids[0], Egid: p.gids[1], Sgid: p.gids[2], FSgid: p.gids[3],
}
m.Unix.Process.Flags = p.flags | flags
m.Unix.Process.Ktime = p.ktime
m.Unix.Msg.Common.Ktime = p.ktime
m.Unix.Process.Filename = filename
m.Unix.Process.Args = args

observer.AllListeners(&m)
observer.AllListeners(&m)
}
}

func updateExecveMapStats(procs int64) {
Expand Down Expand Up @@ -321,14 +361,14 @@ func writeExecveMap(procs []procs) {
updateExecveMapStats(int64(len(procs)))
}

func pushEvents(procs []procs) {
writeExecveMap(procs)
func pushEvents(ps []procs) {
writeExecveMap(ps)

sort.Slice(procs, func(i, j int) bool {
return procs[i].ppid < procs[j].ppid
sort.Slice(ps, func(i, j int) bool {
return ps[i].ppid < ps[j].ppid
})
procs = append(procs, procKernel())
for _, p := range procs {
ps = append([]procs{procKernel()}, ps...)
for _, p := range ps {
pushExecveEvents(p)
}
}
Expand All @@ -352,16 +392,30 @@ func listRunningProcs(procPath string) ([]procs, error) {
continue
}

// All processes have a directory name that consists from a number.
if !regexp.MustCompile(`\d`).MatchString(d.Name()) {
continue
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great addition ;-)


pathName := filepath.Join(procPath, d.Name())

cmdline, err := os.ReadFile(filepath.Join(pathName, "cmdline"))
if err != nil {
continue
}
if string(cmdline) == "" {

// We read comm in the case where cmdling is empty (i.e. kernel thread).
comm, err := os.ReadFile(filepath.Join(pathName, "comm"))
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this only to check if process is a kernel thread? good change ;-) , I thought that we can check /proc/$pid/status for kthread field, but it seems available only on new kernels

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We check that this is a kernel thread by using string(cmdline) == "" in a similar way that we did that before. I use comm just to get a name for the kernel thread in order to provide a reasonable name for the binary field.

continue
}

kernelThread := false
if string(cmdline) == "" {
cmdline = comm
kernelThread = true
}

pid, err := proc.GetProcPid(d.Name())
if err != nil {
logger.GetLogger().WithError(err).Warnf("pid read error")
Expand Down Expand Up @@ -476,6 +530,15 @@ func listRunningProcs(procPath string) ([]procs, error) {
continue
}

pcomm, err := os.ReadFile(filepath.Join(parentPath, "comm"))
if err != nil {
continue
}

if string(pcmdline) == "" {
pcmdline = pcomm
}

pstats, err = proc.GetProcStatStrings(string(parentPath))
if err != nil {
logger.GetLogger().WithError(err).Warnf("parent stats read error")
Expand All @@ -500,13 +563,21 @@ func listRunningProcs(procPath string) ([]procs, error) {

execPath, err := os.Readlink(filepath.Join(procPath, d.Name(), "exe"))
if err != nil {
logger.GetLogger().WithError(err).WithField("process", d.Name()).Warnf("reading process exe error")
if kernelThread {
execPath = strings.TrimSuffix(string(cmdline), "\n")
} else {
logger.GetLogger().WithError(err).WithField("process", d.Name()).Warnf("reading process exe error")
}
}

if _ppid != 0 {
pexecPath, err = os.Readlink(filepath.Join(procPath, ppid, "exe"))
if err != nil {
logger.GetLogger().WithError(err).WithField("process", ppid).Warnf("reading process exe error")
if kernelThread {
pexecPath = strings.TrimSuffix(string(pcmdline), "\n")
} else {
logger.GetLogger().WithError(err).WithField("process", ppid).Warnf("reading process exe error")
}
}
} else {
pexecPath = ""
Expand Down Expand Up @@ -542,6 +613,7 @@ func listRunningProcs(procPath string) ([]procs, error) {
time_for_children_ns: time_for_children_ns,
cgroup_ns: cgroup_ns,
user_ns: user_ns,
kernel_thread: kernelThread,
}

p.size = uint32(processapi.MSG_SIZEOF_EXECVE + len(p.args()) + processapi.MSG_SIZEOF_CWD)
Expand Down
Loading