Populate client side trace's local address via tcp kprobes

[ddelnano]

Summary: Populate client side trace's local address via tcp kprobes

This change populates client side trace's local_addr and local_port columns for the following use cases:

1. To provide more consistency for the protocol data tables. Having columns that are empty make it difficult for end users to understand what is being traced and make them less useful
2. To facilitate addressing a portion of the short lived process problems (Pixie Misses Events from Short Lived Processes/Pods #1638)

For 2, the root of the issue is that df.ctx["pod"] syntax relies on the px.upid_to_pod_name function. If a PEM misses the short lived process during its metadata update, this function fails to resolve the pod name. For client side traces where the pod is making an outbound connection (non localhost), the local_addr column provides an alternative pod name lookup for short lived processes when the pod is long lived. This means the following would be equivalent to the df.ctx["pod"] lookup: px.pod_id_to_pod_name(px.ip_to_pod_id(df.local_addr)).

I intend to follow this PR with a compiler change that will make df.ctx["pod"] try both methods should px.upid_to_pod_name fail to resolve. This will allow the existing pxl scripts to display the previously missed short lived processes.

Alternatives

Another approach I considered was expanding our use of the sock_alloc kprobe. I used ftrace on a simple curl command to see what other options could be used (sudo trace-cmd record -F -p function_graph http://google.com). The socket syscall calls sock_alloc, which would be another mechanism for accessing the struct sock. I decided against this approach because I don't think its viable to assume that the same thread/process that calls socket will be the one that does the later syscalls (how our BPF maps are set up). It's common to have a forking web server model, which means a different process/thread can call socket than the ones that later read/write to it.

Probe stability

These probes appear to be stable from our oldest and newest supported kernel. These functions exist in the tcp_prot, tcpv6_prot structs and I've seen that other projects and bcc tools use these probes. This makes me believe that these functions have a pretty well defined interface.

Relevant Issues: #1829, #1638

Type of change: /kind feature

Test Plan: New tests verify that ipv4 and ipv6 cases work

• Ran for i in $(seq 0 1000); do curl http://google.com/$i; sleep 2; done within a pod and verified that local_addr is populated with this change and px.pod_id_to_pod_name(px.ip_to_pod_id(df.local_addr)) works for pod name resolution.

• Verified the above curl test results in traces without local_addr without this change
  

• Tested on the following k8s offerings and machine images

• GKE COS and Ubuntu

• EKS Amazon Linux 2

Changelog Message: Populate socket tracer data table local_addr and local_port column for client side traces.

[ddelnano]

Since we don't have an easy mechanism for testing this, I tried to test this mid stream case by creating a psql shell prior to the PEM starting and issuing queries after the socket tracer was initialized. I verified that a tcp connection was made and only sendto syscalls were issued when queries were executed (no connect syscalls) to check that this would simulate a mid stream case.

Surprisingly, I saw that the local_addr column was populated before I added the probe on tcp_sendmsg despite my thinking that it should be empty. I went ahead and added this probe since I verified with ftrace on the same psql process that tcp_v4_connect isn't called. Just raising this since I wasn't able to verify that this additional probe adds extra coverage from the testing I did.

[ddelnano]

@oazizi000 would appreciate if you could give this a review. I still need to fix the bpf instruction issue for the 4.x kernels and the incorrect storage of the local port (it is stored in network byte order and is an existing issue), but that shouldn't impact the core details in the PR.

[ddelnano]

The iovec syscall return probes that rely on process_implicit_conn (recvmsg, recvmmsg, sendmsg, sendmmsg) had their instruction count increase beyond the 4.14 limit. This was my solution to getting the BPF instruction count on 4.14 kernels to work. The rationale is that cases that are right on the edge of this limit are likely to have problems already.

The other things I considered were the following:

• Remove match_trace_tgid logic on 4.14 kernels -- did not lower the instruction count enough
• Make the iovec ret syscalls provide a NULL sock for its submit_new_conn call (e.g. submit_new_conn(ctx, tgid, fd, addr, /*sock*/ NULL, role, source_fn)) -- not viable since it partially handles mid stream connections

I'm open to other suggestions if you have any, but this was the set of things I came up with.

[oazizi000]

Approach looks good. We should just manually test on a few more endpoints to gain more confidence. If that all looks good, then this should be good to go.

[ddelnano]

@oazizi000 I've tested this change on EKS (Amazon Linux 2), GKE Ubuntu and GKE COS in addition to tracking down the endianness problem (kernel data structure stores the local port in host byte order).

I've also created #2002 to make that byte order more clear even though that case doesn't have a bug.