Skip to content

Commit

Permalink
docs: getting started with kprofefe
Browse files Browse the repository at this point in the history
Signed-off-by: Gianluca Arbezzano <[email protected]>
  • Loading branch information
Gianluca Arbezzano committed Mar 21, 2020
1 parent 68a2c31 commit 58c4233
Show file tree
Hide file tree
Showing 5 changed files with 151 additions and 176 deletions.
99 changes: 4 additions & 95 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,100 +4,9 @@ continuous profiling and I realized his code was well abstracted and comfortable
to extend. The API server was already done and I decided to write an integration
with Kubernetes.

## kube-profefe
The documentation is located to [./docs/content/docs](./docs/content/docs) or to
[kubernetes.profefe.dev](https://kubernetes.profefe.dev).

This project is a bridge between profefe and Kubernetes. At the moment it serves
two different binaries:
## Install

* `kubectl-profefe` a kubectl plugin that helps you to `caputre` pprof profiles,
storing them locally or in pprofefe. It uses `port-forwarding` to expose the
pprof port locally.
* `kprofefe` is a cli that you can run as a cronjob in your kubernetes cluster.
It discovers running pods in your clusters, it downloads profiles and it
pushes them in profefe.

NB: if your configuration does not allow you to do `kubectl port-forward` the
`kubectl` plugin will not work.

### How it works

Golang has an http handler that exposes pprof over http, via annotation we can
specify if a pod has profiles to capture and with other annotations we can
configure path and port.

The annotations are:

* `profefe.com/enable=true` is the annotation that tells kube-profefe to capture
profiles from that pod.
* `profefe.com/port=8085` tells kube-profefe where to look for a pprof http
server. By default it is 6060.
* `profefe.com/service=frontend` tells kube-profefe the name of the service
usable to lookup the profile. If the annotation is not specified it uses the
pod name. My suggestion is to set this annotation because pods are ephemeral
and lookup by pod name may be hard to do.
* `profefe.com/path=/debug/pprof` tells kube-profefe where to look for a pprof http
server. By default it is `/debug/pprof`.

### Install kubectl-profefe

`kubectl-profefe` is a kubectl plugin designed to gather profiles from your
local laptop and store them locally or in profefe.

In the [release page in
github](https://github.com/gianarb/kube-profefe/releases) we push binaries and
containers (thanks goreleaser), you can download the tar.gz for your
architecture and move the `kubectl-profefe` in your `PATH`. In this way it will
work as kubectl plugin:

```
kubectl profefe --help
```

### Install via Krew

[krew](https://github.com/kubernetes-sigs/krew) is a package manager for kubectl
plugins and you can use it to install kube-profefe:

```
kubectl krew install profefe
```

### Getting Started with kubectl-profefe

Start minikube and deploy this pod:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: influxdb-v2
annotations:
"profefe.com/enable": "true"
"profefe.com/port": "9999"
spec:
containers:
- name: influxdb
image: quay.io/influxdb/influxdb:2.0.0-alpha
ports:
- containerPort: 9999
```
Now you can capture the profiles:
```bash
kubectl profefe capture influxdb-v2
```

The profiles are stored inside your `/tmp` directory (you can change it with
`--output-dir`), so you can read it with `go tool pprof`:

```
go tool pprof /tmp/profile-goroutine-influxdb-v2-1575552135.pb.gz
```

If you have a profefe up and running you can push your profiles there other than
locally:

```
kubectl profefe capture influxdb-v2 --profefe-hostport http://localhost:10100
```
[./docs/content/docs/installation.md](./docs/content/docs/installation.md)
81 changes: 0 additions & 81 deletions docs/content/docs/examples.md

This file was deleted.

92 changes: 92 additions & 0 deletions docs/content/docs/getting-started-with-kprofefe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
+++
title = "Getting Started with kprofefe"
description = "Deploy kprofefe collector in Kubernetes"
weight = 15
draft = false
toc = true
bref = ""
+++

## Goal

The end goal here is to deploy everything in kubernetes, and to get profiles
pushed from our pod into [profefe](https://github.com/profefe/profefe).

## Prerequisites

1. Have a Kubernetes cluster up and running
2. Deploy profefe inside or outside your Kubernetes cluster. But it has to be
reachable from the Kubernetes cluster network. I will assume that the URL for
it is `https://profefe.internal.company.com:10100`
3. You should have the kubectl profefe plugin already installed (checkout the
installation doc)

## Getting Started

### Find a candidate

We have to find a good first candidate, a pod that runs an application that
exposes the pprof handler. I will assume it exposes it on port `9999`. If you do
not have anything in your Kubernetes cluster you can deploy this application as
a test

```
apiVersion: v1
kind: Pod
metadata:
name: influxdb-v2
spec:
containers:
- name: influxdb
image: quay.io/influxdb/influxdb:2.0.0-alpha
ports:
- containerPort: 9999
```

Now that we have the right candidate we should modify the pod adding two
annotations:

```
annotations:
"profefe.com/enable": "true"
"profefe.com/port": "9999"
```

The first one tells kprofefe that this pod can be scraped. By default kprofefe
looks for the port `6060`, with the second annotation we are overriding the
default configuration, because our application exposes profefe to port
`9999`.

### Deploy kprofefe

kprofefe can be deployed as a cronjob in Kubernetes. The one we are deploying
now will scrape codes with the right annotations in all namespaces, every 10
minutes:

```
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: kprofefe-allnamespaces
namespace: default
spec:
concurrencyPolicy: Replace
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
containers:
- args:
# This cronjob will scrape all the pods that has the right
# annotations across all the namespaces
- --all-namespaces
# This url represents the profefe API location.
- --profefe-hostport
- https://profefe.internal.company.com:10100
image: profefe/kprofefe
imagePullPolicy: IfNotPresent
name: kprofefe
schedule: '*/10 * * * *'
successfulJobsHistoryLimit: 3
```
13 changes: 13 additions & 0 deletions docs/content/docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,22 @@ This project is made of two components. A kubectl plugin and the collector.
You can download both of them from the [release
page](https://github.com/profefe/kube-profefe/releases) via GitHub.

## kubectl profefe

The kubectl plugin usually runs from your laptop, and it is built for multiple
platforms: Linux, Mac.

### Install via Krew

[krew](https://github.com/kubernetes-sigs/krew) is a package manager for kubectl
plugins and you can use it to install kube-profefe:

```
kubectl krew install profefe
```

## kprofefe

The collector called `kprofefe` usually runs as a cronjob in Kubernetes. You can
find an example of it in `./contrib/kubernetes/kprofefe.yaml`.

Expand Down
42 changes: 42 additions & 0 deletions docs/content/docs/introduciton.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
+++
title = "Introduction"
description = "Let's sync up on some concepts"
weight = 10
draft = false
toc = true
bref = ""
+++

## Introduction

This project is a bridge between profefe and Kubernetes. At the moment it serves
two different binaries:

* `kubectl-profefe` a kubectl plugin that helps you to `caputre` pprof profiles,
storing them locally or in pprofefe. It uses `port-forwarding` to expose the
pprof port locally.
* `kprofefe` is a cli that you can run as a cronjob in your kubernetes cluster.
It discovers running pods in your clusters, it downloads profiles and it
pushes them in profefe.

NB: if your configuration does not allow you to do `kubectl port-forward` the
`kubectl` plugin will not work.

### How it works

Golang has an http handler that exposes pprof over http, via annotation we can
specify if a pod has profiles to capture and with other annotations we can
configure path and port.

The annotations are:

* `profefe.com/enable=true` is the annotation that tells kube-profefe to capture
profiles from that pod.
* `profefe.com/port=8085` tells kube-profefe where to look for a pprof http
server. By default it is 6060.
* `profefe.com/service=frontend` tells kube-profefe the name of the service
usable to lookup the profile. If the annotation is not specified it uses the
pod name. My suggestion is to set this annotation because pods are ephemeral
and lookup by pod name may be hard to do.
* `profefe.com/path=/debug/pprof` tells kube-profefe where to look for a pprof http
server. By default it is `/debug/pprof`.

0 comments on commit 58c4233

Please sign in to comment.