-
-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Gianluca Arbezzano <[email protected]>
- Loading branch information
Gianluca Arbezzano
committed
Mar 21, 2020
1 parent
68a2c31
commit 58c4233
Showing
5 changed files
with
151 additions
and
176 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
+++ | ||
title = "Getting Started with kprofefe" | ||
description = "Deploy kprofefe collector in Kubernetes" | ||
weight = 15 | ||
draft = false | ||
toc = true | ||
bref = "" | ||
+++ | ||
|
||
## Goal | ||
|
||
The end goal here is to deploy everything in kubernetes, and to get profiles | ||
pushed from our pod into [profefe](https://github.com/profefe/profefe). | ||
|
||
## Prerequisites | ||
|
||
1. Have a Kubernetes cluster up and running | ||
2. Deploy profefe inside or outside your Kubernetes cluster. But it has to be | ||
reachable from the Kubernetes cluster network. I will assume that the URL for | ||
it is `https://profefe.internal.company.com:10100` | ||
3. You should have the kubectl profefe plugin already installed (checkout the | ||
installation doc) | ||
|
||
## Getting Started | ||
|
||
### Find a candidate | ||
|
||
We have to find a good first candidate, a pod that runs an application that | ||
exposes the pprof handler. I will assume it exposes it on port `9999`. If you do | ||
not have anything in your Kubernetes cluster you can deploy this application as | ||
a test | ||
|
||
``` | ||
apiVersion: v1 | ||
kind: Pod | ||
metadata: | ||
name: influxdb-v2 | ||
spec: | ||
containers: | ||
- name: influxdb | ||
image: quay.io/influxdb/influxdb:2.0.0-alpha | ||
ports: | ||
- containerPort: 9999 | ||
``` | ||
|
||
Now that we have the right candidate we should modify the pod adding two | ||
annotations: | ||
|
||
``` | ||
annotations: | ||
"profefe.com/enable": "true" | ||
"profefe.com/port": "9999" | ||
``` | ||
|
||
The first one tells kprofefe that this pod can be scraped. By default kprofefe | ||
looks for the port `6060`, with the second annotation we are overriding the | ||
default configuration, because our application exposes profefe to port | ||
`9999`. | ||
|
||
### Deploy kprofefe | ||
|
||
kprofefe can be deployed as a cronjob in Kubernetes. The one we are deploying | ||
now will scrape codes with the right annotations in all namespaces, every 10 | ||
minutes: | ||
|
||
``` | ||
apiVersion: batch/v1beta1 | ||
kind: CronJob | ||
metadata: | ||
name: kprofefe-allnamespaces | ||
namespace: default | ||
spec: | ||
concurrencyPolicy: Replace | ||
failedJobsHistoryLimit: 1 | ||
jobTemplate: | ||
spec: | ||
template: | ||
spec: | ||
containers: | ||
- args: | ||
# This cronjob will scrape all the pods that has the right | ||
# annotations across all the namespaces | ||
- --all-namespaces | ||
# This url represents the profefe API location. | ||
- --profefe-hostport | ||
- https://profefe.internal.company.com:10100 | ||
image: profefe/kprofefe | ||
imagePullPolicy: IfNotPresent | ||
name: kprofefe | ||
schedule: '*/10 * * * *' | ||
successfulJobsHistoryLimit: 3 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
+++ | ||
title = "Introduction" | ||
description = "Let's sync up on some concepts" | ||
weight = 10 | ||
draft = false | ||
toc = true | ||
bref = "" | ||
+++ | ||
|
||
## Introduction | ||
|
||
This project is a bridge between profefe and Kubernetes. At the moment it serves | ||
two different binaries: | ||
|
||
* `kubectl-profefe` a kubectl plugin that helps you to `caputre` pprof profiles, | ||
storing them locally or in pprofefe. It uses `port-forwarding` to expose the | ||
pprof port locally. | ||
* `kprofefe` is a cli that you can run as a cronjob in your kubernetes cluster. | ||
It discovers running pods in your clusters, it downloads profiles and it | ||
pushes them in profefe. | ||
|
||
NB: if your configuration does not allow you to do `kubectl port-forward` the | ||
`kubectl` plugin will not work. | ||
|
||
### How it works | ||
|
||
Golang has an http handler that exposes pprof over http, via annotation we can | ||
specify if a pod has profiles to capture and with other annotations we can | ||
configure path and port. | ||
|
||
The annotations are: | ||
|
||
* `profefe.com/enable=true` is the annotation that tells kube-profefe to capture | ||
profiles from that pod. | ||
* `profefe.com/port=8085` tells kube-profefe where to look for a pprof http | ||
server. By default it is 6060. | ||
* `profefe.com/service=frontend` tells kube-profefe the name of the service | ||
usable to lookup the profile. If the annotation is not specified it uses the | ||
pod name. My suggestion is to set this annotation because pods are ephemeral | ||
and lookup by pod name may be hard to do. | ||
* `profefe.com/path=/debug/pprof` tells kube-profefe where to look for a pprof http | ||
server. By default it is `/debug/pprof`. |