Skip to content

Commit

Permalink
feat: cherry-pick upstream commits (#3)
Browse files Browse the repository at this point in the history
* Better Jira error handling (prometheus-community#140)

* Better Jira error handling

* Return HTTP 400 Bad Request for non-retriable errors. It is inaccurate, but
  will prevent alertmanager from retrying.
* Turns out go-jira does actually produce useful error messages (and it consumes
  the response body in the process). Log that instead of the empty body.

Signed-off-by: Alin Sinpalean <[email protected]>

* Also include HTTP response status 429 among retriable errors.

Signed-off-by: Alin Sinpalean <[email protected]>

* Include both the go-jira error and the response body in errors. Sometimes go-jira consumes the body and includes it in its error, sometimes it doesn't.

Signed-off-by: Alin Sinpalean <[email protected]>

---------

Signed-off-by: Alin Sinpalean <[email protected]>
Co-authored-by: Alin Sinpalean <[email protected]>

* disable update existing jira issues with parameter (prometheus-community#150)

* Bump all dependencies (prometheus-community#133)

Signed-off-by: Jan-Otto Kröpke <[email protected]>

Signed-off-by: Jan-Otto Kröpke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>

* parameter to disable update jira issues

Signed-off-by: Holger Waschke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>

* rename parameter to make it more clear and avoid double negation. fix bug with missing return value.

Signed-off-by: Holger Waschke <[email protected]>

* Update main.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* Update notify.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* Update main.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* Update main.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* Update main.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* Update notify.go

Signed-off-by: dvag-holger-waschke <[email protected]>

* fix for notify test

Signed-off-by: Holger Waschke <[email protected]>

---------

Signed-off-by: Jan-Otto Kröpke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>
Signed-off-by: dvag-holger-waschke <[email protected]>
Co-authored-by: Jan-Otto Kröpke <[email protected]>
Co-authored-by: Holger Waschke <[email protected]>

* Adding getEnv templating function (prometheus-community#153)

Signed-off-by: Jiri Tyr <[email protected]>

* feat: add support for static jira labels (prometheus-community#154)

Signed-off-by: Herman Ewert <[email protected]>
Co-authored-by: Herman Ewert <[email protected]>

* Fix prometheus-community#146 (safe limit of 200 characters from group label value) (prometheus-community#147)

Signed-off-by: jzajic <[email protected]>

* doc(PAT): Adds doc for PAT usage (prometheus-community#155)

Signed-off-by: Julian Beck <[email protected]>

* truncate descriptions that exceed -max-description-length (default 32KB) (prometheus-community#165)

* truncate descriptions that exceed -max-description-length (default 32,768)

Signed-off-by: Jason Wells <[email protected]>

* Update main.go

size was off by 1

Signed-off-by: Jason Wells <[email protected]>

---------

Signed-off-by: Jason Wells <[email protected]>

* fix: 🐛 Fixes error message for doTransition to display the proper transition state (prometheus-community#176)

Signed-off-by: Nathan Gotz <[email protected]>

* search for existing issue in multiple projects (prometheus-community#162)

* search for existing issue in multiple projects

Signed-off-by: Jason Wells <[email protected]>

* Apply suggestions from code review

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Jason Wells <[email protected]>

---------

Signed-off-by: Jason Wells <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>

* add Fingerprint field to Alert so that it may be used in templates (prometheus-community#152) (prometheus-community#163)

Signed-off-by: Jason Wells <[email protected]>

---------

Signed-off-by: Alin Sinpalean <[email protected]>
Signed-off-by: Jan-Otto Kröpke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>
Signed-off-by: Holger Waschke <[email protected]>
Signed-off-by: dvag-holger-waschke <[email protected]>
Signed-off-by: Jiri Tyr <[email protected]>
Signed-off-by: Herman Ewert <[email protected]>
Signed-off-by: jzajic <[email protected]>
Signed-off-by: Julian Beck <[email protected]>
Signed-off-by: Jason Wells <[email protected]>
Signed-off-by: Nathan Gotz <[email protected]>
Co-authored-by: Alin Sinpalean <[email protected]>
Co-authored-by: Alin Sinpalean <[email protected]>
Co-authored-by: dvag-holger-waschke <[email protected]>
Co-authored-by: Jan-Otto Kröpke <[email protected]>
Co-authored-by: Holger Waschke <[email protected]>
Co-authored-by: Jiri Tyr <[email protected]>
Co-authored-by: Herman <[email protected]>
Co-authored-by: Herman Ewert <[email protected]>
Co-authored-by: Jan Zajic <[email protected]>
Co-authored-by: Julian Beck <[email protected]>
Co-authored-by: Jason Wells <[email protected]>
Co-authored-by: Nathan Gotz <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
  • Loading branch information
14 people authored May 13, 2024
1 parent 2201daf commit e01463c
Show file tree
Hide file tree
Showing 8 changed files with 181 additions and 42 deletions.
18 changes: 12 additions & 6 deletions cmd/jiralert/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ import (
"encoding/json"
"flag"
"fmt"
"github.com/andygrunwald/go-jira"
"net/http"
"os"
"runtime"
"strconv"

"github.com/andygrunwald/go-jira"
"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/prometheus-community/jiralert/pkg/alertmanager"
Expand All @@ -36,9 +36,10 @@ import (
)

const (
unknownReceiver = "<unknown>"
logFormatLogfmt = "logfmt"
logFormatJSON = "json"
unknownReceiver = "<unknown>"
logFormatLogfmt = "logfmt"
logFormatJSON = "json"
defaultMaxDescriptionLength = 32767 // https://jira.atlassian.com/browse/JRASERVER-64351
)

var (
Expand All @@ -48,6 +49,10 @@ var (
logFormat = flag.String("log.format", logFormatLogfmt, "Log format to use ("+logFormatLogfmt+", "+logFormatJSON+")")
hashJiraLabel = flag.Bool("hash-jira-label", false, "if enabled: renames ALERT{...} to JIRALERT{...}; also hashes the key-value pairs inside of JIRALERT{...} in the created jira issue labels"+
"- this ensures that the label text does not overflow the allowed length in jira (255)")
updateSummary = flag.Bool("update-summary", true, "When false, jiralert does not update the summary of the existing jira issue, even when changes are spotted.")
updateDescription = flag.Bool("update-description", true, "When false, jiralert does not update the description of the existing jira issue, even when changes are spotted.")
reopenTickets = flag.Bool("reopen-tickets", true, "When false, jiralert does not reopen tickets.")
maxDescriptionLength = flag.Int("max-description-length", defaultMaxDescriptionLength, "Maximum length of Descriptions. Truncate to this size avoid server errors.")

// Version is the build version, set by make to latest git tag/hash via `-ldflags "-X main.Version=$(VERSION)"`.
Version = "<local build>"
Expand Down Expand Up @@ -121,13 +126,14 @@ func main() {
return
}

if retry, err := notify.NewReceiver(logger, conf, tmpl, client.Issue).Notify(&data, *hashJiraLabel); err != nil {
if retry, err := notify.NewReceiver(logger, conf, tmpl, client.Issue).Notify(&data, *hashJiraLabel, *updateSummary, *updateDescription, *reopenTickets, *maxDescriptionLength); err != nil {
var status int
if retry {
// Instruct Alertmanager to retry.
status = http.StatusServiceUnavailable
} else {
status = http.StatusInternalServerError
// Inaccurate, just letting Alertmanager know that it should not retry.
status = http.StatusBadRequest
}
errorHandler(w, status, err, conf.Name, &data, logger)
return
Expand Down
14 changes: 13 additions & 1 deletion examples/jiralert.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@ defaults:
api_url: https://jiralert.atlassian.net
user: jiralert
password: 'JIRAlert'
# Alternatively to user and password use a Personal Access Token
# personal_access_token: "Your Personal Access Token". See https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

# Exclude labels in JIRA issue.
# Exclude labels in JIRA issue.
exclude_keys: []

# The type of JIRA issue to create. Required.
Expand All @@ -26,6 +28,14 @@ defaults:
# Amount of time after being closed that an issue should be reopened, after which, a new issue is created.
# Optional (default: always reopen)
reopen_duration: 0h
# Static label that will be added to the JIRA ticket alongisde the JIRALERT{...} or ALERT{...} label
static_labels: ["custom"]
# Other projects are the projects to search for existing issues for the given alerts if
# the main project does not have it. If no issue was found in, the main projects will
# be used to create a new one. If the old issue is found in one of the other projects
# (first found is used in case of duplicates) that old project's issue will be used for
# alert updates instead of creating on in the main project.
other_projects: ["OTHER1", "OTHER2"]

# Receiver definitions. At least one must be defined.
receivers:
Expand All @@ -37,6 +47,8 @@ receivers:
add_group_labels: false
exclude_keys:
- pod
# Will be merged with the static_labels from the default map
static_labels: ["anotherLabel"]

- name: 'jira-xy'
project: XY
Expand Down
1 change: 1 addition & 0 deletions pkg/alertmanager/alertmanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ type Alert struct {
StartsAt time.Time `json:"startsAt"`
EndsAt time.Time `json:"endsAt"`
GeneratorURL string `json:"generatorURL"`
Fingerprint string `json:"fingerprint"`
}

// Alerts is a list of Alert objects.
Expand Down
8 changes: 8 additions & 0 deletions pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ type ReceiverConfig struct {

// Required issue fields
Project string `yaml:"project" json:"project"`
OtherProjects []string `yaml:"other_projects" json:"other_projects"`
IssueType string `yaml:"issue_type" json:"issue_type"`
Summary string `yaml:"summary" json:"summary"`
ReopenEnabled *bool `yaml:"reopen_enabled" json:"reopen_enabled"`
Expand All @@ -145,6 +146,7 @@ type ReceiverConfig struct {
WontFixResolution string `yaml:"wont_fix_resolution" json:"wont_fix_resolution"`
Fields map[string]interface{} `yaml:"fields" json:"fields"`
Components []string `yaml:"components" json:"components"`
StaticLabels []string `yaml:"static_labels" json:"static_labels"`

// ExcludeKeys settings
ExcludeKeys []string `yaml:"exclude_keys" json:"exclude_keys"`
Expand Down Expand Up @@ -326,6 +328,12 @@ func (c *Config) UnmarshalYAML(unmarshal func(interface{}) error) error {
}
}
}
if len(c.Defaults.StaticLabels) > 0 {
rc.StaticLabels = append(rc.StaticLabels, c.Defaults.StaticLabels...)
}
if len(c.Defaults.OtherProjects) > 0 {
rc.OtherProjects = append(rc.OtherProjects, c.Defaults.OtherProjects...)
}
}

if len(c.Receivers) == 0 {
Expand Down
53 changes: 48 additions & 5 deletions pkg/config/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ defaults:
# Amount of time after being closed that an issue should be reopened, after which, a new issue is created.
# Optional (default: always reopen)
reopen_duration: 0h
static_labels: ["defaultlabel"]
# Receiver definitions. At least one must be defined.
receivers:
Expand All @@ -55,6 +56,7 @@ receivers:
project: AB
# Copy all Prometheus labels into separate JIRA labels. Optional (default: false).
add_group_labels: false
static_labels: ["somelabel"]
- name: 'jira-xy'
project: XY
Expand Down Expand Up @@ -122,10 +124,11 @@ type receiverTestConfig struct {
ReopenState string `yaml:"reopen_state,omitempty"`
ReopenDuration string `yaml:"reopen_duration,omitempty"`

Priority string `yaml:"priority,omitempty"`
Description string `yaml:"description,omitempty"`
WontFixResolution string `yaml:"wont_fix_resolution,omitempty"`
AddGroupLabels bool `yaml:"add_group_labels,omitempty"`
Priority string `yaml:"priority,omitempty"`
Description string `yaml:"description,omitempty"`
WontFixResolution string `yaml:"wont_fix_resolution,omitempty"`
AddGroupLabels bool `yaml:"add_group_labels,omitempty"`
StaticLabels []string `yaml:"static_labels" json:"static_labels"`

AutoResolve *AutoResolve `yaml:"auto_resolve" json:"auto_resolve"`

Expand Down Expand Up @@ -328,8 +331,9 @@ func TestReceiverOverrides(t *testing.T) {
{"WontFixResolution", "Won't Fix", "Won't Fix"},
{"AddGroupLabels", false, false},
{"AutoResolve", &AutoResolve{State: "Done"}, &autoResolve},
{"StaticLabels", []string{"somelabel"}, []string{"somelabel"}},
} {
optionalFields := []string{"Priority", "Description", "WontFixResolution", "AddGroupLabels", "AutoResolve"}
optionalFields := []string{"Priority", "Description", "WontFixResolution", "AddGroupLabels", "AutoResolve", "StaticLabels"}
defaultsConfig := newReceiverTestConfig(mandatoryReceiverFields(), optionalFields)
receiverConfig := newReceiverTestConfig([]string{"Name"}, optionalFields)

Expand Down Expand Up @@ -383,6 +387,8 @@ func newReceiverTestConfig(mandatory []string, optional []string) *receiverTestC
value = reflect.ValueOf(true)
} else if name == "AutoResolve" {
value = reflect.ValueOf(&AutoResolve{State: "Done"})
} else if name == "StaticLabels" {
value = reflect.ValueOf([]string{})
} else {
value = reflect.ValueOf(name)
}
Expand Down Expand Up @@ -459,3 +465,40 @@ func TestAutoResolveConfigDefault(t *testing.T) {
configErrorTestRunner(t, config, "bad config in defaults section: state cannot be empty")

}

func TestStaticLabelsConfigMerge(t *testing.T) {

for i, test := range []struct {
defaultValue []string
receiverValue []string
expectedElements []string
}{
{[]string{"defaultlabel"}, []string{"receiverlabel"}, []string{"defaultlabel", "receiverlabel"}},
{[]string{}, []string{"receiverlabel"}, []string{"receiverlabel"}},
{[]string{"defaultlabel"}, []string{}, []string{"defaultlabel"}},
{[]string{}, []string{}, []string{}},
} {
mandatory := mandatoryReceiverFields()

defaultsConfig := newReceiverTestConfig(mandatory, []string{})
defaultsConfig.StaticLabels = test.defaultValue

receiverConfig := newReceiverTestConfig([]string{"Name"}, []string{"StaticLabels"})
receiverConfig.StaticLabels = test.receiverValue

config := testConfig{
Defaults: defaultsConfig,
Receivers: []*receiverTestConfig{receiverConfig},
Template: "jiralert.tmpl",
}

yamlConfig, err := yaml.Marshal(&config)
require.NoError(t, err)

cfg, err := Load(string(yamlConfig))
require.NoError(t, err)

receiver := cfg.Receivers[0]
require.ElementsMatch(t, receiver.StaticLabels, test.expectedElements, "Elements should match (failing index: %v)", i)
}
}
77 changes: 50 additions & 27 deletions pkg/notify/notify.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import (
"time"

"github.com/andygrunwald/go-jira"

"github.com/go-kit/log"
"github.com/go-kit/log/level"
"github.com/pkg/errors"
Expand Down Expand Up @@ -61,7 +60,7 @@ func NewReceiver(logger log.Logger, c *config.ReceiverConfig, t *template.Templa
}

// Notify manages JIRA issues based on alertmanager webhook notify message.
func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool) (bool, error) {
func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool, updateSummary bool, updateDescription bool, reopenTickets bool, maxDescriptionLength int) (bool, error) {
project, err := r.tmpl.Execute(r.conf.Project, data)
if err != nil {
return false, errors.Wrap(err, "generate project from template")
Expand All @@ -87,19 +86,30 @@ func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool) (bool, er
return false, errors.Wrap(err, "render issue description")
}

if len(issueDesc) > maxDescriptionLength {
level.Warn(r.logger).Log("msg", "truncating description", "original", len(issueDesc), "limit", maxDescriptionLength)
issueDesc = issueDesc[:maxDescriptionLength]
}

if issue != nil {

// Update summary if needed.
if issue.Fields.Summary != issueSummary {
retry, err := r.updateSummary(issue.Key, issueSummary)
if err != nil {
return retry, err
if updateSummary {
if issue.Fields.Summary != issueSummary {
level.Debug(r.logger).Log("updateSummaryDisabled executing")
retry, err := r.updateSummary(issue.Key, issueSummary)
if err != nil {
return retry, err
}
}
}

if issue.Fields.Description != issueDesc {
retry, err := r.updateDescription(issue.Key, issueDesc)
if err != nil {
return retry, err
if updateDescription {
if issue.Fields.Description != issueDesc {
retry, err := r.updateDescription(issue.Key, issueDesc)
if err != nil {
return retry, err
}
}
}

Expand All @@ -123,18 +133,19 @@ func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool) (bool, er
return false, nil
}

if r.conf.WontFixResolution != "" && issue.Fields.Resolution != nil &&
issue.Fields.Resolution.Name == r.conf.WontFixResolution {
level.Info(r.logger).Log("msg", "issue was resolved as won't fix, not reopening", "key", issue.Key, "label", issueGroupLabel, "resolution", issue.Fields.Resolution.Name)
return false, nil
}
if reopenTickets {
if r.conf.WontFixResolution != "" && issue.Fields.Resolution != nil &&
issue.Fields.Resolution.Name == r.conf.WontFixResolution {
level.Info(r.logger).Log("msg", "issue was resolved as won't fix, not reopening", "key", issue.Key, "label", issueGroupLabel, "resolution", issue.Fields.Resolution.Name)
return false, nil
}

if r.conf.ReopenEnabled != nil && !*r.conf.ReopenEnabled {
level.Debug(r.logger).Log("msg", "reopening disabled, skipping search for existing issue")
} else {
level.Info(r.logger).Log("msg", "issue was recently resolved, reopening", "key", issue.Key, "label", issueGroupLabel)
return r.reopen(issue.Key)
}

level.Debug(r.logger).Log("Did not update anything")
return false, nil
}

if len(data.Alerts.Firing()) == 0 {
Expand All @@ -149,13 +160,15 @@ func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool) (bool, er
return false, errors.Wrap(err, "render issue type")
}

staticLabels := r.conf.StaticLabels

issue = &jira.Issue{
Fields: &jira.IssueFields{
Project: jira.Project{Key: project},
Type: jira.IssueType{Name: issueType},
Description: issueDesc,
Summary: issueSummary,
Labels: []string{issueGroupLabel},
Labels: append(staticLabels, issueGroupLabel),
Unknowns: tcontainer.NewMarshalMap(),
},
}
Expand All @@ -182,7 +195,7 @@ func (r *Receiver) Notify(data *alertmanager.Data, hashJiraLabel bool) (bool, er

if r.conf.AddGroupLabels {
for k, v := range data.GroupLabels {
issue.Fields.Labels = append(issue.Fields.Labels, fmt.Sprintf("%s=%q", k, v))
issue.Fields.Labels = append(issue.Fields.Labels, fmt.Sprintf("%s=%.200q", k, v))
}
}

Expand Down Expand Up @@ -276,8 +289,10 @@ func toGroupTicketLabel(groupLabels alertmanager.KV, hashJiraLabel bool, exclude
return strings.Replace(buf.String(), " ", "", -1)
}

func (r *Receiver) search(project, issueLabel string) (*jira.Issue, bool, error) {
query := fmt.Sprintf("project=\"%s\" and labels=%q order by resolutiondate desc", project, issueLabel)
func (r *Receiver) search(projects []string, issueLabel string) (*jira.Issue, bool, error) {
// Search multiple projects in case issue was moved and further alert firings are desired in existing JIRA.
projectList := "'" + strings.Join(projects, "', '") + "'"
query := fmt.Sprintf("project in(%s) and labels=%q order by resolutiondate desc", projectList, issueLabel)
options := &jira.SearchOptions{
Fields: []string{"summary", "status", "resolution", "resolutiondate"},
MaxResults: 2,
Expand Down Expand Up @@ -305,8 +320,15 @@ func (r *Receiver) search(project, issueLabel string) (*jira.Issue, bool, error)
}

func (r *Receiver) findIssueToReuse(project string, issueGroupLabel string) (*jira.Issue, bool, error) {
projectsToSearch := []string{project}
// In case issue was moved to a different project, include the other configured projects in search (if any).
for _, other := range r.conf.OtherProjects {
if other != project {
projectsToSearch = append(projectsToSearch, other)
}
}

issue, retry, err := r.search(project, issueGroupLabel)
issue, retry, err := r.search(projectsToSearch, issueGroupLabel)
if err != nil {
return nil, retry, err
}
Expand Down Expand Up @@ -383,10 +405,11 @@ func handleJiraErrResponse(api string, resp *jira.Response, err error, logger lo
}

if resp != nil && resp.StatusCode/100 != 2 {
retry := resp.StatusCode == 500 || resp.StatusCode == 503
retry := resp.StatusCode == 500 || resp.StatusCode == 503 || resp.StatusCode == 429
// Sometimes go-jira consumes the body (e.g. in `Search`) and includes it in the error message;
// sometimes (e.g. in `Create`) it doesn't. Include both the error and the body, just in case.
body, _ := io.ReadAll(resp.Body)
// go-jira error message is not particularly helpful, replace it
return retry, errors.Errorf("JIRA request %s returned status %s, body %q", resp.Request.URL, resp.Status, string(body))
return retry, errors.Errorf("JIRA request %s returned status %s, error %q, body %q", resp.Request.URL, resp.Status, err, body)
}
return false, errors.Wrapf(err, "JIRA request %s failed", api)
}
Expand All @@ -413,6 +436,6 @@ func (r *Receiver) doTransition(issueKey string, transitionState string) (bool,
return false, nil
}
}
return false, errors.Errorf("JIRA state %q does not exist or no transition possible for %s", r.conf.ReopenState, issueKey)
return false, errors.Errorf("JIRA state %q does not exist or no transition possible for %s", transitionState, issueKey)

}
Loading

0 comments on commit e01463c

Please sign in to comment.