How to Convert Simple JSON to Prometheus Format

[yaohongkok]

Hello everyone. Dear @rustycl0ck and @SuperQ, hope you are doing well.

I am creating an issue because I am stuck trying to follow @rustycl0ck's suggestion. I hope someone can shed some light on this matter.

I have the following data in bad_examples/mydata1.json

{
    "name": "godlyrustyclock",
    "statusCode": "OK"
}

andbad_examples/mydata2.json

{
    "name": "godlyrustyclock",
    "statusCode": "BAD"
}

I will host this data via a python server:

$ cd bad_examples
$ python -m http.server

So, I can get my data when I do the following:

$ curl http://localhost:8000/mydata1.json
$ curl http://localhost:8000/mydata2.json

I would like convert statusCode of OK to 0 and statusCode of BAD to 1. This is my expected output:

# When statusCode is OK

hc_status{name="godlyrustyclock"} 0


# When statusCode is BAD

hc_status{name="godlyrustyclock"} 1

I am encountering some issues to achieve the above result without using our previously pull request #172.

I tried @rustycl0ck's suggestion and ended up with a simple config known as config_simple.yml stored under bad_examples/:

---
modules:
  default:
    metrics:
    - name: hc
      path: '$[?(@.statusCode == "OK"]'
      type: object
      help: Health Check
      labels:
        name: '${.name}'
      values:
        status: 0

So, I run the json exporter as:

go run main.go --config.file bad_examples/config_simple.yml

and I make the call to the exporter:

curl "http://localhost:7979/probe?module=default&target=http://localhost:8000/mydata1.json"

I get the following output:

hc_status{name=""} 0

This is the first issue I encountered.

It seems like the exporter expect the labels to be under the path and therefore it is return empty string. This is one of the reasons why our pull request #172 use the path of "$", which uses the root.

Ok, now let's expand the YML config further to be bad_examples/config.yml:

---
modules:
  default:
    metrics:
    - name: hc
      path: '$[?(@.statusCode == "OK"]'
      type: object
      help: Health Check
      labels:
        name: '${.name}'
      values:
        status: 0
    - name: hc
      path: '$[?(@.statusCode == "BAD"]'
      type: object
      help: Health Check
      labels:
        name: '${.name}'
      values:
        status: 1

Let's set up the exporter again:

go run main.go --config.file bad_examples/config.yml

Now, let's make a call to the exporter:

curl "http://localhost:7979/probe?module=default&target=http://localhost:8000/mydata1.json"

I get the following error:

An error has occurred while serving metrics:

collected metric "hc_status" { label:<name:"name" value:"" > untyped:<value:1 > } was collected before with the same name and label values

This is issue #2. I looked at the code with extra logging and I cannot really figure out what is happening.

I think the problem here is at the config.yml layer. Maybe I am not familiar enough with how to generate the right YML file. I hope someone can help me with this matter. Thank you.

Or if you see pull request #172 being helpful to solve this problem, this could be a good reason to merge the changes back into this repo.

[rustycl0ck]

Hi @yaohongkok, Just an example (was also mentioned in #172 (comment) briefly as an alternative solution): Let's say some server provides its availability status in several different ways (ACTIVE, INACTIVE, OVERLOADED, SUSPENDED). Then you can just use a single metric with static value of 1 and add the status information in the labels.

• Example data:

  json_exporter/examples/data.json

  Lines 1 to 25 in 1c1ae57

  	{
  	"counter": 1234,
  	"timestamp": 1657568506,
  	"values": [
  	{
  	"id": "id-A",
  	"count": 1,
  	"some_boolean": true,
  	"state": "ACTIVE"
  	},
  	{
  	"id": "id-B",
  	"count": 2,
  	"some_boolean": true,
  	"state": "INACTIVE"
  	},
  	{
  	"id": "id-C",
  	"count": 3,
  	"some_boolean": false,
  	"state": "ACTIVE"
  	}
  	],
  	"location": "mars"
  	}

• Example config:

  ---
  modules:
    default:
      metrics:
      - name: example_state
        type: object
        help: Example of checking availability state
        path: '{.values[*]}'
        labels:
          id: '{.id}'
          state: '{.state}'
        values:
          availability: 1      # static value

• Example scrape:

  $ curl "http://localhost:7979/probe?module=default&target=http://localhost:8000/examples/data.json"
  # HELP example_state_availability Example of checking different states
  # TYPE example_state_availability untyped
  example_state_availability{id="id-A",state="ACTIVE"} 1
  example_state_availability{id="id-B",state="INACTIVE"} 1
  example_state_availability{id="id-C",state="ACTIVE"} 1

• Example alert: An alert can be sent by filtering for the required information. Prometheus query example_state_availability{state="INACTIVE"} != 0 will give information about any servers which are in INACTIVE (or change to any other labels of your interest). You can use this for sending alerts or just gathering metrics about different states of different servers at any given time.

This way, you can keep your prometheus configuration file simple as shown above (don't really need to worry about n number of availability enumerations provided by the server). For alerting, you can send various alerts based on whichever states are of your interest.

Just one thing to note from the error you mentioned in the issue: apply enough labels (or the right labels) for uniquely identifying information. In the given example, if id: '{.id}' is not added as a lable, then the exporter will throw the error you saw above. This is because if there is not enough information to distinguish the value of two metrics, then there is no point of collecting the same information more than once in a single scrape. The exporter would just think that it is a repeated metric, and thus will throw the error you saw above and the scrape will fail.