add CircuitBreaker timesTripped metric

[nginthfs]

This PR adds a metric for each CircuitBreaker that is configured in Solr. The metrics are named like so:

# times_tripped_<name of circuit breaker class>_<request type>
times_tripped_load_average_circuit_breaker_query

All metrics from every CircuitBreaker of the same class and request type are rolled up into this one metric, and reported through the PrometheusMetricsServlet.

Testing

Tested by adding this to a config on my local machine:

...
  <circuitBreaker class="solr.LoadAverageCircuitBreaker">
    <double name="threshold">1</double>
    <arr name="requestTypes">
      <str>update</str>
    </arr>
  </circuitBreaker>

  <circuitBreaker class="solr.LoadAverageCircuitBreaker">
    <double name="threshold">1</double>
    <arr name="requestTypes">
      <str>query</str>
    </arr>
  </circuitBreaker>
...

Then, issuing some requests to a collection, and seeing the circuit breaker activate. Then, I checked http://localhost:8983/solr/metrics to make sure this showed up:

# HELP times_tripped_load_average_circuit_breaker_query number of times circuit has been tripped
# TYPE times_tripped_load_average_circuit_breaker_query counter
times_tripped_load_average_circuit_breaker_query 1
# HELP times_tripped_load_average_circuit_breaker_update number of times circuit has been tripped
# TYPE times_tripped_load_average_circuit_breaker_update counter
times_tripped_load_average_circuit_breaker_update 0

Also tested that Solr functions as normal when no CircuitBreakers are configured.

[nginthfs]

Reworking this a bit, will update when ready for review again.

[nginthfs]

Okay this is ready for review again!

[hiteshk25]

LGTM

[hiteshk25]

QQ: it does add node name as label some where?

[nginthfs]

QQ: it does add node name as label some where?

I believe all Solr metrics have nodename added to them automatically as a label later on in the pipeline. For example, metrics_qtime, which is configured in a similar way in PrometheusMetricsServlet has nodename as a label if you look at it in Grafana. I'm pretty sure this happens in the Prometheus k8s deployment.

[nginthfs]

QQ: it does add node name as label some where?

I believe all Solr metrics have nodename added to them automatically as a label later on in the pipeline. For example, metrics_qtime, which is configured in a similar way in PrometheusMetricsServlet has nodename as a label if you look at it in Grafana. I'm pretty sure this happens in the Prometheus k8s deployment.

Confirmed this by deploying to playpen.