v0.9.0-rc.0

RC release for v0.9.0

See changes here

v0.8.1

Fixed

• #1632 Removes the duplicated external labels detection on Thanos Querier. It is warning only now. Also made Store Gateway compatible with older Querier versions.
  NOTE: thanos_store_nodes_grpc_connections metric is now per external_labels and store_type. It is a recommended metric for Querier storeAPIs. thanos_store_node_info is marked as obsolete and will be removed in next release.
  NOTE2: Store Gateway is now advertising artificial: "@thanos_compatibility_store_type=store" label. This is to have the current Store Gateway compatible with Querier pre v0.8.0.
  This label can be disabled by hidden debug.advertise-compatibility-label=false flag on Store Gateway.

See full CHANGELOG here

v0.8.0

Lots of improvements this release! Outstanding items:

• First Katacoda tutorial! 🐱
• Fixed Deletion order causing Compactor to produce not needed 👻 blocks with missing random files.
• Store GW memory improvements (more to come!).
• Querier allows multiple deduplication labels.
• Both Compactor and Store Gateway can be sharded within the same bucket using relabelling!
• Sidecar exposed data from Prometheus can be now limited to given min-time (e.g 3h only).
• Numerous Thanos Receive improvements.

Make sure you check out Prometheus 2.13.0 as well. New release drastically improves usage and resource consumption of both Prometheus and sidecar with Thanos: https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/

Added

• #1619 Thanos sidecar allows to limit min time range for data it exposes from Prometheus.
• #1583 Thanos sharding:
  • Add relabel config (--selector.relabel-config-file and selector.relabel-config) into Thanos Store and Compact components.
    Selecting blocks to serve depends on the result of block labels relabeling.
  • For store gateway, advertise labels from "approved" blocks.
• #1540 Thanos Downsample added /-/ready and /-/healthy endpoints.
• #1538 Thanos Rule added /-/ready and /-/healthy endpoints.
• #1537 Thanos Receive added /-/ready and /-/healthy endpoints.
• #1460 Thanos Store Added /-/ready and /-/healthy endpoints.
• #1534 Thanos Query Added /-/ready and /-/healthy endpoints.
• #1533 Thanos inspect now supports the timeout flag.
• #1496 Thanos Receive now supports setting block duration.
• #1362 Optional replicaLabels param for /query and
  /query_range querier endpoints. When provided overwrite the query.replica-label cli flags.
• #1482 Thanos now supports Elastic APM as tracing provider.
• #1612 Thanos Rule added resendDelay flag.
• #1480 Thanos Receive flushes storage on hashring change.
• #1613 Thanos Receive now traces forwarded requests.

Changed

• #1362 query.replica-label configuration can be provided more than
  once for multiple deduplication labels like: --query.replica-label=prometheus_replica --query.replica-label=service.
• #1581 Thanos Store now can use smaller buffer sizes for Bytes pool; reducing memory for some requests.
• #1622 & #1590 Updated to Go 1.13.1
• #1498 Thanos Receive change flag labels to label to be consistent with other commands.

Fixed

• #1525 Thanos now deletes block's file in correct order allowing to detect partial blocks without problems.
• #1505 Thanos Store now removes invalid local cache blocks.
• #1587 Thanos Sidecar cleanups all cache dirs after each compaction run.
• #1582 Thanos Rule correctly parses Alertmanager URL if there is more + in it.
• #1544 Iterating over object store is resilient to the edge case for some providers.
• #1469 Fixed Azure potential failures (EOF) when requesting more data then blob has.
• #1512 Thanos Store fixed memory leak for chunk pool.
• #1488 Thanos Rule now now correctly links to query URL from rules and alerts.

See full CHANGELOG here

v0.7.0

v0.7.0

Accepted into CNCF:

• Thanos moved to new repository https://github.com/thanos-io/thanos
• Docker images moved to https://quay.io/thanos/thanos and mirrored at https://hub.docker.com/r/thanosio/thanos
• Slack moved to https://slack.cncf.io #thanos/#thanos-dev / #thanos-prs

Added

• #1378 Thanos Receive now exposes thanos_receive_config_hash, thanos_receive_config_last_reload_successful and thanos_receive_config_last_reload_success_timestamp_seconds metrics to track latest configuration change
• #1268 Thanos Sidecar added support for newest Prometheus streaming remote read added here. This massively improves memory required by single
  request for both Prometheus and sidecar. Single requests now should take constant amount of memory on sidecar, so resource consumption prediction is now straightforward. This will be used if you have Prometheus 2.13 or 2.12-master.
• #1358 Added part_size configuration option for HTTP multipart requests minimum part size for S3 storage type
• #1363 Thanos Receive now exposes thanos_receive_hashring_nodes and thanos_receive_hashring_tenants metrics to monitor status of hash-rings
• #1395 Thanos Sidecar added /-/ready and /-/healthy endpoints to Thanos sidecar.
• #1297 Thanos Compact added /-/ready and /-/healthy endpoints to Thanos compact.
• #1431 Thanos Query added hidden flag to allow the use of downsampled resolution data for instant queries.
• #1408 Thanos Store Gateway can now allow the specifying of supported time ranges it will serve (time sharding). Flags: min-time & max-time

Changed

• #1414 Upgraded important dependencies: Prometheus to 2.12-rc.0. TSDB is now part of Prometheus.
• #1380 Upgraded important dependencies: Prometheus to 2.11.1 and TSDB to 0.9.1. Some changes affecting Querier:
  • [ENHANCEMENT] Query performance improvement: Efficient iteration and search in HashForLabels and HashWithoutLabels. #5707
  • [ENHANCEMENT] Optimize queries using regexp for set lookups. tsdb#602
  • [BUGFIX] prometheus_tsdb_compactions_failed_total is now incremented on any compaction failure. tsdb#613
  • [BUGFIX] PromQL: Correctly display {name="a"}.
• #1338 Thanos Query still warns on store API duplicate, but allows a single one from duplicated set. This is gracefully warn about the problematic logic and not disrupt immediately.
• #1385 Thanos Compact exposes flag to disable downsampling downsampling.disable.

Fixed

• #1327 Thanos Query /series API end-point now properly returns an empty array just like Prometheus if there are no results
• #1302 Thanos now efficiently reuses HTTP keep-alive connections
• #1371 Thanos Receive fixed race condition in hashring
• #1430 Thanos fixed value of GOMAXPROCS inside container.

Deprecated

• #1458 Thanos Query and Receive now use common instrumentation middleware. As as result, for sake of http_requests_total and http_request_duration_seconds_bucket; Thanos Query no longer exposes thanos_query_api_instant_query_duration_seconds, thanos_query_api_range_query_duration_second metrics and Thanos Receive no longer exposes thanos_http_request_duration_seconds, thanos_http_requests_total, thanos_http_response_size_bytes.
• #1423 Thanos Bench deprecated.

v0.7.0-rc.0

TLDR; Move to CNCF, Added steaming between Prometheus and Sidecar, allow time sharding on Store Gateway and many bug fixes.

More detailed information on the release can be found here https://github.com/thanos-io/thanos/blob/master/CHANGELOG.md

v0.6.1

v0.6.1

v0.6.0

Added

TL;DR: Jaeger tracing support (tracing flag changed), various observability improvements, Thanos receiver improvements, improvement external label propagation, including federated Queriers (!) and other fixes.

NOTE: Thanks to improved external labels propagation, if you run have duplicate queries in your Querier configuration with hierarchical federation of multiple Queries, Thanos now will detect this case and block all duplicates. New releases (potentially in v0.6.1) will just warn and block all but one.

• #1097 Added thanos check rules linter for Thanos rule rules files.

• #1253 Add support for specifying a maximum amount of retries when using Azure Blob storage (default: no retries).

• #1244 Thanos Compact now exposes new metrics thanos_compact_downsample_total and thanos_compact_downsample_failures_total which are useful to catch when errors happen

• #1260 Thanos Query/Rule now exposes metrics thanos_querier_store_apis_dns_provider_results and thanos_ruler_query_apis_dns_provider_results which tell how many addresses were configured and how many were actually discovered respectively

• #1248 Add a web UI to show the state of remote storage.

• #1217 Thanos Receive gained basic hashring support

• #1262 Thanos Receive got a new metric thanos_http_requests_total which shows how many requests were handled by it

• #1243 Thanos Receive got an ability to forward time series data between nodes. Now you can pass the hashring configuration via --receive.hashrings-file; the refresh interval --receive.hashrings-file-refresh-interval; the name of the local node's name --receive.local-endpoint; and finally the header's name which is used to determine the tenant --receive.tenant-header.

• #1147 Support for the Jaeger tracer has been added!

breaking New common flags were added for configuring tracing: --tracing.config-file and --tracing.config. You can either pass a file to Thanos with the tracing configuration or pass it in the command line itself. Old --gcloudtrace.* flags were removed ⚠️

To migrate over the old --gcloudtrace.* configuration, your tracing configuration should look like this:

---
type: STACKDRIVER
config:
- service_name: 'foo'
  project_id: '123'
  sample_factor: 123

The other type you can use is JAEGER now. The config keys and values are Jaeger specific and you can find all of the information here.

Changed

• #1284 Add support for multiple label-sets in Info gRPC service. This deprecates the single Labels slice of the InfoResponse, in a future release backward compatible handling for the single set of Labels will be removed. Upgrading to v0.6.0 or higher is advised.

• #1314 Removes http_request_duration_microseconds (Summary) and adds http_request_duration_seconds (Histogram) from http server instrumentation used in Thanos APIs and UIs.

• #1287 Sidecar now waits on Prometheus' external labels before starting the uploading process

• #1261 Thanos Receive now exposes metrics thanos_http_request_duration_seconds and thanos_http_response_size_bytes properly of each handler

• #1274 Iteration limit has been lifted from the LRU cache so there should be no more spam of error messages as they were harmless

• #1321 Thanos Query now fails early on a query which only uses external labels - this improves clarity in certain situations

Fixed

• #1227 Some context handling issues were fixed in Thanos Compact; some unnecessary memory allocations were removed in the hot path of Thanos Store.

• #1183 Compactor now correctly propogates retriable/haltable errors which means that it will not unnecessarily restart if such an error occurs

• #1231 Receive now correctly handles SIGINT and closes without deadlocking

• #1278 Fixed inflated values problem with sum() on Thanos Query

• #1280 Fixed a problem with concurrent writes to a map in Thanos Query while rendering the UI

• #1311 Fixed occasional panics in Compact and Store when using Azure Blob cloud storage caused by lack of error checking in client library.

• #1322 Removed duplicated closing of the gRPC listener - this gets rid of harmless messages like store gRPC listener: close tcp 0.0.0.0:10901: use of closed network connection when those programs are being closed

Deprecated

• #1216 the old "Command-line flags" has been removed from Thanos Query UI since it was not populated and because we are striving for consistency

v0.6.0-rc.0

v0.6.0-rc.0

v0.5.0

TL;DR: Store LRU cache is no longer leaking, Upgraded Thanos UI to Prometheus 2.9, Fixed auto-downsampling, Moved to Go 1.12.5 and more.

This version moved tarballs to Golang 1.12.5 from 1.11 as well, so same warning applies if you use container_memory_usage_bytes from cadvisor. Use container_memory_working_set_bytes instead.

breaking As announced couple of times this release also removes gossip with all configuration flags (--cluster.*).

Fixed

• #1142 fixed major leak on store LRU cache for index items (postings and series).
• #1163 sidecar is no longer blocking for custom Prometheus versions/builds. It only checks if flags return non 404, then it performs optional checks.
• #1146 store/bucket: make getFor() work with interleaved resolutions.
• #1157 querier correctly handles duplicated stores when some store changes external labels in place.

Added

• #1094 Allow configuring the response header timeout for the S3 client.

Changed

• #1118 breaking swift: Added support for cross-domain authentication by introducing userDomainID, userDomainName, projectDomainID, projectDomainName.
  The outdated terms tenantID, tenantName are deprecated and have been replaced by projectID, projectName.

• #1066 Upgrade Thanos ui to Prometheus v2.9.1.

  Changes from the upstream:

  • query:
    • [ENHANCEMENT] Update moment.js and moment-timezone.js PR #4679
    • [ENHANCEMENT] Support to query elements by a specific time PR #4764
    • [ENHANCEMENT] Update to Bootstrap 4.1.3 PR #5192
    • [BUGFIX] Limit number of merics in prometheus UI PR #5139
    • [BUGFIX] Web interface Quality of Life improvements PR #5201
  • rule:
    • [ENHANCEMENT] Improve rule views by wrapping lines PR #4702
    • [ENHANCEMENT] Show rule evaluation errors on rules page PR #4457

• #1156 Moved CI and docker multistage to Golang 1.12.5 for latest mem alloc improvements.

• #1103 Updated go-cos deps. (COS bucket client).

• #1149 Updated google Golang API deps (GCS bucket client).

• #1190 Updated minio deps (S3 bucket client). This fixes minio retries.

• #1133 Use prometheus v2.9.2, common v0.4.0 & tsdb v0.8.0.

  Changes from the upstreams:

  • store gateway:
    • [ENHANCEMENT] Fast path for EmptyPostings cases in Merge, Intersect and Without.
  • store gateway & compactor:
    • [BUGFIX] Fix fd and vm_area leak on error path in chunks.NewDirReader.
    • [BUGFIX] Fix fd and vm_area leak on error path in index.NewFileReader.
  • query:
    • [BUGFIX] Make sure subquery range is taken into account for selection #5467
    • [ENHANCEMENT] Check for cancellation on every step of a range evaluation. #5131
    • [BUGFIX] Exponentation operator to drop metric name in result of operation. #5329
    • [BUGFIX] Fix output sample values for scalar-to-vector comparison operations. #5454
  • rule:
    • [BUGFIX] Reload rules: copy state on both name and labels. #5368

Deprecated

• #1008 breaking Removed Gossip implementation. All --cluster.* flags removed and Thanos will error out if any is provided.

See full CHANGELOG here

v0.5.0-rc.0

v0.5.0-rc.0