v0.14.0

Highlights:

• Upgrade to Prometheus @3268eac2ddda which is after v2.18.1.
  • TSDB now does memory-mapping of Head chunks and reduces memory usage.
• Querier performs concurrent select per query.
• Changed bucket tool bucket verify --id-whitelist flag to --id.
• Store removed support to the legacy index.cache.json. The hidden flag --store.disable-index-header was removed.
• Store decreased memory allocations while querying block's index.

Thanks all for contributing! 🤗

Fixed

• #2637 Compact: Detect retryable errors that are inside of a wrapped tsdb.MultiError.
• #2648 Store: Allow index cache and caching bucket to be configured at the same time.
• #2705 minio-go: Added support for af-south-1 and eu-south-1 regions.
• #2728 Query: Fixed panics when using a larger number of replica labels with short series label sets.
• #2787 Update Prometheus mod to pull in prometheus/prometheus#7414.
• #2807 Store: Decreased memory allocations while querying block's index.
• #2809 Query: /api/v1/stores now guarantees to return a string in the lastError field.

Changed

• #2658 #2703 Upgrade to Prometheus @3268eac2ddda which is after v2.18.1.
  • TSDB now does memory-mapping of Head chunks and reduces memory usage.
• #2667 Store: Removed support to the legacy index.cache.json. The hidden flag --store.disable-index-header was removed.
• #2613 Store: Renamed the caching bucket config option chunk_object_size_ttl to chunk_object_attrs_ttl.
• #2667 Compact: The deprecated flag --index.generate-missing-cache-file and the metric thanos_compact_generated_index_total were removed.
• 2603 Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).
• #2671 breaking Tools: Bucket replicate flag --resolution is now in Go duration format.
• #2671 Tools: Bucket replicate now replicates by default all blocks.
• #2739 Changed bucket tool bucket verify --id-whitelist flag to --id.
• #2748 Upgrade Prometheus to @66dfb951c4ca which is after v2.19.0.
  • PromQL now allows us to executed concurrent selects.

Added

• #2671 Tools: Bucket replicate now allows passing repeated --compaction and --resolution flags.
• #2657 Querier: Add the ability to perform concurrent select request per query.
• #2754 UI: Add stores page in the React UI.
• #2752 Compact: Add flag --block-viewer.global.sync-block-interval to configure metadata sync interval for the bucket UI.

v0.14.0-rc.1

Fixed

• #2848 Querier: Fixed a panic that occured when /api/v1/stores requested.
  • The issue was introduced in #2809 within the current release cycle.

Thanks all for contributing! 🤗

v0.14.0-rc.0

Highlights:

• Upgrade to Prometheus @3268eac2ddda which is after v2.18.1.
  • TSDB now does memory-mapping of Head chunks and reduces memory usage.
• Querier performs concurrent select per query.
• Changed bucket tool bucket verify --id-whitelist flag to --id.
• Store removed support to the legacy index.cache.json. The hidden flag --store.disable-index-header was removed.
• Store decreased memory allocations while querying block's index.

Thanks all for contributing! 🤗

Fixed

• #2637 Compact: Detect retryable errors that are inside of a wrapped tsdb.MultiError.
• #2648 Store: Allow index cache and caching bucket to be configured at the same time.
• #2705 minio-go: Added support for af-south-1 and eu-south-1 regions.
• #2728 Query: Fixed panics when using a larger number of replica labels with short series label sets.
• #2787 Update Prometheus mod to pull in prometheus/prometheus#7414.
• #2807 Store: Decreased memory allocations while querying block's index.
• #2809 Query: /api/v1/stores now guarantees to return a string in the lastError field.

Changed

• #2658 #2703 Upgrade to Prometheus @3268eac2ddda which is after v2.18.1.
  • TSDB now does memory-mapping of Head chunks and reduces memory usage.
• #2667 Store: Removed support to the legacy index.cache.json. The hidden flag --store.disable-index-header was removed.
• #2613 Store: Renamed the caching bucket config option chunk_object_size_ttl to chunk_object_attrs_ttl.
• #2667 Compact: The deprecated flag --index.generate-missing-cache-file and the metric thanos_compact_generated_index_total were removed.
• 2603 Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).
• #2671 breaking Tools: Bucket replicate flag --resolution is now in Go duration format.
• #2671 Tools: Bucket replicate now replicates by default all blocks.
• #2739 Changed bucket tool bucket verify --id-whitelist flag to --id.
• #2748 Upgrade Prometheus to @66dfb951c4ca which is after v2.19.0.
  • PromQL now allows us to executed concurrent selects.

Added

• #2671 Tools: Bucket replicate now allows passing repeated --compaction and --resolution flags.
• #2657 Querier: Add the ability to perform concurrent select request per query.
• #2754 UI: Add stores page in the React UI.
• #2752 Compact: Add flag --block-viewer.global.sync-block-interval to configure metadata sync interval for the bucket UI.

v0.13.0

Highlights:

• Rare overlapping block issue fixed
• Deduplication bug with counters fixed
• Receiver is now multi-tenant (multi TSDB)
• Added @memcached meta and chunk caching. (e.g allows to significantly limit objstore traffic)
• TSDB isolation added for Ruler and Receive
• Querier performance significantly improved especially when used with duplicated Store APIs (e.g for HA purposes).

Thanks all for contributing! 🤗

Fixed

• #2548 Query: Fixed rare cases of double counter reset accounting when querying rate with deduplication enabled.
• #2536 S3: Fixed AWS STS endpoint url to https for Web Identity providers on AWS EKS.
• #2501 Query: Gracefully handle additional fields in SeriesResponse protobuf message that may be added in the future.
• #2568 Query: Don't close the connection of strict, static nodes if establishing a connection had succeeded but Info() call failed.
• #2615 Rule: Fix bugs where rules were out of sync.
• #2614 Tracing: Disabled Elastic APM Go Agent default tracer on initialization to disable the default metric gatherer.
• #2525 Query: Fixed logging for dns resolution error in the Query component.
• #2484 Query/Ruler: Fixed issue #2483, when web.route-prefix is set, it is added twice in HTTP router prefix.
• #2416 Bucket: Fixed issue #2416 bug in inspect --sort-by doesn't work correctly in all cases.
• #2719 Query: irate and resets use now counter downsampling aggregations.
• #2705 minio-go: Added support for af-south-1 and eu-south-1 regions.
• #2753 Sidecar, Receive, Rule: Fixed possibility of out of order uploads in error cases. This could potentially cause Compactor to create overlapping blocks.

Added

• #2012 Receive: Added multi-tenancy support (based on header)
• #2502 StoreAPI: Added hints field to SeriesResponse. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation specific.
• #2521 Sidecar: Added thanos_sidecar_reloader_reloads_failed_total, thanos_sidecar_reloader_reloads_total, thanos_sidecar_reloader_watch_errors_total, thanos_sidecar_reloader_watch_events_total and thanos_sidecar_reloader_watches metrics.
• #2412 UI: Added React UI from Prometheus upstream. Currently only accessible from Query component as only /graph endpoint is migrated.
• #2532 Store: Added hidden option --store.caching-bucket.config=<yaml content> (or --store.caching-bucket.config-file=<file.yaml>) for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage.
• #2579 Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• #2621 Receive: Added flag to configure forward request timeout. Receive write will complete request as soon as a quorum of writes succeeds.

Changed

• #2194 Updated to golang v1.14.2.
• #2505 Store: Removed obsolete thanos_store_node_info metric.
• 2513 Tools: Moved thanos bucket commands to thanos tools bucket, also
  moved thanos check rules to thanos tools rules-check. thanos tools rules-check also takes rules by --rules repeated flag not argument
  anymore.
• #2548 Store, Querier: remove duplicated chunks on StoreAPI.
• #2596 Updated Prometheus dependency to @cd73b3d33e064bbd846fc7a26dc8c313d46af382 which falls in between v2.17.0 and v2.18.0.
  • Receive,Rule: TSDB now supports isolation of append and queries.
  • Receive,Rule: TSDB now holds less WAL files after Head Truncation.
• #2450 Store: Added Regex-set optimization for label=~"a|b|c" matchers.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• 2603 Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).

v0.13.0-rc.2

Fixed

• #2548 Query: Fixed rare cases of double counter reset accounting when querying rate with deduplication enabled.
• #2536 S3: Fixed AWS STS endpoint url to https for Web Identity providers on AWS EKS.
• #2501 Query: Gracefully handle additional fields in SeriesResponse protobuf message that may be added in the future.
• #2568 Query: Don't close the connection of strict, static nodes if establishing a connection had succeeded but Info() call failed.
• #2615 Rule: Fix bugs where rules were out of sync.
• #2614 Tracing: Disabled Elastic APM Go Agent default tracer on initialization to disable the default metric gatherer.
• #2525 Query: Fixed logging for dns resolution error in the Query component.
• #2484 Query/Ruler: Fixed issue #2483, when web.route-prefix is set, it is added twice in HTTP router prefix.
• #2416 Bucket: Fixed issue #2416 bug in inspect --sort-by doesn't work correctly in all cases.
• #2719 Query: irate and resets use now counter downsampling aggregations.
• #2705 minio-go: Added support for af-south-1 and eu-south-1 regions.
• #2753 Sidecar, Receive, Rule: Fixed possibility of out of order uploads in error cases. This could potentially cause Compactor to create overlapping blocks.

Added

• #2012 Receive: Added multi-tenancy support (based on header)
• #2502 StoreAPI: Added hints field to SeriesResponse. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation specific.
• #2521 Sidecar: Added thanos_sidecar_reloader_reloads_failed_total, thanos_sidecar_reloader_reloads_total, thanos_sidecar_reloader_watch_errors_total, thanos_sidecar_reloader_watch_events_total and thanos_sidecar_reloader_watches metrics.
• #2412 UI: Added React UI from Prometheus upstream. Currently only accessible from Query component as only /graph endpoint is migrated.
• #2532 Store: Added hidden option --store.caching-bucket.config=<yaml content> (or --store.caching-bucket.config-file=<file.yaml>) for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage.
• #2579 Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• #2621 Receive: Added flag to configure forward request timeout. Receive write will complete request as soon as quorum of writes succeeds.

Changed

• #2194 Updated to golang v1.14.2.
• #2505 Store: Removed obsolete thanos_store_node_info metric.
• 2513 Tools: Moved thanos bucket commands to thanos tools bucket, also
  moved thanos check rules to thanos tools rules-check. thanos tools rules-check also takes rules by --rules repeated flag not argument
  anymore.
• #2548 Store, Querier: remove duplicated chunks on StoreAPI.
• #2596 Updated Prometheus dependency to @cd73b3d33e064bbd846fc7a26dc8c313d46af382 which falls in between v2.17.0 and v2.18.0.
  • Receive,Rule: TSDB now supports isolation of append and queries.
  • Receive,Rule: TSDB now holds less WAL files after Head Truncation.
• #2450 Store: Added Regex-set optimization for label=~"a|b|c" matchers.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• 2603 Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).

v0.13.0-rc.1

Fixed

• #2548 Query: Fixed rare cases of double counter reset accounting when querying rate with deduplication enabled.
• #2536 S3: Fixed AWS STS endpoint url to https for Web Identity providers on AWS EKS.
• #2501 Query: Gracefully handle additional fields in SeriesResponse protobuf message that may be added in the future.
• #2568 Query: Don't close the connection of strict, static nodes if establishing a connection had succeeded but Info() call failed.
• #2615 Rule: Fix bugs where rules were out of sync.
• #2614 Tracing: Disabled Elastic APM Go Agent default tracer on initialization to disable the default metric gatherer.
• #2525 Query: Fixed logging for dns resolution error in the Query component.
• #2484 Query/Ruler: Fixed issue #2483, when web.route-prefix is set, it is added twice in HTTP router prefix.
• #2416 Bucket: Fixed issue #2416 bug in inspect --sort-by doesn't work correctly in all cases.
• #2719 Query: irate and resets use now counter downsampling aggregations.
• #2705 minio-go: Added support for af-south-1 and eu-south-1 regions.

Added

• #2012 Receive: Added multi-tenancy support (based on header)
• #2502 StoreAPI: Added hints field to SeriesResponse. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation specific.
• #2521 Sidecar: Added thanos_sidecar_reloader_reloads_failed_total, thanos_sidecar_reloader_reloads_total, thanos_sidecar_reloader_watch_errors_total, thanos_sidecar_reloader_watch_events_total and thanos_sidecar_reloader_watches metrics.
• #2412 UI: Added React UI from Prometheus upstream. Currently only accessible from Query component as only /graph endpoint is migrated.
• #2532 Store: Added hidden option --store.caching-bucket.config=<yaml content> (or --store.caching-bucket.config-file=<file.yaml>) for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage.
• #2579 Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• #2621 Receive: Added flag to configure forward request timeout. Receive write will complete request as soon as quorum of writes succeeds.

Changed

• #2194 Updated to golang v1.14.2.
• #2505 Store: Removed obsolete thanos_store_node_info metric.
• 2513 Tools: Moved thanos bucket commands to thanos tools bucket, also
  moved thanos check rules to thanos tools rules-check. thanos tools rules-check also takes rules by --rules repeated flag not argument
  anymore.
• #2548 Store, Querier: remove duplicated chunks on StoreAPI.
• #2596 Updated Prometheus dependency to @cd73b3d33e064bbd846fc7a26dc8c313d46af382 which falls in between v2.17.0 and v2.18.0.
  • Receive,Rule: TSDB now supports isolation of append and queries.
  • Receive,Rule: TSDB now holds less WAL files after Head Truncation.
• #2450 Store: Added Regex-set optimization for label=~"a|b|c" matchers.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.
• 2603 Store/Querier: Significantly optimize cases where StoreAPIs or blocks returns exact overlapping chunks (e.g Store GW and sidecar or brute force Store Gateway HA).

v0.13.0-rc.0

Highlights:

• Fixed very old bug of rare double counter reset accounting when querying rate with deduplication enabled, thanks to @SuperQ who shared raw data, that reproducing this!
• Thanos Receive is now multitenant 🎉 and closer to claiming production stability 💪
• Huge movement in moving Thanos UIs to React thanks to our GSoC mentee @prmsrswt !
• Experimental Memcached caching for metadata and chunks thanks to @pstibrany and Cortex project ❤️
• Optimized querier for various cases of overlapping data (e.g sidecar + store GW overlap) 📉
• Thanos Rule and Receive should use less resources thanks to Prometheus improvements 🎉

Fixed

• #2548 Query: Fixed rare cases of double counter reset accounting when querying rate with deduplication enabled.
• #2536 S3: Fixed AWS STS endpoint URL to https for Web Identity providers on AWS EKS.
• #2501 Query: Gracefully handle additional fields in SeriesResponse protobuf message that may be added in the future.
• #2568 Query: Don't close the connection of strict, static nodes if establishing a connection had succeeded but Info() call failed.
• #2615 Rule: Fix bugs where rules were out of sync.
• #2614 Tracing: Disabled Elastic APM Go Agent default tracer on initialization to disable the default metric gatherer.
• #2525 Query: Fixed logging for dns resolution error in the Query component.
• #2484 Query/Ruler: Fixed issue #2483, when web.route-prefix is set, it is added twice in HTTP router prefix.
• #2416 Bucket: Fixed issue #2416 bug in inspect --sort-by doesn't work correctly in all cases

Added

• #2012 Receive: Added multi-tenancy support (based on the header)
• #2502 StoreAPI: Added hints field to SeriesResponse. Hints in an opaque data structure that can be used to carry additional information from the store and its content is implementation-specific.
• #2521 Sidecar: Added thanos_sidecar_reloader_reloads_failed_total, thanos_sidecar_reloader_reloads_total, thanos_sidecar_reloader_watch_errors_total, thanos_sidecar_reloader_watch_events_total and thanos_sidecar_reloader_watches metrics.
• #2412 UI: Added React UI from Prometheus upstream. Currently only accessible from Query component as only /graph endpoint is migrated.
• #2532 Store: Added hidden option --store.caching-bucket.config=<yaml content> (or --store.caching-bucket.config-file=<file.yaml>) for experimental caching bucket, that can cache chunks into shared memcached. This can speed up querying and reduce number of requests to object storage.
• #2579 Store: Experimental caching bucket can now cache metadata as well. Config has changed from #2532.
• #2621 Receive: Added flag to configure forward request timeout. Receive write will complete request as soon as a quorum of writes succeeds.

Changed

• #2194 Updated to golang v1.14.2.
• #2505 Store: Removed obsolete thanos_store_node_info metric.
• 2513 Tools: Moved thanos bucket commands to thanos tools bucket, also
  moved thanos check rules to thanos tools rules-check. thanos tools rules-check also takes rules by --rules repeated flag not argument
  anymore.
• #2548 Store, Querier: remove duplicated chunks on StoreAPI.
• #2596 Updated Prometheus dependency to @cd73b3d33e064bbd846fc7a26dc8c313d46af382 which falls in between v2.17.0 and v2.18.0.
  • Receive,Rule: TSDB now supports isolation of append and queries.
  • Receive,Rule: TSDB now holds less WAL files after Head Truncation.
• #2450 Store: Added Regex-set optimization for label=~"a|b|c" matchers.
• #2526 Compact: In case there are no labels left after deduplication via --deduplication.replica-label, assign first replica-label with value deduped.

v0.12.2

Fixed

• #2459 Compact: Fixed issue with old blocks being marked and deleted in a (slow) loop.
• #2533 Rule: do not wrap reload endpoint with /. Makes /-/reload accessible again when no prefix has been specified.

v0.12.1

Fixed

• #2411 Query: fix a bug where queries might not time out sometimes due to issues with one or more StoreAPIs.
• #2474 Store: fix a panic caused by concurrent memory access during block filtering.
• #2472 Compact: fix a bug where partial blocks were never deleted, causing spam of warnings.
• #2484 Query/Ruler: fix issue #2483, when web.route-prefix is set, it is added twice in HTTP router prefix.

v0.12.0

Fixed

• #2288 Ruler: fixes issue #2281, a bug causing incorrect parsing of query address with path prefix.
• #2238 Ruler: fixed issue #2204, where a bug in alert queue signaling filled up the queue and alerts were dropped.
• #2231 Bucket Web: sort chunks by thanos.downsample.resolution for better grouping.
• #2254 Bucket: fix issue where metrics were registered multiple times in bucket replicate.
• #2271 Bucket Web: fixed issue #2260, where the bucket passes null when storage is empty.
• #2339 Query: fix a bug where --store.unhealthy-timeout was never respected.
• #2208 Query and Rule: fix handling of web.route-prefix to correctly handle / and prefixes that do not begin with a /.
• #2311 Receive: ensure receive component serves TLS when TLS configuration is provided.
• #2319 Query: fixed inconsistent naming of metrics.
• #2390 Store: fixed bug that was causing all posting offsets to be used instead of only 1/32 as intended; added hidden flag to control this behavior.
• #2393 Store: fixed bug causing certain not-existing label values queried to fail with "invalid-size" error from binary header.
• #2382 Store: fixed bug causing partial writes of index-header.
• #2383 Store: handle expected errors correctly, e.g. do not increment failure counters.

Added

• #2252 Query: add new --store-strict flag. More information available here.
• #2265 Compact: add --wait-interval to specify compaction wait interval between consecutive compact runs when --wait is enabled.
• #2250 Compact: enable vertical compaction for offline deduplication (experimental). Uses --deduplication.replica-label flag to specify the replica label on which to deduplicate (hidden). Please note that this uses a NAIVE algorithm for merging (no smart replica deduplication, just chaining samples together). This works well for deduplication of blocks with precisely the same samples like those produced by Receiver replication. We plan to add a smarter algorithm in the following weeks.
• #1714 Compact: the compact component now exposes the bucket web UI when it is run as a long-lived process.
• #2304 Store: added max_item_size configuration option to memcached-based index cache. This should be set to the max item size configured in memcached (-I flag) in order to not waste network round-trips to cache items larger than the limit configured in memcached.
• #2297 Store: add --experimental.enable-index-cache-postings-compression flag to enable re-encoding and compressing postings before storing them into the cache. Compressed postings take about 10% of the original size.
• #2357 Compact and Store: the compact and store components now serve the bucket UI on :<http-port>/loaded, which shows exactly the blocks that are currently seen by compactor and the store gateway. The compactor also serves a different bucket UI on :<http-port>/global, which shows the status of object storage without any filters.
• #2166 Bucket Web: improve the tooltip for the bucket UI; it was reconstructed and now exposes much more information about blocks.
• #2172 Store: add support for sharding the store component based on the label hash.
• #2113 Bucket: added thanos bucket replicate command to replicate blocks from one bucket to another.
• #1922 Docs: create a new document to explain sharding in Thanos.
• #2230 Store: optimize conversion of labels.

Changed

• #2136 breaking Store, Compact, Bucket: schedule block deletion by adding deletion-mark.json. This adds a consistent way for multiple readers and writers to access object storage.
  Since there are no consistency guarantees provided by some Object Storage providers, this PR adds a consistent lock-free way of dealing with Object Storage irrespective of the choice of object storage. In order to achieve this co-ordination, blocks are not deleted directly. Instead, blocks are marked for deletion by uploading the deletion-mark.json file for the block that was chosen to be deleted. This file contains Unix time of when the block was marked for deletion. If you want to keep existing behavior, you should add --delete-delay=0s as a flag.
• #2090 breaking Downsample command: the downsample command has moved and is now a sub-command of the thanos bucket sub-command; it cannot be called via thanos downsample any more.
• #2294 Store: optimizations for fetching postings. Queries using =~".*" matchers or negation matchers (!=... or !~...) benefit the most.
• #2301 Ruler: exit with an error when initialization fails.
• #2310 Query: report timespan 0 to 0 when discovering no stores.
• #2330 Store: index-header is no longer experimental. It is enabled by default for store Gateway. You can disable it with new hidden flag: --store.disable-index-header. The --experimental.enable-index-header flag was removed.
• #1848 Ruler: allow returning error messages when a reload is triggered via HTTP.
• #2270 All: Thanos components will now print stack traces when they error out.