Component #1823
Merged
Component #1823
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: William Dumont <[email protected]>
Co-authored-by: Piotr <[email protected]>
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
clayton-cornell
approved these changes
Oct 9, 2024
….queue.md Co-authored-by: Clayton Cornell <[email protected]>
ptodev
reviewed
Oct 10, 2024
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
It'd be nice to add a bit more info on how prometheus.remote.queue is different from prometheus.remote_write, and when to use what component.
thampiotr
reviewed
Oct 11, 2024
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
docs/sources/reference/components/prometheus/prometheus.remote.queue.md
Outdated
Show resolved
Hide resolved
| Name | Type | Description | Default | Required | ||
| ---- | ---- | ----------- | ------- | -------- | ||
| `max_signals_to_batch` | `uint` | The maximum number of signals before they are batched to disk. | `10,000` | no | ||
| `batch_frequency` | `duration` | How often to batch signals to disk if `max_signals_to_batch` is not reached. | no |
There was a problem hiding this comment.
Why do we use signals as a name? maybe it should be samples? or is this to encompass histograms samples too? I'm not sure if that's a right name. In current remote_write we call this 'metrics' which makes sense although it's not very precise: https://grafana.com/docs/alloy/next/reference/components/prometheus/prometheus.remote_write/#wal-block
There was a problem hiding this comment.
Subsequent PR will enable metadata storage, so I wanted it to not be just metrics. Sample is to narrow, open to better names here.
There was a problem hiding this comment.
maybe 'entries'? that would be a generic name for things inside a queue.
Comment on lines
+99
to
+111
| if len(s.endpoints) > 0 { | ||
| for _, ep := range s.endpoints { | ||
| ep.Stop() | ||
| } | ||
| s.endpoints = map[string]*endpoint{} | ||
| } | ||
| err := s.createEndpoints() | ||
| if err != nil { | ||
| return err | ||
| } | ||
| for _, ep := range s.endpoints { | ||
| ep.Start() | ||
| } |
There was a problem hiding this comment.
Yeah, maybe Update can signal via channel to Run that will manage everything in one thread. I found this to be a decent pattern in other components.
….queue.md Co-authored-by: Paulin Todev <[email protected]>
….queue.md Co-authored-by: Piotr <[email protected]>
wildum
reviewed
Oct 11, 2024
|
Most of the feedback has been responded so merging. |
mattdurham
added a commit
that referenced
this pull request
Oct 16, 2024
* readme * fix readme * Add filequeue functionality (#1601) * Checkin for file queue * add comment * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * naming and error handling feedback from PR * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/filequeue/filequeue.go Co-authored-by: Piotr <[email protected]> * drop benchmark * rename get to pop --------- Co-authored-by: Piotr <[email protected]> * Adding the serialization features. (#1666) * Adding the serialization features. * Dont test this with race condition since we access vars directly. * Fix test. * Fix typo in file name and return early in DeserializeToSeriesGroup. * Update internal/component/prometheus/remote/queue/serialization/appender.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/serialization/serializer.go Co-authored-by: Piotr <[email protected]> * Rename to indicate that TimeSeries are Put/Get from a pool. * Remove func that was about the same number of lines as inlining. * Update internal/component/prometheus/remote/queue/types/serialization.go Co-authored-by: Piotr <[email protected]> * Update internal/component/prometheus/remote/queue/serialization/serializer.go Co-authored-by: Piotr <[email protected]> * Change benchmark to be more specific. --------- Co-authored-by: Piotr <[email protected]> * Network wal pr (#1717) * Checkin the networking items. * Fix for config updating and tests. * Update internal/component/prometheus/remote/queue/network/loop.go Co-authored-by: William Dumont <[email protected]> * Update internal/component/prometheus/remote/queue/network/loop.go Co-authored-by: Piotr <[email protected]> * pr feedback * pr feedback * simplify stats * PR feedback --------- Co-authored-by: William Dumont <[email protected]> Co-authored-by: Piotr <[email protected]> * Component (#1823) * Checkin the networking items. * Fix for config updating and tests. * Update internal/component/prometheus/remote/queue/network/loop.go Co-authored-by: William Dumont <[email protected]> * Update internal/component/prometheus/remote/queue/network/loop.go Co-authored-by: Piotr <[email protected]> * pr feedback * pr feedback * simplify stats * simplify stats * Initial push. * docs and some renaming * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Changes and testing. * Update docs. * Update docs. * Fix race conditions in unit tests. * Tweaking unit tests. * lower threshold more. * lower threshold more. * Fix deadlock in manager tests. * rollback to previous * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Clayton Cornell <[email protected]> * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Paulin Todev <[email protected]> * Docs PR feedback * Update docs/sources/reference/components/prometheus/prometheus.remote.queue.md Co-authored-by: Piotr <[email protected]> * PR feedback * PR feedback * PR feedback * PR feedback * Fix typo * Fix typo * Fix bug. * Fix docs --------- Co-authored-by: William Dumont <[email protected]> Co-authored-by: Piotr <[email protected]> Co-authored-by: Clayton Cornell <[email protected]> Co-authored-by: Paulin Todev <[email protected]> * Change name to write instead of remote. * Fix issue. * Fix issue. * Dont depend on random sync.pool behavior. * small clarification on changelog. * PR feedback --------- Co-authored-by: Piotr <[email protected]> Co-authored-by: William Dumont <[email protected]> Co-authored-by: Clayton Cornell <[email protected]> Co-authored-by: Paulin Todev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the last PR, aside from the merge to main. This ties everything together and exposes the component. Most of this are very expansive tests. I am likely to change the metrics in subsequent PRs so dont get to in detail about them.
Things to do that I will do after this large merge and create tickets for each.