-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: add json write #4744
base: main
Are you sure you want to change the base?
chore: add json write #4744
Conversation
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
src/servers/src/http/event.rs
Outdated
@@ -410,7 +408,7 @@ fn extract_pipeline_value_by_content_type( | |||
|
|||
async fn ingest_logs_inner( | |||
state: LogHandlerRef, | |||
pipeline_name: String, | |||
pipeline_name: Option<String>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using just Option
to switch modes seems like a weak constraint, which can be unconsciously triggered by mistake. I was thinking of adding a bool flag to switch to 'auto-create' JSON mode by force, but that seems too heavy.
What do you think, @sunng87
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about a enum:
PipelineChoice{
UserDefined(String)
Auto
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The question here is whether to add a parameter to specify the use of auto pipeline or to treat it as if it were used without the pipeline_name. or some other way to specify auto pipeline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, I think we could use None
to indicate direct log write. This might usually happen in debugging or quick demo phases.
Do we need to validate the Content-Type
to be JSON only in direct mode? If not, it would be the whole line as a column field and a greptime_timestamp
as a timestamp index, which seems ok to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we don't need to change the pipeline type to Option
. The user must specify the pipeline name even when using the identify
pipeline. We can ensure that the identity
pipeline is the internal pipeline, and maybe we will add more internal pipelines in the future. Just checking if the pipeline name is reserved when the user creates new pipelines.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I also favor the idea of defining some of the internal pipeline names. I've tentatively set all pipelines starting with greptime_
to be for our internal use, and prohibited users from creating pipeline names starting with greptime_
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add some integration tests if time permits.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4744 +/- ##
==========================================
- Coverage 84.54% 84.22% -0.32%
==========================================
Files 1117 1119 +2
Lines 202738 203631 +893
==========================================
+ Hits 171408 171516 +108
- Misses 31330 32115 +785 |
daadad2
to
6c9f0d6
Compare
); | ||
} | ||
serde_json::Value::Number(n) => { | ||
if n.is_i64() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to do the type conversion here? For example, i64 -> f64 etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The user writes the f64 at first time, but then the value is an integer, I think we should take care of such cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think auto-conversion is the way to go here. If the types don't match, it'll probably report an error. Seems like that'd be a better option.
src/servers/src/http/event.rs
Outdated
@@ -410,7 +408,7 @@ fn extract_pipeline_value_by_content_type( | |||
|
|||
async fn ingest_logs_inner( | |||
state: LogHandlerRef, | |||
pipeline_name: String, | |||
pipeline_name: Option<String>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we don't need to change the pipeline type to Option
. The user must specify the pipeline name even when using the identify
pipeline. We can ensure that the identity
pipeline is the internal pipeline, and maybe we will add more internal pipelines in the future. Just checking if the pipeline name is reserved when the user creates new pipelines.
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
Checklist