avro schema ui and validation #1107

Big-Vi · 2023-07-11T04:05:13Z

1 - Added https://github.com/apache/avro for front-end validation.
2 - Added node_module util to resolve polyfill issue - Stackoverflow
3 - Added https://github.com/hamba/avro for backend schema validation.

In progress #1061

idanasulin2706 · 2023-07-11T12:13:01Z

@Big-Vi hi and thank you for this amazing contribution, we will take it to our side and keep you updated after reviewing your code and test it

yanivbh1 · 2023-07-12T16:36:54Z

@Avitaltrifsik FYI
That is amazing @Big-Vi ! Thank you.
Before exposing and releasing the feature, we also need to add support in the different SDKs. At least the major ones are GO, JS, Python. LMK if that is something you want to take on as well.
Huge thank you again.

Big-Vi · 2023-07-13T01:43:03Z

@yanivbh1 Yes. I've worked on Go SDK already and code is in my local computer. If i get some reviews on this pull request, i can do some needed changes and can create pull request on Go SDK.
Yes. I can work on Python & Javascript SDKs as well. But I'm not familiar with other languages(.Net, Rust, & Kotlin) yet.

Avitaltrifsik · 2023-07-13T10:31:47Z

@Big-Vi Hey I would love it if you can join our Discord channel :)

Big-Vi · 2023-07-13T10:37:59Z

Hi, @Avitaltrifsik I've joined the channel already. Thanks 🙂

Avitaltrifsik · 2023-07-13T10:44:12Z

@Big-Vi Awesome I DMed you :)

idanasulin2706 · 2023-07-23T13:48:22Z

ui_src/package.json

@@ -41,6 +42,7 @@
 "reaflow": "5.0.6",
 "recharts": "^2.1.16",
 "sass": "^1.49.0",
+ "util": "^0.12.5",


I think you are not using it

I had similar experience. as soon as i installed util module, polyfill error disappeared.

But I just pulled the recent code from upstream repo, It seems compiler not throwing any polyfill error locally. I guess something is fixed in upstream repo. I can remove util module i reckon.

idanasulin2706 · 2023-07-23T13:48:40Z

ui_src/src/domain/schema/components/createSchema/index.js

- value: 'avro',
- label: 'Avro (Coming soon)',
+ value: 'Avro',
+ label: 'Avro (New)',


Remove the (New)

idanasulin2706 · 2023-07-23T13:50:32Z

ui_src/src/domain/schema/components/createSchema/index.js

- value: 'avro',
- label: 'Avro (Coming soon)',
+ value: 'Avro',
+ label: 'Avro (New)',
 description: (
 <span>
 The popular. Apache Avro™ is the leading serialization format for record data, and first choice for streaming data pipelines. It offers excellent schema


Add a note saying that at the moment it is only available here to create schemas but they are not enforced yet on the SDK.
Also, make sure that in case of an Avro schema attached to a station not causing issues for connected clients or for clients who trying to produce/consume data

Add a note saying that at the moment it is only available here to create schemas but they are not enforced yet on the SDK. Also, make sure that in case of an Avro schema attached to a station not causing issues for connected clients or for clients who trying to produce/consume data

I can't find any better way to prevent connected clients from error out if the Avro schema is attached to the station. Can we update all the SDKs and merge all PRs at the same time? I'm all ears if you have any other suggestion.
I already worked on Go, JS, and Python SDKs. I can create a pull request today or tomorrow.

So sure please open the PRs on the SDKs and we will merge this one only after it won't crash clients

After merging the PRs of the SDKs please also update the readme file on this repo, you will find there a sdk features table

So sure please open the PRs on the SDKs and we will merge this one only after it won't crash clients

Sweet as. Just to let you know I'm only familiar with the above mentioned languages. If somebody can work on the remaining SDKs(Java, .Net, and Rust), that would be great.

After merging the PRs of the SDKs please also update the readme file on this repo, you will find there a sdk features table

Yep. Sure.

So sure please open the PRs on the SDKs and we will merge this one only after it won't crash clients

Sweet as. Just to let you know I'm only familiar with the above mentioned languages. If somebody can work on the remaining SDKs(Java, .Net, and Rust), that would be great.

Sure would you mind to open issues on those repositories?

Yep. I'm on it.

So sure please open the PRs on the SDKs and we will merge this one only after it won't crash clients

Sweet as. Just to let you know I'm only familiar with the above mentioned languages. If somebody can work on the remaining SDKs(Java, .Net, and Rust), that would be great.

Sure would you mind to open issues on those repositories?

Issues opened:
superstreamlabs/memphis.java#42
superstreamlabs/memphis.net#105

For Rust, there's already an open issue. Not sure i need to open another one.
turulix/memphis-rust-community#15

idanasulin2706 · 2023-07-23T13:58:00Z

server/memphis_handlers_schemas.go

+ if err != nil {
+ return err
+ }
+ return nil


change to
return fmt.Errorf("your Avro file is invalid: %v", err.Error())

idanasulin2706 · 2023-07-23T14:01:04Z

server/memphis_handlers_schemas.go


- if schemaType == "protobuf" || schemaType == "json" || schemaType == "graphql" {
+ if schemaType == "protobuf" || schemaType == "json" || schemaType == "graphql" || schemaType == "avro" {


not related to here but we have the "GetFilterDetails" endpoint in which you should also return the Avro type

Oh. Yep. missed that one.

go.mod

idanasulin2706 · 2023-07-25T08:15:51Z

@Big-Vi I see your comments but not the code fixes, any chances you forgot to commit them?

Big-Vi · 2023-07-25T08:34:59Z

@idanasulinmemphis I like to know if you want me to replace "hamba/avro" with linkedin module before i do code fixes.

idanasulin2706 · 2023-07-25T08:36:23Z

@idanasulinmemphis I like to know if you want me to replace "hamba/avro" with linkedin module before i do code fixes.

No that is ok to leave it as is

Big-Vi · 2023-07-25T08:40:11Z

Great. I'll let you know once the fixes are done.

@idanasulinmemphis I like to know if you want me to replace "hamba/avro" with linkedin module before i do code fixes.

No that is ok to leave it as is

fixed ui filter removed util module better error handling

idanasulin2706 · 2023-07-27T18:28:36Z

Hi @Big-Vi is it already done and awaiting my review?

Big-Vi · 2023-07-28T05:09:26Z

Hi @Big-Vi is it already done and awaiting my review?

I fixed the code that you commented on. Good to review once again.

idanasulin2706

@Big-Vi approved, I'd like to thank you for this amazing contribution, it is much appreciated.
Before merging please resolve the conflict and that's it

Big-Vi · 2023-07-31T15:30:49Z

@Big-Vi approved, I'd like to thank you for this amazing contribution, it is much appreciated. Before merging please resolve the conflict and that's it

Sweet as. Conflict resolved.

idanasulin2706 · 2023-08-01T04:55:21Z

@Big-Vi now it's ok, approved, and merged, will be released in the coming 1.1.2 version - next week

Big-Vi requested review from avrhamNeeman and idanasulin2706 as code owners July 11, 2023 04:05

Big-Vi force-pushed the feature/avro-schema branch from 6a87379 to 6f85542 Compare July 14, 2023 05:21

Big-Vi added 3 commits July 14, 2023 12:48

avro schema ui and validation

4ed7a99

commiting go.mod and go.sum files

79a7dbb

alter schema table with new enum value for backward compatibility

92dcdb2

Big-Vi force-pushed the feature/avro-schema branch from 6f85542 to 92dcdb2 Compare July 14, 2023 07:28

idanasulin2706 reviewed Jul 23, 2023

View reviewed changes

avro schema fixes

236a5d7

fixed ui filter removed util module better error handling

Big-Vi force-pushed the feature/avro-schema branch from 7c5c724 to 236a5d7 Compare July 28, 2023 05:54

Big-Vi mentioned this pull request Jul 28, 2023

avro validation and serialization superstreamlabs/memphis.js#173

Merged

idanasulin2706 approved these changes Jul 31, 2023

View reviewed changes

resolving conflict

46c24dd

idanasulin2706 approved these changes Aug 1, 2023

View reviewed changes

idanasulin2706 merged commit 00b3053 into superstreamlabs:master Aug 1, 2023

Big-Vi mentioned this pull request Aug 1, 2023

Schemaverse: Add support in Avro #1061

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

avro schema ui and validation #1107

avro schema ui and validation #1107

Big-Vi commented Jul 11, 2023 •

edited

Loading

idanasulin2706 commented Jul 11, 2023

yanivbh1 commented Jul 12, 2023

Big-Vi commented Jul 13, 2023 •

edited

Loading

Avitaltrifsik commented Jul 13, 2023

Big-Vi commented Jul 13, 2023 •

edited

Loading

Avitaltrifsik commented Jul 13, 2023

idanasulin2706 Jul 23, 2023

Big-Vi Jul 24, 2023

idanasulin2706 Jul 23, 2023

idanasulin2706 Jul 23, 2023

Big-Vi Jul 26, 2023 •

edited

Loading

idanasulin2706 Jul 26, 2023

idanasulin2706 Jul 26, 2023

Big-Vi Jul 26, 2023

Big-Vi Jul 26, 2023

idanasulin2706 Jul 26, 2023

Big-Vi Jul 26, 2023

Big-Vi Jul 27, 2023

idanasulin2706 Jul 23, 2023

idanasulin2706 Jul 23, 2023

Big-Vi Jul 24, 2023

idanasulin2706 commented Jul 25, 2023

Big-Vi commented Jul 25, 2023

idanasulin2706 commented Jul 25, 2023

Big-Vi commented Jul 25, 2023

idanasulin2706 commented Jul 27, 2023

Big-Vi commented Jul 28, 2023 •

edited

Loading

idanasulin2706 left a comment •

edited

Loading

Big-Vi commented Jul 31, 2023

idanasulin2706 commented Aug 1, 2023


		if schemaType == "protobuf" \|\| schemaType == "json" \|\| schemaType == "graphql" {
		if schemaType == "protobuf" \|\| schemaType == "json" \|\| schemaType == "graphql" \|\| schemaType == "avro" {

avro schema ui and validation #1107

avro schema ui and validation #1107

Conversation

Big-Vi commented Jul 11, 2023 • edited Loading

idanasulin2706 commented Jul 11, 2023

yanivbh1 commented Jul 12, 2023

Big-Vi commented Jul 13, 2023 • edited Loading

Avitaltrifsik commented Jul 13, 2023

Big-Vi commented Jul 13, 2023 • edited Loading

Avitaltrifsik commented Jul 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Big-Vi Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

idanasulin2706 commented Jul 25, 2023

Big-Vi commented Jul 25, 2023

idanasulin2706 commented Jul 25, 2023

Big-Vi commented Jul 25, 2023

idanasulin2706 commented Jul 27, 2023

Big-Vi commented Jul 28, 2023 • edited Loading

idanasulin2706 left a comment • edited Loading

Choose a reason for hiding this comment

Big-Vi commented Jul 31, 2023

idanasulin2706 commented Aug 1, 2023

Big-Vi commented Jul 11, 2023 •

edited

Loading

Big-Vi commented Jul 13, 2023 •

edited

Loading

Big-Vi commented Jul 13, 2023 •

edited

Loading

Big-Vi Jul 26, 2023 •

edited

Loading

Big-Vi commented Jul 28, 2023 •

edited

Loading

idanasulin2706 left a comment •

edited

Loading