Added sagaexecutor example #489

stevef1uk · 2023-12-27T13:12:30Z

Please review and see if this project is a suitable example.

stevef1uk · 2023-12-27T13:34:23Z

Sorry, somehow missed the extra parameter I asked on the NewService call in Poller as that logic hadn't been changed for a while & I must have been testing with an old image without noticing.

stevef1uk · 2023-12-27T13:36:49Z

Also, I just realised that the Tilt files I have used to build & deploy the components are configured for my RPi k3s cluster as I don't have a X86 one available for me to use.

mikeee · 2023-12-30T19:36:24Z

Amazing example, I really like this. Will definitely be useful for showcasing tilt given the push for an improvement in the developer experience.

I think it'd be perfect if you could possibly convert the README.pdf to markdown and combine with the existing README.md?

What would be even more brilliant is if we could get a validation test set up for this example too - mechanical markdown example go-sdk/examples/actor/README.md

I know it's an example but I'd like to commit an improved docker build process - with scratch images. Can contribute and review with suggested changes when I have more bandwith.

codecov · 2023-12-31T06:31:30Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (04f7b59) 70.08% compared to head (9504d8a) 58.04%.
Report is 15 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #489       +/-   ##
===========================================
- Coverage   70.08%   58.04%   -12.04%     
===========================================
  Files          35       55       +20     
  Lines        2841     3568      +727     
===========================================
+ Hits         1991     2071       +80     
- Misses        738     1375      +637     
- Partials      112      122       +10

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

stevef1uk · 2024-01-01T08:18:17Z

Amazing example, I really like this. Will definitely be useful for showcasing tilt given the push for an improvement in the developer experience.

I think it'd be perfect if you could possibly convert the README.pdf to markdown and combine with the existing README.md?

What would be even more brilliant is if we could get a validation test set up for this example too - mechanical markdown example go-sdk/examples/actor/README.md

I know it's an example but I'd like to commit an improved docker build process - with scratch images. Can contribute and review with suggested changes when I have more bandwith.

Thanks.

I have started on this and just committed to my repo the local set-up. I have written a script to automate the creation of the Postgres container & written a Dapr.yaml file to start the poller, a subscriber and the a test server.

Next I will tackle the markup.

When I boot up my proper dev machines again I will look for how I have previously created docker images from scratch. This will be a few years ago as I gave up on k8s a long time ago.

stevef1uk · 2024-01-02T08:39:52Z

Ok. I have completed the documentation consolidation.

I can build a scratch image, but I can't figure out how to do that from Tilt.

stevef1uk · 2024-01-02T11:16:40Z

I have checked my image sizes and they are about 20Mb so I don't think moving to scratch is worth it?

stevef1uk · 2024-01-06T11:25:52Z

I have just included the latest changes I made, which was to remove my Postgres custom code & use the Dapr Postgres Binding instead. The only example for how to use this was in the Bindings test code ;-) Thus, this may be useful o others?

stevef1uk · 2024-01-07T08:57:18Z

OK. I have stopped making changes to this base code. Hopefully, it is ok now?

daixiang0 · 2024-01-08T02:42:43Z

examples/sagaexecutor/cmd/poller/app

Please do not include binary files.

Done. Removed errant binary

daixiang0 · 2024-01-08T02:44:15Z

examples/sagaexecutor/cmd/subscriber/Tiltfile

@@ -0,0 +1,30 @@
+# -*- mode: Python -*-


Please convert it to Makefile, which is more common.

Not keen on doing that. However, ChatGPT makes that easily doable :-)

Makefiles written (mostly) by ChatGPT, tweaked. Testing found issues but fixed & tested on Mac (M1 CPU) and tested build ok with AMD CPU

Tested fixed makefiles on M2 Mac deployed to RPI k3s cluster & Intel Mac with Rancher desktop. All working as expected so I have finished the Makefiles addition request,

Signed-off-by: stevef1uk <[email protected]>

Signed-off-by: stevef1uk <[email protected]> Signed-off-by: stevef1uk <[email protected]>

Just tested on an intel system and realised the README was slightly wrong. Signed-off-by: stevef1uk <[email protected]> Signed-off-by: stevef1uk <[email protected]>

Signed-off-by: stevef1uk <[email protected]>

…plus more simplifications Signed-off-by: stevef1uk <[email protected]>

Signed-off-by: stevef1uk <[email protected]>

stevef1uk · 2024-01-16T07:26:29Z

Anyone got the time to finish review & merge?

yaron2 · 2024-01-19T16:52:56Z

Anyone got the time to finish review & merge?

Yes i'll take a look at this today and early next week.

mikeee · 2024-01-20T09:36:21Z

A saga example in java was added to the quickstarts repo. I'm wondering if this would better be merged there.

stevef1uk · 2024-01-20T11:36:38Z

A saga example in java was added to the quickstarts repo. I'm wondering if this would better be merged there.

I can't see that but I did think about whether this should be in the QuickStarts but decided it wasn't a good fit as this is unashamably a Go project.

mikeee

I think it'd be amazing if you were able to add a git workflow to validate this example continuously

stevef1uk · 2024-02-02T07:33:21Z

I think it'd be amazing if you were able to add a git workflow to validate this example continuously

Sorry, that is something I usually leave to the DevOps team once the Cloud environments have been stood up ;-)

mikeee · 2024-02-02T17:06:13Z

Fair enough, I'm happy to pick that up as long as the example is working and we have an issue raised as a to-do at a later date.

Will review when I get home 👀

stevef1uk · 2024-02-02T20:55:00Z

Fair enough, I'm happy to pick that up as long as the example is working and we have an issue raised as a to-do at a later date.

Will review when I get home 👀

Cool, although I don't want you to work extra hours. I just realised it has been over 24 years since I stopped being a professional programer. These days I do pretty much what I want and can get hands-on again. I have a new contract starting soon where I will have to learn Python. The last scripting language I used seriously was Perl so that dates me even more :-)

Please let me know if the code doesn't work and I will fix it. I have only been able to test on Macs (Intel & M CPUs) and a couple of types of Kubernetes (Rancher & K3s) types. It worked for me :-)

mikeee

First look review comments - I've reviewed without running the example:

This example involves a lot of steps which could potentially be added to a main makefile at the root of the example. I don't see an issue with this example being committed in this way but should in the future be refactored with a few issues being raised for this to be completed at a later date.

I've not commented on the use of snake-case as a convention as this is a sample and is perfectly acceptable and valid.

The comments are a mix of spelling/corrections. A few things I've noticed are:

Imports are currently referencing the repo github.com/stevef1uk/sagaexecutor which should be changed to that of this dapr repo (I've only noted this on the go.mod but this affects each package where the import is made.
I potentially would like to see an env-var to specify the user's dockerhub/registry to push to which may reduce the amount of changes where seahope needs to be changed.
Some absolute paths are specified which should be updated to relative ones.

Looks great, with the minor tweaks it would be a great example to get people started with the saga pattern.

mikeee · 2024-02-03T11:49:50Z

.gitignore

@@ -22,3 +22,5 @@ vendor
 # docs 
 golang.org
 coverage.txt
+
+.dapr


What is this new excludes for - can this be a change limited to that of the example folder for which this PR is made?

What is this new excludes for - can this be a change limited to that of the example folder for which this PR is made?

This change was to stop the logs created when running locally from being committed to GitHub. I can delete the .gitignore from this directory though.

mikeee · 2024-02-03T11:51:44Z

examples/sagaexecutor/LICENSE

I think this licence file should be removed

mikeee · 2024-02-03T11:53:17Z

examples/sagaexecutor/README.md

+2. The Saga Subscriber 
+3. The Saga Poller
+
+The Saga components are shown in Green and the Dapr building blockes in Blue.


sp.

Suggested change

The Saga components are shown in Green and the Dapr building blockes in Blue.

The Saga components are shown in Green and the Dapr building blocks in Blue.

mikeee · 2024-02-03T11:58:07Z

examples/sagaexecutor/README.md

+### Prerequisites
+1. A kubernetes cluster is required with dapr installed (dapr init -k)
+2. Redis & Postgres must be installed on the cluster
+3. Tilt is is used to deply the components (see: https://tilt.dev). However, Makefiles are provided as well. Please note that these files


Suggested change

3. Tilt is is used to deply the components (see: https://tilt.dev). However, Makefiles are provided as well. Please note that these files

3. Tilt is is used to deploy the components (see: https://tilt.dev). However, Makefiles are provided as well. Please note that these files

sp.

mikeee · 2024-02-03T11:58:24Z

examples/sagaexecutor/README.md

+== APP - sagapoller == 2024/01/01 09:14:26 Returned 0 records
+```
+
+You will need to manally delete the postgres container for a full clean-up.


Suggested change

You will need to manally delete the postgres container for a full clean-up.

You will need to manually delete the postgres container for a full clean-up.

sp.

mikeee · 2024-02-04T13:48:31Z

examples/sagaexecutor/go.mod

@@ -0,0 +1,61 @@
+module github.com/stevef1uk/sagaexecutor


This module definition needs to be updated along with any references from each package.

This module definition needs to be updated along with any references from each package.

Replaced all github.com/stevef1uk/sagaexecutor to github.com/dapr/go-sdk/examples/sagaexecutor

mikeee · 2024-02-04T13:57:05Z

examples/sagaexecutor/cmd/subscriber/app.go

+	"fmt"
+	"net/http"
+
+	//"fmt"


Suggested change

//"fmt"

Unused comment

mikeee · 2024-02-04T14:00:31Z

examples/sagaexecutor/components/pubsub.yaml.safe

Appears to be a duplicate spec that could be removed.

I have deleted it, but it was there for my convenience as to run on k8s it needs to be edited,

mikeee · 2024-02-04T14:07:09Z

examples/sagaexecutor/components/secrets.yaml

+  version: v1
+  metadata:
+  - name: secretsFile
+    value: /Users/stevef/dev/sagaexecutor/components/secrets.json


This absolute path to the secrets file appears to be problematic for those attempting to use this. A change required to this file is not documented so I believe it should be relative.

This absolute path to the secrets file appears to be problematic for those attempting to use this. A change required to this file is not documented so I believe it should be relative.

I will look at this later when I have recreated a K8s system to test it against.

This absolute path to the secrets file appears to be problematic for those attempting to use this. A change required to this file is not documented so I believe it should be relative.

I will look at this later when I have recreated a K8s system to test it against.

This absolute path was required for local testing and relative paths wouldn't let me load the component. I have added some code to the setup local.sh file to update this hard path to the one required so that should now work for you.

mikeee · 2024-02-04T14:16:08Z

examples/sagaexecutor/test_clients/test_client3/app.go

+	/*pp_id       string    `json:"app_id"`
+	  Service      string    `json:"service"`
+	  Token        string    `json:"token"`
+	  callback_service string    `json:"callback_service"`
+	  Params       string    `json:"params"`
+	  Timeout      int       `json:"timeout"`
+	  Event        bool      `json:"event"`
+	  LogTime      time.Time `json:"logtime"`*/


Is this comment required?

stevef1uk · 2024-02-04T15:28:20Z

First look review comments - I've reviewed without running the example:

This example involves a lot of steps which could potentially be added to a main makefile at the root of the example. I don't see an issue with this example being committed in this way but should in the future be refactored with a few issues being raised for this to be completed at a later date.

Ok. I use Tilt to deploy so I kept the Makefile minimal as daixiang0 asked for Makefiles, which I first used in 1987 and assumed were no longer a thing :-)

I've not commented on the use of snake-case as a convention as this is a sample and is perfectly acceptable and valid.
Yes a good Go Dev told me I should use golint, which to be honest I didn't this time. As you say it is only an example and I haven't written lots of unit test & used a Mocking framework. As this example is basically heavily using the Dapr Go SDK I focussed on integration tests as it was more fun for me and I am not being paid so I have skipped the boring stuff ;-)

The comments are a mix of spelling/corrections. A few things I've noticed are:
Sorry, I find unless I print things out (which no devs do these days) I read what I expect to see and not what is written, so I will fix those.

Imports are currently referencing the repo github.com/stevef1uk/sagaexecutor which should be changed to that of this dapr repo (I've only noted this on the go.mod but this affects each package where the import is made.
This is something that always confuses me with Go. When I make that change I won't be able to build and test unless I use the replace capability in go.mod, which I think I had to do when I first committed this code to my GitHub repo. Obviously, I can't commit that Go,mod with the replace either.

I potentially would like to see an env-var to specify the user's dockerhub/registry to push to which may reduce the amount of changes where seahope needs to be changed.
Agreed. I was going to do that but forgot.

Some absolute paths are specified which should be updated to relative ones.

Agreed.

Looks great, with the minor tweaks it would be a great example to get people started with the saga pattern.
Thanks. I may be able to start on this tomorrow. I don't yet have a start date for my return to work and when I do anything I do on this will need to be outside working hours. A bit of coding stuff is always more fun for me than powerpoint decks, but generally not what I am paid for anymore :-)

Signed-off-by: stevef1uk <[email protected]>

stevef1uk · 2024-02-05T08:21:40Z

I have made a start and added a commit for most of the minor changes requested. I haven't tried to recompile after changing the imports yet.

Update: I have compiled the code successfully. You will need to edit the replace in Go.mod for your location to try to build this before its is actually added to the GitHub repo and then that replace can be removed.

Signed-off-by: stevef1uk <[email protected]>

stevef1uk · 2024-02-05T15:43:37Z

Just added a commit to enable DockerHub Id to be configured in the Makefiles. Tested this on my repo ok.

Signed-off-by: stevef1uk <[email protected]>

stevef1uk · 2024-02-05T16:53:23Z

OK, all changes requested in the initial review (that I can see above) and have responded to have been addressed. I hope you can now build & test this locally?

For changing the owning repo name for the docker container images simply set the DOCKER_ID environment variable to it.

Signed-off-by: stevef1uk <[email protected]>

…ects stats Signed-off-by: stevef1uk <[email protected]>

stevef1uk · 2024-02-12T15:41:10Z

Je vous attends encore.

Signed-off-by: mikeee <[email protected]>

stevef1uk requested a review from a team as a code owner December 27, 2023 13:12

stevef1uk mentioned this pull request Dec 29, 2023

Example project: Saga Execution Orchestrator #488

Open

stevef1uk force-pushed the main branch from dbf35fe to 3f8a40a Compare January 7, 2024 08:52

daixiang0 reviewed Jan 8, 2024

View reviewed changes

stevef1uk and others added 16 commits January 8, 2024 12:13

Added sagaexecutor example

b269faa

Signed-off-by: stevef1uk <[email protected]>

Missed extra parameter on Poller

2d24a6d

Signed-off-by: stevef1uk <[email protected]>

Update README.md

76a9a7f

Signed-off-by: stevef1uk <[email protected]> Signed-off-by: stevef1uk <[email protected]>

Update README.md

1e88c2b

Just tested on an intel system and realised the README was slightly wrong. Signed-off-by: stevef1uk <[email protected]> Signed-off-by: stevef1uk <[email protected]>

Added consolidated README & local testing capability

b3afa45

Signed-off-by: stevef1uk <[email protected]>

Removing log files

c9b16a1

Signed-off-by: stevef1uk <[email protected]>

deleted log files

75b80bf

Signed-off-by: stevef1uk <[email protected]>

fixed an oops on mock_server dockerfile

9a61ab8

Signed-off-by: stevef1uk <[email protected]>

Updated README for kubernetes

5e5f292

Signed-off-by: stevef1uk <[email protected]>

Removed custom Postgres code and replaced with Dapr Postgres Binding …

a6317c3

…plus more simplifications Signed-off-by: stevef1uk <[email protected]>

removing unnecessary debug logging

a2820dc

Signed-off-by: stevef1uk <[email protected]>

ensuring dapr client closed

2dd7d62

Signed-off-by: stevef1uk <[email protected]>

removed unnecessary makefiles

01c6d8c

Signed-off-by: stevef1uk <[email protected]>

Added signal handlers to Poller & Subscriber

c611de6

Signed-off-by: stevef1uk <[email protected]>

removed errant binary

16c4973

Signed-off-by: stevef1uk <[email protected]>

Added Makefiles & updated README etc.

d368f4b

Signed-off-by: stevef1uk <[email protected]>

stevef1uk force-pushed the main branch from 131fedf to d368f4b Compare January 8, 2024 11:14

stevef1uk added 2 commits January 9, 2024 18:52

Fixed Makesfiles

e92597c

Signed-off-by: stevef1uk <[email protected]>

removed some commented out code

6da8049

Signed-off-by: stevef1uk <[email protected]>

mikeee reviewed Feb 1, 2024

View reviewed changes

mikeee reviewed Feb 4, 2024

View reviewed changes

Most review comments addressed

ef11b9b

Signed-off-by: stevef1uk <[email protected]>

Made GitHub Id configurable in Makefiles

fb07cf1

Signed-off-by: stevef1uk <[email protected]>

stevef1uk added 3 commits February 5, 2024 17:35

worked around absolute path in secret for local testing

c63f56b

Signed-off-by: stevef1uk <[email protected]>

Removed final mentions of my GitHub repo

f731afc

Signed-off-by: stevef1uk <[email protected]>

Deleted errant binary

d84b53f

Signed-off-by: stevef1uk <[email protected]>

stevef1uk added 2 commits February 5, 2024 21:46

Made setting DockerHub ID by one env variable

a1734e4

Signed-off-by: stevef1uk <[email protected]>

Changed comment in poller.app & updated README to contain latest proj…

a7be33c

…ects stats Signed-off-by: stevef1uk <[email protected]>

apply spelling changes

9504d8a

Signed-off-by: mikeee <[email protected]>

stevef1uk closed this May 5, 2024

	The Saga components are shown in Green and the Dapr building blockes in Blue.
	The Saga components are shown in Green and the Dapr building blocks in Blue.

	3. Tilt is is used to deply the components (see: https://tilt.dev). However, Makefiles are provided as well. Please note that these files
	3. Tilt is is used to deploy the components (see: https://tilt.dev). However, Makefiles are provided as well. Please note that these files

	You will need to manally delete the postgres container for a full clean-up.
	You will need to manually delete the postgres container for a full clean-up.

Added sagaexecutor example #489

Added sagaexecutor example #489

Conversation

stevef1uk commented Dec 27, 2023

stevef1uk commented Dec 27, 2023 • edited Loading

stevef1uk commented Dec 27, 2023

mikeee commented Dec 30, 2023

codecov bot commented Dec 31, 2023 • edited Loading

Codecov Report

stevef1uk commented Jan 1, 2024

stevef1uk commented Jan 2, 2024

stevef1uk commented Jan 2, 2024

stevef1uk commented Jan 6, 2024

stevef1uk commented Jan 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevef1uk Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

stevef1uk Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevef1uk commented Jan 16, 2024

yaron2 commented Jan 19, 2024

mikeee commented Jan 20, 2024

stevef1uk commented Jan 20, 2024

mikeee left a comment

Choose a reason for hiding this comment

stevef1uk commented Feb 2, 2024

mikeee commented Feb 2, 2024

stevef1uk commented Feb 2, 2024 • edited Loading

mikeee left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevef1uk commented Feb 4, 2024

stevef1uk commented Feb 5, 2024 • edited Loading

stevef1uk commented Feb 5, 2024

stevef1uk commented Feb 5, 2024 • edited Loading

stevef1uk commented Feb 12, 2024

stevef1uk commented Dec 27, 2023 •

edited

Loading

codecov bot commented Dec 31, 2023 •

edited

Loading

stevef1uk Jan 8, 2024 •

edited

Loading

stevef1uk Jan 8, 2024 •

edited

Loading

stevef1uk commented Feb 2, 2024 •

edited

Loading

stevef1uk commented Feb 5, 2024 •

edited

Loading

stevef1uk commented Feb 5, 2024 •

edited

Loading