Show and modify routing rules from the UI #433

prakhar10 · 2024-08-09T01:50:26Z

Description

Adding a way to display routing rules and also modify them from the UI
Minor change to show a user icon with a gear if user is ADMIN or a regular user icon if its a regular USER on the top right corner of the UI

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(X) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

webapp/src/components/routing-rules.tsx

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

willmostly

This looks like a nice improvement, thank you! I reviewed the Java half, and will try to find someone for the typescript.

My biggest concern here is not with the code, but with allowing users to supply input that will be written to disk on the server, then executed by the jeasy rules engine. We should take a close look at jeasy and mvel to understand what security vulnerabilities we might be introducing. At minimum we should log all of the updates so that they can be audited. @wendigo and @ebyhr wdyt?

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

willmostly · 2024-08-16T01:23:20Z

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

+            for (int i = 0; i < routingRulesList.size(); i++) {
+                if (routingRulesList.get(i).name().equals(routingRules.name())) {
+                    routingRulesList.set(i, routingRules);
+                    break;
+                }
+            }


as the name of this method suggests, this updates an existing rule. It does not appear that this change adds support for deleting an existing rule or adding a new one. I guess this may be due to the UI changes required, but would it be difficult to add this functionality?

I don't think it would be difficult. I thought of initially going with only updates and in a separate PR i will put Add and Delete functionality. Will that be ok? Or should I add it in this PR itself?

that plan sounds fine to me, thanks!

prakhar10 · 2024-08-16T19:15:12Z

This looks like a nice improvement, thank you! I reviewed the Java half, and will try to find someone for the typescript.

My biggest concern here is not with the code, but with allowing users to supply input that will be written to disk on the server, then executed by the jeasy rules engine. We should take a close look at jeasy and mvel to understand what security vulnerabilities we might be introducing. At minimum we should log all of the updates so that they can be audited. @wendigo and @ebyhr wdyt?

This functionality will be accessible only for users with ADMIN privileges. Regular users will have this as Read-Only. I also have a functionality implemented in our internal repo where we are auditing changes being made to the Clusters from the UI. I can raise a PR for that as well and will add functionality to audit routing rules as well. I was thinking of getting this reviewed first and then from comments and suggestions I will raise the Audit logs PR.

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

willmostly

this is looking good, we're almost there. @mosabua and I discussed the need to support this for clustered deployments. We should also provide a way to disable this feature in case someone doesn't want to bother setting up a shared filesystem

docs/gateway-api.md

willmostly · 2024-08-30T19:01:43Z

@prakhar10 please squash these commits

prakhar10 · 2024-08-31T04:53:57Z

this is looking good, we're almost there. @mosabua and I discussed the need to support this for clustered deployments. We should also provide a way to disable this feature in case someone doesn't want to bother setting up a shared filesystem

So this means provide an option or a config to show/disable the Routing Rules option in the left panel of the UI right?

mosabua · 2024-09-03T17:36:56Z

this is looking good, we're almost there. @mosabua and I discussed the need to support this for clustered deployments. We should also provide a way to disable this feature in case someone doesn't want to bother setting up a shared filesystem

So this means provide an option or a config to show/disable the Routing Rules option in the left panel of the UI right?

I think for now we can just say in the docs that you must provide shared storage for that feature to work and in terms of disabling it could be just a config option option .. I think over time we will need various config options for the UI anyway

Remove console logs and renamed api Add api documentation and changes related to PR comments

prakhar10 · 2024-09-24T03:38:08Z

@mosabua @willmostly can you please review?

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

mosabua · 2024-10-03T23:02:25Z

I assume that @prakhar10 and @willmostly are collaborating on this currently .. ping me if review or anything else is needed.

prakhar10 · 2024-10-04T02:41:08Z

I need to make changes according to @ebyhr 's comments. So i will get that done by this weekend.

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/config/HaGatewayConfiguration.java

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

prakhar10 · 2024-10-21T19:55:55Z

@ebyhr can you review now please?

prakhar10 · 2024-10-26T17:24:30Z

@mosabua can you please review?

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java

docs/gateway-api.md

gateway-ha/src/main/java/io/trino/gateway/ha/config/UIConfiguration.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

prakhar10 · 2024-11-18T18:51:57Z

@ebyhr can you take a look now?

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java

prakhar10 · 2024-12-13T09:08:55Z

@willmostly @ebyhr can you review now please?

willmostly · 2025-01-07T15:54:28Z

docs/gateway-api.md

+For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.
+
+```shell
+curl -X POST http://localhost:8080/webapp/updateRoutingRules \


The method definition includes

@Consumes(MediaType.APPLICATION_JSON)

Does this curl command work without

-H 'Content-Type: application/json'

?

willmostly · 2025-01-07T16:15:46Z

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java

+    @RolesAllowed("USER")
+    @Produces(MediaType.APPLICATION_JSON)
+    @Path("/getRoutingRules")
+    public Response getRoutingRules()


Should these methods return the bare type instead of wrapping them in a Response, similar to the Trino UI? E.g. https://github.com/trinodb/trino/blob/bb80ff8be6073bc970501825e2e7f304b3b9d643/core/trino-main/src/main/java/io/trino/server/ui/ClusterResource.java#L51

willmostly · 2025-01-07T16:21:45Z

docs/gateway-api.md

+This API can be used to programmatically update the Routing Rules.
+Rule will be updated based on the rule name.
+
+For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.


Suggested change

For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.

For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage that supports file locking for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.

willmostly · 2025-01-07T16:27:17Z

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

+            throws IOException
+    {
+        ImmutableList.Builder<RoutingRule> updatedRoutingRulesBuilder = ImmutableList.builder();
+        List<RoutingRule> currentRoutingRulesList = new ArrayList<>();


Suggested change

List<RoutingRule> currentRoutingRulesList = new ArrayList<>();

List<RoutingRule> currentRoutingRulesList = getRoutingRules();

also remove the the logic below to read rules

willmostly · 2025-01-07T16:30:31Z

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java

+                yamlContent.append(yamlWriter.writeValueAsString(rule));
+                updatedRoutingRulesBuilder.add(rule);
+            }
+            try (FileChannel fileChannel = FileChannel.open(Paths.get(rulesConfigPath), StandardOpenOption.WRITE, StandardOpenOption.READ);


Lock the file before it is read. Otherwise you allow a sequence like

Gateway 1 reads rules

Gateway 2 reads rules

Gateway 1 locks rules, writes rules, releases lock.

Gateway 2 locks rules, writes rules, releases lock.

Which results in the changes by Gateway 1 being lost.

willmostly · 2025-01-07T16:35:24Z

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+final class TestRoutingRulesManager


Please add tests that use the REST API

willmostly · 2025-01-07T16:39:40Z

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRule.java

+public record RoutingRule(
+        String name,
+        @Nullable String description,
+        @Nullable Integer priority,


priority should not be nullable. It may be defaulted to 0.

willmostly · 2025-01-07T16:42:18Z

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRule.java

+        requireNonNull(name, "name is null");
+        actions = ImmutableList.copyOf(actions);
+        requireNonNull(condition, "condition is null");
+    }


Set description to "" if it is null

cla-bot bot added the cla-signed label Aug 9, 2024

Chaho12 reviewed Aug 12, 2024

View reviewed changes

willmostly requested changes Aug 16, 2024

View reviewed changes

willmostly requested changes Aug 20, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/resource/GatewayWebAppResource.java Outdated Show resolved Hide resolved

willmostly requested changes Aug 30, 2024

View reviewed changes

docs/gateway-api.md Show resolved Hide resolved

Show and modify routing rules from the UI

d3c7855

Remove console logs and renamed api Add api documentation and changes related to PR comments

prakhar10 force-pushed the routing-rules-ui branch from 85287b9 to d3c7855 Compare September 9, 2024 00:21

Add option to disable pages and update API docs

ba2e92f

prakhar10 requested a review from willmostly September 24, 2024 03:43

ebyhr reviewed Sep 25, 2024

View reviewed changes

ebyhr assigned prakhar10 Oct 1, 2024

prakhar10 added 2 commits October 7, 2024 21:43

Move functions to service class and add tests and other fixes

034963d

Fix build and remove json property

38023fc

ebyhr reviewed Oct 9, 2024

View reviewed changes

prakhar10 added 3 commits October 16, 2024 16:04

Fix PR comments

546b85f

Remove unused import

7cf593d

Fix CI Build

e3ea3cd

ebyhr requested changes Oct 27, 2024

View reviewed changes

willmostly requested changes Oct 29, 2024

View reviewed changes

willmostly requested changes Oct 31, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/domain/RoutingRules.java Outdated Show resolved Hide resolved

willmostly reviewed Oct 31, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java Outdated Show resolved Hide resolved

prakhar10 added 4 commits November 11, 2024 22:42

Add some PR comment fixes

3c98006

Move code to service class

b8ba509

Fix conflict

5799ce8

Merge branch 'main' into routing-rules-ui

1fb283d

ebyhr reviewed Nov 19, 2024

View reviewed changes

prakhar10 added 2 commits November 20, 2024 19:46

Fix PR comments

de2a34c

Fix CI build

2c31fef

ebyhr reviewed Nov 21, 2024

View reviewed changes

gateway-ha/src/main/java/io/trino/gateway/ha/router/RoutingRulesManager.java Outdated Show resolved Hide resolved

gateway-ha/src/test/java/io/trino/gateway/ha/router/TestRoutingRulesManager.java Outdated Show resolved Hide resolved

Use Filesystem locking to avoid concurrent updates

c0bb63d

fixup! Use Filesystem locking to avoid concurrent updates

765da61

willmostly requested changes Jan 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Show and modify routing rules from the UI #433

Show and modify routing rules from the UI #433

prakhar10 commented Aug 9, 2024

willmostly left a comment

willmostly Aug 16, 2024

prakhar10 Aug 16, 2024

willmostly Aug 20, 2024

prakhar10 commented Aug 16, 2024 •

edited

Loading

willmostly left a comment

willmostly commented Aug 30, 2024

prakhar10 commented Aug 31, 2024

mosabua commented Sep 3, 2024

prakhar10 commented Sep 24, 2024

mosabua commented Oct 3, 2024

prakhar10 commented Oct 4, 2024

prakhar10 commented Oct 21, 2024

prakhar10 commented Oct 26, 2024

prakhar10 commented Nov 18, 2024

prakhar10 commented Dec 13, 2024

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

willmostly Jan 7, 2025

	For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.
	For this feature to work with multiple replicas of the Trino Gateway, you will need to provide a shared storage that supports file locking for the routing rules file. If multiple replicas are used with local storage, then rules will get out of sync when updated.

	List<RoutingRule> currentRoutingRulesList = new ArrayList<>();
	List<RoutingRule> currentRoutingRulesList = getRoutingRules();

Show and modify routing rules from the UI #433

Are you sure you want to change the base?

Show and modify routing rules from the UI #433

Conversation

prakhar10 commented Aug 9, 2024

Description

Additional context and related issues

Release notes

willmostly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prakhar10 commented Aug 16, 2024 • edited Loading

willmostly left a comment

Choose a reason for hiding this comment

willmostly commented Aug 30, 2024

prakhar10 commented Aug 31, 2024

mosabua commented Sep 3, 2024

prakhar10 commented Sep 24, 2024

mosabua commented Oct 3, 2024

prakhar10 commented Oct 4, 2024

prakhar10 commented Oct 21, 2024

prakhar10 commented Oct 26, 2024

prakhar10 commented Nov 18, 2024

prakhar10 commented Dec 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prakhar10 commented Aug 16, 2024 •

edited

Loading