WIP : Ranger HTTP client implementations #41

koushikr · 2021-10-18T07:16:42Z

Apologies for the massive-ish PR. I understand the incredible burden of having to go through it.

Split the T of serviceNode and Criteria and that contributed to one ridiculous spur of changes. Didn't make sense breaking that in parts, and the tests to support the same have been added too - again didn't make sense to submit code and test in parts.

The additional ones are the additional modules, ranger-http-client, ranger-zk-client, and ranger-server (Really small interface classes). The split of criteria and T of serviceNode has bloated the PR up.

This is an extension of, #31

Why separate T and Criteria

To facilitate for clients to dynamically bind inline criteria when implementing shard selectors, to decouple the type of such criteria bound with the generic T type that has been saved as a part of serviceNode.
This will help clients further wing out their own implementations of criteria basis their individual parameters and supply a matcher during node group (shard) selection.

Major LLD Callouts

The shardSelector implementation after the introduction of Criteria can be either of ShardSelector<T, C extends Criteria<T>> implements ShardSelector<T, C, R extends Registry<T>> or ShardSelector<T> implements ShardSelector<T, Criteria<T>, R extends Registry<T>>. Chose the former to keep the Criteria implementation fluid and to be able to have clients define their shardSelector within the bound of any Criteria implementation.
Added clients as different modules to avoid everyone having to implement the same boilerplate code to create hubs (zk or http). Creating a hub is now just a new instance creation.

Changelist

The T in the ServiceNode has been broken from the Criteria used in the shard selectors, to help for dynamic binding of criteria, so node filters can be dictated by the clients who are using the bundle, basis any of the client-specific parameters.
The unclusteredFinder has been renamed to unsharededFinder and a list-based shardSelector to work with the criteria has been introduced.
The ranger module has been further broken down into ranger-core (To define the core framework of ranger), ranger-zookeeper, and ranger-HTTP (To appropriate the ranger-core with the respective implementations for zookeeper and HTTP data sources). ranger-zookeeper-client and ranger-HTTP-client(To appropriate the finder-hub implementations for zookeeper and HTTP data sources)
A sample ranger-server has been added to facilitate writing an HTTP finder back-end, atop a service hub.
An extensive test suite to test the multiple modules has been added as well.
Added license in the files missing license.

…erviceprovider and servicehub package structures, each workflow has a containment 2. Added an implementation for http node data sink and registration

Added a dummy project structure for ranger-server

…, finderhub and health

… hub

…t custom criteria could be passed to the shardSelector which can help in shardSelection

…implementations

… one for mapBasedRegistry and one for listBasedServiceRegistry b) Tests added for the broken down criteria across all the modules

…y purposes and to manage the lifecycle within

…ce info

…clients

…e you may want to build

…valMs to have it consistent with the rest of the code

…ere acquired. 3. Addee license on the missing files. 4. Sonar fixes

santanusinha · 2021-10-18T09:01:02Z

ranger-client/src/main/java/com/flipkart/ranger/client/Constants.java


-package com.flipkart.ranger.core.finder.unsharded;
+public class Constants {


Introduce a private Constructor

santanusinha · 2021-10-18T09:01:20Z

ranger-client/src/main/java/com/flipkart/ranger/client/Constants.java

@@ -13,17 +13,10 @@
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
+package com.flipkart.ranger.client;


Why is it showing up as a rename of the file?

santanusinha · 2021-10-18T09:12:29Z

...est/java/com/flipkart/ranger/core/finder/nodeselector/RoundRobinServiceNodeSelectorTest.java

+        serviceNodes.add(new ServiceNode<>("localhost-2", 9001, TestNodeData.builder().nodeId(2).build()));
+        serviceNodes.add(new ServiceNode<>("localhost-3", 9002, TestNodeData.builder().nodeId(3).build()));
+        ServiceNode<TestNodeData> select = roundRobinSelector.select(serviceNodes);
+        Assert.assertTrue(select.getHost().equalsIgnoreCase("localhost-2"));


why equalsIgnoreCase etc? use Assert.assertEquals

Fix in all places

santanusinha · 2021-10-18T09:15:08Z

ranger-http/src/main/java/com/flipkart/ranger/http/common/HttpNodeDataStoreConnector.java

@@ -39,7 +48,7 @@ public HttpNodeDataStoreConnector(
                .connectionPool(new ConnectionPool(1, 30, TimeUnit.SECONDS))
                .build();
        this.config = config;
-        this.mapper = mapper;
+        this.mapper = null != mapper ? mapper : new ObjectMapper();


Do not create object mapper inside constructor

santanusinha · 2021-10-18T09:15:36Z

ranger-http/src/main/java/com/flipkart/ranger/http/model/ServiceDataSourceResponse.java

+@Builder
+public class ServiceDataSourceResponse {
+    private boolean success;
+    private List<Service> data;


Add an error message here

santanusinha · 2021-10-18T09:21:26Z

ranger-http/src/main/java/com/flipkart/ranger/http/servicefinder/HttpNodeDataSource.java

-        log.error("No data returned from server: " + httpUrl);
+
+        if (null != serviceNodesResponse && null != serviceNodesResponse.getData() &&
+                !serviceNodesResponse.getData().isEmpty()) {


If there is error message in the response, log that

santanusinha · 2021-10-18T09:22:41Z

...est/java/com/flipkart/ranger/http/serviceprovider/HttpShardedServiceProviderBuilderTest.java

+
+    @Test
+    public void testProvider() throws Exception {
+        TestNodeData nm5NodeData = TestNodeData.builder().farmId("nm5").build();


santanusinha · 2021-10-18T09:23:01Z

...est/java/com/flipkart/ranger/http/serviceprovider/HttpShardedServiceProviderBuilderTest.java

+    @Test
+    public void testProvider() throws Exception {
+        TestNodeData nm5NodeData = TestNodeData.builder().farmId("nm5").build();
+        ServiceNode<TestNodeData> testNode = new ServiceNode<>("localhost-1", 80, nm5NodeData);


val .. make things consistent

santanusinha · 2021-10-18T09:24:29Z

ranger-server/src/main/java/com/flipkart/ranger/server/resources/RangerResource.java

+    @GET
+    @Path("/nodes/{namespace}/{serviceName}")
+    @Timed
+    @ExceptionMetered


Just do @metered and removed the above two anotations

santanusinha · 2021-10-18T09:26:28Z

ranger-server/src/main/java/com/flipkart/ranger/server/manager/RangerClientManager.java

+
+    private final RangerConfiguration rangerConfiguration;
+
+    private final UnshardedRangerZKHubClient<T, C> zkHubClient;


You might want to rethink this. Ideally this should be abstract here, so that the server can, if it wants, add some in-memory cached decorator to the hub

koushikr · 2021-10-18T11:13:23Z

Closing this PR. Broke this into #42 and #43 to be able to track better.

Koushik R and others added 30 commits September 23, 2021 18:56

1. Reorganized ranger-zookeeper and ranger-http into servicefinder, s…

4ab3e8e

…erviceprovider and servicehub package structures, each workflow has a containment 2. Added an implementation for http node data sink and registration

Moved ZKWatcherRegistryUpdateSignal into signals in finder

36429bb

Moved ZKNodeDataStoreConnector to commons in ZK

39d6ba9

Added a dummy project structure for ranger-server

Restored the java version to 1.8 for backward compatibility

86d1638

Optimized imports

bf3325c

Fixed HttpShardedServiceFinderBuilderTest

7210e86

Added healthcheck handlers for service provision

b7b68ce

Cleaned up the health update

c51f153

Added a nodeselector and moved nodeselectors underneath it

a65346c

Reorganized the structure of ranger core into finder, serviceprovider…

c850aba

…, finderhub and health

Added a finderSelector - for custom finder selection from the service…

37d763b

… hub

Reverted the finderSelector

96d910f

Added a criteria holder, make it return the relevant nodeData. So tha…

79a6f58

…t custom criteria could be passed to the shardSelector which can help in shardSelection

Added tests for httpModule of ranger

cdae648

Renamed unsharded serviceregistry to list based service registry

e8d11f5

Added an abstract nodeList to serviceRegistry and implemented in its …

e384d07

…implementations

a) Criteria has been broken down into sharded and unsharded criteria,…

e4a6b4a

… one for mapBasedRegistry and one for listBasedServiceRegistry b) Tests added for the broken down criteria across all the modules

Removed the server module

d8e33d5

Removed the unnecessary server

0d59101

Added http service finder hub

f0e07cd

Added tests for http hub

d3e1c72

Changed ZKServiceDataSource to take in connectionString for consitenc…

72820c8

…y purposes and to manage the lifecycle within

Updated lombok version

69f4883

Made Criteria as interfaces instead. @FunctionalInterface at that

3806e9f

Fixed lambda expressions under criteria in tests

bfd19e6

ServiceRegistry type removed from criteria

773091a

Optimized imports

c50cc9a

Optimized imports

1eac3ed

Added a noop shard selector

5af8ff6

Unsharded criteria renamed to filter criteria

d93bdc9

koushikr added 23 commits October 14, 2021 16:55

Normalized criteria into a single interface

37f5f8e

Added curator to datasource

0e3d7a5

Changed matching selector to support a missing criteria

831863e

Committing locally to save stash

fc21d46

Criteria is now part of the parent closure

ada0f6d

Simplified the static builders

b23cfcc

Fixed tests in service discovery bundle

37ee531

Added the required licence to the classes that were missing the licen…

c8432f4

…ce info

Added ranger clients. Http and zk, including the simple and http hub …

1180c9b

…clients

Optimized imports

f9a6150

Added ranger server bundle to be used as a finder for any http servic…

2165ae8

…e you may want to build

Optimized imports and fixed tests for URL path changes

431ada0

Changed the access qualifier to protected in clients

20c73ee

Restructured package structure for server bundle

54fa826

Added tests for abstract hub client test

75e0c8a

Added tests for sharded zk client

b248b2e

Added deserializer to the http bundle as well

202c819

Added tests for ranger http client

ac1360a

Simplified the extension tests in http module

ab63403

Optimized imports

6558249

Added changelog

5090dcf

Changed the refreshIntervalMs in AbstractZKClient to nodeRefreshInter…

6eaad53

…valMs to have it consistent with the rest of the code

1. Changed the license to javadoc format. 2. Made attributes final wh…

1773c67

…ere acquired. 3. Addee license on the missing files. 4. Sonar fixes

santanusinha requested changes Oct 18, 2021

View reviewed changes

This was referenced Oct 18, 2021

Separated T in nodeData from Criteria #42

Merged

Added ranger-clients [Dependent on #42] #43

Merged

koushikr closed this Oct 18, 2021

koushikr mentioned this pull request Oct 18, 2021

Ranger server [Dependent on PR #42, #43] #44

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP : Ranger HTTP client implementations #41

WIP : Ranger HTTP client implementations #41

koushikr commented Oct 18, 2021 •

edited

Loading

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

santanusinha Oct 18, 2021

koushikr commented Oct 18, 2021


		package com.flipkart.ranger.core.finder.unsharded;
		public class Constants {


		private final RangerConfiguration rangerConfiguration;

		private final UnshardedRangerZKHubClient<T, C> zkHubClient;

WIP : Ranger HTTP client implementations #41

WIP : Ranger HTTP client implementations #41

Conversation

koushikr commented Oct 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koushikr commented Oct 18, 2021

koushikr commented Oct 18, 2021 •

edited

Loading