diff --git a/docs/connection/new-connection-framework-metamodel.png b/docs/connection/new-connection-framework-metamodel.png new file mode 100644 index 00000000000..151efcc4de2 Binary files /dev/null and b/docs/connection/new-connection-framework-metamodel.png differ diff --git a/docs/connection/new-connection-framework.md b/docs/connection/new-connection-framework.md new file mode 100644 index 00000000000..54c5edf9fca --- /dev/null +++ b/docs/connection/new-connection-framework.md @@ -0,0 +1,166 @@ +# New Connection Framework (PoC) + +This is _an attempt_ to cleanup the existing connection acquisition framework to make it (1) more extensible, (2) integrate better with work that has been done on [identity/credential management](https://github.com/finos/legend/wiki/Legend-Authentication) by @epsstan as well as feedback from @pierredebelen, and (3) reduce code complexity. The core mission centers around: **_"How do we make it easy for people (both platform developers and users) to acquire database connection given their identity?"_** + +> This is still a work in progress, we haven't enabled this in production. To use/test, set environment variable `org.finos.legend.engine.enableNewConnectionFramework=true` when starting the server. To execute, since at the time of speaking, we haven't include the new connection models in latest production protocol, choose `clientVersion=vX_X_X` + +## Overview + +Fundamentally, the new connection framework improves DX for integrating new type of connection by making it consistent across different types of databases (not limited to only relational databases); in other words, adding support for `MongoDB` or `ElasticSearch` should require similar set of change to adding support for `Snowflake` or `BigQuery`, for example. + +The new API is also designed to be more declarative about the connection acquisition flows that it supports, hence, it's a continuation of the work that has been done with [DatabaseAuthenticationFlow](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan-connection-authentication/src/main/java/org/finos/legend/engine/authentication/DatabaseAuthenticationFlow.java#L42) by @epsstan. In the new framework, when we setup the connection manager/factory, we make it very explicit which flows are supported (which types of credentials can be resolved given an identity, and which those credentials, which types of connection can be acquired). + +Last but not least, we want to clean up the existing connection pooling code using [HikariCP](https://github.com/brettwooldridge/HikariCP) which has become fairly complex after many iterations. + +For more details on design and implementation, see the following sub-sections: + +- [Connection Factory](#connection-factory) +- [Connection Pooling](#connection-pooling) +- [Migration Strategy](#migration-strategy) +- [Future Work / TODO](#future-work-/-todo) + +## Connection Factory + +`ConnectionFactory` is the API entry of the new connection framework. It is simplified to specifically addresses the main objective: **given an identity, obtain a connection to a database**. A new set of (meta)model have been created to serve this new API. + +These models are heavily inspired by [RelationalDatabaseConnection](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/RelationalDatabaseConnection.java#L25); a `Connection` in the new framework comes with `AuthenticationConfiguration` corresponding to [AuthenticationStrategy](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/authentication/AuthenticationStrategy.java#L21) and `ConnectionSpecification` corresponding to [DatasourceSpecification](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/specification/DatasourceSpecification.java#L21). The former is responsible for providing extra metadata needed to obtain necessary credential from the user's identity while the latter provides metadata to obtain the actual connection to the database given the obtained credentials. + +![framework-metamodel](./new-connection-framework-metamodel.png) + +> We believe in this abstraction since concepts like `authentication` and `connection specification` are certainly applicable to NoSQL databases like `MongoDB` and `ElasticSearch` + +Each connection targets a particular type of database (similar to [RelationalDatabaseConnection.databaseType](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/RelationalDatabaseConnection.java#L29)), which comes with a set of database support defined in the platform configuration. As of now, this database support only includes information about the various authentication mechanisms that a database supports in Legend. For example, for `Snowflake`, we only support `OAuth` and `KeyPair`; for `Postgres`, we only support `UsernamePassword`. Since `AuthenticationConfiguration` provides extra metadata to obtain credentials, each type of configuration therefore is bound to an `AuthenticationMechanism` -- this specification is also something defined explicitly at top-level. + +> Unlike [RelationalDatabaseConnection.databaseType](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/RelationalDatabaseConnection.java#L29)) which is practically an enum which is not modularizable and hardcoded to Relational types, `database support` is more extensible and a useful concept on its own, in the future, we can add more database capabilities here as needed + +As for the `ConnectionFactory`, it's configured with 2 types of components:`CredentialBuilder` - (can be chained) taking the identity and build a credential, `ConnectionBuilder` - taking a credential and acquire a connection. + +```java +class ConnectionBuilder +... + +class CredentialBuilder +... +``` + +> Notice how the credential builder is bound to the authentication configuration as some credential builders need extra metadata from the configuration to obtain the credential, for example, `UserPasswordCredentialBuilder` needs the name of the user and the reference to the password in the secret vault. @epsstan mentioned here that we should also tie `CrendetialBuilder` by the `AuthenticationMechanism` which is a point worths exploring. + +The factory exposes 2 methods: `getAuthenticator(Identity, ConnectionInfo): Authenticator` and `getConnection(Identity, Authenticator): Connection`. Given connection information (i.e. `Connection` metamodel) and user's identity, the factory will resolve a chain of credential builders that sequentially can _authenticate_ users and help them obtain the necessary credentials to acquire a connection to the specified database. The second method is self-explantory, the factory can use the authenticator to establish the connection given user's identity. + +> The resolution process of the credential builders chain is implemented in `breadth first search` manner, and the fact that we configure the factory with small components makes it convenient to reuse these pieces in different flows, easier to debug, and flexible enough to support fairly complex flows: e.g. given a `kerberos` identity, obtain a password from the vault which can be used to obtain a `single-sign-on` token, and so on... + +```mermaid +sequenceDiagram + participant USER + participant factory as ConnectionFactory + participant connBuilder as ConnectionBuilder + participant pool as ConnectionPool + participant credBuilder as CredentialBuilder + participant database as Database + + Note over USER: (comes with an Identity) + USER ->>+ factory: getAuthenticator(identity, connectionInfo) + factory --)- USER: authenticator + USER ->>+ factory: getConnection(identity, authenticator) + factory ->>+ connBuilder: getConnection(connectionInfo, authenticator, identity) + connBuilder ->>+ pool: getConnectionFromPool(identity, authenticator, connectionInfo) + alt get available connection in pool + pool --)- connBuilder: connection + else establish new connection + connBuilder ->>+ pool: getConnectionFromPool(identity, authenticator, ...) + loop each credential builder from authenticator + pool ->>+ credBuilder: getCredential(identity, credentials) + credBuilder --)- pool: credentials + end + pool ->>+ database: getConnection(credentials, connectionInfo) + database --)- pool: connection + pool --)- connBuilder: connection + end + connBuilder --)- factory: connection + factory --)- USER: connection +``` + +## Connection Pooling + +The current implementaion of the connection pool is fairly simple, we maintain a hashmap that indexes the connection by the database and user's identity. This hasn't been able to address @pierredebelen biggest concern that is when a connection from the pool dies (e.g. it coudl timeout/closed by the database), we don't have a way to re-establish that connection with the user's identity since `HikariCP` creates connection asynchronously; caching identity across thread is potentially security-risk. + +> Right now, we cache the identity in the connection pool for a brief period of time and have a cleanup process that runs on a separate thread to remove connections/pools with expired/invalid credentials. + +## Migration Strategy + +The bulk of the work is to migrate existing relational database connection framework to the new framework. To do that, we opt for a gradual migration strategy, convert one [DatabaseAuthenticationFlow](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan-connection-authentication/src/main/java/org/finos/legend/engine/authentication/DatabaseAuthenticationFlow.java#L42) [flow at a time](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan-connection-authentication-default/src/main/java/org/finos/legend/engine/authentication/LegendDefaultDatabaseAuthenticationFlowProvider.java#L50). We create new subtypes for `DatasourceSpecification` and `AuthenticationStrategy` that wraps new models `ConnectionSpecification` and `AuthenticationConfiguration` respectively. + +```java +public class ConnectionSpecificationWrapper extends DatasourceSpecification +{ + public ConnectionSpecification value; +} + +public class AuthenticationConfigurationWrapper extends AuthenticationStrategy +{ + public AuthenticationConfiguration value; +} +``` + +In `RelationalExecutor`'s `ConnectionManagerSelector.getDatabaseConnectionImpl(...)`, we check for the presence of the wrapper types to determine if we can opt-in to the new connection framework to acquire connections. + +On top level, we configure a custom deserializer for [RelationalDatabaseConnection](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/RelationalDatabaseConnection.java#L25) if it matches a particular combination, we will convert its datasource specification and authentication strategy to wrapper type. For example, if the connection database type is [Snowflake](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/DatabaseType.java#L20) with [SnowflakeDatasourceSpecification](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/specification/SnowflakeDatasourceSpecification.java#L17) and [SnowflakePublicAuthenticationStrategy](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/store/relational/connection/authentication/SnowflakePublicAuthenticationStrategy.java#L17) (which corresponds to [SnowflakeWithKeyPairFlow](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-execution/src/main/java/org/finos/legend/engine/authentication/flows/SnowflakeWithKeyPairFlow.java#L49)), we can convert this to use wrapper type, such as: + +``` +RelationalDatabaseConnection connection::snowflake +{ + store: store::MyDatabase; + type: Snowflake; + specification: ConnectionSpecification // wrapper type + { + rawValue: #{ + { + "_type" : "snowflake", + "accountName" : "ki79827", + "region" : "us-east-2", + "warehouseName" : "SUMMIT_DEV", + "databaseName" : "SUMMIT_DEV", + "cloudType" : "aws", + "role" : "SUMMIT_DEV" + } + }#; + }; + auth: AuthenticationConfiguration // wrapper type + { + rawValue: #{ + { + "_type" : "KeyPair", + "userName" : "SUMMIT_DEV1", + "privateKey" : { + "_type" : "properties", + "propertyName" : "TEST_SNOWFLAKE_PK" + }, + "passphrase" : { + "_type" : "properties", + "propertyName" : "TEST_SNOWFLAKE_PK_PASSPHRASE" + } + } + }#; + }; +} +``` + +> Note: since this is still a PoC, we _hack_ a fair bit to make things just work while keeping things loose enough to adapt to new feedback, therefore, we haven't settled with a particular grammar/Pure protocol/Pure metamodel, that's something we have to come back to change + +> When migrate to the new models, we also use new `CredentialVault` mechanism created by @epsstan, which requires specifying the type of [CredentialVaultSecret](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-authentication/legend-engine-xt-authentication-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/authentication/vault/CredentialVaultSecret.java#L23), e.g. [PropertiesFileSecret](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-authentication/legend-engine-xt-authentication-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/authentication/vault/PropertiesFileSecret.java#L21C14-L21C34), [EnvironmentCredentialVaultSecret](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-authentication/legend-engine-xt-authentication-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/packageableElement/authentication/vault/EnvironmentCredentialVaultSecret.java#L21C14-L21C46), etc. When we configure the custom deserializer, we need to be aware of this though, for legacy infrastructure, most likely, we will use their proprietary secret vault implementation. + +Last but not least, since right now, this POC supports only one flow `SnowflakeWithKeyPair`, to test it out, we need to set environment variable `org.finos.legend.engine.enableNewConnectionFramework=true` when starting the server. +For the execution flow, since we need to transform Pure metamodel into protocol as part of building the execution plan, we need to make change to `Pure protocol`, right now those changes are in `vX_X_X`, when execute, we need to set `clientVersion=vX_X_X`. + +## Future Work / TODO + +- [ ] Settle on the metamodels and implement grammar/compiler/Pure protocol/metamodel code for these models (search for marker **`@HACKY: new-connection-framework`** in the codebase) +- [ ] Improve connection pooling recovery mechanism +- [ ] Improve connection acquisition flow coverage, see [LegendDefaultDatabaseAuthenticationFlowProvider](https://github.com/finos/legend-engine/blob/07e7129ec2b68d7c1606dd157c23e65d7cd6857b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan-connection-authentication-default/src/main/java/org/finos/legend/engine/authentication/LegendDefaultDatabaseAuthenticationFlowProvider.java#L50) +- [ ] Make use of this new connection framework in data-push server to battle-test the implementation +- [ ] Design parameterized tests for new connection framework flows. These tests should be parameterized on (potentially) `database type`, `credential vault type`, `connection specification`, `authentication mechanism`, `authentication configuration` and should have a simple `READ` operation to ensure the connection is working. +- [ ] Consistenyly support `aws-sdk S3`-styled builder patterns +- [ ] Cleanups: + - [ ] Move `GCPApplicatonDefaultsCredential` to `bigquery` db extension + - [ ] Cleanup logic for `IdentityFactory` + - [ ] Potentially unify the work done on `xt-authenticaton` with the new connection framework diff --git a/legend-engine-config/legend-engine-connection-integration-tests/pom.xml b/legend-engine-config/legend-engine-connection-integration-tests/pom.xml index 3fb7a35c689..82f6fb194d7 100644 --- a/legend-engine-config/legend-engine-connection-integration-tests/pom.xml +++ b/legend-engine-config/legend-engine-connection-integration-tests/pom.xml @@ -46,7 +46,12 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory + test + + + org.finos.legend.engine + legend-engine-xt-connection-protocol test @@ -54,11 +59,21 @@ legend-engine-xt-relationalStore-connection test + + org.finos.legend.engine + legend-engine-xt-relationalStore-protocol + test + org.finos.legend.engine legend-engine-xt-relationalStore-postgres-test-support test + + org.finos.legend.engine + legend-engine-xt-relationalStore-snowflake-protocol + test + diff --git a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/AbstractConnectionFactoryTest.java b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/AbstractConnectionFactoryTest.java index e5b6435ece9..27b8e42a8c6 100644 --- a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/AbstractConnectionFactoryTest.java +++ b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/AbstractConnectionFactoryTest.java @@ -17,25 +17,26 @@ import org.finos.legend.authentication.vault.CredentialVault; import org.finos.legend.authentication.vault.impl.EnvironmentCredentialVault; import org.finos.legend.authentication.vault.impl.SystemPropertiesCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; +import org.finos.legend.connection.AuthenticationMechanism; import org.finos.legend.connection.Authenticator; +import org.finos.legend.connection.Connection; import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.connection.DatabaseSupport; import org.finos.legend.connection.DatabaseType; import org.finos.legend.connection.IdentityFactory; import org.finos.legend.connection.IdentitySpecification; -import org.finos.legend.connection.impl.InstrumentedStoreInstanceProvider; import org.finos.legend.connection.LegendEnvironment; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.impl.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; import org.finos.legend.connection.impl.KerberosCredentialExtractor; import org.finos.legend.connection.impl.KeyPairCredentialBuilder; +import org.finos.legend.connection.impl.RelationalDatabaseType; import org.finos.legend.connection.impl.SnowflakeConnectionBuilder; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; -import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; import org.finos.legend.connection.impl.StaticJDBCConnectionBuilder; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; +import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeEach; @@ -43,11 +44,8 @@ public abstract class AbstractConnectionFactoryTest { - protected static final String TEST_STORE_INSTANCE_NAME = "test-store"; - protected LegendEnvironment environment; protected IdentityFactory identityFactory; - protected InstrumentedStoreInstanceProvider storeInstanceProvider; protected ConnectionFactory connectionFactory; @BeforeEach @@ -55,26 +53,30 @@ public void initialize() { this.setup(); - LegendEnvironment.Builder environmentBuilder = new LegendEnvironment.Builder() - .withVaults( + LegendEnvironment.Builder environmentBuilder = LegendEnvironment.builder() + .vaults( new SystemPropertiesCredentialVault(), new EnvironmentCredentialVault() ) - .withStoreSupports( - new RelationalDatabaseStoreSupport.Builder(DatabaseType.POSTGRES) - .withIdentifier("Postgres") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).withAuthenticationConfigurationTypes( - UserPasswordAuthenticationConfiguration.class - ).build() + .databaseSupports( + DatabaseSupport.builder() + .type(RelationalDatabaseType.POSTGRES) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class + ).build() ) .build(), - new RelationalDatabaseStoreSupport.Builder(DatabaseType.SNOWFLAKE) - .withIdentifier("Snowflake") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KEY_PAIR).withAuthenticationConfigurationTypes( - EncryptedPrivateKeyPairAuthenticationConfiguration.class - ).build() + DatabaseSupport.builder() + .type(RelationalDatabaseType.SNOWFLAKE) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.KEY_PAIR) + .authenticationConfigurationTypes( + EncryptedPrivateKeyPairAuthenticationConfiguration.class + ).build() ) .build() ); @@ -82,22 +84,23 @@ public void initialize() CredentialVault credentialVault = this.getCredentialVault(); if (credentialVault != null) { - environmentBuilder.withVault(credentialVault); + environmentBuilder.vault(credentialVault); } this.environment = environmentBuilder.build(); - this.identityFactory = new IdentityFactory.Builder(this.environment) + this.identityFactory = IdentityFactory.builder() + .environment(this.environment) .build(); - this.storeInstanceProvider = new InstrumentedStoreInstanceProvider(); - this.connectionFactory = new ConnectionFactory.Builder(this.environment, this.storeInstanceProvider) - .withCredentialBuilders( + this.connectionFactory = ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders( new KerberosCredentialExtractor(), new UserPasswordCredentialBuilder(), new KeyPairCredentialBuilder() ) - .withConnectionBuilders( + .connectionBuilders( new StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword(), new SnowflakeConnectionBuilder.WithKeyPair() ) @@ -119,10 +122,12 @@ public CredentialVault getCredentialVault() return null; } - public abstract StoreInstance getStoreInstance(); - public abstract Identity getIdentity(); + public abstract DatabaseType getDatabaseType(); + + public abstract ConnectionSpecification getConnectionSpecification(); + public abstract AuthenticationConfiguration getAuthenticationConfiguration(); public abstract void runTestWithConnection(T connection) throws Exception; @@ -130,11 +135,19 @@ public CredentialVault getCredentialVault() @Test public void runTest() throws Exception { - this.storeInstanceProvider.injectStoreInstance(this.getStoreInstance()); Identity identity = this.getIdentity(); + DatabaseType databaseType = this.getDatabaseType(); + ConnectionSpecification connectionSpecification = this.getConnectionSpecification(); AuthenticationConfiguration authenticationConfiguration = this.getAuthenticationConfiguration(); - Authenticator authenticator = this.connectionFactory.getAuthenticator(identity, TEST_STORE_INSTANCE_NAME, authenticationConfiguration); + Connection databaseConnection = Connection.builder() + .databaseSupport(this.environment.getDatabaseSupport(databaseType)) + .identifier("test::connection") + .connectionSpecification(connectionSpecification) + .authenticationConfiguration(authenticationConfiguration) + .build(); + + Authenticator authenticator = this.connectionFactory.getAuthenticator(identity, databaseConnection); T connection = this.connectionFactory.getConnection(identity, authenticator); this.runTestWithConnection(connection); @@ -146,8 +159,8 @@ public void runTest() throws Exception protected static Identity getAnonymousIdentity(IdentityFactory identityFactory) { return identityFactory.createIdentity( - new IdentitySpecification.Builder() - .withName("test-user") + IdentitySpecification.builder() + .name("test-user") .build() ); } diff --git a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestJDBCConnectionManager.java b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestJDBCConnectionManager.java index 7abf243d608..dc08b4dc741 100644 --- a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestJDBCConnectionManager.java +++ b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestJDBCConnectionManager.java @@ -16,47 +16,43 @@ import net.bytebuddy.asm.Advice; import org.finos.legend.authentication.vault.impl.PropertiesFileCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; +import org.finos.legend.connection.AuthenticationMechanism; import org.finos.legend.connection.Authenticator; +import org.finos.legend.connection.Connection; import org.finos.legend.connection.ConnectionFactory; -import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.DatabaseSupport; import org.finos.legend.connection.IdentityFactory; import org.finos.legend.connection.IdentitySpecification; -import org.finos.legend.connection.JDBCConnectionBuilder; import org.finos.legend.connection.LegendEnvironment; import org.finos.legend.connection.PostgresTestContainerWrapper; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.impl.InstrumentedStoreInstanceProvider; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; +import org.finos.legend.connection.impl.JDBCConnectionBuilder; import org.finos.legend.connection.impl.JDBCConnectionManager; +import org.finos.legend.connection.impl.RelationalDatabaseType; import org.finos.legend.connection.impl.StaticJDBCConnectionBuilder; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; -import org.finos.legend.connection.protocol.ConnectionSpecification; -import org.finos.legend.connection.protocol.StaticJDBCConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.model.connection.StaticJDBCConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.PropertiesFileSecret; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; -import java.sql.Connection; import java.sql.SQLTransientConnectionException; import java.util.Properties; public class TestJDBCConnectionManager { PostgresTestContainerWrapper postgresContainer; - private static final String TEST_STORE_INSTANCE_NAME = "test-store"; private LegendEnvironment environment; private IdentityFactory identityFactory; - private InstrumentedStoreInstanceProvider storeInstanceProvider; private ConnectionFactory connectionFactory; - private StoreInstance storeInstance; + private Connection connection; @BeforeEach public void setup() @@ -67,32 +63,37 @@ public void setup() Properties properties = new Properties(); properties.put("passwordRef", this.postgresContainer.getPassword()); - LegendEnvironment.Builder environmentBuilder = new LegendEnvironment.Builder() - .withVaults(new PropertiesFileCredentialVault(properties)) - .withStoreSupports( - new RelationalDatabaseStoreSupport.Builder(DatabaseType.POSTGRES) - .withIdentifier("Postgres") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).withAuthenticationConfigurationTypes( - UserPasswordAuthenticationConfiguration.class - ).build() + LegendEnvironment.Builder environmentBuilder = LegendEnvironment.builder() + .vaults(new PropertiesFileCredentialVault(properties)) + .databaseSupports( + DatabaseSupport.builder() + .type(RelationalDatabaseType.POSTGRES) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD).authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class + ).build() ) .build() ); this.environment = environmentBuilder.build(); - this.identityFactory = new IdentityFactory.Builder(this.environment) + this.identityFactory = IdentityFactory.builder() + .environment(this.environment) .build(); - this.storeInstanceProvider = new InstrumentedStoreInstanceProvider(); ConnectionSpecification connectionSpecification = new StaticJDBCConnectionSpecification( this.postgresContainer.getHost(), this.postgresContainer.getPort(), this.postgresContainer.getDatabaseName() ); - this.storeInstance = new StoreInstance.Builder(this.environment) - .withIdentifier(TEST_STORE_INSTANCE_NAME) - .withStoreSupportIdentifier("Postgres") - .withConnectionSpecification(connectionSpecification) + this.connection = Connection.builder() + .databaseSupport(this.environment.getDatabaseSupport(RelationalDatabaseType.POSTGRES)) + .identifier("test::connection") + .connectionSpecification(connectionSpecification) + .authenticationConfiguration(new UserPasswordAuthenticationConfiguration( + postgresContainer.getUser(), + new PropertiesFileSecret("passwordRef") + )) .build(); } @@ -114,39 +115,37 @@ public void testBasicConnectionPooling() throws Exception .withConnectionTimeout(1000L) .build() ); - this.connectionFactory = new ConnectionFactory.Builder(this.environment, this.storeInstanceProvider) - .withCredentialBuilders( + this.connectionFactory = ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders( new UserPasswordCredentialBuilder() ) - .withConnectionBuilders( + .connectionBuilders( customizedJDBCConnectionBuilder ) .build(); - this.storeInstanceProvider.injectStoreInstance(this.storeInstance); Identity identity = identityFactory.createIdentity( - new IdentitySpecification.Builder() - .withName("test-user") + IdentitySpecification.builder() + .name("test-user") .build() ); - ConnectionSpecification connectionSpecification = this.storeInstance.getConnectionSpecification(); - AuthenticationConfiguration authenticationConfiguration = new UserPasswordAuthenticationConfiguration( - postgresContainer.getUser(), - new PropertiesFileSecret("passwordRef") - ); - Authenticator authenticator = this.connectionFactory.getAuthenticator(identity, TEST_STORE_INSTANCE_NAME, authenticationConfiguration); + ConnectionSpecification connectionSpecification = this.connection.getConnectionSpecification(); + AuthenticationConfiguration authenticationConfiguration = this.connection.getAuthenticationConfiguration(); + + Authenticator authenticator = this.connectionFactory.getAuthenticator(identity, this.connection); JDBCConnectionManager connectionManager = JDBCConnectionManager.getInstance(); Assertions.assertEquals(0, connectionManager.getPoolSize()); // 1. Get a connection, this should initialize the pool as well as create a new connection in the empty pool // this connection should be active - Connection connection0 = this.connectionFactory.getConnection(identity, authenticator); + java.sql.Connection connection0 = this.connectionFactory.getConnection(identity, authenticator); String poolName = JDBCConnectionManager.getPoolName(identity, connectionSpecification, authenticationConfiguration); JDBCConnectionManager.ConnectionPool connectionPool = connectionManager.getPool(poolName); // 2. Close the connection, verify that the pool keeps this connection around in idle state - Connection underlyingConnection0 = connection0.unwrap(Connection.class); + java.sql.Connection underlyingConnection0 = connection0.unwrap(java.sql.Connection.class); connection0.close(); Assertions.assertEquals(1, connectionPool.getTotalConnections()); @@ -154,9 +153,9 @@ public void testBasicConnectionPooling() throws Exception Assertions.assertEquals(1, connectionPool.getIdleConnections()); // 3. Get a new connection, the pool should return the idle connection and create no new connection - Connection connection1 = this.connectionFactory.getConnection(identity, authenticator); + java.sql.Connection connection1 = this.connectionFactory.getConnection(identity, authenticator); - Assertions.assertEquals(underlyingConnection0, connection1.unwrap(Connection.class)); + Assertions.assertEquals(underlyingConnection0, connection1.unwrap(java.sql.Connection.class)); Assertions.assertEquals(1, connectionPool.getTotalConnections()); Assertions.assertEquals(1, connectionPool.getActiveConnections()); Assertions.assertEquals(0, connectionPool.getIdleConnections()); @@ -180,36 +179,33 @@ public void testBasicConnectionPooling() throws Exception @Test public void testConnectionPoolingForDifferentIdentities() throws Exception { - this.connectionFactory = new ConnectionFactory.Builder(this.environment, this.storeInstanceProvider) - .withCredentialBuilders( + this.connectionFactory = ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders( new UserPasswordCredentialBuilder() ) - .withConnectionBuilders( + .connectionBuilders( new StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword() ) .build(); - this.storeInstanceProvider.injectStoreInstance(this.storeInstance); Identity identity1 = identityFactory.createIdentity( - new IdentitySpecification.Builder() - .withName("testUser1") + IdentitySpecification.builder() + .name("testUser1") .build() ); Identity identity2 = identityFactory.createIdentity( - new IdentitySpecification.Builder() - .withName("testUser2") + IdentitySpecification.builder() + .name("testUser2") .build() ); - ConnectionSpecification connectionSpecification = this.storeInstance.getConnectionSpecification(); - AuthenticationConfiguration authenticationConfiguration = new UserPasswordAuthenticationConfiguration( - postgresContainer.getUser(), - new PropertiesFileSecret("passwordRef") - ); + ConnectionSpecification connectionSpecification = this.connection.getConnectionSpecification(); + AuthenticationConfiguration authenticationConfiguration = this.connection.getAuthenticationConfiguration(); JDBCConnectionManager connectionManager = JDBCConnectionManager.getInstance(); Assertions.assertEquals(0, connectionManager.getPoolSize()); // 1. Get a new connection for identity1, which should initialize a pool - this.connectionFactory.getConnection(identity1, this.connectionFactory.getAuthenticator(identity1, TEST_STORE_INSTANCE_NAME, authenticationConfiguration)); + this.connectionFactory.getConnection(identity1, this.connectionFactory.getAuthenticator(identity1, this.connection)); String poolName1 = JDBCConnectionManager.getPoolName(identity1, connectionSpecification, authenticationConfiguration); JDBCConnectionManager.ConnectionPool connectionPool1 = connectionManager.getPool(poolName1); @@ -220,7 +216,7 @@ public void testConnectionPoolingForDifferentIdentities() throws Exception Assertions.assertEquals(0, connectionPool1.getIdleConnections()); // 2. Get a new connection for identity2, which should initialize another pool - this.connectionFactory.getConnection(identity2, this.connectionFactory.getAuthenticator(identity2, TEST_STORE_INSTANCE_NAME, authenticationConfiguration)); + this.connectionFactory.getConnection(identity2, this.connectionFactory.getAuthenticator(identity2, this.connection)); String poolName2 = JDBCConnectionManager.getPoolName(identity2, connectionSpecification, authenticationConfiguration); JDBCConnectionManager.ConnectionPool connectionPool2 = connectionManager.getPool(poolName2); @@ -234,7 +230,7 @@ public void testConnectionPoolingForDifferentIdentities() throws Exception @Test public void testRetryOnBrokenConnection() { - // + // TODO } public static class CustomAdvice @@ -246,50 +242,4 @@ public static void intercept(@Advice.Return(readOnly = false) String value) value = "hi: " + value; } } - -// public static class MyWay -// { -// } -// -// private static class InstrumentedStaticJDBCConnectionBuilder -// { -// static class WithPlaintextUsernamePassword extends StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword -// { -// WithPlaintextUsernamePassword(Function hikariConfigHandler) -// { -// this.connectionManager = new InstrumentedJDBCConnectionManager(hikariConfigHandler); -// } -// -// @Override -// public JDBCConnectionManager getConnectionManager() -// { -// return this.connectionManager; -// } -// -// @Override -// protected Type[] actualTypeArguments() -// { -// Type genericSuperClass = this.getClass().getSuperclass().getGenericSuperclass(); -// ParameterizedType parameterizedType = (ParameterizedType) genericSuperClass; -// return parameterizedType.getActualTypeArguments(); -// } -// } -// } -// -// private static class InstrumentedJDBCConnectionManager extends JDBCConnectionManager -// { -// private final Function hikariConfigHandler; -// -// InstrumentedJDBCConnectionManager(Function hikariConfigHandler) -// { -// this.hikariConfigHandler = hikariConfigHandler; -// } -// -//// @Override -//// protected void handleHikariConfig(HikariConfig config) -//// { -//// config.setRegisterMbeans(true); -//// this.hikariConfigHandler.apply(config); -//// } -// } } diff --git a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestPostgresConnection.java b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestPostgresConnection.java index 9e311f05fa8..6c9addedd6b 100644 --- a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestPostgresConnection.java +++ b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestPostgresConnection.java @@ -16,18 +16,16 @@ import org.finos.legend.authentication.vault.CredentialVault; import org.finos.legend.authentication.vault.impl.PropertiesFileCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; +import org.finos.legend.connection.DatabaseType; import org.finos.legend.connection.PostgresTestContainerWrapper; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; -import org.finos.legend.connection.protocol.StaticJDBCConnectionSpecification; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.connection.impl.RelationalDatabaseType; +import org.finos.legend.engine.protocol.pure.v1.model.connection.StaticJDBCConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.PropertiesFileSecret; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; -import java.sql.Connection; import java.sql.Statement; import java.util.Properties; @@ -35,7 +33,7 @@ public class TestPostgresConnection { - public static class WithUserPassword extends AbstractConnectionFactoryTest + public static class WithUserPassword extends AbstractConnectionFactoryTest { private PostgresTestContainerWrapper postgresContainer; @@ -71,27 +69,25 @@ public CredentialVault getCredentialVault() } @Override - public StoreInstance getStoreInstance() + public Identity getIdentity() { - ConnectionSpecification connectionSpecification = new StaticJDBCConnectionSpecification( - this.postgresContainer.getHost(), - this.postgresContainer.getPort(), - this.postgresContainer.getDatabaseName() - ); - return new StoreInstance.Builder(this.environment) - .withIdentifier(TEST_STORE_INSTANCE_NAME) - .withStoreSupportIdentifier("Postgres") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).build() - ) - .withConnectionSpecification(connectionSpecification) - .build(); + return getAnonymousIdentity(this.identityFactory); } @Override - public Identity getIdentity() + public DatabaseType getDatabaseType() { - return getAnonymousIdentity(this.identityFactory); + return RelationalDatabaseType.POSTGRES; + } + + @Override + public ConnectionSpecification getConnectionSpecification() + { + return new StaticJDBCConnectionSpecification( + this.postgresContainer.getHost(), + this.postgresContainer.getPort(), + this.postgresContainer.getDatabaseName() + ); } @Override @@ -104,7 +100,7 @@ public AuthenticationConfiguration getAuthenticationConfiguration() } @Override - public void runTestWithConnection(Connection connection) throws Exception + public void runTestWithConnection(java.sql.Connection connection) throws Exception { Statement statement = connection.createStatement(); statement.setMaxRows(10); diff --git a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestSnowflakeConnection.java b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestSnowflakeConnection.java index 825725a5eea..b0874a64e54 100644 --- a/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestSnowflakeConnection.java +++ b/legend-engine-config/legend-engine-connection-integration-tests/src/test/java/org/finos/legend/engine/connection/test/TestSnowflakeConnection.java @@ -16,17 +16,16 @@ import org.finos.legend.authentication.vault.CredentialVault; import org.finos.legend.authentication.vault.impl.PropertiesFileCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.impl.EncryptedPrivateKeyPairAuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; -import org.finos.legend.connection.protocol.SnowflakeConnectionSpecification; +import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.impl.RelationalDatabaseType; +import org.finos.legend.engine.protocol.pure.v1.connection.SnowflakeConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.EnvironmentCredentialVaultSecret; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.PropertiesFileSecret; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; -import java.sql.Connection; import java.sql.Statement; import java.util.Properties; @@ -34,7 +33,7 @@ public class TestSnowflakeConnection { - public static class WithKeyPair extends AbstractConnectionFactoryTest + public static class ForSnowflakeWithKeyPairFlow extends AbstractConnectionFactoryTest { private static final String CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK = "CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK"; private static final String CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK_PASSPHRASE = "CONNECTION_INTEGRATION_TEST__SNOWFLAKE_PK_PASSPHRASE"; @@ -71,7 +70,19 @@ public CredentialVault getCredentialVault() } @Override - public StoreInstance getStoreInstance() + public Identity getIdentity() + { + return getAnonymousIdentity(this.identityFactory); + } + + @Override + public DatabaseType getDatabaseType() + { + return RelationalDatabaseType.SNOWFLAKE; + } + + @Override + public ConnectionSpecification getConnectionSpecification() { SnowflakeConnectionSpecification connectionSpecification = new SnowflakeConnectionSpecification(); connectionSpecification.databaseName = "SUMMIT_DEV"; @@ -80,20 +91,7 @@ public StoreInstance getStoreInstance() connectionSpecification.region = "us-east-2"; connectionSpecification.cloudType = "aws"; connectionSpecification.role = "SUMMIT_DEV"; - return new StoreInstance.Builder(this.environment) - .withIdentifier(TEST_STORE_INSTANCE_NAME) - .withStoreSupportIdentifier("Snowflake") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KEY_PAIR).build() - ) - .withConnectionSpecification(connectionSpecification) - .build(); - } - - @Override - public Identity getIdentity() - { - return getAnonymousIdentity(this.identityFactory); + return connectionSpecification; } @Override @@ -107,7 +105,7 @@ public AuthenticationConfiguration getAuthenticationConfiguration() } @Override - public void runTestWithConnection(Connection connection) throws Exception + public void runTestWithConnection(java.sql.Connection connection) throws Exception { Statement statement = connection.createStatement(); statement.setMaxRows(10); diff --git a/legend-engine-config/legend-engine-extensions-collection-generation/pom.xml b/legend-engine-config/legend-engine-extensions-collection-generation/pom.xml index 5e5b259bc69..e702972ddc2 100644 --- a/legend-engine-config/legend-engine-extensions-collection-generation/pom.xml +++ b/legend-engine-config/legend-engine-extensions-collection-generation/pom.xml @@ -250,6 +250,21 @@ + + + org.finos.legend.engine + legend-engine-xt-connection-grammar + + + org.finos.legend.engine + legend-engine-xt-connection-compiler + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + + + org.finos.legend.engine diff --git a/legend-engine-config/legend-engine-extensions-collection-generation/src/test/java/org/finos/legend/engine/extensions/collection/generation/TestExtensions.java b/legend-engine-config/legend-engine-extensions-collection-generation/src/test/java/org/finos/legend/engine/extensions/collection/generation/TestExtensions.java index c1865923cf6..84d174859f4 100644 --- a/legend-engine-config/legend-engine-extensions-collection-generation/src/test/java/org/finos/legend/engine/extensions/collection/generation/TestExtensions.java +++ b/legend-engine-config/legend-engine-extensions-collection-generation/src/test/java/org/finos/legend/engine/extensions/collection/generation/TestExtensions.java @@ -33,6 +33,7 @@ import org.finos.legend.engine.language.bigqueryFunction.grammar.to.BigQueryFunctionGrammarComposer; import org.finos.legend.engine.language.graphQL.grammar.integration.GraphQLGrammarParserExtension; import org.finos.legend.engine.language.graphQL.grammar.integration.GraphQLPureGrammarComposerExtension; +import org.finos.legend.engine.language.pure.compiler.toPureGraph.ConnectionCompilerExtension; import org.finos.legend.engine.language.pure.compiler.toPureGraph.PureModel; import org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.CompilerExtension; import org.finos.legend.engine.language.pure.dsl.authentication.grammar.from.AuthenticationGrammarParserExtension; @@ -51,6 +52,7 @@ import org.finos.legend.engine.language.pure.dsl.persistence.relational.grammar.to.PersistenceRelationalComposerExtension; import org.finos.legend.engine.language.pure.dsl.service.grammar.from.ServiceParserExtension; import org.finos.legend.engine.language.pure.dsl.service.grammar.to.ServiceGrammarComposerExtension; +import org.finos.legend.engine.language.pure.grammar.from.ConnectionParserExtension; import org.finos.legend.engine.language.pure.grammar.from.CorePureGrammarParser; import org.finos.legend.engine.language.pure.grammar.from.DataSpaceParserExtension; import org.finos.legend.engine.language.pure.grammar.from.DiagramParserExtension; @@ -60,6 +62,7 @@ import org.finos.legend.engine.language.pure.grammar.from.TextParserExtension; import org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtension; import org.finos.legend.engine.language.pure.grammar.to.BigQueryGrammarComposerExtension; +import org.finos.legend.engine.language.pure.grammar.to.ConnectionGrammarComposerExtension; import org.finos.legend.engine.language.pure.grammar.to.CorePureGrammarComposer; import org.finos.legend.engine.language.pure.grammar.to.DataSpaceGrammarComposerExtension; import org.finos.legend.engine.language.pure.grammar.to.DiagramGrammarComposerExtension; @@ -269,6 +272,7 @@ protected Iterable> getExpected return Lists.mutable.>empty() .with(org.finos.legend.engine.protocol.pure.v1.CorePureProtocolExtension.class) .with(org.finos.legend.engine.protocol.pure.v1.DataSpaceProtocolExtension.class) + .with(org.finos.legend.engine.protocol.pure.v1.ConnectionProtocolExtension.class) .with(SnowflakeAppProtocolExtension.class) .with(BigQueryFunctionProtocolExtension.class) .with(org.finos.legend.engine.protocol.pure.v1.DiagramProtocolExtension.class) @@ -314,6 +318,7 @@ protected Iterable> getExp return Lists.mutable.>empty() .with(CorePureGrammarParser.class) .with(DataSpaceParserExtension.class) + .with(ConnectionParserExtension.class) .with(SnowflakeAppGrammarParserExtension.class) .with(BigQueryFunctionGrammarParserExtension.class) .with(DiagramParserExtension.class) @@ -341,6 +346,7 @@ protected Iterable> getE return Lists.mutable.>empty() .with(CorePureGrammarComposer.class) .with(DataSpaceGrammarComposerExtension.class) + .with(ConnectionGrammarComposerExtension.class) .with(SnowflakeAppGrammarComposer.class) .with(BigQueryFunctionGrammarComposer.class) .with(DiagramGrammarComposerExtension.class) @@ -376,6 +382,7 @@ protected Iterable> getExpectedComp .with(SnowflakeAppCompilerExtension.class) .with(BigQueryFunctionCompilerExtension.class) .with(org.finos.legend.engine.language.pure.compiler.toPureGraph.DataSpaceCompilerExtension.class) + .with(ConnectionCompilerExtension.class) .with(org.finos.legend.engine.language.pure.compiler.toPureGraph.TextCompilerExtension.class) .with(org.finos.legend.engine.language.pure.compiler.toPureGraph.CoreCompilerExtension.class) .with(org.finos.legend.engine.language.pure.dsl.generation.compiler.toPureGraph.GenerationCompilerExtensionImpl.class) @@ -485,6 +492,7 @@ protected Iterable getExpectedCodeRepositories() .with("core_analytics_lineage") .with("core_analytics_mapping") .with("core_analytics_search") + .with("core_connection_metamodel") .with("core_data_space") .with("core_data_space_metamodel") .with("core_diagram") diff --git a/legend-engine-config/legend-engine-server/pom.xml b/legend-engine-config/legend-engine-server/pom.xml index 6b5bc974bce..37ed8f3b41e 100644 --- a/legend-engine-config/legend-engine-server/pom.xml +++ b/legend-engine-config/legend-engine-server/pom.xml @@ -263,7 +263,16 @@ runtime - + + org.finos.legend.engine + legend-engine-xt-connection-compiler + runtime + + + org.finos.legend.engine + legend-engine-xt-connection-grammar + runtime + org.finos.legend.engine legend-engine-xt-persistence-grammar @@ -447,10 +456,6 @@ org.finos.legend.engine legend-engine-xt-analytics-binding-api - - org.finos.legend.engine - legend-engine-xt-analytics-binding-api - org.finos.legend.engine legend-engine-xt-relationalStore-grammar @@ -748,7 +753,11 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory + + + org.finos.legend.engine + legend-engine-xt-connection-protocol org.finos.legend.engine diff --git a/legend-engine-config/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java b/legend-engine-config/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java index 5e185307baa..035ec9fd3e1 100644 --- a/legend-engine-config/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java +++ b/legend-engine-config/legend-engine-server/src/main/java/org/finos/legend/engine/server/Server.java @@ -43,22 +43,18 @@ import org.finos.legend.authentication.vault.impl.EnvironmentCredentialVault; import org.finos.legend.authentication.vault.impl.PropertiesFileCredentialVault; import org.finos.legend.authentication.vault.impl.SystemPropertiesCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; +import org.finos.legend.connection.AuthenticationMechanism; import org.finos.legend.connection.ConnectionFactory; -import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.DatabaseSupport; import org.finos.legend.connection.LegendEnvironment; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstanceProvider; -import org.finos.legend.connection.impl.DefaultStoreInstanceProvider; -import org.finos.legend.connection.impl.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; import org.finos.legend.connection.impl.HACKY__SnowflakeConnectionAdapter; import org.finos.legend.connection.impl.KerberosCredentialExtractor; import org.finos.legend.connection.impl.KeyPairCredentialBuilder; +import org.finos.legend.connection.impl.RelationalDatabaseType; import org.finos.legend.connection.impl.SnowflakeConnectionBuilder; import org.finos.legend.connection.impl.StaticJDBCConnectionBuilder; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; import org.finos.legend.engine.api.analytics.BindingAnalytics; import org.finos.legend.engine.api.analytics.ClassAnalytics; import org.finos.legend.engine.api.analytics.DataSpaceAnalytics; @@ -126,6 +122,8 @@ import org.finos.legend.engine.protocol.bigqueryFunction.metamodel.BigQueryFunctionDeploymentConfiguration; import org.finos.legend.engine.protocol.pure.v1.PureProtocolObjectMapperFactory; import org.finos.legend.engine.protocol.pure.v1.model.PureProtocol; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.pure.code.core.PureCoreExtensionLoader; import org.finos.legend.engine.query.graphQL.api.debug.GraphQLDebug; import org.finos.legend.engine.query.graphQL.api.execute.GraphQLExecute; @@ -399,7 +397,7 @@ public void run(T serverConfiguration, Environment environment) new RelationalStoreSQLSourceProvider(projectCoordinateLoader), new FunctionSQLSourceProvider(projectCoordinateLoader), new LegendServiceSQLSourceProvider(projectCoordinateLoader)), - generatorExtensions.flatCollect(PlanGeneratorExtension::getExtraPlanTransformers)))); + generatorExtensions.flatCollect(PlanGeneratorExtension::getExtraPlanTransformers)))); environment.jersey().register(new SqlGrammar()); // Service @@ -439,39 +437,48 @@ public void run(T serverConfiguration, Environment environment) // TODO: @akphi - this is temporary, rework when we find a better way to handle the initialization of connection factory from config or some external source. private ConnectionFactory setupConnectionFactory(List vaultConfigurations) { - LegendEnvironment environment = new LegendEnvironment.Builder() - .withVaults( + LegendEnvironment environment = LegendEnvironment + .builder() + .vaults( new SystemPropertiesCredentialVault(), new EnvironmentCredentialVault(), new PropertiesFileCredentialVault(this.buildVaultProperties(vaultConfigurations)) ) - .withStoreSupports( - new RelationalDatabaseStoreSupport.Builder(DatabaseType.POSTGRES) - .withIdentifier("Postgres") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).withAuthenticationConfigurationTypes( - UserPasswordAuthenticationConfiguration.class - ).build() + .databaseSupports( + DatabaseSupport + .builder() + .type(RelationalDatabaseType.POSTGRES) + .authenticationMechanisms( + AuthenticationMechanism + .builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class + ).build() ) .build(), - new RelationalDatabaseStoreSupport.Builder(DatabaseType.SNOWFLAKE) - .withIdentifier("Snowflake") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KEY_PAIR).withAuthenticationConfigurationTypes( - EncryptedPrivateKeyPairAuthenticationConfiguration.class - ).build() + DatabaseSupport + .builder() + .type(RelationalDatabaseType.SNOWFLAKE) + .authenticationMechanisms( + AuthenticationMechanism + .builder() + .type(CoreAuthenticationMechanismType.KEY_PAIR) + .authenticationConfigurationTypes( + EncryptedPrivateKeyPairAuthenticationConfiguration.class + ).build() ) .build() ).build(); - StoreInstanceProvider storeInstanceProvider = new DefaultStoreInstanceProvider.Builder().build(); - return new ConnectionFactory.Builder(environment, storeInstanceProvider) - .withCredentialBuilders( + return ConnectionFactory.builder() + .environment(environment) + .credentialBuilders( new KerberosCredentialExtractor(), new UserPasswordCredentialBuilder(), new KeyPairCredentialBuilder() ) - .withConnectionBuilders( + .connectionBuilders( new StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword(), new SnowflakeConnectionBuilder.WithKeyPair() ) diff --git a/legend-engine-xts-analytics/legend-engine-xts-analytics-mapping/legend-engine-xt-analytics-mapping-api/pom.xml b/legend-engine-xts-analytics/legend-engine-xts-analytics-mapping/legend-engine-xt-analytics-mapping-api/pom.xml index ec437e0118d..55b054729fe 100644 --- a/legend-engine-xts-analytics/legend-engine-xts-analytics-mapping/legend-engine-xt-analytics-mapping-api/pom.xml +++ b/legend-engine-xts-analytics/legend-engine-xts-analytics-mapping/legend-engine-xt-analytics-mapping-api/pom.xml @@ -62,10 +62,6 @@ org.finos.legend.engine legend-engine-xt-analytics-function-pure - - org.finos.legend.engine - legend-engine-xt-analytics-function-pure - org.finos.legend.engine legend-engine-xt-analytics-binding-pure diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismConfiguration.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismConfiguration.java deleted file mode 100644 index 11ca50b9ce1..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismConfiguration.java +++ /dev/null @@ -1,102 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.block.function.Function0; -import org.eclipse.collections.api.factory.Lists; -import org.eclipse.collections.api.list.ImmutableList; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; - -import java.util.ArrayList; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Objects; -import java.util.Set; - -public class AuthenticationMechanismConfiguration -{ - private final AuthenticationMechanism authenticationMechanism; - private final ImmutableList> authenticationConfigurationTypes; - private final Function0 defaultAuthenticationConfigurationGenerator; - - private AuthenticationMechanismConfiguration(AuthenticationMechanism authenticationMechanism, List> authenticationConfigurationTypes, Function0 defaultAuthenticationConfigurationGenerator) - { - this.authenticationMechanism = Objects.requireNonNull(authenticationMechanism, "Authentication mechanism is missing"); - this.authenticationConfigurationTypes = Lists.immutable.withAll(authenticationConfigurationTypes); - this.defaultAuthenticationConfigurationGenerator = defaultAuthenticationConfigurationGenerator; - } - - public AuthenticationMechanism getAuthenticationMechanism() - { - return authenticationMechanism; - } - - public ImmutableList> getAuthenticationConfigurationTypes() - { - return authenticationConfigurationTypes; - } - - public Function0 getDefaultAuthenticationConfigurationGenerator() - { - return defaultAuthenticationConfigurationGenerator; - } - - public static class Builder - { - private final AuthenticationMechanism authenticationMechanism; - private final Set> authenticationConfigurationTypes = new LinkedHashSet<>(); - private Function0 defaultAuthenticationConfigurationGenerator; - - public Builder(AuthenticationMechanism authenticationMechanism) - { - this.authenticationMechanism = authenticationMechanism; - } - - public Builder withAuthenticationConfigurationType(Class authenticationConfigurationType) - { - this.authenticationConfigurationTypes.add(authenticationConfigurationType); - return this; - } - - public Builder withAuthenticationConfigurationTypes(List> authenticationConfigurationTypes) - { - this.authenticationConfigurationTypes.addAll(authenticationConfigurationTypes); - return this; - } - - @SafeVarargs - public final Builder withAuthenticationConfigurationTypes(Class... authenticationConfigurationTypes) - { - this.authenticationConfigurationTypes.addAll(Lists.mutable.of(authenticationConfigurationTypes)); - return this; - } - - public Builder withDefaultAuthenticationConfigurationGenerator(Function0 defaultAuthenticationConfigurationGenerator) - { - this.defaultAuthenticationConfigurationGenerator = defaultAuthenticationConfigurationGenerator; - return this; - } - - public AuthenticationMechanismConfiguration build() - { - return new AuthenticationMechanismConfiguration( - this.authenticationMechanism, - new ArrayList<>(this.authenticationConfigurationTypes), - this.defaultAuthenticationConfigurationGenerator - ); - } - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java deleted file mode 100644 index 7751f5bdecc..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java +++ /dev/null @@ -1,129 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.factory.Lists; -import org.eclipse.collections.api.list.ImmutableList; -import org.eclipse.collections.api.map.ImmutableMap; -import org.eclipse.collections.api.map.MutableMap; -import org.eclipse.collections.impl.factory.Maps; -import org.eclipse.collections.impl.utility.ListIterate; -import org.finos.legend.authentication.vault.CredentialVault; -import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.CredentialVaultSecret; -import org.finos.legend.engine.shared.core.identity.Identity; - -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; - -/** - * This is the runtime instance of configuration for Legend Engine, the place we package common configs, - * such as vaults, that can be passed to various parts of engine, authentication, connection factory, etc. - */ -public class LegendEnvironment -{ - protected final ImmutableList vaults; - protected final ImmutableMap, CredentialVault> vaultsIndex; - protected final ImmutableMap storeSupportsIndex; - - protected LegendEnvironment(List vaults, Map storeSupportsIndex) - { - this.vaults = Lists.immutable.withAll(vaults); - MutableMap, CredentialVault> vaultsIndex = Maps.mutable.empty(); - for (CredentialVault vault : vaults) - { - vaultsIndex.put(vault.getSecretType(), vault); - } - this.vaultsIndex = vaultsIndex.toImmutable(); - this.storeSupportsIndex = Maps.immutable.withAll(storeSupportsIndex); - } - - public String lookupVaultSecret(CredentialVaultSecret credentialVaultSecret, Identity identity) throws Exception - { - Class secretClass = credentialVaultSecret.getClass(); - if (!this.vaultsIndex.containsKey(secretClass)) - { - throw new RuntimeException(String.format("Can't find secret: credential vault for secret of type '%s' has not been registered", secretClass.getSimpleName())); - } - CredentialVault vault = this.vaultsIndex.get(secretClass); - return vault.lookupSecret(credentialVaultSecret, identity); - } - - public StoreSupport findStoreSupport(String identifier) - { - return Objects.requireNonNull(this.storeSupportsIndex.get(identifier), String.format("Can't find store support with identifier '%s'", identifier)); - } - - public static class Builder - { - private final List vaults = Lists.mutable.empty(); - private final Map storeSupportsIndex = new LinkedHashMap<>(); - - public Builder() - { - } - - public Builder withVaults(List vaults) - { - this.vaults.addAll(vaults); - return this; - } - - public Builder withVaults(CredentialVault... vaults) - { - this.vaults.addAll(Lists.mutable.with(vaults)); - return this; - } - - public Builder withVault(CredentialVault vault) - { - this.vaults.add(vault); - return this; - } - - public Builder withStoreSupports(List storeSupports) - { - storeSupports.forEach(this::registerStoreSupport); - return this; - } - - public Builder withStoreSupports(StoreSupport... storeSupports) - { - ListIterate.forEach(Lists.mutable.with(storeSupports), this::registerStoreSupport); - return this; - } - - public Builder withStoreSupport(StoreSupport storeSupport) - { - this.registerStoreSupport(storeSupport); - return this; - } - - private void registerStoreSupport(StoreSupport storeSupport) - { - if (this.storeSupportsIndex.containsKey(storeSupport.getIdentifier())) - { - throw new RuntimeException(String.format("Can't register store support: found multiple store supports with identifier '%s'", storeSupport.getIdentifier())); - } - this.storeSupportsIndex.put(storeSupport.getIdentifier(), storeSupport); - } - - public LegendEnvironment build() - { - return new LegendEnvironment(this.vaults, this.storeSupportsIndex); - } - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstance.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstance.java deleted file mode 100644 index 2ced76c35a8..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstance.java +++ /dev/null @@ -1,221 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.factory.Lists; -import org.eclipse.collections.api.list.ImmutableList; -import org.eclipse.collections.impl.utility.ListIterate; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; -import org.finos.legend.connection.protocol.ConnectionSpecification; - -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; - -/** - * A StoreInstance represents a named instance of a Store. - */ -public final class StoreInstance -{ - private final String identifier; - private final StoreSupport storeSupport; - private final ConnectionSpecification connectionSpecification; - private final Map authenticationMechanismConfigurationIndex; - private final Map, AuthenticationMechanism> authenticationMechanismIndex; - - private StoreInstance(String identifier, StoreSupport storeSupport, List authenticationMechanismConfigurations, ConnectionSpecification connectionSpecification) - { - this.identifier = Objects.requireNonNull(identifier, "Can't create store instance: identifier is missing"); - this.storeSupport = storeSupport; - this.connectionSpecification = Objects.requireNonNull(connectionSpecification, "Connection specification is missing"); - - Map authenticationMechanismConfigurationIndex = new LinkedHashMap<>(); - - if (authenticationMechanismConfigurations.isEmpty()) - { - for (AuthenticationMechanism authenticationMechanism : this.storeSupport.getAuthenticationMechanisms()) - { - authenticationMechanismConfigurationIndex.put(authenticationMechanism, this.storeSupport.getAuthenticationMechanismConfiguration(authenticationMechanism)); - } - } - else - { - for (AuthenticationMechanismConfiguration authenticationMechanismConfiguration : authenticationMechanismConfigurations) - { - AuthenticationMechanism authenticationMechanism = authenticationMechanismConfiguration.getAuthenticationMechanism(); - if (authenticationMechanismConfigurationIndex.containsKey(authenticationMechanism)) - { - throw new RuntimeException(String.format("Found multiple configurations for authentication mechanism '%s'", authenticationMechanism.getLabel())); - } - AuthenticationMechanismConfiguration configFromStoreSupport = this.storeSupport.getAuthenticationMechanismConfiguration(authenticationMechanism); - if (configFromStoreSupport == null) - { - throw new RuntimeException(String.format("Authentication mechanism '%s' is not covered by store support '%s'. Supported mechanism(s):\n%s", - authenticationMechanism.getLabel(), - this.storeSupport.getIdentifier(), - ListIterate.collect(this.storeSupport.getAuthenticationMechanisms(), mechanism -> "- " + mechanism.getLabel()).makeString("\n") - )); - } - ImmutableList> authenticationConfigTypesFromStoreSupport = configFromStoreSupport.getAuthenticationConfigurationTypes(); - List> authenticationConfigurationTypes = Lists.mutable.empty(); - for (Class authenticationConfigurationType : authenticationMechanismConfiguration.getAuthenticationConfigurationTypes()) - { - if (!authenticationConfigTypesFromStoreSupport.contains(authenticationConfigurationType)) - { - throw new RuntimeException(String.format("Authentication configuration type '%s' is not covered by store support '%s' for authentication mechanism '%s'. Supported configuration type(s):\n%s", - authenticationConfigurationType.getSimpleName(), - this.storeSupport.getIdentifier(), - authenticationMechanism.getLabel(), - authenticationConfigTypesFromStoreSupport.collect(type -> "- " + type.getSimpleName()).makeString("\n") - )); - } - else - { - authenticationConfigurationTypes.add(authenticationConfigurationType); - } - } - authenticationMechanismConfigurationIndex.put(authenticationMechanism, new AuthenticationMechanismConfiguration.Builder(authenticationMechanism) - // NOTE: if no configuration type is specified, it means the store instance supports all configuration types configured for that mechanism in the store support - .withAuthenticationConfigurationTypes(!authenticationConfigurationTypes.isEmpty() ? authenticationConfigurationTypes : authenticationConfigTypesFromStoreSupport.toList()) - .withDefaultAuthenticationConfigurationGenerator(authenticationMechanismConfiguration.getDefaultAuthenticationConfigurationGenerator() != null ? authenticationMechanismConfiguration.getDefaultAuthenticationConfigurationGenerator() : configFromStoreSupport.getDefaultAuthenticationConfigurationGenerator()) - .build()); - - } - - } - - this.authenticationMechanismConfigurationIndex = authenticationMechanismConfigurationIndex; - Map, AuthenticationMechanism> authenticationMechanismIndex = new LinkedHashMap<>(); - authenticationMechanismConfigurationIndex.forEach((authenticationMechanism, authenticationMechanismConfiguration) -> - { - if (authenticationMechanismConfiguration.getAuthenticationConfigurationTypes().isEmpty()) - { - throw new RuntimeException(String.format("No authentication configuration type is associated with authentication mechanism '%s'", authenticationMechanism.getLabel())); - } - authenticationMechanismConfiguration.getAuthenticationConfigurationTypes().forEach(configurationType -> - { - authenticationMechanismIndex.put(configurationType, authenticationMechanism); - }); - }); - this.authenticationMechanismIndex = authenticationMechanismIndex; - } - - public String getIdentifier() - { - return identifier; - } - - public StoreSupport getStoreSupport() - { - return storeSupport; - } - - public List getAuthenticationMechanisms() - { - return new ArrayList<>(this.authenticationMechanismConfigurationIndex.keySet()); - } - - public List> getAuthenticationConfigurationTypes() - { - return new ArrayList<>(this.authenticationMechanismIndex.keySet()); - } - - public AuthenticationMechanism getAuthenticationMechanism(Class authenticationConfigurationType) - { - return this.authenticationMechanismIndex.get(authenticationConfigurationType); - } - - public ConnectionSpecification getConnectionSpecification() - { - return connectionSpecification; - } - - public AuthenticationMechanismConfiguration getAuthenticationMechanismConfiguration(AuthenticationMechanism authenticationMechanism) - { - return authenticationMechanismConfigurationIndex.get(authenticationMechanism); - } - - public T getConnectionSpecification(Class clazz) - { - if (!this.connectionSpecification.getClass().equals(clazz)) - { - throw new RuntimeException(String.format("Can't get connection specification of type '%s' for store '%s'", clazz.getSimpleName(), this.identifier)); - } - return (T) this.connectionSpecification; - } - - public static class Builder - { - private final LegendEnvironment environment; - private String identifier; - private String storeSupportIdentifier; - private final List authenticationMechanismConfigurations = Lists.mutable.empty(); - private ConnectionSpecification connectionSpecification; - - public Builder(LegendEnvironment environment) - { - this.environment = environment; - } - - public Builder withIdentifier(String identifier) - { - this.identifier = identifier; - return this; - } - - public Builder withStoreSupportIdentifier(String storeSupportIdentifier) - { - this.storeSupportIdentifier = storeSupportIdentifier; - return this; - } - - public Builder withAuthenticationMechanismConfiguration(AuthenticationMechanismConfiguration authenticationMechanismConfiguration) - { - this.authenticationMechanismConfigurations.add(authenticationMechanismConfiguration); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(List authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(authenticationMechanismConfigurations); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(AuthenticationMechanismConfiguration... authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(Lists.mutable.of(authenticationMechanismConfigurations)); - return this; - } - - public Builder withConnectionSpecification(ConnectionSpecification connectionSpecification) - { - this.connectionSpecification = connectionSpecification; - return this; - } - - public StoreInstance build() - { - return new StoreInstance( - this.identifier, - this.environment.findStoreSupport(Objects.requireNonNull(this.storeSupportIdentifier, "Store support identifier is missing")), - this.authenticationMechanismConfigurations, - this.connectionSpecification - ); - } - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreSupport.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreSupport.java deleted file mode 100644 index fb36a14dc05..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreSupport.java +++ /dev/null @@ -1,122 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.factory.Lists; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; - -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; - -/** - * A StoreSupport describes the capabilities supported by a Store. - * For now, it describes the authentication mechanisms. - */ -public class StoreSupport -{ - private final String identifier; - private final Map authenticationMechanismConfigurationIndex; - - protected StoreSupport(String identifier, List authenticationMechanismConfigurations) - { - this.identifier = Objects.requireNonNull(identifier, "Identifier is missing"); - - Map authenticationMechanismConfigurationIndex = new LinkedHashMap<>(); - Map, AuthenticationMechanism> authenticationConfigurationTypeIndex = new LinkedHashMap<>(); - for (AuthenticationMechanismConfiguration authenticationMechanismConfiguration : authenticationMechanismConfigurations) - { - AuthenticationMechanism authenticationMechanism = authenticationMechanismConfiguration.getAuthenticationMechanism(); - if (authenticationMechanismConfigurationIndex.containsKey(authenticationMechanism)) - { - throw new RuntimeException(String.format("Found multiple configurations for authentication mechanism '%s'", authenticationMechanism.getLabel())); - } - authenticationMechanismConfigurationIndex.put(authenticationMechanism, authenticationMechanismConfiguration); - authenticationMechanismConfiguration.getAuthenticationConfigurationTypes().forEach(authenticationConfigurationType -> - { - if (authenticationConfigurationTypeIndex.containsKey(authenticationConfigurationType)) - { - throw new RuntimeException(String.format("Authentication configuration type '%s' is associated with multiple authentication mechanisms", authenticationConfigurationType.getSimpleName())); - } - authenticationConfigurationTypeIndex.put(authenticationConfigurationType, authenticationMechanism); - }); - } - - this.authenticationMechanismConfigurationIndex = authenticationMechanismConfigurationIndex; - this.authenticationMechanismConfigurationIndex.forEach((authenticationMechanism, authenticationMechanismConfiguration) -> - { - if (authenticationMechanismConfiguration.getAuthenticationConfigurationTypes().isEmpty()) - { - throw new RuntimeException(String.format("No authentication configuration type is associated with authentication mechanism '%s'", authenticationMechanism.getLabel())); - } - }); - } - - public String getIdentifier() - { - return identifier; - } - - public AuthenticationMechanismConfiguration getAuthenticationMechanismConfiguration(AuthenticationMechanism authenticationMechanism) - { - return authenticationMechanismConfigurationIndex.get(authenticationMechanism); - } - - public List getAuthenticationMechanisms() - { - return new ArrayList<>(this.authenticationMechanismConfigurationIndex.keySet()); - } - - public static class Builder - { - private String identifier; - private final List authenticationMechanismConfigurations = Lists.mutable.empty(); - - public Builder withIdentifier(String identifier) - { - this.identifier = identifier; - return this; - } - - public Builder withAuthenticationMechanismConfiguration(AuthenticationMechanismConfiguration authenticationMechanismConfiguration) - { - this.authenticationMechanismConfigurations.add(authenticationMechanismConfiguration); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(List authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(authenticationMechanismConfigurations); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(AuthenticationMechanismConfiguration... authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(Lists.mutable.of(authenticationMechanismConfigurations)); - return this; - } - - public StoreSupport build() - { - return new StoreSupport( - this.identifier, - this.authenticationMechanismConfigurations - ); - } - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/DefaultStoreInstanceProvider.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/DefaultStoreInstanceProvider.java deleted file mode 100644 index 718b6404929..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/DefaultStoreInstanceProvider.java +++ /dev/null @@ -1,85 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection.impl; - -import org.eclipse.collections.api.factory.Lists; -import org.eclipse.collections.api.factory.Maps; -import org.eclipse.collections.api.map.ImmutableMap; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.StoreInstanceProvider; - -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.Objects; - -public class DefaultStoreInstanceProvider implements StoreInstanceProvider -{ - private final ImmutableMap storeInstancesIndex; - - private DefaultStoreInstanceProvider(Map storeInstancesIndex) - { - - this.storeInstancesIndex = Maps.immutable.withAll(storeInstancesIndex); - } - - @Override - public StoreInstance lookup(String identifier) - { - return Objects.requireNonNull(this.storeInstancesIndex.get(identifier), String.format("Can't find store instance with identifier '%s'", identifier)); - } - - public static class Builder - { - private final Map storeInstancesIndex = new HashMap<>(); - - public Builder() - { - - } - - public Builder withStoreInstances(List storeInstances) - { - storeInstances.forEach(this::registerStoreInstance); - return this; - } - - public Builder withStoreInstances(StoreInstance... storeInstances) - { - Lists.mutable.with(storeInstances).forEach(this::registerStoreInstance); - return this; - } - - public Builder withStoreInstance(StoreInstance storeInstance) - { - this.registerStoreInstance(storeInstance); - return this; - } - - private void registerStoreInstance(StoreInstance storeInstance) - { - if (this.storeInstancesIndex.containsKey(storeInstance.getIdentifier())) - { - throw new RuntimeException(String.format("Found multiple store instances with identifier '%s'", storeInstance.getIdentifier())); - } - this.storeInstancesIndex.put(storeInstance.getIdentifier(), storeInstance); - } - - public DefaultStoreInstanceProvider build() - { - return new DefaultStoreInstanceProvider(this.storeInstancesIndex); - } - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/InstrumentedStoreInstanceProvider.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/InstrumentedStoreInstanceProvider.java deleted file mode 100644 index c3438eab9fd..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/InstrumentedStoreInstanceProvider.java +++ /dev/null @@ -1,45 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection.impl; - -import org.eclipse.collections.api.factory.Maps; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.StoreInstanceProvider; - -import java.util.Map; -import java.util.Objects; - -/** - * This is the instrumented version of {@link StoreInstanceProvider} which is used for testing. - */ -public class InstrumentedStoreInstanceProvider implements StoreInstanceProvider -{ - private final Map storeInstancesIndex = Maps.mutable.empty(); - - public void injectStoreInstance(StoreInstance storeInstance) - { - if (this.storeInstancesIndex.containsKey(storeInstance.getIdentifier())) - { - throw new RuntimeException(String.format("Found multiple store instances with identifier '%s'", storeInstance.getIdentifier())); - } - this.storeInstancesIndex.put(storeInstance.getIdentifier(), storeInstance); - } - - @Override - public StoreInstance lookup(String identifier) - { - return Objects.requireNonNull(this.storeInstancesIndex.get(identifier), String.format("Can't find store instance with identifier '%s'", identifier)); - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanismType.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanismType.java deleted file mode 100644 index b60214e8c54..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanismType.java +++ /dev/null @@ -1,48 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection.protocol; - -import org.finos.legend.connection.impl.ApiKeyAuthenticationConfiguration; -import org.finos.legend.connection.impl.EncryptedPrivateKeyPairAuthenticationConfiguration; -import org.finos.legend.connection.impl.KerberosAuthenticationConfiguration; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; -import org.finos.legend.engine.shared.core.identity.Credential; -import org.finos.legend.engine.shared.core.identity.credential.ApiTokenCredential; -import org.finos.legend.engine.shared.core.identity.credential.LegendKerberosCredential; -import org.finos.legend.engine.shared.core.identity.credential.OAuthCredential; -import org.finos.legend.engine.shared.core.identity.credential.PlaintextUserPasswordCredential; -import org.finos.legend.engine.shared.core.identity.credential.PrivateKeyCredential; - -public enum AuthenticationMechanismType implements AuthenticationMechanism -{ - USER_PASSWORD("UsernamePassword"), - API_KEY("APIKey"), - KEY_PAIR("KeyPair"), - KERBEROS("Kerberos"), - OAUTH("OAuth"); - - private final String label; - - private AuthenticationMechanismType(String label) - { - this.label = label; - } - - @Override - public String getLabel() - { - return this.label; - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/StoreSupportTest.java b/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/StoreSupportTest.java deleted file mode 100644 index 3a7f85d05ff..00000000000 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/StoreSupportTest.java +++ /dev/null @@ -1,254 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.factory.Lists; -import org.finos.legend.connection.impl.DefaultStoreInstanceProvider; -import org.finos.legend.connection.impl.EncryptedPrivateKeyPairAuthenticationConfiguration; -import org.finos.legend.connection.impl.KerberosAuthenticationConfiguration; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; -import org.finos.legend.connection.protocol.ConnectionSpecification; -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -public class StoreSupportTest -{ - @Test - public void testValidateStoreSupportBuilder() - { - // success - new StoreSupport.Builder() - .withIdentifier("test") - .build(); - - // failure - Exception exception; - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreSupport.Builder().build(); - }); - Assertions.assertEquals("Identifier is missing", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) - .build(), - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) - .build() - ).build(); - }); - Assertions.assertEquals("Found multiple configurations for authentication mechanism 'UsernamePassword'", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) - .build(), - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KERBEROS) - .withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) - .build() - ).build(); - }); - Assertions.assertEquals("Authentication configuration type 'UserPasswordAuthenticationConfiguration' is associated with multiple authentication mechanisms", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .build() - ).build(); - }); - Assertions.assertEquals("No authentication configuration type is associated with authentication mechanism 'UsernamePassword'", exception.getMessage()); - } - - @Test - public void testValidateStoreInstanceBuilder() - { - LegendEnvironment environment = new LegendEnvironment.Builder() - .withStoreSupport(new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes( - UserPasswordAuthenticationConfiguration.class, - EncryptedPrivateKeyPairAuthenticationConfiguration.class - ) - .build(), - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KERBEROS) - .withAuthenticationConfigurationTypes(KerberosAuthenticationConfiguration.class) - .build() - ) - .build()) - .build(); - - // success - StoreInstance testStore = new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .build() - ) - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - Assertions.assertArrayEquals(new AuthenticationMechanism[]{AuthenticationMechanismType.USER_PASSWORD}, testStore.getAuthenticationMechanisms().toArray()); - - // make sure if no auth mechanisms is specified, all mechanisms will be supported - StoreInstance testStore2 = new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - Assertions.assertArrayEquals(new AuthenticationMechanism[]{AuthenticationMechanismType.USER_PASSWORD, AuthenticationMechanismType.KERBEROS}, testStore2.getAuthenticationMechanisms().toArray()); - - // make sure if no authentication configuration type is specified, all types will be supported - StoreInstance testStore3 = new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withConnectionSpecification(new TestConnectionSpecification()) - .withAuthenticationMechanismConfiguration(new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).build()) - .build(); - Assertions.assertArrayEquals(Lists.mutable.of( - UserPasswordAuthenticationConfiguration.class, - EncryptedPrivateKeyPairAuthenticationConfiguration.class - ).toArray(), testStore3.getAuthenticationMechanismConfiguration(AuthenticationMechanismType.USER_PASSWORD).getAuthenticationConfigurationTypes().toArray()); - - // failure - Exception exception; - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .build(); - }); - Assertions.assertEquals("Connection specification is missing", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .build(), - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .build() - ) - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - }); - Assertions.assertEquals("Found multiple configurations for authentication mechanism 'UsernamePassword'", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.API_KEY) - .build() - ) - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - }); - Assertions.assertEquals("Authentication mechanism 'APIKey' is not covered by store support 'test'. Supported mechanism(s):\n" + - "- UsernamePassword\n" + - "- Kerberos", exception.getMessage()); - - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes(KerberosAuthenticationConfiguration.class) - .build() - ) - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - }); - Assertions.assertEquals("Authentication configuration type 'KerberosAuthenticationConfiguration' is not covered by store support 'test' for authentication mechanism 'UsernamePassword'. Supported configuration type(s):\n" + - "- UserPasswordAuthenticationConfiguration\n" + - "- EncryptedPrivateKeyPairAuthenticationConfiguration", exception.getMessage()); - } - - private static class TestConnectionSpecification extends ConnectionSpecification - { - @Override - public String shortId() - { - return null; - } - } - - @Test - public void testStoreInstanceManagement() - { - LegendEnvironment environment = new LegendEnvironment.Builder() - .withStoreSupport(new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD) - .withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) - .build(), - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.KERBEROS) - .withAuthenticationConfigurationTypes(KerberosAuthenticationConfiguration.class) - .build() - ) - .build()) - .build(); - - StoreInstance storeInstance = new StoreInstance.Builder(environment) - .withIdentifier("test-store") - .withStoreSupportIdentifier("test") - .withConnectionSpecification(new TestConnectionSpecification()) - .build(); - - StoreInstanceProvider storeInstanceProvider = new DefaultStoreInstanceProvider.Builder().withStoreInstance(storeInstance).build(); - - // failure - Exception exception; - - // error: store already registered - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - new DefaultStoreInstanceProvider.Builder().withStoreInstances(storeInstance, storeInstance).build(); - }); - Assertions.assertEquals("Found multiple store instances with identifier 'test-store'", exception.getMessage()); - - // error: store not found - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - storeInstanceProvider.lookup("unknown"); - }); - Assertions.assertEquals("Can't find store instance with identifier 'unknown'", exception.getMessage()); - } -} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-implementation-vault-aws/pom.xml b/legend-engine-xts-authentication/legend-engine-xt-authentication-implementation-vault-aws/pom.xml index 1e8353adb54..681b8673670 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-implementation-vault-aws/pom.xml +++ b/legend-engine-xts-authentication/legend-engine-xt-authentication-implementation-vault-aws/pom.xml @@ -54,7 +54,7 @@ software.amazon.awssdk auth - 2.17.129 + ${amazon.awssdk.version} software.amazon.awssdk @@ -101,7 +101,7 @@ software.amazon.awssdk secretsmanager - 2.17.129 + ${amazon.awssdk.version} io.netty diff --git a/legend-engine-xts-authentication/pom.xml b/legend-engine-xts-authentication/pom.xml index 1613896c51c..6df60d90e1c 100644 --- a/legend-engine-xts-authentication/pom.xml +++ b/legend-engine-xts-authentication/pom.xml @@ -14,7 +14,8 @@ See the License for the specific language governing permissions and limitations under the License. --> - + org.finos.legend.engine legend-engine @@ -28,7 +29,6 @@ legend-engine-xt-authentication-grammar - legend-engine-xt-authentication-connection-factory legend-engine-xt-authentication-implementation-core legend-engine-xt-authentication-implementation-gcp-federation legend-engine-xt-authentication-implementation-vault-aws diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-compiler/pom.xml b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/pom.xml new file mode 100644 index 00000000000..b9f95efb51d --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/pom.xml @@ -0,0 +1,73 @@ + + + + + + org.finos.legend.engine + legend-engine-xts-connection + 4.35.4-SNAPSHOT + + 4.0.0 + + legend-engine-xt-connection-compiler + jar + Legend Engine - XT - Connection - Compiler + + + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + + + org.finos.legend.engine + legend-engine-xt-connection-pure-metamodel + + + org.finos.legend.engine + legend-engine-language-pure-compiler + + + + + + org.finos.legend.pure + legend-pure-m3-core + + + + + + org.eclipse.collections + eclipse-collections-api + + + + + + org.junit.jupiter + junit-jupiter-api + test + + + org.junit.jupiter + junit-jupiter-engine + test + + + + \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/java/org/finos/legend/engine/language/pure/compiler/toPureGraph/ConnectionCompilerExtension.java b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/java/org/finos/legend/engine/language/pure/compiler/toPureGraph/ConnectionCompilerExtension.java new file mode 100644 index 00000000000..3276c0f3d49 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/java/org/finos/legend/engine/language/pure/compiler/toPureGraph/ConnectionCompilerExtension.java @@ -0,0 +1,51 @@ +// Copyright 2020 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.language.pure.compiler.toPureGraph; + +import org.eclipse.collections.api.factory.Maps; +import org.eclipse.collections.api.map.MutableMap; +import org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.CompilerExtension; +import org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.Processor; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection; +import org.finos.legend.pure.generated.Root_meta_pure_metamodel_connection_Connection; +import org.finos.legend.pure.generated.Root_meta_pure_metamodel_connection_Connection_Impl; + +import java.util.Collections; + +public class ConnectionCompilerExtension implements CompilerExtension +{ + static final MutableMap connectionsIndex = Maps.mutable.empty(); + + @Override + public CompilerExtension build() + { + return new ConnectionCompilerExtension(); + } + + @Override + public Iterable> getExtraProcessors() + { + return Collections.singletonList(Processor.newProcessor( + Connection.class, + (element, context) -> + { + // @HACKY: new-connection-framework + Root_meta_pure_metamodel_connection_Connection metamodel = new Root_meta_pure_metamodel_connection_Connection_Impl(element.name, null, context.pureModel.getClass("meta::pure::metamodel::connection::Connection"))._name(element.name); + connectionsIndex.put(context.pureModel.buildPackageString(element._package, element.name), metamodel); + metamodel._rawValue(element); + return metamodel; + })); + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.CompilerExtension b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.CompilerExtension new file mode 100644 index 00000000000..c6f728c13a9 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-compiler/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.compiler.toPureGraph.extension.CompilerExtension @@ -0,0 +1 @@ +org.finos.legend.engine.language.pure.compiler.toPureGraph.ConnectionCompilerExtension diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/pom.xml b/legend-engine-xts-connection/legend-engine-xt-connection-factory/pom.xml similarity index 85% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/pom.xml rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/pom.xml index 2683a301d47..363f8907ab9 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/pom.xml +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/pom.xml @@ -18,14 +18,14 @@ org.finos.legend.engine - legend-engine-xts-authentication + legend-engine-xts-connection 4.35.4-SNAPSHOT 4.0.0 - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory jar - Legend Engine - XT - Authentication - Connection Factory + Legend Engine - XT - Connection - Factory @@ -41,6 +41,14 @@ org.finos.legend.engine legend-engine-xt-authentication-protocol + + org.finos.legend.engine + legend-engine-protocol-pure + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanism.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanism.java new file mode 100644 index 00000000000..7e982189eba --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanism.java @@ -0,0 +1,95 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection; + +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.api.list.ImmutableList; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; + +import java.util.ArrayList; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Objects; +import java.util.Set; + +public class AuthenticationMechanism +{ + private final AuthenticationMechanismType authenticationMechanismType; + private final ImmutableList> authenticationConfigurationTypes; + + private AuthenticationMechanism(AuthenticationMechanismType authenticationMechanismType, List> authenticationConfigurationTypes) + { + this.authenticationMechanismType = Objects.requireNonNull(authenticationMechanismType, "Authentication mechanism is missing"); + this.authenticationConfigurationTypes = Lists.immutable.withAll(authenticationConfigurationTypes); + } + + public AuthenticationMechanismType getAuthenticationMechanismType() + { + return authenticationMechanismType; + } + + public ImmutableList> getAuthenticationConfigurationTypes() + { + return authenticationConfigurationTypes; + } + + public static Builder builder() + { + return new Builder(); + } + + public static class Builder + { + private AuthenticationMechanismType authenticationMechanismType; + private final Set> authenticationConfigurationTypes = new LinkedHashSet<>(); + + private Builder() + { + } + + public Builder type(AuthenticationMechanismType authenticationMechanismType) + { + this.authenticationMechanismType = authenticationMechanismType; + return this; + } + + public Builder authenticationConfigurationType(Class authenticationConfigurationType) + { + this.authenticationConfigurationTypes.add(authenticationConfigurationType); + return this; + } + + public Builder authenticationConfigurationTypes(List> authenticationConfigurationTypes) + { + this.authenticationConfigurationTypes.addAll(authenticationConfigurationTypes); + return this; + } + + @SafeVarargs + public final Builder authenticationConfigurationTypes(Class... authenticationConfigurationTypes) + { + this.authenticationConfigurationTypes.addAll(Lists.mutable.of(authenticationConfigurationTypes)); + return this; + } + + public AuthenticationMechanism build() + { + return new AuthenticationMechanism( + this.authenticationMechanismType, + new ArrayList<>(this.authenticationConfigurationTypes) + ); + } + } +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstanceProvider.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismType.java similarity index 88% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstanceProvider.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismType.java index 8b83db33c91..15c061ee6f9 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/StoreInstanceProvider.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/AuthenticationMechanismType.java @@ -14,7 +14,7 @@ package org.finos.legend.connection; -public interface StoreInstanceProvider +public interface AuthenticationMechanismType { - StoreInstance lookup(String identifier); + String getIdentifier(); } diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java similarity index 80% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java index 90e10882b38..96f879da05f 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Authenticator.java @@ -16,8 +16,7 @@ import org.eclipse.collections.api.factory.Lists; import org.eclipse.collections.api.list.ImmutableList; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; @@ -26,8 +25,8 @@ public class Authenticator { - private final StoreInstance storeInstance; - private final AuthenticationMechanism authenticationMechanism; + private final Connection connection; + private final AuthenticationMechanismType authenticationMechanismType; private final AuthenticationConfiguration authenticationConfiguration; private final Class sourceCredentialType; private final Class targetCredentialType; @@ -35,10 +34,10 @@ public class Authenticator private final ConnectionBuilder connectionBuilder; private final LegendEnvironment environment; - public Authenticator(StoreInstance storeInstance, AuthenticationMechanism authenticationMechanism, AuthenticationConfiguration authenticationConfiguration, Class sourceCredentialType, Class targetCredentialType, List credentialBuilders, ConnectionBuilder connectionBuilder, LegendEnvironment environment) + public Authenticator(Connection connection, AuthenticationMechanismType authenticationMechanismType, AuthenticationConfiguration authenticationConfiguration, Class sourceCredentialType, Class targetCredentialType, List credentialBuilders, ConnectionBuilder connectionBuilder, LegendEnvironment environment) { - this.storeInstance = storeInstance; - this.authenticationMechanism = authenticationMechanism; + this.connection = connection; + this.authenticationMechanismType = authenticationMechanismType; this.authenticationConfiguration = authenticationConfiguration; this.sourceCredentialType = sourceCredentialType; this.targetCredentialType = targetCredentialType; @@ -74,9 +73,9 @@ public CRED makeCredential(Identity identity) throws Exception return (CRED) credential; } - public AuthenticationMechanism getAuthenticationMechanism() + public AuthenticationMechanismType getAuthenticationMechanism() { - return authenticationMechanism; + return authenticationMechanismType; } public AuthenticationConfiguration getAuthenticationConfiguration() @@ -84,9 +83,9 @@ public AuthenticationConfiguration getAuthenticationConfiguration() return authenticationConfiguration; } - public StoreInstance getStoreInstance() + public Connection getConnection() { - return storeInstance; + return connection; } public Class getSourceCredentialType() diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Connection.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Connection.java new file mode 100644 index 00000000000..a352b763636 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/Connection.java @@ -0,0 +1,271 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection; + +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.api.factory.Maps; +import org.eclipse.collections.api.list.ImmutableList; +import org.eclipse.collections.api.map.ImmutableMap; +import org.eclipse.collections.impl.utility.ListIterate; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +public final class Connection +{ + private final String identifier; + private final DatabaseSupport databaseSupport; + private final ConnectionSpecification connectionSpecification; + private final AuthenticationConfiguration authenticationConfiguration; + private final ImmutableMap authenticationMechanismsIndex; + private final ImmutableList authenticationMechanismTypes; + private final ImmutableMap, AuthenticationMechanismType> authenticationConfigurationTypesMap; + + private final ImmutableList> authenticationConfigurationTypes; + + private Connection(String identifier, DatabaseSupport databaseSupport, List authenticationMechanisms, ConnectionSpecification connectionSpecification, AuthenticationConfiguration authenticationConfiguration) + { + this.identifier = Objects.requireNonNull(identifier, "Can't create connection: identifier is missing"); + this.databaseSupport = databaseSupport; + this.connectionSpecification = Objects.requireNonNull(connectionSpecification, "Connection specification is missing"); + this.authenticationConfiguration = Objects.requireNonNull(authenticationConfiguration, "Authentication configuration is missing"); + + Map authenticationMechanismsIndex = new LinkedHashMap<>(); + List authenticationMechanismTypes = Lists.mutable.empty(); + if (authenticationMechanisms.isEmpty()) + { + for (AuthenticationMechanismType authenticationMechanismType : this.databaseSupport.getAuthenticationMechanismTypes()) + { + authenticationMechanismsIndex.put(authenticationMechanismType.getIdentifier(), this.databaseSupport.getAuthenticationMechanism(authenticationMechanismType)); + } + authenticationMechanismTypes.addAll(this.databaseSupport.getAuthenticationMechanismTypes().toList()); + } + else + { + for (AuthenticationMechanism authenticationMechanism : authenticationMechanisms) + { + AuthenticationMechanismType authenticationMechanismType = authenticationMechanism.getAuthenticationMechanismType(); + // if no mechanism is specified, it means the connection supports all mechanisms specified in the database support + if (authenticationMechanismsIndex.containsKey(authenticationMechanismType.getIdentifier())) + { + throw new RuntimeException(String.format("Found multiple configurations for authentication mechanism '%s'", authenticationMechanismType.getIdentifier())); + } + AuthenticationMechanism authenticationMechanismsFromDatabaseSupport = this.databaseSupport.getAuthenticationMechanism(authenticationMechanismType); + if (authenticationMechanismsFromDatabaseSupport == null) + { + throw new RuntimeException(String.format("Authentication mechanism '%s' is not covered by database support '%s'. Supported mechanism(s):\n%s", + authenticationMechanismType.getIdentifier(), + this.databaseSupport.getDatabaseType().getIdentifier(), + this.databaseSupport.getAuthenticationMechanismTypes().collect(mechanism -> "- " + mechanism.getIdentifier()).makeString("\n") + )); + } + ImmutableList> authenticationConfigTypesFromDatabaseSupport = authenticationMechanismsFromDatabaseSupport.getAuthenticationConfigurationTypes(); + List> authenticationConfigurationTypes = Lists.mutable.empty(); + for (Class authenticationConfigurationType : authenticationMechanism.getAuthenticationConfigurationTypes()) + { + if (!authenticationConfigTypesFromDatabaseSupport.contains(authenticationConfigurationType)) + { + throw new RuntimeException(String.format("Authentication configuration type '%s' is not covered by database support '%s' for authentication mechanism '%s'. Supported configuration type(s):\n%s", + authenticationConfigurationType.getSimpleName(), + this.databaseSupport.getDatabaseType().getIdentifier(), + authenticationMechanismType.getIdentifier(), + authenticationConfigTypesFromDatabaseSupport.collect(type -> "- " + type.getSimpleName()).makeString("\n") + )); + } + else + { + authenticationConfigurationTypes.add(authenticationConfigurationType); + } + } + authenticationMechanismsIndex.put(authenticationMechanismType.getIdentifier(), AuthenticationMechanism + .builder() + .type(authenticationMechanismType) + // if no configuration type is specified, it means the connection supports all configuration types configured for that mechanism in the database support + .authenticationConfigurationTypes(!authenticationConfigurationTypes.isEmpty() ? authenticationConfigurationTypes : authenticationConfigTypesFromDatabaseSupport.toList()) + .build()); + authenticationMechanismTypes.add(authenticationMechanismType); + } + } + this.authenticationMechanismsIndex = Maps.immutable.withAll(authenticationMechanismsIndex); + this.authenticationMechanismTypes = Lists.immutable.withAll(authenticationMechanismTypes); + + Map, AuthenticationMechanismType> authenticationConfigurationTypesMap = new LinkedHashMap<>(); + List> authenticationConfigurationTypes = Lists.mutable.empty(); + authenticationMechanismsIndex.values().forEach((authenticationMechanism) -> + { + authenticationMechanism.getAuthenticationConfigurationTypes().forEach(configurationType -> + { + authenticationConfigurationTypesMap.put(configurationType, authenticationMechanism.getAuthenticationMechanismType()); + authenticationConfigurationTypes.add(configurationType); + }); + }); + this.authenticationConfigurationTypesMap = Maps.immutable.withAll(authenticationConfigurationTypesMap); + this.authenticationConfigurationTypes = Lists.immutable.withAll(authenticationConfigurationTypes); + + if (!this.authenticationConfigurationTypesMap.containsKey(this.authenticationConfiguration.getClass())) + { + throw new RuntimeException(String.format("Specified authentication configuration of type '%s' is not compatible. Supported configuration type(s):\n%s", + this.authenticationConfiguration.getClass().getSimpleName(), + this.getAuthenticationConfigurationTypes().collect(type -> "- " + type.getSimpleName()).makeString("\n") + )); + } + } + + public String getIdentifier() + { + return identifier; + } + + public DatabaseSupport getDatabaseSupport() + { + return databaseSupport; + } + + public ConnectionSpecification getConnectionSpecification() + { + return connectionSpecification; + } + + public AuthenticationConfiguration getAuthenticationConfiguration() + { + return authenticationConfiguration; + } + + public ImmutableList getAuthenticationMechanisms() + { + return this.authenticationMechanismTypes; + } + + public ImmutableList> getAuthenticationConfigurationTypes() + { + return this.authenticationConfigurationTypes; + } + + public AuthenticationMechanismType getAuthenticationMechanism(Class authenticationConfigurationType) + { + return this.authenticationConfigurationTypesMap.get(authenticationConfigurationType); + } + + public AuthenticationMechanism getAuthenticationMechanism(AuthenticationMechanismType authenticationMechanismType) + { + return authenticationMechanismsIndex.get(authenticationMechanismType.getIdentifier()); + } + + public T getConnectionSpecification(Class clazz) + { + if (!this.connectionSpecification.getClass().equals(clazz)) + { + throw new RuntimeException(String.format("Can't get connection specification of type '%s' for store '%s'", clazz.getSimpleName(), this.identifier)); + } + return (T) this.connectionSpecification; + } + + public static Builder builder() + { + return new Builder(); + } + + public static class Builder + { + private DatabaseSupport databaseSupport; + private String identifier; + private final List authenticationMechanisms = Lists.mutable.empty(); + private ConnectionSpecification connectionSpecification; + private AuthenticationConfiguration authenticationConfiguration; + + private Builder() + { + } + + public Builder databaseSupport(DatabaseSupport databaseSupport) + { + this.databaseSupport = databaseSupport; + return this; + } + + public Builder identifier(String identifier) + { + this.identifier = identifier; + return this; + } + + public Builder authenticationMechanism(AuthenticationMechanism authenticationMechanism) + { + this.authenticationMechanisms.add(authenticationMechanism); + return this; + } + + public Builder authenticationMechanisms(List authenticationMechanisms) + { + this.authenticationMechanisms.addAll(authenticationMechanisms); + return this; + } + + public Builder authenticationMechanisms(AuthenticationMechanism... authenticationMechanisms) + { + this.authenticationMechanisms.addAll(Lists.mutable.of(authenticationMechanisms)); + return this; + } + + public Builder connectionSpecification(ConnectionSpecification connectionSpecification) + { + this.connectionSpecification = connectionSpecification; + return this; + } + + public Builder authenticationConfiguration(AuthenticationConfiguration authenticationConfiguration) + { + this.authenticationConfiguration = authenticationConfiguration; + return this; + } + + public Builder fromProtocol(org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection protocol, LegendEnvironment environment) + { + return this + .databaseSupport(environment.getDatabaseSupport(environment.getDatabaseType(protocol.databaseType))) + .identifier(protocol.getPath()) + .authenticationMechanisms( + protocol.authenticationMechanisms != null + ? ListIterate.collect(protocol.authenticationMechanisms, mechanism -> + AuthenticationMechanism + .builder() + .type(environment.getAuthenticationMechanism(mechanism.authenticationMechanismType)) + .authenticationConfigurationTypes( + ListIterate.collect(mechanism.configurationTypes, environment::getAuthenticationConfigurationType) + ) + .build()) + : Lists.mutable.empty() + ) + .connectionSpecification(protocol.connectionSpecification) + .authenticationConfiguration(protocol.authenticationConfiguration); + } + + + public Connection build() + { + return new Connection( + this.identifier, + this.databaseSupport, + this.authenticationMechanisms, + this.connectionSpecification, + this.authenticationConfiguration + ); + } + } +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java similarity index 97% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java index 49de64a0f9c..344982d55eb 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionBuilder.java @@ -14,7 +14,7 @@ package org.finos.legend.connection; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseType.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionExtension.java similarity index 68% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseType.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionExtension.java index 3982c57e35e..58fb2362ee1 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseType.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionExtension.java @@ -14,23 +14,18 @@ package org.finos.legend.connection; -public enum DatabaseType implements Database -{ - H2("H2"), - POSTGRES("Postgres"), - BIG_QUERY("BigQuery"), - SNOWFLAKE("Snowflake"); - - private final String label; +import java.util.Collections; +import java.util.List; - private DatabaseType(String label) +public interface ConnectionExtension +{ + default List getExtraDatabaseTypes() { - this.label = label; + return Collections.emptyList(); } - @Override - public String getLabel() + default List getExtraAuthenticationMechanismTypes() { - return this.label; + return Collections.emptyList(); } } diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java similarity index 65% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java index 5eca86a7c23..aedb2732661 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionFactory.java @@ -14,12 +14,9 @@ package org.finos.legend.connection; -import org.eclipse.collections.api.block.function.Function0; import org.eclipse.collections.api.factory.Lists; -import org.eclipse.collections.impl.utility.ListIterate; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; @@ -39,14 +36,12 @@ public class ConnectionFactory { private final LegendEnvironment environment; - private final StoreInstanceProvider storeInstanceProvider; private final Map credentialBuildersIndex = new LinkedHashMap<>(); private final Map connectionBuildersIndex = new LinkedHashMap<>(); - private ConnectionFactory(LegendEnvironment environment, StoreInstanceProvider storeInstanceProvider, List credentialBuilders, List connectionBuilders) + private ConnectionFactory(LegendEnvironment environment, List credentialBuilders, List connectionBuilders) { - this.environment = environment; - this.storeInstanceProvider = storeInstanceProvider; + this.environment = Objects.requireNonNull(environment, "environment is missing"); for (ConnectionBuilder builder : connectionBuilders) { this.connectionBuildersIndex.put(new ConnectionBuilder.Key(builder.getConnectionSpecificationType(), builder.getCredentialType()), builder); @@ -62,84 +57,52 @@ public LegendEnvironment getEnvironment() return environment; } - public Authenticator getAuthenticator(Identity identity, String storeInstanceIdentifier, AuthenticationMechanism authenticationMechanism) + public Authenticator getAuthenticator(Identity identity, Connection connection, AuthenticationConfiguration authenticationConfiguration) { - return this.getAuthenticator(identity, this.storeInstanceProvider.lookup(storeInstanceIdentifier), authenticationMechanism); - } - - public Authenticator getAuthenticator(Identity identity, StoreInstance storeInstance, AuthenticationMechanism authenticationMechanism) - { - AuthenticationMechanismConfiguration authenticationMechanismConfiguration = Objects.requireNonNull(storeInstance.getAuthenticationMechanismConfiguration(authenticationMechanism), String.format("Store '%s' does not support authentication mechanism '%s'. Supported mechanism(s):\n%s", - storeInstance.getIdentifier(), - authenticationMechanism.getLabel(), - ListIterate.collect(storeInstance.getAuthenticationMechanisms(), mechanism -> "- " + mechanism.getLabel()).makeString("\n") - )); - Function0 generator = authenticationMechanismConfiguration.getDefaultAuthenticationConfigurationGenerator(); - AuthenticationConfiguration authenticationConfiguration = Objects.requireNonNull(generator != null ? generator.get() : null, String.format("Can't auto-generate authentication configuration for store '%s' with authentication mechanism '%s'. Please provide a configuration of one of the following type(s):\n%s", - storeInstance.getIdentifier(), - authenticationMechanism.getLabel(), - authenticationMechanismConfiguration.getAuthenticationConfigurationTypes().collect(configType -> "- " + configType.getSimpleName()).makeString("\n") - )); - return this.getAuthenticator(identity, storeInstance, authenticationMechanism, authenticationConfiguration); - } - - public Authenticator getAuthenticator(Identity identity, String storeInstanceIdentifier, AuthenticationConfiguration authenticationConfiguration) - { - return this.getAuthenticator(identity, this.storeInstanceProvider.lookup(storeInstanceIdentifier), authenticationConfiguration); - } - - public Authenticator getAuthenticator(Identity identity, StoreInstance storeInstance, AuthenticationConfiguration authenticationConfiguration) - { - AuthenticationMechanism authenticationMechanism = Objects.requireNonNull(storeInstance.getAuthenticationMechanism(authenticationConfiguration.getClass()), String.format("Store '%s' does not accept authentication configuration type '%s'. Supported configuration type(s):\n%s", - storeInstance.getIdentifier(), + AuthenticationMechanismType authenticationMechanismType = Objects.requireNonNull(connection.getAuthenticationMechanism(authenticationConfiguration.getClass()), String.format("Connection '%s' is not compatible with authentication configuration type '%s'. Supported configuration type(s):\n%s", + connection.getIdentifier(), authenticationConfiguration.getClass().getSimpleName(), - ListIterate.collect(storeInstance.getAuthenticationConfigurationTypes(), configType -> "- " + configType.getSimpleName()).makeString("\n") + connection.getAuthenticationConfigurationTypes().collect(configType -> "- " + configType.getSimpleName()).makeString("\n") )); - return this.getAuthenticator(identity, storeInstance, authenticationMechanism, authenticationConfiguration); + return this.getAuthenticator(identity, connection, authenticationMechanismType, authenticationConfiguration); } - private Authenticator getAuthenticator(Identity identity, StoreInstance storeInstance, AuthenticationMechanism authenticationMechanism, AuthenticationConfiguration authenticationConfiguration) + private Authenticator getAuthenticator(Identity identity, Connection connection, AuthenticationMechanismType authenticationMechanismType, AuthenticationConfiguration authenticationConfiguration) { - AuthenticationFlowResolver.ResolutionResult result = AuthenticationFlowResolver.run(this.credentialBuildersIndex, this.connectionBuildersIndex, identity, authenticationMechanism, authenticationConfiguration, storeInstance.getConnectionSpecification()); + AuthenticationFlowResolver.ResolutionResult result = AuthenticationFlowResolver.run(this.credentialBuildersIndex, this.connectionBuildersIndex, identity, authenticationMechanismType, authenticationConfiguration, connection.getConnectionSpecification()); if (result == null) { - throw new RuntimeException(String.format("No authentication flow for store '%s' can be resolved for the specified identity (authentication configuration: %s, connection specification: %s)", - storeInstance.getIdentifier(), + throw new RuntimeException(String.format("No authentication flow for connection '%s' can be resolved for the specified identity (authentication configuration: %s, connection specification: %s)", + connection.getIdentifier(), authenticationConfiguration.getClass().getSimpleName(), - storeInstance.getConnectionSpecification().getClass().getSimpleName() + connection.getConnectionSpecification().getClass().getSimpleName() )); } - return new Authenticator(storeInstance, authenticationMechanism, authenticationConfiguration, result.sourceCredentialType, result.targetCredentialType, result.flow, connectionBuildersIndex.get(new ConnectionBuilder.Key(storeInstance.getConnectionSpecification().getClass(), result.targetCredentialType)), this.environment); + return new Authenticator(connection, authenticationMechanismType, authenticationConfiguration, result.sourceCredentialType, result.targetCredentialType, result.flow, connectionBuildersIndex.get(new ConnectionBuilder.Key(connection.getConnectionSpecification().getClass(), result.targetCredentialType)), this.environment); } - public Authenticator getAuthenticator(Identity identity, String storeInstanceIdentifier) - { - return this.getAuthenticator(identity, this.storeInstanceProvider.lookup(storeInstanceIdentifier)); - } - - public Authenticator getAuthenticator(Identity identity, StoreInstance storeInstance) + public Authenticator getAuthenticator(Identity identity, Connection connection) { Authenticator authenticator = null; - for (AuthenticationMechanism authenticationMechanism : storeInstance.getAuthenticationMechanisms()) + for (AuthenticationMechanismType authenticationMechanismType : connection.getAuthenticationMechanisms()) { - AuthenticationMechanismConfiguration authenticationMechanismConfiguration = storeInstance.getAuthenticationMechanismConfiguration(authenticationMechanism); - Function0 generator = authenticationMechanismConfiguration.getDefaultAuthenticationConfigurationGenerator(); - AuthenticationConfiguration authenticationConfiguration = generator != null ? generator.get() : null; + AuthenticationMechanism authenticationMechanism = connection.getAuthenticationMechanism(authenticationMechanismType); + AuthenticationConfiguration authenticationConfiguration = connection.getAuthenticationConfiguration(); if (authenticationConfiguration != null) { - AuthenticationFlowResolver.ResolutionResult result = AuthenticationFlowResolver.run(this.credentialBuildersIndex, this.connectionBuildersIndex, identity, authenticationMechanism, authenticationConfiguration, storeInstance.getConnectionSpecification()); + AuthenticationFlowResolver.ResolutionResult result = AuthenticationFlowResolver.run(this.credentialBuildersIndex, this.connectionBuildersIndex, identity, authenticationMechanismType, authenticationConfiguration, connection.getConnectionSpecification()); if (result != null) { - authenticator = new Authenticator(storeInstance, authenticationMechanism, authenticationConfiguration, result.sourceCredentialType, result.targetCredentialType, result.flow, connectionBuildersIndex.get(new ConnectionBuilder.Key(storeInstance.getConnectionSpecification().getClass(), result.targetCredentialType)), this.environment); + authenticator = new Authenticator(connection, authenticationMechanismType, authenticationConfiguration, result.sourceCredentialType, result.targetCredentialType, result.flow, connectionBuildersIndex.get(new ConnectionBuilder.Key(connection.getConnectionSpecification().getClass(), result.targetCredentialType)), this.environment); break; } } } if (authenticator == null) { - throw new RuntimeException(String.format("No authentication flow for store '%s' can be resolved for the specified identity. Try specifying an authentication mechanism or authentication configuration. Supported configuration type(s):\n%s", - storeInstance.getIdentifier(), - ListIterate.collect(storeInstance.getAuthenticationConfigurationTypes(), configType -> "- " + configType.getSimpleName() + " (" + storeInstance.getAuthenticationMechanism(configType).getLabel() + ")").makeString("\n") + throw new RuntimeException(String.format("No authentication flow for connection '%s' can be resolved for the specified identity. Try specifying another authentication configuration. Supported configuration type(s):\n%s", + connection.getIdentifier(), + connection.getAuthenticationConfigurationTypes().collect(configType -> "- " + configType.getSimpleName() + " (" + connection.getAuthenticationMechanism(configType).getIdentifier() + ")").makeString("\n") )); } return authenticator; @@ -175,7 +138,7 @@ private static class AuthenticationFlowResolver *

* With this setup, we can use a basic graph search algorithm (e.g. BFS) to resolve the shortest path to build a connection */ - private AuthenticationFlowResolver(Map credentialBuildersIndex, Map connectionBuildersIndex, Identity identity, AuthenticationConfiguration authenticationConfiguration, AuthenticationMechanism authenticationMechanism, ConnectionSpecification connectionSpecification) + private AuthenticationFlowResolver(Map credentialBuildersIndex, Map connectionBuildersIndex, Identity identity, AuthenticationConfiguration authenticationConfiguration, AuthenticationMechanismType authenticationMechanismType, ConnectionSpecification connectionSpecification) { // add start node (i.e. identity node) this.startNode = new FlowNode(identity); @@ -228,10 +191,10 @@ private void processEdge(FlowNode node, FlowNode adjacentNode) /** * Resolves the authentication flow in order to build a connection for a specified identity */ - public static ResolutionResult run(Map credentialBuildersIndex, Map connectionBuildersIndex, Identity identity, AuthenticationMechanism authenticationMechanism, AuthenticationConfiguration authenticationConfiguration, ConnectionSpecification connectionSpecification) + public static ResolutionResult run(Map credentialBuildersIndex, Map connectionBuildersIndex, Identity identity, AuthenticationMechanismType authenticationMechanismType, AuthenticationConfiguration authenticationConfiguration, ConnectionSpecification connectionSpecification) { // using BFS algo to search for the shortest (non-cyclic) path - AuthenticationFlowResolver state = new AuthenticationFlowResolver(credentialBuildersIndex, connectionBuildersIndex, identity, authenticationConfiguration, authenticationMechanism, connectionSpecification); + AuthenticationFlowResolver state = new AuthenticationFlowResolver(credentialBuildersIndex, connectionBuildersIndex, identity, authenticationConfiguration, authenticationMechanismType, connectionSpecification); boolean found = false; Set visitedNodes = new HashSet<>(); // Create a set to keep track of visited vertices @@ -371,76 +334,74 @@ public ResolutionResult(List flow, Class T getConnection(Identity identity, StoreInstance storeInstance, AuthenticationConfiguration authenticationConfiguration) throws Exception - { - return this.getConnection(identity, this.getAuthenticator(identity, storeInstance, authenticationConfiguration)); - } - - public T getConnection(Identity identity, String storeInstanceIdentifier, AuthenticationConfiguration authenticationConfiguration) throws Exception + public T getConnection(Identity identity, Connection connection, AuthenticationConfiguration authenticationConfiguration) throws Exception { - return this.getConnection(identity, this.getAuthenticator(identity, storeInstanceIdentifier, authenticationConfiguration)); + return this.getConnection(identity, this.getAuthenticator(identity, connection, authenticationConfiguration)); } - public T getConnection(Identity identity, StoreInstance storeInstance) throws Exception + public T getConnection(Identity identity, Connection connection) throws Exception { - return this.getConnection(identity, this.getAuthenticator(identity, storeInstance)); + return this.getConnection(identity, this.getAuthenticator(identity, connection)); } - public T getConnection(Identity identity, String storeInstanceIdentifier) throws Exception + public T getConnection(Identity identity, Authenticator authenticator) throws Exception { - return this.getConnection(identity, this.getAuthenticator(identity, storeInstanceIdentifier)); + ConnectionBuilder flow = (ConnectionBuilder) authenticator.getConnectionBuilder(); + return flow.getConnection(authenticator.getConnection().getConnectionSpecification(), flow.getAuthenticatorCompatible(authenticator), identity); } - public T getConnection(Identity identity, Authenticator authenticator) throws Exception + public static Builder builder() { - ConnectionBuilder flow = (ConnectionBuilder) authenticator.getConnectionBuilder(); - return flow.getConnection(authenticator.getStoreInstance().getConnectionSpecification(), flow.getAuthenticatorCompatible(authenticator), identity); + return new Builder(); } public static class Builder { - private final LegendEnvironment environment; - private final StoreInstanceProvider storeInstanceProvider; + private LegendEnvironment environment; private final List credentialBuilders = Lists.mutable.empty(); private final List connectionBuilders = Lists.mutable.empty(); - public Builder(LegendEnvironment environment, StoreInstanceProvider storeInstanceProvider) + private Builder() + { + } + + public Builder environment(LegendEnvironment environment) { this.environment = environment; - this.storeInstanceProvider = storeInstanceProvider; + return this; } - public Builder withCredentialBuilders(List credentialBuilders) + public Builder credentialBuilders(List credentialBuilders) { this.credentialBuilders.addAll(credentialBuilders); return this; } - public Builder withCredentialBuilders(CredentialBuilder... credentialBuilders) + public Builder credentialBuilders(CredentialBuilder... credentialBuilders) { this.credentialBuilders.addAll(Lists.mutable.with(credentialBuilders)); return this; } - public Builder withCredentialBuilder(CredentialBuilder credentialBuilder) + public Builder credentialBuilder(CredentialBuilder credentialBuilder) { this.credentialBuilders.add(credentialBuilder); return this; } - public Builder withConnectionBuilders(List connectionBuilders) + public Builder connectionBuilders(List connectionBuilders) { this.connectionBuilders.addAll(connectionBuilders); return this; } - public Builder withConnectionBuilders(ConnectionBuilder... connectionBuilders) + public Builder connectionBuilders(ConnectionBuilder... connectionBuilders) { this.connectionBuilders.addAll(Lists.mutable.with(connectionBuilders)); return this; } - public Builder withConnectionBuilder(ConnectionBuilder connectionBuilder) + public Builder connectionBuilder(ConnectionBuilder connectionBuilder) { this.connectionBuilders.add(connectionBuilder); return this; @@ -459,7 +420,6 @@ public ConnectionFactory build() return new ConnectionFactory( this.environment, - this.storeInstanceProvider, this.credentialBuilders, this.connectionBuilders ); diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionManager.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionManager.java similarity index 100% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/ConnectionManager.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/ConnectionManager.java diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java similarity index 96% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java index 0f4726abfd7..404074863a1 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/CredentialBuilder.java @@ -14,7 +14,7 @@ package org.finos.legend.connection; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseSupport.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseSupport.java new file mode 100644 index 00000000000..7649cb9aedc --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseSupport.java @@ -0,0 +1,165 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection; + +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.api.factory.Maps; +import org.eclipse.collections.api.list.ImmutableList; +import org.eclipse.collections.api.map.ImmutableMap; +import org.eclipse.collections.impl.utility.ListIterate; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * A DatabaseSupport describes the capabilities supported by a database. + * For now, it describes the authentication mechanisms. + */ +public final class DatabaseSupport +{ + private final DatabaseType databaseType; + private final ImmutableMap authenticationMechanismsIndex; + private final ImmutableList authenticationMechanismTypes; + + private DatabaseSupport(DatabaseType databaseType, List authenticationMechanisms) + { + this.databaseType = Objects.requireNonNull(databaseType, "Database type is missing"); + + Map authenticationMechanismsIndex = new LinkedHashMap<>(); + List authenticationMechanismTypes = Lists.mutable.empty(); + Map, AuthenticationMechanismType> authenticationConfigurationTypeIndex = new LinkedHashMap<>(); + for (AuthenticationMechanism authenticationMechanism : authenticationMechanisms) + { + AuthenticationMechanismType authenticationMechanismType = authenticationMechanism.getAuthenticationMechanismType(); + if (authenticationMechanismsIndex.containsKey(authenticationMechanismType.getIdentifier())) + { + throw new RuntimeException(String.format("Found multiple authentication mechanisms with type '%s'", authenticationMechanismType.getIdentifier())); + } + authenticationMechanismsIndex.put(authenticationMechanismType.getIdentifier(), authenticationMechanism); + authenticationMechanism.getAuthenticationConfigurationTypes().forEach(authenticationConfigurationType -> + { + if (authenticationConfigurationTypeIndex.containsKey(authenticationConfigurationType)) + { + throw new RuntimeException(String.format("Authentication configuration type '%s' is associated with multiple authentication mechanisms", authenticationConfigurationType.getSimpleName())); + } + authenticationConfigurationTypeIndex.put(authenticationConfigurationType, authenticationMechanismType); + }); + authenticationMechanismTypes.add(authenticationMechanism.getAuthenticationMechanismType()); + } + + this.authenticationMechanismTypes = Lists.immutable.withAll(authenticationMechanismTypes); + this.authenticationMechanismsIndex = Maps.immutable.withAll(authenticationMechanismsIndex); + + authenticationMechanisms.forEach((authenticationMechanism) -> + { + if (authenticationMechanism.getAuthenticationConfigurationTypes().isEmpty()) + { + throw new RuntimeException(String.format("No authentication configuration type is associated with authentication mechanism '%s'", authenticationMechanism.getAuthenticationMechanismType().getIdentifier())); + } + }); + } + + public DatabaseType getDatabaseType() + { + return this.databaseType; + } + + public AuthenticationMechanism getAuthenticationMechanism(AuthenticationMechanismType authenticationMechanismType) + { + return authenticationMechanismsIndex.get(authenticationMechanismType.getIdentifier()); + } + + public ImmutableList getAuthenticationMechanismTypes() + { + return this.authenticationMechanismTypes; + } + + public static void verifyDatabaseType(DatabaseSupport databaseSupport, DatabaseType databaseType) + { + if (!databaseType.equals(databaseSupport.getDatabaseType())) + { + + throw new RuntimeException(String.format("Expected database type '%s'", databaseType.getIdentifier())); + } + } + + public static Builder builder() + { + return new Builder(); + } + + public static class Builder + { + private DatabaseType databaseType; + private final List authenticationMechanisms = Lists.mutable.empty(); + + private Builder() + { + } + + public Builder type(DatabaseType databaseType) + { + this.databaseType = databaseType; + return this; + } + + public Builder authenticationMechanism(AuthenticationMechanism authenticationMechanism) + { + this.authenticationMechanisms.add(authenticationMechanism); + return this; + } + + public Builder authenticationMechanisms(List authenticationMechanisms) + { + this.authenticationMechanisms.addAll(authenticationMechanisms); + return this; + } + + public Builder authenticationMechanisms(AuthenticationMechanism... authenticationMechanisms) + { + this.authenticationMechanisms.addAll(Lists.mutable.of(authenticationMechanisms)); + return this; + } + + public Builder fromProtocol(org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.DatabaseSupport databaseSupport, LegendEnvironment environment) + { + return this + .type(environment.getDatabaseType(databaseSupport.databaseType)) + .authenticationMechanisms( + databaseSupport.authenticationMechanisms != null + ? ListIterate.collect(databaseSupport.authenticationMechanisms, mechanism -> + AuthenticationMechanism + .builder() + .type(environment.getAuthenticationMechanism(mechanism.authenticationMechanismType)) + .authenticationConfigurationTypes( + ListIterate.collect(mechanism.configurationTypes, environment::getAuthenticationConfigurationType) + ) + .build()) + : Lists.mutable.empty() + ); + } + + public DatabaseSupport build() + { + return new DatabaseSupport( + this.databaseType, + this.authenticationMechanisms + ); + } + } +} diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/Database.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseType.java similarity index 91% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/Database.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseType.java index e999c689c69..19292292433 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/Database.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/DatabaseType.java @@ -14,7 +14,7 @@ package org.finos.legend.connection; -public interface Database +public interface DatabaseType { - String getLabel(); + String getIdentifier(); } diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java similarity index 80% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java index a02600a70a6..71065b9e4fc 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentityFactory.java @@ -21,6 +21,7 @@ import org.finos.legend.engine.shared.core.identity.factory.DefaultIdentityFactory; import java.util.List; +import java.util.Objects; public class IdentityFactory { @@ -28,7 +29,7 @@ public class IdentityFactory private IdentityFactory(LegendEnvironment environment) { - this.environment = environment; + this.environment = Objects.requireNonNull(environment, "environment is missing"); } // TODO: @akphi - this clones the logic from IdentityFactoryProvider, we should think about unifying them @@ -41,11 +42,11 @@ public Identity createIdentity(IdentitySpecification identitySpecification) // TODO: @akphi - should we restrict here that we can only either specify the subject/profiles? if (identitySpecification.getSubject() != null) { - credentials.addAll(DEFAULT.makeIdentity(identitySpecification.getSubject()).getCredentials().toList()); + return DEFAULT.makeIdentity(identitySpecification.getSubject()); } if (!identitySpecification.getProfiles().isEmpty()) { - credentials.addAll(DEFAULT.makeIdentity(Lists.mutable.withAll(identitySpecification.getProfiles())).getCredentials().toList()); + return DEFAULT.makeIdentity(Lists.mutable.withAll(identitySpecification.getProfiles())); } if (credentials.isEmpty()) { @@ -54,13 +55,23 @@ public Identity createIdentity(IdentitySpecification identitySpecification) return new Identity(identitySpecification.getName(), credentials); } + public static Builder builder() + { + return new Builder(); + } + public static class Builder { - private final LegendEnvironment environment; + private LegendEnvironment environment; + + private Builder() + { + } - public Builder(LegendEnvironment environment) + public Builder environment(LegendEnvironment environment) { this.environment = environment; + return this; } public IdentityFactory build() diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java similarity index 85% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java index 7a07c2f6989..50e60a670e8 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/IdentitySpecification.java @@ -20,7 +20,6 @@ import javax.security.auth.Subject; import java.util.List; -import java.util.Objects; public class IdentitySpecification { @@ -57,6 +56,11 @@ public List getCredentials() return credentials; } + public static Builder builder() + { + return new Builder(); + } + public static class Builder { private String name; @@ -64,37 +68,41 @@ public static class Builder private Subject subject; private final List credentials = Lists.mutable.empty(); - public Builder withName(String name) + private Builder() + { + } + + public Builder name(String name) { this.name = name; return this; } - public Builder withProfiles(List profiles) + public Builder profiles(List profiles) { this.profiles.addAll(profiles); return this; } - public Builder withProfile(CommonProfile profile) + public Builder profile(CommonProfile profile) { this.profiles.add(profile); return this; } - public Builder withSubject(Subject subject) + public Builder subject(Subject subject) { this.subject = subject; return this; } - public Builder withCredentials(List credentials) + public Builder credentials(List credentials) { this.credentials.addAll(credentials); return this; } - public Builder withCredential(Credential credential) + public Builder credentials(Credential credential) { this.credentials.add(credential); return this; diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java new file mode 100644 index 00000000000..6a9d44531d0 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/LegendEnvironment.java @@ -0,0 +1,209 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection; + +import org.eclipse.collections.api.block.function.Function0; +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.api.list.ImmutableList; +import org.eclipse.collections.api.list.MutableList; +import org.eclipse.collections.api.map.ImmutableMap; +import org.eclipse.collections.impl.factory.Maps; +import org.eclipse.collections.impl.utility.LazyIterate; +import org.finos.legend.authentication.vault.CredentialVault; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; +import org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtensionLoader; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.CredentialVaultSecret; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.shared.core.identity.Identity; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.ServiceLoader; + +/** + * This is the runtime instance of configuration for Legend Engine, the place we package common configs, + * such as vaults, that can be passed to various parts of engine, authentication, connection factory, etc. + */ +public class LegendEnvironment +{ + protected final ImmutableList vaults; + protected final ImmutableList connectionExtensions; + protected final ImmutableMap authenticationMechanismsTypesIndex; + protected final ImmutableMap databaseTypesIndex; + protected final ImmutableMap> authenticationConfigurationTypesIndex; + protected final ImmutableMap, CredentialVault> vaultsIndex; + protected final ImmutableMap databaseSupportsIndex; + + protected LegendEnvironment(List vaults, List databaseSupports) + { + this.vaults = Lists.immutable.withAll(vaults); + Map, CredentialVault> vaultsIndex = new LinkedHashMap<>(); + for (CredentialVault vault : vaults) + { + vaultsIndex.put(vault.getSecretType(), vault); + } + this.vaultsIndex = Maps.immutable.withAll(vaultsIndex); + + MutableList connectionExtensions = Lists.mutable.withAll(ServiceLoader.load(ConnectionExtension.class)); + this.connectionExtensions = connectionExtensions.toImmutable(); + + // load authentication mechanism types + List authenticationMechanismTypes = Lists.mutable.of(CoreAuthenticationMechanismType.values()); + authenticationMechanismTypes.addAll(connectionExtensions.flatCollect(ConnectionExtension::getExtraAuthenticationMechanismTypes)); + Map authenticationMechanismsTypesIndex = new LinkedHashMap<>(); + authenticationMechanismTypes.forEach(authenticationMechanism -> + { + if (authenticationMechanismsTypesIndex.containsKey(authenticationMechanism.getIdentifier())) + { + throw new RuntimeException(String.format("Found multiple authentication mechanisms with label '%s'", authenticationMechanism.getIdentifier())); + } + authenticationMechanismsTypesIndex.put(authenticationMechanism.getIdentifier(), authenticationMechanism); + }); + this.authenticationMechanismsTypesIndex = Maps.immutable.withAll(authenticationMechanismsTypesIndex); + + // load database types + List databaseTypes = connectionExtensions.flatCollect(ConnectionExtension::getExtraDatabaseTypes); + Map databaseTypesIndex = new LinkedHashMap<>(); + databaseTypes.forEach(databaseType -> + { + if (databaseTypesIndex.containsKey(databaseType.getIdentifier())) + { + throw new RuntimeException(String.format("Found multiple authentication mechanisms with label '%s'", databaseType.getIdentifier())); + } + databaseTypesIndex.put(databaseType.getIdentifier(), databaseType); + }); + this.databaseTypesIndex = Maps.immutable.withAll(databaseTypesIndex); + + // load authentication configuration types + Map> authenticationConfigurationTypesIndex = new LinkedHashMap<>(); + PureProtocolExtensionLoader.extensions().forEach(extension -> + LazyIterate.flatCollect(extension.getExtraProtocolSubTypeInfoCollectors(), Function0::value).forEach(info -> + { + info.getSubTypes().forEach(subType -> + { + if (AuthenticationConfiguration.class.isAssignableFrom(subType.getOne())) + { + if (authenticationConfigurationTypesIndex.containsKey(subType.getTwo())) + { + throw new RuntimeException(String.format("Found multiple authentication configuration types with identifier '%s'", subType.getTwo())); + } + authenticationConfigurationTypesIndex.put(subType.getTwo(), (Class) subType.getOne()); + } + }); + })); + this.authenticationConfigurationTypesIndex = Maps.immutable.withAll(authenticationConfigurationTypesIndex); + + // load database supports + Map databaseSupportsIndex = new LinkedHashMap<>(); + databaseSupports.forEach(databaseSupport -> + { + if (databaseSupportsIndex.containsKey(databaseSupport.getDatabaseType().getIdentifier())) + { + throw new RuntimeException(String.format("Found multiple database supports for type '%s'", databaseSupport.getDatabaseType().getIdentifier())); + } + databaseSupportsIndex.put(databaseSupport.getDatabaseType().getIdentifier(), databaseSupport); + }); + this.databaseSupportsIndex = Maps.immutable.withAll(databaseSupportsIndex); + } + + public String lookupVaultSecret(CredentialVaultSecret credentialVaultSecret, Identity identity) throws Exception + { + Class secretClass = credentialVaultSecret.getClass(); + if (!this.vaultsIndex.containsKey(secretClass)) + { + throw new RuntimeException(String.format("Can't find secret: credential vault for secret of type '%s' has not been registered", secretClass.getSimpleName())); + } + CredentialVault vault = this.vaultsIndex.get(secretClass); + return vault.lookupSecret(credentialVaultSecret, identity); + } + + public AuthenticationMechanismType getAuthenticationMechanism(String identifier) + { + return Objects.requireNonNull(this.authenticationMechanismsTypesIndex.get(identifier), String.format("Can't find authentication mechanism with label '%s'", identifier)); + } + + public DatabaseType getDatabaseType(String identifier) + { + return Objects.requireNonNull(this.databaseTypesIndex.get(identifier), String.format("Can't find database type with identifier '%s'", identifier)); + } + + public Class getAuthenticationConfigurationType(String identifier) + { + return Objects.requireNonNull(this.authenticationConfigurationTypesIndex.get(identifier), String.format("Can't find authentication configuration type with identifier '%s'", identifier)); + } + + public DatabaseSupport getDatabaseSupport(DatabaseType databaseType) + { + return Objects.requireNonNull(this.databaseSupportsIndex.get(databaseType.getIdentifier()), String.format("Can't find database support with database type '%s'", databaseType.getIdentifier())); + } + + public static Builder builder() + { + return new Builder(); + } + + public static class Builder + { + private final List vaults = Lists.mutable.empty(); + private final List databaseSupports = Lists.mutable.empty(); + + private Builder() + { + } + + public Builder vaults(List vaults) + { + this.vaults.addAll(vaults); + return this; + } + + public Builder vaults(CredentialVault... vaults) + { + this.vaults.addAll(Lists.mutable.with(vaults)); + return this; + } + + public Builder vault(CredentialVault vault) + { + this.vaults.add(vault); + return this; + } + + public Builder databaseSupports(List databaseSupports) + { + this.databaseSupports.addAll(databaseSupports); + return this; + } + + public Builder databaseSupports(DatabaseSupport... databaseSupports) + { + this.databaseSupports.addAll(Lists.mutable.with(databaseSupports)); + return this; + } + + public Builder databaseSupport(DatabaseSupport databaseSupport) + { + this.databaseSupports.add(databaseSupport); + return this; + } + + public LegendEnvironment build() + { + return new LegendEnvironment(this.vaults, this.databaseSupports); + } + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/CoreAuthenticationMechanismType.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/CoreAuthenticationMechanismType.java new file mode 100644 index 00000000000..860b89cf708 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/CoreAuthenticationMechanismType.java @@ -0,0 +1,39 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection.impl; + +import org.finos.legend.connection.AuthenticationMechanismType; + +public enum CoreAuthenticationMechanismType implements AuthenticationMechanismType +{ + USER_PASSWORD("UsernamePassword"), + API_KEY("APIKey"), + KEY_PAIR("KeyPair"), + KERBEROS("Kerberos"), + OAUTH("OAuth"); + + private final String identifier; + + private CoreAuthenticationMechanismType(String identifier) + { + this.identifier = identifier; + } + + @Override + public String getIdentifier() + { + return this.identifier; + } +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java similarity index 93% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java index a36c3add1cc..1dd67d45e49 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosCredentialExtractor.java @@ -16,6 +16,7 @@ import org.finos.legend.connection.CredentialBuilder; import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.KerberosAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; import org.finos.legend.engine.shared.core.identity.credential.LegendKerberosCredential; diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java similarity index 97% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java index 46cc4ddc880..29c084f136c 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/KeyPairCredentialBuilder.java @@ -26,6 +26,7 @@ import org.eclipse.collections.impl.factory.Strings; import org.finos.legend.connection.CredentialBuilder; import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import org.finos.legend.engine.shared.core.identity.credential.PrivateKeyCredential; diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java similarity index 92% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java index 7cec18bd446..98fa2b6d675 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordCredentialBuilder.java @@ -16,6 +16,7 @@ import org.finos.legend.connection.CredentialBuilder; import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import org.finos.legend.engine.shared.core.identity.credential.PlaintextUserPasswordCredential; diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java similarity index 68% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java rename to legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java index df1398edd7c..7c935f6cdc5 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/ConnectionFactoryTest.java @@ -15,10 +15,8 @@ package org.finos.legend.connection; import org.eclipse.collections.api.factory.Lists; -import org.finos.legend.connection.impl.InstrumentedStoreInstanceProvider; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanism; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import org.junit.jupiter.api.Assertions; @@ -40,61 +38,39 @@ public void testGetConnection_WithFailures() throws Exception new ConnectionBuilder_A() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build(), - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.Y) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_Y.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.Y) + .authenticationConfigurationTypes(AuthenticationConfiguration_Y.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test"); // success - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X())); + env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X()))); Exception exception; - // error: store not found - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "unknown", new AuthenticationConfiguration_X())); - }); - Assertions.assertEquals("Can't find store instance with identifier 'unknown'", exception.getMessage()); - - // error: unsupported authentication mechanism - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", TestAuthenticationMechanismType.Z)); - }); - Assertions.assertEquals("Store 'test' does not support authentication mechanism 'Z'. Supported mechanism(s):\n" + - "- X\n" + - "- Y", exception.getMessage()); - - // error: authentication mechanism does not come with a default config generator - exception = Assertions.assertThrows(RuntimeException.class, () -> - { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", TestAuthenticationMechanismType.X)); - }); - Assertions.assertEquals("Can't auto-generate authentication configuration for store 'test' with authentication mechanism 'X'. Please provide a configuration of one of the following type(s):\n" + - "- AuthenticationConfiguration_X", exception.getMessage()); - // error: unsupported authentication configuration exception = Assertions.assertThrows(RuntimeException.class, () -> { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_Z())); + env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X()), new AuthenticationConfiguration_Z())); }); - Assertions.assertEquals("Store 'test' does not accept authentication configuration type 'AuthenticationConfiguration_Z'. Supported configuration type(s):\n" + + Assertions.assertEquals("Connection 'test::connectionX' is not compatible with authentication configuration type 'AuthenticationConfiguration_Z'. Supported configuration type(s):\n" + "- AuthenticationConfiguration_X\n" + "- AuthenticationConfiguration_Y", exception.getMessage()); // error: unresolvable authentication flow exception = Assertions.assertThrows(RuntimeException.class, () -> { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_Y())); + env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X()), new AuthenticationConfiguration_Y())); }); - Assertions.assertEquals("No authentication flow for store 'test' can be resolved for the specified identity (authentication configuration: AuthenticationConfiguration_Y, connection specification: TestConnectionSpecification)", exception.getMessage()); + Assertions.assertEquals("No authentication flow for connection 'test::connectionX' can be resolved for the specified identity (authentication configuration: AuthenticationConfiguration_Y, connection specification: TestConnectionSpecification)", exception.getMessage()); } /** @@ -113,17 +89,19 @@ public void testGetConnection_WithSimpleFlow() throws Exception new ConnectionBuilder_B() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build(), - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.Y) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_Y.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.Y) + .authenticationConfigurationTypes(AuthenticationConfiguration_Y.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test"); - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential.class, Lists.mutable.with( "Credential->Credential_A [AuthenticationConfiguration_X]" ), ConnectionBuilder_A.class); @@ -146,14 +124,15 @@ public void testGetConnection_WithSpecificBuilderOrder() throws Exception new ConnectionBuilder_B() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test"); - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential.class, Lists.mutable.with( "Credential->Credential_B [AuthenticationConfiguration_X]" ), ConnectionBuilder_B.class); @@ -175,14 +154,15 @@ public void testGetConnection_WithChainFlow() throws Exception new ConnectionBuilder_C() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test"); - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential.class, Lists.mutable.with( "Credential->Credential_A [AuthenticationConfiguration_X]", "Credential_A->Credential_B [AuthenticationConfiguration_X]", @@ -206,14 +186,15 @@ public void testGetConnection_WithShortestFlowResolved() throws Exception new ConnectionBuilder_C() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test", new Credential_B()); - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential_B.class, Lists.mutable.with( "Credential_B->Credential_C [AuthenticationConfiguration_X]" ), ConnectionBuilder_C.class); @@ -223,7 +204,7 @@ public void testGetConnection_WithShortestFlowResolved() throws Exception * Test Case: A -> B -> [Connection] */ @Test - public void testGetConnection_WithNoAuthConfigProvided() throws Exception + public void testGetConnection_WithCustomAuthConfigProvided() throws Exception { TestEnv env = TestEnv.create( Lists.mutable.with( @@ -234,23 +215,25 @@ public void testGetConnection_WithNoAuthConfigProvided() throws Exception new ConnectionBuilder_B() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build(), - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.Y) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_Y.class) - .withDefaultAuthenticationConfigurationGenerator(AuthenticationConfiguration_Y::new) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.Y) + .authenticationConfigurationTypes(AuthenticationConfiguration_Y.class) .build(), - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.Z) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_Z.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.Z) + .authenticationConfigurationTypes(AuthenticationConfiguration_Z.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test", new Credential_A()); // success - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test"); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X()), new AuthenticationConfiguration_Y()); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential_A.class, Lists.mutable.with( "Credential_A->Credential_B [AuthenticationConfiguration_Y]" ), ConnectionBuilder_B.class); @@ -258,9 +241,9 @@ public void testGetConnection_WithNoAuthConfigProvided() throws Exception // error: unresolvable authentication flow Exception exception = Assertions.assertThrows(RuntimeException.class, () -> { - env.connectionFactory.getAuthenticator(new Identity("test"), "test"); + env.connectionFactory.getAuthenticator(new Identity("test"), env.newStore("test::connectionX", new AuthenticationConfiguration_X())); }); - Assertions.assertEquals("No authentication flow for store 'test' can be resolved for the specified identity. Try specifying an authentication mechanism or authentication configuration. Supported configuration type(s):\n" + + Assertions.assertEquals("No authentication flow for connection 'test::connectionX' can be resolved for the specified identity. Try specifying another authentication configuration. Supported configuration type(s):\n" + "- AuthenticationConfiguration_X (X)\n" + "- AuthenticationConfiguration_Y (Y)\n" + "- AuthenticationConfiguration_Z (Z)", exception.getMessage()); @@ -280,14 +263,15 @@ public void testGetConnection_WithCredentialExtractor() throws Exception new ConnectionBuilder_A() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test", new Credential_A()); - Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + Authenticator authenticator = env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env.connectionFactory, authenticator, Credential_A.class, Lists.mutable.with( "Credential_A->Credential_A [AuthenticationConfiguration_X]" ), ConnectionBuilder_A.class); @@ -301,13 +285,14 @@ public void testGetConnection_WithCredentialExtractor() throws Exception new ConnectionBuilder_A() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); - authenticator = env2.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X()); + authenticator = env2.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X())); assertAuthenticator(identity, env2.connectionFactory, authenticator, Credential_A.class, Lists.mutable.with( "Credential_A->Credential_A [AuthenticationConfiguration_X]" ), ConnectionBuilder_A.class); @@ -328,18 +313,20 @@ public void testGetConnection_WithoutCredentialExtractor() throws Exception new ConnectionBuilder_A() ), Lists.mutable.with( - new AuthenticationMechanismConfiguration.Builder(TestAuthenticationMechanismType.X) - .withAuthenticationConfigurationTypes(AuthenticationConfiguration_X.class) + AuthenticationMechanism.builder() + .type(TestAuthenticationMechanismType.X) + .authenticationConfigurationTypes(AuthenticationConfiguration_X.class) .build() ) - ).newStore("test", Lists.mutable.empty()); + ); Identity identity = new Identity("test", new Credential_A()); Exception exception = Assertions.assertThrows(RuntimeException.class, () -> { - env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, "test", new AuthenticationConfiguration_X())); + env.connectionFactory.getConnection(identity, env.connectionFactory.getAuthenticator(identity, env.newStore("test::connectionX", new AuthenticationConfiguration_X()))); }); - Assertions.assertEquals("No authentication flow for store 'test' can be resolved for the specified identity (authentication configuration: AuthenticationConfiguration_X, connection specification: TestConnectionSpecification)", exception.getMessage()); + Assertions.assertEquals("No authentication flow for connection 'test::connectionX' can be resolved for the specified identity. Try specifying another authentication configuration. Supported configuration type(s):\n" + + "- AuthenticationConfiguration_X (X)", exception.getMessage()); } private void assertAuthenticator(Identity identity, ConnectionFactory connectionFactory, Authenticator authenticator, Class sourceCredentialType, List credentialBuilders, Class connectionBuilderType) throws Exception @@ -353,33 +340,38 @@ private void assertAuthenticator(Identity identity, ConnectionFactory connection private static class TestEnv { final LegendEnvironment environment; - final InstrumentedStoreInstanceProvider storeInstanceProvider; final ConnectionFactory connectionFactory; - private TestEnv(List credentialBuilders, List connectionBuilders, List authenticationMechanismConfigurations) + private TestEnv(List credentialBuilders, List connectionBuilders, List authenticationMechanisms) { - this.environment = new LegendEnvironment.Builder() - .withStoreSupport(new StoreSupport.Builder() - .withIdentifier("test") - .withAuthenticationMechanismConfigurations(authenticationMechanismConfigurations) + this.environment = LegendEnvironment.builder() + .databaseSupport(DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .authenticationMechanisms(authenticationMechanisms) .build()) .build(); - this.storeInstanceProvider = new InstrumentedStoreInstanceProvider(); - this.connectionFactory = new ConnectionFactory.Builder(this.environment, this.storeInstanceProvider) - .withCredentialBuilders(credentialBuilders) - .withConnectionBuilders(connectionBuilders) + this.connectionFactory = ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders(credentialBuilders) + .connectionBuilders(connectionBuilders) + .build(); + } + + Connection newStore(String identifier, AuthenticationConfiguration authenticationConfiguration, List authenticationMechanisms) + { + DatabaseSupport databaseSupport = this.environment.getDatabaseSupport(TestDatabaseType.TEST); + return Connection.builder() + .databaseSupport(databaseSupport) + .identifier(identifier) + .authenticationMechanisms(authenticationMechanisms) + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(authenticationConfiguration) .build(); } - TestEnv newStore(String identifier, List authenticationMechanismConfigurations) + Connection newStore(String identifier, AuthenticationConfiguration authenticationConfiguration) { - this.storeInstanceProvider.injectStoreInstance(new StoreInstance.Builder(this.environment) - .withIdentifier(identifier) - .withStoreSupportIdentifier("test") - .withAuthenticationMechanismConfigurations(authenticationMechanismConfigurations) - .withConnectionSpecification(new TestConnectionSpecification()) - .build()); - return this; + return newStore(identifier, authenticationConfiguration, Lists.mutable.empty()); } static TestEnv create() @@ -387,12 +379,26 @@ static TestEnv create() return new TestEnv(Lists.mutable.empty(), Lists.mutable.empty(), Lists.mutable.empty()); } - static TestEnv create(List credentialBuilders, List connectionBuilders, List authenticationMechanismConfigurations) + static TestEnv create(List credentialBuilders, List connectionBuilders, List authenticationMechanisms) { - return new TestEnv(credentialBuilders, connectionBuilders, authenticationMechanismConfigurations); + return new TestEnv(credentialBuilders, connectionBuilders, authenticationMechanisms); } } + // -------------------------- Database Type ------------------------------- + + private enum TestDatabaseType implements DatabaseType + { + TEST() + { + @Override + public String getIdentifier() + { + return "Test"; + } + } + } + // -------------------------- Credential ------------------------------- private static class Credential_A implements Credential @@ -436,14 +442,14 @@ public String shortId() } } - private enum TestAuthenticationMechanismType implements AuthenticationMechanism + private enum TestAuthenticationMechanismType implements AuthenticationMechanismType { X, Y, Z; @Override - public String getLabel() + public String getIdentifier() { return this.toString(); } diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/DatabaseSupportTest.java b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/DatabaseSupportTest.java new file mode 100644 index 00000000000..42c8f979abb --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-factory/src/test/java/org/finos/legend/connection/DatabaseSupportTest.java @@ -0,0 +1,273 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection; + +import org.eclipse.collections.api.factory.Lists; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.PropertiesFileSecret; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ApiKeyAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.KerberosAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +public class DatabaseSupportTest +{ + @Test + public void testValidateDatabaseSupportBuilder() + { + // success + DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .build(); + + Exception exception; + + // error: missing database type + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + DatabaseSupport.builder().build(); + }); + Assertions.assertEquals("Database type is missing", exception.getMessage()); + + // error: multiple authentication configurations for one authentication mechanism + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) + .build(), + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) + .build() + ).build(); + }); + Assertions.assertEquals("Found multiple authentication mechanisms with type 'UsernamePassword'", exception.getMessage()); + + // error: one authentication configuration is associated with multiple authentication mechanisms + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) + .build(), + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.KERBEROS) + .authenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class) + .build() + ).build(); + }); + Assertions.assertEquals("Authentication configuration type 'UserPasswordAuthenticationConfiguration' is associated with multiple authentication mechanisms", exception.getMessage()); + + // error: no authentication configurations is associated with an authentication mechanism + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .build() + ).build(); + }); + Assertions.assertEquals("No authentication configuration type is associated with authentication mechanism 'UsernamePassword'", exception.getMessage()); + } + + @Test + public void testValidateConnectionBuilder() + { + DatabaseSupport databaseSupport = DatabaseSupport.builder() + .type(TestDatabaseType.TEST) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class, + EncryptedPrivateKeyPairAuthenticationConfiguration.class + ) + .build(), + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.KERBEROS) + .authenticationConfigurationTypes(KerberosAuthenticationConfiguration.class) + .build() + ) + .build(); + + // success + UserPasswordAuthenticationConfiguration userPasswordAuthenticationConfiguration = new UserPasswordAuthenticationConfiguration("some-user", new PropertiesFileSecret("some-secret")); + Connection testStore = Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .build() + ) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .connectionSpecification(new TestConnectionSpecification()) + .build(); + Assertions.assertArrayEquals(new AuthenticationMechanismType[]{CoreAuthenticationMechanismType.USER_PASSWORD}, testStore.getAuthenticationMechanisms().toArray()); + + // success: make sure if no auth mechanisms is specified, all mechanisms will be supported + Connection testStore2 = Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .build(); + Assertions.assertArrayEquals(new AuthenticationMechanismType[]{CoreAuthenticationMechanismType.USER_PASSWORD, CoreAuthenticationMechanismType.KERBEROS}, testStore2.getAuthenticationMechanisms().toArray()); + + // success: make sure if no authentication configuration type is specified, all types will be supported + Connection testStore3 = Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .authenticationMechanism(AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD).build()) + .build(); + Assertions.assertArrayEquals(Lists.mutable.of( + UserPasswordAuthenticationConfiguration.class, + EncryptedPrivateKeyPairAuthenticationConfiguration.class + ).toArray(), testStore3.getAuthenticationMechanism(CoreAuthenticationMechanismType.USER_PASSWORD).getAuthenticationConfigurationTypes().toArray()); + + // failure: missing connection specification + Exception exception; + + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .build(); + }); + Assertions.assertEquals("Connection specification is missing", exception.getMessage()); + + // failure: missing authentication configuration + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(new ApiKeyAuthenticationConfiguration()) + .build(); + }); + Assertions.assertEquals("Specified authentication configuration of type 'ApiKeyAuthenticationConfiguration' is not compatible. Supported configuration type(s):\n" + + "- UserPasswordAuthenticationConfiguration\n" + + "- EncryptedPrivateKeyPairAuthenticationConfiguration\n" + + "- KerberosAuthenticationConfiguration", exception.getMessage()); + + // failure: specified authentication configuration is not covered by any authentication mechanisms + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .connectionSpecification(new TestConnectionSpecification()) + .build(); + }); + Assertions.assertEquals("Authentication configuration is missing", exception.getMessage()); + + // failure: multiple configurations for one authentication mechanism + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .build(), + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .build() + ) + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .build(); + }); + Assertions.assertEquals("Found multiple configurations for authentication mechanism 'UsernamePassword'", exception.getMessage()); + + // failure: specified an authentication mechanism that is not covered by the database support + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.API_KEY) + .build() + ) + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .build(); + }); + Assertions.assertEquals("Authentication mechanism 'APIKey' is not covered by database support 'Test'. Supported mechanism(s):\n" + + "- UsernamePassword\n" + + "- Kerberos", exception.getMessage()); + + // failure: mismatch in authentication configuration types coverage with database support for an authentication mechanism + exception = Assertions.assertThrows(RuntimeException.class, () -> + { + Connection.builder() + .databaseSupport(databaseSupport) + .identifier("test::connection") + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes(KerberosAuthenticationConfiguration.class) + .build() + ) + .connectionSpecification(new TestConnectionSpecification()) + .authenticationConfiguration(userPasswordAuthenticationConfiguration) + .build(); + }); + Assertions.assertEquals("Authentication configuration type 'KerberosAuthenticationConfiguration' is not covered by database support 'Test' for authentication mechanism 'UsernamePassword'. Supported configuration type(s):\n" + + "- UserPasswordAuthenticationConfiguration\n" + + "- EncryptedPrivateKeyPairAuthenticationConfiguration", exception.getMessage()); + } + + private enum TestDatabaseType implements DatabaseType + { + TEST() + { + @Override + public String getIdentifier() + { + return "Test"; + } + } + } + + private static class TestConnectionSpecification extends ConnectionSpecification + { + @Override + public String shortId() + { + return null; + } + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/pom.xml b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/pom.xml new file mode 100644 index 00000000000..02369bc2133 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/pom.xml @@ -0,0 +1,140 @@ + + + + + org.finos.legend.engine + legend-engine-xts-connection + 4.35.4-SNAPSHOT + + 4.0.0 + + legend-engine-xt-connection-grammar + jar + Legend Engine - XT - Connection - Grammar + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + initialize + + unpack + + + + + org.finos.legend.engine + legend-engine-language-pure-grammar + jar + false + ${project.build.directory} + antlr/*.g4 + + + + + + + + org.antlr + antlr4-maven-plugin + + + + antlr4 + + + true + true + true + target/antlr + ${project.build.directory}/generated-sources + + + + + + + + + + + org.finos.legend.engine + legend-engine-protocol-pure + + + org.finos.legend.engine + legend-engine-protocol + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + + + org.finos.legend.engine + legend-engine-language-pure-grammar + + + + + + org.eclipse.collections + eclipse-collections-api + + + org.eclipse.collections + eclipse-collections + + + + + + com.fasterxml.jackson.core + jackson-databind + + + com.fasterxml.jackson.core + jackson-annotations + + + + + + org.antlr + antlr4-runtime + compile + + + + + + junit + junit + test + + + org.finos.legend.engine + legend-engine-language-pure-grammar + test-jar + test + + + + diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionLexerGrammar.g4 b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionLexerGrammar.g4 new file mode 100644 index 00000000000..1e4efe86491 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionLexerGrammar.g4 @@ -0,0 +1,10 @@ +lexer grammar ConnectionLexerGrammar; + +import M3LexerGrammar; + + +// ------------------------------------ KEYWORD -------------------------------------- + +DATABASE_CONNECTION: 'DatabaseConnection'; + +RAW_VALUE: 'rawValue'; diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionParserGrammar.g4 b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionParserGrammar.g4 new file mode 100644 index 00000000000..502449fee80 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/antlr4/org/finos/legend/engine/language/pure/grammar/from/antlr4/ConnectionParserGrammar.g4 @@ -0,0 +1,32 @@ +parser grammar ConnectionParserGrammar; + +import M3ParserGrammar; + +options +{ + tokenVocab = ConnectionLexerGrammar; +} + +// -------------------------------------- IDENTIFIER -------------------------------------- + +identifier: VALID_STRING | STRING + | ALL | LET | ALL_VERSIONS | ALL_VERSIONS_IN_RANGE | TO_BYTES_FUNCTION // from M3Parser + | DATABASE_CONNECTION | RAW_VALUE +; + +// -------------------------------------- DEFINITION -------------------------------------- + +definition: (databaseConnectionElement)* + EOF +; +databaseConnectionElement: DATABASE_CONNECTION qualifiedName + BRACE_OPEN + ( + rawValue + )* + BRACE_CLOSE +; +rawValue: RAW_VALUE COLON ISLAND_OPEN (rawValueContent)* SEMI_COLON +; +rawValueContent: ISLAND_START | ISLAND_BRACE_OPEN | ISLAND_CONTENT | ISLAND_HASH | ISLAND_BRACE_CLOSE | ISLAND_END +; \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParseTreeWalker.java b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParseTreeWalker.java new file mode 100644 index 00000000000..4f39265160a --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParseTreeWalker.java @@ -0,0 +1,73 @@ +// Copyright 2020 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.language.pure.grammar.from; + +import org.antlr.v4.runtime.CharStream; +import org.finos.legend.engine.language.pure.grammar.from.antlr4.ConnectionParserGrammar; +import org.finos.legend.engine.protocol.pure.v1.PureProtocolObjectMapperFactory; +import org.finos.legend.engine.protocol.pure.v1.model.SourceInformation; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.PackageableElement; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.section.DefaultCodeSection; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection; + +import java.util.function.Consumer; + +public class ConnectionParseTreeWalker +{ + private final CharStream input; + private final ParseTreeWalkerSourceInformation walkerSourceInformation; + private final Consumer elementConsumer; + private final DefaultCodeSection section; + + public ConnectionParseTreeWalker(CharStream input, ParseTreeWalkerSourceInformation walkerSourceInformation, Consumer elementConsumer, DefaultCodeSection section) + { + this.input = input; + this.walkerSourceInformation = walkerSourceInformation; + this.elementConsumer = elementConsumer; + this.section = section; + } + + public void visit(ConnectionParserGrammar.DefinitionContext ctx) + { + ctx.databaseConnectionElement().stream().map(this::visitElement).peek(e -> this.section.elements.add(e.getPath())).forEach(this.elementConsumer); + } + + private Connection visitElement(ConnectionParserGrammar.DatabaseConnectionElementContext ctx) + { + SourceInformation sourceInformation = walkerSourceInformation.getSourceInformation(ctx); + ConnectionParserGrammar.RawValueContext rawValueContext = PureGrammarParserUtility.validateAndExtractRequiredField(ctx.rawValue(), "rawValue", sourceInformation); + Connection connection; + try + { + StringBuilder text = new StringBuilder(); + for (ConnectionParserGrammar.RawValueContentContext fragment : rawValueContext.rawValueContent()) + { + text.append(fragment.getText()); + } + String rawValueText = text.length() > 0 ? text.substring(0, text.length() - 2) : text.toString(); + connection = PureProtocolObjectMapperFactory.getNewObjectMapper().readValue(rawValueText, Connection.class); + } + catch (Exception e) + { + throw new RuntimeException(e); + } + + connection.name = PureGrammarParserUtility.fromIdentifier(ctx.qualifiedName().identifier()); + connection._package = ctx.qualifiedName().packagePath() == null ? "" : PureGrammarParserUtility.fromPath(ctx.qualifiedName().packagePath().identifier()); + connection.sourceInformation = sourceInformation; + + return connection; + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParserExtension.java b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParserExtension.java new file mode 100644 index 00000000000..c2718631b4c --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/from/ConnectionParserExtension.java @@ -0,0 +1,64 @@ +// Copyright 2020 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.language.pure.grammar.from; + +import org.antlr.v4.runtime.CharStream; +import org.antlr.v4.runtime.CharStreams; +import org.antlr.v4.runtime.CommonTokenStream; +import org.eclipse.collections.impl.factory.Lists; +import org.finos.legend.engine.language.pure.grammar.from.antlr4.ConnectionLexerGrammar; +import org.finos.legend.engine.language.pure.grammar.from.antlr4.ConnectionParserGrammar; +import org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtension; +import org.finos.legend.engine.language.pure.grammar.from.extension.SectionParser; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.PackageableElement; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.section.DefaultCodeSection; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.section.Section; + +import java.util.function.Consumer; + +public class ConnectionParserExtension implements PureGrammarParserExtension +{ + public static final String NAME = "DatabaseConnection"; + + @Override + public Iterable getExtraSectionParsers() + { + return Lists.immutable.with(SectionParser.newParser(NAME, ConnectionParserExtension::parseSection)); + } + + private static Section parseSection(SectionSourceCode sectionSourceCode, Consumer elementConsumer, PureGrammarParserContext pureGrammarParserContext) + { + SourceCodeParserInfo parserInfo = getConnectionParserInfo(sectionSourceCode); + DefaultCodeSection section = new DefaultCodeSection(); + section.parserName = sectionSourceCode.sectionType; + section.sourceInformation = parserInfo.sourceInformation; + ConnectionParseTreeWalker walker = new ConnectionParseTreeWalker(parserInfo.input, parserInfo.walkerSourceInformation, elementConsumer, section); + walker.visit((ConnectionParserGrammar.DefinitionContext) parserInfo.rootContext); + return section; + } + + private static SourceCodeParserInfo getConnectionParserInfo(SectionSourceCode sectionSourceCode) + { + CharStream input = CharStreams.fromString(sectionSourceCode.code); + ParserErrorListener errorListener = new ParserErrorListener(sectionSourceCode.walkerSourceInformation); + ConnectionLexerGrammar lexer = new ConnectionLexerGrammar(input); + lexer.removeErrorListeners(); + lexer.addErrorListener(errorListener); + ConnectionParserGrammar parser = new ConnectionParserGrammar(new CommonTokenStream(lexer)); + parser.removeErrorListeners(); + parser.addErrorListener(errorListener); + return new SourceCodeParserInfo(sectionSourceCode.code, input, sectionSourceCode.sourceInformation, sectionSourceCode.walkerSourceInformation, lexer, parser, parser.definition()); + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/to/ConnectionGrammarComposerExtension.java b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/to/ConnectionGrammarComposerExtension.java new file mode 100644 index 00000000000..02608afd877 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/java/org/finos/legend/engine/language/pure/grammar/to/ConnectionGrammarComposerExtension.java @@ -0,0 +1,90 @@ +// Copyright 2020 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.language.pure.grammar.to; + +import com.fasterxml.jackson.annotation.JsonInclude; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import org.eclipse.collections.api.block.function.Function3; +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.impl.utility.LazyIterate; +import org.eclipse.collections.impl.utility.ListIterate; +import org.finos.legend.engine.language.pure.grammar.from.ConnectionParserExtension; +import org.finos.legend.engine.language.pure.grammar.to.extension.PureGrammarComposerExtension; +import org.finos.legend.engine.protocol.pure.v1.PureProtocolObjectMapperFactory; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.PackageableElement; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection; + +import java.util.List; + +import static org.finos.legend.engine.language.pure.grammar.to.PureGrammarComposerUtility.getTabString; + +public class ConnectionGrammarComposerExtension implements PureGrammarComposerExtension +{ + @Override + public List, PureGrammarComposerContext, String, String>> getExtraSectionComposers() + { + return Lists.fixedSize.with((elements, context, sectionName) -> + { + if (!ConnectionParserExtension.NAME.equals(sectionName)) + { + return null; + } + return ListIterate.collect(elements, element -> + { + if (element instanceof Connection) + { + return renderElement((Connection) element, context); + } + return "/* Can't transform element '" + element.getPath() + "' in this section */"; + }).makeString("\n\n"); + }); + } + + @Override + public List, PureGrammarComposerContext, List, PureFreeSectionGrammarComposerResult>> getExtraFreeSectionComposers() + { + return Lists.fixedSize.with((elements, context, composedSections) -> + { + List composableElements = ListIterate.selectInstancesOf(elements, Connection.class); + return composableElements.isEmpty() ? null : new PureFreeSectionGrammarComposerResult(LazyIterate.collect(composableElements, el -> renderElement(el, context)).makeString("###" + ConnectionParserExtension.NAME + "\n", "\n\n", ""), composableElements); + }); + } + + private static String renderElement(Connection element, PureGrammarComposerContext context) + { + String value; + try + { + // @HACKY: new-connection-framework + element.sourceInformation = null; + ObjectMapper objectMapper = PureProtocolObjectMapperFactory.getNewObjectMapper(); + objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL); + objectMapper.enable(SerializationFeature.INDENT_OUTPUT); + value = objectMapper.writeValueAsString(element); + } + catch (Exception e) + { + throw new RuntimeException(e); + } + + return "DatabaseConnection " + PureGrammarComposerUtility.convertPath(element.getPath()) + "\n" + + "{\n" + + (getTabString() + "rawValue: #{\n" + + ListIterate.collect(Lists.mutable.of(value.split("\n")), line -> getTabString() + line).makeString("\n") + "\n" + + getTabString() + "}#;\n") + + "}"; + } +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtension b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtension new file mode 100644 index 00000000000..747e1deb95e --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.from.extension.PureGrammarParserExtension @@ -0,0 +1 @@ +org.finos.legend.engine.language.pure.grammar.from.ConnectionParserExtension \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.to.extension.PureGrammarComposerExtension b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.to.extension.PureGrammarComposerExtension new file mode 100644 index 00000000000..889fae4c0bc --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-grammar/src/main/resources/META-INF/services/org.finos.legend.engine.language.pure.grammar.to.extension.PureGrammarComposerExtension @@ -0,0 +1 @@ +org.finos.legend.engine.language.pure.grammar.to.ConnectionGrammarComposerExtension \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/pom.xml b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/pom.xml new file mode 100644 index 00000000000..ca8c0255238 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/pom.xml @@ -0,0 +1,69 @@ + + + + + + org.finos.legend.engine + legend-engine-xts-connection + 4.35.4-SNAPSHOT + + 4.0.0 + + legend-engine-xt-connection-protocol + jar + Legend Engine - XT - Connection - Protocol + + + + + org.finos.legend.engine + legend-engine-xt-authentication-protocol + + + org.finos.legend.engine + legend-engine-protocol-pure + + + + + + com.fasterxml.jackson.core + jackson-annotations + + + + + + org.eclipse.collections + eclipse-collections-api + + + + + + org.junit.jupiter + junit-jupiter-api + test + + + org.junit.jupiter + junit-jupiter-engine + test + + + + \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/ConnectionProtocolExtension.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/ConnectionProtocolExtension.java new file mode 100644 index 00000000000..10963ed2bc9 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/ConnectionProtocolExtension.java @@ -0,0 +1,61 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.protocol.pure.v1; + +import org.eclipse.collections.api.block.function.Function0; +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.api.factory.Maps; +import org.finos.legend.engine.protocol.pure.v1.extension.ProtocolSubTypeInfo; +import org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension; +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.PackageableElement; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.KerberosAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; + +import java.util.List; +import java.util.Map; + +public class ConnectionProtocolExtension implements PureProtocolExtension +{ + public static final String CONNECTION_CLASSIFIER_PATH = "meta::pure::metamodel::connection::Connection"; + + @Override + public List>>> getExtraProtocolSubTypeInfoCollectors() + { + return Lists.fixedSize.of(() -> Lists.fixedSize.of( + // Packageable Element + ProtocolSubTypeInfo.newBuilder(PackageableElement.class) + // TODO: ideally we should be able to set this as `connection`, but this will clash with `PackageableElement` + .withSubtype(Connection.class, "databaseConnection") + .build(), + // Authentication + ProtocolSubTypeInfo.newBuilder(AuthenticationConfiguration.class) + .withSubtype(EncryptedPrivateKeyPairAuthenticationConfiguration.class, "KeyPair") + .withSubtype(UserPasswordAuthenticationConfiguration.class, "UserPassword") + .withSubtype(KerberosAuthenticationConfiguration.class, "Kerberos") + .build() + )); + } + + @Override + public Map, String> getExtraProtocolToClassifierPathMap() + { + return Maps.mutable.with( + Connection.class, CONNECTION_CLASSIFIER_PATH + ); + } +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/ApiKeyAuthenticationConfiguration.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ApiKeyAuthenticationConfiguration.java similarity index 92% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/ApiKeyAuthenticationConfiguration.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ApiKeyAuthenticationConfiguration.java index 4e22064c7c7..bda1af4cabe 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/ApiKeyAuthenticationConfiguration.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ApiKeyAuthenticationConfiguration.java @@ -12,9 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.impl; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.CredentialVaultSecret; public class ApiKeyAuthenticationConfiguration extends AuthenticationConfiguration diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationConfiguration.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationConfiguration.java similarity index 77% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationConfiguration.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationConfiguration.java index 10774902332..4ae3b429833 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationConfiguration.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationConfiguration.java @@ -12,8 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.protocol; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; +import com.fasterxml.jackson.annotation.JsonTypeInfo; + +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "_type") public abstract class AuthenticationConfiguration { public abstract String shortId(); diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationMechanism.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationMechanism.java new file mode 100644 index 00000000000..af5c3ad460a --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/AuthenticationMechanism.java @@ -0,0 +1,23 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; + +import java.util.List; + +public final class AuthenticationMechanism +{ + public String authenticationMechanismType; + public List configurationTypes; +} diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/Connection.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/Connection.java new file mode 100644 index 00000000000..99d989c6be3 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/Connection.java @@ -0,0 +1,28 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; + +import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.PackageableElement; + +import java.util.List; + +public final class Connection extends PackageableElement +{ + public String databaseType; + public List authenticationMechanisms; + public ConnectionSpecification connectionSpecification; + public AuthenticationConfiguration authenticationConfiguration; +} + diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/ConnectionSpecification.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ConnectionSpecification.java similarity index 77% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/ConnectionSpecification.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ConnectionSpecification.java index 908ab661327..451553e9afb 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/ConnectionSpecification.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/ConnectionSpecification.java @@ -12,8 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.protocol; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; +import com.fasterxml.jackson.annotation.JsonTypeInfo; + +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "_type") public abstract class ConnectionSpecification { public abstract String shortId(); diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/DatabaseSupport.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/DatabaseSupport.java new file mode 100644 index 00000000000..fcd17a2c3d0 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/DatabaseSupport.java @@ -0,0 +1,26 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; + +import com.fasterxml.jackson.annotation.JsonTypeInfo; + +import java.util.List; + +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "_type") +public class DatabaseSupport +{ + public String databaseType; + public List authenticationMechanisms; +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/EncryptedPrivateKeyPairAuthenticationConfiguration.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/EncryptedPrivateKeyPairAuthenticationConfiguration.java similarity index 93% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/EncryptedPrivateKeyPairAuthenticationConfiguration.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/EncryptedPrivateKeyPairAuthenticationConfiguration.java index 0bd50613949..e6409c1d142 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/EncryptedPrivateKeyPairAuthenticationConfiguration.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/EncryptedPrivateKeyPairAuthenticationConfiguration.java @@ -12,9 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.impl; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.CredentialVaultSecret; public class EncryptedPrivateKeyPairAuthenticationConfiguration extends AuthenticationConfiguration @@ -25,6 +24,7 @@ public class EncryptedPrivateKeyPairAuthenticationConfiguration extends Authenti public EncryptedPrivateKeyPairAuthenticationConfiguration() { + // jackson } public EncryptedPrivateKeyPairAuthenticationConfiguration(String userName, CredentialVaultSecret privateKey, CredentialVaultSecret passphrase) diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosAuthenticationConfiguration.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/KerberosAuthenticationConfiguration.java similarity index 87% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosAuthenticationConfiguration.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/KerberosAuthenticationConfiguration.java index 61b6ac6594a..ad2dcdfc702 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/KerberosAuthenticationConfiguration.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/KerberosAuthenticationConfiguration.java @@ -12,14 +12,13 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.impl; - -import org.finos.legend.connection.protocol.AuthenticationConfiguration; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; public class KerberosAuthenticationConfiguration extends AuthenticationConfiguration { public KerberosAuthenticationConfiguration() { + // authentication } @Override diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordAuthenticationConfiguration.java b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/UserPasswordAuthenticationConfiguration.java similarity index 91% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordAuthenticationConfiguration.java rename to legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/UserPasswordAuthenticationConfiguration.java index 61e8fb5c096..756de8dd6fd 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/impl/UserPasswordAuthenticationConfiguration.java +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/packageableElement/connection/UserPasswordAuthenticationConfiguration.java @@ -12,9 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.impl; +package org.finos.legend.engine.protocol.pure.v1.packageableElement.connection; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.CredentialVaultSecret; public class UserPasswordAuthenticationConfiguration extends AuthenticationConfiguration @@ -24,6 +23,7 @@ public class UserPasswordAuthenticationConfiguration extends AuthenticationConfi public UserPasswordAuthenticationConfiguration() { + // jackson } public UserPasswordAuthenticationConfiguration(String username, CredentialVaultSecret password) diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/resources/META-INF/services/org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/resources/META-INF/services/org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension new file mode 100644 index 00000000000..dfa880debb4 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-protocol/src/main/resources/META-INF/services/org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension @@ -0,0 +1 @@ +org.finos.legend.engine.protocol.pure.v1.ConnectionProtocolExtension \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/pom.xml b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/pom.xml new file mode 100644 index 00000000000..0052f847951 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/pom.xml @@ -0,0 +1,158 @@ + + + + + + org.finos.legend.engine + legend-engine-xts-connection + 4.35.4-SNAPSHOT + + 4.0.0 + + legend-engine-xt-connection-pure-metamodel + jar + Legend Engine - XT - Connection - Pure - Metamodel + + + + + org.finos.legend.pure + legend-pure-maven-generation-par + + src/main/resources + ${legend.pure.version} + + platform + core + core_connection_metamodel + + + ${project.basedir}/src/main/resources/core_connection_metamodel.definition.json + + + + + + generate-sources + + build-pure-jar + + + + + + org.finos.legend.pure + legend-pure-m2-dsl-diagram-grammar + ${legend.pure.version} + + + org.finos.legend.pure + legend-pure-m2-dsl-graph-grammar + ${legend.pure.version} + + + org.finos.legend.engine + legend-engine-pure-code-compiled-core + ${project.version} + + + org.finos.legend.engine + legend-engine-xt-json-pure + ${project.version} + + + + + org.finos.legend.pure + legend-pure-maven-generation-java + + + compile + + build-pure-compiled-jar + + + true + true + modular + true + + core_connection_metamodel + + + + + + + org.finos.legend.pure + legend-pure-m2-dsl-diagram-grammar + ${legend.pure.version} + + + org.finos.legend.pure + legend-pure-m2-dsl-graph-grammar + ${legend.pure.version} + + + org.finos.legend.engine + legend-engine-pure-code-compiled-core + ${project.version} + + + + + + + + + + org.finos.legend.pure + legend-pure-m4 + + + org.finos.legend.pure + legend-pure-m3-core + + + org.finos.legend.pure + legend-pure-runtime-java-engine-compiled + + + + + + org.finos.legend.engine + legend-engine-pure-code-compiled-core + + + org.finos.legend.engine + legend-engine-pure-platform-java + + + + + + org.eclipse.collections + eclipse-collections + + + org.eclipse.collections + eclipse-collections-api + + + + diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/java/org/finos/legend/pure/code/core/CoreConnectionRepositoryProvider.java b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/java/org/finos/legend/pure/code/core/CoreConnectionRepositoryProvider.java new file mode 100644 index 00000000000..bfa033889df --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/java/org/finos/legend/pure/code/core/CoreConnectionRepositoryProvider.java @@ -0,0 +1,29 @@ +// Copyright 2021 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.pure.code.core; + +import org.finos.legend.pure.m3.serialization.filesystem.repository.CodeRepository; +import org.finos.legend.pure.m3.serialization.filesystem.repository.CodeRepositoryProvider; +import org.finos.legend.pure.m3.serialization.filesystem.repository.GenericCodeRepository; + +public class CoreConnectionRepositoryProvider implements CodeRepositoryProvider +{ + @Override + public CodeRepository repository() + { + return GenericCodeRepository.build("core_connection_metamodel.definition.json"); + } +} + diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/META-INF/services/org.finos.legend.pure.m3.serialization.filesystem.repository.CodeRepositoryProvider b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/META-INF/services/org.finos.legend.pure.m3.serialization.filesystem.repository.CodeRepositoryProvider new file mode 100644 index 00000000000..05623448288 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/META-INF/services/org.finos.legend.pure.m3.serialization.filesystem.repository.CodeRepositoryProvider @@ -0,0 +1 @@ +org.finos.legend.pure.code.core.CoreConnectionRepositoryProvider \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel.definition.json b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel.definition.json new file mode 100644 index 00000000000..63239e854b0 --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel.definition.json @@ -0,0 +1,9 @@ +{ + "name": "core_connection_metamodel", + "pattern": "(meta::pure::metamodel::connection)(::.*)?", + "dependencies": [ + "platform", + "platform_functions", + "core" + ] +} \ No newline at end of file diff --git a/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel/metamodel.pure b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel/metamodel.pure new file mode 100644 index 00000000000..b6cd3dcfe4a --- /dev/null +++ b/legend-engine-xts-connection/legend-engine-xt-connection-pure-metamodel/src/main/resources/core_connection_metamodel/metamodel.pure @@ -0,0 +1,19 @@ +// Copyright 2022 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +Class meta::pure::metamodel::connection::Connection extends PackageableElement +{ + // @HACKY: new-connection-framework + rawValue: Any[1]; +} diff --git a/legend-engine-xts-connection/pom.xml b/legend-engine-xts-connection/pom.xml new file mode 100644 index 00000000000..3dda8473e68 --- /dev/null +++ b/legend-engine-xts-connection/pom.xml @@ -0,0 +1,36 @@ + + + + + org.finos.legend.engine + legend-engine + 4.35.4-SNAPSHOT + + 4.0.0 + + legend-engine-xts-connection + pom + Legend Engine - XTS - Connection + + + legend-engine-xt-connection-factory + legend-engine-xt-connection-compiler + legend-engine-xt-connection-grammar + legend-engine-xt-connection-protocol + legend-engine-xt-connection-pure-metamodel + + \ No newline at end of file diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/pom.xml b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/pom.xml index 082cb07bbd5..e4e0d7652e9 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/pom.xml +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/pom.xml @@ -1,5 +1,6 @@ - + legend-engine-xts-data-push org.finos.legend.engine @@ -37,24 +38,57 @@ + + org.finos.legend.engine + legend-engine-shared-core + org.finos.legend.engine legend-engine-xt-authentication-implementation-core org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory org.finos.legend.engine legend-engine-xt-relationalStore-connection + + org.finos.legend.engine + legend-engine-xt-relationalStore-snowflake-connection + + + org.finos.legend.engine + legend-engine-language-pure-modelManager-sdlc + + + org.finos.legend.engine + legend-engine-protocol-pure + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + + + + + + org.finos.legend.engine + legend-engine-xt-relationalStore-protocol + runtime + + + org.finos.legend.engine + legend-engine-xt-relationalStore-snowflake-protocol + runtime + org.finos.legend.engine legend-engine-xt-relationalStore-postgres-connection runtime - + @@ -73,14 +107,6 @@ - - - - org.eclipse.collections - eclipse-collections-api - - - com.fasterxml.jackson.core @@ -96,12 +122,27 @@ - + - org.apache.httpcomponents - httpclient + org.eclipse.collections + eclipse-collections-api + + + org.eclipse.collections + eclipse-collections - + + + + + org.pac4j.jax-rs + core + + + org.pac4j + pac4j-core + + @@ -124,6 +165,65 @@ + + + org.slf4j + slf4j-api + + + + + javax.servlet + javax.servlet-api + + + org.apache.httpcomponents + httpclient + + + commons-codec + commons-codec + + + + + org.apache.httpcomponents + httpcore + + + commons-logging + commons-logging + + + + + + software.amazon.awssdk + s3 + + + software.amazon.awssdk + sdk-core + + + software.amazon.awssdk + regions + + + software.amazon.awssdk + auth + + + + io.deephaven + deephaven-csv + + + io.deephaven + deephaven-csv-fast-double-parser + runtime + + com.fasterxml.jackson.dataformat @@ -178,6 +278,21 @@ dropwizard-testing test + + io.minio + minio + test + + + org.finos.legend.engine + legend-engine-xt-relationalStore-postgres-test-support + test + + + org.testcontainers + testcontainers + test + \ No newline at end of file diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusher.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusher.java new file mode 100644 index 00000000000..b8abecd5280 --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusher.java @@ -0,0 +1,32 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush; + +import org.finos.legend.connection.Connection; +import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.engine.datapush.data.Data; +import org.finos.legend.engine.shared.core.identity.Identity; + +public abstract class DataPusher +{ + protected ConnectionFactory connectionFactory; + + public void configure(ConnectionFactory connectionFactory) + { + this.connectionFactory = connectionFactory; + } + + public abstract void writeCSV(Identity identity, Connection connection, Data data) throws Exception; +} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusherProvider.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusherProvider.java new file mode 100644 index 00000000000..c1b56623f75 --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/DataPusherProvider.java @@ -0,0 +1,22 @@ +// Copyright 2022 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush; + +import org.finos.legend.connection.Connection; + +public interface DataPusherProvider +{ + DataPusher getDataPusher(Connection connection); +} diff --git a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanism.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/CSVData.java similarity index 84% rename from legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanism.java rename to legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/CSVData.java index 7a104453eee..250cc0d0d7a 100644 --- a/legend-engine-xts-authentication/legend-engine-xt-authentication-connection-factory/src/main/java/org/finos/legend/connection/protocol/AuthenticationMechanism.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/CSVData.java @@ -12,9 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.protocol; +package org.finos.legend.engine.datapush.data; -public interface AuthenticationMechanism +public class CSVData extends Data { - String getLabel(); + public String value; } diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/Data.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/Data.java new file mode 100644 index 00000000000..e8350c31bca --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/data/Data.java @@ -0,0 +1,26 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush.data; + +import com.fasterxml.jackson.annotation.JsonSubTypes; +import com.fasterxml.jackson.annotation.JsonTypeInfo; + +@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "_type") +@JsonSubTypes({ + @JsonSubTypes.Type(value = CSVData.class, name = "csv"), +}) +public class Data +{ +} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/impl/SnowflakeWithS3StageDataPusher.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/impl/SnowflakeWithS3StageDataPusher.java new file mode 100644 index 00000000000..e7c0cf26734 --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/impl/SnowflakeWithS3StageDataPusher.java @@ -0,0 +1,183 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush.impl; + +import io.deephaven.csv.CsvSpecs; +import io.deephaven.csv.reading.CsvReader; +import io.deephaven.csv.sinks.SinkFactory; +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.impl.utility.ListIterate; +import org.finos.legend.connection.Connection; +import org.finos.legend.engine.datapush.DataPusher; +import org.finos.legend.engine.datapush.data.CSVData; +import org.finos.legend.engine.datapush.data.Data; +import org.finos.legend.engine.shared.core.identity.Identity; +import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3ClientBuilder; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; +import software.amazon.awssdk.services.s3.model.PutObjectResponse; + +import java.io.ByteArrayInputStream; +import java.net.URI; +import java.sql.Statement; +import java.util.UUID; + +public class SnowflakeWithS3StageDataPusher extends DataPusher +{ + private final String tableName; + private final String stageName; + private final S3DataStage s3DataStage; + + public SnowflakeWithS3StageDataPusher(String s3StageBucketName, String s3StageEndpoint, AwsCredentialsProvider s3StageCredentialProvider, String tableName, String stageName) + { + this.tableName = tableName; + this.stageName = stageName; + this.s3DataStage = new S3DataStage(s3StageBucketName, s3StageEndpoint, s3StageCredentialProvider); + } + + @Override + public void writeCSV(Identity identity, Connection connection, Data data) throws Exception + { + // TODO: this is probably not performant for streaming/large CSV, we should think of how to optimize this later + CSVData csvData = (CSVData) data; + CsvSpecs specs = CsvSpecs.csv(); + CsvReader.Result csvParserResult = CsvReader.read(specs, new ByteArrayInputStream(csvData.value.getBytes()), SinkFactory.arrays()); + String filePath = this.s3DataStage.write(identity, csvData); + this.uploadCSVToSnowflake(identity, connection, filePath, csvParserResult); + } + + public void uploadCSVToSnowflake(Identity identity, Connection connection, String filePath, CsvReader.Result csvParserResult) throws Exception + { + java.sql.Connection jdbcConnection = this.connectionFactory.getConnection(identity, connection); + + String tableCreationQuery = String.format("CREATE TABLE %s (%s);", this.tableName, ListIterate.collect( + Lists.mutable.of(csvParserResult.columns()), column -> + { + String dataType = null; + // NOTE: these are specific to Snowflake + // See https://docs.snowflake.com/en/sql-reference/data-types + switch (column.dataType()) + { + case BOOLEAN_AS_BYTE: + dataType = "BOOLEAN"; + break; + case BYTE: + case SHORT: + case INT: + case LONG: + case DATETIME_AS_LONG: + case TIMESTAMP_AS_LONG: + dataType = "INTEGER"; + break; + case FLOAT: + case DOUBLE: + dataType = "FLOAT"; + break; + case STRING: + case CHAR: + dataType = "STRING"; + break; + case CUSTOM: + throw new RuntimeException("Not possible"); + } + // Put quote around table name to avoid problems with column names with spaces + return String.format("\"%s\" %s", column.name(), dataType); + } + ).makeString(",")); + // Give Snowflake the full s3 path to improve performance as no lookup is necessary + // See https://community.snowflake.com/s/question/0D50Z00009Y7eCRSAZ/copy-from-s3-into-table-command-is-extremely-slow + String insertQuery = String.format("COPY INTO %s FROM @%s/%s file_format = (type = csv skip_header = 1);", this.tableName, this.stageName, filePath); + + try + { + Statement statement = jdbcConnection.createStatement(); + statement.execute(String.format("DROP TABLE IF EXISTS %s;", this.tableName)); + statement.execute(tableCreationQuery); + statement.execute(insertQuery); + statement.close(); + } + catch (Exception e) + { + e.printStackTrace(); + throw new RuntimeException(e); + } + } + + public static class S3DataStage + { + private final String bucket; + private final String endpoint; + private final AwsCredentialsProvider credentialsProvider; + + public S3DataStage(String bucket, String endpoint, AwsCredentialsProvider credentialsProvider) + { + this.bucket = bucket; + this.endpoint = endpoint; + this.credentialsProvider = credentialsProvider; + } + + private static String generateBucketName(Identity identity) + { + return identity.getName().replaceAll("_", "").toLowerCase(); + } + + private static String generateObjectKey() + { + return UUID.randomUUID().toString(); + } + + private static String generateObjectPrefix(Identity identity) + { + return identity.getName(); + } + + private S3Client getS3Client() + { + S3ClientBuilder clientBuilder = S3Client + .builder() + .credentialsProvider(this.credentialsProvider) + .region(Region.US_EAST_1); + if (this.endpoint != null) + { + clientBuilder.endpointOverride(URI.create(this.endpoint)); + } + return clientBuilder.build(); + } + + public String write(Identity identity, Data data) throws Exception + { + CSVData csvData = (CSVData) data; + S3Client s3Client = this.getS3Client(); + String key = String.format("%s/%s", generateObjectPrefix(identity), generateObjectKey()); + try + { + PutObjectRequest putObjectRequest = PutObjectRequest.builder() + .bucket(this.bucket) + .key(key).build(); + RequestBody requestBody = RequestBody.fromString(csvData.value); + PutObjectResponse putObjectResponse = s3Client.putObject(putObjectRequest, requestBody); + } + catch (Exception e) + { + e.printStackTrace(); + throw new RuntimeException(e); + } + return key; + } + } +} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/BaseDataPushServer.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/BaseDataPushServer.java index 4959fee8f09..1129ce8f1bf 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/BaseDataPushServer.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/BaseDataPushServer.java @@ -15,26 +15,46 @@ package org.finos.legend.engine.datapush.server; import io.dropwizard.setup.Environment; -import org.finos.legend.engine.datapush.server.config.DataPushServerConfiguration; +import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.connection.IdentityFactory; +import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.engine.datapush.DataPusherProvider; +import org.finos.legend.engine.datapush.server.configuration.DataPushServerConfiguration; import org.finos.legend.engine.datapush.server.resources.DataPushResource; +import org.finos.legend.engine.protocol.pure.v1.PureProtocolObjectMapperFactory; import org.finos.legend.engine.server.support.server.BaseServer; - -import java.net.InetAddress; -import java.net.UnknownHostException; +import org.finos.legend.engine.shared.core.ObjectMapperFactory; public abstract class BaseDataPushServer extends BaseServer { - protected ServerInfo serverInfo; + protected LegendEnvironment environment; + protected IdentityFactory identityFactory; + protected ConnectionFactory connectionFactory; + protected DataPusherProvider dataPushProvider; + + @Override + public void initialize(io.dropwizard.setup.Bootstrap bootstrap) + { + super.initialize(bootstrap); + + PureProtocolObjectMapperFactory.withPureProtocolExtensions(bootstrap.getObjectMapper()); + ObjectMapperFactory.withStandardConfigurations(bootstrap.getObjectMapper()); + } - private static String getLocalHostName() throws UnknownHostException + @Override + public void run(DataPushServerConfiguration configuration, Environment environment) { - return InetAddress.getLocalHost().getHostName(); + this.environment = this.buildLegendEnvironment(configuration); + this.identityFactory = this.buildIdentityFactory(configuration, this.environment); + this.connectionFactory = this.buildConnectionFactory(configuration, this.environment); + this.dataPushProvider = this.buildDataPushProvider(); + super.run(configuration, environment); } @Override protected void configureServerCore(DataPushServerConfiguration configuration, Environment environment) { - environment.jersey().register(DataPushResource.class); + environment.jersey().register(new DataPushResource(configuration.getMetadataServerConfiguration(), this.environment, this.identityFactory, this.connectionFactory, this.dataPushProvider)); } @Override @@ -43,32 +63,11 @@ protected void configureServerExtension(DataPushServerConfiguration configuratio super.configureServerExtension(configuration, environment); } - public static final class ServerInfo - { - private final String hostName; - private final String initTime; - private final ServerPlatformInfo serverPlatformInfo; + public abstract LegendEnvironment buildLegendEnvironment(DataPushServerConfiguration configuration); - private ServerInfo(String hostName, String initTime, ServerPlatformInfo serverPlatformInfo) - { - this.hostName = hostName; - this.initTime = initTime; - this.serverPlatformInfo = (serverPlatformInfo == null) ? new ServerPlatformInfo(null, null, null) : serverPlatformInfo; - } + public abstract IdentityFactory buildIdentityFactory(DataPushServerConfiguration configuration, LegendEnvironment environment); - public String getHostName() - { - return this.hostName; - } + public abstract ConnectionFactory buildConnectionFactory(DataPushServerConfiguration configuration, LegendEnvironment environment); - public String getInitTime() - { - return this.initTime; - } - - public ServerPlatformInfo getPlatform() - { - return this.serverPlatformInfo; - } - } + public abstract DataPusherProvider buildDataPushProvider(); } diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryBundle.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryBundle.java deleted file mode 100644 index 2b96dfd3940..00000000000 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryBundle.java +++ /dev/null @@ -1,113 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.engine.datapush.server; - -import io.dropwizard.Configuration; -import io.dropwizard.ConfiguredBundle; -import io.dropwizard.setup.Bootstrap; -import io.dropwizard.setup.Environment; -import org.finos.legend.authentication.vault.CredentialVault; -import org.finos.legend.authentication.vault.impl.EnvironmentCredentialVault; -import org.finos.legend.authentication.vault.impl.SystemPropertiesCredentialVault; -import org.finos.legend.connection.AuthenticationMechanismConfiguration; -import org.finos.legend.connection.ConnectionFactory; -import org.finos.legend.connection.DatabaseType; -import org.finos.legend.connection.impl.DefaultStoreInstanceProvider; -import org.finos.legend.connection.IdentityFactory; -import org.finos.legend.connection.LegendEnvironment; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstanceProvider; -import org.finos.legend.connection.impl.UserPasswordAuthenticationConfiguration; -import org.finos.legend.connection.protocol.AuthenticationMechanismType; - -import java.util.List; -import java.util.function.Function; - -public class ConnectionFactoryBundle implements ConfiguredBundle -{ - private static LegendEnvironment environment; - private static IdentityFactory identityFactory; - private static StoreInstanceProvider storeInstanceProvider; - private static ConnectionFactory connectionFactory; - private final List credentialVaults; - private final Function configSupplier; - - public ConnectionFactoryBundle(Function configSupplier, List credentialVaults) - { - this.configSupplier = configSupplier; - this.credentialVaults = credentialVaults; - } - - @Override - public void initialize(Bootstrap bootstrap) - { - } - - @Override - public void run(C configuration, Environment environment) - { - ConnectionFactoryBundle.environment = new LegendEnvironment.Builder() - // TODO: @akphi - add a property credential vault and load its content up from the config -// .withVault(propertiesFileCredentialVault) - .withVault(new SystemPropertiesCredentialVault()) - .withVault(new EnvironmentCredentialVault()) - .withVaults(this.credentialVaults) - .withStoreSupport(new RelationalDatabaseStoreSupport.Builder(DatabaseType.POSTGRES) - .withIdentifier("Postgres") - .withAuthenticationMechanismConfigurations( - new AuthenticationMechanismConfiguration.Builder(AuthenticationMechanismType.USER_PASSWORD).withAuthenticationConfigurationTypes(UserPasswordAuthenticationConfiguration.class).build() - ).build()) - .build(); - - identityFactory = new IdentityFactory.Builder(ConnectionFactoryBundle.environment) - .build(); - - storeInstanceProvider = new DefaultStoreInstanceProvider.Builder().build(); - - connectionFactory = new ConnectionFactory.Builder(ConnectionFactoryBundle.environment, storeInstanceProvider) -// .withCredentialBuilderProvider(new DefaultCredentialBuilderProvider()) // can also use service loader -// .withConnectionBuilderProvider(new DefaultConnectionBuilderProvider()) // can also use service loader - .build(); - - // TODO: register store instances - } - - public static LegendEnvironment getEnvironment() - { - if (environment == null) - { - throw new IllegalStateException("Environment configuration has not been set!"); - } - return environment; - } - - public static IdentityFactory getIdentityFactory() - { - if (identityFactory == null) - { - throw new IllegalStateException("Identity factory has not been configured properly!"); - } - return identityFactory; - } - - public static ConnectionFactory getConnectionFactory() - { - if (connectionFactory == null) - { - throw new IllegalStateException("Connection factory has not been configured properly!"); - } - return connectionFactory; - } -} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionModelLoader.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionModelLoader.java new file mode 100644 index 00000000000..0b6daa1ec43 --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionModelLoader.java @@ -0,0 +1,182 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush.server; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.http.client.CookieStore; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpGet; +import org.apache.http.cookie.Cookie; +import org.apache.http.impl.client.BasicCookieStore; +import org.apache.http.impl.client.CloseableHttpClient; +import org.eclipse.collections.api.factory.Lists; +import org.eclipse.collections.impl.utility.ArrayIterate; +import org.eclipse.collections.impl.utility.ListIterate; +import org.finos.legend.engine.language.pure.modelManager.sdlc.SDLCLoader; +import org.finos.legend.engine.language.pure.modelManager.sdlc.configuration.MetaDataServerConfiguration; +import org.finos.legend.engine.protocol.pure.v1.model.context.AlloySDLC; +import org.finos.legend.engine.protocol.pure.v1.model.context.PureModelContextData; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection; +import org.finos.legend.engine.shared.core.ObjectMapperFactory; +import org.finos.legend.engine.shared.core.kerberos.HttpClientBuilder; +import org.finos.legend.engine.shared.core.operational.Assert; +import org.finos.legend.engine.shared.core.operational.errorManagement.EngineException; +import org.finos.legend.engine.shared.core.operational.logs.LoggingEventType; +import org.pac4j.core.profile.CommonProfile; + +import javax.servlet.http.HttpServletRequest; +import java.util.Date; +import java.util.List; + +public class ConnectionModelLoader +{ + private final MetaDataServerConfiguration metaDataServerConfiguration; + + public ConnectionModelLoader(MetaDataServerConfiguration metaDataServerConfiguration) + { + this.metaDataServerConfiguration = metaDataServerConfiguration; + } + + public Connection getConnectionFromSDLCWorkspace(HttpServletRequest request, String projectId, String workspaceId, boolean isGroupWorkspace, String connectionPath) + { + // NOTE: this flow is really meant only for development, here we have to + CookieStore cookieStore = new BasicCookieStore(); + ArrayIterate.forEach(request.getCookies(), c -> cookieStore.addCookie(new DEV_ONLY__HttpClientBuilderCookie(c))); + + try (CloseableHttpClient client = (CloseableHttpClient) HttpClientBuilder.getHttpClient(cookieStore)) + { + if (metaDataServerConfiguration == null || metaDataServerConfiguration.getSdlc() == null) + { + throw new EngineException("Please specify the metadataserver.sdlc information in the server configuration"); + } + HttpGet req = new HttpGet("http://" + metaDataServerConfiguration.getSdlc().host + ":" + metaDataServerConfiguration.getSdlc().port + "/api/projects/" + projectId + (isGroupWorkspace ? "/groupWorkspaces/" : "/workspaces/") + workspaceId + "/pureModelContextData"); + try (CloseableHttpResponse res = client.execute(req)) + { + ObjectMapper mapper = ObjectMapperFactory.getNewStandardObjectMapperWithPureProtocolExtensionSupports(); + PureModelContextData pureModelContextData = mapper.readValue(res.getEntity().getContent(), PureModelContextData.class); + return ListIterate.select(pureModelContextData.getElements(), element -> element.getPath().equals(connectionPath)).selectInstancesOf(Connection.class).getAny(); + } + } + catch (Exception e) + { + throw new RuntimeException(e); + } + } + + public Connection getConnectionFromProject(List profiles, String groupId, String artifactId, String versionId, String connectionPath) + { + AlloySDLC sdlcInfo = new AlloySDLC(); + sdlcInfo.groupId = groupId; + sdlcInfo.artifactId = artifactId; + sdlcInfo.version = versionId; + Assert.assertTrue(sdlcInfo.project == null, () -> "Accessing metadata services using project id was demised. Please update AlloySDLC to provide group and artifact IDs"); + Assert.assertTrue(sdlcInfo.groupId != null && sdlcInfo.artifactId != null, () -> "AlloySDLC info must contain and group and artifact IDs to access metadata services"); + PureModelContextData pureModelContextData = SDLCLoader.loadMetadataFromHTTPURL(Lists.mutable.withAll(profiles), LoggingEventType.METADATA_REQUEST_ALLOY_PROJECT_START, LoggingEventType.METADATA_REQUEST_ALLOY_PROJECT_STOP, (isLatestRevision(sdlcInfo)) ? + metaDataServerConfiguration.getAlloy().getBaseUrl() + "/projects/" + sdlcInfo.groupId + "/" + sdlcInfo.artifactId + "/revisions/latest/pureModelContextData" : + metaDataServerConfiguration.getAlloy().getBaseUrl() + "/projects/" + sdlcInfo.groupId + "/" + sdlcInfo.artifactId + "/versions/" + sdlcInfo.version + "/pureModelContextData"); + return ListIterate.select(pureModelContextData.getElements(), element -> element.getPath().equals(connectionPath)).selectInstancesOf(Connection.class).getAny(); + } + + private boolean isLatestRevision(AlloySDLC alloySDLC) + { + return alloySDLC.version == null || alloySDLC.version.equals("none") || alloySDLC.version.equals("master-SNAPSHOT"); + } + + private static class DEV_ONLY__HttpClientBuilderCookie implements Cookie + { + private final javax.servlet.http.Cookie cookie; + + public DEV_ONLY__HttpClientBuilderCookie(javax.servlet.http.Cookie cookie) + { + this.cookie = cookie; + } + + @Override + public String getName() + { + return this.cookie.getName(); + } + + @Override + public String getValue() + { + return this.cookie.getValue(); + } + + @Override + public String getComment() + { + return this.cookie.getComment(); + } + + @Override + public String getCommentURL() + { + return ""; + } + + @Override + public Date getExpiryDate() + { + if (this.cookie.getMaxAge() >= 0) + { + return new Date(System.currentTimeMillis() + this.cookie.getMaxAge() * 1000L); + } + throw new RuntimeException(""); + } + + @Override + public boolean isPersistent() + { + return true; + } + + @Override + public String getDomain() + { + return "localhost"; + } + + @Override + public String getPath() + { + return "/"; + } + + @Override + public int[] getPorts() + { + return new int[]{}; + } + + @Override + public boolean isSecure() + { + return false; + } + + @Override + public int getVersion() + { + return this.cookie.getVersion(); + } + + @Override + public boolean isExpired(Date date) + { + return false; + } + } +} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/DataPushServer.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/DataPushServer.java index 3f939298be9..c9e566ca3d2 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/DataPushServer.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/DataPushServer.java @@ -15,10 +15,32 @@ package org.finos.legend.engine.datapush.server; import io.dropwizard.setup.Bootstrap; -import org.eclipse.collections.api.factory.Lists; -import org.finos.legend.engine.datapush.server.config.DataPushServerConfiguration; +import org.finos.legend.authentication.vault.impl.EnvironmentCredentialVault; +import org.finos.legend.authentication.vault.impl.SystemPropertiesCredentialVault; +import org.finos.legend.connection.AuthenticationMechanism; +import org.finos.legend.connection.Connection; +import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.connection.DatabaseSupport; +import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.IdentityFactory; +import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; +import org.finos.legend.connection.impl.KerberosCredentialExtractor; +import org.finos.legend.connection.impl.KeyPairCredentialBuilder; +import org.finos.legend.connection.impl.RelationalDatabaseType; +import org.finos.legend.connection.impl.SnowflakeConnectionBuilder; +import org.finos.legend.connection.impl.StaticJDBCConnectionBuilder; +import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; +import org.finos.legend.engine.datapush.DataPusher; +import org.finos.legend.engine.datapush.DataPusherProvider; +import org.finos.legend.engine.datapush.impl.SnowflakeWithS3StageDataPusher; +import org.finos.legend.engine.datapush.server.configuration.DataPushServerConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; import org.finos.legend.engine.server.support.server.config.BaseServerConfiguration; import org.finos.legend.server.pac4j.LegendPac4jBundle; +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; public class DataPushServer extends BaseDataPushServer { @@ -33,7 +55,6 @@ public void initialize(Bootstrap bootstrap) { super.initialize(bootstrap); - bootstrap.addBundle(new ConnectionFactoryBundle<>(DataPushServerConfiguration::getConnectionFactoryConfiguration, Lists.mutable.empty())); bootstrap.addBundle(new LegendPac4jBundle<>(BaseServerConfiguration::getPac4jConfiguration)); } @@ -41,4 +62,83 @@ public static void main(String... args) throws Exception { new DataPushServer().run(args); } + + @Override + public LegendEnvironment buildLegendEnvironment(DataPushServerConfiguration configuration) + { + return LegendEnvironment.builder() + .vaults( + new SystemPropertiesCredentialVault(), + new EnvironmentCredentialVault() + ) + .databaseSupports( + DatabaseSupport.builder() + .type(RelationalDatabaseType.POSTGRES) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD).authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class + ).build() + ) + .build(), + DatabaseSupport.builder() + .type(RelationalDatabaseType.SNOWFLAKE) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.KEY_PAIR).authenticationConfigurationTypes( + EncryptedPrivateKeyPairAuthenticationConfiguration.class + ).build() + ) + .build() + ).build(); + } + + @Override + public IdentityFactory buildIdentityFactory(DataPushServerConfiguration configuration, LegendEnvironment environment) + { + return IdentityFactory.builder() + .environment(environment) + .build(); + } + + @Override + public ConnectionFactory buildConnectionFactory(DataPushServerConfiguration configuration, LegendEnvironment environment) + { + return ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders( + new KerberosCredentialExtractor(), + new UserPasswordCredentialBuilder(), + new KeyPairCredentialBuilder() + ) + .connectionBuilders( + new StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword(), + new SnowflakeConnectionBuilder.WithKeyPair() + ) + .build(); + } + + @Override + public DataPusherProvider buildDataPushProvider() + { + return new DataPusherProvider() + { + @Override + public DataPusher getDataPusher(Connection connection) + { + DatabaseType databaseType = connection.getDatabaseSupport().getDatabaseType(); + if (RelationalDatabaseType.SNOWFLAKE.equals(databaseType)) + { + String tableName = "DEMO_DB.SCHEMA1.TABLE1"; + String stageName = "DEMO_DB.SCHEMA1.STAGE1"; + return new SnowflakeWithS3StageDataPusher("legend-dpsh1", null, StaticCredentialsProvider.create( + AwsBasicCredentials.create( + "xxxxx", // NOTE: secret - to be removed when committed + "xxxxx" // NOTE: secret - to be removed when committed + )), tableName, stageName); + } + throw new UnsupportedOperationException("Unsupported database type: " + databaseType.getIdentifier()); + } + }; + } } diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryConfiguration.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/ConnectionFactoryConfiguration.java similarity index 91% rename from legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryConfiguration.java rename to legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/ConnectionFactoryConfiguration.java index f3feddde5b1..e5e22b76bb6 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/ConnectionFactoryConfiguration.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/ConnectionFactoryConfiguration.java @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.engine.datapush.server; +package org.finos.legend.engine.datapush.server.configuration; public class ConnectionFactoryConfiguration { diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/config/DataPushServerConfiguration.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/DataPushServerConfiguration.java similarity index 73% rename from legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/config/DataPushServerConfiguration.java rename to legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/DataPushServerConfiguration.java index 3c0a4fbf1a8..1934f36e667 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/config/DataPushServerConfiguration.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/configuration/DataPushServerConfiguration.java @@ -12,19 +12,26 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.engine.datapush.server.config; +package org.finos.legend.engine.datapush.server.configuration; import com.fasterxml.jackson.annotation.JsonProperty; -import org.finos.legend.engine.datapush.server.ConnectionFactoryConfiguration; +import org.finos.legend.engine.language.pure.modelManager.sdlc.configuration.MetaDataServerConfiguration; import org.finos.legend.engine.server.support.server.config.BaseServerConfiguration; public class DataPushServerConfiguration extends BaseServerConfiguration { @JsonProperty("connection") private ConnectionFactoryConfiguration connectionFactoryConfiguration; + @JsonProperty("metadata-server") + private MetaDataServerConfiguration metadataserver; public ConnectionFactoryConfiguration getConnectionFactoryConfiguration() { return connectionFactoryConfiguration; } + + public MetaDataServerConfiguration getMetadataServerConfiguration() + { + return metadataserver; + } } diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/resources/DataPushResource.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/resources/DataPushResource.java index 2490782efc3..6bffdce91bd 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/resources/DataPushResource.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/main/java/org/finos/legend/engine/datapush/server/resources/DataPushResource.java @@ -15,49 +15,206 @@ package org.finos.legend.engine.datapush.server.resources; import io.swagger.annotations.Api; -import io.swagger.annotations.ApiOperation; +import io.swagger.annotations.ApiParam; +import org.finos.legend.connection.Connection; +import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.connection.IdentityFactory; +import org.finos.legend.connection.IdentitySpecification; +import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.engine.datapush.DataPusher; +import org.finos.legend.engine.datapush.DataPusherProvider; +import org.finos.legend.engine.datapush.data.CSVData; +import org.finos.legend.engine.datapush.server.ConnectionModelLoader; +import org.finos.legend.engine.language.pure.modelManager.sdlc.configuration.MetaDataServerConfiguration; import org.finos.legend.engine.server.support.server.resources.BaseResource; +import org.finos.legend.engine.shared.core.identity.Identity; +import org.finos.legend.engine.shared.core.kerberos.ProfileManagerHelper; +import org.finos.legend.engine.shared.core.operational.errorManagement.EngineException; +import org.finos.legend.engine.shared.core.operational.errorManagement.ExceptionTool; +import org.finos.legend.engine.shared.core.operational.logs.LoggingEventType; +import org.pac4j.core.profile.CommonProfile; +import org.pac4j.core.profile.ProfileManager; +import org.pac4j.jax.rs.annotations.Pac4JProfileManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import javax.servlet.http.HttpServletRequest; import javax.ws.rs.Consumes; +import javax.ws.rs.DefaultValue; import javax.ws.rs.POST; import javax.ws.rs.Path; import javax.ws.rs.PathParam; import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; -import java.io.IOException; +import java.util.List; -@Path("/data/push") +@Path("/data-push") @Api("Data Push") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) public class DataPushResource extends BaseResource { - public DataPushResource() + private static final String TEXT_CSV = "text/csv"; + private static final Logger LOGGER = LoggerFactory.getLogger(DataPushResource.class); + + private final ConnectionModelLoader connectionModelLoader; + private final LegendEnvironment environment; + private final IdentityFactory identityFactory; + private final ConnectionFactory connectionFactory; + private final DataPusherProvider dataPusherProvider; + + public DataPushResource(MetaDataServerConfiguration metadataserver, LegendEnvironment environment, IdentityFactory identityFactory, ConnectionFactory connectionFactory, DataPusherProvider dataPusherProvider) { + this.environment = environment; + this.identityFactory = identityFactory; + this.connectionFactory = connectionFactory; + this.dataPusherProvider = dataPusherProvider; + this.connectionModelLoader = new ConnectionModelLoader(metadataserver); } - @Path("/location/{location}/datastore/{datastore}/dataset/{dataset}") + @Path("/push/{groupId}/{artifactId}/{versionId}/{connectionPath}") @POST - @ApiOperation("Push data") - public Response push(@PathParam("location") String location, @PathParam("datastore") String datastore, @PathParam("dataset") String dataset) throws IOException + @Consumes({ + // TODO: content type will drive how we interpret the data, right now + // we only support CSV + MediaType.TEXT_PLAIN, + MediaType.TEXT_XML, + MediaType.APPLICATION_JSON, + TEXT_CSV + }) + @Produces(MediaType.APPLICATION_JSON) + public Response pushData( + @PathParam("groupId") String groupId, + @PathParam("artifactId") String artifactId, + @PathParam("versionId") String versionId, + @PathParam("connectionPath") String connectionPath, + String data, + @Context HttpServletRequest request, + @ApiParam(hidden = true) @Pac4JProfileManager ProfileManager profileManager + ) { - return executeWithLogging( - "pushing data\"", - () -> Response.ok().entity(this.pushData(location, datastore, dataset)).build() + List profiles = ProfileManagerHelper.extractProfiles(profileManager); + Identity identity = this.identityFactory.createIdentity( + IdentitySpecification.builder().profiles(profiles).build() ); + org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection connection = this.connectionModelLoader.getConnectionFromProject(profiles, groupId, artifactId, versionId, connectionPath); + + CSVData csvData = new CSVData(); + csvData.value = data; + + try + { + this.pushCSVData(identity, connection, csvData); + return Response.noContent().build(); + } + catch (Exception exception) + { + LOGGER.error("Can't push data:\n", exception); + return handleException(profiles, exception); + } } - private String pushData(String location, String datastore, String dataset) + @Path("/pushDev/{projectId}/{workspaceId}/{connectionPath}") + @POST + @Consumes({ + // TODO: content type will drive how we interpret the data, right now + // we only support CSV + MediaType.TEXT_PLAIN, + MediaType.TEXT_XML, + MediaType.APPLICATION_JSON, + TEXT_CSV + }) + @Produces(MediaType.APPLICATION_JSON) + public Response pushData_Dev( + @PathParam("projectId") String projectId, + @PathParam("workspaceId") String workspaceId, + @PathParam("connectionPath") String connectionPath, + @QueryParam("isGroupWorkspace") @DefaultValue("false") boolean isGroupWorkspace, + String data, + @Context HttpServletRequest request, + @ApiParam(hidden = true) @Pac4JProfileManager ProfileManager profileManager + ) { + List profiles = ProfileManagerHelper.extractProfiles(profileManager); + Identity identity = this.identityFactory.createIdentity( + IdentitySpecification.builder().profiles(profiles).build() + ); + org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection connection = this.connectionModelLoader.getConnectionFromSDLCWorkspace(request, projectId, workspaceId, isGroupWorkspace, connectionPath); + + CSVData csvData = new CSVData(); + csvData.value = data; + try { - // TODO - actually push the data - return "ok"; + this.pushCSVData(identity, connection, csvData); + return Response.noContent().build(); + } + catch (Exception exception) + { + LOGGER.error("Can't push data:\n", exception); + return handleException(profiles, exception); + } + } + + private Response handleException(List profiles, Exception exception) + { + Response.Status status = exception instanceof EngineException ? Response.Status.BAD_REQUEST : Response.Status.INTERNAL_SERVER_ERROR; + return ExceptionTool.exceptionManager(exception, LoggingEventType.ERROR_MANAGEMENT_ERROR, status, profiles); + } + + private void pushCSVData(Identity identity, org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection connectionProtocol, CSVData csvData) + { + Connection connection = Connection.builder().fromProtocol(connectionProtocol, this.environment).build(); + try + { + DataPusher dataPusher = this.dataPusherProvider.getDataPusher(connection); + dataPusher.configure(this.connectionFactory); + dataPusher.writeCSV(identity, connection, csvData); } catch (Exception e) { throw new RuntimeException(e); } } + + // ------------------------ DEBUG ----------------------- + // TO BE REMOVED when we stabilize the API and models + + @Path("/pushDev/debug") + @POST + @Consumes(MediaType.APPLICATION_JSON) + @Produces(MediaType.APPLICATION_JSON) + public Response TEMPORARY__pushData_Debug( + DebugInput input, + @Context HttpServletRequest request, + @ApiParam(hidden = true) @Pac4JProfileManager ProfileManager profileManager + ) + { + List profiles = ProfileManagerHelper.extractProfiles(profileManager); + Identity identity = this.identityFactory.createIdentity( + IdentitySpecification.builder().profiles(profiles).build() + ); + CSVData csvData = new CSVData(); + csvData.value = input.data; + + try + { + this.pushCSVData(identity, input.connection, csvData); + return Response.noContent().build(); + } + catch (Exception exception) + { + LOGGER.error("Can't push data:\n", exception); + return handleException(profiles, exception); + } + } + + public static class DebugInput + { + public org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.Connection connection; + public String data; + } } diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/TestDataPushResource.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/TestDataPushResource.java deleted file mode 100644 index d0187255cf0..00000000000 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/TestDataPushResource.java +++ /dev/null @@ -1,43 +0,0 @@ -// Copyright 2020 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.engine.datapush.server; - -import org.apache.http.client.HttpResponseException; -import org.finos.legend.engine.datapush.server.test.AbstractDataPushServerResourceTest; -import org.junit.Test; - -import javax.ws.rs.client.Entity; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; - -import static org.junit.Assert.assertEquals; - -public class TestDataPushResource extends AbstractDataPushServerResourceTest -{ - @Test - public void testPost() throws HttpResponseException - { - Response response = this.clientFor("/api/data/push/location/LOCATION/datastore/STORE/dataset/DATASET").request().post(Entity.entity("{}", MediaType.APPLICATION_JSON_TYPE)); - - String responseText = response.readEntity(String.class); - - if (response.getStatus() != 200) - { - throw new HttpResponseException(response.getStatus(), "Error during http call with status: " + response.getStatus() + " , entity: " + responseText); - } - - assertEquals("ok", responseText); - } -} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/resources/DataPushTestResource.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/resources/DataPushTestResource.java deleted file mode 100644 index 6053bf0afe3..00000000000 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/resources/DataPushTestResource.java +++ /dev/null @@ -1,61 +0,0 @@ -// Copyright 2020 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.engine.datapush.server.resources; - -import io.swagger.annotations.Api; -import io.swagger.annotations.ApiOperation; -import org.finos.legend.engine.server.support.server.resources.BaseResource; - -import javax.ws.rs.Consumes; -import javax.ws.rs.GET; -import javax.ws.rs.POST; -import javax.ws.rs.Path; -import javax.ws.rs.Produces; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; - -@Path("/tests/data/push") -@Api("Data Push - tests") -@Consumes(MediaType.APPLICATION_JSON) -@Produces(MediaType.APPLICATION_JSON) -public class DataPushTestResource extends BaseResource -{ - public DataPushTestResource() - { - - } - - @Path("/postSomething") - @POST - @ApiOperation("Test POST") - public Response postSomething(Object object) - { - return executeWithLogging( - "testing post \"", - () -> Response.ok().entity("{\"post\" : \"ok\"}").build() - ); - } - - @Path("/getSomething") - @GET - @ApiOperation("Test GET") - public Response getSomething() - { - return executeWithLogging( - "testing post \"", - () -> Response.ok().entity("{\"get\" : \"ok\"}").build() - ); - } -} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/AbstractDataPushServerResourceTest.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/AbstractDataPushServerResourceTest.java index f83f18063dc..ef44e5f161f 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/AbstractDataPushServerResourceTest.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/AbstractDataPushServerResourceTest.java @@ -16,7 +16,7 @@ import io.dropwizard.testing.ResourceHelpers; import io.dropwizard.testing.junit.DropwizardAppRule; -import org.finos.legend.engine.datapush.server.config.DataPushServerConfiguration; +import org.finos.legend.engine.datapush.server.configuration.DataPushServerConfiguration; import org.junit.Before; import org.junit.ClassRule; import org.junit.Rule; @@ -50,6 +50,11 @@ private void configureClient() this.client.target(getServerUrl()).request().get(); } + protected DataPushServerForTest getApplicationInstance() + { + return APP_RULE.getApplication(); + } + protected WebTarget clientFor(String url) { return this.client.target(getServerUrl(url)); diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/DataPushServerForTest.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/DataPushServerForTest.java index 212c55dc8b6..70ab113b50c 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/DataPushServerForTest.java +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/DataPushServerForTest.java @@ -14,32 +14,71 @@ package org.finos.legend.engine.datapush.server.test; -import io.dropwizard.setup.Environment; -import org.finos.legend.engine.datapush.server.BaseDataPushServer; -import org.finos.legend.engine.datapush.server.config.DataPushServerConfiguration; -import org.finos.legend.engine.datapush.server.resources.DataPushTestResource; -import org.finos.legend.engine.server.support.server.BaseServer; +import org.finos.legend.authentication.vault.impl.EnvironmentCredentialVault; +import org.finos.legend.authentication.vault.impl.SystemPropertiesCredentialVault; +import org.finos.legend.connection.AuthenticationMechanism; +import org.finos.legend.connection.ConnectionFactory; +import org.finos.legend.connection.DatabaseSupport; +import org.finos.legend.connection.LegendEnvironment; +import org.finos.legend.connection.impl.CoreAuthenticationMechanismType; +import org.finos.legend.connection.impl.KerberosCredentialExtractor; +import org.finos.legend.connection.impl.RelationalDatabaseType; +import org.finos.legend.connection.impl.StaticJDBCConnectionBuilder; +import org.finos.legend.connection.impl.UserPasswordCredentialBuilder; +import org.finos.legend.engine.datapush.server.DataPushServer; +import org.finos.legend.engine.datapush.server.configuration.DataPushServerConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.UserPasswordAuthenticationConfiguration; -public class DataPushServerForTest extends BaseDataPushServer +public class DataPushServerForTest extends DataPushServer { public DataPushServerForTest() { } + public static void main(String... args) throws Exception + { + new DataPushServerForTest().run(args); + } + @Override - protected void configureServerExtension(DataPushServerConfiguration configuration, Environment environment) + public LegendEnvironment buildLegendEnvironment(DataPushServerConfiguration configuration) { - environment.jersey().register(new DataPushTestResource()); + return LegendEnvironment.builder() + .vaults( + new SystemPropertiesCredentialVault(), + new EnvironmentCredentialVault() + ) + .databaseSupports( + DatabaseSupport.builder() + .type(RelationalDatabaseType.POSTGRES) + .authenticationMechanisms( + AuthenticationMechanism.builder() + .type(CoreAuthenticationMechanismType.USER_PASSWORD) + .authenticationConfigurationTypes( + UserPasswordAuthenticationConfiguration.class + ).build() + ) + .build() + ).build(); } @Override - protected BaseServer.ServerPlatformInfo newServerPlatformInfo() + public ConnectionFactory buildConnectionFactory(DataPushServerConfiguration configuration, LegendEnvironment environment) { - return new ServerPlatformInfo(null, null, null); + return ConnectionFactory.builder() + .environment(this.environment) + .credentialBuilders( + new KerberosCredentialExtractor(), + new UserPasswordCredentialBuilder() + ) + .connectionBuilders( + new StaticJDBCConnectionBuilder.WithPlaintextUsernamePassword() + ) + .build(); } - public static void main(String... args) throws Exception + public LegendEnvironment getEnvironment() { - new DataPushServerForTest().run(args); + return environment; } } \ No newline at end of file diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/MinIOS3TestContainerWrapper.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/MinIOS3TestContainerWrapper.java new file mode 100644 index 00000000000..de1e4d3f2af --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/MinIOS3TestContainerWrapper.java @@ -0,0 +1,86 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +package org.finos.legend.engine.datapush.server.test; + +import org.junit.Assert; +import org.testcontainers.DockerClientFactory; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.Network; + +public class MinIOS3TestContainerWrapper +{ + private Network network; + private GenericContainer minioContainer; + + private MinIOS3TestContainerWrapper() + { + } + + public static MinIOS3TestContainerWrapper build() + { + return new MinIOS3TestContainerWrapper(); + } + + public void start() throws Exception + { + Assert.assertTrue("Docker environment not properly setup", DockerClientFactory.instance().isDockerAvailable()); + + this.network = Network.newNetwork(); + this.initMinio(); + } + + private void initMinio() + { +// GenericContainer mc = new GenericContainer<>("minio/mc:RELEASE.2023-08-08T17-23-59Z") +// .withNetwork(this.network) +// .withEnv("AWS_ACCESS_KEY_ID", "admin") +// .withEnv("AWS_SECRET_ACCESS_KEY", "password") +// .withEnv("AWS_REGION", "us-east-1") +// .withCreateContainerCmdModifier(x -> x.withEntrypoint( +// "/bin/sh", +// "-c", +// "until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; " + +// "/usr/bin/mc rm -r --force minio/" + this.getBucketName() + "; " + +// "/usr/bin/mc mb minio/" + this.getBucketName() + "; " + +// "/usr/bin/mc policy set public minio/" + this.getBucketName() + "; " + +// "tail -f /dev/null" +// ) +// ); + + this.minioContainer = new GenericContainer<>("minio/minio:RELEASE.2023-08-09T23-30-22Z") + .withNetwork(this.network) + .withNetworkAliases("minio", "warehouse.minio") + .withEnv("MINIO_ROOT_USER", "admin") + .withEnv("MINIO_ROOT_PASSWORD", "password") + .withEnv("MINIO_DOMAIN", "minio") + .withExposedPorts(9000, 9001) + .withCommand("server", "/data", "--console-address", ":9001") +// .dependsOn(mc) + ; + + } + + public void stop() throws Exception + { + try ( + Network ignored = this.network; + AutoCloseable ignored1 = this.minioContainer; + ) + { + // done + } + } +} \ No newline at end of file diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/TestDataPushServer.java b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/TestDataPushServer.java new file mode 100644 index 00000000000..19a78f80f8f --- /dev/null +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/java/org/finos/legend/engine/datapush/server/test/TestDataPushServer.java @@ -0,0 +1,92 @@ +// Copyright 2020 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.engine.datapush.server.test; + +import org.finos.legend.connection.PostgresTestContainerWrapper; +import org.junit.After; +import org.junit.Assume; +import org.junit.Before; +import org.junit.Ignore; +import org.junit.Test; + +import javax.ws.rs.client.Entity; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response; + +import static org.junit.Assert.assertEquals; + +public class TestDataPushServer extends AbstractDataPushServerResourceTest +{ + private MinIOS3TestContainerWrapper minioContainer; + + private PostgresTestContainerWrapper postgresContainer; + + @Before + public void setUp() + { + try + { + // TODO: use @ClassRule + this.minioContainer = MinIOS3TestContainerWrapper.build(); + this.minioContainer.start(); + } + catch (Exception e) + { + Assume.assumeTrue("Can't start MinIO", false); + } + + try + { + // TODO: use @ClassRule + this.postgresContainer = PostgresTestContainerWrapper.build(); + this.postgresContainer.start(); + } + catch (Exception e) + { + Assume.assumeTrue("Can't start PostgreSQLContainer", false); + } + } + + @After + public void tearDown() throws Exception + { + if (this.minioContainer != null) + { + this.minioContainer.stop(); + } + + if (this.postgresContainer != null) + { + this.postgresContainer.stop(); + } + + System.clearProperty("passwordRef"); + } + + @Test + @Ignore + public void test() + { + Response response = this.clientFor("/api/data-push/stage").request().post(Entity.entity("{\n" + + " \"_type\": \"sql\",\n" + + " \"statements\": [\"Drop table if exists FirmTable;\n" + + "Create Table FirmTable(id INT, Legal_Name VARCHAR(200));\n" + + "Insert into FirmTable (id, Legal_Name) values (1, 'FINOS');\"]\n" + + "}", MediaType.APPLICATION_JSON_TYPE)); + String responseText = response.readEntity(String.class); + + assertEquals("ok", responseText); + } +} diff --git a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/resources/config-test.yaml b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/resources/config-test.yaml index 0e02a41dfc4..41376c8a8dc 100644 --- a/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/resources/config-test.yaml +++ b/legend-engine-xts-data-push/legend-engine-xt-data-push-server/src/test/resources/config-test.yaml @@ -35,6 +35,17 @@ filterPriorities: org.pac4j.j2e.filter.SecurityFilter: 3 CORS: 4 +metadata-server: + pure: + host: 127.0.0.1 + port: 8080 + alloy: + host: 127.0.0.1 + port: 6200 + prefix: "/depot/api" + sdlc: + host: localhost + port: 6100 logging: # Change this to affect library class logging @@ -43,8 +54,13 @@ logging: - type: console logFormat: "%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p [%thread] %c - %m%n" +pac4j: + callbackPrefix: /api/pac4j + clients: + - org.pac4j.core.client.direct.AnonymousClient: {} + swagger: resourcePackage: org.finos.legend.engine.datapush.server.resources - title: Legend Data Push + title: Legend Data Push (Test) version: local-snapshot schemes: [] \ No newline at end of file diff --git a/legend-engine-xts-hostedService/legend-engine-xt-hostedService-generation/pom.xml b/legend-engine-xts-hostedService/legend-engine-xt-hostedService-generation/pom.xml index f86fcc91c82..ca33ecc7591 100644 --- a/legend-engine-xts-hostedService/legend-engine-xt-hostedService-generation/pom.xml +++ b/legend-engine-xts-hostedService/legend-engine-xt-hostedService-generation/pom.xml @@ -134,11 +134,6 @@ org.finos.legend.engine legend-engine-language-pure-dsl-generation - - org.finos.legend.engine - legend-engine-language-pure-dsl-generation - - com.fasterxml.jackson.core jackson-databind diff --git a/legend-engine-xts-iceberg/legend-engine-xt-iceberg-test-support/pom.xml b/legend-engine-xts-iceberg/legend-engine-xt-iceberg-test-support/pom.xml index 6a5fb022481..6ef693b35e1 100644 --- a/legend-engine-xts-iceberg/legend-engine-xt-iceberg-test-support/pom.xml +++ b/legend-engine-xts-iceberg/legend-engine-xt-iceberg-test-support/pom.xml @@ -122,13 +122,6 @@ io.minio minio - 8.5.5 - - - org.jetbrains - * - - diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/pom.xml b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/pom.xml index 78bffc2431f..4ad10b16753 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/pom.xml +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/pom.xml @@ -35,7 +35,11 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory + + + org.finos.legend.engine + legend-engine-xt-connection-protocol org.finos.legend.engine diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/HACKY__RelationalDatabaseConnectionAdapter.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/HACKY__RelationalDatabaseConnectionAdapter.java index 172bdd2a8d2..a924ad9b89a 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/HACKY__RelationalDatabaseConnectionAdapter.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/HACKY__RelationalDatabaseConnectionAdapter.java @@ -14,7 +14,7 @@ package org.finos.legend.connection; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.RelationalDatabaseConnection; import org.finos.legend.engine.shared.core.identity.Identity; @@ -29,12 +29,12 @@ public interface HACKY__RelationalDatabaseConnectionAdapter class ConnectionFactoryMaterial { - public final StoreInstance storeInstance; + public final Connection connection; public final AuthenticationConfiguration authenticationConfiguration; - public ConnectionFactoryMaterial(StoreInstance storeInstance, AuthenticationConfiguration authenticationConfiguration) + public ConnectionFactoryMaterial(Connection connection, AuthenticationConfiguration authenticationConfiguration) { - this.storeInstance = storeInstance; + this.connection = connection; this.authenticationConfiguration = authenticationConfiguration; } } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseManager.java similarity index 95% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseManager.java index d7320f0a503..7ff111b7394 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/DatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseManager.java @@ -17,7 +17,7 @@ import java.util.List; import java.util.Properties; -public interface DatabaseManager +public interface RelationalDatabaseManager { List getIds(); diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseStoreSupport.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseStoreSupport.java deleted file mode 100644 index b81cb654a97..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/RelationalDatabaseStoreSupport.java +++ /dev/null @@ -1,101 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -import org.eclipse.collections.api.factory.Lists; - -import java.util.List; -import java.util.Objects; - -public class RelationalDatabaseStoreSupport extends StoreSupport -{ - private final Database database; - - private RelationalDatabaseStoreSupport(String identifier, Database database, List authenticationMechanismConfigurations) - { - super(identifier, authenticationMechanismConfigurations); - this.database = Objects.requireNonNull(database, "Relational database store support database type is missing"); - } - - public Database getDatabase() - { - return database; - } - - public static RelationalDatabaseStoreSupport cast(StoreSupport storeSupport) - { - return cast(storeSupport, null); - } - - public static RelationalDatabaseStoreSupport cast(StoreSupport storeSupport, Database database) - { - if (!(storeSupport instanceof RelationalDatabaseStoreSupport)) - { - throw new RuntimeException("Expected store support for relational databases"); - } - RelationalDatabaseStoreSupport relationalDatabaseStoreSupport = (RelationalDatabaseStoreSupport) storeSupport; - if (database != null && !database.equals(relationalDatabaseStoreSupport.getDatabase())) - { - - throw new RuntimeException(String.format("Expected relational database store support for '%s'", database.getLabel())); - } - return relationalDatabaseStoreSupport; - } - - public static class Builder - { - private final Database database; - private String identifier; - private final List authenticationMechanismConfigurations = Lists.mutable.empty(); - - public Builder(Database database) - { - this.database = database; - } - - public Builder withIdentifier(String identifier) - { - this.identifier = identifier; - return this; - } - - public Builder withAuthenticationMechanismConfiguration(AuthenticationMechanismConfiguration authenticationMechanismConfiguration) - { - this.authenticationMechanismConfigurations.add(authenticationMechanismConfiguration); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(List authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(authenticationMechanismConfigurations); - return this; - } - - public Builder withAuthenticationMechanismConfigurations(AuthenticationMechanismConfiguration... authenticationMechanismConfigurations) - { - this.authenticationMechanismConfigurations.addAll(Lists.mutable.of(authenticationMechanismConfigurations)); - return this; - } - - public RelationalDatabaseStoreSupport build() - { - return new RelationalDatabaseStoreSupport( - this.identifier, - this.database, - this.authenticationMechanismConfigurations - ); - } - } -} diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2DatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2RelationalDatabaseManager.java similarity index 83% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2DatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2RelationalDatabaseManager.java index 44fdf427975..6a44059a547 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2DatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/H2RelationalDatabaseManager.java @@ -15,18 +15,17 @@ package org.finos.legend.connection.impl; import org.eclipse.collections.impl.factory.Lists; -import org.finos.legend.connection.DatabaseManager; -import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.RelationalDatabaseManager; import java.util.List; import java.util.Properties; -public class H2DatabaseManager implements DatabaseManager +public class H2RelationalDatabaseManager implements RelationalDatabaseManager { @Override public List getIds() { - return Lists.mutable.with(DatabaseType.H2.name()); + return Lists.mutable.with(RelationalDatabaseType.H2.name()); } @Override diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/JDBCConnectionBuilder.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionBuilder.java similarity index 89% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/JDBCConnectionBuilder.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionBuilder.java index 0c09de0784e..d380bb07a9c 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/JDBCConnectionBuilder.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionBuilder.java @@ -12,10 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection; +package org.finos.legend.connection.impl; +import org.finos.legend.connection.ConnectionBuilder; import org.finos.legend.connection.impl.JDBCConnectionManager; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import java.sql.Connection; diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionManager.java index 8b96bf0aa7a..3fd59cf54fd 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/JDBCConnectionManager.java @@ -21,18 +21,17 @@ import org.eclipse.collections.impl.map.mutable.ConcurrentHashMap; import org.finos.legend.connection.Authenticator; import org.finos.legend.connection.ConnectionManager; -import org.finos.legend.connection.Database; -import org.finos.legend.connection.DatabaseManager; +import org.finos.legend.connection.DatabaseType; +import org.finos.legend.connection.RelationalDatabaseManager; import org.finos.legend.connection.LegendEnvironment; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.protocol.AuthenticationConfiguration; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.connection.Connection; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.AuthenticationConfiguration; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import javax.sql.DataSource; import java.io.PrintWriter; -import java.sql.Connection; import java.sql.Driver; import java.sql.SQLException; import java.sql.SQLFeatureNotSupportedException; @@ -50,7 +49,7 @@ public class JDBCConnectionManager implements ConnectionManager private static final int HIKARICP_DEFAULT_MAX_POOL_SIZE = 100; private static final int HIKARICP_DEFAULT_MIN_POOL_SIZE = 0; - private static final ConcurrentHashMap managerByName = ConcurrentHashMap.newMap(); + private static final ConcurrentHashMap managerByName = ConcurrentHashMap.newMap(); private static final AtomicBoolean isInitialized = new AtomicBoolean(); private static JDBCConnectionManager INSTANCE; @@ -79,7 +78,7 @@ private static void setup() { if (!isInitialized.get()) { - for (DatabaseManager manager : ServiceLoader.load(DatabaseManager.class)) + for (RelationalDatabaseManager manager : ServiceLoader.load(RelationalDatabaseManager.class)) { manager.getIds().forEach(i -> managerByName.put(i, manager)); } @@ -95,24 +94,24 @@ public void initialize(LegendEnvironment environment) JDBCConnectionManager.setup(); } - public Connection getConnection(Database database, - String host, - int port, - String databaseName, - Properties connectionProperties, - ConnectionPoolConfig connectionPoolConfig, - Function authenticationPropertiesSupplier, - Authenticator authenticator, - Identity identity + public java.sql.Connection getConnection(DatabaseType databaseType, + String host, + int port, + String databaseName, + Properties connectionProperties, + ConnectionPoolConfig connectionPoolConfig, + Function authenticationPropertiesSupplier, + Authenticator authenticator, + Identity identity ) throws SQLException { - StoreInstance storeInstance = authenticator.getStoreInstance(); - ConnectionSpecification connectionSpecification = storeInstance.getConnectionSpecification(); + Connection connection = authenticator.getConnection(); + ConnectionSpecification connectionSpecification = connection.getConnectionSpecification(); AuthenticationConfiguration authenticationConfiguration = authenticator.getAuthenticationConfiguration(); String poolName = getPoolName(identity, connectionSpecification, authenticationConfiguration); // TODO: @akphi - this is simplistic, we need to handle concurrency and errors - Supplier dataSourceSupplier = () -> this.buildDataSource(database, host, port, databaseName, connectionProperties, connectionPoolConfig, authenticationPropertiesSupplier, authenticator, identity); + Supplier dataSourceSupplier = () -> this.buildDataSource(databaseType, host, port, databaseName, connectionProperties, connectionPoolConfig, authenticationPropertiesSupplier, authenticator, identity); Function0 connectionPoolSupplier = () -> new ConnectionPool(dataSourceSupplier.get()); ConnectionPool connectionPool = this.poolIndex.getIfAbsentPut(poolName, connectionPoolSupplier); @@ -147,7 +146,7 @@ public Connection getConnection(Database database, } protected HikariDataSource buildDataSource( - Database database, + DatabaseType databaseType, String host, int port, String databaseName, @@ -158,16 +157,16 @@ protected HikariDataSource buildDataSource( Identity identity ) { - StoreInstance storeInstance = authenticator.getStoreInstance(); - ConnectionSpecification connectionSpecification = storeInstance.getConnectionSpecification(); + Connection connection = authenticator.getConnection(); + ConnectionSpecification connectionSpecification = connection.getConnectionSpecification(); AuthenticationConfiguration authenticationConfiguration = authenticator.getAuthenticationConfiguration(); - DatabaseManager databaseManager = getManagerForDatabase(database); + RelationalDatabaseManager relationalDatabaseManager = getManagerForDatabase(databaseType); - String jdbcUrl = databaseManager.buildURL(host, port, databaseName, connectionProperties); + String jdbcUrl = relationalDatabaseManager.buildURL(host, port, databaseName, connectionProperties); String poolName = getPoolName(identity, connectionSpecification, authenticationConfiguration); HikariConfig jdbcConfig = new HikariConfig(); - jdbcConfig.setDriverClassName(databaseManager.getDriver()); + jdbcConfig.setDriverClassName(relationalDatabaseManager.getDriver()); jdbcConfig.setPoolName(poolName); jdbcConfig.setJdbcUrl(jdbcUrl); @@ -184,7 +183,7 @@ protected HikariDataSource buildDataSource( jdbcConfig.addDataSourceProperty("prepStmtCacheSqlLimit", 0); jdbcConfig.addDataSourceProperty("useServerPrepStmts", false); - jdbcConfig.setDataSource(new DataSourceWrapper(jdbcUrl, connectionProperties, databaseManager, authenticationPropertiesSupplier, authenticator, identity)); + jdbcConfig.setDataSource(new DataSourceWrapper(jdbcUrl, connectionProperties, relationalDatabaseManager, authenticationPropertiesSupplier, authenticator, identity)); return new HikariDataSource(jdbcConfig); } @@ -213,16 +212,16 @@ public static String getPoolName(Identity identity, ConnectionSpecification conn ); } - private static DatabaseManager getManagerForDatabase(Database database) + private static RelationalDatabaseManager getManagerForDatabase(DatabaseType databaseType) { if (!isInitialized.get()) { throw new IllegalStateException("JDBC connection manager has not been configured properly"); } - DatabaseManager manager = managerByName.get(database.getLabel()); + RelationalDatabaseManager manager = managerByName.get(databaseType.getIdentifier()); if (manager == null) { - throw new RuntimeException(String.format("Can't find any matching managers for database type '%s'", database.getLabel())); + throw new RuntimeException(String.format("Can't find any matching managers for database type '%s'", databaseType.getIdentifier())); } return manager; } @@ -334,7 +333,7 @@ private static class DataSourceWrapper implements DataSource public DataSourceWrapper( String url, Properties connectionProperties, - DatabaseManager databaseManager, + RelationalDatabaseManager relationalDatabaseManager, Function authenticationPropertiesSupplier, Authenticator authenticator, Identity identity @@ -344,7 +343,7 @@ public DataSourceWrapper( this.connectionProperties = connectionProperties; try { - this.driver = (Driver) Class.forName(databaseManager.getDriver()).getDeclaredConstructor().newInstance(); + this.driver = (Driver) Class.forName(relationalDatabaseManager.getDriver()).getDeclaredConstructor().newInstance(); } catch (Exception e) { @@ -356,7 +355,7 @@ public DataSourceWrapper( } @Override - public Connection getConnection() throws SQLException + public java.sql.Connection getConnection() throws SQLException { Properties properties = new Properties(); properties.putAll(this.connectionProperties); @@ -377,7 +376,7 @@ public Connection getConnection() throws SQLException } @Override - public Connection getConnection(String username, String password) throws SQLException + public java.sql.Connection getConnection(String username, String password) throws SQLException { throw new RuntimeException(); } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalConnectionExtension.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalConnectionExtension.java new file mode 100644 index 00000000000..138e56e5506 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalConnectionExtension.java @@ -0,0 +1,31 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection.impl; + +import org.eclipse.collections.api.factory.Lists; +import org.finos.legend.connection.AuthenticationMechanismType; +import org.finos.legend.connection.ConnectionExtension; +import org.finos.legend.connection.DatabaseType; + +import java.util.List; + +public class RelationalConnectionExtension implements ConnectionExtension +{ + @Override + public List getExtraDatabaseTypes() + { + return Lists.mutable.of(RelationalDatabaseType.values()); + } +} diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalDatabaseType.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalDatabaseType.java new file mode 100644 index 00000000000..e0cdd1811f8 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/RelationalDatabaseType.java @@ -0,0 +1,38 @@ +// Copyright 2023 Goldman Sachs +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package org.finos.legend.connection.impl; + +import org.finos.legend.connection.DatabaseType; + +public enum RelationalDatabaseType implements DatabaseType +{ + H2("H2"), + POSTGRES("Postgres"), + BIG_QUERY("BigQuery"), + SNOWFLAKE("Snowflake"); + + private final String identifier; + + private RelationalDatabaseType(String identifier) + { + this.identifier = identifier; + } + + @Override + public String getIdentifier() + { + return this.identifier; + } +} diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/StaticJDBCConnectionBuilder.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/StaticJDBCConnectionBuilder.java index 1cbe12abf4f..6d12df9cdae 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/StaticJDBCConnectionBuilder.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/impl/StaticJDBCConnectionBuilder.java @@ -15,15 +15,13 @@ package org.finos.legend.connection.impl; import org.finos.legend.connection.Authenticator; -import org.finos.legend.connection.JDBCConnectionBuilder; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.protocol.StaticJDBCConnectionSpecification; +import org.finos.legend.connection.Connection; +import org.finos.legend.connection.DatabaseSupport; +import org.finos.legend.engine.protocol.pure.v1.model.connection.StaticJDBCConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import org.finos.legend.engine.shared.core.identity.credential.PlaintextUserPasswordCredential; -import java.sql.Connection; import java.util.Properties; import java.util.function.Function; @@ -31,10 +29,10 @@ public class StaticJDBCConnectionBuilder { public static class WithPlaintextUsernamePassword extends JDBCConnectionBuilder { - public Connection getConnection(StaticJDBCConnectionSpecification connectionSpecification, Authenticator authenticator, Identity identity) throws Exception + public java.sql.Connection getConnection(StaticJDBCConnectionSpecification connectionSpecification, Authenticator authenticator, Identity identity) throws Exception { - StoreInstance storeInstance = authenticator.getStoreInstance(); - RelationalDatabaseStoreSupport storeSupport = RelationalDatabaseStoreSupport.cast(storeInstance.getStoreSupport()); + Connection connection = authenticator.getConnection(); + DatabaseSupport databaseSupport = connection.getDatabaseSupport(); Properties connectionProperties = new Properties(); Function authenticationPropertiesSupplier = cred -> @@ -46,7 +44,7 @@ public Connection getConnection(StaticJDBCConnectionSpecification connectionSpec return properties; }; - return this.getConnectionManager().getConnection(storeSupport.getDatabase(), connectionSpecification.host, connectionSpecification.port, connectionSpecification.databaseName, connectionProperties, this.getConnectionPoolConfig(), authenticationPropertiesSupplier, authenticator, identity); + return this.getConnectionManager().getConnection(databaseSupport.getDatabaseType(), connectionSpecification.host, connectionSpecification.port, connectionSpecification.databaseName, connectionProperties, this.getConnectionPoolConfig(), authenticationPropertiesSupplier, authenticator, identity); } } } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.ConnectionExtension b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.ConnectionExtension new file mode 100644 index 00000000000..460c36f956e --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.ConnectionExtension @@ -0,0 +1 @@ +org.finos.legend.connection.impl.RelationalConnectionExtension \ No newline at end of file diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager deleted file mode 100644 index 069422f1bf1..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager +++ /dev/null @@ -1 +0,0 @@ -org.finos.legend.connection.impl.H2DatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager new file mode 100644 index 00000000000..8a0c42480c5 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager @@ -0,0 +1 @@ +org.finos.legend.connection.impl.H2RelationalDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLDatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLRelationalDatabaseManager.java similarity index 90% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLDatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLRelationalDatabaseManager.java index a9a81b97815..a4dda5f09e3 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLDatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/java/org/finos/legend/connection/jdbc/driver/MemSQLRelationalDatabaseManager.java @@ -15,13 +15,13 @@ package org.finos.legend.connection.jdbc.driver; import org.eclipse.collections.impl.factory.Lists; -import org.finos.legend.connection.DatabaseManager; +import org.finos.legend.connection.RelationalDatabaseManager; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType; import java.util.List; import java.util.Properties; -public class MemSQLDatabaseManager implements DatabaseManager +public class MemSQLRelationalDatabaseManager implements RelationalDatabaseManager { @Override public List getIds() diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager deleted file mode 100644 index 29c0a912ba4..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager +++ /dev/null @@ -1 +0,0 @@ -org.finos.legend.connection.jdbc.driver.MemSQLDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager new file mode 100644 index 00000000000..e83f44fb392 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-memsql/legend-engine-xt-relationalStore-memsql-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager @@ -0,0 +1 @@ +org.finos.legend.connection.jdbc.driver.MemSQLRelationalDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresDatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresRelationalDatabaseManager.java similarity index 81% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresDatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresRelationalDatabaseManager.java index 26565642f82..12947ba478c 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresDatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/java/org/finos/legend/connection/impl/PostgresRelationalDatabaseManager.java @@ -15,18 +15,17 @@ package org.finos.legend.connection.impl; import org.eclipse.collections.impl.factory.Lists; -import org.finos.legend.connection.DatabaseType; -import org.finos.legend.connection.DatabaseManager; +import org.finos.legend.connection.RelationalDatabaseManager; import java.util.List; import java.util.Properties; -public class PostgresDatabaseManager implements DatabaseManager +public class PostgresRelationalDatabaseManager implements RelationalDatabaseManager { @Override public List getIds() { - return Lists.mutable.with("PostgreSQL", DatabaseType.POSTGRES.getLabel()); + return Lists.mutable.with("PostgreSQL", RelationalDatabaseType.POSTGRES.getIdentifier()); } @Override diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager deleted file mode 100644 index 29f38d7a5c6..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager +++ /dev/null @@ -1 +0,0 @@ -org.finos.legend.connection.impl.PostgresDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager new file mode 100644 index 00000000000..ed38b2b933e --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-postgres/legend-engine-xt-relationalStore-postgres-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager @@ -0,0 +1 @@ +org.finos.legend.connection.impl.PostgresRelationalDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/pom.xml b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/pom.xml index 7caa328a241..bd87a1ee8ef 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/pom.xml +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/pom.xml @@ -35,7 +35,11 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory + + + org.finos.legend.engine + legend-engine-xt-connection-protocol org.finos.legend.engine diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/SnowflakeAccountType.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/SnowflakeAccountType.java deleted file mode 100644 index ddc11f3bd40..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/SnowflakeAccountType.java +++ /dev/null @@ -1,21 +0,0 @@ -// Copyright 2023 Goldman Sachs -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package org.finos.legend.connection; - -public enum SnowflakeAccountType -{ - VPS, - MultiTenant -} diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/HACKY__SnowflakeConnectionAdapter.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/HACKY__SnowflakeConnectionAdapter.java index 32c89b91622..dbbbe7a9297 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/HACKY__SnowflakeConnectionAdapter.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/HACKY__SnowflakeConnectionAdapter.java @@ -14,15 +14,16 @@ package org.finos.legend.connection.impl; +import org.finos.legend.connection.Connection; import org.finos.legend.connection.HACKY__RelationalDatabaseConnectionAdapter; import org.finos.legend.connection.LegendEnvironment; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.protocol.SnowflakeConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.connection.SnowflakeConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.authentication.vault.PropertiesFileSecret; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.RelationalDatabaseConnection; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.authentication.SnowflakePublicAuthenticationStrategy; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.specification.SnowflakeDatasourceSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.EncryptedPrivateKeyPairAuthenticationConfiguration; import org.finos.legend.engine.shared.core.identity.Identity; public class HACKY__SnowflakeConnectionAdapter @@ -56,10 +57,10 @@ public ConnectionFactoryMaterial adapt(RelationalDatabaseConnection relationalDa connectionSpecification.accountType = datasourceSpecification.accountType; connectionSpecification.role = datasourceSpecification.role; - StoreInstance storeInstance = new StoreInstance.Builder(environment) - .withIdentifier("adapted-store") - .withStoreSupportIdentifier("Snowflake") - .withConnectionSpecification(connectionSpecification) + Connection connection = Connection.builder() + .databaseSupport(environment.getDatabaseSupport(RelationalDatabaseType.SNOWFLAKE)) + .identifier("adapted-store") + .connectionSpecification(connectionSpecification) .build(); EncryptedPrivateKeyPairAuthenticationConfiguration authenticationConfiguration = new EncryptedPrivateKeyPairAuthenticationConfiguration(); @@ -67,7 +68,7 @@ public ConnectionFactoryMaterial adapt(RelationalDatabaseConnection relationalDa authenticationConfiguration.privateKey = new PropertiesFileSecret(authenticationStrategy.privateKeyVaultReference); authenticationConfiguration.passphrase = new PropertiesFileSecret(authenticationStrategy.passPhraseVaultReference); - return new ConnectionFactoryMaterial(storeInstance, authenticationConfiguration); + return new ConnectionFactoryMaterial(connection, authenticationConfiguration); } return null; } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeConnectionBuilder.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeConnectionBuilder.java index 7c6d11bcaec..9b01d03bd6d 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeConnectionBuilder.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeConnectionBuilder.java @@ -15,31 +15,28 @@ package org.finos.legend.connection.impl; import org.finos.legend.connection.Authenticator; -import org.finos.legend.connection.DatabaseType; -import org.finos.legend.connection.JDBCConnectionBuilder; -import org.finos.legend.connection.RelationalDatabaseStoreSupport; -import org.finos.legend.connection.StoreInstance; -import org.finos.legend.connection.protocol.SnowflakeConnectionSpecification; +import org.finos.legend.connection.Connection; +import org.finos.legend.connection.DatabaseSupport; +import org.finos.legend.engine.protocol.pure.v1.connection.SnowflakeConnectionSpecification; import org.finos.legend.engine.shared.core.identity.Credential; import org.finos.legend.engine.shared.core.identity.Identity; import org.finos.legend.engine.shared.core.identity.credential.PrivateKeyCredential; -import java.sql.Connection; import java.util.Optional; import java.util.Properties; import java.util.function.Function; -import static org.finos.legend.connection.impl.SnowflakeDatabaseManager.*; +import static org.finos.legend.connection.impl.SnowflakeRelationalDatabaseManager.*; public class SnowflakeConnectionBuilder { public static class WithKeyPair extends JDBCConnectionBuilder { @Override - public Connection getConnection(SnowflakeConnectionSpecification connectionSpecification, Authenticator authenticator, Identity identity) throws Exception + public java.sql.Connection getConnection(SnowflakeConnectionSpecification connectionSpecification, Authenticator authenticator, Identity identity) throws Exception { - StoreInstance storeInstance = authenticator.getStoreInstance(); - RelationalDatabaseStoreSupport.cast(storeInstance.getStoreSupport(), DatabaseType.SNOWFLAKE); + Connection connection = authenticator.getConnection(); + DatabaseSupport.verifyDatabaseType(connection.getDatabaseSupport(), RelationalDatabaseType.SNOWFLAKE); Properties connectionProperties = generateJDBCConnectionProperties(connectionSpecification); Function authenticationPropertiesSupplier = cred -> @@ -51,7 +48,7 @@ public Connection getConnection(SnowflakeConnectionSpecification connectionSpeci return properties; }; - return this.getConnectionManager().getConnection(DatabaseType.SNOWFLAKE, null, 0, connectionSpecification.databaseName, connectionProperties, this.getConnectionPoolConfig(), authenticationPropertiesSupplier, authenticator, identity); + return this.getConnectionManager().getConnection(RelationalDatabaseType.SNOWFLAKE, null, 0, connectionSpecification.databaseName, connectionProperties, this.getConnectionPoolConfig(), authenticationPropertiesSupplier, authenticator, identity); } } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeDatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeRelationalDatabaseManager.java similarity index 93% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeDatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeRelationalDatabaseManager.java index 81f78f2a73e..dafc4d2470c 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeDatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/impl/SnowflakeRelationalDatabaseManager.java @@ -15,15 +15,13 @@ package org.finos.legend.connection.impl; import org.eclipse.collections.impl.factory.Lists; -import org.finos.legend.connection.DatabaseType; -import org.finos.legend.connection.SnowflakeAccountType; -import org.finos.legend.connection.DatabaseManager; +import org.finos.legend.connection.RelationalDatabaseManager; import org.finos.legend.engine.shared.core.operational.Assert; import java.util.List; import java.util.Properties; -public class SnowflakeDatabaseManager implements DatabaseManager +public class SnowflakeRelationalDatabaseManager implements RelationalDatabaseManager { private static final String PRIVATELINK_SNOWFLAKECOMPUTING_COM = ".privatelink.snowflakecomputing.com"; private static final String SNOWFLAKECOMPUTING_COM = ".snowflakecomputing.com"; @@ -48,7 +46,7 @@ public class SnowflakeDatabaseManager implements DatabaseManager @Override public List getIds() { - return Lists.mutable.with(DatabaseType.SNOWFLAKE.getLabel()); + return Lists.mutable.with(RelationalDatabaseType.SNOWFLAKE.getIdentifier()); } @Override @@ -104,4 +102,10 @@ public void buildMultiTenantHostname(String accountName, String region, StringBu { url.append(accountName).append(".").append(region); } + + public static enum SnowflakeAccountType + { + VPS, + MultiTenant + } } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager deleted file mode 100644 index fa4a918a8d0..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager +++ /dev/null @@ -1 +0,0 @@ -org.finos.legend.connection.impl.SnowflakeDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager new file mode 100644 index 00000000000..aaffb6f32f8 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager @@ -0,0 +1 @@ +org.finos.legend.connection.impl.SnowflakeRelationalDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/pom.xml b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/pom.xml index a0421587336..c3232be1328 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/pom.xml +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/pom.xml @@ -37,6 +37,10 @@ org.finos.legend.engine legend-engine-xt-relationalStore-protocol + + org.finos.legend.engine + legend-engine-xt-connection-protocol + diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/SnowflakeProtocolExtension.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/SnowflakeProtocolExtension.java index f4dd2829f20..7d1b06fd077 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/SnowflakeProtocolExtension.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/SnowflakeProtocolExtension.java @@ -16,6 +16,8 @@ import org.eclipse.collections.api.block.function.Function0; import org.eclipse.collections.impl.factory.Lists; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.connection.SnowflakeConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.extension.ProtocolSubTypeInfo; import org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.authentication.AuthenticationStrategy; @@ -31,14 +33,18 @@ public class SnowflakeProtocolExtension implements PureProtocolExtension public List>>> getExtraProtocolSubTypeInfoCollectors() { return Lists.fixedSize.with(() -> Lists.fixedSize.with( - //DatasourceSpecification - ProtocolSubTypeInfo.newBuilder(DatasourceSpecification.class) - .withSubtype(SnowflakeDatasourceSpecification.class, "snowflake") - .build(), - // AuthenticationStrategy - ProtocolSubTypeInfo.newBuilder(AuthenticationStrategy.class) - .withSubtype(SnowflakePublicAuthenticationStrategy.class, "snowflakePublic") - .build() + // DatasourceSpecification + ProtocolSubTypeInfo.newBuilder(DatasourceSpecification.class) + .withSubtype(SnowflakeDatasourceSpecification.class, "snowflake") + .build(), + // AuthenticationStrategy + ProtocolSubTypeInfo.newBuilder(AuthenticationStrategy.class) + .withSubtype(SnowflakePublicAuthenticationStrategy.class, "snowflakePublic") + .build(), + // ConnectionSpecification + ProtocolSubTypeInfo.newBuilder(ConnectionSpecification.class) + .withSubtype(SnowflakeConnectionSpecification.class, "snowflake") + .build() )); } } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/protocol/SnowflakeConnectionSpecification.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/connection/SnowflakeConnectionSpecification.java similarity index 91% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/protocol/SnowflakeConnectionSpecification.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/connection/SnowflakeConnectionSpecification.java index 555f44c7979..0e2118f6551 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-connection/src/main/java/org/finos/legend/connection/protocol/SnowflakeConnectionSpecification.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-snowflake/legend-engine-xt-relationalStore-snowflake-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/connection/SnowflakeConnectionSpecification.java @@ -12,7 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.protocol; +package org.finos.legend.engine.protocol.pure.v1.connection; + +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; public class SnowflakeConnectionSpecification extends ConnectionSpecification { diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerDatabaseManager.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerRelationalDatabaseManager.java similarity index 90% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerDatabaseManager.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerRelationalDatabaseManager.java index f22628f7a8b..df5c20172cc 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerDatabaseManager.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/java/org/finos/legend/connection/jdbc/driver/SQLServerRelationalDatabaseManager.java @@ -15,13 +15,13 @@ package org.finos.legend.connection.jdbc.driver; import org.eclipse.collections.impl.factory.Lists; -import org.finos.legend.connection.DatabaseManager; +import org.finos.legend.connection.RelationalDatabaseManager; import org.finos.legend.engine.protocol.pure.v1.model.packageableElement.store.relational.connection.DatabaseType; import java.util.List; import java.util.Properties; -public class SQLServerDatabaseManager implements DatabaseManager +public class SQLServerRelationalDatabaseManager implements RelationalDatabaseManager { @Override public List getIds() diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager deleted file mode 100644 index c4d426cd4b6..00000000000 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.DatabaseManager +++ /dev/null @@ -1 +0,0 @@ -org.finos.legend.connection.jdbc.driver.SQLServerDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager new file mode 100644 index 00000000000..120a1866f63 --- /dev/null +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-dbExtension/legend-engine-xt-relationalStore-sqlserver/legend-engine-xt-relationalStore-sqlserver-connection/src/main/resources/META-INF/services/org.finos.legend.connection.RelationalDatabaseManager @@ -0,0 +1 @@ +org.finos.legend.connection.jdbc.driver.SQLServerRelationalDatabaseManager diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/pom.xml b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/pom.xml index 9f7ee8a879f..292995db609 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/pom.xml +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/pom.xml @@ -129,7 +129,7 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-connection-factory org.bouncycastle diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/manager/ConnectionManagerSelector.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/manager/ConnectionManagerSelector.java index c8a4be55180..5155617da16 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/manager/ConnectionManagerSelector.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-execution/legend-engine-xt-relationalStore-executionPlan/src/main/java/org/finos/legend/engine/plan/execution/stores/relational/connection/manager/ConnectionManagerSelector.java @@ -155,7 +155,7 @@ public Connection getDatabaseConnectionImpl(Identity identity, DatabaseConnectio { try { - return this.connectionFactory.getConnection(identity, connectionFactoryMaterial.storeInstance, connectionFactoryMaterial.authenticationConfiguration); + return this.connectionFactory.getConnection(identity, connectionFactoryMaterial.connection, connectionFactoryMaterial.authenticationConfiguration); } catch (Exception exception) { diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/pom.xml b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/pom.xml index 857b0736ee7..67221f7ee43 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/pom.xml +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/pom.xml @@ -83,6 +83,10 @@ junit test + + org.finos.legend.engine + legend-engine-xt-connection-protocol + \ No newline at end of file diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/RelationalProtocolExtension.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/RelationalProtocolExtension.java index 13228724c59..2a9067ca3b6 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/RelationalProtocolExtension.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/RelationalProtocolExtension.java @@ -17,8 +17,10 @@ import org.eclipse.collections.api.block.function.Function0; import org.eclipse.collections.api.factory.Lists; import org.eclipse.collections.api.factory.Maps; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.extension.ProtocolSubTypeInfo; import org.finos.legend.engine.protocol.pure.v1.extension.PureProtocolExtension; +import org.finos.legend.engine.protocol.pure.v1.model.connection.StaticJDBCConnectionSpecification; import org.finos.legend.engine.protocol.pure.v1.model.data.EmbeddedData; import org.finos.legend.engine.protocol.pure.v1.model.executionPlan.RelationResultType; import org.finos.legend.engine.protocol.pure.v1.model.executionPlan.nodes.CreateAndPopulateTempTableExecutionNode; @@ -184,6 +186,11 @@ public List>>> getExtraProtocolSubTypeInfo .withSubtype(BusinessMilestoning.class, "businessMilestoning") .withSubtype(BusinessSnapshotMilestoning.class, "businessSnapshotMilestoning") .withSubtype(ProcessingMilestoning.class, "processingMilestoning") + .build(), + + // Connection Specification + ProtocolSubTypeInfo.newBuilder(ConnectionSpecification.class) + .withSubtype(StaticJDBCConnectionSpecification.class, "staticJDBC") .build() )); } diff --git a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/protocol/StaticJDBCConnectionSpecification.java b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/connection/StaticJDBCConnectionSpecification.java similarity index 82% rename from legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/protocol/StaticJDBCConnectionSpecification.java rename to legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/connection/StaticJDBCConnectionSpecification.java index 373e76c7c42..1082bacc174 100644 --- a/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-connection/src/main/java/org/finos/legend/connection/protocol/StaticJDBCConnectionSpecification.java +++ b/legend-engine-xts-relationalStore/legend-engine-xt-relationalStore-generation/legend-engine-xt-relationalStore-protocol/src/main/java/org/finos/legend/engine/protocol/pure/v1/model/connection/StaticJDBCConnectionSpecification.java @@ -12,9 +12,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -package org.finos.legend.connection.protocol; +package org.finos.legend.engine.protocol.pure.v1.model.connection; -import org.finos.legend.connection.protocol.ConnectionSpecification; +import org.finos.legend.engine.protocol.pure.v1.packageableElement.connection.ConnectionSpecification; public class StaticJDBCConnectionSpecification extends ConnectionSpecification { @@ -22,6 +22,11 @@ public class StaticJDBCConnectionSpecification extends ConnectionSpecification public int port; public String databaseName; + public StaticJDBCConnectionSpecification() + { + // jackson + } + public StaticJDBCConnectionSpecification(String host, int port, String databaseName) { this.host = host; diff --git a/pom.xml b/pom.xml index b2ac32003ca..609904e7d25 100644 --- a/pom.xml +++ b/pom.xml @@ -92,6 +92,7 @@ legend-engine-xts-authentication + legend-engine-xts-connection legend-engine-config @@ -157,6 +158,7 @@ 2.6 3.7 1.10.0 + 0.12.0 1.3.17-1 4.1.16 1.3.29 @@ -188,6 +190,7 @@ 1.2.17 0.1.5 1.2.3 + 8.5.5 4.4.0 3.12.8 4.6 @@ -695,22 +698,42 @@ org.finos.legend.engine - legend-engine-xt-authentication-connection-factory + legend-engine-xt-authentication-implementation-core ${project.version} org.finos.legend.engine - legend-engine-xt-authentication-implementation-core + legend-engine-xt-authentication-implementation-vault-aws ${project.version} org.finos.legend.engine - legend-engine-xt-authentication-implementation-vault-aws + legend-engine-xt-authentication-implementation-gcp-federation ${project.version} org.finos.legend.engine - legend-engine-xt-authentication-implementation-gcp-federation + legend-engine-xt-connection-factory + ${project.version} + + + org.finos.legend.engine + legend-engine-xt-connection-compiler + ${project.version} + + + org.finos.legend.engine + legend-engine-xt-connection-grammar + ${project.version} + + + org.finos.legend.engine + legend-engine-xt-connection-protocol + ${project.version} + + + org.finos.legend.engine + legend-engine-xt-connection-pure-metamodel ${project.version} @@ -3029,6 +3052,17 @@ docker-java-api 3.3.0 + + io.minio + minio + ${minio.version} + + + org.jetbrains + * + + + @@ -3150,6 +3184,19 @@ + + + io.deephaven + deephaven-csv + ${deephaven-csv.version} + + + io.deephaven + deephaven-csv-fast-double-parser + ${deephaven-csv.version} + runtime + + org.apache.arrow @@ -3193,7 +3240,6 @@ - @@ -3234,9 +3280,7 @@ vX_X_X v1_33_0 - -