You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the interface to the different databases backend is through the StorageManager. It was built assuming SQL databases (mySQL, postgresQL,...), offering functions that takes SQL queries as input and JDBC drivers are doing the connection to the databases.
But if we want to introduce NoSQL backends, like HBASE, or even memcached, the process of understanding the SQL query and performing it is done at the JDBC driver level (like Apache Phoenix project). Having such adapters would work out almost of the box, but when looking at the whole streaming process would introduce a huge and useless overhead.
Indeed, a lot of functions are just building SQL queries to be able to interact with the DB, but if we know that the DB is just a K-V store, and that the query will just need to be parsed in the next function call, why still using SQL internally?
SQL is very handy for using in the VS configuration files and should be kept here, but it should be parsed much sooner and replaced by an abstraction that can be efficiently passed to SQL and NoSQL backends.
This is just a suggestion and it is not clear for me yet at which level this should be implemented (JDBC, StorageManager, SlidingTimeWindows, VSConfig) ?
The text was updated successfully, but these errors were encountered:
Currently the interface to the different databases backend is through the StorageManager. It was built assuming SQL databases (mySQL, postgresQL,...), offering functions that takes SQL queries as input and JDBC drivers are doing the connection to the databases.
But if we want to introduce NoSQL backends, like HBASE, or even memcached, the process of understanding the SQL query and performing it is done at the JDBC driver level (like Apache Phoenix project). Having such adapters would work out almost of the box, but when looking at the whole streaming process would introduce a huge and useless overhead.
Indeed, a lot of functions are just building SQL queries to be able to interact with the DB, but if we know that the DB is just a K-V store, and that the query will just need to be parsed in the next function call, why still using SQL internally?
SQL is very handy for using in the VS configuration files and should be kept here, but it should be parsed much sooner and replaced by an abstraction that can be efficiently passed to SQL and NoSQL backends.
This is just a suggestion and it is not clear for me yet at which level this should be implemented (JDBC, StorageManager, SlidingTimeWindows, VSConfig) ?
The text was updated successfully, but these errors were encountered: