Dynamic Address Allocation for Local DNS in Nebula Spark Connector

[vaibhavkgiradkar]

Description:
When attempting to pull or push data from another cluster to the Nebula cluster, we often encounter errors related to DNS unreachability.

Motivation:

This enhancement is necessary because these DNS-related errors can hinder data transfer operations between clusters. By modifying the Nebula Spark Connector build to dynamically replace the DNS addresses during connection attempts, we can overcome these issues and facilitate smooth data transfer between the Nebula cluster and external clusters.

Proposed Solution:
We propose modifying the Nebula Spark Connector build to dynamically replace the DNS addresses during both read and write operations. In read operations, the connector will be enhanced to include configurable storage addresses that can be replaced with the appropriate local DNS addresses at the time of connection. Similarly, in write operations, we will implement dynamic address allocation to ensure that data is pushed to the Nebula cluster through a mechanism that seamlessly handles local DNS resolution.

For read operations, this dynamic addressing will significantly improve the reliability and flexibility of retrieving data from the Nebula cluster, especially when dealing with remote clusters or external systems with varying DNS configurations.

For write operations, dynamic address allocation will enable the Nebula Spark Connector to adapt to changing DNS configurations during data transfers. This enhancement ensures a consistent and error-free data transfer process.

Additional Information:
This enhancement will significantly enhance the reliability and flexibility of data read and write operations within the Nebula Spark Connector. It will be especially beneficial when dealing with remote clusters or external systems that may require dynamic address allocation to resolve DNS issues.

[koustreak]

@wey-gu need your help .

[wey-gu]

Dear @koustreak , @vaibhavkgiradkar

Sorry to know that you had to hack the DNS in such a dirty way to enable this!

And I share similar ideas(not DNS but to allow replacing host list) as in #39, too! And that involves java client changing, as well.

@Nicole00 @HarrisChu @xtcyclist @MuYiYong @yixinglu what do you think of this, (hands up the topic, again)

This will help not just ETL pipeline but also analytical/graph computation, but just enabling how storaged instances will be network addressed.

We just optionally decouple how service instances discovery/identified with how they are routed/addressed, a pure client opt-in behavior change, things could be much different then.

BR//Wey

[vaibhavkgiradkar]

@wey-gu
Will support be added to enable reading/writing from a new cluster?