[fix][sec] Drop hdfs2 support, Upgrade hadoop3 to 3.4.0 and dnsjava to 3.6.2 to address CVE-2024-25638

[lhotari]

Motivation

In pulsar-io connectors, Hadoop 2 and Hadoop 3 versions before 3.4.0 depend on dnsjava 2.1.7 which is outdated and contains CVE-2024-25638 .

Upgrading dnsjava 2.1.7 to dnsjava 3.6.x isn't possible directly since dnsjava 3.x API isn't compatible with dnsjava 2.1.x API. It is necessary to ensure that all dependencies are at a level where dnsjava can be upgraded.

Modifications

• remove pulsar-io/hdfs2 connector since it depends on Hadoop 2 which doesn't support dnsjava 3.x
• upgrade hadoop3 version 3.4.0 which supports dnsjava 3.x
• upgrade hbase version used in pulsar-io/hbase to 2.6.0-hadoop3 since then it can use hadoop3 3.4.0 which supports dnsjava 3.x
• use dependencyManagement to enforce the versions so that hadoop3 3.4.0 libraries are used consistently
  • hadoop3 doesn't have a bom so each depended dependency will have to be included in dependencyManagement

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.56%. Comparing base (bbc6224) to head (47ca891).
Report is 643 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23411      +/-   ##
============================================
+ Coverage     73.57%   74.56%   +0.99%     
- Complexity    32624    34517    +1893     
============================================
  Files          1877     1924      +47     
  Lines        139502   145251    +5749     
  Branches      15299    15873     +574     
============================================
+ Hits         102638   108309    +5671     
+ Misses        28908    28645     -263     
- Partials       7956     8297     +341     

Flag	Coverage Δ
inttests	27.61% <ø> (+3.02%)	⬆️
systests	24.52% <ø> (+0.20%)	⬆️
unittests	73.92% <ø> (+1.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 625 files with indirect coverage changes

[lhotari]

It looks like we need to wait for Hadoop 3.4.1 release to complete: https://issues.apache.org/jira/browse/HADOOP-19237

[lhotari]

"Exclude the META-INF/services/java.net.spi.InetAddressResolverProvider from your project" is mentioned as a workaround in dnsjava/dnsjava#338. Related Hadoop PR https://github.com/apache/hadoop/pull/7070/files . I guess that there might be ways to exclude and disable dnsjava too. However Hadoop 3.4.1 release would be useful.

[lhotari]

Let's wait for Hadoop 3.4.1 release that is in voting: https://lists.apache.org/thread/6dshj3nb26mlhzhmd8xkmw78l3zk896t