Add Hbase new version 2.4.17 and compression #494

maltesander · 2023-11-02T10:24:55Z

Description

Add HBase 2.4.17
Bump jmx-exporter to 0.20.0 and use softlink
Add HBase package upload script
Add snappy compression

fixes #332

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Tasks

Give feedback

Changes are OpenShift compatible
All added packages (via microdnf or otherwise) have a comment on why they are added
Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
All packages should have (if available) signatures/hashes verified
Does your change affect an SBOM? Make sure to update all SBOMs
Options

# Conflicts: # CHANGELOG.md

- Upload HBase sources to Nexus as well - Patch HBase build to fix HBASE-27860 - Custom build HBase against Hadoop 3.3.6

- Include some patches for 2.4.12 but it doesn't build yet due to another issue (HBASE-25983 probably) - Remove unneeded packages Note: The builder part is a bit torn to pieces due to debugging and needs to be cleaned up

…mpression

lfrancke · 2023-11-06T09:18:56Z

When we have more than one image that requires patching it probably makes sense to move apply_patches.sh outside of the HBase image to prevent duplication but for now I was too lazy to figure out where it should go etc.

hbase/Dockerfile

razvan

The images build successfully. I left some comments and will have a look at the related operator PR again

hadoop/Dockerfile

conf.py

hbase/Dockerfile

razvan · 2023-11-06T15:59:25Z

I get the following error when running the kuttl smoke test with the operator from stackabletech/hbase-operator#403

2023-11-06 15:54:43,540 WARN  [master/test-hbase-master-default-0:16000:becomeActiveMaster] wal.AbstractProtobufLogWriter: Init output failed, path=hdfs://test-hdfs/hbase/MasterData/WALs/test-hbase-master-default-0.test-hbase-master-default.kuttl-test-learning-gannet.svc.cluster.local,16000,1699286079146/test-hbase-master-default-0.test-hbase-master-default.kuttl-test-learning-gannet.svc.cluster.local%2C16000%2C1699286079146.1699286083449
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2693)
	at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.getWALCellCodecClass(WALCellCodec.java:82)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.buildWALHeader0(AbstractProtobufLogWriter.java:81)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.buildWALHeader(AbstractProtobufLogWriter.java:88)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:164)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:719)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:881)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:574)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:515)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295)
	at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200)
	at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:263)
	at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:344)
	at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528)
	at java.base/java.lang.Thread.run(Thread.java:829)

maltesander · 2023-11-06T16:16:27Z

I get the following error when running the kuttl smoke test with the operator from stackabletech/hbase-operator#403

2023-11-06 15:54:43,540 WARN  [master/test-hbase-master-default-0:16000:becomeActiveMaster] wal.AbstractProtobufLogWriter: Init output failed, path=hdfs://test-hdfs/hbase/MasterData/WALs/test-hbase-master-default-0.test-hbase-master-default.kuttl-test-learning-gannet.svc.cluster.local,16000,1699286079146/test-hbase-master-default-0.test-hbase-master-default.kuttl-test-learning-gannet.svc.cluster.local%2C16000%2C1699286079146.1699286083449
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec not found
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2693)
	at org.apache.hadoop.hbase.regionserver.wal.WALCellCodec.getWALCellCodecClass(WALCellCodec.java:82)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.buildWALHeader0(AbstractProtobufLogWriter.java:81)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.buildWALHeader(AbstractProtobufLogWriter.java:88)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:164)
	at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:116)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:719)
	at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:128)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:881)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:574)
	at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.init(AbstractFSWAL.java:515)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:160)
	at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:62)
	at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:295)
	at org.apache.hadoop.hbase.master.region.MasterRegion.createWAL(MasterRegion.java:200)
	at org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:263)
	at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:344)
	at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
	at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855)
	at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193)
	at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528)
	at java.base/java.lang.Thread.run(Thread.java:829)

Sorry should have mentioned this. Images are not build yet, please try with:

docker pull stackable-experimental/hbase:2.4.17-stackable23.11.0-compression-fix

razvan · 2023-11-06T16:19:54Z

docker pull stackable-experimental/hbase:2.4.17-stackable23.11.0-compression-fix

I built the images myself.

razvan · 2023-11-06T16:24:08Z

I believe the problem is that the environment variable HBASE_CONF_DIR is missing from the final docker stage. I will test.

razvan · 2023-11-06T16:48:12Z

The env var is not the problem (but should probably be set anyway).

The classpath only contains the following:

/stackable/hbase/conf
/usr/lib/jvm/jre-11/lib/tools.jar
/stackable/hbase/bin/..
/stackable/hbase/bin/../lib/shaded-clients/hbase-shaded-client-2.4.17.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/audience-annotations-0.13.0.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/reload4j-1.2.22.jar
/stackable/hbase/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar

and is missing the important part ( /stackable/hbase/lib )

➜  ~ k run hbase --image docker.stackable.tech/stackable/hbase:2.4.17-stackable0.0.0-dev -- hbase classpath    
pod/hbase created
➜  ~ k logs hbase
/stackable/hbase/conf:/usr/lib/jvm/jre-11/lib/tools.jar:/stackable/hbase/bin/..:/stackable/hbase/bin/../lib/shaded-clients/hbase-shaded-client-2.4.17.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/audience-annotations-0.13.0.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/commons-logging-1.2.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/reload4j-1.2.22.jar:/stackable/hbase/bin/../lib/client-facing-thirdparty/slf4j-api-1.7.33.jar

razvan · 2023-11-07T10:33:17Z

The docker.stackable.tech/stackable-experimental/hbase:2.4.17-stackable23.11.0-compression-fix uses java 8. The HEAD version uses java 11.

razvan · 2023-11-07T10:37:51Z

And the error is that the HEAD version doesn't include the phoenix-server-hbase-2.4-5.1.3.jar file which is where the IndexedWALEditCodec class is located.

bash-4.4# ls -l /stackable/hbase/lib/phoenix-server-hbase-2.4-5.1.3.jar 
lrwxrwxrwx 1 stackable stackable 53 Nov  6 15:00 /stackable/hbase/lib/phoenix-server-hbase-2.4-5.1.3.jar -> /stackable/phoenix/phoenix-server-hbase-2.4-5.1.3.jar
bash-4.4# ls -l /stackable 
total 12
lrwxrwxrwx 1 root      root        23 Nov  6 15:00 hbase -> /stackable/hbase-2.4.17
drwxr-xr-x 6 stackable stackable 4096 Nov  6 14:59 hbase-2.4.17
drwxr-xr-x 2 stackable stackable 4096 Nov  6 14:59 jmx
drwxr-xr-x 3 stackable stackable 4096 Nov  6 14:19 vector

razvan · 2023-11-07T11:47:53Z

After adding Phoenix to the image, the smoke test fails because apparently Phoenix needs tar which is not available

   logger.go:42: 12:40:40 | smoke_hbase-2.4.17_hdfs-3.2.4_zookeeper-3.8.3_listener-class-external-unstable/60-test-phoenix | error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d0ad2265043eb78e4a1a306d497a48a547fff6f1f8b6a25a92bbaf68eced81e5": OCI runtime exec failed: exec failed: unable to start container process: exec: "tar": executable file not found in $PATH: unknown

lfrancke · 2023-11-07T11:54:08Z

Thank you! That one should be easy to fix, when you do, please add a comment what it is needed for so we don't forget!

hbase/Dockerfile

razvan

The integration tests now work. The comments regarding the handling of the JAVA version will be added to new issue

maltesander · 2023-11-07T13:57:45Z

Merge for now, track improvements here #495

maltesander and others added 15 commits October 6, 2023 17:19

add hbase 2.4.17, update jmx-exporter to 0.20.0

6c69c4d

add upload script

5170387

adapted changelog

1b5b965

use softlink for jmx exporter

e390882

add --fail to curl commands

8ede222

add legacy symlink for jmx-exporter

d966dc4

add quotes to fix linter

3e802ec

fix copy paste

c742e34

adapt jmx-exporter configs to match role names from operator

8c2e250

add to changelog

0aea88e

add jmx exporter arg

a29bdd4

Merge remote-tracking branch 'origin/main' into hbase-new-version

5267127

# Conflicts: # CHANGELOG.md

WIP towards HBase compression

2edb0d1

- Upload HBase sources to Nexus as well - Patch HBase build to fix HBASE-27860 - Custom build HBase against Hadoop 3.3.6

Add patch file for HBASE-27860

120326e

Split into Builder & Final image

90dc502

- Include some patches for 2.4.12 but it doesn't build yet due to another issue (HBASE-25983 probably) - Remove unneeded packages Note: The builder part is a bit torn to pieces due to debugging and needs to be cleaned up

maltesander assigned lfrancke and maltesander Nov 2, 2023

This was referenced Nov 2, 2023

Hbase new version 2.4.17 #469

Closed

Update product versions for release 23.11 stackabletech/issues#446

Closed

maltesander and others added 4 commits November 2, 2023 11:28

Merge remote-tracking branch 'origin/main' into hbase-compression

2e8236f

adapt changelog to new PR number

c2ab7f5

HBase 2.4.12 builds as well now

6a6d209

Merge remote-tracking branch 'origin/hbase-compression' into hbase-co…

d3515d8

…mpression

lfrancke marked this pull request as ready for review November 6, 2023 09:17

maltesander commented Nov 6, 2023

View reviewed changes

hbase/Dockerfile Outdated Show resolved Hide resolved

hbase/Dockerfile Show resolved Hide resolved

hbase/Dockerfile Show resolved Hide resolved

clean up

064e2b2

razvan self-requested a review November 6, 2023 14:13

razvan reviewed Nov 6, 2023

View reviewed changes

hadoop/Dockerfile Show resolved Hide resolved

conf.py Show resolved Hide resolved

hbase/Dockerfile Show resolved Hide resolved

Copy Phoenix fro builder

bbb65ad

add tar to final image

bbfa0f6

lfrancke reviewed Nov 7, 2023

View reviewed changes

hbase/Dockerfile Outdated Show resolved Hide resolved

Add Phoenix requirements.

8572ada

razvan approved these changes Nov 7, 2023

View reviewed changes

maltesander mentioned this pull request Nov 7, 2023

Improve building Java projects from source #495

Closed

maltesander enabled auto-merge November 7, 2023 13:57

maltesander added this pull request to the merge queue Nov 7, 2023

Merged via the queue into main with commit 750923e Nov 7, 2023
11 of 12 checks passed

maltesander deleted the hbase-compression branch November 7, 2023 13:59

maltesander mentioned this pull request Nov 10, 2023

Add HBASE-25292 and HBASE-25336 to HBase 2.4 #500

Merged

maltesander mentioned this pull request May 15, 2024

Build all product and intermediate/dependencies images from source #674

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Hbase new version 2.4.17 and compression #494

Add Hbase new version 2.4.17 and compression #494

maltesander commented Nov 2, 2023

Tasks

lfrancke commented Nov 6, 2023

razvan left a comment

razvan commented Nov 6, 2023

maltesander commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 7, 2023

razvan commented Nov 7, 2023

razvan commented Nov 7, 2023

lfrancke commented Nov 7, 2023

razvan left a comment

maltesander commented Nov 7, 2023

Add Hbase new version 2.4.17 and compression #494

Add Hbase new version 2.4.17 and compression #494

Conversation

maltesander commented Nov 2, 2023

Description

Definition of Done Checklist

Tasks

lfrancke commented Nov 6, 2023

razvan left a comment

Choose a reason for hiding this comment

razvan commented Nov 6, 2023

maltesander commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 6, 2023

razvan commented Nov 7, 2023

razvan commented Nov 7, 2023

razvan commented Nov 7, 2023

lfrancke commented Nov 7, 2023

razvan left a comment

Choose a reason for hiding this comment

maltesander commented Nov 7, 2023