Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

chore: support for hbase 2.6.1 #627

Merged
merged 2 commits into from
Feb 12, 2025
Merged

chore: support for hbase 2.6.1 #627

merged 2 commits into from
Feb 12, 2025

Conversation

dervoeti
Copy link
Member

@dervoeti dervoeti commented Feb 11, 2025

Description

Part of stackabletech/docker-images#972

I had to add the --no-checksum-verify flag when exporting snapshots to/from S3, because our snapshot-export test was failing for HBase 2.6.1.
From what I see, checksum verification was merged and released with 2.6.1: https://issues.apache.org/jira/browse/HBASE-28625
However, it seems to have problems when comparing snapshots on different filesystems (like HDFS and S3).

https://issues.apache.org/jira/browse/HBASE-28702 was also merged into 2.6.1 and seems to introduce a flag to skip checksum-checks between snapshots when exporting tables. I'm not sure if this flag is supposed to work with snapshot exports as well, at least I could not make it work. Setting fs.checksum.combine.mode to COMPOSITE_CRC also does not seem to work.

https://issues.apache.org/jira/browse/HBASE-28998 (still open) also mentions that it does not work with COMPOSITE_CRC and it specificly mentions problems with HDFS in combination with S3.

So I decided to add the --no-checksum-verify flag in our tests when exporting snapshots. From what I understand snapshot checksums where not validated before 2.6.1 either. I don't have too much experience with HBase and thus am not sure if this is really the best solution. Could one of our HBase experts (@razvan @siegfriedweber @lfrancke ?) take a look at this as well? If this solution is fine I would adapt the documentation accordingly.

Definition of Done Checklist

Author

Preview Give feedback

Reviewer

Preview Give feedback

Acceptance

Preview Give feedback

@razvan razvan self-requested a review February 12, 2025 08:48
@razvan
Copy link
Member

razvan commented Feb 12, 2025

Indeed, it seems there is no workaround to the checksum problem currently.

The tests are green:

--- PASS: kuttl (4600.24s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (210.98s)
        --- PASS: kuttl/harness/shutdown_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (503.10s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-external-unstable_openshift-false (420.45s)
        --- PASS: kuttl/harness/smoke_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-3.4.0_zookeeper-3.9.2_listener-class-cluster-internal_openshift-false (299.91s)
        --- PASS: kuttl/harness/cluster-operation_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (259.18s)
        --- PASS: kuttl/harness/omid_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false_omid-1.1.2 (262.05s)
        --- PASS: kuttl/harness/logging_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (229.79s)
        --- PASS: kuttl/harness/overrides_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (213.66s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-cluster-internal_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (448.90s)
        --- PASS: kuttl/harness/orphaned_resources_hbase-latest-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (240.87s)
        --- PASS: kuttl/harness/kerberos_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_listener-class-external-unstable_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (481.88s)
        --- PASS: kuttl/harness/profiling_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (226.43s)
        --- PASS: kuttl/harness/snapshot-export_hbase-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_openshift-false (288.24s)
        --- PASS: kuttl/harness/opa_hbase-opa-2.6.1,docker.stackable.tech_stackable_hbase_2.6.1-stackable0.0.0-dev_hdfs-latest-3.4.0_zookeeper-latest-3.9.2_krb5-1.21.1_opa-1.0.0_openshift-false (514.78s)
PASS

Copy link
Member

@razvan razvan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm with a small fix

@dervoeti dervoeti added this pull request to the merge queue Feb 12, 2025
Merged via the queue into main with commit 63b9bb8 Feb 12, 2025
17 checks passed
@dervoeti dervoeti deleted the chore/add-hbase-2.6.1 branch February 12, 2025 14:13
@dervoeti dervoeti restored the chore/add-hbase-2.6.1 branch February 14, 2025 11:04
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants