perf(hadoop): Use precompiled/hadoop for faster image builds

[NickLarsenNZ]

Part of #1465

The precompiled/hadoop was done in #1466

[NickLarsenNZ]

Waiting on #1474

[NickLarsenNZ]

Caution

BLOCKER

We've run into an issue. It is mostly a cosmetic thing (albeit important to customers), but we also need to consider what this means in terms of SBOMs and lineage.

Note

I think this didn't appear on my initial PoC because I happened to use 0.0.0-dev until I thought of using a timestamp.

As part of the faster builds work, we compile a version+patchset of some code (eg: what was hadoop/hadoop, but moving to precompiled/hadoop - both of which exist on main). The precompiled/* images aren't tied to a release, but instead use a timestamp.

This requires some changes (see: 0e79589), which leads to the timestamped version in the HDFS web UI.

---

@lfrancke found where in the code the web UI gets the version string from.
You can use these steps to check for yourself.

# Run as root so we can install unzip
docker run --rm -it --user 0 oci.stackable.tech/precompiled/hadoop:3.4.2-stackable1776755865
microdnf update && microdnf install unzip
cd /stackable/hadoop-3.4.2-stackable1776755865/share/hadoop/common
unzip hadoop-common-3.4.2-stackable1776755865.jar
cat common-version-info.properties

common-version-info.properties contains details used in the web UI:

# ...
version=3.4.2-stackable1776755865
revision=08a7206a29f212e1f0e3bd81e0cb0be7907907a4
branch=patchable/3.4.2
user=stackable
date=2026-04-21T07:38Z
url=Unknown
srcChecksum=e573e429c0e2b636ead65e1a9b8bd46a
protocVersion=3.23.4
compilePlatform=linux-x86_64

We could add a light build stage to the product image that extracts the file, updates the version string, and archives it again.

---

This needs further discussion with @lfrancke, @StefanFl, @dervoeti.
Until then, I will stop any work on this.

[NickLarsenNZ]

Just noting that tests pass at 0e79589

Details

cargo boil build hadoop=3.4.2 --strip-architecture
# Yes, I forgot to tell boil to do this
docker tag oci.stackable.tech/sdp/hadoop:3.4.2-stackable0.0.0-dev \
  localboi/hadoop:3.4.2-stackable0.0.0-dev

--- PASS: kuttl (3564.94s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/orphaned-resources_hadoop-latest-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_openshift-false (139.47s)
        --- PASS: kuttl/harness/kerberos_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_krb5-1.21.1_opa-1.12.3_kerberos-realm-CLUSTER.LOCAL_kerberos-backend-mit_openshift-false (1471.72s)
        --- PASS: kuttl/harness/kerberos_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_krb5-1.21.1_opa-1.12.3_kerberos-realm-PROD.MYCORP_kerberos-backend-mit_openshift-false (1502.26s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable_openshift-false (251.15s)
        --- PASS: kuttl/harness/cluster-operation_hadoop-latest-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_openshift-false (245.91s)
        --- PASS: kuttl/harness/topology-provider_hadoop-latest-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_krb5-1.21.1_kerberos-backend-mit_openshift-false (361.85s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-2_datanode-pvcs-default_listener-class-external-unstable_openshift-false (241.20s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-2_datanode-pvcs-default_listener-class-cluster-internal_openshift-false (228.57s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-1_datanode-pvcs-default_listener-class-cluster-internal_openshift-false (167.15s)
        --- PASS: kuttl/harness/logging_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_openshift-false (680.84s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-1_datanode-pvcs-default_listener-class-external-unstable_openshift-false (155.63s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal_openshift-false (170.71s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-1_datanode-pvcs-2hdd-1ssd_listener-class-external-unstable_openshift-false (148.68s)
        --- PASS: kuttl/harness/profiling_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-latest-3.9.4_openshift-false (200.01s)
        --- PASS: kuttl/harness/smoke_hadoop-3.4.2,localboi_hadoop_3.4.2-stackable0.0.0-dev_zookeeper-3.9.4_zookeeper-latest-3.9.4_number-of-datanodes-2_datanode-pvcs-2hdd-1ssd_listener-class-cluster-internal_openshift-false (924.41s)
PASS

[dervoeti]

We could add a light build stage to the product image that extracts the file, updates the version string, and archives it again.

For projects that use Maven, the version is set via mvn versions:set:

docker-images/hadoop/hadoop/Dockerfile

Line 64 in c98be7e

mvn versions:set -DnewVersion=${NEW_VERSION}

I think just patching it in one place (the web UI) would not be sufficient. One reason for the custom versions was that customers are able to pull in our patched libraries from a custom Stackable repository in the future. For that, the JARs have to have the correct metadata and ideally a matching filename. I guess we prefer having the release version in there so it's easier for customers to use a patched lib from a specific SDP release.

Some vague ideas for solutions:

• Use mvn versions:set for the light build. I fear this will trigger a complete recompile, which would defeat the purpose of the faster image builds efforts, but I haven't tried it.
• Use the latest commit hash of docker-images at build time instead of the SDP version as version suffix (makes using it harder for customers though)
• Force a full rebuild during a release, using the SDP version instead of the timestamp
• Not sure if you've tried it but it might be possible to use Harbor as Docker build cache, then everything could work as before (in theory):

     cache-from: type=registry,ref=oci.stackable.tech/my-project/my-app:buildcache
     cache-to: type=registry,ref=oci.stackable.tech/my-project/my-app:buildcache,mode=max
  

  But IIRC we tried this once and there were problems with it.