Skip to content

Hive 4.0.1 support #626

Closed
Closed
Hive 4.0.1 support#626
Assignees
razvan
Labels
release/25.11.0

Description

@lfrancke
lfrancke
opened on Aug 12, 2025

Description

We want to upgrade our Apache Hive we ship from version 4.0.0 to 4.0.1 as Hive 4.0.x is our LTS line.
Currently we mark 4.0.1 as experimental due to issues we encountered when we tried it.

Value

Hive 4.0.1 contains bug fixes and security patches that our customers need. Since we're committed to supporting the 4.0.x line for LTS reasons for at least until the 26.3 release, we need to provide the most stable version available in that line.

Dependencies

Tasks

Acceptance Criteria

  • Trino integration works properly or documented limitations/workarounds are provided
  • All integration tests pass
  • All demos using Hive work

Activity

  1. lfrancke
    added this to Stackable End-to-End Coordinationon Aug 12, 2025
  2. lfrancke
    moved this to Selected for Development in Stackable End-to-End Coordinationon Aug 12, 2025
  3. lfrancke
    added
    scheduled-for/25.11.0
    on Aug 12, 2025
  4. razvan
    self-assigned this
    on Aug 28, 2025
  5. razvan
    added this to Stackable Engineeringon Aug 28, 2025
  6. razvan
    moved this to Development: In Progress in Stackable Engineeringon Aug 28, 2025

  7. razvan commented on Aug 28, 2025

    @razvan
    razvan
    on Aug 28, 2025
    Member

    There seems to be no easy workaround for any of these problems. A proper fix upstream requires the Iceberg catalog model to change as described below.

    Spark/Iceberg/Hive

    Investigate the demo issue in stackabletech/demos#173

    Problem

    When using Hive 4.0.1 for the hive-iceberg-metastore the Spark application spark-ad fails with the error:

    $ k logs spark-ad-driver
...
2025-08-29T07:49:01,678 INFO [Thread-4] hive.metastore - Connected to metastore.
Traceback (most recent call last):
  File "/spark-scripts/spark-ad.py", line 13, in <module>
    spark.sql("CREATE TABLE IF NOT EXISTS prediction.ad.iforest (pickup_ts TIMESTAMP, pickup_minute_group VARCHAR(4), pickup_hour INT, pickup_year INT, pickup_month INT, pickup_dayofmonth INT, pickup_dayofweek INT, norides INT, total_bill DOUBLE, avg_bill DOUBLE, norides_lag INT, pred INT) USING iceberg")
  File "/stackable/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1631, in sql
  File "/stackable/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/stackable/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/stackable/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o77.sql.
: java.lang.RuntimeException: Failed to get table info from metastore ad.iforest
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:160)
...
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table'
...

    This is problem with Iceberg and there is an open issue for it: apache/iceberg#12878

    Workaround

    Patch and build Hive 2 as described in this comment . Then replace the Hive jars shipped by Spark.

    Spark 3.5 ships Hive 2.3.9, while Spark 4.0 ships Hive 2.3.10.

    See this PR for an attempt to build Hive 2.

    Trino/Hive

    Understand the Trino integration problem (trinodb/trino#26214)

    This comment explains the problem. Hive 4 added a new type for timestamp fields that is not supported by Trino (as of v476).

    Workaround

    ???

  8. sbernauer
    moved this from Development: In Progress to Refinement: In Progress in Stackable Engineeringon Sep 1, 2025
  9. razvan
    moved this from Refinement: In Progress to Refinement Acceptance: Waiting for in Stackable Engineeringon Sep 1, 2025

  10. sbernauer commented on Sep 3, 2025

    @sbernauer
    sbernauer
    on Sep 3, 2025
    Member

    Just throwing in spark.sql.hive.metastore.jars.path

    Location of the jars that should be used to instantiate the HiveMetastoreClient

    It maybe won't help, but I wanted to mention that property

  11. lfrancke
    moved this from Selected for Development to In Refinement in Stackable End-to-End Coordinationon Sep 3, 2025

  12. lfrancke commented on Sep 9, 2025

    @lfrancke
    lfrancke
    on Sep 9, 2025
    MemberAuthor

    Okay. My current plan would be: Continue shipping Hive 3 and 4 at the same time.

    • Spark/Iceberg/Hive is not something we can untangle anytime soon even if we wanted to, and we'll need to jump to later Hive versions than 4 at some point
    • Trino issue is something we might be able to fix upstream but that's also not going to be trivial

    So, my suggestion: Follow the relevant upstream discussions and jump in and help if we can and in the meantime clearly document where Hive 3 is still needed and what works/doesn't work with Hive 4. If that is fine then I'd make two issues out of this:

    • Update Hive to 4.0.1
    • Document limitations

    Opinions?

  13. lfrancke
    moved this from Refinement Acceptance: Waiting for to Refinement Acceptance: In Progress in Stackable Engineeringon Sep 9, 2025

  14. sbernauer commented on Sep 10, 2025

    @sbernauer
    sbernauer
    on Sep 10, 2025
    Member

    I think what would be great is to have a story on downgrading. E.g. users bump 3.6.3 -> 4.0.0 or 4.0.0 -> 4.0.1 and notice that things broke.
    Can they downgrade again? I recall for some tools the database migration script would panic, not sure about HMS.
    I think we should add docs on downgrading as part of this issue, WDYT?

  15. lfrancke
    moved this from In Refinement to In Progress in Stackable End-to-End Coordinationon Sep 25, 2025

  16. lfrancke commented on Oct 2, 2025

    @lfrancke
    lfrancke
    on Oct 2, 2025
    MemberAuthor

    We just clarified in the lifecycle policies (or will) that downgrades are not supported.
    It'd be nice to have the support but we currently just don't have the resources to test all of those paths :(

  17. sbernauer
    moved this from Refinement Acceptance: In Progress to Development: Done in Stackable Engineeringon Oct 6, 2025
  18. sbernauer
    moved this from In Progress to Done in Stackable End-to-End Coordinationon Oct 6, 2025
  19. razvan
    closed this as completedon Oct 13, 2025
  20. lfrancke
    added
    release/25.11.0
    and removed
    scheduled-for/25.11.0
    on Oct 14, 2025
  21. lfrancke
    moved this from Development: Done to Done in Stackable Engineeringon Oct 14, 2025

  22. sbernauer commented on Nov 26, 2025

    @sbernauer
    sbernauer
    on Nov 26, 2025
    Member

    I think what would be great is to have a story on downgrading. E.g. users bump 3.6.3 -> 4.0.0 or 4.0.0 -> 4.0.1 and notice that things broke.
    Can they downgrade again? I recall for some tools the database migration script would panic, not sure about HMS.
    I think we should add docs on downgrading as part of this issue, WDYT?

    A customer was running into exactly this problem.
    They upgraded from HMS 4.0.0 to 4.1.0, so the HMS ran the database migration from https://github.com/apache/hive/blob/6b361ac3582a18852a6f090a9a20f522b84c8135/standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-4.0.0-to-4.1.0.postgres.sql

    Some statements are especially breaking, namely

    ALTER TABLE "TAB_COL_STATS" DROP COLUMN "CAT_NAME", DROP COLUMN "DB_NAME", DROP COLUMN "TABLE_NAME";
ALTER TABLE "PART_COL_STATS" DROP COLUMN "CAT_NAME", DROP COLUMN "DB_NAME", DROP COLUMN "TABLE_NAME", DROP COLUMN "PARTITION_NAME";
UPDATE "VERSION" SET "SCHEMA_VERSION"='4.1.0', "VERSION_COMMENT"='Hive release version 4.1.0' where "VER_ID"=1;

    They drop columns, which can't be restored.

    Luckily, in the particular case of the customer, the TAB_COL_STATS and PART_COL_STATS tables where both empty, so we where able to re-add the columns (empty):

    ALTER TABLE "PART_COL_STATS"
    ADD COLUMN "CAT_NAME" character varying(256) DEFAULT NULL::character varying,
    ADD COLUMN "DB_NAME" character varying(128) DEFAULT NULL::character varying,
    ADD COLUMN "TABLE_NAME" character varying(256) DEFAULT NULL::character varying,
    ADD COLUMN "PARTITION_NAME" character varying(767) DEFAULT NULL::character varying;

ALTER TABLE "TAB_COL_STATS"
    ADD COLUMN "CAT_NAME" character varying(256) DEFAULT NULL::character varying,
    ADD COLUMN "DB_NAME" character varying(128) DEFAULT NULL::character varying,
    ADD COLUMN "TABLE_NAME" character varying(256) DEFAULT NULL::character varying;

UPDATE "VERSION" SET "SCHEMA_VERSION"='4.0.0', "VERSION_COMMENT"='Hive release version 4.0.0, manually downgraded from 4.1.0 again';

    That way we brought the Hive metastore 4.0.0 back to live, but this is not recommended.
    I created stackabletech/documentation#810 to warn users more prominently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

release/25.11.0

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions