GH-55478: [R] Vendored dependencies to enable parquet (smallest change, biggest tarball) by jonkeane · Pull Request #55491 · apache/arrow

jonkeane · 2026-11-11T01:38:02Z

Rationale for this change

Vendor (and trim) C++ dependencies for offline builds that would allow us to build Parquet even when CMake cannot reach the internet. This is an R-package-only change, but in doing so can't trim quite as much (specifically building thrift without boost). The resulting tarball is larger than strictly necessary (~13.1 MB).

Resolves #55478

What changes are included in this PR?

Scripts for vendoring and trimming dependencies, tests

Are these changes tested?

Yes

Are there any user-facing changes?

More full-feature arrow builds

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [R] Vendor trimmed thirdparty sources so offline (CRAN) source builds can include Parquet #55478

github-actions · 2026-11-11T01:38:28Z

⚠️ GitHub issue #55478 has been automatically assigned in GitHub to PR creator.

jonkeane · 2026-11-11T01:40:46Z

@github-actions crossbow submit test-r-offline-vendored

Copilot

🟡 Changes recommended

The build leaves rewritten checksums in tools/cpp/thirdparty/versions.txt, causing subsequent installs from the same source tree with upstream archives to fail checksum validation.

1 open issue

Restore versions.txt checksum after the build · New

What changed in this PR

Enables more capable offline R source builds, including Parquet, by shipping trimmed third-party dependencies alongside the bundled Arrow C++ source.

Changes:

Adds dependency caching, trimming, and upstream/vendored checksum tracking.
Preserves optional features whose dependencies are available offline.
Expands offline CI validation and records dependency licensing information.

File	Description
`r/tools/vendor_dependencies.sh`	Downloads, caches, trims, and records checksums for vendored dependencies.
`r/tools/trim/boost.sh`	Retains the Boost libraries needed by R builds.
`r/tools/test-nixlibs.R`	Tests dependency selection, checksum rewriting, and offline feature handling.
`r/tools/nixlibs.R`	Selects compatible vendored archives and preserves supported offline features.
`r/inst/COPYRIGHTS`	Documents vendored dependency licenses and copyrights.
`r/DESCRIPTION`	References the copyright information.
`r/.Rbuildignore`	Excludes download caches and trimming scripts from source packages.
`r/.gitignore`	Ignores cached upstream archives.
`dev/tasks/tasks.yml`	Configures offline CI to exercise vendored dependencies and expected features.
`dev/release/rat_exclude_files.txt`	Excludes the copyright inventory from license-header checks.
`ci/scripts/r_test.sh`	Verifies expected build capabilities and allocator backends.

_{Review effort: Balanced}

github-actions · 2026-11-11T01:43:28Z

Revision: dda15ba

Submitted crossbow builds: ursacomputing/crossbow @ actions-7820f0a5a5

Task	Status
test-r-offline-vendored

jonkeane · 2026-11-11T13:08:14Z

+        -e NOT_CRAN=false
+        -e SKIP_VIGNETTES=true
+        -e ARROW_SOURCE_HOME=''
+        -e ARROW_WITH_BZ2=ON


I included bz2 here not because I think it's useful, but it's actually tiny (0.09MB). But we could save some bytes by removing it.

jonkeane · 2026-11-11T13:09:39Z

@@ -0,0 +1,205 @@
+#!/usr/bin/env bash


I am mostly relying on the fact that our tests pass to confirm that this file is doing what we expect it to. But it does seem to work. If someone is interested in going over it with a fine(r) tooth comb, that would be appreciate, though given that it's part of our build system, I'm not sure that's required.

Copilot

🟡 Changes recommended

Critical issues remain: the trimmed Boost archive breaks bundled UBSan builds by excluding multiprecision, and persisted checksum rewrites cause subsequent builds using upstream archives to fail hash validation.

3 open issues

_{Review effort: Balanced}

+#   needs it uses this tarball, configuring libarrow fails in build_boost()
+#   because the boost_multiprecision target doesn't exist.
+# - beast, variant, xpressive: only for ARROW_FLIGHT_SQL_ODBC.
+EXCLUDED_LIBRARIES=(multiprecision beast variant xpressive)


jonkeane · 2026-11-11T13:18:20Z

+# Returns the paths of the vendored tarballs that the libarrow source in
+# `src_dir` can use, named by their ARROW_*_URL.
+# The tarballs are trimmed, so the libarrow build only accepts them once
+# `use_vendored_checksums()` has put their checksums in versions.txt. We only
+# do that to the package's own copy of the source (tools/cpp), not e.g. ../cpp
+# of a git checkout, and only for tarballs that were trimmed from the version
+# that versions.txt pins (according to the checksums.txt file that
+# tools/vendor_dependencies.sh writes).
+usable_vendored_dependencies <- function(src_dir) {


Checking the vendored checksums is honestly a little bit silly — we compute them ourselves and include them with the package, so it's not really an independent verification (if someone wanted to make a change to them and distribute the source package, they could also just update the hash). But I do think it's ok to have as an integrity check or if we mess up something in our vendoring it'll fall back to downloading.

We also already do an integrity check when we download the dependencies before trimming.

Copilot

🔵 Needs a closer look

The trimmed Boost tarball omits Multiprecision but is still selected for UBSAN builds that require it, causing configuration to fail even when online.

1 open issue

UBSan builds fail when bundled Boost omits multiprecision

2 resolved since last review

_{Review effort: Balanced}

jonkeane · 2026-11-11T13:21:01Z

I would love anyone's thoughts on this who has been tagged already, but I would especially love your thoughts on this @kou (and if this or the similar, but more far-reaching changes in #55492 would actually be better? It makes a much smaller tarball but with more to maintain and possible impacts for other C++ implementations)

jonkeane · 2026-11-11T16:18:39Z

@github-actions crossbow submit test-r-linux-sanitizers test-r-m1-san

github-actions · 2026-11-11T16:20:56Z

Revision: 11267e0

Submitted crossbow builds: ursacomputing/crossbow @ actions-93b6ed324a

Task	Status
test-r-linux-sanitizers
test-r-m1-san

Copilot

🟢 Approval recommended

No active review issues are reported for the offline dependency-vendoring changes.

1 open issue

UBSan builds fail when bundled Boost omits multiprecision

_{Review effort: Balanced}

…ecs, compute and JSON Ship trimmed source tarballs of the thirdparty dependencies needed for Parquet, the compression codecs, RE2/utf8proc, JSON and mimalloc in the R package so that offline source builds (e.g. on CRAN) can include those features instead of falling back to a minimal build. Brotli and Zstandard aren't vendored (yet) to keep the package small, so offline builds turn them off. * tools/vendor_dependencies.sh (run by `make sync-cpp`) downloads, trims and vendors the tarballs into tools/vendored, and records each one's upstream and trimmed checksums in tools/vendored/checksums.txt. A dependency can provide its own trim script at tools/trim/<name>.sh. * When building the package's copy of the C++ source (tools/cpp), nixlibs.R uses the vendored tarballs that were trimmed from the versions that versions.txt pins, replacing their checksums in versions.txt with the trimmed ones. When offline, it only turns off the features whose dependencies aren't vendored. * The offline CI job (now test-r-offline-vendored) checks that the expected features are available after the build. * Add inst/COPYRIGHTS listing the vendored dependencies' licenses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…eeds The generic trim leaves a 29 MB Boost tarball. tools/trim/boost.sh keeps only the Boost libraries that build_boost() in ThirdpartyToolchain.cmake includes (minus ones the R build doesn't use, e.g. multiprecision) and their dependencies, as resolved by Boost's own CMake, and drops their tests, docs and examples. This brings the vendored Boost down to ~5 MB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

🔵 Needs a closer look

Interrupted builds can leave vendored checksums in versions.txt that are restored unchanged, causing later builds using upstream tarballs to fail checksum verification.

1 open issue

UBSan builds fail when bundled Boost omits multiprecision

_{Review effort: Balanced}

jonkeane · 2026-11-12T00:04:41Z

@github-actions crossbow submit test-r-linux-sanitizers test-r-m1-san

github-actions · 2026-11-12T00:06:49Z

Revision: d70832b

Submitted crossbow builds: ursacomputing/crossbow @ actions-c59b7f870a

Task	Status
test-r-linux-sanitizers
test-r-m1-san

kou

I prefer this to #55492 because #55492 includes code that is only for R in cpp/.

kou · 2026-11-12T07:14:50Z

+  # Some of them are required even for a minimal build, so they can't be turned
+  # off like the ones above.
+  env_var_list <- with_vendored_dependencies(env_var_list, vendored)
+  original_versions <- use_vendored_checksums(src_dir, env_var_list, vendored)


We can add support for changing ARROW_XXX_BUILD_SHA256_CHECKSUM by environment variables in cpp/ like we did for ARROW_XXX_URL. For example, we can change ARROW_XSIMD_URL by an environment variable:

arrow/cpp/cmake_modules/ThirdpartyToolchain.cmake

Lines 847 to 848 in 7eb865c

if(DEFINED ENV{ARROW_XSIMD_URL})

set(XSIMD_SOURCE_URL "$ENV{ARROW_XSIMD_URL}")

Oh, yes that's nice. I'll do that + update here.

Copilot

🟢 Approval recommended

No active review issues are recorded for the changes enabling richer offline R source builds with vendored dependencies.

1 open issue

UBSan builds fail when bundled Boost omits multiprecision

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The generated trimmed dependency archives are absent, so confirming that host-determined pruning preserves platform-specific build inputs requires human review of offline source-build results across supported platforms.

1 open issue

UBSan builds fail when bundled Boost omits multiprecision

_{Review effort: Balanced}

jonkeane requested review from assignUser, kou, raulcd and thisisnic as code owners November 11, 2026 01:38

Copilot AI balanced review requested due to automatic review settings November 11, 2026 01:38

github-actions Bot added the Component: R label Nov 11, 2026

Copilot started reviewing on behalf of jonkeane November 11, 2026 01:38

View session

jonkeane mentioned this pull request Nov 11, 2026

[R] Vendor trimmed thirdparty sources so offline (CRAN) source builds can include Parquet #55478

Open

Copilot AI reviewed Nov 11, 2026

View reviewed changes

Comment thread r/tools/nixlibs.R Resolved

jonkeane force-pushed the vendored_parquet branch from dda15ba to 5ad1f96 Compare November 11, 2026 13:06

Copilot AI balanced review requested due to automatic review settings November 11, 2026 13:06

Copilot started reviewing on behalf of jonkeane November 11, 2026 13:07

View session

jonkeane commented Nov 11, 2026

View reviewed changes

Copilot AI reviewed Nov 11, 2026

View reviewed changes

jonkeane force-pushed the vendored_parquet branch from 5ad1f96 to 11267e0 Compare November 11, 2026 13:17

Copilot AI balanced review requested due to automatic review settings November 11, 2026 13:17

Copilot started reviewing on behalf of jonkeane November 11, 2026 13:18

View session

jonkeane commented Nov 11, 2026

View reviewed changes

Copilot AI reviewed Nov 11, 2026

View reviewed changes

jonkeane force-pushed the vendored_parquet branch from 11267e0 to e821d72 Compare November 11, 2026 16:51

Copilot AI balanced review requested due to automatic review settings November 11, 2026 16:51

Copilot started reviewing on behalf of jonkeane November 11, 2026 16:54

View session

Copilot AI reviewed Nov 11, 2026

View reviewed changes

jonkeane and others added 2 commits November 11, 2026 17:57

jonkeane force-pushed the vendored_parquet branch from e821d72 to d70832b Compare November 11, 2026 23:57

Copilot AI balanced review requested due to automatic review settings November 11, 2026 23:57

Copilot started reviewing on behalf of jonkeane November 11, 2026 23:57

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

kou reviewed Nov 12, 2026

View reviewed changes

Use env vars for overriding shas

aebc626

Copilot AI balanced review requested due to automatic review settings November 12, 2026 14:29

github-actions Bot added Component: C++ Component: Documentation labels Nov 12, 2026

Copilot started reviewing on behalf of jonkeane November 12, 2026 14:30

View session

Copilot AI reviewed Nov 12, 2026

View reviewed changes

Local review iteration

dcab783

Copilot AI balanced review requested due to automatic review settings November 13, 2026 20:01

Copilot started reviewing on behalf of jonkeane November 13, 2026 20:02

View session

Copilot AI reviewed Nov 13, 2026

View reviewed changes

	if(DEFINED ENV{ARROW_XSIMD_URL})
	set(XSIMD_SOURCE_URL "$ENV{ARROW_XSIMD_URL}")

Conversation

jonkeane commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

jonkeane commented Nov 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

jonkeane Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

jonkeane Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

jonkeane Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

jonkeane commented Nov 11, 2026

Uh oh!

jonkeane commented Nov 11, 2026

Uh oh!

github-actions Bot commented Nov 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

jonkeane commented Nov 12, 2026

Uh oh!

github-actions Bot commented Nov 12, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

jonkeane Nov 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

jonkeane commented Nov 11, 2026 •

edited

Loading