SPAR is a content-addressed archive format optimized for Python package distribution. It features metadata-at-front layout for efficient dependency resolution, solid streaming compression for superior ratios, and CID-based content addressing for deduplication across package versions.
- Metadata at the front — Package name, version, and dependencies sit at a fixed offset (byte 17), enabling resolvers to read metadata with a single streaming HTTP request without downloading the full archive.
- Solid compression — The entire content block is compressed as a single zstd stream, achieving dramatically better ratios for packages with many small files (e.g., plotly: 9.5MB → 6.6MB solid vs 9.3MB per-file).
- Content addressing — Files are identified by CIDv1 (multicodec 0x55 raw + multihash SHA-256/BLAKE3), enabling deduplication on repositories and locally on disk.
- Local deduplication — A content store with hardlink-based materialization means multiple package versions sharing large files (ML models, GPU kernels) don't consume duplicate disk space.
- One format, multiple distribution types — Binary (
.sparb) and source (.spars) distributions use the same archive structure, distinguished by a format identifier in the header. - Deterministic output — Identical inputs always produce byte-identical archives.
- Security by default — Archive-level checksums, per-entry CID verification, decompression bomb protection, and strict path validation.
| Property | ZIP | tar | wheel (ZIP) | SPAR |
|---|---|---|---|---|
| Metadata without full download | No | No | Partial | Yes (fixed offset) |
| Content deduplication | No | No | No | Yes (CID-based) |
| Deterministic output | No | No | No | Yes |
| Per-entry integrity | CRC-32 | No | CRC-32 | SHA-256/BLAKE3 CID |
| Archive-level checksum | No | No | No | SHA-256 |
| Solid compression | No | Yes (tar.gz) | No | Yes (zstd) |
| Random access by name | Yes | No | Yes | Yes |
| Path traversal hardened | No | No | No | Yes |
| Decompression bomb protection | No | No | No | Yes |
┌─────────────────────────────────────┐ offset 0
│ Header (17 bytes): │
│ Magic (4 bytes): "SPAR" │
│ Version (4 bytes): u32 LE = 1 │
│ Format ID (1-9 bytes): varint │
│ Zero-padding to byte 17 │
├─────────────────────────────────────┤ offset 17 (fixed)
│ Metadata Block: │
│ length: u32 LE │
│ metadata_bytes: JSON object │
│ (name, version, requires_dist, │
│ requires_python, provides_extra)│
├─────────────────────────────────────┤ index_offset
│ Index Block: │
│ entry_count: u32 LE │
│ compression_mode: u8 │
│ (0x00 = per-entry, │
│ 0x01 = solid zstd) │
│ entries[]: IndexRecord[] │
├─────────────────────────────────────┤ content_offset
│ Content Data Block: │
│ Solid: single zstd stream │
│ Per-entry: independent blobs │
├─────────────────────────────────────┤ EOF - 60
│ Footer (60 bytes): │
│ metadata_offset: u64 LE │
│ index_offset: u64 LE │
│ content_offset: u64 LE │
│ checksum: [u8; 32] SHA-256 │
│ magic: [u8; 4] "SPAR" │
└─────────────────────────────────────┘ EOF
Each IndexRecord:
path_length: u16 LE
path_bytes: [u8; path_length] (UTF-8)
cid: [u8; 36] (CIDv1: version + codec + hash_code + digest_len + digest)
content_offset: u64 LE (offset within content block)
raw_size: u64 LE
compressed_size: u64 LE
permissions: u32 LE (Unix mode)
compression: u8 (0x00 = stored, 0x01 = zstd)
| Type | Varint | Extension | Description |
|---|---|---|---|
| Binary Distribution | 0x01 |
.sparb |
Pre-built package (wheel equivalent) |
| Source Distribution | 0x02 |
.spars |
Source requiring build step |
| Generic Archive | 0x03 |
— | Plain content-addressed archive |
0x01 CID version 1
0x55 multicodec: raw binary
0x12 or 0x1e multihash: SHA-256 or BLAKE3
0x20 digest length: 32 bytes
<32 bytes> hash of uncompressed content
spar/
├── Cargo.toml # Workspace root
├── crates/
│ ├── spar-core/ # Core library (format, CID, index, reader, writer, validation)
│ ├── spar-store/ # Content store for local deduplication
│ ├── spar-cli/ # Command-line tool
│ └── spar-ffi/ # C ABI for foreign language bindings
# Build everything
cargo build --workspace --release
# Run tests (including property-based tests)
cargo test --workspace
# The CLI binary is at target/release/spar# Archive a directory (solid compression, SHA-256, generic format)
spar create output.spar ./my-package/
# Binary distribution with BLAKE3
spar create -f binary -a blake3 output.sparb ./dist/
# Source distribution with per-entry compression
spar create -f source -c per-entry output.spars ./src/
# With metadata from a file
spar create -f binary -m METADATA output.sparb ./package/
# Multiple inputs with prefix stripping
spar create --strip-prefix ./build output.spar ./build/dist/# Simple listing
spar list archive.sparb
# Detailed view (path, CID, sizes, permissions, compression)
spar list -l archive.sparb
# JSON output for scripting
spar list --json archive.sparb# Extract everything to current directory
spar extract archive.sparb
# Extract to a specific directory
spar extract archive.sparb -o ./output/
# Extract with local content store for deduplication
spar extract archive.sparb -o ./output/ --store ~/.spar/store
# Use hardlink strategy for shared blobs
spar extract archive.sparb -o ./output/ --store ~/.spar/store --strategy hardlink
# Extract specific entries only
spar extract archive.sparb src/main.py README.mdspar inspect archive.sparbOutput:
SPAR Archive: archive.sparb
Format version: 1
Format identifier: binary
Metadata:
Name: numpy
Version: 1.26.4
Requires-Python: >=3.9
Dependencies:
- setuptools
- cython>=3.0
Index:
Entry count: 42
Compression mode: solid
Footer offsets:
metadata_offset: 17
index_offset: 1234
content_offset: 5678
Archive checksum: a1b2c3d4...
Total raw size: 256000 bytes
Total compressed: 128456 bytes
Compression ratio: 1.99x
| Code | Meaning |
|---|---|
| 0 | Success |
| 1 | General error (I/O, invalid arguments) |
| 2 | Integrity error (corrupted archive, checksum/CID mismatch) |
| 3 | Security error (decompression bomb, path traversal) |
Add spar-core to your Cargo.toml:
[dependencies]
spar-core = { path = "crates/spar-core" }use spar_core::{SparWriter, FormatIdentifier, HashAlgorithm,
DistributionMetadata, ContentBlockCompression};
use std::collections::BTreeMap;
let metadata = DistributionMetadata {
name: "my-package".to_string(),
version: "1.0.0".to_string(),
requires_python: Some(">=3.9".to_string()),
requires_dist: vec!["requests>=2.28".to_string()],
provides_extra: vec!["dev".to_string()],
extra: BTreeMap::new(),
};
let mut writer = SparWriter::new(
FormatIdentifier::BinaryDistribution,
HashAlgorithm::Sha256,
metadata,
);
// Default is solid compression; switch to per-entry if needed:
// writer.set_compression_mode(ContentBlockCompression::PerEntry);
writer.add_entry("src/main.py", b"print('hello')", 0o100644)?;
writer.add_entry("README.md", b"# My Package", 0o100644)?;
let mut output = Vec::new();
let result = writer.finish(&mut output)?;
// result.cids contains the CID for each entry
// result.index contains the full index
std::fs::write("my-package.sparb", &output)?;use spar_core::SparReader;
use std::io::Cursor;
let data = std::fs::read("my-package.sparb")?;
let mut reader = SparReader::open(Cursor::new(data))?;
// Access metadata without extracting
println!("Package: {} v{}", reader.metadata().name, reader.metadata().version);
println!("Format: {:?}", reader.format_id());
// List entries from the index
for record in &reader.index().entries {
println!("{} (CID: {:?}, {} bytes)", record.path, record.cid.digest, record.raw_size);
}
// Read a specific entry (decompresses and verifies CID)
let content = reader.read_entry("src/main.py")?;
// Extract all to a directory
reader.extract_all(std::path::Path::new("./output"), None)?;use spar_core::{SparReader, ReaderConfig};
let config = ReaderConfig {
max_uncompressed_size: 100 * 1024 * 1024, // 100 MiB per entry
max_compression_ratio: 100, // max 100:1 ratio
};
let reader = SparReader::open_with_config(Cursor::new(data), config)?;use spar_store::{LocalStore, ContentStore, MaterializeStrategy};
use spar_core::{HashAlgorithm, Cid};
// Open or create a local content store
let store = LocalStore::open("~/.spar/store".into(), HashAlgorithm::Sha256)?;
// Store a blob (idempotent — no-op if already exists)
let cid = store.store(b"file content", HashAlgorithm::Sha256)?;
// Check existence before extracting
if !store.exists(&cid)? {
// Extract from archive...
}
// Materialize a directory tree from the store using hardlinks
store.materialize(&index, Path::new("./target"), MaterializeStrategy::HardlinkWithFallback)?;The spar-ffi crate produces a shared/static library with a generated C header (spar.h).
cargo build -p spar-ffi --release
# Outputs: libspar_ffi.dylib/.so/.a + crates/spar-ffi/spar.h#include "spar.h"
// Constants
#define SPAR_FORMAT_BINARY 1
#define SPAR_FORMAT_SOURCE 2
#define SPAR_FORMAT_GENERIC 3
#define SPAR_ALGORITHM_SHA256 0
#define SPAR_ALGORITHM_BLAKE3 1
#define SPAR_COMPRESSION_PER_ENTRY 0
#define SPAR_COMPRESSION_SOLID 1
// Writer
SparWriterHandle *spar_writer_new(uint8_t format_id, uint8_t algorithm);
int spar_writer_set_metadata(SparWriterHandle *w, const char *name,
const char *version, const char *requires_python);
int spar_writer_set_compression_mode(SparWriterHandle *w, uint8_t mode);
int spar_writer_add_entry(SparWriterHandle *w, const char *name,
const uint8_t *data, size_t len,
uint32_t permissions, uint8_t compression);
int spar_writer_finish(SparWriterHandle *w, uint8_t **buf, size_t *len);
void spar_writer_free(SparWriterHandle *w);
// Reader
int spar_reader_open(const uint8_t *data, size_t len, SparReaderHandle **out);
uint32_t spar_reader_entry_count(const SparReaderHandle *r);
int spar_reader_entry_name(const SparReaderHandle *r, uint32_t idx, const char **out);
int spar_reader_read_entry(SparReaderHandle *r, const char *name,
uint8_t **data, size_t *len);
const char *spar_reader_metadata_name(const SparReaderHandle *r);
const char *spar_reader_metadata_version(const SparReaderHandle *r);
uint8_t spar_reader_format_id(const SparReaderHandle *r);
void spar_reader_free(SparReaderHandle *r);
// Utilities
void spar_free_buf(uint8_t *buf, size_t len);
const char *spar_last_error(void);#include "spar.h"
#include <stdio.h>
#include <string.h>
int main() {
// Create a binary distribution archive with SHA-256
SparWriterHandle *writer = spar_writer_new(SPAR_FORMAT_BINARY, SPAR_ALGORITHM_SHA256);
spar_writer_set_metadata(writer, "my-package", "1.0.0", ">=3.9");
const char *content = "print('hello')";
spar_writer_add_entry(writer, "main.py",
(const uint8_t *)content, strlen(content), 0100644, 0);
uint8_t *buf = NULL;
size_t len = 0;
spar_writer_finish(writer, &buf, &len);
// Open and read back
SparReaderHandle *reader = NULL;
spar_reader_open(buf, len, &reader);
printf("Package: %s v%s\n",
spar_reader_metadata_name(reader),
spar_reader_metadata_version(reader));
printf("Format: %d\n", spar_reader_format_id(reader));
printf("Entries: %u\n", spar_reader_entry_count(reader));
spar_reader_free(reader);
spar_free_buf(buf, len);
spar_writer_free(writer);
return 0;
}SPAR is designed to be safe on untrusted input:
- Archive integrity: SHA-256 checksum over header + metadata + index + content, verified on open.
- Per-entry integrity: CID (SHA-256 or BLAKE3) of each entry's uncompressed content is verified on extraction.
- Path traversal prevention: No
.., no leading slashes, no backslashes, no null bytes, no control characters (0x00-0x1F), no Windows-reserved device names. - Decompression bomb protection: Configurable max uncompressed size (default 4 GiB) and max compression ratio (default 1000:1).
- Canonicalization: Strict lexicographic order, no duplicates, single valid representation.
- No dangerous metadata: No UIDs, GIDs, xattrs, ACLs, symlinks, device nodes, or setuid/setgid.
- Format evolution via version only: No extension points or optional fields.
| Limit | Value |
|---|---|
| Max entries per archive | 16,777,216 (2²⁴) |
| Max entry path length | 4,096 bytes |
| Max uncompressed entry size (default) | 4 GiB |
| Max compression ratio (default) | 1000:1 |
| Header size (fixed) | 17 bytes |
| Footer size (fixed) | 60 bytes |
| CID size (fixed) | 36 bytes |
Metadata-first. Resolvers can read package metadata (name, version, dependencies) with a single HTTP request for the first ~1KB of the archive. No range requests, no full download.
Reproducibility. Entries sorted lexicographically, deterministic compression (zstd level 3, single-threaded, no dictionary), no timestamps or environment-dependent data. Same inputs → same bytes.
Content addressing. Every file is identified by its CID. Identical content across package versions shares storage both on repositories and locally on disk.
Security by default. The reader validates everything before exposing any data. Malformed archives are rejected early and completely.
Simplicity. One archive structure for all distribution types. Two compression modes (solid, per-entry). Two hash algorithms (SHA-256, BLAKE3). No optional features.
MIT