Run SQL from anything that speaks HTTP. Get Arrow back.

---

⚡ What is Warp SQL?

Warp SQL is an open protocol that lets any HTTP client run SQL against a database server and get Apache Arrow data back: a browser fetch, curl, or any language's standard HTTP library. No gRPC, no protobuf, no generated code.

• 🏹 Arrow on the wire. Results, query parameters and bulk uploads are Arrow IPC streams (application/vnd.apache.arrow.stream), so every Arrow library reads them directly. Control messages are small JSON documents.
• 🚀 Short queries stay one round trip. One POST runs a query and streams its result.
• 🔀 Big results go wide. A client can ask for a large result as several partitions and read them over separate connections at once, past what one TCP stream can carry, with the order kept when the query has one. Warp SQL's drivers do it by default (8 partitions).
• 📤 Big uploads go wide too. One bulk ingest can be sent as several parts over separate connections at once and committed as one transaction, all or nothing.
• 🌐 Long queries survive real networks. Submit a query, get a handle, poll it and fetch the result later. A dropped download resumes from the batch where it stopped, and no request has to outlive a proxy timeout.
• 🔐 Safe by construction. Every handle is bound to the identity that created it and checked on every use. Credentials are bearer tokens, never cookies. Results and metadata follow the caller's authorization.
• 🧩 Engine-neutral and open. Apache-2.0 licensed: any engine can implement it, and a black-box conformance suite tells you when you have.

👀 A quick look

# Get a token, open a session, run a query, read the result with pyarrow.
TOKEN=$(curl -s https://db.example.com/warp/v1/auth/token \
  -H 'Content-Type: application/json' \
  -d '{"grant_type":"password","username":"joe","password":"…"}' | jq -r .access_token)

SESSION=$(curl -s https://db.example.com/warp/v1/sessions \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{}' | jq -r .session_id)

curl -s https://db.example.com/warp/v1/sessions/$SESSION/statements \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -H 'Accept: application/vnd.apache.arrow.stream' \
  -d '{"sql":"SELECT 42 AS answer"}' > result.arrows

python -c "import pyarrow.ipc as ipc; print(ipc.open_stream(source='result.arrows').read_all())"

Every server describes itself at GET /warp, so a client knows what it may use:

{
  "protocol": "warp-sql",
  "protocol_versions": ["1"],
  "server": {"name": "warp-duckdb", "version": "1.0.1 (DuckDB v2.0.0)"},
  "capabilities": ["prepared_statements", "transactions", "ingest", "partitioned_ingest",
                   "long_poll", "ipc_compression_zstd", "ipc_compression_lz4_frame",
                   "result_partitions", "openapi"],
  "auth": {"grant_types": ["password"]},
  "links": {"openapi": "/warp/v1/openapi.json"}
}

and serves the protocol's OpenAPI 3.1 description at GET /warp/v1/openapi.json, for API browsers and client generators.

(Both run against warp-duckdb, the reference server in cpp/.)

🔄 How it works

sequenceDiagram
  participant C as Client
  participant S as Warp SQL server
  C->>S: POST /warp/v1/auth/token
  S-->>C: bearer token
  C->>S: POST /warp/v1/sessions
  S-->>C: session_id
  alt short query: one round trip
    C->>S: POST …/statements  (Accept: Arrow stream)
    S-->>C: Arrow IPC stream
  else long query: submit, poll, fetch
    C->>S: POST …/statements  (Prefer: respond-async)
    S-->>C: 201 statement document
    C->>S: GET …/statements/{id}  (long-poll)
    S-->>C: succeeded: result partitions
    C->>S: GET …/statements/{id}/result?from_batch=N
    S-->>C: Arrow IPC stream (resumable)
  end

Loading

📦 What the protocol covers

	Spec
🔎 Discovery, health and capabilities	§3
📜 An OpenAPI 3.1 description, served at GET /warp/v1/openapi.json	§3.3, openapi.yaml
🔑 Token sign-in (password, token exchange), browser sign-in	§4
🗂️ Sessions with options, and transactions	§5
▶️ Statements: streamed or submit/poll/fetch, positional and named bind parameters	§6
♻️ Retained, resumable results; errors mid-stream; idempotent submission	§6.5–6.7
🛑 Cancellation (statement and session) and timeouts	§6.8–6.9
🔀 Partitioned results, read concurrently over several connections, in order or not	§6.10
🧪 Prepared statements with JSON or Arrow parameter batches	§7
📥 Streaming bulk ingest	§8
🗜️ Arrow buffer compression, lz4_frame or zstd, negotiated by preference	§8.2
📤 Partitioned ingest: one upload over several connections, committed atomically	§8.3
🔐 Security model: principals, bound handles, governed results	§9
📚 Metadata with Flight SQL's result schemas, so driver code carries over	§10
⚠️ Problem-document errors and graceful shutdown	§11
🧭 Versioning rules	§12

Results are described as partitions, each with a location. Today a server splits a large result into partitions on itself, so a client reads it over several connections at once (§6.10). Tomorrow a distributed engine can point partitions at other machines, without changing the protocol (docs/distributed.md).

🌍 Implementations

	Project
Server	Hardcharger: GizmoData's DuckDB 2.0 server
Reference server	warp-duckdb (this repo): DuckDB 2.0 through its v2 C API; for testing and development only, not production
Server SDK	libwarpsql (C++20, this repo): engines implement a Backend; the SDK owns authentication, handle binding, retention, cancellation and draining, and streams every result through an asynchronous pipeline (produce, compress on a worker pool, encrypt and send, each stage overlapped)
ADBC driver	warp-sql-adbc: Go, with Python bindings, loadable by any ADBC driver manager
JDBC driver	warp-sql-jdbc: one jar for Java 17+, no gRPC or Netty
JavaScript client	warp-sql-js: @gizmodata/warp-sql, TypeScript over plain fetch and Arrow JS, for Node, browsers, Deno, Bun and Cloudflare Workers

JavaScript

A Warp SQL server needs nothing but fetch from JavaScript. With @gizmodata/warp-sql, rows come back as an Arrow JS Table (or stream batch by batch), and resumable downloads, idempotent retries, cancellation through an AbortSignal and split results come with it:

import { connect } from "@gizmodata/warp-sql";

const conn = await connect({ url: "warp://localhost:31337?tls=false", username: "joe", password: "s3:cret" });
const table = await conn.query("SELECT 42 AS answer, ? AS greeting", ["hello"]);
console.log(table.toArray().map((row) => row.toJSON()));
await conn.close();

[ { answer: 42, greeting: 'hello' } ]

(Run against warp-duckdb --username joe, with WARP_DUCKDB_PASSWORD='s3:cret'.)

✅ Conformance

The suite in conformance/ is black-box: it talks to a server only through the protocol, so any implementation can run it. It starts the server itself (and fails, never skips, without one):

cd conformance
uv venv .venv && uv pip install --python .venv/bin/python -e .
WARP_SQL_TEST_SERVER_BINARY=../cpp/build/warp-duckdb .venv/bin/python -m pytest -q

For another server, tell the suite which variable carries its administrator's password (WARP_SQL_TEST_SERVER_PASSWORD_ENV), the user name (WARP_SQL_TEST_SERVER_USERNAME) and the flag naming its client CA for the mutual-TLS tests (WARP_SQL_TEST_SERVER_MTLS_FLAG, default --tls-client-ca; the other --mtls-* flags are the same on every server), or point it at a running server with WARP_SQL_TEST_URL (and -m "not mtls": those tests start servers of their own).

🛠️ Build the SDK and reference server

cd cpp
./scripts/build-duckdb.sh     # DuckDB 2.0 at the pinned commit
./scripts/build-arrow.sh      # Arrow C++, IPC only
OPENSSL_ROOT_DIR=$(brew --prefix openssl@3) cmake -S . -B build -G Ninja
cmake --build build
WARP_DUCKDB_PASSWORD='…' ./build/warp-duckdb --database my.duckdb --port 31337

See cpp/README.md for TLS, mTLS and the other options.

🗺️ Status and roadmap

1.0.1. Version 1 is stable: it changes only by addition (new endpoints, fields, capabilities and error codes, §12), so a client written for 1.0.0 keeps working.

1. Spec draft.
2. Spike: which Arrow library does the IPC, checked value by value for every DuckDB type. Arrow C++ with IPC only (results).
3. Reference server SDK (C++) and conformance suite.
4. ADBC (Go, Python) and JDBC drivers.
5. A production server on the SDK: Hardcharger, which runs GizmoSQL 1.x's test battery over Warp SQL alone.
6. Parallel multi-partition results (§6.10, with the SDK's pipelined streaming).
7. A JavaScript client (warp-sql-js) and partitioned uploads (§8.3).
8. Spec 1.0.0. Appendix A lists what may come next, each behind a capability.

📂 In this repo

Path	What
spec/warp-sql.md	The protocol specification
spec/openapi.yaml	Its OpenAPI 3.1 description (linted by spec/scripts/lint-openapi.sh)
docs/server-sdk.md	Design of the C++ server SDK
docs/distributed.md	How partitioned results leave room for distributed query
cpp/	libwarpsql and warp-duckdb
conformance/	The conformance suite, and the MUSTs it doesn't test yet
site/	The protocol's site: the spec, every flow with a sequence diagram and commands that run, the OpenAPI reference in Swagger UI
CHANGELOG.md	Changes by version

🏹 Relationship to Apache Arrow

Warp SQL is an independent project; it is not part of, or endorsed by, the Apache Software Foundation. It uses the Arrow IPC format and borrows Arrow Flight SQL's metadata schemas so existing driver code carries over. The Arrow community's HTTP data transport work is the reference for how Arrow streams should travel over HTTP.

Apache Arrow, Arrow, Apache, the Apache logo, and the Apache Arrow project logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

📄 License

Apache License 2.0; see LICENSE.

---

_{Warp SQL is developed by GizmoData}

---

_{Hardcharger™ and GizmoData™ are trademarks of GizmoData LLC.}