Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -431,25 +431,6 @@ steps = [
], comment = "Failed stores did not publish a secondary association." },
]

[[e2e]]
name = "command_wrapper"
cfg = "not(windows)"
ignore = true
steps = [
{ argv = [
"remote-cache-server",
"node",
"-e",
"console.log(new URL(process.env.VP_REMOTE_CACHE_URL).pathname)",
], comment = "The endpoint includes a namespace base path." },
{ argv = [
"remote-cache-server",
"vtt",
"exit",
"3",
], comment = "The wrapper exits with the command's exit code." },
]

[[e2e]]
name = "state_across_invocations"
cfg = "not(windows)"
Expand Down

This file was deleted.

4 changes: 2 additions & 2 deletions packages/tools/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ remote-cache-server cbor-http POST /store --form-cbor "metadata={\"key\": 'A', \
remote-cache-server cbor-http POST /fetch --cbor "{\"key\": 'A', \"secondary_key\": 'S'}"
```

`remote-cache-server COMMAND [ARGS...]` starts the backend on a free loopback port and runs the command with `VP_REMOTE_CACHE_URL` set to the endpoint, `http://127.0.0.1:<port>/projects/test`. The fixed base path gives every endpoint a namespace path. The wrapper takes no options and passes all arguments to the command unchanged. The command inherits stdio. When it exits, the server stops and the wrapper exits with the command's exit code, or 128 plus the signal number if a signal ended it. The terminal delivers Ctrl-C to the command too, so the wrapper keeps serving until the command exits. The wrapper forwards `SIGTERM` to the command.
`remote-cache-server COMMAND [ARGS...]` starts the backend on a free loopback port and runs the command with `VP_REMOTE_CACHE_URL` set to the endpoint, `http://127.0.0.1:<port>/projects/test`. The fixed base path gives every endpoint a namespace path. The wrapper takes no options and passes all arguments to the command unchanged. The command inherits stdio. When it exits, the server stops and the wrapper exits with the command's exit code.

State persists in `remote-cache/` in the current directory, so consecutive commands share it. Each E2E case has its own directory and state. `state.json` holds the entries, associations, and next blob ID, with keys and values hex-encoded. Each blob is a file in `remote-cache/blobs/` named by its blob ID. Blob IDs are sequential strings and continue across invocations, keeping snapshots deterministic.

The backend implements `POST /fetch`, `POST /store`, and `GET /blob/{blob_id}` from the [remote cache server API](https://github.com/voidzero-dev/vite-task/pull/713). Keys, values, and blobs are opaque bytes without length limits. There is no authentication.

## CBOR HTTP client

`cbor-http METHOD PATH` joins the path onto `VP_REMOTE_CACHE_URL`, including its base path, so run it through `remote-cache-server`. An absolute HTTP URL works without the variable. Request bodies use [CBOR extended diagnostic notation (EDN)](https://www.rfc-editor.org/rfc/rfc8610.html#appendix-G): `"text"` is text, `'bytes'` is a UTF-8 byte string, and `b64'AP+A'` contains arbitrary binary bytes.
`cbor-http METHOD PATH` appends the path to `VP_REMOTE_CACHE_URL`, so run it through `remote-cache-server`. Request bodies use [CBOR extended diagnostic notation (EDN)](https://www.rfc-editor.org/rfc/rfc8610.html#appendix-G): `"text"` is text, `'bytes'` is a UTF-8 byte string, and `b64'AP+A'` contains arbitrary binary bytes.

| Option | Body |
| ----------------------- | --------------------------------------------------------- |
Expand Down
8 changes: 1 addition & 7 deletions packages/tools/src/cbor-http.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,7 @@ async function main(): Promise<void> {
contentType = `multipart/form-data; boundary=${boundary}`;
}
contentType = values['content-type'] ?? contentType;
let url = path;
if (!/^https?:\/\//.test(path)) {
const endpoint = process.env['VP_REMOTE_CACHE_URL'];
if (!endpoint) throw new Error('VP_REMOTE_CACHE_URL is not set');
url = `${endpoint.replace(/\/$/, '')}/${path.replace(/^\//, '')}`;
}
const response = await fetch(url, {
const response = await fetch(`${process.env['VP_REMOTE_CACHE_URL']}${path}`, {
method,
...(body === undefined ? {} : { body }),
...(contentType === undefined ? {} : { headers: { 'content-type': contentType } }),
Expand Down
44 changes: 12 additions & 32 deletions packages/tools/src/remote-cache/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,19 @@
import { spawn } from 'node:child_process';
import { once } from 'node:events';
import type { AddressInfo } from 'node:net';
import { constants } from 'node:os';
import { createCacheServer } from './server.ts';

const basePath = '/projects/test';
const directory = 'remote-cache';

async function main(): Promise<void> {
const [command, ...args] = process.argv.slice(2);
if (command === undefined) throw new Error('Usage: remote-cache-server COMMAND [ARGS...]');
const server = createCacheServer({ basePath, directory });
server.listen(0, '127.0.0.1');
await once(server, 'listening');
try {
const { port } = server.address() as AddressInfo;
const child = spawn(command, args, {
stdio: 'inherit',
env: { ...process.env, VP_REMOTE_CACHE_URL: `http://127.0.0.1:${port}${basePath}` },
});
// The terminal also delivers Ctrl-C to the command. Keep serving until it exits.
process.on('SIGINT', () => {});
process.on('SIGTERM', () => child.kill('SIGTERM'));
const [code, signal] = (await once(child, 'exit')) as [number | null, NodeJS.Signals | null];
process.exitCode = code ?? 128 + constants.signals[signal!];
} finally {
const closed = server[Symbol.asyncDispose]();
server.closeAllConnections();
await closed;
}
}

try {
await main();
} catch (error) {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
}
const [command, ...args] = process.argv.slice(2);
const server = createCacheServer({ basePath, directory: 'remote-cache' });
server.listen(0, '127.0.0.1');
await once(server, 'listening');
const { port } = server.address() as AddressInfo;
const child = spawn(command!, args, {
stdio: 'inherit',
env: { ...process.env, VP_REMOTE_CACHE_URL: `http://127.0.0.1:${port}${basePath}` },
});
const [code] = (await once(child, 'exit')) as [number | null];
server.close();
process.exitCode = code;
36 changes: 7 additions & 29 deletions packages/tools/src/remote-cache/server.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
import { Busboy } from '@fastify/busboy';
import { decode } from 'cbor2/decoder';
import { encode } from 'cbor2/encoder';
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { readFile } from 'node:fs/promises';
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { createServer, type IncomingMessage, type ServerResponse } from 'node:http';
import { join } from 'node:path';

Expand Down Expand Up @@ -95,24 +94,6 @@ function cbor(response: ServerResponse, value: unknown): void {
response.end(encode(value));
}

function loadState(file: string): State {
try {
return JSON.parse(readFileSync(file, 'utf8')) as State;
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
return { next_blob_id: 1, entries: {}, associations: {} };
}
}

async function readBlob(file: string): Promise<Buffer | undefined> {
try {
return await readFile(file);
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return undefined;
throw error;
}
}

/**
* A test backend that keeps its state in `directory`: entries and associations
* in `state.json`, and each blob in `blobs/` under its ID. Keys, values, and
Expand All @@ -127,20 +108,20 @@ export function createCacheServer({
}) {
const stateFile = join(directory, 'state.json');
const blobDirectory = join(directory, 'blobs');
const state = loadState(stateFile);
const state: State = existsSync(stateFile)
? JSON.parse(readFileSync(stateFile, 'utf8'))
: { next_blob_id: 1, entries: {}, associations: {} };
const entries = new Map(Object.entries(state.entries));
const associations = new Map(Object.entries(state.associations));
let nextBlobId = state.next_blob_id;

async function handle(request: IncomingMessage, response: ServerResponse): Promise<void> {
const path = new URL(request.url ?? '/', 'http://localhost').pathname;
if (request.method === 'GET' && path.startsWith(`${basePath}/blob/`)) {
const blobId = path.slice(`${basePath}/blob/`.length);
// Blob IDs are sequential numbers, so other IDs cannot name a blob file.
const blob = /^\d+$/.test(blobId) ? await readBlob(join(blobDirectory, blobId)) : undefined;
if (blob === undefined) throw new RequestError(404, 'Blob not found');
const file = join(blobDirectory, path.slice(`${basePath}/blob/`.length));
if (!existsSync(file)) throw new RequestError(404, 'Blob not found');
response.writeHead(200, { 'content-type': 'application/octet-stream' });
response.end(blob);
response.end(readFileSync(file));
return;
}
if (request.method !== 'POST' || ![`${basePath}/fetch`, `${basePath}/store`].includes(path)) {
Expand Down Expand Up @@ -186,9 +167,6 @@ export function createCacheServer({
const fields = byteFields(metadata, ['key', 'secondary_key', 'value']);
const key = toHex(fields.get('key')!);
const blob = parts.get('blob');

// Publish only after the complete request is validated. The writes are
// synchronous, so concurrent stores cannot interleave them.
mkdirSync(blobDirectory, { recursive: true });
const blobId = blob === undefined ? null : String(nextBlobId++);
if (blobId !== null) writeFileSync(join(blobDirectory, blobId), blob!);
Expand Down
Loading