[hasumikin]

Problem

Three things stop Prism 2.0.0 from being embedded on a microcontroller as Prism 1.9 was. mruby's compiler (mruby/mruby#7707) and PicoRuby, which runs that compiler on RP2040/RP2350 boards with a few hundred KB of RAM, need both.

1. The arena's first block is 8 KB, and the size is fixed. PM_ARENA_INITIAL_SIZE in src/arena.c is a plain #define. A parser has two arenas (the caller's node arena and its own metadata arena), so parsing one REPL line costs two 8 KB blocks, 16 KB plus headers, for about 6 KB of data (most of it the constant pool, which mruby pre-seeds with its symbol table). Measured on a 32-bit build: 16 KB held per parse, against 6 KB with Prism 1.9.

2. The 64-bit PRI* macros are undefined on a common cross toolchain. 61ebe51 ("Fix cross-compile issue with newlib on Ubuntu 24.04") made defines.h include <sys/types.h> ahead of <inttypes.h>: with an arm-none-eabi GCC whose <stdint.h> is GCC's own, newlib's <inttypes.h> never sees the marker it tests before it defines the 64-bit macros, so PRIu64 is undefined while PRIu32 is defined. 400b217 ("Remove defines") removed the header and the workaround with it. Prism 2.0.0 fails to cross-compile there:

   src/integer.c:645:44: error: expected ')' before 'PRIu64'
   

   Seen with Debian's gcc-arm-none-eabi 13.2.1 and its newlib.

3. The constant pool starts at 4 entries for a short source, and every growth step leaves its old arrays in the arena. pm_parser_init sizes the pool from the source length (size / 120, at least 4). The pool doubles when it is 3/4 full, and since the arrays live in the metadata arena, the outgrown ones are never freed until the parser is cleaned up. mruby interns its 58 pre-defined symbols into the pool before every parse, so a REPL line goes 4 -> 8 -> 16 -> 32 -> 64 -> 128 and carries all six generations of the hash and the constant array. That is most of the 10 KB a REPL line still holds with 2 KB arena blocks.

Fix

• src/arena.c: PM_ARENA_INITIAL_SIZE under #ifndef, so an embedder can build with -DPM_ARENA_INITIAL_SIZE=2048. The default stays 8 KB and nothing else changes. With 2 KB blocks the same REPL line holds 10 KB on the device (5 blocks); an 18 KB source file is unaffected, since pm_parser_init reserves its blocks from the source size.
• src/integer.c, src/errors_format.c: include <sys/types.h> before <inttypes.h> again, in the two files that include <inttypes.h> now, with the note that defines.h carried.
• src/prism.c: a PM_CONSTANT_POOL_MIN_CAPACITY macro under #ifndef, default 4, as the lower bound of the pool capacity that pm_parser_init computes. An embedder that seeds the pool builds with -DPM_CONSTANT_POOL_MIN_CAPACITY=128; the default build is unchanged. With it, the REPL line on the device holds 6 KB per parse, the same as with Prism 1.9.

Verification

• A PicoRuby firmware for the Raspberry Pi Pico 2 W (RP2350, arm-none-eabi 13.2.1, newlib) builds and runs its REPL with the three changes; without the second it does not compile, without the first it holds 16 KB per parsed line instead of 10 KB, and without the third 10 KB instead of 6 KB. An 18 KB source file goes from 158 KB (2.0.0 as released) to 128 KB.
• bundle exec rake compile test on the host passes (22098 tests, 2243812 assertions, 0 failures): the default block size and pool capacity are unchanged and the extra include is the one 1.9 had.

Request

These are the only changes mruby and PicoRuby need from Prism to move to 2.0. Once this is merged, could you cut a 2.0.1 release? mruby vendors a released Prism, so the move waits on a tag that includes these commits.

[kddnewton]

@hasumikin happy to release a new version for this. Thanks for the quick fixes and testing of 2.0.0! A quick question: I added a scratch arena to the parser for stuff that shouldn't outlive the parse. It's freed at the end of parsing. Is that going to be alright? It's initialized and freed within the bounds of pm_parser_new...pm_parser_free.

[hasumikin]

@kddnewton
Yes, that is fine for us, and I measured it. mruby's compiler reads only the tree and the constant pool after pm_parse returns; the scopes, lex modes and context nodes that moved to the scratch arena are internal to the parse.

On a 32-bit build with PM_ARENA_INITIAL_SIZE=2048 and PM_CONSTANT_POOL_MIN_CAPACITY=128 (the PicoRuby device settings), current main against the three commits alone:

• the scratch arena is one 2 KB block for every input I tried (a REPL line, an 18 KB source file), freed at the end of pm_parse, so a parse peaks 2 KB higher while it runs;
• the memory a parse leaves behind for code generation is unchanged (6 KB for a REPL line, 128 KB for the 18 KB file), and the bytes used inside those blocks go down by up to 1 KB, since the scopes and locals no longer sit in the metadata arena.

So the scratch arena is a small win for us. Thank you for your work!!!