Skip to content

Commit 4bf1dd4

Browse files
authored
format: evaluate a scope's define in dependency order (#408)
* pointers: evaluate a scope's define in dependency order A member sees the other members of the same define by name, and its own name means the variable of the enclosing scope. Members are queued and retried, as region fields are for ~this; a cycle throws. * format: define a scope's variables without member order JSON objects are unordered, so the members of a define may use each other in any order. Reorder the long string example's define so the integration test resolves a member before the one it uses. * Add changelog entries for define order * Link the define order changelog entries to #408
1 parent 53bb262 commit 4bf1dd4

8 files changed

Lines changed: 197 additions & 30 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,20 @@ The sections do not signal obligations; the prefixes do.
5555

5656
### Changed
5757

58+
- The members of a `define` no longer depend on their order, since JSON
59+
objects are unordered. A member may use any other member of the same
60+
`define`. Inside its own expression, its own name means the variable of the
61+
enclosing scope. A cycle of members makes the pointer invalid ([#408]).
62+
- Schemas: **ethdebug/format/pointer/collection/scope**
63+
- Producers: required: a name of a later member of the same `define` now
64+
means that member, not the enclosing scope's variable with that name. Data
65+
that used such a name for the outer variable must put the member in a
66+
nested scope. Data that uses only earlier members, a member's own name, or
67+
other outer names resolves as before.
68+
- Consumers: required: "A consumer evaluates each member after the members
69+
it uses. If members depend on each other in a cycle, the pointer is
70+
invalid."
71+
5872
- Variables now have a stated sort. A list's `each` variable is an integer.
5973
A `define` variable has the sort of its expression, so a bytes variable
6074
keeps its width. A template's `expect`ed variable takes the sort of the
@@ -748,3 +762,4 @@ First published version of the specification.
748762
[#325]: https://github.com/ethdebug/format/pull/325
749763
[#402]: https://github.com/ethdebug/format/pull/402
750764
[#404]: https://github.com/ethdebug/format/pull/404
765+
[#408]: https://github.com/ethdebug/format/pull/408

‎packages/pointers/CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@ itself are tracked in the root [`CHANGELOG.md`](../../CHANGELOG.md).
66

77
## Unreleased
88

9+
### Changed
10+
11+
- `dereference` evaluates the members of a scope's `define` in dependency
12+
order, not member order. A member sees the other members by name, and its
13+
own name means the variable of the enclosing scope. A cycle of members throws
14+
"Circular reference detected" ([#408]).
15+
916
### Fixed
1017

1118
- The published type declarations no longer reference Node's `util` types.
@@ -147,3 +154,4 @@ First publication.
147154
[#317]: https://github.com/ethdebug/format/pull/317
148155
[#323]: https://github.com/ethdebug/format/pull/323
149156
[#325]: https://github.com/ethdebug/format/pull/325
157+
[#408]: https://github.com/ethdebug/format/pull/408

‎packages/pointers/src/dereference/index.test.ts‎

Lines changed: 78 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { vitest, expect, describe, it, beforeEach } from "vitest";
2-
import { Pointer } from "@ethdebug/format";
2+
import { Pointer, schemas } from "@ethdebug/format";
33
import { Machine } from "#machine";
44
import { Data } from "#data";
55
import { dereference } from "./index.js";
@@ -336,6 +336,83 @@ describe("dereference", () => {
336336
expect(regions[1]).toMatchObject({ slot: Data.fromNumber(2) });
337337
});
338338

339+
describe("define member order", () => {
340+
// the region of a scope whose `in` is at storage slot "result"
341+
const regionOf = async (
342+
define: Pointer.Collection.Scope["define"],
343+
outer?: Pointer.Collection.Scope["define"],
344+
) => {
345+
const scope: Pointer = {
346+
define,
347+
in: { location: "storage", slot: "result" },
348+
};
349+
const pointer: Pointer = outer ? { define: outer, in: scope } : scope;
350+
const { regions } = await (await dereference(pointer)).view(state);
351+
return regions[0];
352+
};
353+
354+
it("evaluates members after the members they use", async () => {
355+
expect(
356+
await regionOf({ result: { "~sum": ["a", 1] }, a: 5 }),
357+
).toMatchObject({ slot: Data.fromNumber(6) });
358+
});
359+
360+
it("evaluates a longer chain in any order", async () => {
361+
expect(
362+
await regionOf({
363+
result: { "~sum": ["c", 1] },
364+
c: { "~product": ["b", { "~sum": ["a", 1] }] },
365+
b: { "~sum": ["a", 2] },
366+
a: 1,
367+
}),
368+
).toMatchObject({ slot: Data.fromNumber(7) });
369+
});
370+
371+
it("binds a member's own name to the enclosing scope", async () => {
372+
expect(
373+
await regionOf({ result: { "~sum": ["result", 1] } }, { result: 4 }),
374+
).toMatchObject({ slot: Data.fromNumber(5) });
375+
});
376+
377+
it("hides an outer variable from the other members", async () => {
378+
expect(
379+
await regionOf({ result: { "~sum": ["a", 1] }, a: 5 }, { a: 100 }),
380+
).toMatchObject({ slot: Data.fromNumber(6) });
381+
});
382+
383+
it("resolves the scope schema's out-of-order example", async () => {
384+
const { examples } = schemas[
385+
"schema:ethdebug/format/pointer/collection/scope"
386+
] as { examples: Pointer[] };
387+
const example = examples.find((pointer) =>
388+
JSON.stringify(pointer).includes('"end"'),
389+
)!;
390+
const { regions } = await (await dereference(example)).view(state);
391+
expect(regions[0]).toMatchObject({
392+
offset: Data.fromNumber(96),
393+
length: Data.fromNumber(32),
394+
});
395+
});
396+
397+
it("throws on a cycle between members", async () => {
398+
await expect(regionOf({ result: "a", a: "b", b: "a" })).rejects.toThrow(
399+
"Circular reference detected",
400+
);
401+
});
402+
403+
it("throws on a cycle even when outer variables match", async () => {
404+
await expect(
405+
regionOf({ result: "a", a: "b", b: "a" }, { a: 1, b: 2 }),
406+
).rejects.toThrow("Circular reference detected");
407+
});
408+
409+
it("still throws on an unknown variable", async () => {
410+
await expect(regionOf({ result: "a", b: 1 })).rejects.toThrow(
411+
"Unknown variable with identifier a",
412+
);
413+
});
414+
});
415+
339416
it("scopes variables defined inside referenced templates", async () => {
340417
const pointer: Pointer = {
341418
define: { slot: "0x00" },

‎packages/pointers/src/dereference/process.ts‎

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -138,19 +138,51 @@ async function* processScope(
138138
): Process {
139139
const { define: variableExpressions, in: in_ } = collection;
140140

141-
const allVariables: Record<string, Value> = Object.assign(
142-
Object.create(null),
143-
options.variables,
144-
);
145-
const newVariables: { [identifier: string]: Value } = {};
146-
for (const [identifier, expression] of Object.entries(variableExpressions)) {
147-
const value = await evaluate(expression, {
148-
...options,
149-
variables: allVariables,
141+
const newVariables: { [identifier: string]: Value } = Object.create(null);
142+
const isMember = (name: PropertyKey): name is string =>
143+
Object.prototype.hasOwnProperty.call(variableExpressions, name);
144+
145+
// the variables one member sees: a name of another member means that
146+
// member, so it must be evaluated first; any other name, the member's
147+
// own included, means a variable of the enclosing scope
148+
const variablesFor = (identifier: string) =>
149+
new Proxy(options.variables, {
150+
get(target, name) {
151+
if (name === identifier || !isMember(name)) {
152+
return target[name as string];
153+
}
154+
if (name in newVariables) {
155+
return newVariables[name];
156+
}
157+
throw new Error(`Variable not evaluated yet: ${name}`);
158+
},
150159
});
151160

152-
allVariables[identifier] = value;
153-
newVariables[identifier] = value;
161+
// evaluate members in dependency order, independent of member order.
162+
// retry a member that uses one not evaluated yet; when every remaining
163+
// member has failed since the last success, they form a cycle
164+
const queue = Object.keys(variableExpressions);
165+
let failures = 0;
166+
while (queue.length > 0) {
167+
const identifier = queue.shift()!;
168+
try {
169+
newVariables[identifier] = await evaluate(
170+
variableExpressions[identifier],
171+
{ ...options, variables: variablesFor(identifier) },
172+
);
173+
failures = 0;
174+
} catch (error) {
175+
if (
176+
!(error instanceof Error) ||
177+
!error.message.startsWith("Variable not evaluated yet: ")
178+
) {
179+
throw error;
180+
}
181+
if (++failures > queue.length) {
182+
throw new Error(`Circular reference detected: ${identifier}`);
183+
}
184+
queue.push(identifier);
185+
}
154186
}
155187

156188
// variables defined here are visible only within `in`

‎packages/web/docs/core-schemas/pointers/collections.mdx‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -84,10 +84,13 @@ evaluates to a non-zero value.
8484

8585
## Scope
8686

87-
A **scope** defines variables that can be used in a nested pointer. Variables
88-
are evaluated in order, so later variables can reference earlier ones. They are
89-
visible only within `in`: pointers outside the scope, such as later members of
90-
an enclosing group, do not see them.
87+
A **scope** defines variables that can be used in a nested pointer. A variable
88+
can use the other variables of the same `define`, in any order; member order
89+
has no effect. Inside its own expression, a variable's own name means the outer
90+
variable with that name, so a variable never depends on itself. A cycle of
91+
variables makes the pointer invalid. The variables are visible only within
92+
`in`: pointers outside the scope, such as later members of an enclosing group,
93+
do not see them.
9194

9295
Scopes help break complex pointer definitions into readable steps. For examples
9396
combining scopes with keccak256 for storage slot computation, see the

‎packages/web/docs/implementation-guides/pointers/dereference-logic/generating-regions.mdx‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -227,10 +227,12 @@ Finally, the last kind of collection defined by this schema is for defining
227227
a scope of variables by identifier by specifying the expression values for
228228
each of those variables.
229229

230-
Since this schema takes advantage of how JSON objects are ordered lists of
231-
key/value pairs, variables specified later may reference variables specified
232-
earlier. The only trickiness in implementing `processScope` is ensuring that
233-
variable values are available immediately.
230+
JSON objects are unordered, so the members of a `define` may use each other
231+
in any order. `processScope` evaluates each member after the members it uses.
232+
Like the fields of a region that refer to each other with `~this`, it queues
233+
the members and retries one that uses a member not evaluated yet. Each member
234+
sees the other members by name, but its own name means the variable of the
235+
enclosing scope. If every remaining member fails in turn, they form a cycle.
234236

235237
<CodeListing
236238
packageName="@ethdebug/pointers"

‎schemas/pointer.schema.yaml‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,15 @@ examples:
211211
length: ~wordsize
212212

213213
- define:
214+
"total-slots":
215+
# account for both zero and nonzero slot remainders by
216+
# adding ~wordsize-1 to the length before dividing. this
217+
# uses "string-length", defined below: the members of a
218+
# define may appear in any order
219+
~quotient:
220+
- ~sum: ["string-length", { ~difference: [~wordsize, 1] }]
221+
- ~wordsize
222+
214223
"string-length":
215224
~quotient:
216225
- ~difference:
@@ -221,13 +230,6 @@ examples:
221230
"start-slot":
222231
~keccak256:
223232
- ~wordsized: "string-storage-contract-variable-slot"
224-
225-
"total-slots":
226-
# account for both zero and nonzero slot remainders by adding
227-
# ~wordsize-1 to the length before dividing
228-
~quotient:
229-
- ~sum: ["string-length", { ~difference: [~wordsize, 1] }]
230-
- ~wordsize
231233
in:
232234
list:
233235
count: "total-slots"

‎schemas/pointer/collection/scope.schema.yaml‎

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,18 @@ description: |
77
as expressions.
88
99
Variables are specified by the `define` field as an object mapping of
10-
expression by identifier. Variables are specified **in order**, so that
11-
later appearing variables may reference earlier ones in the same object.
10+
expression by identifier. The members of one `define` can use each other
11+
in any order. Member order has no effect.
12+
13+
- A name in a variable's expression that matches **another** member of
14+
the same `define` means that member.
15+
- A name that matches the variable **itself** means the variable with
16+
that name in the enclosing scope. So a variable can never depend on
17+
itself.
18+
- Any other name means a variable of the enclosing scope.
19+
20+
A consumer evaluates each member after the members it uses. If members
21+
depend on each other in a cycle, the pointer is invalid.
1222
1323
Each variable has the sort of the expression that defines it (see
1424
**ethdebug/format/pointer/expression**). A variable defined by an integer
@@ -19,7 +29,8 @@ description: |
1929
The variables are visible only within `in`. Pointers outside this scope
2030
do not see them: later members of an enclosing group, for example, see
2131
only the variables of their own enclosing scopes. A variable defined here
22-
with the same identifier as an outer variable shadows it within `in`.
32+
with the same identifier as an outer variable shadows it within `in` and
33+
within the expressions of the other members.
2334
2435
type: object
2536

@@ -52,6 +63,23 @@ examples:
5263
offset: example-offset
5364
length: example-length
5465

66+
- # example: `end` uses `offset`, a member that appears after it. in its
67+
# own expression, `offset` means the outer `offset` (64), so the inner
68+
# `offset` is 96 and `end` is 128: the region is 32 bytes at 96.
69+
define:
70+
offset: 64
71+
in:
72+
define:
73+
end:
74+
~sum: [offset, ~wordsize]
75+
offset:
76+
~sum: [offset, 32]
77+
in:
78+
location: memory
79+
offset: offset
80+
length:
81+
~difference: [end, offset]
82+
5583
- # example: a variable with a bytes value keeps its width, so it can be
5684
# hashed directly. this is the storage slot of `m["abc"]` for a Solidity
5785
# `mapping(string => uint256) m` at slot 0: the key's raw bytes followed

0 commit comments

Comments
 (0)