trim_trailing/1 walked the string with binary_part/3, which allocated the
three byte lookahead plus a fresh prefix on every step even though every
prefix but the last one is discarded, and paid one allocation even when
there was nothing to trim.
Carry the position instead and slice once at the end. The six one-byte
whitespace codepoints, which are the common case, now need no lookahead at
all, so the lookahead tables only hold the multi-byte ones.
Dispatching on the byte with :binary.at/2 instead of matching the string is
what makes the common path free. Matching turns the argument into a match
context, and a clause returning the string unchanged then has to materialize
it again, which is the allocation we are trying to avoid.
Measured on OTP 29, ns/op and words allocated per call:
before after
100B, nothing to trim 47.6 / 8w 16.0 / 0w
1KB, nothing to trim 47.4 / 8w 16.2 / 0w
100B + one space 73.2 / 21w 41.4 / 5w
100B + four spaces 104.4 / 34w 83.2 / 5w
100B + NBSP 71.8 / 21w 60.9 / 13w
100B + 64 spaces 754.8 / 294w 887.8 / 5w
Trimming every line of lib/elixir/lib/kernel.ex drops from 0.442ms to
0.134ms, and 2000 lines ending in a newline from 0.282ms to 0.094ms.
The trade-off is the last row. The old code advanced up to three bytes per
iteration through the lookahead table where this one advances one byte at a
time, so runs longer than about five whitespace bytes lose time, 15% to 20%
on a 64 byte run. Allocation on that run goes from O(n) to O(1), and long
trailing runs are rare next to lines ending in a single newline or in
nothing at all.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Atoms made exclusively of ASCII letters, numbers and underscores, beginning
with a lowercase letter or underscore and optionally ending with ? or !, are
always identifiers, and only atoms prefixed by "Elixir" are candidates for
aliases. Recognizing both directly on the binary avoids building a charlist
and running the unicode aware identifier tokenizer, which dominates the cost
of classifying keyword list, map and struct keys when inspecting them.
Both encoders were built on a binary comprehension whose body calls
`percent/2` through a captured predicate:
for <<byte <- string>>, into: "" do
case percent(byte, &char_unreserved?/1) do
"%20" -> "+"
percent -> percent
end
end
`for ... into: ""` over a binary generator compiles to a native Erlang
bitstring comprehension, so the comprehension itself is fine, but in the
body it pays per byte a `make_fun3` to build the captured predicate, a
call to `percent/2` plus an indirect call through it, a freshly
allocated one- or three-byte binary that is discarded immediately, an
`is_eq_exact` against the literal "%20", and a runtime-sized
`bs_create_bin` to append the result.
Rewrite it as a tail-recursive loop that writes the escape straight into
the accumulator. The unreserved test moves into a `defguardp` so it
inlines instead of going through a closure, `+` is handled by its own
clause rather than by comparing the produced binary against "%20", and
the appended segments have compile-time-known sizes.
Barna Kovacs
José Valim
Dino Kovač
Daniel KukulaandClaude Opus 5
Thomas Cioppettini
Hasan mohamed
Eric Meadows-Jönsson
Alexander Gubarev
Andrea Leopardi
Nelson Vides
José Valim
Mikhail Nekrashevich
makoto-developer
Jonatan Männchen