* update support for uts39 from unicode 15
* follow uts39's recco that it's not necessary to require
idents to be single-script (they call out proglang idents,
reference the new uts55-5). We use a heuristic derived from
the concept of identifier chunks from uts55-5, to allow
idents like foo_bar_baz where each chunk around the _ can be
single-or-highly-restrictive
* provide directional confusability detection, by reversing
spans of direction-changed chars in idents for bidi_skeleton,
see issue #12929
When capitalize/2 was written, Erlang did not provide titlecase
functions. capitalize/2 also downcases the rest of the string
while Erlang doesn't, which is a common contention point.
Given our titlecase implementation requires 40kb of additional
code in .beam files, it makes sense to unify it with Erlang's
(which we have already done for most non-critical functionality
in unicode).
* nfc by default
* suggest nfkc on unexpected tokens when nfkc would have worked
* support additional normalizations (just micro/mu for now)
* parser, formatter tests
* tweak Macro.inner_classify(atom) not to rely on not_nfc causing error
The patch computes byte lookups based on the prefix. For example,
Á, É, etc all have the same prefix <<195>>, so they are lumped
together for lookup and then we just do a byte lookup later. We
tried doing the byte lookup on 64-element tuple (since the byte
is always within 0b10000000 and 0b10111111) but that's slower,
especially because we need to check the byte range for invalid
Unicode, so instead the last byte lookup is a case. Grouping the
top-level lookup makes the cost of a miss 3x cheaper albeit a
hit is 10% more expensive and reduces bytecode size.
Erlang ships with its own embedding of the Unicode
Codebase for a couple releases and this release changes
Elixir to depend on it in order to compute grapheme
clusters.
The Erlang implementation was up to 2x faster in low
codepoints (such as latin1) while the Elixir one could
be faster up to 3x in high codepoints (such as emoji)
so at the end the performance results are roughly the
same. As a benefit, we no longer need to ship our copy
of the grapheme cluster algorithm, which would take up
to 250kB in disk and more than 15 seconds to compile.
Note we still keep our own String downcase and upcase
algorithms, as our version is considerably more efficient
on all cases since it works exclusively with binaries
(more than 5x faster).
José Valim
Jonatan Männchen
maintenance-beam-app
Luc Fueston
EksperimentalandMaintenance App
Andrea Leopardi
sabiwara
John Bampton
Enrico Rivarola
Cẩm Huỳnh
Alt D. Soy
Eksperimental
José Valim
Eric Meadows-Jönsson