José Valim
51884ca511
Fix regression on identifier\\ being interpreted as a single identifier
2022-06-30 15:52:18 +02:00
José Valim
5deafbdc89
Normalize unicode characters once in tokenizer
2022-05-23 12:14:39 +03:00
José Valim
f431aefcac
Do not attempt to suggest starting tokens
...
It is hard to suggest because it is not possible to
know if the user intended an operator, alias, or
identifier, so we can give incorrect feedback.
2022-05-23 12:14:39 +03:00
Luc Fueston
e7001455d7
nfc and additional normalizations for identifiers ( #11859 )
...
* nfc by default
* suggest nfkc on unexpected tokens when nfkc would have worked
* support additional normalizations (just micro/mu for now)
* parser, formatter tests
* tweak Macro.inner_classify(atom) not to rely on not_nfc causing error
2022-05-23 09:44:44 +02:00
José Valim
e8b24cd3f0
Remove duplication in unicode tokenizer
2022-04-09 09:46:56 +02:00
José Valim
9e0ae6140a
Allow only single- and highly restricted mixed-scripts in identifiers (UTS39 C3) ( #11621 )
2022-02-11 20:44:06 +01:00
Luc Fueston
4a5fcbe04a
Don't allow restricted characters in identifiers (UTS 39, C1) ( #11580 )
2022-01-17 19:42:36 +01:00
José Valim
151f11e0b8
Split on \r\n and \n accordingly during unicode compilation, closes #10991
2021-05-20 12:12:42 +02:00
Enrico Rivarola
218c35a041
Sync unicode data to Unicode 13.0.0 ( #10866 )
2021-04-03 15:20:05 +02:00
José Valim
c36791091f
Consider CRLF on Windows, closes #10470
2020-11-01 17:25:06 +01:00
Eric Meadows-Jönsson
9b6ec7bae7
Move unreachable function check from xref to group pass ( #9168 )
2019-06-28 10:25:49 +02:00
Frank McGeough
ba9d08a54b
Run formatter on lib/elixir/unicode/tokenizer.ex ( #6813 )
2017-10-10 18:10:35 +02:00
José Valim
3c9b191ec1
Do not rely on on_load because it breaks releases
2017-05-29 15:50:06 +02:00
José Valim
9873e4239f
Add non-quoted Unicode atoms and variables ( #6158 )
...
It follows Unicode Annex #31 .
See the Unicode Syntax document for a more in-depth reference.
2017-05-27 22:43:05 +02:00