Commit Graph
14 Commits
Author SHA1 Message Date
José Valim 51884ca511 Fix regression on identifier\\ being interpreted as a single identifier 2022-06-30 15:52:18 +02:00
José Valim 5deafbdc89 Normalize unicode characters once in tokenizer 2022-05-23 12:14:39 +03:00
José Valim f431aefcac Do not attempt to suggest starting tokens
It is hard to suggest because it is not possible to
know if the user intended an operator, alias, or
identifier, so we can give incorrect feedback.
2022-05-23 12:14:39 +03:00
Luc Fueston e7001455d7 nfc and additional normalizations for identifiers (#11859)
* nfc by default
* suggest nfkc on unexpected tokens when nfkc would have worked
* support additional normalizations (just micro/mu for now)
* parser, formatter tests
* tweak Macro.inner_classify(atom) not to rely on not_nfc causing error
2022-05-23 09:44:44 +02:00
José Valim e8b24cd3f0 Remove duplication in unicode tokenizer 2022-04-09 09:46:56 +02:00
José Valim 9e0ae6140a Allow only single- and highly restricted mixed-scripts in identifiers (UTS39 C3) (#11621) 2022-02-11 20:44:06 +01:00
Luc Fueston 4a5fcbe04a Don't allow restricted characters in identifiers (UTS 39, C1) (#11580) 2022-01-17 19:42:36 +01:00
José Valim 151f11e0b8 Split on \r\n and \n accordingly during unicode compilation, closes #10991 2021-05-20 12:12:42 +02:00
Enrico Rivarola 218c35a041 Sync unicode data to Unicode 13.0.0 (#10866) 2021-04-03 15:20:05 +02:00
José Valim c36791091f Consider CRLF on Windows, closes #10470 2020-11-01 17:25:06 +01:00
Eric Meadows-Jönsson 9b6ec7bae7 Move unreachable function check from xref to group pass (#9168) 2019-06-28 10:25:49 +02:00
Frank McGeough ba9d08a54b Run formatter on lib/elixir/unicode/tokenizer.ex (#6813) 2017-10-10 18:10:35 +02:00
José Valim 3c9b191ec1 Do not rely on on_load because it breaks releases 2017-05-29 15:50:06 +02:00
José Valim 9873e4239f Add non-quoted Unicode atoms and variables (#6158)
It follows Unicode Annex #31.

See the Unicode Syntax document for a more in-depth reference.
2017-05-27 22:43:05 +02:00