For providing better feedback in credo and tools like it we should be
able to map a token back to its original source. This makes sure that
`_` characters in numbers are properly accounted for so that they stay
in sync after we've encountered something like 123_456_789.
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
Standardizes the use of comma leaving a white space after it whenever applicable.
Note: It does not enforce this in quantifiers in regular expressions such as in: `x{1,3}`
Before this commit, we handled char literals (like `?a`) in the
tokenizer, turning a literal like `?a` into the token `{:number, _,
97}` (thus indistinguishable from the literal `97` at the parsing
stage). This led to error messages with the integer for the character
instead of the character literal, e.g.:
iex> :ok ?a
** (SyntaxError) iex:11: syntax error before: 97
With this commit, we now turn `?a` into the token `{:char, _,
97}` (which is the same token used by Erlang for Erlang char literals
like `$a`); since it's the same token as in Erlang, the parser will now
output the char literal as an Erlang char (`?a` would be printed as
`$a`). We hijack the error message in elixir_errors.erl to end up with
the correct message:
iex> :ok ?a
** (SyntaxError) iex:11: syntax error before: ?a
Note a field named safe_by_default has been added so we don't
warn on variables defined inside EXPR in "case EXPR, do: ...".
The proper solution would be to convert && and || into special
forms and they would be free of the warning. However,
safe_by_default is a welcome temporary solution until we see
how the warning will fare on actual applications.
Closes#3113:
- Fix tokenization of `a@b: ` and `A!: `
- Add error for `a@b` (previously tokenized like `a @b`)
- Extend error for `a:b` to cover `a:+`, etc.
- Unify tokenization of upper- and lower-case `k: v` keys
- Fix end-column value for `a: `
- Add tests as appropriate
Add column info for each token in elixir_tokenizer.
Change the format of location info from `Line` to `[Line, BeginColumn,
EndColumn]`. Pass the current column after the current line in
`elixir_tokenizer:tokenize`. Reflect the change in related modules.
Mike Stok
José Valim
eksperimental
Andrea Leopardi
ꗈ ꘐ ꗈ
Roger Braun
Or Neeman
José Valim
Alexander Ivanov
Eric Meadows-Jönsson
niku
Yuki Ito
Matheus Lima