For providing better feedback in credo and tools like it we should be
able to map a token back to its original source. This makes sure that
`_` characters in numbers are properly accounted for so that they stay
in sync after we've encountered something like 123_456_789.
Signed-off-by: José Valim <jose.valim@plataformatec.com.br>
Standardizes the use of comma leaving a white space after it whenever applicable.
Note: It does not enforce this in quantifiers in regular expressions such as in: `x{1,3}`
Before this commit, we handled char literals (like `?a`) in the
tokenizer, turning a literal like `?a` into the token `{:number, _,
97}` (thus indistinguishable from the literal `97` at the parsing
stage). This led to error messages with the integer for the character
instead of the character literal, e.g.:
iex> :ok ?a
** (SyntaxError) iex:11: syntax error before: 97
With this commit, we now turn `?a` into the token `{:char, _,
97}` (which is the same token used by Erlang for Erlang char literals
like `$a`); since it's the same token as in Erlang, the parser will now
output the char literal as an Erlang char (`?a` would be printed as
`$a`). We hijack the error message in elixir_errors.erl to end up with
the correct message:
iex> :ok ?a
** (SyntaxError) iex:11: syntax error before: ?a
This is a large commit that does the following:
* Removes backticks from messages and comments.
All references to backticks have been replaced with double quotes everywhere the
code is not interpreted as Markdown, (ie. anywhere outside documentation and
markdown files, such as in code comments, or error messages).
* Variables in Exception messages are printed using `inspect`
The way no-matching error message are printed, have changed because now we use inspect for printing
variables. The following file and their respective test have been changed:
- lib/elixir/lib/exception.ex
- lib/elixir/lib/inspect/algebra.ex
- lib/elixir/test/elixir/inspect_test.exs
- lib/ex_unit/test/ex_unit/formatter_test.exs
- lib/elixir/test/elixir/exception_test.exs
* Properly use Title Case for Mix, Git, Dializer
* Use backticks when citing a command
****************************************************
CONVENTION FOR RENAMING USING BACKTICKS AND QUOTES
https://github.com/elixir-lang/elixir/pull/3697#issuecomment-138811747
1. Backticks should never be printed in error messages, neither be included anywhere where Markdown code is not interpreted as such.
2. Do not use single quotes anywhere. We should favor double quotes everywhere, to avoid confusion
3. If you want to format something in error messages, use inspect. For example, if you want to show the dependency name and that is an atom, instead of the dependency "foo", let's show the dependency :foo. Less noise and may click better
4. Similarly, if you want to show something with double quotes, call inspect, as it handles escaping as well as the quotes
5. Things like "--all" just add verbosity, we can definitely read --all without ambiguity.
6. When using switches with a single hyphen (such as "-o"), we can make it explicit in the text: e.g. "... give the switch -o when choosing ..."
****************************************************
SHELL COMMANDS TO DETECT CODE BREAKING THE RULES
* Detect values surrounded by single-quotes where we want double-quotes.
# regular variables
ag "'#{"
# constants or ENV variables
ag "'[A-Z][A-Z0-9_-]+"
# switches
ag -i "(?<!(\[))'--?[a-z0-9][a-z0-9_-]+'"
# atoms
ag -i "(?<!(\[))':[a-z0-9][a-z0-9_-]+'"
# DETECT BACKTICKS OUTSIDE DOCS
ag "^\s+(?<!(#))#[^#\r\n]*\`" --ignore "*.md"
ag '^\s+["%].*`' --ignore "*.md"
ag -s '(raise|Error)\b(?!(`|/)).*\`'
* Spot mentions to running a command that is not using backticks:
commands="elixir|elixirc|mix|iex|ex_doc|git|make|rebar|dialyzer|erl|rm|cd|mkdir|rmdir|ln|ls|pwd"
actions="runs?|running|executes?|executing|types?|typing|enters?|entering|calls?|calling"
ag '(?i)('${actions}')(?-i)\b[^`\r\n/]+[\ \t]+(?!(`))('${commands}')'
ag '(?-i)\b(?!(`))('${commands}')[\ \t]+[^`\r\n/]+(?i)(commands?)'
* Spot where a command is mentioned:
#commands="erlang|elixir|eex|iex|ex_unit|ex_doc|logger|elixirc|git|make|rebar|dialyzer|erl|rm|cd|mkdir|rmdir|ln|ls|pwd"
commands="mix|elixirc|git|make|rebar|dialyzer|erl|rm|cd|mkdir|rmdir|ln|ls|pwd"
ag -s '(?<!(\`))(?<!(\.))(?<!(/))(?<!(:))(?<!(_))\b('${commands}')\b(?!(:))(?!(\?))(?!(_))(?!(-))(?!(\.))(?!(\`))(?!(/))(?!(>))' \
--ignore "*.erl" --ignore "*.yrl" --ignore "*.src"
Closes#3113:
- Fix tokenization of `a@b: ` and `A!: `
- Add error for `a@b` (previously tokenized like `a @b`)
- Extend error for `a:b` to cover `a:+`, etc.
- Unify tokenization of upper- and lower-case `k: v` keys
- Fix end-column value for `a: `
- Add tests as appropriate
Add column info for each token in elixir_tokenizer.
Change the format of location info from `Line` to `[Line, BeginColumn,
EndColumn]`. Pass the current column after the current line in
`elixir_tokenizer:tokenize`. Reflect the change in related modules.
Andrea Leopardi
Steven Blowers
Aleksei Magusev
ottolin
Mike Stok
eksperimental
Martin Frost
José Valim
ꗈ ꘐ ꗈ
Roger Braun
Or Neeman
Xavier Defrang
Alexander Ivanov
James Brennan
Alex Rønne Petersen
Eric Meadows-Jönsson
Yuki Ito