Release v1.16.0

Follow up to/extension of #13173

.github/workflows/ci.yml

@@ -3,10 +3,10 @@ name: CI
 on:
   push:
     paths-ignore:
-      - 'lib/**/*.md'
+      - "lib/**/*.md"
   pull_request:
     paths-ignore:
-      - 'lib/**/*.md'
+      - "lib/**/*.md"
 
 env:
   ELIXIR_ASSERT_TIMEOUT: 2000
@@ -24,12 +24,12 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - otp_version: '26.0'
+          - otp_version: "26.0"
             otp_latest: true
-          - otp_version: '25.3'
-          - otp_version: '25.0'
-          - otp_version: '24.3'
-          - otp_version: '24.0'
+          - otp_version: "25.3"
+          - otp_version: "25.0"
+          - otp_version: "24.3"
+          - otp_version: "24.0"
           - otp_version: master
             development: true
           - otp_version: maint
@@ -77,7 +77,7 @@ jobs:
     name: Windows Server 2019, Erlang/OTP ${{ matrix.otp_version }}
     strategy:
       matrix:
-        otp_version: ['24', '25', '26']
+        otp_version: ["24", "25", "26.0"]
     runs-on: windows-2019
     steps:
       - name: Configure Git

CHANGELOG.md

@@ -29,6 +29,21 @@ For mismatched delimiters, it now shows both delimiters:
     └─ lib/my_app.ex:1:18
 ```
 
+For unclosed delimiters, it now shows where the unclosed delimiter starts:
+
+```
+** (TokenMissingError) token missing on lib/my_app:8:23:
+    error: missing terminator: )
+    │
+  1 │ my_numbers = (1, 2, 3, 4, 5, 6
+    │              └ unclosed delimiter
+ ...
+  8 │ IO.inspect(my_numbers)
+    │                       └ missing closing delimiter (expected ")")
+    │
+    └─ lib/my_app:8:23
+```
+
 Errors and warnings diagnostics also include code snippets. When possible, we will show precise spans, such as on undefined variables:
 
 ```
@@ -51,19 +66,19 @@ error: function names should start with lowercase characters or underscore, inva
   └─ lib/sample.ex:3
 ```
 
-A huge thank you to Vinícius Muller for working on the new diagnostics.
+A huge thank you to Vinícius Müller for working on the new diagnostics.
 
 ## Revamped documentation
 
-Elixir's Getting Started guided has been made part of the Elixir repository and incorporated into ExDoc. This was an opportunity to revisit and unify all official guides and references.
+Elixir's Getting Started guides have been made part of the Elixir repository and incorporated into ExDoc. This was an opportunity to revisit and unify all official guides and references.
 
-We have also incorporated and extended the work on [Understanding Code Smells in Elixir Functional Language](https://github.com/lucasvegi/Elixir-Code-Smells/blob/main/etc/2023-emse-code-smells-elixir.pdf), by Lucas Vegi and Marco Tulio Valente, from [ASERG/DCC/UFMG](http://aserg.labsoft.dcc.ufmg.br/), into the official document in the form of anti-patterns. The anti-patterns are divided into four categories: code-related, design-related, process-related, and meta-programming. Our goal is to give all developers with both positive and negative examples of Elixir code, with context and examples on how to improve their codebases.
+We have also incorporated and extended the work on [Understanding Code Smells in Elixir Functional Language](https://github.com/lucasvegi/Elixir-Code-Smells/blob/main/etc/2023-emse-code-smells-elixir.pdf), by Lucas Vegi and Marco Tulio Valente, from [ASERG/DCC/UFMG](http://aserg.labsoft.dcc.ufmg.br/), into the official document in the form of anti-patterns. The anti-patterns are divided into four categories: code-related, design-related, process-related, and meta-programming. Our goal is to give all developers examples of potential anti-patterns, with context and examples on how to improve their codebases.
 
 Another [ExDoc](https://github.com/elixir-lang/ex_doc) feature we have incorporated in this release is the addition of cheatsheets, starting with [a cheatsheet for the Enum module](https://hexdocs.pm/elixir/main/enum-cheat.html). If you would like to contribute future cheatsheets to Elixir itself, feel free to start a discussion with an issue.
 
 Finally, we have started enriching our documentation with [Mermaid.js](https://mermaid.js.org/) diagrams. You can find examples in the [GenServer](https://hexdocs.pm/elixir/main/GenServer.html) and [Supervisor](https://hexdocs.pm/elixir/main/Supervisor.html) docs.
 
-## v1.16.0-dev
+## v1.16.0 (2023-12-22)
 
 ### 1. Enhancements
 
@@ -73,9 +88,14 @@ Finally, we have started enriching our documentation with [Mermaid.js](https://m
 
 #### Elixir
 
+  * [Code] Add `:emit_warnings` for `Code.string_to_quoted/2`
   * [Code] Automatically include columns in parsing options
   * [Code] Introduce `MismatchedDelimiterError` for handling mismatched delimiter exceptions
   * [Code.Fragment] Handle anonymous calls in fragments
+  * [Code.Formatter] Trim trailing whitespace on heredocs with `\r\n`
+  * [File] Add `:offset` option to `File.stream!/2`
+  * [Kernel] Auto infer size of matched variable in bitstrings
+  * [Kernel] Preserve column information when translating typespecs
   * [Kernel] Suggest module names based on suffix and casing errors when the module does not exist in `UndefinedFunctionError`
   * [Kernel.ParallelCompiler] Introduce `Kernel.ParallelCompiler.pmap/2` to compile multiple additional entries in parallel
   * [Kernel.SpecialForms] Warn if `True`/`False`/`Nil` are used as aliases and there is no such alias
@@ -83,34 +103,56 @@ Finally, we have started enriching our documentation with [Mermaid.js](https://m
   * [Module] Add support for `@nifs` annotation from Erlang/OTP 25
   * [Module] Add support for missing `@dialyzer` configuration
   * [String] Update to Unicode 15.1.0
+  * [String] Add `String.replace_invalid/2`
   * [Task] Add `:limit` option to `Task.yield_many/2`
 
+#### Logger
+
+  * [Logger] Add `Logger.levels/0`
+
 #### Mix
 
   * [mix] Add `MIX_PROFILE` to profile a list of comma separated tasks
+  * [mix archive.install] Support `--sparse` option
+  * [mix compile.app] Warn if both `:applications` and `:extra_applications` are used
+  * [mix compile.elixir] Pass original exception down to diagnostic `:details` when possible
   * [mix compile.elixir] Optimize scenario where there are thousands of files in `lib/` and one of them is changed
+  * [mix deps.clean] Emit a warning instead of crashing when a dependency cannot be removed
+  * [mix escript.install] Support `--sparse` option
+  * [mix release] Include `include/` directory in releases
   * [mix test] Allow testing multiple file:line at once, such as `mix test test/foo_test.exs:13 test/bar_test.exs:27`
 
 ### 2. Bug fixes
 
 #### Elixir
 
+  * [Code] Keep quotes for atom keys in formatter
   * [Code.Fragment] Fix crash in `Code.Fragment.surround_context/2` when matching on `->`
   * [IO] Raise when using `IO.binwrite/2` on terminated device (mirroring `IO.write/2`)
   * [Kernel] Do not expand aliases recursively (the alias stored in Macro.Env is already expanded)
   * [Kernel] Ensure `dbg` module is a compile-time dependency
   * [Kernel] Warn when a private function or macro uses `unquote/1` and the function/macro itself is unused
+  * [Kernel] Re-enabled compiler optimizations for top level functions in scripts (disabled in v1.14.0 but shouldn't impact most programs)
   * [Kernel] Do not define an alias for nested modules starting with `Elixir.` in their definition
   * [Kernel.ParallelCompiler] Consider a module has been defined in `@after_compile` callbacks to avoid deadlocks
+  * [Macro] Address exception on `Macro.to_string/1` for certain ASTs
+  * [Path] Lazily evaluate `File.cwd!/0` in `Path.expand/1` and `Path.absname/1`
   * [Path] Ensure `Path.relative_to/2` returns a relative path when the given argument does not share a common prefix with `cwd`
 
 #### ExUnit
 
   * [ExUnit] Raise on incorrectly dedented doctests
 
+#### IEx
+
+  * [IEx.Pry] Fix prying functions with only literals in their body
+
 #### Mix
 
-  * [Mix] Ensure files with duplicate modules are recompiled whenever any of the files change
+  * [mix archive.install] Restore code paths after `mix archive.install`
+  * [mix compile] Ensure files with duplicate modules are recompiled whenever any of the files change
+  * [mix compile] Update Mix compiler diagnostics documentation and typespecs to match the Elixir compiler behaviour where both lines and columns start from one (before it inaccurately said that columns started from zero)
+  * [mix escript.install] Restore code paths after `mix escript.install`
 
 ### 3. Soft deprecations (no warnings emitted)
 
@@ -120,6 +162,10 @@ Finally, we have started enriching our documentation with [Mermaid.js](https://m
   * [Kernel.ParallelCompiler] Deprecate `Kernel.ParallelCompiler.async/1` in favor of `Kernel.ParallelCompiler.pmap/2`
   * [Path] Deprecate `Path.safe_relative_to/2` in favor of `Path.safe_relative/2`
 
+#### Mix
+
+  * [mix compile] Returning a four-element tuple as a position in `Mix.Task.Compiler.Diagnostic`
+
 ### 4. Hard deprecations
 
 #### Elixir

Makefile

@@ -2,7 +2,7 @@ PREFIX ?= /usr/local
 TEST_FILES ?= "*_test.exs"
 SHARE_PREFIX ?= $(PREFIX)/share
 MAN_PREFIX ?= $(SHARE_PREFIX)/man
-CANONICAL := main/
+# CANONICAL := main/
 ELIXIRC := bin/elixirc --ignore-module-conflict $(ELIXIRC_OPTS)
 ERLC := erlc -I lib/elixir/include
 ERL_MAKE := if [ -n "$(ERLC_OPTS)" ]; then ERL_COMPILER_OPTIONS=$(ERLC_OPTS) erl -make; else erl -make; fi

SECURITY.md

@@ -6,12 +6,11 @@ Elixir applies bug fixes only to the latest minor branch. Security patches are a
 
 Elixir version | Support
 :------------- | :-----------------------------
-1.16           | Development
-1.15           | Bug fixes and security patches
+1.16           | Bug fixes and security patches
+1.15           | Security patches only
 1.14           | Security patches only
 1.13           | Security patches only
 1.12           | Security patches only
-1.11           | Security patches only
 
 ## Announcements
 

VERSION

@@ -1 +1 @@
-1.16.0-dev
+1.16.0

bin/elixir

@@ -1,7 +1,7 @@
 #!/bin/sh
 set -e
 
-ELIXIR_VERSION=1.16.0-dev
+ELIXIR_VERSION=1.16.0
 
 if [ $# -eq 0 ] || { [ $# -eq 1 ] && { [ "$1" = "--help" ] || [ "$1" = "-h" ]; }; }; then
   cat <<USAGE >&2

bin/elixir.bat

@@ -1,6 +1,6 @@
 @if defined ELIXIR_CLI_ECHO (@echo on) else (@echo off)
 
-set ELIXIR_VERSION=1.16.0-dev
+set ELIXIR_VERSION=1.16.0
 
 setlocal enabledelayedexpansion
 if    ""%1""==""""                if ""%2""=="""" goto documentation
@@ -140,16 +140,16 @@ if ""==!par:--remsh=!     (set "parsElixir=!parsElixir! --remsh %~1" && shift &&
 if ""==!par:--dot-iex=!   (set "parsElixir=!parsElixir! --dot-iex %~1" && shift && goto startloop)
 if ""==!par:--dbg=!       (set "parsElixir=!parsElixir! --dbg %~1" && shift && goto startloop)
 rem ******* ERLANG PARAMETERS **********************
-if ""==!par:--boot=!                (set "parsErlang=!parsErlang! -boot %~1" && shift && goto startloop)
-if ""==!par:--boot-var=!            (set "parsErlang=!parsErlang! -boot_var %~1 %~2" && shift && shift && goto startloop)
-if ""==!par:--cookie=!              (set "parsErlang=!parsErlang! -setcookie %~1" && shift && goto startloop)
+if ""==!par:--boot=!                (set "parsErlang=!parsErlang! -boot "%~1"" && shift && goto startloop)
+if ""==!par:--boot-var=!            (set "parsErlang=!parsErlang! -boot_var "%~1" "%~2"" && shift && shift && goto startloop)
+if ""==!par:--cookie=!              (set "parsErlang=!parsErlang! -setcookie "%~1"" && shift && goto startloop)
 if ""==!par:--hidden=!              (set "parsErlang=!parsErlang! -hidden" && goto startloop)
-if ""==!par:--erl-config=!          (set "parsErlang=!parsErlang! -config %~1" && shift && goto startloop)
+if ""==!par:--erl-config=!          (set "parsErlang=!parsErlang! -config "%~1"" && shift && goto startloop)
 if ""==!par:--logger-otp-reports=!  (set "parsErlang=!parsErlang! -logger handle_otp_reports %1" && shift && goto startloop)
 if ""==!par:--logger-sasl-reports=! (set "parsErlang=!parsErlang! -logger handle_sasl_reports %1" && shift && goto startloop)
-if ""==!par:--name=!                (set "parsErlang=!parsErlang! -name %~1" && shift && goto startloop)
-if ""==!par:--sname=!               (set "parsErlang=!parsErlang! -sname %~1" && shift && goto startloop)
-if ""==!par:--vm-args=!             (set "parsErlang=!parsErlang! -args_file %~1" && shift && goto startloop)
+if ""==!par:--name=!                (set "parsErlang=!parsErlang! -name "%~1"" && shift && goto startloop)
+if ""==!par:--sname=!               (set "parsErlang=!parsErlang! -sname "%~1"" && shift && goto startloop)
+if ""==!par:--vm-args=!             (set "parsErlang=!parsErlang! -args_file "%~1"" && shift && goto startloop)
 if ""==!par:--erl=!                 (set "beforeExtra=!beforeExtra! %~1" && shift && goto startloop)
 if ""==!par:--pipe-to=!             (echo --pipe-to : Option is not supported on Windows && goto end)
 set endLoop=1

lib/eex/test/eex_test.exs

@@ -761,38 +761,31 @@ defmodule EExTest do
     end
 
     test "line and column meta" do
-      parser_options = Code.get_compiler_option(:parser_options)
-      Code.put_compiler_option(:parser_options, columns: true)
+      indentation = 12
 
-      try do
-        indentation = 12
+      ast =
+        EEx.compile_string(
+          """
+          <%= f() %> <% f() %>
+            <%= f fn -> %>
+              <%= f() %>
+            <% end %>
+          """,
+          indentation: indentation
+        )
 
-        ast =
-          EEx.compile_string(
-            """
-            <%= f() %> <% f() %>
-              <%= f fn -> %>
-                <%= f() %>
-              <% end %>
-            """,
-            indentation: indentation
-          )
+      {_, calls} =
+        Macro.prewalk(ast, [], fn
+          {:f, meta, _args} = expr, acc -> {expr, [meta | acc]}
+          other, acc -> {other, acc}
+        end)
 
-        {_, calls} =
-          Macro.prewalk(ast, [], fn
-            {:f, meta, _args} = expr, acc -> {expr, [meta | acc]}
-            other, acc -> {other, acc}
-          end)
-
-        assert Enum.reverse(calls) == [
-                 [line: 1, column: indentation + 5],
-                 [line: 1, column: indentation + 15],
-                 [line: 2, column: indentation + 7],
-                 [line: 3, column: indentation + 9]
-               ]
-      after
-        Code.put_compiler_option(:parser_options, parser_options)
-      end
+      assert Enum.reverse(calls) == [
+               [line: 1, column: indentation + 5],
+               [line: 1, column: indentation + 15],
+               [line: 2, column: indentation + 7],
+               [line: 3, column: indentation + 9]
+             ]
     end
   end
 

lib/elixir/lib/code.ex

@@ -196,20 +196,44 @@ defmodule Code do
 
   @typedoc """
   Diagnostics returned by the compiler and code evaluation.
+
+  The file and position relate to where the diagnostic should be shown.
+  If there is a file and position, then the diagnostic is precise
+  and you can use the given file and position for generating snippets,
+  IDEs annotations, and so on. An optional span is available with
+  the line and column the diagnostic ends.
+
+  Otherwise, a stacktrace may be given, which you can place your own
+  heuristics to provide better reporting.
+
+  The source field points to the source file the compiler tracked
+  the error to. For example, a file `lib/foo.ex` may embed `.eex`
+  templates from `lib/foo/bar.eex`. A syntax error on the EEx template
+  will point to file `lib/foo/bar.eex` but the source is `lib/foo.ex`.
   """
   @type diagnostic(severity) :: %{
-          required(:file) => Path.t(),
+          required(:source) => Path.t() | nil,
+          required(:file) => Path.t() | nil,
           required(:severity) => severity,
           required(:message) => String.t(),
-          required(:position) => position,
+          required(:position) => position(),
           required(:stacktrace) => Exception.stacktrace(),
-          required(:span) => {non_neg_integer, non_neg_integer} | nil,
+          required(:span) => {line :: pos_integer(), column :: pos_integer()} | nil,
+          optional(:details) => term(),
           optional(any()) => any()
         }
 
   @typedoc "The line. 0 indicates no line."
   @type line() :: non_neg_integer()
-  @type position() :: line() | {pos_integer(), column :: non_neg_integer}
+
+  @typedoc """
+  The position of the diagnostic.
+
+  Can be either a line number or a `{line, column}`.
+  Line and columns numbers are one-based.
+  A position of `0` represents unknown.
+  """
+  @type position() :: line() | {line :: pos_integer(), column :: pos_integer()}
 
   @boolean_compiler_options [
     :docs,
@@ -962,10 +986,9 @@ defmodule Code do
     to_quoted_opts =
       [
         unescape: false,
-        warn_on_unnecessary_quotes: false,
         literal_encoder: &{:ok, {:__block__, &2, [&1]}},
         token_metadata: true,
-        warnings: false
+        emit_warnings: false
       ] ++ opts
 
     {forms, comments} = string_to_quoted_with_comments!(string, to_quoted_opts)
@@ -1126,9 +1149,8 @@ defmodule Code do
       atoms but `:existing_atoms_only` is still used for dynamic atoms,
       such as atoms with interpolations.
 
-    * `:warn_on_unnecessary_quotes` - when `false`, does not warn
-      when atoms, keywords or calls have unnecessary quotes on
-      them. Defaults to `true`.
+    * `:emit_warnings` (since v1.16.0) - when `false`, does not emit
+      tokenizing/parsing related warnings. Defaults to `true`.
 
   ## `Macro.to_string/2`
 
@@ -2057,7 +2079,8 @@ defmodule Code do
 
   defp get_beam_and_path(module) do
     with {^module, beam, filename} <- :code.get_object_code(module),
-         {:ok, ^module} <- beam |> :beam_lib.info() |> Keyword.fetch(:module) do
+         info_pairs when is_list(info_pairs) <- :beam_lib.info(beam),
+         {:ok, ^module} <- Keyword.fetch(info_pairs, :module) do
       {beam, filename}
     else
       _ -> :error

lib/elixir/lib/code/formatter.ex

@@ -520,14 +520,15 @@ defmodule Code.Formatter do
               {string(~S{"..//":}), state}
 
             {:__block__, _, [atom]} when is_atom(atom) ->
-              key =
+              iodata =
                 if Macro.classify_atom(atom) in [:identifier, :unquoted] do
-                  IO.iodata_to_binary([Atom.to_string(atom), ?:])
+                  [Atom.to_string(atom), ?:]
                 else
-                  IO.iodata_to_binary([?", Atom.to_string(atom), ?", ?:])
+                  [?", atom |> Atom.to_string() |> String.replace("\"", "\\\""), ?", ?:]
                 end
 
-              {string(key) |> color(:atom, state.inspect_opts), state}
+              {iodata |> IO.iodata_to_binary() |> string() |> color(:atom, state.inspect_opts),
+               state}
 
             {{:., _, [:erlang, :binary_to_atom]}, _, [{:<<>>, _, entries}, :utf8]} ->
               interpolation_to_algebra(entries, @double_quote, state, "\"", "\":")

lib/elixir/lib/code/fragment.ex

@@ -636,7 +636,7 @@ defmodule Code.Fragment do
     {reversed_pre, post} = adjust_position(reversed_pre, post)
 
     case take_identifier(post, []) do
-      :none ->
+      {_, [], _} ->
         maybe_operator(reversed_pre, post, line, opts)
 
       {:identifier, reversed_post, rest} ->
@@ -644,7 +644,7 @@ defmodule Code.Fragment do
         reversed = reversed_post ++ reversed_pre
 
         case codepoint_cursor_context(reversed, opts) do
-          {{:struct, acc}, offset} when acc != [] ->
+          {{:struct, acc}, offset} ->
             build_surround({:struct, acc}, reversed, line, offset)
 
           {{:alias, acc}, offset} ->
@@ -749,27 +749,11 @@ defmodule Code.Fragment do
     do: take_identifier(t, [h | acc])
 
   defp take_identifier(rest, acc) do
-    {stripped, _} = strip_spaces(rest, 0)
-
-    with [?. | t] <- stripped,
+    with {[?. | t], _} <- strip_spaces(rest, 0),
          {[h | _], _} when h in ?A..?Z <- strip_spaces(t, 0) do
       take_alias(rest, acc)
     else
-      # Consider it an identifier if we are at the end of line
-      # or if we have spaces not followed by . (call) or / (arity)
-      _ when acc == [] and (rest == [] or (hd(rest) in @space and hd(stripped) not in ~c"/.")) ->
-        {:identifier, acc, rest}
-
-      # If we are immediately followed by a container, we are still part of the identifier.
-      # We don't consider << as it _may_ be an operator.
-      _ when acc == [] and hd(stripped) in ~c"({[" ->
-        {:identifier, acc, rest}
-
-      _ when acc == [] ->
-        :none
-
-      _ ->
-        {:identifier, acc, rest}
+      _ -> {:identifier, acc, rest}
     end
   end
 
@@ -1107,6 +1091,6 @@ defmodule Code.Fragment do
     opts =
       Keyword.take(opts, [:file, :line, :column, :columns, :token_metadata, :literal_encoder])
 
-    Code.string_to_quoted(fragment, [cursor_completion: true, warnings: false] ++ opts)
+    Code.string_to_quoted(fragment, [cursor_completion: true, emit_warnings: false] ++ opts)
   end
 end

lib/elixir/lib/code/normalizer.ex

@@ -349,13 +349,15 @@ defmodule Code.Normalizer do
         meta
       end
 
+    last = List.last(args)
+
     cond do
-      Keyword.has_key?(meta, :do) or match?([{{:__block__, _, [:do]}, _} | _], List.last(args)) ->
+      Keyword.has_key?(meta, :do) or match?([{{:__block__, _, [:do]}, _} | _], last) ->
         # def foo do :ok end
         # def foo, do: :ok
         normalize_kw_blocks(form, meta, args, state)
 
-      match?([{:do, _} | _], List.last(args)) ->
+      match?([{:do, _} | _], last) and Keyword.keyword?(last) ->
         # Non normalized kw blocks
         line = state.parent_meta[:line]
         meta = meta ++ [do: [line: line], end: [line: line]]

lib/elixir/lib/code/typespec.ex

@@ -175,7 +175,8 @@ defmodule Code.Typespec do
 
   defp get_module_and_beam(module) when is_atom(module) do
     with {^module, beam, _filename} <- :code.get_object_code(module),
-         {:ok, ^module} <- beam |> :beam_lib.info() |> Keyword.fetch(:module) do
+         info_pairs when is_list(info_pairs) <- :beam_lib.info(beam),
+         {:ok, ^module} <- Keyword.fetch(info_pairs, :module) do
       {module, beam}
     else
       _ -> :error
@@ -419,5 +420,13 @@ defmodule Code.Typespec do
     :error
   end
 
-  defp meta(anno), do: [line: :erl_anno.line(anno)]
+  defp meta(anno) do
+    case :erl_anno.location(anno) do
+      {line, column} ->
+        [line: line, column: column]
+
+      line when is_integer(line) ->
+        [line: line]
+    end
+  end
 end

lib/elixir/lib/config.ex

@@ -46,9 +46,9 @@ defmodule Config do
   ## Migrating from `use Mix.Config`
 
   The `Config` module in Elixir was introduced in v1.9 as a replacement to
-  `Mix.Config`, which was specific to Mix and has been deprecated.
+  `use Mix.Config`, which was specific to Mix and has been deprecated.
 
-  You can leverage `Config` instead of `Mix.Config` in three steps. The first
+  You can leverage `Config` instead of `use Mix.Config` in three steps. The first
   step is to replace `use Mix.Config` at the top of your config files by
   `import Config`.
 

lib/elixir/lib/exception.ex

@@ -793,6 +793,194 @@ defmodule Exception do
       col -> format_file_line_column(Keyword.get(opts, :file), Keyword.get(opts, :line), col, " ")
     end
   end
+
+  @doc false
+  def format_delimiter(delimiter) do
+    if delimiter |> Atom.to_string() |> String.contains?(["\"", "'"]),
+      do: delimiter,
+      else: ~s("#{delimiter}")
+  end
+
+  @doc false
+  def format_snippet(
+        {start_line, _start_column} = start_pos,
+        {end_line, end_column} = end_pos,
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
+      when start_line < end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+
+    relevant_lines =
+      if end_line - start_line < 5 do
+        line_range(lines, start_pos, end_pos, padding, max_digits, start_message, end_message)
+      else
+        trimmed_inbetween_lines(
+          lines,
+          start_pos,
+          end_pos,
+          padding,
+          max_digits,
+          start_message,
+          end_message
+        )
+      end
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{relevant_lines}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  def format_snippet(
+        {start_line, start_column},
+        {end_line, end_column},
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
+      when start_line == end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+    formatted_line = [line_padding(end_line, max_digits), to_string(end_line), " │ ", hd(lines)]
+
+    mismatched_closing_line =
+      [
+        n_spaces(start_column - 1),
+        red("│"),
+        format_end_message(end_column - start_column, end_message)
+      ]
+
+    unclosed_delimiter_line =
+      [padding, " │ ", format_start_message(start_column, start_message)]
+
+    below_line = [padding, " │ ", mismatched_closing_line, "\n", unclosed_delimiter_line]
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{formatted_line}
+    #{below_line}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  defp line_padding(line_number, max_digits) do
+    line_digits = digits(line_number)
+
+    spacing =
+      if line_digits == 1 do
+        max(2, max_digits)
+      else
+        max_digits - line_digits + 1
+      end
+
+    n_spaces(spacing)
+  end
+
+  defp n_spaces(n), do: String.duplicate(" ", n)
+
+  defp digits(number, acc \\ 1)
+  defp digits(number, acc) when number < 10, do: acc
+  defp digits(number, acc), do: digits(div(number, 10), acc + 1)
+
+  defp trimmed_inbetween_lines(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         start_message,
+         end_message
+       ) do
+    start_padding = line_padding(start_line, max_digits)
+    end_padding = line_padding(end_line, max_digits)
+    first_line = hd(lines)
+    last_line = List.last(lines)
+
+    """
+    #{start_padding}#{start_line} │ #{first_line}
+     #{padding}│ #{format_start_message(start_column, start_message)}
+     ...
+    #{end_padding}#{end_line} │ #{last_line}
+     #{padding}│ #{format_end_message(end_column, end_message)}\
+    """
+  end
+
+  defp line_range(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         start_message,
+         end_message
+       ) do
+    Enum.zip_with(lines, start_line..end_line, fn line, line_number ->
+      line_padding = line_padding(line_number, max_digits)
+
+      cond do
+        line_number == start_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            format_start_message(start_column, start_message)
+          ]
+
+        line_number == end_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            format_end_message(end_column, end_message)
+          ]
+
+        true ->
+          [line_padding, to_string(line_number), " │ ", line]
+      end
+    end)
+    |> Enum.intersperse("\n")
+  end
+
+  defp format_end_message(end_column, message),
+    do: [
+      n_spaces(end_column - 1),
+      red(message)
+    ]
+
+  defp format_start_message(start_column, message),
+    do: [n_spaces(start_column - 1), red(message)]
+
+  defp pad_message(message, padding), do: String.replace(message, "\n", "\n #{padding}")
+
+  defp red(string) do
+    if IO.ANSI.enabled?() do
+      [IO.ANSI.red(), string, IO.ANSI.reset()]
+    else
+      string
+    end
+  end
 end
 
 # Some exceptions implement "message/1" instead of "exception/1" mostly
@@ -944,11 +1132,24 @@ defmodule MismatchedDelimiterError do
   An exception raised when a mismatched delimiter is found when parsing code.
 
   For example:
-  - `[1, 2, 3}`
-  - `fn a -> )`
-  """
 
-  @max_lines_shown 5
+    * `[1, 2, 3}`
+    * `fn a -> )`
+
+  The following fields of this exceptions are public and can be accessed freely:
+
+    * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
+      the error occurred in code that did not come from a file
+    * `:line` - the line for the opening delimiter
+    * `:column` - the column for the opening delimiter
+    * `:end_line` - the line for the mismatched closing delimiter
+    * `:end_column` - the column for the mismatched closing delimiter
+    * `:opening_delimiter` - an atom representing the opening delimiter
+    * `:closing_delimiter` - an atom representing the mismatched closing delimiter
+    * `:expected_delimiter` - an atom representing the closing delimiter
+    * `:description` - a description of the mismatched delimiter error
+
+  """
 
   defexception [
     :file,
@@ -958,6 +1159,7 @@ defmodule MismatchedDelimiterError do
     :end_column,
     :opening_delimiter,
     :closing_delimiter,
+    :expected_delimiter,
     :snippet,
     description: "mismatched delimiter error"
   ]
@@ -969,206 +1171,32 @@ defmodule MismatchedDelimiterError do
         end_line: end_line,
         end_column: end_column,
         description: description,
-        opening_delimiter: opening_delimiter,
-        closing_delimiter: _closing_delimiter,
+        expected_delimiter: expected_delimiter,
         file: file,
         snippet: snippet
       }) do
     start_pos = {start_line, start_column}
     end_pos = {end_line, end_column}
     lines = String.split(snippet, "\n")
-    expected_delimiter = :elixir_tokenizer.terminator(opening_delimiter)
+    expected_delimiter = Exception.format_delimiter(expected_delimiter)
+
+    start_message = "└ unclosed delimiter"
+    end_message = ~s/└ mismatched closing delimiter (expected #{expected_delimiter})/
+
+    snippet =
+      Exception.format_snippet(
+        start_pos,
+        end_pos,
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
 
-    snippet = format_snippet(start_pos, end_pos, description, file, lines, expected_delimiter)
     format_message(file, end_line, end_column, snippet)
   end
 
-  defp format_snippet(
-         {start_line, _start_column} = start_pos,
-         {end_line, end_column} = end_pos,
-         description,
-         file,
-         lines,
-         expected_delimiter
-       )
-       when start_line < end_line do
-    max_digits = digits(end_line)
-    general_padding = max(2, max_digits) + 1
-    padding = n_spaces(general_padding)
-
-    relevant_lines =
-      if end_line - start_line < @max_lines_shown do
-        line_range(lines, start_pos, end_pos, padding, max_digits, expected_delimiter)
-      else
-        trimmed_inbetween_lines(
-          lines,
-          start_pos,
-          end_pos,
-          padding,
-          max_digits,
-          expected_delimiter
-        )
-      end
-
-    """
-     #{padding}#{red("error:")} #{pad_message(description, padding)}
-     #{padding}│
-    #{relevant_lines}
-     #{padding}│
-     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
-    """
-  end
-
-  defp format_snippet(
-         {start_line, start_column},
-         {end_line, end_column},
-         description,
-         file,
-         lines,
-         expected_delimiter
-       )
-       when start_line == end_line do
-    max_digits = digits(end_line)
-    general_padding = max(2, max_digits) + 1
-    padding = n_spaces(general_padding)
-
-    line = Enum.fetch!(lines, end_line - 1)
-    formatted_line = [line_padding(end_line, max_digits), to_string(end_line), " │ ", line]
-
-    mismatched_closing_line =
-      [
-        n_spaces(start_column - 1),
-        red("│"),
-        mismatched_closing_delimiter(end_column - start_column, expected_delimiter)
-      ]
-
-    unclosed_delimiter_line =
-      [padding, " │ ", unclosed_delimiter(start_column)]
-
-    below_line = [padding, " │ ", mismatched_closing_line, "\n", unclosed_delimiter_line]
-
-    """
-     #{padding}#{red("error:")} #{pad_message(description, padding)}
-     #{padding}│
-    #{formatted_line}
-    #{below_line}
-     #{padding}│
-     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
-    """
-  end
-
-  defp line_padding(line_number, max_digits) do
-    line_digits = digits(line_number)
-
-    spacing =
-      if line_digits == 1 do
-        max(2, max_digits)
-      else
-        max_digits - line_digits + 1
-      end
-
-    n_spaces(spacing)
-  end
-
-  defp n_spaces(n), do: String.duplicate(" ", n)
-
-  defp digits(number, acc \\ 1)
-  defp digits(number, acc) when number < 10, do: acc
-  defp digits(number, acc), do: digits(div(number, 10), acc + 1)
-
-  defp trimmed_inbetween_lines(
-         lines,
-         {start_line, start_column},
-         {end_line, end_column},
-         padding,
-         max_digits,
-         expected_delimiter
-       ) do
-    start_padding = line_padding(start_line, max_digits)
-    end_padding = line_padding(end_line, max_digits)
-    first_line = Enum.fetch!(lines, start_line - 1)
-    last_line = Enum.fetch!(lines, end_line - 1)
-
-    """
-    #{start_padding}#{start_line} │ #{first_line}
-     #{padding}│ #{unclosed_delimiter(start_column)}
-     ...
-    #{end_padding}#{end_line} │ #{last_line}
-     #{padding}│ #{mismatched_closing_delimiter(end_column, expected_delimiter)}\
-    """
-  end
-
-  defp line_range(
-         lines,
-         {start_line, start_column},
-         {end_line, end_column},
-         padding,
-         max_digits,
-         expected_delimiter
-       ) do
-    start_line = start_line - 1
-    end_line = end_line - 1
-
-    lines
-    |> Enum.slice(start_line..end_line)
-    |> Enum.zip_with(start_line..end_line, fn line, line_number ->
-      line_number = line_number + 1
-      start_line = start_line + 1
-      end_line = end_line + 1
-
-      line_padding = line_padding(line_number, max_digits)
-
-      cond do
-        line_number == start_line ->
-          [
-            line_padding,
-            to_string(line_number),
-            " │ ",
-            line,
-            "\n",
-            padding,
-            " │ ",
-            unclosed_delimiter(start_column)
-          ]
-
-        line_number == end_line ->
-          [
-            line_padding,
-            to_string(line_number),
-            " │ ",
-            line,
-            "\n",
-            padding,
-            " │ ",
-            mismatched_closing_delimiter(end_column, expected_delimiter)
-          ]
-
-        true ->
-          [line_padding, to_string(line_number), " │ ", line]
-      end
-    end)
-    |> Enum.intersperse("\n")
-  end
-
-  defp mismatched_closing_delimiter(end_column, expected_closing_delimiter),
-    do: [
-      n_spaces(end_column - 1),
-      red(~s/└ mismatched closing delimiter (expected "#{expected_closing_delimiter}")/)
-    ]
-
-  defp unclosed_delimiter(start_column),
-    do: [n_spaces(start_column - 1), red("└ unclosed delimiter")]
-
-  defp pad_message(message, padding), do: String.replace(message, "\n", "\n #{padding}")
-
-  defp red(string) do
-    if IO.ANSI.enabled?() do
-      [IO.ANSI.red(), string, IO.ANSI.reset()]
-    else
-      string
-    end
-  end
-
   defp format_message(file, line, column, message) do
     location = Exception.format_file_line_column(Path.relative_to_cwd(file), line, column)
     "mismatched delimiter found on " <> location <> "\n" <> message
@@ -1183,8 +1211,9 @@ defmodule SyntaxError do
 
     * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
       the error occurred in code that did not come from a file
-    * `:line` (`t:non_neg_integer/0`) - the line where the error occurred
-    * `:column` (`t:non_neg_integer/0`) - the column where the error occurred
+    * `:line` - the line where the error occurred
+    * `:column` - the column where the error occurred
+    * `:description` - a description of the syntax error
 
   """
 
@@ -1233,17 +1262,24 @@ defmodule TokenMissingError do
 
     * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
       the error occurred in code that did not come from a file
-    * `:line` (`t:non_neg_integer/0`) - the line where the error occurred
-    * `:column` (`t:non_neg_integer/0`) - the column where the error occurred
-
+    * `:line` - the line for the opening delimiter
+    * `:column` - the column for the opening delimiter
+    * `:end_line` - the line for the end of the string
+    * `:end_column` - the column for the end of the string
+    * `:opening_delimiter` - an atom representing the opening delimiter
+    * `:expected_delimiter` - an atom representing the expected delimiter
+    * `:description` - a description of the missing token error
   """
 
   defexception [
     :file,
     :line,
-    :snippet,
     :column,
+    :end_line,
+    :end_column,
+    :snippet,
     :opening_delimiter,
+    :expected_delimiter,
     description: "expression is incomplete"
   ]
 
@@ -1252,14 +1288,40 @@ defmodule TokenMissingError do
         file: file,
         line: line,
         column: column,
+        end_line: end_line,
         description: description,
+        expected_delimiter: expected_delimiter,
         snippet: snippet
       })
-      when not is_nil(snippet) and not is_nil(column) do
-    snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil)
+      when not is_nil(snippet) and not is_nil(column) and not is_nil(end_line) do
+    {trimmed, [last_line | _] = reversed_lines} =
+      snippet
+      |> String.split("\n")
+      |> Enum.reverse()
+      |> Enum.split_while(&(&1 == ""))
 
-    format_message(file, line, column, snippet)
+    end_line = end_line - length(trimmed)
+    end_column = String.length(last_line) + 1
+
+    start_pos = {line, column}
+    end_pos = {end_line, end_column}
+    expected_delimiter = Exception.format_delimiter(expected_delimiter)
+
+    start_message = ~s/└ unclosed delimiter/
+    end_message = ~s/└ missing closing delimiter (expected #{expected_delimiter})/
+
+    snippet =
+      Exception.format_snippet(
+        start_pos,
+        end_pos,
+        description,
+        file,
+        Enum.reverse(reversed_lines),
+        start_message,
+        end_message
+      )
+
+    format_message(file, end_line, end_column, snippet)
   end
 
   @impl true
@@ -1267,13 +1329,13 @@ defmodule TokenMissingError do
         file: file,
         line: line,
         column: column,
+        snippet: snippet,
         description: description
       }) do
     snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, nil, :error, [], nil)
+      :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil)
 
-    padded = "   " <> String.replace(snippet, "\n", "\n   ")
-    format_message(file, line, column, padded)
+    format_message(file, line, column, snippet)
   end
 
   defp format_message(file, line, column, message) do

lib/elixir/lib/file.ex

@@ -133,6 +133,7 @@ defmodule File do
 
   @type stream_mode ::
           encoding_mode()
+          | read_offset_mode()
           | :append
           | :compressed
           | :delayed_write
@@ -140,6 +141,8 @@ defmodule File do
           | {:read_ahead, pos_integer | false}
           | {:delayed_write, non_neg_integer, non_neg_integer}
 
+  @type read_offset_mode :: {:read_offset, non_neg_integer()}
+
   @type erlang_time ::
           {{year :: non_neg_integer(), month :: 1..12, day :: 1..31},
            {hour :: 0..23, minute :: 0..59, second :: 0..59}}
@@ -1748,13 +1751,15 @@ defmodule File do
   One may also consider passing the `:delayed_write` option if the stream
   is meant to be written to under a tight loop.
 
-  ## Byte order marks
+  ## Byte order marks and read offset
 
   If you pass `:trim_bom` in the modes parameter, the stream will
   trim UTF-8, UTF-16 and UTF-32 byte order marks when reading from file.
 
   Note that this function does not try to discover the file encoding
-  based on BOM.
+  based on BOM. From Elixir v1.16.0, you may also pass a `:read_offset`
+  that is skipped whenever enumerating the stream (if both `:read_offset`
+  and `:trim_bom` are given, the offset is skipped after the BOM).
 
   ## Examples
 

lib/elixir/lib/file/stream.ex

@@ -18,6 +18,12 @@ defmodule File.Stream do
 
   @doc false
   def __build__(path, line_or_bytes, modes) do
+    with {:read_offset, offset} <- :lists.keyfind(:read_offset, 1, modes),
+         false <- is_integer(offset) and offset >= 0 do
+      raise ArgumentError,
+            "expected :read_offset to be a non-negative integer, got: #{inspect(offset)}"
+    end
+
     raw = :lists.keyfind(:encoding, 1, modes) == false
 
     modes =
@@ -88,7 +94,7 @@ defmodule File.Stream do
       start_fun = fn ->
         case File.Stream.__open__(stream, read_modes(modes)) do
           {:ok, device} ->
-            if :trim_bom in modes, do: trim_bom(device, raw) |> elem(0), else: device
+            skip_bom_and_offset(device, raw, modes)
 
           {:error, reason} ->
             raise File.Error, reason: reason, action: "stream", path: stream.path
@@ -104,9 +110,14 @@ defmodule File.Stream do
       Stream.resource(start_fun, next_fun, &:file.close/1).(acc, fun)
     end
 
-    def count(%{modes: modes, line_or_bytes: :line, path: path} = stream) do
+    def count(%{modes: modes, line_or_bytes: :line, path: path, raw: raw} = stream) do
       pattern = :binary.compile_pattern("\n")
-      counter = &count_lines(&1, path, pattern, read_function(stream), 0)
+
+      counter = fn device ->
+        device = skip_bom_and_offset(device, raw, modes)
+        count_lines(device, path, pattern, read_function(stream), 0)
+      end
+
       {:ok, open!(stream, modes, counter)}
     end
 
@@ -116,8 +127,11 @@ defmodule File.Stream do
           {:error, __MODULE__}
 
         {:ok, %{size: size}} ->
+          bom_offset = count_raw_bom(stream, modes)
+          offset = get_read_offset(modes)
+          size = max(size - bom_offset - offset, 0)
           remainder = if rem(size, bytes) == 0, do: 0, else: 1
-          {:ok, div(size, bytes) + remainder - count_raw_bom(stream, modes)}
+          {:ok, div(size, bytes) + remainder}
 
         {:error, reason} ->
           raise File.Error, reason: reason, action: "stream", path: path
@@ -158,6 +172,23 @@ defmodule File.Stream do
       end
     end
 
+    defp skip_bom_and_offset(device, raw, modes) do
+      device =
+        if :trim_bom in modes do
+          device |> trim_bom(raw) |> elem(0)
+        else
+          device
+        end
+
+      offset = get_read_offset(modes)
+
+      if offset > 0 do
+        {:ok, _} = :file.position(device, {:cur, offset})
+      end
+
+      device
+    end
+
     defp trim_bom(device, true) do
       bom_length = device |> IO.binread(4) |> bom_length()
       {:ok, new_pos} = :file.position(device, bom_length)
@@ -183,6 +214,13 @@ defmodule File.Stream do
     defp bom_length(<<254, 255, 0, 0, _rest::binary>>), do: 4
     defp bom_length(_binary), do: 0
 
+    def get_read_offset(modes) do
+      case :lists.keyfind(:read_offset, 1, modes) do
+        {:read_offset, offset} -> offset
+        false -> 0
+      end
+    end
+
     defp read_modes(modes) do
       for mode <- modes, mode not in [:write, :append, :trim_bom], do: mode
     end

lib/elixir/lib/float.ex

@@ -198,7 +198,7 @@ defmodule Float do
   defp add_dot(acc, false), do: acc <> ".0"
 
   @doc """
-  Rounds a float to the largest number less than or equal to `num`.
+  Rounds a float to the largest float less than or equal to `number`.
 
   `floor/2` also accepts a precision to round a floating-point value down
   to an arbitrary number of fractional digits (between 0 and 15).
@@ -246,7 +246,7 @@ defmodule Float do
   end
 
   @doc """
-  Rounds a float to the smallest integer greater than or equal to `num`.
+  Rounds a float to the smallest float greater than or equal to `number`.
 
   `ceil/2` also accepts a precision to round a floating-point value down
   to an arbitrary number of fractional digits (between 0 and 15).

lib/elixir/lib/gen_server.ex

@@ -9,14 +9,10 @@ defmodule GenServer do
   tracing and error reporting. It will also fit into a supervision tree.
 
   ```mermaid
-  graph TD
-      GenServer
-      GenServer -. reply -.-> A
-      GenServer -. reply -.-> B
-      GenServer -. reply -.-> C
-      A(Client #1) -- request --> GenServer
-      B(Client #2) -- request --> GenServer
-      C(Client #3) -- request --> GenServer
+  graph BT
+      C(Client #3) ~~~ B(Client #2) ~~~ A(Client #1)
+      A & B & C -->|request| GenServer
+      GenServer -.->|reply| A & B & C
   ```
 
   ## Example
@@ -475,7 +471,7 @@ defmodule GenServer do
   guide provides a tutorial-like introduction. The documentation and links
   in Erlang can also provide extra insight.
 
-    * [GenServer - Elixir's Getting Started Guide](https://elixir-lang.org/getting-started/mix-otp/genserver.html)
+    * [GenServer - Elixir's Getting Started Guide](genservers.md)
     * [`:gen_server` module documentation](`:gen_server`)
     * [gen_server Behaviour - OTP Design Principles](https://www.erlang.org/doc/design_principles/gen_server_concepts.html)
     * [Clients and Servers - Learn You Some Erlang for Great Good!](http://learnyousomeerlang.com/clients-and-servers)

lib/elixir/lib/inspect/algebra.ex

@@ -143,9 +143,8 @@ defmodule Inspect.Opts do
   function as this must be controlled by applications. Libraries
   should instead define their own structs with custom inspect
   implementations. If a library must change the default inspect
-  function, then it is best to define to ask users of your library
-  to explicitly call `default_inspect_fun/1` with your function of
-  choice.
+  function, then it is best to ask users of your library to explicitly
+  call `default_inspect_fun/1` with your function of choice.
 
   The default is `Inspect.inspect/2`.
 

lib/elixir/lib/io.ex

@@ -308,17 +308,20 @@ defmodule IO do
       entry from the compilation environment will be used
 
     * a keyword list with at least the `:file` option representing
-      a single stacktrace entry (since v1.14.0). The `:line`, `:module`,
-      `:function` options are also supported
+      a single stacktrace entry (since v1.14.0). The `:line`, `:column`,
+      `:module`, and `:function` options are also supported
 
-  This function also notifies the compiler a warning was printed
-  (in case --warnings-as-errors was enabled). It returns `:ok`
-  if it succeeds.
+  This function notifies the compiler a warning was printed
+  and emits a compiler diagnostic (`t:Code.diagnostic/1`).
+  The diagnostic will include precise file and location information
+  if a `Macro.Env` is given or those values have been passed as
+  keyword list, but not for stacktraces, as they are often imprecise.
+
+  It returns `:ok` if it succeeds.
 
   ## Examples
 
-      stacktrace = [{MyApp, :main, 1, [file: 'my_app.ex', line: 4]}]
-      IO.warn("variable bar is unused", stacktrace)
+      IO.warn("variable bar is unused", module: MyApp, function: {:main, 1}, line: 4, file: "my_app.ex")
       #=> warning: variable bar is unused
       #=>   my_app.ex:4: MyApp.main/1
 
@@ -337,15 +340,22 @@ defmodule IO do
 
   def warn(message, [{_, _} | _] = keyword) do
     if file = keyword[:file] do
-      warn(
-        message,
-        %{
+      line = keyword[:line]
+      column = keyword[:column]
+      position = if line && column, do: {line, column}, else: line
+      message = to_chardata(message)
+
+      stacktrace =
+        Macro.Env.stacktrace(%{
           __ENV__
           | module: keyword[:module],
             function: keyword[:function],
-            line: keyword[:line],
+            line: line,
             file: file
-        }
+        })
+
+      :elixir_errors.emit_diagnostic(:warning, position, file, message, stacktrace,
+        read_snippet: true
       )
     else
       warn(message, [])

lib/elixir/lib/kernel.ex

@@ -2064,9 +2064,6 @@ defmodule Kernel do
       {var, _, nil} when is_atom(var) ->
         invalid_concat_left_argument_error(Atom.to_string(var))
 
-      {:^, _, [{var, _, nil}]} when is_atom(var) ->
-        invalid_concat_left_argument_error("^#{Atom.to_string(var)}")
-
       _ ->
         expanded_arg
     end
@@ -3625,7 +3622,7 @@ defmodule Kernel do
   defp do_at([], meta, name, function?, env) do
     IO.warn(
       "the @#{name}() notation (with parentheses) is deprecated, please use @#{name} (without parentheses) instead",
-      Macro.Env.stacktrace(env)
+      env
     )
 
     do_at(nil, meta, name, function?, env)

lib/elixir/lib/kernel/parallel_compiler.ex

@@ -859,9 +859,12 @@ defmodule Kernel.ParallelCompiler do
     )
 
     for {file, _, description, stacktrace} <- deadlock do
+      file = Path.absname(file)
+
       %{
         severity: :error,
-        file: Path.absname(file),
+        file: file,
+        source: file,
         position: nil,
         message: description,
         stacktrace: stacktrace,
@@ -889,50 +892,64 @@ defmodule Kernel.ParallelCompiler do
     ])
   end
 
-  defp to_error(file, kind, reason, stack) do
-    line = get_line(file, reason, stack)
-    file = Path.absname(file)
+  defp to_error(source, kind, reason, stack) do
+    {file, line, span} = get_snippet_info(source, reason, stack)
+    source = Path.absname(source)
     message = :unicode.characters_to_binary(Kernel.CLI.format_error(kind, reason, stack))
 
     %{
-      file: file,
+      file: file || source,
+      source: source,
       position: line || 0,
       message: message,
       severity: :error,
       stacktrace: stack,
-      span: nil
+      span: span,
+      details: {kind, reason}
     }
   end
 
-  defp get_line(_file, %{line: line, column: column}, _stack)
+  defp get_snippet_info(
+         _file,
+         %{file: file, line: line, column: column, end_line: end_line, end_column: end_column},
+         _stack
+       )
+       when is_integer(line) and line > 0 and is_integer(column) and column >= 0 and
+              is_integer(end_line) and end_line > 0 and is_integer(end_column) and end_column >= 0 do
+    {Path.absname(file), {line, column}, {end_line, end_column}}
+  end
+
+  defp get_snippet_info(_file, %{file: file, line: line, column: column}, _stack)
        when is_integer(line) and line > 0 and is_integer(column) and column >= 0 do
-    {line, column}
+    {Path.absname(file), {line, column}, nil}
   end
 
-  defp get_line(_file, %{line: line}, _stack) when is_integer(line) and line > 0 do
-    line
+  defp get_snippet_info(_file, %{line: line}, _stack) when is_integer(line) and line > 0 do
+    {nil, line, nil}
   end
 
-  defp get_line(file, :undef, [{_, _, _, []}, {_, _, _, info} | _]) do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+  defp get_snippet_info(file, :undef, [{_, _, _, []}, {_, _, _, info} | _]) do
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(file, _reason, [{_, _, _, [file: expanding]}, {_, _, _, info} | _])
+  defp get_snippet_info(file, _reason, [{_, _, _, [file: expanding]}, {_, _, _, info} | _])
        when expanding in [~c"expanding macro", ~c"expanding struct"] do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(file, _reason, [{_, _, _, info} | _]) do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+  defp get_snippet_info(file, _reason, [{_, _, _, info} | _]) do
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(_, _, _) do
-    nil
+  defp get_snippet_info(_, _, _) do
+    {nil, nil, nil}
+  end
+
+  defp get_snippet_info_from_stacktrace_info(info, file) do
+    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
+      {nil, Keyword.get(info, :line), nil}
+    else
+      {nil, nil, nil}
+    end
   end
 end

lib/elixir/lib/kernel/typespec.ex

@@ -385,17 +385,17 @@ defmodule Kernel.Typespec do
       compile_error(caller, error)
     end
 
-    line = line(meta)
+    location = location(meta)
     vars = Keyword.keys(guard)
 
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
     {return, state} = typespec(return, vars, caller, state)
-    spec = {:type, line, :fun, [{:type, line, :product, args}, return]}
+    spec = {:type, location, :fun, [{:type, location, :product, args}, return]}
 
     {spec, state} =
       case guard_to_constraints(guard, vars, meta, caller, state) do
         {[], state} -> {spec, state}
-        {constraints, state} -> {{:type, line, :bounded_fun, [spec, constraints]}, state}
+        {constraints, state} -> {{:type, location, :bounded_fun, [spec, constraints]}, state}
       end
 
     ensure_no_unused_local_vars!(caller, state.local_vars)
@@ -437,7 +437,7 @@ defmodule Kernel.Typespec do
   defp ensure_not_default(_), do: :ok
 
   defp guard_to_constraints(guard, vars, meta, caller, state) do
-    line = line(meta)
+    location = location(meta)
 
     fun = fn
       {_name, {:var, _, context}}, {constraints, state} when is_atom(context) ->
@@ -445,9 +445,9 @@ defmodule Kernel.Typespec do
 
       {name, type}, {constraints, state} ->
         {spec, state} = typespec(type, vars, caller, state)
-        constraint = [{:atom, line, :is_subtype}, [{:var, line, name}, spec]]
+        constraint = [{:atom, location, :is_subtype}, [{:var, location, name}, spec]]
         state = update_local_vars(state, name)
-        {[{:type, line, :constraint, constraint} | constraints], state}
+        {[{:type, location, :constraint, constraint} | constraints], state}
     end
 
     {constraints, state} = :lists.foldl(fun, {[], state}, guard)
@@ -456,21 +456,27 @@ defmodule Kernel.Typespec do
 
   ## To typespec conversion
 
-  defp line(meta) do
-    Keyword.get(meta, :line, 0)
+  defp location(meta) do
+    line = Keyword.get(meta, :line, 0)
+
+    if column = Keyword.get(meta, :column) do
+      {line, column}
+    else
+      line
+    end
   end
 
   # Handle unions
   defp typespec({:|, meta, [_, _]} = exprs, vars, caller, state) do
     exprs = collect_union(exprs)
     {union, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, exprs)
-    {{:type, line(meta), :union, union}, state}
+    {{:type, location(meta), :union, union}, state}
   end
 
   # Handle binaries
   defp typespec({:<<>>, meta, []}, _, _, state) do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line, 0}, {:integer, line, 0}]}, state}
+    location = location(meta)
+    {{:type, location, :binary, [{:integer, location, 0}, {:integer, location, 0}]}, state}
   end
 
   defp typespec(
@@ -480,14 +486,18 @@ defmodule Kernel.Typespec do
          state
        )
        when is_atom(ctx1) and is_atom(ctx2) and unit in 1..256 do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line, 0}, {:integer, line(unit_meta), unit}]}, state}
+    location = location(meta)
+
+    {{:type, location, :binary, [{:integer, location, 0}, {:integer, location(unit_meta), unit}]},
+     state}
   end
 
   defp typespec({:<<>>, meta, [{:"::", size_meta, [{:_, _, ctx}, size]}]}, _, _, state)
        when is_atom(ctx) and is_integer(size) and size >= 0 do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line(size_meta), size}, {:integer, line, 0}]}, state}
+    location = location(meta)
+
+    {{:type, location, :binary, [{:integer, location(size_meta), size}, {:integer, location, 0}]},
+     state}
   end
 
   defp typespec(
@@ -505,8 +515,8 @@ defmodule Kernel.Typespec do
        )
        when is_atom(ctx1) and is_atom(ctx2) and is_atom(ctx3) and is_integer(size) and
               size >= 0 and unit in 1..256 do
-    args = [{:integer, line(size_meta), size}, {:integer, line(unit_meta), unit}]
-    {{:type, line(meta), :binary, args}, state}
+    args = [{:integer, location(size_meta), size}, {:integer, location(unit_meta), unit}]
+    {{:type, location(meta), :binary, args}, state}
   end
 
   defp typespec({:<<>>, _meta, _args}, _vars, caller, _state) do
@@ -519,7 +529,7 @@ defmodule Kernel.Typespec do
 
   ## Handle maps and structs
   defp typespec({:map, meta, args}, _vars, _caller, state) when args == [] or is_atom(args) do
-    {{:type, line(meta), :map, :any}, state}
+    {{:type, location(meta), :map, :any}, state}
   end
 
   defp typespec({:%{}, meta, fields} = map, vars, caller, state) do
@@ -527,17 +537,17 @@ defmodule Kernel.Typespec do
       {{:required, meta2, [k]}, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta2), :map_field_exact, [arg1, arg2]}, state}
+        {{:type, location(meta2), :map_field_exact, [arg1, arg2]}, state}
 
       {{:optional, meta2, [k]}, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta2), :map_field_assoc, [arg1, arg2]}, state}
+        {{:type, location(meta2), :map_field_assoc, [arg1, arg2]}, state}
 
       {k, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta), :map_field_exact, [arg1, arg2]}, state}
+        {{:type, location(meta), :map_field_exact, [arg1, arg2]}, state}
 
       {:|, _, [_, _]}, _state ->
         error =
@@ -551,7 +561,7 @@ defmodule Kernel.Typespec do
     end
 
     {fields, state} = :lists.mapfoldl(fun, state, fields)
-    {{:type, line(meta), :map, fields}, state}
+    {{:type, location(meta), :map, fields}, state}
   end
 
   defp typespec({:%, _, [name, {:%{}, meta, fields}]} = node, vars, caller, state) do
@@ -644,7 +654,7 @@ defmodule Kernel.Typespec do
     {right, state} = typespec(right, vars, caller, state)
     :ok = validate_range(left, right, caller)
 
-    {{:type, line(meta), :range, [left, right]}, state}
+    {{:type, location(meta), :range, [left, right]}, state}
   end
 
   # Handle special forms
@@ -668,7 +678,7 @@ defmodule Kernel.Typespec do
         pair -> pair
       end
 
-    {{:type, line(meta), :fun, fun_args}, state}
+    {{:type, location(meta), :fun, fun_args}, state}
   end
 
   # Handle type operator
@@ -691,10 +701,10 @@ defmodule Kernel.Typespec do
         # This may be generating an invalid typespec but we need to generate it
         # to avoid breaking existing code that was valid but only broke Dialyzer
         {right, state} = typespec(expr, vars, caller, state)
-        {{:ann_type, line(meta), [{:var, line(var_meta), var_name}, right]}, state}
+        {{:ann_type, location(meta), [{:var, location(var_meta), var_name}, right]}, state}
 
       {right, state} ->
-        {{:ann_type, line(meta), [{:var, line(var_meta), var_name}, right]}, state}
+        {{:ann_type, location(meta), [{:var, location(var_meta), var_name}, right]}, state}
     end
   end
 
@@ -723,13 +733,13 @@ defmodule Kernel.Typespec do
     {left, state} = typespec(left, vars, caller, state)
     state = %{state | undefined_type_error_enabled?: true}
     {right, state} = typespec(right, vars, caller, state)
-    {{:ann_type, line(meta), [left, right]}, state}
+    {{:ann_type, location(meta), [left, right]}, state}
   end
 
   # Handle unary ops
   defp typespec({op, meta, [integer]}, _, _, state) when op in [:+, :-] and is_integer(integer) do
-    line = line(meta)
-    {{:op, line, op, {:integer, line, integer}}, state}
+    location = location(meta)
+    {{:op, location, op, {:integer, location, integer}}, state}
   end
 
   # Handle remote calls in the form of @module_attribute.type.
@@ -778,12 +788,12 @@ defmodule Kernel.Typespec do
 
   # Handle tuples
   defp typespec({:tuple, meta, []}, _vars, _caller, state) do
-    {{:type, line(meta), :tuple, :any}, state}
+    {{:type, location(meta), :tuple, :any}, state}
   end
 
   defp typespec({:{}, meta, t}, vars, caller, state) when is_list(t) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, t)
-    {{:type, line(meta), :tuple, args}, state}
+    {{:type, location(meta), :tuple, args}, state}
   end
 
   defp typespec({left, right}, vars, caller, state) do
@@ -799,7 +809,7 @@ defmodule Kernel.Typespec do
   defp typespec({name, meta, atom}, vars, caller, state) when is_atom(atom) do
     if :lists.member(name, vars) do
       state = update_local_vars(state, name)
-      {{:var, line(meta), name}, state}
+      {{:var, location(meta), name}, state}
     else
       typespec({name, meta, []}, vars, caller, state)
     end
@@ -814,7 +824,7 @@ defmodule Kernel.Typespec do
 
     IO.warn(warning, caller)
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :string, args}, state}
+    {{:type, location(meta), :string, args}, state}
   end
 
   defp typespec({:nonempty_string, meta, args}, vars, caller, state) do
@@ -825,7 +835,7 @@ defmodule Kernel.Typespec do
 
     IO.warn(warning, caller)
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :nonempty_string, args}, state}
+    {{:type, location(meta), :nonempty_string, args}, state}
   end
 
   defp typespec({type, _meta, []}, vars, caller, state) when type in [:charlist, :char_list] do
@@ -855,7 +865,7 @@ defmodule Kernel.Typespec do
 
   defp typespec({:fun, meta, args}, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :fun, args}, state}
+    {{:type, location(meta), :fun, args}, state}
   end
 
   defp typespec({:..., _meta, _args}, _vars, caller, _state) do
@@ -872,7 +882,7 @@ defmodule Kernel.Typespec do
 
     case :erl_internal.is_type(name, arity) do
       true ->
-        {{:type, line(meta), name, args}, state}
+        {{:type, location(meta), name, args}, state}
 
       false ->
         if state.undefined_type_error_enabled? and
@@ -890,7 +900,7 @@ defmodule Kernel.Typespec do
             %{state | used_type_pairs: [{name, arity} | state.used_type_pairs]}
           end
 
-        {{:user_type, line(meta), name, args}, state}
+        {{:user_type, location(meta), name, args}, state}
     end
   end
 
@@ -963,7 +973,7 @@ defmodule Kernel.Typespec do
 
   defp remote_type({remote, meta, name, args}, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:remote_type, line(meta), [remote, name, args]}, state}
+    {{:remote_type, location(meta), [remote, name, args]}, state}
   end
 
   defp collect_union({:|, _, [a, b]}), do: [a | collect_union(b)]
@@ -996,16 +1006,16 @@ defmodule Kernel.Typespec do
   end
 
   defp fn_args(meta, [{:..., _, _}], _vars, _caller, state) do
-    {{:type, line(meta), :any}, state}
+    {{:type, location(meta), :any}, state}
   end
 
   defp fn_args(meta, args, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :product, args}, state}
+    {{:type, location(meta), :product, args}, state}
   end
 
   defp variable({name, meta, args}) when is_atom(name) and is_atom(args) do
-    {:var, line(meta), name}
+    {:var, location(meta), name}
   end
 
   defp variable(expr), do: expr

lib/elixir/lib/kernel/utils.ex

@@ -36,14 +36,14 @@ defmodule Kernel.Utils do
     if is_list(funs) do
       IO.warn(
         "passing a list to Kernel.defdelegate/2 is deprecated, please define each delegate separately",
-        Macro.Env.stacktrace(env)
+        env
       )
     end
 
     if Keyword.has_key?(opts, :append_first) do
       IO.warn(
         "Kernel.defdelegate/2 :append_first option is deprecated",
-        Macro.Env.stacktrace(env)
+        env
       )
     end
 

lib/elixir/lib/module.ex

@@ -326,8 +326,17 @@ defmodule Module do
   ### `@nifs` (since v1.16.0)
 
   A list of functions and their arities which will be overridden
-  by a native implementation (NIF). See the Erlang documentation
-  for more information: https://www.erlang.org/doc/man/erl_nif
+  by a native implementation (NIF).
+
+      defmodule MyLibrary.MyModule do
+        @nifs [foo: 1, bar: 2]
+
+        def foo(arg1), do: :erlang.nif_error(:not_loaded)
+        def bar(arg1, arg2), do: :erlang.nif_error(:not_loaded)
+      end
+
+  See the Erlang documentation for more information:
+  https://www.erlang.org/doc/man/erl_nif
 
   ### `@on_definition`
 
@@ -2124,8 +2133,7 @@ defmodule Module do
   end
 
   defp attribute_stack(module, line) do
-    file = String.to_charlist(Path.relative_to_cwd(:elixir_module.file(module)))
-    [{module, :__MODULE__, 0, file: file, line: line}]
+    struct!(Macro.Env, module: module, file: :elixir_module.file(module), line: line)
   end
 
   ## Helpers

lib/elixir/lib/module/parallel_checker.ex

@@ -324,6 +324,7 @@ defmodule Module.ParallelChecker do
   defp to_diagnostic(message, {file, position, mfa}) when is_list(position) do
     %{
       severity: :warning,
+      source: file,
       file: file,
       position: position_to_tuple(position),
       message: IO.iodata_to_binary(message),
@@ -334,7 +335,7 @@ defmodule Module.ParallelChecker do
 
   defp position_to_tuple(position) do
     case position[:column] do
-      nil -> position[:line]
+      nil -> position[:line] || 0
       col -> {position[:line], col}
     end
   end

lib/elixir/lib/path.ex

@@ -44,17 +44,17 @@ defmodule Path do
   """
   @spec absname(t) :: binary
   def absname(path) do
-    absname(path, File.cwd!())
+    absname(path, &File.cwd!/0)
   end
 
   @doc """
   Builds a path from `relative_to` to `path`.
 
   If `path` is already an absolute path, `relative_to` is ignored. See also
-  `relative_to/3`.
+  `relative_to/3`. `relative_to` is either a path or an anonymous function,
+  which is invoked only when necessary, that returns a path.
 
-  Unlike `expand/2`, no attempt is made to
-  resolve `..`, `.` or `~`.
+  Unlike `expand/2`, no attempt is made to resolve `..`, `.` or `~`.
 
   ## Examples
 
@@ -65,19 +65,33 @@ defmodule Path do
       "bar/../x"
 
   """
-  @spec absname(t, t) :: binary
+  @spec absname(t, t | (-> t)) :: binary
   def absname(path, relative_to) do
     path = IO.chardata_to_string(path)
 
     case type(path) do
       :relative ->
+        relative_to =
+          if is_function(relative_to, 0) do
+            relative_to.()
+          else
+            relative_to
+          end
+
         absname_join([relative_to, path])
 
       :absolute ->
         absname_join([path])
 
       :volumerelative ->
-        relative_to = IO.chardata_to_string(relative_to)
+        relative_to =
+          if is_function(relative_to, 0) do
+            relative_to.()
+          else
+            relative_to
+          end
+          |> IO.chardata_to_string()
+
         absname_vr(split(path), split(relative_to), relative_to)
     end
   end
@@ -163,7 +177,7 @@ defmodule Path do
   """
   @spec expand(t) :: binary
   def expand(path) do
-    expand_dot(absname(expand_home(path), File.cwd!()))
+    expand_dot(absname(expand_home(path), &File.cwd!/0))
   end
 
   @doc """
@@ -192,7 +206,7 @@ defmodule Path do
   """
   @spec expand(t, t) :: binary
   def expand(path, relative_to) do
-    expand_dot(absname(absname(expand_home(path), expand_home(relative_to)), File.cwd!()))
+    expand_dot(absname(absname(expand_home(path), expand_home(relative_to)), &File.cwd!/0))
   end
 
   @doc """
@@ -745,9 +759,9 @@ defmodule Path do
   You may call `Path.expand/1` to normalize the path before invoking
   this function.
 
-  A character preceded by \ loses its special meaning.
-  Note that \ must be written as \\ in a string literal.
-  For example, "\\?*" will match any filename starting with ?.
+  A character preceded by `\\` loses its special meaning.
+  Note that `\\` must be written as `\\\\` in a string literal.
+  For example, `"\\\\?*"` will match any filename starting with `?.`.
 
   By default, the patterns `*` and `?` do not match files starting
   with a dot `.`. See the `:match_dot` option in the "Options" section

lib/elixir/lib/process.ex

@@ -504,7 +504,7 @@ defmodule Process do
   If the process is already dead when calling `Process.monitor/1`, a
   `:DOWN` message is delivered immediately.
 
-  See ["The need for monitoring"](https://elixir-lang.org/getting-started/mix-otp/genserver.html#the-need-for-monitoring)
+  See ["The need for monitoring"](genservers.md#the-need-for-monitoring)
   for an example. See `:erlang.monitor/2` for more information.
 
   Inlined by the compiler.

lib/elixir/lib/registry.ex

@@ -27,8 +27,8 @@ defmodule Registry do
   `Registry.start_link/1`, it can be used to register and access named
   processes using the `{:via, Registry, {registry, key}}` tuple:
 
-      {:ok, _} = Registry.start_link(keys: :unique, name: Registry.ViaTest)
-      name = {:via, Registry, {Registry.ViaTest, "agent"}}
+      {:ok, _} = Registry.start_link(keys: :unique, name: MyApp.Registry)
+      name = {:via, Registry, {MyApp.Registry, "agent"}}
       {:ok, _} = Agent.start_link(fn -> 0 end, name: name)
       Agent.get(name, & &1)
       #=> 0
@@ -39,22 +39,22 @@ defmodule Registry do
   In the previous example, we were not interested in associating a value to the
   process:
 
-      Registry.lookup(Registry.ViaTest, "agent")
+      Registry.lookup(MyApp.Registry, "agent")
       #=> [{self(), nil}]
 
   However, in some cases it may be desired to associate a value to the process
   using the alternate `{:via, Registry, {registry, key, value}}` tuple:
 
-      {:ok, _} = Registry.start_link(keys: :unique, name: Registry.ViaTest)
-      name = {:via, Registry, {Registry.ViaTest, "agent", :hello}}
+      {:ok, _} = Registry.start_link(keys: :unique, name: MyApp.Registry)
+      name = {:via, Registry, {MyApp.Registry, "agent", :hello}}
       {:ok, agent_pid} = Agent.start_link(fn -> 0 end, name: name)
-      Registry.lookup(Registry.ViaTest, "agent")
+      Registry.lookup(MyApp.Registry, "agent")
       #=> [{agent_pid, :hello}]
 
   To this point, we have been starting `Registry` using `start_link/1`.
   Typically the registry is started as part of a supervision tree though:
 
-      {Registry, keys: :unique, name: Registry.ViaTest}
+      {Registry, keys: :unique, name: MyApp.Registry}
 
   Only registries with unique keys can be used in `:via`. If the name is
   already taken, the case-specific `start_link` function (`Agent.start_link/2`

lib/elixir/lib/string.ex

@@ -1871,6 +1871,104 @@ defmodule String do
     end
   end
 
+  defguardp replace_invalid_ii_of_iii(i, ii)
+            when Bitwise.bor(Bitwise.bsl(i, 6), ii) in 32..863 or
+                   Bitwise.bor(Bitwise.bsl(i, 6), ii) in 896..1023
+
+  defguardp replace_invalid_ii_of_iv(i, ii)
+            when Bitwise.bor(Bitwise.bsl(i, 6), ii) in 16..271
+
+  defguardp replace_invalid_iii_of_iv(i, ii, iii)
+            when Bitwise.bor(Bitwise.bor(Bitwise.bsl(i, 12), Bitwise.bsl(ii, 6)), iii) in 1024..17407
+
+  defguardp replace_invalid_is_next(next) when Bitwise.bsr(next, 6) !== 0b10
+
+  @doc ~S"""
+  Returns a new string created by replacing all invalid bytes with `replacement` (`"�"` by default).
+
+  ## Examples
+
+      iex> String.replace_invalid("asd" <> <<0xFF::8>>)
+      "asd�"
+
+      iex> String.replace_invalid("nem rán bề bề")
+      "nem rán bề bề"
+
+      iex> String.replace_invalid("nem rán b" <> <<225, 187>> <> " bề")
+      "nem rán b� bề"
+
+      iex> String.replace_invalid("nem rán b" <> <<225, 187>> <> " bề", "ERROR!")
+      "nem rán bERROR! bề"
+  """
+  @doc since: "1.16.0"
+  def replace_invalid(bytes, replacement \\ "�")
+      when is_binary(bytes) and is_binary(replacement) do
+    do_replace_invalid(bytes, replacement, <<>>)
+  end
+
+  # Valid ASCII (for better average speed)
+  defp do_replace_invalid(<<ascii::8, next::8, _::bytes>> = rest, rep, acc)
+       when ascii in 0..127 and replace_invalid_is_next(next) do
+    <<_::8, rest::bytes>> = rest
+    do_replace_invalid(rest, rep, acc <> <<ascii::8>>)
+  end
+
+  # Valid UTF-8
+  defp do_replace_invalid(<<grapheme::utf8, rest::bytes>>, rep, acc) do
+    do_replace_invalid(rest, rep, acc <> <<grapheme::utf8>>)
+  end
+
+  # 2/3 truncated sequence
+  defp do_replace_invalid(<<0b1110::4, i::4, 0b10::2, ii::6>>, rep, acc)
+       when replace_invalid_ii_of_iii(i, ii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(<<0b1110::4, i::4, 0b10::2, ii::6, next::8, _::bytes>> = rest, rep, acc)
+       when replace_invalid_ii_of_iii(i, ii) and replace_invalid_is_next(next) do
+    <<_::16, rest::bytes>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # 2/4
+  defp do_replace_invalid(<<0b11110::5, i::3, 0b10::2, ii::6>>, rep, acc)
+       when replace_invalid_ii_of_iv(i, ii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(
+         <<0b11110::5, i::3, 0b10::2, ii::6, next::8, _::bytes>> = rest,
+         rep,
+         acc
+       )
+       when replace_invalid_ii_of_iv(i, ii) and replace_invalid_is_next(next) do
+    <<_::16, rest::bytes>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # 3/4
+  defp do_replace_invalid(<<0b11110::5, i::3, 0b10::2, ii::6, 0b10::2, iii::6>>, rep, acc)
+       when replace_invalid_iii_of_iv(i, ii, iii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(
+         <<0b11110::5, i::3, 0b10::2, ii::6, 0b10::2, iii::6, next::8, _::bytes>> = rest,
+         rep,
+         acc
+       )
+       when replace_invalid_iii_of_iv(i, ii, iii) and replace_invalid_is_next(next) do
+    <<_::24, rest::bytes>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # Everything else
+  defp do_replace_invalid(<<_, rest::bytes>>, rep, acc),
+    do: do_replace_invalid(rest, rep, acc <> rep)
+
+  # Final
+  defp do_replace_invalid(<<>>, _, acc), do: acc
+
   @doc ~S"""
   Splits the string into chunks of characters that share a common trait.
 

lib/elixir/lib/supervisor.ex

@@ -413,10 +413,10 @@ defmodule Supervisor do
   The difference between the two approaches is that a module-based
   supervisor gives you more direct control over how the supervisor
   is initialized. Instead of calling `Supervisor.start_link/2` with
-  a list of child specifications that are automatically initialized, we manually
-  initialize the children by calling `Supervisor.init/2` inside its
-  `c:init/1` callback. `Supervisor.init/2` accepts the same `:strategy`,
-  `:max_restarts`, and `:max_seconds` options as `start_link/2`.
+  a list of child specifications that are implicitly initialized for us,
+  we must explicitly initialize the children by calling `Supervisor.init/2`
+  inside its `c:init/1` callback. `Supervisor.init/2` accepts the same
+  `:strategy`, `:max_restarts`, and `:max_seconds` options as `start_link/2`.
 
   > #### `use Supervisor` {: .info}
   >

lib/elixir/lib/task.ex

@@ -44,10 +44,38 @@ defmodule Task do
        means that, if the caller crashes, the task will crash
        too and vice-versa. This is on purpose: if the process
        meant to receive the result no longer exists, there is
-       no purpose in completing the computation.
+       no purpose in completing the computation. If this is not
+       desired, you will want to use supervised tasks, described
+       in a subsequent section.
 
-       If this is not desired, you will want to use supervised
-       tasks, described next.
+  ## Tasks are processes
+
+  Tasks are processes and so data will need to be completely copied
+  to them. Take the following code as an example:
+
+      large_data = fetch_large_data()
+      task = Task.async(fn -> do_some_work(large_data) end)
+      res = do_some_other_work()
+      res + Task.await(task)
+
+  The code above copies over all of `large_data`, which can be
+  resource intensive depending on the size of the data.
+  There are two ways to address this.
+
+  First, if you need to access only part of `large_data`,
+  consider extracting it before the task:
+
+      large_data = fetch_large_data()
+      subset_data = large_data.some_field
+      task = Task.async(fn -> do_some_work(subset_data) end)
+
+  Alternatively, if you can move the data loading altogether
+  to the task, it may be even better:
+
+      task = Task.async(fn ->
+        large_data = fetch_large_data()
+        do_some_work(large_data)
+      end)
 
   ## Dynamically supervised tasks
 

lib/elixir/lib/uri.ex

@@ -362,22 +362,24 @@ defmodule URI do
   end
 
   @doc """
-  Percent-escapes all characters that require escaping in `string`.
+  Percent-encodes all characters that require escaping in `string`.
 
-  This means reserved characters, such as `:` and `/`, and the
-  so-called unreserved characters, which have the same meaning both
-  escaped and unescaped, won't be escaped by default.
+  By default, this function is meant to escape the whole URI, and
+  therefore it will escape all characters which are foreign to the
+  URI specification. Reserved characters (such as `:` and `/`) or
+  unreserved (such as letters and numbers) are not escaped.
+
+  Because different components of a URI require different escaping
+  rules, this function also accepts a `predicate` function as an optional
+  argument. If passed, this function will be called with each byte
+  in `string` as its argument and should return a truthy value (anything other
+  than `false` or `nil`) if the given byte should be left as is, or
+  return a falsy value (`false` or `nil`) if the character should be
+  escaped. Defaults to `URI.char_unescaped?/1`.
 
   See `encode_www_form/1` if you are interested in escaping reserved
   characters too.
 
-  This function also accepts a `predicate` function as an optional
-  argument. If passed, this function will be called with each byte
-  in `string` as its argument and should return a truthy value (anything other
-  than `false` or `nil`) if the given byte should be left as is, or return a
-  falsy value (`false` or `nil`) if the character should be escaped. Defaults
-  to `URI.char_unescaped?/1`.
-
   ## Examples
 
       iex> URI.encode("ftp://s-ite.tld/?value=put it+й")

lib/elixir/pages/anti-patterns/code-anti-patterns.md

@@ -1,6 +1,6 @@
 # Code-related anti-patterns
 
-This document outlines anti-patterns related to your code and particular Elixir idioms and features.
+This document outlines potential anti-patterns related to your code and particular Elixir idioms and features.
 
 ## Comments
 
@@ -34,7 +34,8 @@ You could refactor the code above like this:
 @five_min_in_seconds 60 * 5
 
 defp unix_five_min_from_now do
-  unix_now = DateTime.to_unix(DateTime.utc_now(), :second)
+  now = DateTime.utc_now()
+  unix_now = DateTime.to_unix(now, :second)
   unix_now + @five_min_in_seconds
 end
 ```
@@ -45,75 +46,6 @@ We removed the unnecessary comments. We also added a `@five_min_in_seconds` modu
 
 Elixir makes a clear distinction between **documentation** and code comments. The language has built-in first-class support for documentation through `@doc`, `@moduledoc`, and more. See the ["Writing documentation"](../getting-started/writing-documentation.md) guide for more information.
 
-## Complex branching
-
-#### Problem
-
-When a function assumes the responsibility of handling multiple errors alone, it can increase its cyclomatic complexity (metric of control-flow) and become incomprehensible. This situation can configure a specific instance of "Long function", a traditional anti-pattern, but has implications of its own. Under these circumstances, this function could get very confusing, difficult to maintain and test, and therefore bug-proneness.
-
-#### Example
-
-An example of this anti-pattern is when a function uses the `case` control-flow structure or other similar constructs (for example, `cond` or `receive`) to handle variations of a return type. This practice can make the function more complex, long, and difficult to understand, as shown next.
-
-```elixir
-def get_customer(customer_id) do
-  case SomeHTTPClient.get("/customers/#{customer_id}") do
-    {:ok, %{status: 200, body: body}} ->
-      case Jason.decode(body) do
-        {:ok, decoded} ->
-          %{
-            "first_name" => first_name,
-            "last_name" => last_name,
-            "company" => company
-          } = decoded
-
-          customer =
-            %Customer{
-              id: customer_id,
-              name: "#{first_name} #{last_name}",
-              company: company
-            }
-
-          {:ok, customer}
-
-        {:error, _} ->
-          {:error, "invalid response body"}
-      end
-
-    {:error, %{status: status, body: body}} ->
-      case Jason.decode(body) do
-        %{"error" => message} when is_binary(message) ->
-          {:error, message}
-
-        %{} ->
-          {:error, "invalid response with status #{status}"}
-      end
-  end
-end
-```
-
-The code above is complex because the `case` clauses are long and often have their own branching logic in them. With the clauses spread out, it is hard to understand what each clause does individually and it is hard to see all of the different scenarios your code pattern matches on.
-
-#### Refactoring
-
-As shown below, in this situation, instead of concentrating all error handling within the same function, creating complex branches, it is better to delegate each branch to a different private function. In this way, the code will be cleaner and more readable.
-
-```elixir
-def get_customer(customer_id) do
-  case SomeHTTPClient.get("/customers/#{customer_id}") do
-    {:ok, %{status: 200, body: body}} ->
-      http_customer_to_struct(customer_id, body)
-
-    {:error, %{status: status, body: body}} ->
-      http_error(status, body)
-  end
-end
-```
-
-Both `http_customer_to_struct(customer_id, body)` and `http_error(status, body)` above contain the previous branches refactored into private functions.
-
-It is worth noting that this refactoring is trivial to perform in Elixir because clauses cannot define variables or otherwise affect their parent scope. Therefore, extracting any clause or branch to a private function is a matter of gathering all variables used in that branch and passing them as arguments to the new function.
-
 ## Complex `else` clauses in `with`
 
 #### Problem
@@ -169,11 +101,11 @@ end
 
 #### Problem
 
-When we use multi-clause functions, it is possible to extract values in the clauses for further usage and for pattern matching/guard checking. This extraction itself does not represent an anti-pattern, but when you have too many clauses or too many arguments, it becomes hard to know which extracted parts are used for pattern/guards and what is used only inside the function body. This anti-pattern is related to [Unrelated multi-clause function](design-anti-patterns.md#unrelated-multi-clause-function), but with implications of its own. It impairs the code readability in a different way.
+When we use multi-clause functions, it is possible to extract values in the clauses for further usage and for pattern matching/guard checking. This extraction itself does not represent an anti-pattern, but when you have *extractions made across several clauses and several arguments of the same function*, it becomes hard to know which extracted parts are used for pattern/guards and what is used only inside the function body. This anti-pattern is related to [Unrelated multi-clause function](design-anti-patterns.md#unrelated-multi-clause-function), but with implications of its own. It impairs the code readability in a different way.
 
 #### Example
 
-The multi-clause function `drive/1` is extracting fields of an `%User{}` struct for usage in the clause expression (`age`) and for usage in the function body (`name`). Ideally, a function should not mix pattern matching extractions for usage in its guard expressions and also in its body.
+The multi-clause function `drive/1` is extracting fields of an `%User{}` struct for usage in the clause expression (`age`) and for usage in the function body (`name`):
 
 ```elixir
 def drive(%User{name: name, age: age}) when age >= 18 do
@@ -185,7 +117,7 @@ def drive(%User{name: name, age: age}) when age < 18 do
 end
 ```
 
-While the example is small and looks like a clear code, try to imagine a situation where `drive/1` was more complex, having many more clauses, arguments, and extractions.
+While the example above is small and does not configure an anti-pattern, it is an example of mixed extraction and pattern matching. A situation where `drive/1` was more complex, having many more clauses, arguments, and extractions, would make it hard to know at a glance which variables are used for pattern/guards and which ones are not.
 
 #### Refactoring
 
@@ -288,7 +220,7 @@ defmodule MyRequestHandler do
 end
 ```
 
-All valid statuses all defined as atoms within the same module, and that's enough. If you want to be explicit, you could also have a function that lists them:
+All valid statuses are defined as atoms within the same module, and that's enough. If you want to be explicit, you could also have a function that lists them:
 
 ```elixir
 def valid_statuses do
@@ -363,11 +295,13 @@ defmodule PlugAuth do
 end
 ```
 
+#### Additional remarks
+
 There are few known exceptions to this anti-pattern:
 
   * [Protocol implementations](`Kernel.defimpl/2`) are, by design, defined under the protocol namespace
 
-  * [Custom Mix tasks](`Mix.Task`) are always defined under the `Mix.Tasks` namespace, such as `Mix.Tasks.PlugAuth`
+  * In some scenarios, the namespace owner may allow exceptions to this rule. For example, in Elixir itself, you defined [custom Mix tasks](`Mix.Task`) by placing them under the `Mix.Tasks` namespace, such as `Mix.Tasks.PlugAuth`
 
   * If you are the maintainer for both `plug` and `plug_auth`, then you may allow `plug_auth` to define modules with the `Plug` namespace, such as `Plug.Auth`. However, you are responsible for avoiding or managing any conflicts that may arise in the future
 
@@ -375,9 +309,13 @@ There are few known exceptions to this anti-pattern:
 
 #### Problem
 
-In Elixir, it is possible to access values from `Map`s, which are key-value data structures, either statically or dynamically. When a key is expected to exist in the map, it must be accessed using the `map.key` notation, which asserts the key exists. If the key does not exist, an exception is raised (and in some situations also compiler warnings), allowing developers to catch bugs early on.
+In Elixir, it is possible to access values from `Map`s, which are key-value data structures, either statically or dynamically.
 
-`map[:key]` must be used with optional keys. This way, if the informed key does not exist, `nil` is returned. When used with required keys, this return can be confusing and allow `nil` values to pass through the system, while `map.key` would raise upfront. In this way, this anti-pattern may cause bugs in the code.
+When a key is expected to exist in a map, it must be accessed using the `map.key` notation, making it clear to developers (and the compiler) that the key must exist. If the key does not exist, an exception is raised (and in some cases also compiler warnings). This is also known as the static notation, as the key is known at the time of writing the code.
+
+When a key is optional, the `map[:key]` notation must be used instead. This way, if the informed key does not exist, `nil` is returned. This is the dynamic notation, as it also supports dynamic key access, such as `map[some_var]`.
+
+When you use `map[:key]` to access a key that always exists in the map, you are making the code less clear for developers and for the compiler, as they now need to work with the assumption the key may not be there. This mismatch may also make it harder to track certain bugs. If the key is unexpected missing, you will have a `nil` value propagate through the system, instead of raising on map access.
 
 #### Example
 
@@ -387,8 +325,6 @@ The function `plot/1` tries to draw a graphic to represent the position of a poi
 defmodule Graphics do
   def plot(point) do
     # Some other code...
-
-    # Dynamic access to use point values
     {point[:x], point[:y], point[:z]}
   end
 end
@@ -397,90 +333,77 @@ end
 ```elixir
 iex> point_2d = %{x: 2, y: 3}
 %{x: 2, y: 3}
-iex> point_3d = %{x: 5, y: 6, z: nil}
-%{x: 5, y: 6, z: nil}
+iex> point_3d = %{x: 5, y: 6, z: 7}
+%{x: 5, y: 6, z: 7}
 iex> Graphics.plot(point_2d)
-{2, 3, nil}   # <= ambiguous return
+{2, 3, nil}
 iex> Graphics.plot(point_3d)
-{5, 6, nil}
+{5, 6, 7}
 ```
 
-As can be seen in the example above, even when the key `:z` does not exist in the map (`point_2d`), dynamic access returns the value `nil`. This return can be dangerous because of its ambiguity. It is not possible to conclude from it whether the map has the key `:z` or not. If the function relies on the return value to make decisions about how to plot a point, this can be problematic and even cause errors when testing the code.
+Given we want to plot both 2D and 3D points, the behaviour above is expected. But what happens if we forget to pass a point with either `:x` or `:y`?
+
+```elixir
+iex> bad_point = %{y: 3, z: 4}
+%{y: 3, z: 4}
+iex> Graphics.plot(bad_point)
+{nil, 3, 4}
+```
+
+The behaviour above is unexpected because our function should not work with points without a `:x` key. This leads to subtle bugs, as we may now pass `nil` to another function, instead of raising early on.
 
 #### Refactoring
 
-To remove this anti-pattern, whenever accessing an existing key of `Atom` type in the map, replace the dynamic `map[:key]` syntax by the static `map.key` notation. This way, when a non-existent key is accessed, Elixir raises an error immediately, allowing developers to find bugs faster. The next code illustrates the refactoring of `plot/1`, removing this anti-pattern:
+To remove this anti-pattern, we must use the dynamic `map[:key]` syntax and the static `map.key` notation according to our requirements. We expect `:x` and `:y` to always exist, but not `:z`. The next code illustrates the refactoring of `plot/1`, removing this anti-pattern:
 
 ```elixir
 defmodule Graphics do
   def plot(point) do
     # Some other code...
-
-    # Strict access to use point values
-    {point.x, point.y, point.z}
+    {point.x, point.y, point[:z]}
   end
 end
 ```
 
 ```elixir
-iex> point_2d = %{x: 2, y: 3}
-%{x: 2, y: 3}
-iex> point_3d = %{x: 5, y: 6, z: nil}
-%{x: 5, y: 6, z: nil}
 iex> Graphics.plot(point_2d)
-** (KeyError) key :z not found in: %{x: 2, y: 3} # <= explicitly warns that
-  graphic.ex:6: Graphics.plot/1                  # <= the :z key does not exist!
-iex> Graphics.plot(point_3d)
-{5, 6, nil}
+{2, 3, nil}
+iex> Graphics.plot(bad_point)
+** (KeyError) key :x not found in: %{y: 3, z: 4} # <= explicitly warns that
+  graphic.ex:4: Graphics.plot/1                  # <= the :z key does not exist!
 ```
 
 Overall, the usage of `map.key` and `map[:key]` encode important information about your data structure, allowing developers to be clear about their intent. See both `Map` and `Access` module documentation for more information and examples.
 
-Another alternative to refactor this anti-pattern is to use pattern matching:
+An alternative to refactor this anti-pattern is to use pattern matching, defining explicit clauses for 2d vs 3d points:
 
 ```elixir
 defmodule Graphics do
+  # 2d
+  def plot(%{x: x, y: y}) do
+    # Some other code...
+    {x, y}
+  end
+
+  # 3d
   def plot(%{x: x, y: y, z: z}) do
     # Some other code...
-
-    # Strict access to use point values
     {x, y, z}
   end
 end
 ```
 
-```elixir
-iex> point_2d = %{x: 2, y: 3}
-%{x: 2, y: 3}
-iex> point_3d = %{x: 5, y: 6, z: nil}
-%{x: 5, y: 6, z: nil}
-iex> Graphics.plot(point_2d)
-** (FunctionClauseError)  no function clause matching in Graphics.plot/1
-  graphic.ex:2: Graphics.plot/1                  # <= the :z key does not exist!
-iex> Graphics.plot(point_3d)
-{5, 6, nil}
-```
+Pattern-matching is specially useful when matching over multiple keys as well as on the values themselves at once.
 
-Another alternative is to use structs. By default, structs only support static access to its fields:
+Another option is to use structs. By default, structs only support static access to its fields. In such scenarios, you may consider defining structs for both 2D and 3D points:
 
 ```elixir
-defmodule Point.2D do
+defmodule Point2D do
   @enforce_keys [:x, :y]
   defstruct [x: nil, y: nil]
 end
 ```
 
-```elixir
-iex> point = %Point.2D{x: 2, y: 3}
-%Point.2D{x: 2, y: 3}
-iex> point.x   # <= strict access to use point values
-2
-iex> point.z   # <= trying to access a non-existent key
-** (KeyError) key :z not found in: %Point{x: 2, y: 3}
-iex> point[:x] # <= by default, struct does not support dynamic access
-** (UndefinedFunctionError) ... (Point does not implement the Access behaviour)
-```
-
 Generally speaking, structs are useful when sharing data structures across modules, at the cost of adding a compile time dependency between these modules. If module `A` uses a struct defined in module `B`, `A` must be recompiled if the fields in the struct `B` change.
 
 #### Additional remarks
@@ -562,7 +485,7 @@ case some_function(arg) do
 end
 ```
 
-In particular, avoid matching solely on `_`, as shown below, as it is less clear in intent and it may hide bugs if `some_function/1` adds new return values in the future:
+In particular, avoid matching solely on `_`, as shown below:
 
 ```elixir
 case some_function(arg) do
@@ -571,6 +494,8 @@ case some_function(arg) do
 end
 ```
 
+ Matching on `_` is less clear in intent and it may hide bugs if `some_function/1` adds new return values in the future.
+
 #### Additional remarks
 
 This anti-pattern was formerly known as [Speculative assumptions](https://github.com/lucasvegi/Elixir-Code-Smells#speculative-assumptions).

lib/elixir/pages/anti-patterns/design-anti-patterns.md

@@ -1,7 +1,6 @@
 # Design-related anti-patterns
 
-This document outlines anti-patterns related to your modules, functions, and the role they
-play within a codebase.
+This document outlines potential anti-patterns related to your modules, functions, and the role they play within a codebase.
 
 ## Alternative return types
 
@@ -126,7 +125,7 @@ Remember booleans are internally represented as atoms. Therefore there is no per
 
 #### Problem
 
-This anti-pattern refers to code that uses exceptions for control flow. Exception handling itself does not represent an anti-pattern, but developers must prefer to use `case` and pattern matching to change the flow of their code, instead of `try/rescue`. In turn, library authors should provide developers with APIs to handle errors without relying on exception handling. When developers have no freedom to decide if an error is exceptional or not, this is considered an anti-pattern.
+This anti-pattern refers to code that uses `Exception`s for control flow. Exception handling itself does not represent an anti-pattern, but developers must prefer to use `case` and pattern matching to change the flow of their code, instead of `try/rescue`. In turn, library authors should provide developers with APIs to handle errors without relying on exception handling. When developers have no freedom to decide if an error is exceptional or not, this is considered an anti-pattern.
 
 #### Example
 
@@ -186,63 +185,15 @@ end
 
 A common practice followed by the community is to make the non-raising version return `{:ok, result}` or `{:error, Exception.t}`. For example, an HTTP client may return `{:ok, %HTTP.Response{}}` on success cases and `{:error, %HTTP.Error{}}` for failures, where `HTTP.Error` is [implemented as an exception](`Kernel.defexception/1`). This makes it convenient for anyone to raise an exception by simply calling `Kernel.raise/1`.
 
-## Feature envy
+#### Additional remarks
 
-#### Problem
-
-This anti-pattern occurs when a function accesses more data or calls more functions from another module than from its own. The presence of this anti-pattern can make a module less cohesive and increase code coupling.
-
-#### Example
-
-In the following code, all the data used in the `calculate_total_item/1` function of the module `Order` comes from the `OrderItem` module. This increases coupling and decreases code cohesion unnecessarily.
-
-```elixir
-defmodule Order do
-  # Some functions...
-
-  def calculate_total_item(id) do
-    item = OrderItem.find_item(id)
-    total = (item.price + item.taxes) * item.amount
-
-    if discount = OrderItem.find_discount(item) do
-      total - total * discount
-    else
-      total
-    end
-  end
-end
-```
-
-#### Refactoring
-
-To remove this anti-pattern we can move `calculate_total_item/1` to `OrderItem`, decreasing coupling:
-
-```elixir
-defmodule OrderItem do
-  def find_item(id)
-  def find_discount(item)
-
-  def calculate_total_item(id) do   # <= function moved from Order!
-    item = find_item(id)
-    total = (item.price + item.taxes) * item.amount
-    discount = find_discount(item)
-
-    unless is_nil(discount) do
-      total - total * discount
-    else
-      total
-    end
-  end
-end
-```
-
-This refactoring is only possible when you own both modules. If the module you are invoking belongs to another application, then it is not possible to add new functions to it, and your only option is to define an additional module that augments the third-party module.
+This anti-pattern was formerly known as [Using exceptions for control-flow](https://github.com/lucasvegi/Elixir-Code-Smells#using-exceptions-for-control-flow).
 
 ## Primitive obsession
 
 #### Problem
 
-This anti-pattern happens when Elixir basic types (for example, *integer*, *float*, and *string*) are abusively used in function parameters and code variables, rather than creating specific composite data types (for example, *tuples*, *maps*, and *structs*) that can better represent a domain.
+This anti-pattern happens when Elixir basic types (for example, *integer*, *float*, and *string*) are excessively used to carry structured information, rather than creating specific composite data types (for example, *tuples*, *maps*, and *structs*) that can better represent a domain.
 
 #### Example
 
@@ -250,17 +201,23 @@ An example of this anti-pattern is the use of a single *string* to represent an
 
 ```elixir
 defmodule MyApp do
-  def process_address(address) when is_binary(address) do
-    # Do something with address...
+  def extract_postal_code(address) when is_binary(address) do
+    # Extract postal code with address...
+  end
+
+  def fill_in_country(address) when is_binary(address) do
+    # Fill in missing country...
   end
 end
 ```
 
+While you may receive the `address` as a string from a database, web request, or a third-party, if you find yourself frequently manipulating or extracting information from the string, it is a good indicator you should convert the address into structured data:
+
 Another example of this anti-pattern is using floating numbers to model money and currency, when [richer data structures should be preferred](https://hexdocs.pm/ex_money/).
 
 #### Refactoring
 
-Possible solutions to this anti-pattern is to use maps or structs to model our address. The example below creates an `Address` struct, better representing this domain through a composite type. Additionally, we can modify the `process_address/1` function to accept a parameter of type `Address` instead of a *string*. With this modification, we can extract each field of this composite type individually when needed.
+Possible solutions to this anti-pattern is to use maps or structs to model our address. The example below creates an `Address` struct, better representing this domain through a composite type. Additionally, we introduce a `parse/1` function, that converts the string into an `Address`, which will simplify the logic of remainng functions. With this modification, we can extract each field of this composite type individually when needed.
 
 ```elixir
 defmodule Address do
@@ -270,129 +227,62 @@ end
 
 ```elixir
 defmodule MyApp do
-  def process_address(%Address{} = address) do
-    # Do something with address...
+  def parse(address) when is_binary(address) do
+    # Returns %Address{}
+  end
+
+  def extract_postal_code(%Address{} = address) do
+    # Extract postal code with address...
+  end
+
+  def fill_in_country(%Address{} = address) do
+    # Fill in missing country...
   end
 end
 ```
 
-## Propagating invalid data
-
-#### Problem
-
-This anti-pattern refers to a function that does not validate its parameters and propagates them to other functions, which can produce internal unexpected behavior. When an error is raised inside a function due to an invalid parameter value, it can be confusing for developers and make it harder to locate and fix the error.
-
-#### Example
-
-An example of this anti-pattern is when a function receives an invalid parameter and then passes it to other functions, either in the same library or in a third-party library. This can cause an error to be raised deep inside the call stack, which may be confusing for the developer who is working with invalid data. As shown next, the function `foo/1` is a user-facing API which doesn't validate its parameters at the boundary. In this way, it is possible that invalid data will be passed through, causing an error that is obscure and hard to debug.
-
-```elixir
-defmodule MyLibrary do
-  def foo(invalid_data) do
-    # Some other code...
-
-    MyLibrary.Internal.sum(1, invalid_data)
-  end
-end
-```
-
-```elixir
-iex> MyLibrary.foo(2)
-3
-iex> MyLibrary.foo("José") # With invalid data
-** (ArithmeticError) bad argument in arithmetic expression: 1 + "José"
-  :erlang.+(1, "José")
-  my_library.ex:4: MyLibrary.Internal.sum/2
-```
-
-#### Refactoring
-
-To remove this anti-pattern, the client code must validate input parameters at the boundary with the user, via guard clauses, pattern matching, or conditionals. This prevents errors from occurring elsewhere in the call stack, making them easier to understand and debug. This refactoring also allows libraries to be implemented without worrying about creating internal protection mechanisms. The next code snippet illustrates the refactoring of `foo/1`, removing this anti-pattern:
-
-```elixir
-defmodule MyLibrary do
-  def foo(data) when is_integer(data) do
-    # Some other code
-
-    MyLibrary.Internal.sum(1, data)
-  end
-end
-```
-
-```elixir
-iex> MyLibrary.foo(2) # With valid data
-3
-iex> MyLibrary.foo("José") # With invalid data
-** (FunctionClauseError) no function clause matching in MyLibrary.foo/1.
-The following arguments were given to MyLibrary.foo/1:
-
-      # 1
-      "José"
-
-  my_library.ex:2: MyLibrary.foo/1
-```
-
 ## Unrelated multi-clause function
 
 #### Problem
 
-Using multi-clause functions in Elixir, to group functions of the same name, is not an anti-pattern in itself. However, due to the great flexibility provided by this programming feature, some developers may abuse the number of guard clauses and pattern matches to group *unrelated* functionality.
+Using multi-clause functions in Elixir, to group functions of the same name, is a powerful Elixir feature. However, some developers may abuse this feature to group *unrelated* functionality, which configures an anti-pattern.
 
 #### Example
 
-A frequent example of this usage of multi-clause functions is when developers mix unrelated business logic into the same function definition. Such functions often have generic names or too broad specifications, making it difficult for maintainers and users of said functions to maintain and understand them.
+A frequent example of this usage of multi-clause functions is when developers mix unrelated business logic into the same function definition, in a way the behaviour of each clause is completely distinct from the other ones. Such functions often have too broad specifications, making it difficult for other developers to understand and maintain them.
 
-Some developers may use documentation mechanisms such as `@doc` annotations to compensate for poor code readability, however the documentation itself may end-up full of conditionals to describe how the function behaves for each different argument combination.
+Some developers may use documentation mechanisms such as `@doc` annotations to compensate for poor code readability, however the documentation itself may end-up full of conditionals to describe how the function behaves for each different argument combination. This is a good indicator that the clauses are ultimately unrelated.
 
 ```elixir
 @doc """
 Updates a struct.
 
-If given a "sharp" product (metal or glass with empty count),
-it will...
-
-If given a blunt product, it will...
+If given a product, it will...
 
 If given an animal, it will...
 """
-def update(%Product{count: nil, material: material})
-    when material in ["metal", "glass"] do
+def update(%Product{count: count, material: material})  do
   # ...
 end
 
-def update(%Product{count: count, material: material})
-    when count > 0 and material not in ["metal", "glass"] do
-  # ...
-end
-
-def update(%Animal{count: 1, skin: skin})
-    when skin in ["fur", "hairy"] do
+def update(%Animal{count: count, skin: skin})  do
   # ...
 end
 ```
 
+If updating an animal is completely different from updating a product and requires a different set of rules, it may be worth splitting those over different functions or even different modules.
+
 #### Refactoring
 
-As shown below, a possible solution to this anti-pattern is to break the business rules that are mixed up in a single unrelated multi-clause function in several different simple functions. More precise names make the scope of the function clear. Each function can have a specific `@doc`, describing its behavior and parameters received. While this refactoring sounds simple, it can have a lot of impact on the function's current users, so be careful!
+As shown below, a possible solution to this anti-pattern is to break the business rules that are mixed up in a single unrelated multi-clause function in simple functions. Each function can have a specific name and `@doc`, describing its behavior and parameters received. While this refactoring sounds simple, it can impact the function's current users, so be careful!
 
 ```elixir
 @doc """
-Updates a "sharp" product.
+Updates a product.
 
 It will...
 """
-def update_sharp_product(%Product{count: nil, material: material})
-    when material in ["metal", "glass"] do
-  # ...
-end
-
-@doc """
-Updates a "blunt" product.
-
-It will...
-"""
-def update_blunt_product(%Product{count: count, material: material})
-    when count > 0 and material not in ["metal", "glass"] do
+def update_product(%Product{count: count, material: material}) do
   # ...
 end
 
@@ -401,12 +291,48 @@ Updates an animal.
 
 It will...
 """
-def update_animal(%Animal{count: 1, skin: skin})
-    when skin in ["fur", "hairy"] do
+def update_animal(%Animal{count: count, skin: skin}) do
   # ...
 end
 ```
 
+These functions may still be implemented with multiple clauses, as long as the clauses group related funtionality. For example, `update_product` could be in practice implemented as follows:
+
+```elixir
+def update_product(%Product{count: 0}) do
+  # ...
+end
+
+def update_product(%Product{material: material})
+    when material in ["metal", "glass"] do
+  # ...
+end
+
+def update_product(%Product{material: material})
+    when material not in ["metal", "glass"] do
+  # ...
+end
+```
+
+You can see this pattern in practice within Elixir itself. The `+/2` operator can add `Integer`s and `Float`s together, but not `String`s, which instead use the `<>/2` operator. In this sense, it is reasonable to handle integers and floats in the same operation, but strings are unrelated enough to deserve their own function.
+
+You will also find examples in Elixir of functions that work with any struct, which would seemingly be an occurrence of this anti-pattern, such as `struct/2`:
+
+```elixir
+iex> struct(URI.parse("/foo/bar"), path: "/bar/baz")
+%URI{
+  scheme: nil,
+  userinfo: nil,
+  host: nil,
+  port: nil,
+  path: "/bar/baz",
+  query: nil,
+  fragment: nil
+}
+```
+
+The difference here is that the `struct/2` function behaves precisely the same for any struct given, therefore there is no question of how the function handles different inputs. If the behaviour is clear and consistent for all inputs, then the anti-pattern does not take place.
+
 ## Using application configuration for libraries
 
 #### Problem
@@ -465,3 +391,35 @@ iex> DashSplitter.split("Lucas-Francisco-da-Matta-Vegi", [parts: 5])
 iex> DashSplitter.split("Lucas-Francisco-da-Matta-Vegi") #<= default config is used!
 ["Lucas", "Francisco-da-Matta-Vegi"]
 ```
+
+#### Additional Remarks
+
+For Mix tasks and related tools, it may be necessary to provide per-project configuration. For example, imagine you have a `:linter` project, which supports setting the output file and the verbosity level. You may choose to configure it through application environment:
+
+```elixir
+config :linter,
+  output_file: "/path/to/output.json",
+  verbosity: 3
+```
+
+However, `Mix` allows tasks to read per-project configuration via `Mix.Project.config/0`. In this case, you can configure the `:linter` directly in the `mix.exs` file:
+
+```elixir
+def project do
+  [
+    app: :my_app,
+    version: "1.0.0",
+    linter: [
+      output_file: "/path/to/output.json",
+      verbosity: 3
+    ],
+    ...
+  ]
+end
+```
+
+Additonally, if a Mix task is available, you can also accept these options as command line arguments (see `OptionParser`):
+
+```bash
+mix linter --output-file /path/to/output.json --verbosity 3
+```

lib/elixir/pages/anti-patterns/macro-anti-patterns.md

@@ -1,6 +1,6 @@
 # Meta-programming anti-patterns
 
-This document outlines anti-patterns related to meta-programming.
+This document outlines potential anti-patterns related to meta-programming.
 
 ## Large code generation by macros
 
@@ -63,7 +63,7 @@ end
 
 #### Problem
 
-**Macros** are powerful meta-programming mechanisms that can be used in Elixir to extend the language. While using macros is not an anti-pattern in itself, this meta-programming mechanism should only be used when absolutely necessary. Whenever a macro is used, but it would have been possible to solve the same problem using functions or other existing Elixir structures, the code becomes unnecessarily more complex and less readable. Because macros are more difficult to implement and reason about, their indiscriminate use can compromise the evolution of a system, reducing its maintainability.
+*Macros* are powerful meta-programming mechanisms that can be used in Elixir to extend the language. While using macros is not an anti-pattern in itself, this meta-programming mechanism should only be used when absolutely necessary. Whenever a macro is used, but it would have been possible to solve the same problem using functions or other existing Elixir structures, the code becomes unnecessarily more complex and less readable. Because macros are more difficult to implement and reason about, their indiscriminate use can compromise the evolution of a system, reducing its maintainability.
 
 #### Example
 

lib/elixir/pages/anti-patterns/process-anti-patterns.md

@@ -1,12 +1,12 @@
 # Process-related anti-patterns
 
-This document outlines anti-patterns related to processes and process-based abstractions.
+This document outlines potential anti-patterns related to processes and process-based abstractions.
 
 ## Code organization by process
 
 #### Problem
 
-This anti-pattern refers to code that is unnecessarily organized by processes. A process itself does not represent an anti-pattern, but it should only be used to model runtime properties (such as concurrency, access to shared resources, and event scheduling). When you use a process for code organization, it can create bottlenecks in the system.
+This anti-pattern refers to code that is unnecessarily organized by processes. A process itself does not represent an anti-pattern, but it should only be used to model runtime properties (such as concurrency, access to shared resources, error isolation, etc). When you use a process for code organization, it can create bottlenecks in the system.
 
 #### Example
 
@@ -188,6 +188,60 @@ iex> Foo.Bucket.get(bucket, "milk")
 
 This anti-pattern was formerly known as [Agent obsession](https://github.com/lucasvegi/Elixir-Code-Smells/tree/main#agent-obsession).
 
+## Sending unnecessary data
+
+#### Problem
+
+Sending a message to a process can be an expensive operation if the message is big enough. That's because that message will be fully copied to the receiving process, which may be CPU and memory intensive. This is due to Erlang's "share nothing" architecture, where each process has its own memory, which simplifies and speeds up garbage collection.
+
+This is more obvious when using `send/2`, `GenServer.call/3`, or the initial data in `GenServer.start_link/3`. Notably this also happens when using `spawn/1`, `Task.async/1`, `Task.async_stream/3`, and so on. It is more subtle here as the anonymous function passed to these functions captures the variables it references, and all captured variables will be copied over. By doing this, you can accidentally send way more data to a process than you actually need.
+
+#### Example
+
+Imagine you were to implement some simple reporting of IP addresses that made requests against your application. You want to do this asynchronously and not block processing, so you decide to use `spawn/1`. It may seem like a good idea to hand over the whole connection because we might need more data later. However passing the connection results in copying a lot of unnecessary data like the request body, params, etc.
+
+```elixir
+# log_request_ip send the ip to some external service
+spawn(fn -> log_request_ip(conn) end)
+```
+
+This problem also occurs when accessing only the relevant parts:
+
+```elixir
+spawn(fn -> log_request_ip(conn.remote_ip) end)
+```
+
+This will still copy over all of `conn`, because the `conn` variable is being captured inside the spawned function. The function then extracts the `remote_ip` field, but only after the whole `conn` has been copied over.
+
+`send/2` and the `GenServer` APIs also rely on message passing. In the example below, the `conn` is once again copied to the underlying `GenServer`:
+
+```elixir
+GenServer.cast(pid, {:report_ip_address, conn})
+```
+
+#### Refactoring
+
+This anti-pattern has many potential remedies:
+
+  * Limit the data you send to the absolute necessary minimum instead of sending an entire struct. For example, don't send an entire `conn` struct if all you need is a couple of fields.
+
+  * If the only process that needs data is the one you are sending to, consider making the process fetch that data instead of passing it.
+
+  * Some abstractions, such as [`:persistent_term`](https://www.erlang.org/doc/man/persistent_term.html), allows you to share data between processes, as long as such data changes infrequently.
+
+In our case, limiting the input data is a reasonable strategy. If all we need *right now* is the IP address, then let's only work with that and make sure we're only passing the IP address into the closure, like so:
+
+```elixir
+ip_address = conn.remote_ip
+spawn(fn -> log_request_ip(ip_address) end)
+```
+
+Or in the `GenServer` case:
+
+```elixir
+GenServer.cast(pid, {:report_ip_address, conn.remote_ip})
+```
+
 ## Unsupervised processes
 
 #### Problem
@@ -207,7 +261,9 @@ defmodule Counter do
   use GenServer
 
   @doc "Starts a counter process."
-  def start(initial_value, name \\ __MODULE__) when is_integer(initial_value) do
+  def start_link(opts \\ []) do
+    initial_valye = Keyword.get(opts, :initial_value, 0)
+    name = Keywoird.get(opts, :name, __MODULE__)
     GenServer.start(__MODULE__, initial_value, name: name)
   end
 
@@ -217,7 +273,7 @@ defmodule Counter do
   end
 
   @doc "Bumps the value of the given counter."
-  def bump(value, pid_name \\ __MODULE__) do
+  def bump(pid_name \\ __MODULE__, value) do
     GenServer.call(pid_name, {:bump, value})
   end
 
@@ -238,17 +294,17 @@ end
 ```
 
 ```elixir
-iex> Counter.start(0)
+iex> Counter.start_link()
 {:ok, #PID<0.115.0>}
 iex> Counter.get()
 0
-iex> Counter.start(15, :other_counter)
+iex> Counter.start_link(initial_value: 15, name: :other_counter)
 {:ok, #PID<0.120.0>}
 iex> Counter.get(:other_counter)
 15
-iex> Counter.bump(-3, :other_counter)
+iex> Counter.bump(:other_counter, -3)
 12
-iex> Counter.bump(7)
+iex> Counter.bump(Counter, 7)
 7
 ```
 
@@ -263,8 +319,13 @@ defmodule SupervisedProcess.Application do
   @impl true
   def start(_type, _args) do
     children = [
-      %{id: Counter, start: {Counter, :start, [0]}},
-      %{id: :other_counter, start: {Counter, :start, [0, :other_counter]}}
+      # With the default values for counter and name
+      Counter,
+      # With custom values for counter, name, and a custom ID
+      Supervisor.child_spec(
+        {Counter, name: :other_counter, initial_value: 15},
+        id: :other_counter
+      )
     ]
 
     Supervisor.start_link(children, strategy: :one_for_one, name: App.Supervisor)
@@ -278,8 +339,8 @@ iex> Supervisor.count_children(App.Supervisor)
 iex> Counter.get(Counter)
 0
 iex> Counter.get(:other_counter)
-0
-iex> Counter.bump(7, Counter)
+15
+iex> Counter.bump(Counter, 7)
 7
 iex> Supervisor.terminate_child(App.Supervisor, Counter)
 iex> Supervisor.count_children(App.Supervisor) # Only one active child

lib/elixir/pages/anti-patterns/what-anti-patterns.md

@@ -1,12 +1,14 @@
 # What are anti-patterns?
 
-Anti-patterns describe common mistakes or indicators of potential problems in code.
+Anti-patterns describe common mistakes or indicators of problems in code.
 They are also known as "code smells".
 
-The goal of these guides is to document known anti-patterns found in Elixir software
-and teach developers how to identify and correct them. If an existing piece of code
-matches an anti-pattern, it does not mean your code must be rewritten. However, you
-should take its potential pitfalls and alternatives into consideration.
+The goal of these guides is to document potential anti-patterns found in Elixir software
+and teach developers how to identify them and their pitfalls. If an existing piece
+of code matches an anti-pattern, it does not mean your code must be rewritten.
+Sometimes, even if a snippet matches a potential anti-pattern and its limitations,
+it may be the best approach to the problem at hand. No codebase is free of anti-patterns
+and one should not aim to remove all of them.
 
 The anti-patterns in these guides are broken into 4 main categories:
 
@@ -34,6 +36,8 @@ Each anti-pattern is documented using the following structure:
   * **Refactoring:** Ways to change your code to improve its qualities. Examples of refactored
     code are presented to illustrate these changes.
 
+An additional section with "Additional Remarks" may be provided. Those may include known scenarios where the anti-pattern does not apply.
+
 The initial catalog of anti-patterns was proposed by Lucas Vegi and Marco Tulio Valente, from [ASERG/DCC/UFMG](http://aserg.labsoft.dcc.ufmg.br/). For more info, see [Understanding Code Smells in Elixir Functional Language](https://github.com/lucasvegi/Elixir-Code-Smells/blob/main/etc/2023-emse-code-smells-elixir.pdf) and [the associated code repository](https://github.com/lucasvegi/Elixir-Code-Smells).
 
-Additionally, the Security Working Group of the [Erlang Ecosystem Foundation](https://erlef.github.io/security-wg/) publishes [documents with security resources and best-practices of both Erland and Elixir, including detailed guides for web applications](https://erlef.github.io/security-wg/).
+Additionally, the Security Working Group of the [Erlang Ecosystem Foundation](https://erlef.github.io/security-wg/) publishes [documents with security resources and best-practices of both Erlang and Elixir, including detailed guides for web applications](https://erlef.github.io/security-wg/).

lib/elixir/pages/cheatsheets/enum-cheat.cheatmd

@@ -820,7 +820,10 @@ Negative indexes count from the back:
 
 ```elixir
 iex> Enum.drop(cart, -1)
-[%{fruit: "orange", count: 6}]
+[
+  %{fruit: "apple", count: 3},
+  %{fruit: "banana", count: 1}
+]
 ```
 
 ### [`drop_every(enum, nth)`](`Enum.drop_every/2`)
@@ -851,10 +854,7 @@ Negative indexes count from the back:
 
 ```elixir
 iex> Enum.take(cart, -1)
-[
-  %{fruit: "apple", count: 3},
-  %{fruit: "banana", count: 1}
-]
+[%{fruit: "orange", count: 6}]
 ```
 
 ### [`take_every(enum, nth)`](`Enum.take_every/2`)

lib/elixir/pages/getting-started/alias-require-and-import.md

@@ -112,7 +112,7 @@ end
 
 In the example above, the imported `List.duplicate/2` is only visible within that specific function. `duplicate/2` won't be available in any other function in that module (or any other module for that matter).
 
-Note that `import`s are generally discouraged in the language. When working on your own code, prefer `alias` to `import`.
+While `import`s can be a useful for frameworks and libraries to build abstractions, developers should generally prefer `alias` to `import` on their own codebases, as aliases make the origin of the function being invoked clearer.
 
 ## use
 

lib/elixir/pages/getting-started/anonymous-functions.md

@@ -17,7 +17,7 @@ true
 
 In the example above, we defined an anonymous function that receives two arguments, `a` and `b`, and returns the result of `a + b`. The arguments are always on the left-hand side of `->` and the code to be executed on the right-hand side. The anonymous function is stored in the variable `add`.
 
-We can invoke anonymous functions by passing arguments to it. Note that a dot (`.`) between the variable and parentheses is required to invoke an anonymous function. The dot ensures there is no ambiguity between calling the anonymous function matched to a variable `add` and a named function `add/2`. We will write our own named functions when dealing with [Modules and Functions](modules-and-functions.md). For now, just remember that Elixir makes a clear distinction between anonymous functions and named functions.
+We can invoke anonymous functions by passing arguments to it. Note that a dot (`.`) between the variable and parentheses is required to invoke an anonymous function. The dot makes it clear when you are calling an anonymous function, stored in the variable `add`, opposed to a function named `add/2`. For example, if you have an anonymous function stored in the variable `is_atom`, there is no ambiguity between `is_atom.(:foo)` and `is_atom(:foo)`. If both used the same `is_atom(:foo)` syntax, the only way to know the actual behaviour of `is_atom(:foo)` would be by scanning all code thus far for a possible definition of the `is_atom` variable. This scanning hurts maintainability as it requires developers to track additional context in their head when reading and writing code.
 
 Anonymous functions in Elixir are also identified by the number of arguments they receive. We can check if a function is of any given arity by using `is_function/2`:
 

lib/elixir/pages/getting-started/basic-types.md

@@ -83,7 +83,7 @@ You can also use `is_float` or `is_number` to check, respectively, if an argumen
 
 ## Identifying functions and documentation
 
-Before we move on to the next data type, let's talk about how Elixir identity functions.
+Before we move on to the next data type, let's talk about how Elixir identifies functions.
 
 Functions in Elixir are identified by both their name and their arity. The arity of a function describes the number of arguments that the function takes. From this point on we will use both the function name and its arity to describe functions throughout the documentation. `trunc/1` identifies the function which is named `trunc` and takes `1` argument, whereas `trunc/2` identifies a different (nonexistent) function with the same name but with an arity of `2`.
 

lib/elixir/pages/getting-started/case-cond-and-if.md

@@ -65,7 +65,7 @@ iex> case :ok do
 ** (CaseClauseError) no case clause matching: :ok
 ```
 
-The documentation for the `Kernel` module lists all available guards in its sidebar. You can also consult the complete [Patterns and Guards](../references/patterns-and-guards.html#guards) reference for in-depth documentation.
+The documentation for the `Kernel` module lists all available guards in its sidebar. You can also consult the complete [Patterns and Guards](../references/patterns-and-guards.md#guards) reference for in-depth documentation.
 
 ## cond
 

lib/elixir/pages/getting-started/enumerable-and-streams.md

@@ -119,6 +119,6 @@ iex> Enum.take(stream, 10)
 
 The example above will fetch the first 10 lines of the file you have selected. This means streams can be very useful for handling large files or even slow resources like network resources.
 
-The `Enum` and `Stream` modules provide a wide-range functions but know all of them at heart. Familiarize yourself with `Enum.map/2`, `Enum.reduce/3` and other functions with either `map` or `reduce` in their names, and you will naturally build an intuition around the most important use cases. You may also focus on the `Enum` module first and only move to `Stream` for the particular scenarios where laziness is required, to either deal with slow resources or large, possibly infinite, collections.
+The `Enum` and `Stream` modules provide a wide range of functions, but you don't have to know all of them by heart. Familiarize yourself with `Enum.map/2`, `Enum.reduce/3` and other functions with either `map` or `reduce` in their names, and you will naturally build an intuition around the most important use cases. You may also focus on the `Enum` module first and only move to `Stream` for the particular scenarios where laziness is required, to either deal with slow resources or large, possibly infinite, collections.
 
 Next, we'll look at a feature central to Elixir, Processes, which allows us to write concurrent, parallel and distributed programs in an easy and understandable way.

lib/elixir/pages/getting-started/io-and-the-file-system.md

@@ -129,7 +129,7 @@ iex> IO.write(pid, "hello")
 
 After `IO.write/2`, we can see the request sent by the `IO` module printed out (a four-elements tuple). Soon after that, we see that it fails since the `IO` module expected some kind of result, which we did not supply.
 
-By modeling IO devices with processes, the Erlang VM allows is to even read and write to files across nodes. Neat!
+By modeling IO devices with processes, the Erlang VM allows us to even read and write to files across nodes. Neat!
 
 ## `iodata` and `chardata`
 

lib/elixir/pages/getting-started/keywords-and-maps.md

@@ -127,7 +127,7 @@ iex> if true, do: "This will be seen", else: "This won't"
 
 Pay close attention to both syntaxes. In the keyword list format, we separate each key-value pair with commas, and each key is followed by `:`. In the `do`-blocks, we get rid of the colons, the commas, and separate each keyword by a newline. They are useful exactly because they remove the verbosity when writing blocks of code. Most of the time, you will use the block syntax, but it is good to know they are equivalent.
 
-This plays an important role in the language as it allows Elixir syntax to stay small but still expressive. We only need few data structures to represent the language, a topic we will come back to when talking about [optional syntax](optional-syntax.md) and go in-depth when discussing [meta-programming](../quote-and-unquote.md).
+This plays an important role in the language as it allows Elixir syntax to stay small but still expressive. We only need few data structures to represent the language, a topic we will come back to when talking about [optional syntax](optional-syntax.md) and go in-depth when discussing [meta-programming](../meta-programming/quote-and-unquote.md).
 
 With this out of the way, let's talk about maps.
 

lib/elixir/pages/getting-started/sigils.md

@@ -2,7 +2,7 @@
 
 Elixir provides double-quoted strings as well as a concept called charlists, which are defined using the `~c"hello world"` sigil syntax. In this chapter, we will learn more about sigils and how to define our own.
 
-One of Elixir's goals is extensibility: developers should be able to extend the language to fit any particular domain. Sigils provide the foundation for extending the language with custom textual representations. Sigils start with the tilde (`~`) character which is followed by a one lower-case letter or several upper-case ones and then a delimiter. Optionally, modifiers can be added after the final delimiter.
+One of Elixir's goals is extensibility: developers should be able to extend the language to fit any particular domain. Sigils provide the foundation for extending the language with custom textual representations. Sigils start with the tilde (`~`) character which is followed by either a single lower-case letter or one or more upper-case letters, and then a delimiter. Optional modifiers are added after the final delimiter.
 
 ## Regular expressions
 

lib/elixir/pages/mix-and-otp/agents.md

@@ -124,7 +124,7 @@ The first step in our implementation is to call `use Agent`. Most of the functio
 
 Then we define a `start_link/1` function, which will effectively start the agent. It is a convention to define a `start_link/1` function that always accepts a list of options. We don't plan on using any options right now, but we might later on. We then proceed to call `Agent.start_link/1`, which receives an anonymous function that returns the Agent's initial state.
 
-We are keeping a map inside the agent to store our keys and values. Getting and putting values on the map is done with the Agent API and the capture operator `&`, introduced in [the Getting Started guide](../getting-started/modules-and-functions.md#function-capturing). The agent passes its state to the anonymous function via the `&1` argument when `Agent.get/2` and `Agent.update/2` are called.
+We are keeping a map inside the agent to store our keys and values. Getting and putting values on the map is done with the Agent API and the capture operator `&`, introduced in [the Getting Started guide](../getting-started/anonymous-functions.md#the-capture-operator). The agent passes its state to the anonymous function via the `&1` argument when `Agent.get/2` and `Agent.update/2` are called.
 
 Now that the `KV.Bucket` module has been defined, our test should pass! You can try it yourself by running: `mix test`.
 

lib/elixir/pages/mix-and-otp/distributed-tasks.md

@@ -90,7 +90,7 @@ There are three better alternatives to `Node.spawn_link/2` that we could use in
 
 2. We could have a server running on the other node and send requests to that node via the `GenServer` API. For example, you can call a server on a remote node by using `GenServer.call({name, node}, arg)` or passing the remote process PID as the first argument
 
-3. We could use [tasks](`Task`), which we have learned about in [a previous chapter](../getting-started/mix-otp/task-and-gen-tcp.md), as they can be spawned on both local and remote nodes
+3. We could use [tasks](`Task`), which we have learned about in [a previous chapter](task-and-gen-tcp.md), as they can be spawned on both local and remote nodes
 
 The options above have different properties. The GenServer would serialize your requests on a single server, while tasks are effectively running asynchronously on the remote node, with the only serialization point being the spawning done by the supervisor.
 

lib/elixir/pages/mix-and-otp/genservers.md

@@ -1,6 +1,6 @@
 # Client-server communication with GenServer
 
-In the [previous chapter](../agents.md), we used agents to represent our buckets. In the [introduction to mix](../introduction-to-mix.md), we specified we would like to name each bucket so we can do the following:
+In the [previous chapter](agents.md), we used agents to represent our buckets. In the [introduction to mix](introduction-to-mix.md), we specified we would like to name each bucket so we can do the following:
 
 ```elixir
 CREATE shopping
@@ -320,7 +320,7 @@ So far we have used three callbacks: `handle_call/3`, `handle_cast/2` and `handl
 
 Since any message, including the ones sent via `send/2`, go to `handle_info/2`, there is a chance that unexpected messages will arrive to the server. Therefore, if we don't define the catch-all clause, those messages could cause our registry to crash, because no clause would match. We don't need to worry about such cases for `handle_call/3` and `handle_cast/2` though. Calls and casts are only done via the `GenServer` API, so an unknown message is quite likely a developer mistake.
 
-To help developers remember the differences between call, cast and info, the supported return values and more, we have a tiny [GenServer cheat sheet](/downloads/cheatsheets/gen-server.pdf).
+To help developers remember the differences between call, cast and info, the supported return values and more, we have a tiny [GenServer cheat sheet](https://elixir-lang.org/downloads/cheatsheets/gen-server.pdf).
 
 ## Monitors or links?
 

lib/elixir/pages/references/compatibility-and-deprecations.md

@@ -8,12 +8,11 @@ Elixir applies bug fixes only to the latest minor branch. Security patches are a
 
 Elixir version | Support
 :------------- | :-----------------------------
-1.16           | Development
-1.15           | Bug fixes and security patches
+1.16           | Bug fixes and security patches
+1.15           | Security patches only
 1.14           | Security patches only
 1.13           | Security patches only
 1.12           | Security patches only
-1.11           | Security patches only
 
 New releases are announced in the read-only [announcements mailing list](https://groups.google.com/group/elixir-lang-ann). All security releases [will be tagged with `[security]`](https://groups.google.com/forum/#!searchin/elixir-lang-ann/%5Bsecurity%5D%7Csort:date).
 
@@ -43,8 +42,9 @@ Erlang/OTP versioning is independent from the versioning of Elixir. Erlang relea
 
 Elixir version | Supported Erlang/OTP versions
 :------------- | :-------------------------------
+1.16           | 24 - 26
 1.15           | 24 - 26
-1.14           | 23 - 25
+1.14           | 23 - 25 (and Erlang/OTP 26 from v1.14.5)
 1.13           | 22 - 24 (and Erlang/OTP 25 from v1.13.4)
 1.12           | 22 - 24
 1.11           | 21 - 23 (and Erlang/OTP 24 from v1.11.4)

lib/elixir/pages/references/syntax-reference.md

@@ -395,7 +395,7 @@ end
 
 All of the constructs above are part of Elixir's syntax and have their own representation as part of the Elixir AST. This section will discuss the remaining constructs that are alternative representations of the constructs above. In other words, the constructs below can be represented in more than one way in your Elixir code and retain AST equivalence. We call this "Optional Syntax".
 
-For a lightweight introduction to Elixir's Optional Syntax, [see this document](https://elixir-lang.org/getting-started/optional-syntax.html). Below we continue with a more complete reference.
+For a lightweight introduction to Elixir's Optional Syntax, [see this document](../getting-started/optional-syntax.md). Below we continue with a more complete reference.
 
 ### Integers in other bases and Unicode code points
 

lib/elixir/pages/references/typespecs.md

@@ -17,7 +17,7 @@ Type specifications (most often referred to as *typespecs*) are defined in diffe
   * `@callback`
   * `@macrocallback`
 
-In addition, you can use `@typedoc` to describe a custom `@type` definition.
+In addition, you can use `@typedoc` to document a custom `@type` definition.
 
 See the "User-defined types" and "Defining a specification" sub-sections below for more information on defining types and typespecs.
 
@@ -250,19 +250,114 @@ Behaviours in Elixir (and Erlang) are a way to separate and abstract the generic
 
 A behaviour module defines a set of functions and macros (referred to as *callbacks*) that callback modules implementing that behaviour must export. This "interface" identifies the specific part of the component. For example, the `GenServer` behaviour and functions abstract away all the message-passing (sending and receiving) and error reporting that a "server" process will likely want to implement from the specific parts such as the actions that this server process has to perform.
 
-To define a behaviour module, it's enough to define one or more callbacks in that module. To define callbacks, the `@callback` and `@macrocallback` module attributes can be used (for function callbacks and macro callbacks respectively).
+Say we want to implement a bunch of parsers, each parsing structured data: for example, a JSON parser and a MessagePack parser. Each of these two parsers will *behave* the same way: both will provide a `parse/1` function and an `extensions/0` function. The `parse/1` function will return an Elixir representation of the structured data, while the `extensions/0` function will return a list of file extensions that can be used for each type of data (e.g., `.json` for JSON files).
 
-    defmodule MyBehaviour do
-      @callback my_fun(arg :: any) :: any
-      @macrocallback my_macro(arg :: any) :: Macro.t
-    end
+We can create a `Parser` behaviour:
+
+```elixir
+defmodule Parser do
+  @doc """
+  Parses a string.
+  """
+  @callback parse(String.t) :: {:ok, term} | {:error, atom}
+
+  @doc """
+  Lists all supported file extensions.
+  """
+  @callback extensions() :: [String.t]
+end
+```
 
 As seen in the example above, defining a callback is a matter of defining a specification for that callback, made of:
 
-  * the callback name (`my_fun` or `my_macro` in the example)
-  * the arguments that the callback must accept (`arg :: any` in the example)
+  * the callback name (`parse` or `extensions` in the example)
+  * the arguments that the callback must accept (`String.t`)
   * the *expected* type of the callback return value
 
+Modules adopting the `Parser` behaviour will have to implement all the functions defined with the `@callback` attribute. As you can see, `@callback` expects a function name but also a function specification like the ones used with the `@spec` attribute we saw above.
+
+### Implementing behaviours
+
+Implementing a behaviour is straightforward:
+
+```elixir
+defmodule JSONParser do
+  @behaviour Parser
+
+  @impl Parser
+  def parse(str), do: {:ok, "some json " <> str} # ... parse JSON
+
+  @impl Parser
+  def extensions, do: [".json"]
+end
+```
+
+```elixir
+defmodule CSVParser do
+  @behaviour Parser
+
+  @impl Parser
+  def parse(str), do: {:ok, "some csv " <> str} # ... parse CSV
+
+  @impl Parser
+  def extensions, do: [".csv"]
+end
+```
+
+If a module adopting a given behaviour doesn't implement one of the callbacks required by that behaviour, a compile-time warning will be generated.
+
+Furthermore, with `@impl` you can also make sure that you are implementing the **correct** callbacks from the given behaviour in an explicit manner. For example, the following parser implements both `parse` and `extensions`. However, thanks to a typo, `BADParser` is implementing `parse/0` instead of `parse/1`.
+
+```elixir
+defmodule BADParser do
+  @behaviour Parser
+
+  @impl Parser
+  def parse, do: {:ok, "something bad"}
+
+  @impl Parser
+  def extensions, do: ["bad"]
+end
+```
+
+This code generates a warning letting you know that you are mistakenly implementing `parse/0` instead of `parse/1`.
+You can read more about `@impl` in the [module documentation](Module.html#module-impl).
+
+### Using behaviours
+
+Behaviours are useful because you can pass modules around as arguments and you can then *call back* to any of the functions specified in the behaviour. For example, we can have a function that receives a filename, several parsers, and parses the file based on its extension:
+
+```elixir
+@spec parse_path(Path.t(), [module()]) :: {:ok, term} | {:error, atom}
+def parse_path(filename, parsers) do
+  with {:ok, ext} <- parse_extension(filename),
+       {:ok, parser} <- find_parser(ext, parsers),
+       {:ok, contents} <- File.read(filename) do
+    parser.parse(contents)
+  end
+end
+
+defp parse_extension(filename) do
+  if ext = Path.extname(filename) do
+    {:ok, ext}
+  else
+    {:error, :no_extension}
+  end
+end
+
+defp find_parser(ext, parsers) do
+  if parser = Enum.find(parsers, fn parser -> ext in parser.extensions() end) do
+    {:ok, parser}
+  else
+    {:error, :no_matching_parser}
+  end
+end
+```
+
+You could also invoke any parser directly: `CSVParser.parse(...)`.
+
+Note you don't need to define a behaviour in order to dynamically dispatch on a module, but those features often go hand in hand.
+
 ### Optional callbacks
 
 Optional callbacks are callbacks that callback modules may implement if they want to, but are not required to. Usually, behaviour modules know if they should call those callbacks based on configuration, or they check if the callbacks are defined with `function_exported?/3` or `macro_exported?/3`.
@@ -278,34 +373,6 @@ Optional callbacks can be defined through the `@optional_callbacks` module attri
 
 One example of optional callback in Elixir's standard library is `c:GenServer.format_status/2`.
 
-### Implementing behaviours
-
-To specify that a module implements a given behaviour, the `@behaviour` attribute must be used:
-
-    defmodule MyBehaviour do
-      @callback my_fun(arg :: any) :: any
-    end
-
-    defmodule MyCallbackModule do
-      @behaviour MyBehaviour
-      def my_fun(arg), do: arg
-    end
-
-If a callback module that implements a given behaviour doesn't export all the functions and macros defined by that behaviour, the user will be notified through warnings during the compilation process (no errors will happen).
-
-You can also use the `@impl` attribute before a function to denote that particular function is implementation a behaviour:
-
-    defmodule MyCallbackModule do
-      @behaviour MyBehaviour
-
-      @impl true
-      def my_fun(arg), do: arg
-    end
-
-You can also use `@impl MyBehaviour` to make clearer from which behaviour the callbacks comes from, providing even more context for future readers of your code.
-
-Elixir's standard library contains a few frequently used behaviours such as `GenServer`, `Supervisor`, and `Application`.
-
 ### Inspecting behaviours
 
 The `@callback` and `@optional_callbacks` attributes are used to create a `behaviour_info/1` function available on the defining module. This function can be used to retrieve the callbacks and optional callbacks defined by that module.
@@ -319,6 +386,10 @@ For example, for the `MyBehaviour` module defined in "Optional callbacks" above:
 
 When using `iex`, the `IEx.Helpers.b/1` helper is also available.
 
+## Pitfalls
+
+There are some known pitfalls when using typespecs, they are documented next.
+
 ## The `string()` type
 
 Elixir discourages the use of the `string()` type. The `string()` type refers to Erlang strings, which are known as "charlists" in Elixir. They do not refer to Elixir strings, which are UTF-8 encoded binaries. To avoid confusion, if you attempt to use the type `string()`, Elixir will emit a warning. You should use `charlist()`, `nonempty_charlist()`, `binary()` or `String.t()` accordingly, or any of the several literal representations for these types.

lib/elixir/scripts/windows_installer/build.sh

@@ -4,43 +4,27 @@
 # With Elixir archive:
 #
 #     ELIXIR_ZIP=Precompiled.zip OTP_VERSION=25.3.2.2 ./build.sh
-#
-# With Elixir version:
-#
-#     ELIXIR_VERSION=1.14.5 OTP_VERSION=25.3.2.2 ./build.sh
 set -euo pipefail
 
-OTP_VERSION="${OTP_VERSION:-26.0}"
-otp_release=`echo "${OTP_VERSION}" | cut -d. -f1`
-
 mkdir -p tmp
+rm -rf tmp/elixir
+unzip -d "tmp/elixir" "${ELIXIR_ZIP}"
 
-ELIXIR_VERSION="${ELIXIR_VERSION:-}"
-if [ -n "${ELIXIR_VERSION}" ]; then
-  ELIXIR_ZIP="tmp/elixir-${ELIXIR_VERSION}-otp-${otp_release}.zip"
-  if [ ! -f "${ELIXIR_ZIP}" ]; then
-    url="https://github.com/elixir-lang/elixir/releases/download/v${ELIXIR_VERSION}/elixir-otp-${otp_release}.zip"
-    echo "downloading ${url}"
-    curl --fail -L -o "${ELIXIR_ZIP}" "${url}"
-  fi
-  basename=elixir-${ELIXIR_VERSION}-otp-${otp_release}
-else
-  basename=elixir-otp-${otp_release}
-fi
-
-if [ ! -d "tmp/${basename}" ]; then
-  unzip -d "tmp/${basename}" "${ELIXIR_ZIP}"
-fi
+elixir_version=`cat tmp/elixir/VERSION`
+otp_release=`erl -noshell -eval 'io:put_chars(erlang:system_info(otp_release)), halt().'`
+otp_version=`erl -noshell -eval '{ok, Vsn} = file:read_file(code:root_dir() ++ "/releases/" ++ erlang:system_info(otp_release) ++ "/OTP_VERSION"), io:put_chars(Vsn), halt().'`
+elixir_exe=elixir-otp-${otp_release}.exe
 
 # brew install makensis
 # apt install -y nsis
 # choco install -y nsis
 export PATH="/c/Program Files (x86)/NSIS:${PATH}"
 makensis \
-  -X"OutFile tmp\\${basename}.exe" \
-  -DOTP_VERSION=${OTP_VERSION} \
+  -X"OutFile tmp\\${elixir_exe}" \
   -DOTP_RELEASE="${otp_release}" \
-  -DELIXIR_DIR=tmp\\${basename} \
+  -DOTP_VERSION=${otp_version} \
+  -DELIXIR_DIR=tmp\\elixir \
+  -DELIXIR_VERSION=${elixir_version} \
   installer.nsi
 
-echo "Installer path: tmp/${basename}.exe"
+echo "Installer path: tmp/${elixir_exe}"

lib/elixir/scripts/windows_installer/installer.nsi

@@ -6,7 +6,8 @@ Name "Elixir"
 ManifestDPIAware true
 Unicode True
 InstallDir "$PROGRAMFILES64\Elixir"
-!define MUI_ICON "assets\drop.ico"
+!define MUI_ICON "assets\Elixir.ico"
+!define MUI_UNICON "assets\Elixir.ico"
 
 ; Install Page: Install Erlang/OTP
 
@@ -191,7 +192,17 @@ FunctionEnd
 Section "Install Elixir" SectionElixir
   SetOutPath "$INSTDIR"
   File /r "${ELIXIR_DIR}\"
+  File "assets\Elixir.ico"
   File "update_system_path.erl"
+  WriteRegStr   HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "DisplayName" "Elixir"
+  WriteRegStr   HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "DisplayVersion" "${ELIXIR_VERSION}"
+  WriteRegStr   HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "DisplayIcon" "$INSTDIR\Elixir.ico"
+  WriteRegStr   HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "Publisher" "The Elixir Team"
+  WriteRegStr   HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "UninstallString" '"$INSTDIR\Uninstall.exe"'
+  WriteRegDWORD HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "NoModify" 1
+  WriteRegDWORD HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir" "NoRepair" 1
+
+  WriteRegStr   HKLM "Software\Elixir\Elixir" "InstallRoot" "$INSTDIR"
 
   WriteUninstaller "Uninstall.exe"
 SectionEnd
@@ -270,6 +281,9 @@ UninstPage custom un.FinishPageShow un.FinishPageLeave
 
 Section "Uninstall"
   RMDir /r "$INSTDIR"
+  DeleteRegKey HKLM "Software\Microsoft\Windows\CurrentVersion\Uninstall\Elixir"
+  DeleteRegKey HKLM "Software\Elixir\Elixir"
+  DeleteRegKey /ifempty HKLM "Software\Elixir"
 SectionEnd
 
 !insertmacro MUI_LANGUAGE "English"

lib/elixir/src/elixir.erl

@@ -486,7 +486,7 @@ string_to_tokens(String, StartLine, StartColumn, File, Opts) when is_integer(Sta
     {ok, _Line, _Column, [], Tokens} ->
       {ok, Tokens};
     {ok, _Line, _Column, Warnings, Tokens} ->
-      (lists:keyfind(warnings, 1, Opts) /= {warnings, false}) andalso
+      (lists:keyfind(emit_warnings, 1, Opts) /= {emit_warnings, false}) andalso
         [elixir_errors:erl_warn(L, File, M) || {L, M} <- lists:reverse(Warnings)],
       {ok, Tokens};
     {error, {Location, {ErrorPrefix, ErrorSuffix}, Token}, _Rest, _Warnings, _SoFar} ->
@@ -544,8 +544,8 @@ to_binary(Atom) when is_atom(Atom) -> atom_to_binary(Atom).
 
 handle_parsing_opts(File, Opts) ->
   WarningFile =
-    case lists:keyfind(warnings, 1, Opts) of
-      {warnings, false} -> nil;
+    case lists:keyfind(emit_warnings, 1, Opts) of
+      {emit_warnings, false} -> nil;
       _ -> File
     end,
   LiteralEncoder =

lib/elixir/src/elixir.hrl

@@ -35,7 +35,7 @@
   identifier_tokenizer=elixir_tokenizer,
   ascii_identifiers_only=true,
   indentation=0,
+  column=1,
   mismatch_hints=[],
-  warn_on_unnecessary_quotes=true,
   warnings=[]
 }).

lib/elixir/src/elixir_bitstring.erl

@@ -35,7 +35,12 @@ expand(BitstrMeta, Fun, [{'::', Meta, [Left, Right]} | T], Acc, S, E, Alignment,
 
   MatchOrRequireSize = RequireSize or is_match_size(T, EL),
   EType = expr_type(ELeft),
-  {ERight, EAlignment, SS, ES} = expand_specs(EType, Meta, Right, SL, OriginalS, EL, MatchOrRequireSize),
+  ExpectSize = case ELeft of
+    {'^', _, [{_, _, _}]} -> {infer, ELeft};
+    _ when MatchOrRequireSize -> required;
+    _ -> optional
+  end,
+  {ERight, EAlignment, SS, ES} = expand_specs(EType, Meta, Right, SL, OriginalS, EL, ExpectSize),
 
   EAcc = concat_or_prepend_bitstring(Meta, ELeft, ERight, Acc, ES, MatchOrRequireSize),
   expand(BitstrMeta, Fun, T, EAcc, {SS, OriginalS}, ES, alignment(Alignment, EAlignment), RequireSize);
@@ -147,7 +152,7 @@ expand_expr(Meta, Component, Fun, S, E) ->
 
 %% Expands and normalizes types of a bitstring.
 
-expand_specs(ExprType, Meta, Info, S, OriginalS, E, RequireSize) ->
+expand_specs(ExprType, Meta, Info, S, OriginalS, E, ExpectSize) ->
   Default =
     #{size => default,
       unit => default,
@@ -158,11 +163,17 @@ expand_specs(ExprType, Meta, Info, S, OriginalS, E, RequireSize) ->
     expand_each_spec(Meta, unpack_specs(Info, []), Default, S, OriginalS, E),
 
   MergedType = type(Meta, ExprType, Type, E),
-  validate_size_required(Meta, RequireSize, ExprType, MergedType, Size, ES),
+  validate_size_required(Meta, ExpectSize, ExprType, MergedType, Size, ES),
   SizeAndUnit = size_and_unit(Meta, ExprType, Size, Unit, ES),
   Alignment = compute_alignment(MergedType, Size, Unit),
 
-  [H | T] = build_spec(Meta, Size, Unit, MergedType, Endianness, Sign, SizeAndUnit, ES),
+  MaybeInferredSize = case {ExpectSize, MergedType, SizeAndUnit} of
+    {{infer, PinnedVar}, binary, []} -> [{size, Meta, [{{'.', Meta, [erlang, byte_size]}, Meta, [PinnedVar]}]}];
+    {{infer, PinnedVar}, bitstring, []} -> [{size, Meta, [{{'.', Meta, [erlang, bit_size]}, Meta, [PinnedVar]}]}];
+    _ -> SizeAndUnit
+  end,
+
+  [H | T] = build_spec(Meta, Size, Unit, MergedType, Endianness, Sign, MaybeInferredSize, ES),
   {lists:foldl(fun(I, Acc) -> {'-', Meta, [Acc, I]} end, H, T), Alignment, SS, ES}.
 
 type(_, default, default, _) ->
@@ -276,7 +287,7 @@ validate_spec_arg(Meta, unit, Value, _S, _OriginalS, E) when not is_integer(Valu
 validate_spec_arg(_Meta, _Key, _Value, _S, _OriginalS, _E) ->
   ok.
 
-validate_size_required(Meta, true, default, Type, default, E) when Type == binary; Type == bitstring ->
+validate_size_required(Meta, required, default, Type, default, E) when Type == binary; Type == bitstring ->
   function_error(Meta, E, ?MODULE, unsized_binary),
   ok;
 validate_size_required(_, _, _, _, _, _) ->

lib/elixir/src/elixir_compiler.erl

@@ -1,6 +1,6 @@
 %% Elixir compiler front-end to the Erlang backend.
 -module(elixir_compiler).
--export([string/3, quoted/3, bootstrap/0, file/2, compile/3]).
+-export([string/3, quoted/3, bootstrap/0, file/2, compile/4]).
 -include("elixir.hrl").
 
 string(Contents, File, Callback) ->
@@ -36,22 +36,22 @@ maybe_fast_compile(Forms, Args, E) ->
   case (?key(E, module) == nil) andalso allows_fast_compilation(Forms) andalso
         (not elixir_config:is_bootstrap()) of
     true  -> fast_compile(Forms, E);
-    false -> compile(Forms, Args, E)
+    false -> compile(Forms, Args, [], E)
   end,
   ok.
 
-compile(Quoted, ArgsList, #{line := Line} = E) ->
+compile(Quoted, ArgsList, CompilerOpts, #{line := Line} = E) ->
   Block = no_tail_optimize([{line, Line}], Quoted),
   {Expanded, SE, EE} = elixir_expand:expand(Block, elixir_env:env_to_ex(E), E),
   elixir_env:check_unused_vars(SE, EE),
 
   {Module, Fun, Purgeable} =
-    elixir_erl_compiler:spawn(fun() -> spawned_compile(Expanded, E) end),
+    elixir_erl_compiler:spawn(fun() -> spawned_compile(Expanded, CompilerOpts, E) end),
 
   Args = list_to_tuple(ArgsList),
   {dispatch(Module, Fun, Args, Purgeable), SE, EE}.
 
-spawned_compile(ExExprs, #{line := Line, file := File} = E) ->
+spawned_compile(ExExprs, CompilerOpts, #{line := Line, file := File} = E) ->
   {Vars, S} = elixir_erl_var:from_env(E),
   {ErlExprs, _} = elixir_erl_pass:translate(ExExprs, erl_anno:new(Line), S),
 
@@ -59,7 +59,7 @@ spawned_compile(ExExprs, #{line := Line, file := File} = E) ->
   Fun = code_fun(?key(E, module)),
   Forms = code_mod(Fun, ErlExprs, Line, File, Module, Vars),
 
-  {Module, Binary} = elixir_erl_compiler:noenv_forms(Forms, File, [nowarn_nomatch, no_bool_opt, no_ssa_opt]),
+  {Module, Binary} = elixir_erl_compiler:noenv_forms(Forms, File, [nowarn_nomatch | CompilerOpts]),
   code:load_binary(Module, "", Binary),
   {Module, Fun, is_purgeable(Module, Binary)}.
 

lib/elixir/src/elixir_errors.erl

@@ -26,7 +26,7 @@ print_warning(Position, File, Message) ->
 %% Called by Module.ParallelChecker.
 print_warning(Message, Diagnostic) ->
   #{file := File, position := Position, stacktrace := S} = Diagnostic,
-  Snippet = get_snippet(File, Position),
+  Snippet = read_snippet(File, Position),
   Span = get_span(Diagnostic),
   Output = format_snippet(Position, File, Message, Snippet, warning, S, Span),
   io:put_chars(standard_error, [Output, $\n, $\n]).
@@ -34,7 +34,7 @@ print_warning(Message, Diagnostic) ->
 %% Called by Module.ParallelChecker.
 print_warning_group(Message, [Diagnostic | Others]) ->
   #{file := File, position := Position, stacktrace := S} = Diagnostic,
-  Snippet = get_snippet(File, Position),
+  Snippet = read_snippet(File, Position),
   Span = get_span(Diagnostic),
   Formatted = format_snippet(Position, File, Message, Snippet, warning, S, Span),
   LineNumber = extract_line(Position),
@@ -49,16 +49,15 @@ print_warning_group(Message, [Diagnostic | Others]) ->
 get_span(#{span := nil}) -> nil;
 get_span(#{span := Span}) -> Span.
 
-get_snippet(nil, _Position) ->
+read_snippet(nil, _Position) ->
   nil;
-get_snippet(<<"nofile">>, _Position) ->
+read_snippet(<<"nofile">>, _Position) ->
   nil;
-get_snippet(File, Position) ->
+read_snippet(File, Position) ->
   LineNumber = extract_line(Position),
   get_file_line(File, LineNumber).
 
-get_file_line(_, 0) -> nil;
-get_file_line(File, LineNumber) ->
+get_file_line(File, LineNumber) when is_integer(LineNumber), LineNumber > 0 ->
   case file:open(File, [read, binary]) of
     {ok, IoDevice} ->
       Line = traverse_file_line(IoDevice, LineNumber),
@@ -66,7 +65,8 @@ get_file_line(File, LineNumber) ->
       Line;
     {error, _} ->
       nil
-  end.
+  end;
+get_file_line(_, _) -> nil.
 
 traverse_file_line(IoDevice, 1) ->
   case file:read_line(IoDevice) of
@@ -80,7 +80,7 @@ traverse_file_line(IoDevice, N) ->
 print_diagnostic(#{severity := Severity, message := M, stacktrace := Stacktrace, position := P, file := F} = Diagnostic, ReadSnippet) ->
   Snippet =
     case ReadSnippet of
-      true -> get_snippet(F, P);
+      true -> read_snippet(F, P);
       false -> nil
     end,
 
@@ -106,6 +106,7 @@ emit_diagnostic(Severity, Position, File, Message, Stacktrace, Options) ->
 
   Diagnostic = #{
     severity => Severity,
+    source => File,
     file => File,
     position => Position,
     message => unicode:characters_to_binary(Message),
@@ -319,6 +320,7 @@ parse_error(Location, File, Error, <<>>, Input) ->
     <<"syntax error before: ">> -> <<"syntax error: expression is incomplete">>;
     _ -> <<Error/binary>>
   end,
+
   raise_snippet(Location, File, Input, 'Elixir.TokenMissingError', Message);
 
 %% Show a nicer message for end of line
@@ -394,8 +396,10 @@ parse_error(Location, File, <<"syntax error before: ">>, <<$$, Char/binary>>, In
 parse_error(Location, File, Error, Token, Input) when is_binary(Error), is_binary(Token) ->
   Message = <<Error/binary, Token/binary>>,
   case lists:keytake(error_type, 1, Location) of
-    {value, {error_type, mismatched_delimiter}, Loc} -> raise_mismatched_delimiter(Loc, File, Input, Message);
-    _ -> raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message)
+    {value, {error_type, mismatched_delimiter}, Loc} ->
+      raise_snippet(Loc, File, Input, 'Elixir.MismatchedDelimiterError', Message);
+    _ ->
+      raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message)
   end.
 
 parse_erl_term(Term) ->
@@ -403,11 +407,6 @@ parse_erl_term(Term) ->
   {ok, Parsed} = erl_parse:parse_term(Tokens ++ [{dot, 1}]),
   Parsed.
 
-raise_mismatched_delimiter(Location, File, Input, Message) ->
-  {InputString, _, _} = Input,
-  InputBinary = elixir_utils:characters_to_binary(InputString),
-  raise('Elixir.MismatchedDelimiterError', Message,  [{file, File}, {snippet, InputBinary} | Location]).
-
 raise_reserved(Location, File, Input, Keyword) ->
   raise_snippet(Location, File, Input, 'Elixir.SyntaxError',
         <<"syntax error before: ", Keyword/binary, ". \"", Keyword/binary, "\" is a "
@@ -415,25 +414,43 @@ raise_reserved(Location, File, Input, Keyword) ->
           "it can't be used as a variable or be defined nor invoked as a regular function">>).
 
 raise_snippet(Location, File, Input, Kind, Message) when is_binary(File) ->
-  {InputString, StartLine, _} = Input,
-  Snippet = snippet_line(InputString, Location, StartLine),
+  Snippet = cut_snippet(Location, Input),
   raise(Kind, Message, [{file, File}, {snippet, Snippet} | Location]).
 
-snippet_line(InputString, Location, StartLine) ->
-  {line, Line} = lists:keyfind(line, 1, Location),
+cut_snippet(Location, Input) ->
   case lists:keyfind(column, 1, Location) of
     {column, _} ->
-      Lines = string:split(InputString, "\n", all),
-      Snippet = (lists:nth(Line - StartLine + 1, Lines)),
-      case string:trim(Snippet, leading) of
-        [] -> nil;
-        _ -> elixir_utils:characters_to_binary(Snippet)
+      {line, Line} = lists:keyfind(line, 1, Location),
+
+      case lists:keyfind(end_line, 1, Location) of
+        {end_line, EndLine} ->
+          cut_snippet(Input, Line, EndLine - Line + 1);
+
+        false ->
+          Snippet = cut_snippet(Input, Line, 1),
+          case string:trim(Snippet, leading) of
+            <<>> -> nil;
+            _ -> Snippet
+          end
       end;
 
     false ->
       nil
   end.
 
+cut_snippet({InputString, StartLine, StartColumn}, Line, Span) ->
+  %% In case the code is indented, we need to add the indentation back
+  %% for the snippets to match the reported columns.
+  Indent = binary:copy(<<" ">>, StartColumn - 1),
+  Lines = string:split(InputString, "\n", all),
+  [Head | Tail] = lists:nthtail(Line - StartLine, Lines),
+  IndentedTail = indent_n(Tail, Span - 1, <<"\n", Indent/binary>>),
+  elixir_utils:characters_to_binary([Indent, Head, IndentedTail]).
+
+indent_n([], _Count, _Indent) -> [];
+indent_n(_Lines, 0, _Indent) -> [];
+indent_n([H | T], Count, Indent) -> [Indent, H | indent_n(T, Count - 1, Indent)].
+
 %% Helpers
 
 prefix(warning) -> highlight(<<"warning:">>, warning);

lib/elixir/src/elixir_expand.erl

@@ -402,16 +402,12 @@ expand({Atom, Meta, Args}, S, E) when is_atom(Atom), is_list(Meta), is_list(Args
 
 %% Remote calls
 
-expand({{'.', DotMeta, [Left, Right]}, Meta, Args} = Expr, S, E)
+expand({{'.', DotMeta, [Left, Right]}, Meta, Args}, S, E)
     when (is_tuple(Left) orelse is_atom(Left)), is_atom(Right), is_list(Meta), is_list(Args) ->
   {ELeft, SL, EL} = expand(Left, elixir_env:prepare_write(S), E),
-  NoParens = lists:keyfind(no_parens, 1, Meta),
-
-  (is_atom(ELeft) and (Args =:= []) and (NoParens =:= {no_parens, true})) andalso
-    elixir_errors:file_warn(Meta, E, ?MODULE, {remote_nullary_no_parens, Expr}),
 
   elixir_dispatch:dispatch_require(Meta, ELeft, Right, Args, S, EL, fun(AR, AF, AA) ->
-    expand_remote(AR, DotMeta, AF, Meta, NoParens, AA, S, SL, EL)
+    expand_remote(AR, DotMeta, AF, Meta, AA, S, SL, EL)
   end);
 
 %% Anonymous calls
@@ -858,12 +854,13 @@ expand_local(Meta, Name, Args, _S, #{function := nil} = E) ->
 
 %% Remote
 
-expand_remote(Receiver, DotMeta, Right, Meta, NoParens, Args, S, SL, #{context := Context} = E) when is_atom(Receiver) or is_tuple(Receiver) ->
+expand_remote(Receiver, DotMeta, Right, Meta, Args, S, SL, #{context := Context} = E)
+    when is_atom(Receiver) or is_tuple(Receiver) ->
   assert_no_clauses(Right, Meta, Args, E),
 
   if
     Context =:= guard, is_tuple(Receiver) ->
-      (NoParens /= {no_parens, true}) andalso
+      (lists:keyfind(no_parens, 1, Meta) /= {no_parens, true}) andalso
         function_error(Meta, E, ?MODULE, {parens_map_lookup, Receiver, Right, guard_context(S)}),
 
       {{{'.', DotMeta, [Receiver, Right]}, Meta, []}, SL, E};
@@ -881,7 +878,7 @@ expand_remote(Receiver, DotMeta, Right, Meta, NoParens, Args, S, SL, #{context :
           file_error(Meta, E, elixir_rewrite, Error)
       end
   end;
-expand_remote(Receiver, DotMeta, Right, Meta, _NoParens, Args, _, _, E) ->
+expand_remote(Receiver, DotMeta, Right, Meta, Args, _, _, E) ->
   Call = {{'.', DotMeta, [Receiver, Right]}, Meta, Args},
   file_error(Meta, E, ?MODULE, {invalid_call, Call}).
 
@@ -1171,9 +1168,6 @@ assert_no_underscore_clause_in_cond(_Other, _E) ->
 guard_context(#elixir_ex{prematch={_, _, {bitsize, _}}}) -> "bitstring size specifier";
 guard_context(_) -> "guards".
 
-format_error({remote_nullary_no_parens, Expr}) ->
-  String = 'Elixir.String':replace_suffix('Elixir.Macro':to_string(Expr), <<"()">>, <<>>),
-  io_lib:format("parentheses are required for function calls with no arguments, got: ~ts", [String]);
 format_error(invalid_match_on_zero_float) ->
   "pattern matching on 0.0 is equivalent to matching only on +0.0 from Erlang/OTP 27+. Instead you must match on +0.0 or -0.0";
 format_error({useless_literal, Term}) ->

lib/elixir/src/elixir_import.erl

@@ -9,24 +9,26 @@ import(Meta, Ref, Opts, E) ->
   {Functions, Macros, Added} =
     case keyfind(only, Opts) of
       {only, functions} ->
-        {Added1, Funs} = import_functions(Meta, Ref, Opts, E),
+        {Added1, _Used1, Funs} = import_functions(Meta, Ref, Opts, E),
         {Funs, keydelete(Ref, ?key(E, macros)), Added1};
       {only, macros} ->
-        {Added2, Macs} = import_macros(true, Meta, Ref, Opts, E),
+        {Added2, _Used2, Macs} = import_macros(true, Meta, Ref, Opts, E),
         {keydelete(Ref, ?key(E, functions)), Macs, Added2};
       {only, sigils} ->
-        {Added1, Funs} = import_sigil_functions(Meta, Ref, Opts, E),
-        {Added2, Macs} = import_sigil_macros(Meta, Ref, Opts, E),
+        {Added1, _Used1, Funs} = import_sigil_functions(Meta, Ref, Opts, E),
+        {Added2, _Used2, Macs} = import_sigil_macros(Meta, Ref, Opts, E),
         {Funs, Macs, Added1 or Added2};
       {only, List} when is_list(List) ->
-        {Added1, Funs} = import_functions(Meta, Ref, Opts, E),
-        {Added2, Macs} = import_macros(false, Meta, Ref, Opts, E),
+        {Added1, Used1, Funs} = import_functions(Meta, Ref, Opts, E),
+        {Added2, Used2, Macs} = import_macros(false, Meta, Ref, Opts, E),
+        [elixir_errors:file_warn(Meta, ?key(E, file), ?MODULE, {invalid_import, {Ref, Name, Arity}}) ||
+         {Name, Arity} <- (List -- Used1) -- Used2],
         {Funs, Macs, Added1 or Added2};
       {only, Other} ->
         elixir_errors:file_error(Meta, E, ?MODULE, {invalid_option, only, Other});
       false ->
-        {Added1, Funs} = import_functions(Meta, Ref, Opts, E),
-        {Added2, Macs} = import_macros(false, Meta, Ref, Opts, E),
+        {Added1, _Used1, Funs} = import_functions(Meta, Ref, Opts, E),
+        {Added2, _Used2, Macs} = import_macros(false, Meta, Ref, Opts, E),
         {Funs, Macs, Added1 or Added2}
     end,
 
@@ -95,9 +97,6 @@ calculate(Meta, Key, Opts, Old, File, Existing) ->
         _ -> elixir_errors:file_error(Meta, File, ?MODULE, only_and_except_given)
       end,
 
-      [elixir_errors:file_warn(Meta, File, ?MODULE, {invalid_import, {Key, Name, Arity}}) ||
-       {Name, Arity} <- Only -- get_exports(Key)],
-
       intersection(Only, Existing());
 
     _ ->
@@ -125,17 +124,14 @@ calculate(Meta, Key, Opts, Old, File, Existing) ->
   %% Normalize the data before storing it
   case ordsets:from_list(New) of
     [] ->
-      {false, keydelete(Key, Old)};
+      {false, [], keydelete(Key, Old)};
     Set  ->
       ensure_no_special_form_conflict(Meta, File, Key, Set),
-      {true, [{Key, Set} | keydelete(Key, Old)]}
+      {true, Set, [{Key, Set} | keydelete(Key, Old)]}
   end.
 
 %% Retrieve functions and macros from modules
 
-get_exports(Module) ->
-  get_functions(Module) ++ get_macros(Module).
-
 get_functions(Module) ->
   try
     Module:'__info__'(functions)
@@ -143,14 +139,6 @@ get_functions(Module) ->
     error:undef -> remove_internals(Module:module_info(exports))
   end.
 
-get_macros(Module) ->
-  case fetch_macros(Module) of
-    {ok, Macros} ->
-      Macros;
-    error ->
-      []
-  end.
-
 fetch_macros(Module) ->
   try
     {ok, Module:'__info__'(macros)}

lib/elixir/src/elixir_interpolation.erl

@@ -97,7 +97,7 @@ extract_nl(Rest, Buffer, Output, Line, Scope, Interpol, [H,H,H] = Last) ->
       extract(NewRest, NewBuffer, Output, Line + 1, Column, Scope, Interpol, Last)
   end;
 extract_nl(Rest, Buffer, Output, Line, Scope, Interpol, Last) ->
-  extract(Rest, Buffer, Output, Line + 1, 1, Scope, Interpol, Last).
+  extract(Rest, Buffer, Output, Line + 1, Scope#elixir_tokenizer.column, Scope, Interpol, Last).
 
 strip_horizontal_space([H | T], Buffer, Counter) when H =:= $\s; H =:= $\t ->
   strip_horizontal_space(T, [H | Buffer], Counter + 1);

lib/elixir/src/elixir_module.erl

@@ -421,7 +421,9 @@ build(Module, Line, File, E) ->
 %% Handles module and callback evaluations.
 
 eval_form(Line, Module, DataBag, Block, Vars, Prune, E) ->
-  {Value, ExS, EE} = elixir_compiler:compile(Block, Vars, E),
+  %% Given Elixir modules can get very long to compile due to metaprogramming,
+  %% we disable expansions that take linear time to code size.
+  {Value, ExS, EE} = elixir_compiler:compile(Block, Vars, [no_bool_opt, no_ssa_opt], E),
   elixir_overridable:store_not_overridden(Module),
   EV = (elixir_env:reset_vars(EE))#{line := Line},
   EC = eval_callbacks(Line, DataBag, before_compile, [EV], EV),

lib/elixir/src/elixir_parser.yrl

@@ -910,11 +910,6 @@ build_identifier({'.', Meta, _} = Dot, Args) ->
 build_identifier({op_identifier, Location, Identifier}, [Arg]) ->
   {Identifier, [{ambiguous_op, nil} | meta_from_location(Location)], [Arg]};
 
-%% TODO: Either remove ... or make it an operator on v2.0
-build_identifier({_, {Line, Column, _} = Location, '...'}, Args) when is_list(Args) ->
-  warn({Line, Column}, "... is no longer supported as a function call and it must receive no arguments"),
-  {'...', meta_from_location(Location), Args};
-
 build_identifier({_, Location, Identifier}, Args) ->
   {Identifier, meta_from_location(Location), Args}.
 

lib/elixir/src/elixir_tokenizer.erl

@@ -123,11 +123,9 @@ tokenize(String, Line, Column, Opts) ->
         Acc#elixir_tokenizer{preserve_comments=PreserveComments};
       ({unescape, Unescape}, Acc) when is_boolean(Unescape) ->
         Acc#elixir_tokenizer{unescape=Unescape};
-      ({warn_on_unnecessary_quotes, Unnecessary}, Acc) when is_boolean(Unnecessary) ->
-        Acc#elixir_tokenizer{warn_on_unnecessary_quotes=Unnecessary};
       (_, Acc) ->
         Acc
-    end, #elixir_tokenizer{identifier_tokenizer=IdentifierTokenizer}, Opts),
+    end, #elixir_tokenizer{identifier_tokenizer=IdentifierTokenizer, column=Column}, Opts),
 
   tokenize(String, Line, Column, Scope, []).
 
@@ -144,12 +142,19 @@ tokenize([], Line, Column, #elixir_tokenizer{cursor_completion=Cursor} = Scope,
   AccTokens = cursor_complete(Line, CursorColumn, CursorTerminators, CursorTokens),
   {ok, Line, Column, AllWarnings, AccTokens};
 
-tokenize([], EndLine, Column, #elixir_tokenizer{terminators=[{Start, {StartLine, _, _}, _} | _]} = Scope, Tokens) ->
+tokenize([], EndLine, EndColumn, #elixir_tokenizer{terminators=[{Start, {StartLine, StartColumn, _}, _} | _]} = Scope, Tokens) ->
   End = terminator(Start),
   Hint = missing_terminator_hint(Start, End, Scope),
-  Message = "missing terminator: ~ts (for \"~ts\" starting at line ~B)",
-  Formatted = io_lib:format(Message, [End, Start, StartLine]),
-  Meta = [{opening_delimiter, Start} | ?LOC(EndLine, Column)],
+  Message = "missing terminator: ~ts",
+  Formatted = io_lib:format(Message, [End]),
+  Meta = [
+    {opening_delimiter, Start},
+    {expected_delimiter, End},
+    {line, StartLine},
+    {column, StartColumn},
+    {end_line, EndLine},
+    {end_column, EndColumn}
+  ],
   error({Meta, [Formatted, Hint], []}, [], Scope, Tokens);
 
 tokenize([], Line, Column, #elixir_tokenizer{} = Scope, Tokens) ->
@@ -527,7 +532,7 @@ tokenize([$:, H | T] = Original, Line, Column, Scope, Tokens) when ?is_quote(H)
 
     {error, Reason} ->
       Message = " (for atom starting at line ~B)",
-      interpolation_error(Reason, Original, Scope, Tokens, Message, [Line])
+      interpolation_error(Reason, Original, Scope, Tokens, Message, [Line], Line, Column + 1, [H], [H])
   end;
 
 tokenize([$: | String] = Original, Line, Column, Scope, Tokens) ->
@@ -706,9 +711,9 @@ unexpected_token([T | Rest], Line, Column, Scope, Tokens) ->
   error({?LOC(Line, Column), "unexpected token: ", Message}, Rest, Scope, Tokens).
 
 tokenize_eol(Rest, Line, Scope, Tokens) ->
-  {StrippedRest, Indentation} = strip_horizontal_space(Rest, 0),
-  IndentedScope = Scope#elixir_tokenizer{indentation=Indentation},
-  tokenize(StrippedRest, Line + 1, Indentation + 1, IndentedScope, Tokens).
+  {StrippedRest, Column} = strip_horizontal_space(Rest, Scope#elixir_tokenizer.column),
+  IndentedScope = Scope#elixir_tokenizer{indentation=Column-1},
+  tokenize(StrippedRest, Line + 1, Column, IndentedScope, Tokens).
 
 strip_horizontal_space([H | T], Counter) when ?is_horizontal_space(H) ->
   strip_horizontal_space(T, Counter + 1);
@@ -724,12 +729,12 @@ tokenize_dot(T, Line, Column, DotInfo, Scope, Tokens) ->
 
         {Rest, Comment} ->
           preserve_comments(Line, Column, Tokens, Comment, Rest, Scope),
-          tokenize_dot(Rest, Line, 1, DotInfo, Scope, Tokens)
+          tokenize_dot(Rest, Line, Scope#elixir_tokenizer.column, DotInfo, Scope, Tokens)
       end;
     {"\r\n" ++ Rest, _} ->
-      tokenize_dot(Rest, Line + 1, 1, DotInfo, Scope, Tokens);
+      tokenize_dot(Rest, Line + 1, Scope#elixir_tokenizer.column, DotInfo, Scope, Tokens);
     {"\n" ++ Rest, _} ->
-      tokenize_dot(Rest, Line + 1, 1, DotInfo, Scope, Tokens);
+      tokenize_dot(Rest, Line + 1, Scope#elixir_tokenizer.column, DotInfo, Scope, Tokens);
     {Rest, Length} ->
       handle_dot([$. | Rest], Line, Column + Length, DotInfo, Scope, Tokens)
   end.
@@ -768,7 +773,7 @@ handle_heredocs(T, Line, Column, H, Scope, Tokens) ->
 handle_strings(T, Line, Column, H, Scope, Tokens) ->
   case elixir_interpolation:extract(Line, Column, Scope, true, T, H) of
     {error, Reason} ->
-      interpolation_error(Reason, [H | T], Scope, Tokens, " (for string starting at line ~B)", [Line]);
+      interpolation_error(Reason, [H | T], Scope, Tokens, " (for string starting at line ~B)", [Line], Line, Column-1, [H], [H]);
 
     {NewLine, NewColumn, Parts, [$: | Rest], InterScope} when ?is_space(hd(Rest)) ->
       NewScope = case is_unnecessary_quote(Parts, InterScope) of
@@ -794,7 +799,7 @@ handle_strings(T, Line, Column, H, Scope, Tokens) ->
               Token = {kw_identifier, {Line, Column - 1, nil}, Atom},
               tokenize(Rest, NewLine, NewColumn + 1, NewScope, [Token | Tokens]);
             {error, Reason} ->
-              {error, Reason, Rest, Tokens}
+              error(Reason, Rest, NewScope, Tokens)
           end;
 
         {ok, Unescaped} ->
@@ -921,7 +926,7 @@ handle_dot([$., H | T] = Original, Line, Column, DotInfo, Scope, Tokens) when ?i
       Message = "interpolation is not allowed when calling function/macro. Found interpolation in a call starting with: ",
       error({?LOC(Line, Column), Message, [H]}, Rest, NewScope, Tokens);
     {error, Reason} ->
-      interpolation_error(Reason, Original, Scope, Tokens, " (for function name starting at line ~B)", [Line])
+      interpolation_error(Reason, Original, Scope, Tokens, " (for function name starting at line ~B)", [Line], Line, Column, [H], [H])
   end;
 
 handle_dot([$. | Rest], Line, Column, DotInfo, Scope, Tokens) ->
@@ -964,12 +969,11 @@ eol(_Line, _Column, [{eol, {Line, Column, Count}} | Tokens]) ->
 eol(Line, Column, Tokens) ->
   [{eol, {Line, Column, 1}} | Tokens].
 
-is_unnecessary_quote([Part], #elixir_tokenizer{warn_on_unnecessary_quotes=true} = Scope) when is_list(Part) ->
+is_unnecessary_quote([Part], Scope) when is_list(Part) ->
   case (Scope#elixir_tokenizer.identifier_tokenizer):tokenize(Part) of
     {identifier, _, [], _, true, Special} -> not lists:member(at, Special);
     _ -> false
   end;
-
 is_unnecessary_quote(_Parts, _Scope) ->
   false.
 
@@ -1027,7 +1031,7 @@ extract_heredoc_with_interpolation(Line, Column, Scope, Interpol, T, H) ->
           {ok, NewLine, NewColumn, tokens_to_binary(Parts2), Rest, NewScope};
 
         {error, Reason} ->
-          {error, interpolation_format(Reason, " (for heredoc starting at line ~B)", [Line])}
+          {error, interpolation_format(Reason, " (for heredoc starting at line ~B)", [Line], Line, Column, [H, H, H], [H, H, H])}
       end;
 
     error ->
@@ -1340,12 +1344,20 @@ previous_was_eol(_) -> nil.
 
 %% Error handling
 
-interpolation_error(Reason, Rest, Scope, Tokens, Extension, Args) ->
-  error(interpolation_format(Reason, Extension, Args), Rest, Scope, Tokens).
+interpolation_error(Reason, Rest, Scope, Tokens, Extension, Args, Line, Column, Opening, Closing) ->
+  error(interpolation_format(Reason, Extension, Args, Line, Column, Opening, Closing), Rest, Scope, Tokens).
 
-interpolation_format({string, Line, Column, Message, Token}, Extension, Args) ->
-  {?LOC(Line, Column), [Message, io_lib:format(Extension, Args)], Token};
-interpolation_format({_, _, _} = Reason, _Extension, _Args) ->
+interpolation_format({string, EndLine, EndColumn, Message, Token}, Extension, Args, Line, Column, Opening, Closing) ->
+  Meta = [
+    {opening_delimiter, list_to_atom(Opening)},
+    {expected_delimiter, list_to_atom(Closing)},
+    {line, Line},
+    {column, Column},
+    {end_line, EndLine},
+    {end_column, EndColumn}
+  ],
+  {Meta, [Message, io_lib:format(Extension, Args)], Token};
+interpolation_format({_, _, _} = Reason, _Extension, _Args, _Line, _Column, _Opening, _Closing) ->
   Reason.
 
 %% Terminators
@@ -1417,7 +1429,7 @@ check_terminator({End, {EndLine, EndColumn, _}}, [{Start, {StartLine, StartColum
     End ->
       {ok, Scope#elixir_tokenizer{terminators=Terminators}};
 
-    _ExpectedEnd ->
+    ExpectedEnd ->
       Meta = [
         {line, StartLine},
         {column, StartColumn},
@@ -1425,7 +1437,8 @@ check_terminator({End, {EndLine, EndColumn, _}}, [{Start, {StartLine, StartColum
         {end_column, EndColumn},
         {error_type, mismatched_delimiter},
         {opening_delimiter, Start},
-        {closing_delimiter, End}
+        {closing_delimiter, End},
+        {expected_delimiter, ExpectedEnd}
      ],
      {error, {Meta, unexpected_token_or_reserved(End), [atom_to_list(End)]}}
   end;
@@ -1583,7 +1596,7 @@ tokenize_sigil_contents([H | T] = Original, [S | _] = SigilName, Line, Column, S
     {error, Reason} ->
       Sigil = [$~, S, H],
       Message = " (for sigil ~ts starting at line ~B)",
-      interpolation_error(Reason, [$~] ++ SigilName ++ Original, Scope, Tokens, Message, [Sigil, Line])
+      interpolation_error(Reason, [$~] ++ SigilName ++ Original, Scope, Tokens, Message, [Sigil, Line], Line, Column, [H], [sigil_terminator(H)])
   end;
 
 tokenize_sigil_contents([H | _] = Original, SigilName, Line, Column, Scope, Tokens) ->

lib/elixir/test/elixir/code_formatter/containers_test.exs

@@ -203,6 +203,7 @@ defmodule Code.Formatter.ContainersTest do
       assert_same ~S(["\w": 1, "\\w": 2])
       assert_same ~S(["Elixir.Foo": 1, "Elixir.Bar": 2])
       assert_format ~S(["Foo": 1, "Bar": 2]), ~S([Foo: 1, Bar: 2])
+      assert_same ~S(["with \"scare quotes\"": 1])
     end
 
     test "with operators keyword lists" do

lib/elixir/test/elixir/code_fragment_test.exs

@@ -443,12 +443,11 @@ defmodule CodeFragmentTest do
     end
 
     test "column out of range" do
-      assert CF.surround_context("hello", {1, 20}) ==
-               %{begin: {1, 1}, context: {:local_or_var, ~c"hello"}, end: {1, 6}}
+      assert CF.surround_context("hello", {1, 20}) == :none
     end
 
     test "local_or_var" do
-      for i <- 1..9 do
+      for i <- 1..8 do
         assert CF.surround_context("hello_wo", {1, i}) == %{
                  context: {:local_or_var, ~c"hello_wo"},
                  begin: {1, 1},
@@ -456,9 +455,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("hello_wo ", {1, 10}) == :none
+      assert CF.surround_context("hello_wo", {1, 9}) == :none
 
-      for i <- 2..10 do
+      for i <- 2..9 do
         assert CF.surround_context(" hello_wo", {1, i}) == %{
                  context: {:local_or_var, ~c"hello_wo"},
                  begin: {1, 2},
@@ -466,9 +465,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context(" hello_wo ", {1, 11}) == :none
+      assert CF.surround_context(" hello_wo", {1, 10}) == :none
 
-      for i <- 1..7 do
+      for i <- 1..6 do
         assert CF.surround_context("hello!", {1, i}) == %{
                  context: {:local_or_var, ~c"hello!"},
                  begin: {1, 1},
@@ -476,9 +475,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("hello! ", {1, 8}) == :none
+      assert CF.surround_context("hello!", {1, 7}) == :none
 
-      for i <- 1..6 do
+      for i <- 1..5 do
         assert CF.surround_context("안녕_세상", {1, i}) == %{
                  context: {:local_or_var, ~c"안녕_세상"},
                  begin: {1, 1},
@@ -486,7 +485,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("안녕_세상 ", {1, 6}) == :none
+      assert CF.surround_context("안녕_세상", {1, 6}) == :none
 
       # Keywords are not local or var
       for keyword <- ~w(do end after catch else rescue fn true false nil)c do
@@ -500,38 +499,8 @@ defmodule CodeFragmentTest do
       end
     end
 
-    test "local + operator" do
-      for i <- 1..8 do
-        assert CF.surround_context("hello_wo+", {1, i}) == %{
-                 context: {:local_or_var, ~c"hello_wo"},
-                 begin: {1, 1},
-                 end: {1, 9}
-               }
-      end
-
-      assert CF.surround_context("hello_wo+", {1, 9}) == %{
-               begin: {1, 9},
-               context: {:operator, ~c"+"},
-               end: {1, 10}
-             }
-
-      for i <- 1..9 do
-        assert CF.surround_context("hello_wo +", {1, i}) == %{
-                 context: {:local_or_var, ~c"hello_wo"},
-                 begin: {1, 1},
-                 end: {1, 9}
-               }
-      end
-
-      assert CF.surround_context("hello_wo +", {1, 10}) == %{
-               begin: {1, 10},
-               context: {:operator, ~c"+"},
-               end: {1, 11}
-             }
-    end
-
     test "local call" do
-      for i <- 1..9 do
+      for i <- 1..8 do
         assert CF.surround_context("hello_wo(", {1, i}) == %{
                  context: {:local_call, ~c"hello_wo"},
                  begin: {1, 1},
@@ -539,9 +508,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("hello_wo(", {1, 10}) == :none
+      assert CF.surround_context("hello_wo(", {1, 9}) == :none
 
-      for i <- 1..9 do
+      for i <- 1..8 do
         assert CF.surround_context("hello_wo (", {1, i}) == %{
                  context: {:local_call, ~c"hello_wo"},
                  begin: {1, 1},
@@ -549,10 +518,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("hello_wo (", {1, 10}) == :none
-      assert CF.surround_context("hello_wo (", {1, 11}) == :none
+      assert CF.surround_context("hello_wo (", {1, 9}) == :none
 
-      for i <- 1..7 do
+      for i <- 1..6 do
         assert CF.surround_context("hello!(", {1, i}) == %{
                  context: {:local_call, ~c"hello!"},
                  begin: {1, 1},
@@ -560,9 +528,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("hello!(", {1, 8}) == :none
+      assert CF.surround_context("hello!(", {1, 7}) == :none
 
-      for i <- 1..6 do
+      for i <- 1..5 do
         assert CF.surround_context("안녕_세상(", {1, i}) == %{
                  context: {:local_call, ~c"안녕_세상"},
                  begin: {1, 1},
@@ -570,7 +538,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("안녕_세상(", {1, 7}) == :none
+      assert CF.surround_context("안녕_세상(", {1, 6}) == :none
     end
 
     test "local arity" do
@@ -698,7 +666,7 @@ defmodule CodeFragmentTest do
     end
 
     test "alias" do
-      for i <- 1..9 do
+      for i <- 1..8 do
         assert CF.surround_context("HelloWor", {1, i}) == %{
                  context: {:alias, ~c"HelloWor"},
                  begin: {1, 1},
@@ -706,9 +674,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("HelloWor ", {1, 10}) == :none
+      assert CF.surround_context("HelloWor", {1, 9}) == :none
 
-      for i <- 2..10 do
+      for i <- 2..9 do
         assert CF.surround_context(" HelloWor", {1, i}) == %{
                  context: {:alias, ~c"HelloWor"},
                  begin: {1, 2},
@@ -716,9 +684,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context(" HelloWor ", {1, 11}) == :none
+      assert CF.surround_context(" HelloWor", {1, 10}) == :none
 
-      for i <- 1..10 do
+      for i <- 1..9 do
         assert CF.surround_context("Hello.Wor", {1, i}) == %{
                  context: {:alias, ~c"Hello.Wor"},
                  begin: {1, 1},
@@ -726,9 +694,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("Hello.Wor ", {1, 11}) == :none
+      assert CF.surround_context("Hello.Wor", {1, 10}) == :none
 
-      for i <- 1..12 do
+      for i <- 1..11 do
         assert CF.surround_context("Hello . Wor", {1, i}) == %{
                  context: {:alias, ~c"Hello.Wor"},
                  begin: {1, 1},
@@ -736,9 +704,9 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("Hello . Wor ", {1, 13}) == :none
+      assert CF.surround_context("Hello . Wor", {1, 12}) == :none
 
-      for i <- 1..16 do
+      for i <- 1..15 do
         assert CF.surround_context("Foo . Bar . Baz", {1, i}) == %{
                  context: {:alias, ~c"Foo.Bar.Baz"},
                  begin: {1, 1},
@@ -770,7 +738,7 @@ defmodule CodeFragmentTest do
                end: {1, 11}
              }
 
-      for i <- 1..15 do
+      for i <- 1..14 do
         assert CF.surround_context("__MODULE__.Foo", {1, i}) == %{
                  context: {:alias, {:local_or_var, ~c"__MODULE__"}, ~c"Foo"},
                  begin: {1, 1},
@@ -778,7 +746,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      for i <- 1..19 do
+      for i <- 1..18 do
         assert CF.surround_context("__MODULE__.Foo.Sub", {1, i}) == %{
                  context: {:alias, {:local_or_var, ~c"__MODULE__"}, ~c"Foo.Sub"},
                  begin: {1, 1},
@@ -830,7 +798,7 @@ defmodule CodeFragmentTest do
     end
 
     test "attribute submodules" do
-      for i <- 1..10 do
+      for i <- 1..9 do
         assert CF.surround_context("@some.Foo", {1, i}) == %{
                  context: {:alias, {:module_attribute, ~c"some"}, ~c"Foo"},
                  begin: {1, 1},
@@ -838,7 +806,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      for i <- 1..14 do
+      for i <- 1..13 do
         assert CF.surround_context("@some.Foo.Sub", {1, i}) == %{
                  context: {:alias, {:module_attribute, ~c"some"}, ~c"Foo.Sub"},
                  begin: {1, 1},
@@ -921,7 +889,7 @@ defmodule CodeFragmentTest do
                end: {1, 15}
              }
 
-      for i <- 2..10 do
+      for i <- 2..9 do
         assert CF.surround_context("%HelloWor", {1, i}) == %{
                  context: {:struct, ~c"HelloWor"},
                  begin: {1, 1},
@@ -929,7 +897,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("%HelloWor ", {1, 11}) == :none
+      assert CF.surround_context("%HelloWor", {1, 10}) == :none
 
       # With dot
       assert CF.surround_context("%Hello.Wor", {1, 1}) == %{
@@ -938,7 +906,7 @@ defmodule CodeFragmentTest do
                end: {1, 11}
              }
 
-      for i <- 2..11 do
+      for i <- 2..10 do
         assert CF.surround_context("%Hello.Wor", {1, i}) == %{
                  context: {:struct, ~c"Hello.Wor"},
                  begin: {1, 1},
@@ -946,7 +914,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("%Hello.Wor ", {1, 12}) == :none
+      assert CF.surround_context("%Hello.Wor", {1, 11}) == :none
 
       # With spaces
       assert CF.surround_context("% Hello . Wor", {1, 1}) == %{
@@ -955,7 +923,7 @@ defmodule CodeFragmentTest do
                end: {1, 14}
              }
 
-      for i <- 2..14 do
+      for i <- 2..13 do
         assert CF.surround_context("% Hello . Wor", {1, i}) == %{
                  context: {:struct, ~c"Hello.Wor"},
                  begin: {1, 1},
@@ -963,7 +931,7 @@ defmodule CodeFragmentTest do
                }
       end
 
-      assert CF.surround_context("% Hello . Wor ", {1, 15}) == :none
+      assert CF.surround_context("% Hello . Wor", {1, 14}) == :none
     end
 
     test "module attributes" do

lib/elixir/test/elixir/code_normalizer/quoted_ast_test.exs

@@ -229,6 +229,12 @@ defmodule Code.Normalizer.QuotedASTTest do
     test "invalid block" do
       assert quoted_to_string({:__block__, [], {:bar, [], []}}) ==
                "{:__block__, [], {:bar, [], []}}"
+
+      assert quoted_to_string({:foo, [], [{:do, :ok}, :not_keyword]}) ==
+               "foo({:do, :ok}, :not_keyword)"
+
+      assert quoted_to_string({:foo, [], [[{:do, :ok}, :not_keyword]]}) ==
+               "foo([{:do, :ok}, :not_keyword])"
     end
 
     test "not in" do

lib/elixir/test/elixir/code_test.exs

@@ -252,6 +252,7 @@ defmodule CodeTest do
                  message: "undefined variable \"x\"",
                  position: 1,
                  file: "nofile",
+                 source: "nofile",
                  stacktrace: [],
                  severity: :error
                }
@@ -483,7 +484,7 @@ defmodule CodeTest do
 
     assert_exception(
       SyntaxError,
-      ["nofile:11:5:", "syntax error before:", "1 + * 3", "^"],
+      ["nofile:11:7:", "syntax error before:", "1 + * 3", "^"],
       fn ->
         Code.string_to_quoted!(":ok\n1 + * 3", line: 10, column: 3)
       end
@@ -498,9 +499,13 @@ defmodule CodeTest do
   end
 
   test "string_to_quoted returns error on incomplete escaped string" do
-    assert Code.string_to_quoted("\"\\") ==
-             {:error,
-              {[line: 1, column: 3], "missing terminator: \" (for string starting at line 1)", ""}}
+    assert {:error, {meta, "missing terminator: \" (for string starting at line 1)", ""}} =
+             Code.string_to_quoted("\"\\")
+
+    assert meta[:line] == 1
+    assert meta[:column] == 1
+    assert meta[:end_line] == 1
+    assert meta[:end_column] == 3
   end
 
   test "compile source" do

lib/elixir/test/elixir/file/stream_test.exs

@@ -141,6 +141,35 @@ defmodule File.StreamTest do
         end
       end
 
+      test "supports byte offset" do
+        src = fixture_path("file.txt")
+
+        assert @node
+               |> stream!(src, [{:read_offset, 0}])
+               |> Enum.take(1) == ["FOO\n"]
+
+        assert @node
+               |> stream!(src, [{:read_offset, 1}])
+               |> Enum.take(1) == ["OO\n"]
+
+        assert @node
+               |> stream!(src, [{:read_offset, 4}])
+               |> Enum.take(1) == []
+
+        assert @node |> stream!(src, 1, [{:read_offset, 1}]) |> Enum.count() == 3
+        assert @node |> stream!(src, 1, [{:read_offset, 4}]) |> Enum.count() == 0
+      end
+
+      test "applies offset after trimming BOM" do
+        src = fixture_path("utf8_bom.txt")
+
+        assert @node
+               |> stream!(src, [:trim_bom, {:read_offset, 4}])
+               |> Enum.take(1) == ["сский\n"]
+
+        assert @node |> stream!(src, 1, [:trim_bom, {:read_offset, 4}]) |> Enum.count() == 15
+      end
+
       test "keeps BOM when raw" do
         src = fixture_path("utf8_bom.txt")
 
@@ -169,6 +198,7 @@ defmodule File.StreamTest do
 
         assert @node |> stream!(src, [:trim_bom]) |> Enum.count() == 2
         assert @node |> stream!(src, 1, [:trim_bom]) |> Enum.count() == 19
+        assert @node |> stream!(src, 2, [:trim_bom]) |> Enum.count() == 10
       end
 
       test "keeps BOM with utf8 encoding" do

lib/elixir/test/elixir/kernel/binary_test.exs

@@ -128,20 +128,6 @@ defmodule Kernel.BinaryTest do
     assert_raise ArgumentError, message, fn ->
       Code.eval_string(~s["a" <> b <> "c" = "abc"])
     end
-
-    assert_raise ArgumentError, message, fn ->
-      Code.eval_string(~s[
-        a = "a"
-        ^a <> "b" = "ab"
-      ])
-    end
-
-    assert_raise ArgumentError, message, fn ->
-      Code.eval_string(~s[
-        b = "b"
-        "a" <> ^b <> "c" = "abc"
-      ])
-    end
   end
 
   test "hex" do
@@ -269,6 +255,28 @@ defmodule Kernel.BinaryTest do
     assert <<1::size((^foo).bar)>> = <<1::5>>
   end
 
+  test "automatic size computation of matched bitsyntax variable" do
+    var = "foo"
+    <<^var::binary, rest::binary>> = "foobar"
+    assert rest == "bar"
+
+    <<^var::bytes, rest::bytes>> = "foobar"
+    assert rest == "bar"
+
+    ^var <> rest = "foobar"
+    assert rest == "bar"
+
+    var = <<0, 1>>
+    <<^var::bitstring, rest::bitstring>> = <<0, 1, 2, 3>>
+    assert rest == <<2, 3>>
+
+    <<^var::bits, rest::bits>> = <<0, 1, 2, 3>>
+    assert rest == <<2, 3>>
+
+    ^var <> rest = <<0, 1, 2, 3>>
+    assert rest == <<2, 3>>
+  end
+
   defmacro signed_16 do
     quote do
       big - signed - integer - unit(16)

lib/elixir/test/elixir/kernel/diagnostics_test.exs

@@ -53,6 +53,28 @@ defmodule Kernel.DiagnosticsTest do
              """
     end
 
+    test "same line with offset" do
+      output =
+        capture_raise(
+          """
+          [1, 2, 3, 4, 5, 6)
+          """,
+          MismatchedDelimiterError,
+          line: 3
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:3:18:
+                 error: unexpected token: )
+                 │
+               3 │ [1, 2, 3, 4, 5, 6)
+                 │ │                └ mismatched closing delimiter (expected "]")
+                 │ └ unclosed delimiter
+                 │
+                 └─ nofile:3:18\
+             """
+    end
+
     test "two-line span" do
       output =
         capture_raise(
@@ -76,6 +98,30 @@ defmodule Kernel.DiagnosticsTest do
              """
     end
 
+    test "two-line span with offset" do
+      output =
+        capture_raise(
+          """
+          [a, b, c
+           d, f, g}
+          """,
+          MismatchedDelimiterError,
+          line: 3
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:4:9:
+                 error: unexpected token: }
+                 │
+               3 │ [a, b, c
+                 │ └ unclosed delimiter
+               4 │  d, f, g}
+                 │         └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:4:9\
+             """
+    end
+
     test "many-line span" do
       output =
         capture_raise(
@@ -103,7 +149,9 @@ defmodule Kernel.DiagnosticsTest do
                  │
                  └─ nofile:5:5\
              """
+    end
 
+    test "many-line span with offset" do
       output =
         capture_raise(
           """
@@ -111,20 +159,21 @@ defmodule Kernel.DiagnosticsTest do
             IO.inspect(2 + 2) + 2
           )
           """,
-          MismatchedDelimiterError
+          MismatchedDelimiterError,
+          line: 3
         )
 
       assert output == """
-             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:3:1:
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:5:1:
                  error: unexpected token: )
                  │
-               1 │ fn always_forget_end ->
+               3 │ fn always_forget_end ->
                  │ └ unclosed delimiter
-               2 │   IO.inspect(2 + 2) + 2
-               3 │ )
+               4 │   IO.inspect(2 + 2) + 2
+               5 │ )
                  │ └ mismatched closing delimiter (expected "end")
                  │
-                 └─ nofile:3:1\
+                 └─ nofile:5:1\
              """
     end
 
@@ -153,7 +202,7 @@ defmodule Kernel.DiagnosticsTest do
              """
     end
 
-    test "trim inbetween lines if too many" do
+    test "trim in between lines if too many" do
       output =
         capture_raise(
           """
@@ -288,18 +337,247 @@ defmodule Kernel.DiagnosticsTest do
     end
   end
 
+  describe "token missing error" do
+    test "missing parens terminator" do
+      output =
+        capture_raise(
+          """
+          my_numbers = [1, 2, 3, 4, 5, 6
+          IO.inspect(my_numbers)
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:2:23:
+                 error: missing terminator: ]
+                 │
+               1 │ my_numbers = [1, 2, 3, 4, 5, 6
+                 │              └ unclosed delimiter
+               2 │ IO.inspect(my_numbers)
+                 │                       └ missing closing delimiter (expected "]")
+                 │
+                 └─ nofile:2:23\
+             """
+    end
+
+    test "missing heredoc terminator" do
+      output =
+        capture_raise(
+          """
+          a = \"""
+          test string
+
+          IO.inspect(10 + 20)
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:4:20:
+                 error: missing terminator: \""" (for heredoc starting at line 1)
+                 │
+               1 │ a = \"""
+                 │     └ unclosed delimiter
+               2 │ test string
+               3 │\s
+               4 │ IO.inspect(10 + 20)
+                 │                    └ missing closing delimiter (expected \""")
+                 │
+                 └─ nofile:4:20\
+             """
+    end
+
+    test "missing sigil terminator" do
+      output =
+        capture_raise("~s<foobar", TokenMissingError)
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:10:
+                 error: missing terminator: > (for sigil ~s< starting at line 1)
+                 │
+               1 │ ~s<foobar
+                 │   │      └ missing closing delimiter (expected ">")
+                 │   └ unclosed delimiter
+                 │
+                 └─ nofile:1:10\
+             """
+
+      output =
+        capture_raise("~s|foobar", TokenMissingError)
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:10:
+                 error: missing terminator: | (for sigil ~s| starting at line 1)
+                 │
+               1 │ ~s|foobar
+                 │   │      └ missing closing delimiter (expected "|")
+                 │   └ unclosed delimiter
+                 │
+                 └─ nofile:1:10\
+             """
+    end
+
+    test "missing string terminator" do
+      output =
+        capture_raise("\"foobar", TokenMissingError)
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:8:
+                 error: missing terminator: " (for string starting at line 1)
+                 │
+               1 │ "foobar
+                 │ │      └ missing closing delimiter (expected ")
+                 │ └ unclosed delimiter
+                 │
+                 └─ nofile:1:8\
+             """
+    end
+
+    test "missing atom terminator" do
+      output =
+        capture_raise(":\"foobar", TokenMissingError)
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:9:
+                 error: missing terminator: " (for atom starting at line 1)
+                 │
+               1 │ :"foobar
+                 │  │      └ missing closing delimiter (expected ")
+                 │  └ unclosed delimiter
+                 │
+                 └─ nofile:1:9\
+             """
+    end
+
+    test "missing function terminator" do
+      output =
+        capture_raise("K.\"foobar", TokenMissingError)
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:10:
+                 error: missing terminator: " (for function name starting at line 1)
+                 │
+               1 │ K."foobar
+                 │   │      └ missing closing delimiter (expected ")
+                 │   └ unclosed delimiter
+                 │
+                 └─ nofile:1:10\
+             """
+    end
+
+    test "shows in between lines if EOL is not far below" do
+      output =
+        capture_raise(
+          """
+          my_numbers = [1, 2, 3, 4, 5, 6
+          my_numbers
+          |> Enum.map(&(&1 + 1))
+          |> Enum.map(&(&1 * &1))
+          |> IO.inspect()
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:5:16:
+                 error: missing terminator: ]
+                 │
+               1 │ my_numbers = [1, 2, 3, 4, 5, 6
+                 │              └ unclosed delimiter
+               2 │ my_numbers
+               3 │ |> Enum.map(&(&1 + 1))
+               4 │ |> Enum.map(&(&1 * &1))
+               5 │ |> IO.inspect()
+                 │                └ missing closing delimiter (expected "]")
+                 │
+                 └─ nofile:5:16\
+             """
+    end
+
+    test "trims lines" do
+      output =
+        capture_raise(
+          """
+          my_numbers = (1, 2, 3, 4, 5, 6
+
+
+
+
+
+
+          IO.inspect(my_numbers)
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:8:23:
+                 error: missing terminator: )
+                 │
+               1 │ my_numbers = (1, 2, 3, 4, 5, 6
+                 │              └ unclosed delimiter
+              ...
+               8 │ IO.inspect(my_numbers)
+                 │                       └ missing closing delimiter (expected ")")
+                 │
+                 └─ nofile:8:23\
+             """
+    end
+
+    test "shows the last non-empty line of a file" do
+      output =
+        capture_raise(
+          """
+          my_numbers = {1, 2, 3, 4, 5, 6
+          IO.inspect(my_numbers)
+
+
+
+
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:2:23:
+                 error: missing terminator: }
+                 │
+               1 │ my_numbers = {1, 2, 3, 4, 5, 6
+                 │              └ unclosed delimiter
+               2 │ IO.inspect(my_numbers)
+                 │                       └ missing closing delimiter (expected "}")
+                 │
+                 └─ nofile:2:23\
+             """
+    end
+
+    test "supports unicode" do
+      output =
+        capture_raise(
+          """
+          my_emojis = [1, 2, 3, 4 # ⚗️
+          IO.inspect(my_numbers)
+          """,
+          TokenMissingError
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:2:23:
+                 error: missing terminator: ]
+                 │
+               1 │ my_emojis = [1, 2, 3, 4 # ⚗️
+                 │             └ unclosed delimiter
+               2 │ IO.inspect(my_numbers)
+                 │                       └ missing closing delimiter (expected "]")
+                 │
+                 └─ nofile:2:23\
+             """
+    end
+  end
+
   describe "compile-time exceptions" do
     test "SyntaxError (snippet)" do
-      expected = """
-      ** (SyntaxError) invalid syntax found on nofile:1:17:
-          error: syntax error before: '*'
-          │
-        1 │ [1, 2, 3, 4, 5, *]
-          │                 ^
-          │
-          └─ nofile:1:17\
-      """
-
       output =
         capture_raise(
           """
@@ -308,20 +586,39 @@ defmodule Kernel.DiagnosticsTest do
           SyntaxError
         )
 
-      assert output == expected
+      assert output == """
+             ** (SyntaxError) invalid syntax found on nofile:1:17:
+                 error: syntax error before: '*'
+                 │
+               1 │ [1, 2, 3, 4, 5, *]
+                 │                 ^
+                 │
+                 └─ nofile:1:17\
+             """
+    end
+
+    test "SyntaxError (snippet) with offset" do
+      output =
+        capture_raise(
+          """
+          [1, 2, 3, 4, 5, *]
+          """,
+          SyntaxError,
+          line: 3
+        )
+
+      assert output == """
+             ** (SyntaxError) invalid syntax found on nofile:3:17:
+                 error: syntax error before: '*'
+                 │
+               3 │ [1, 2, 3, 4, 5, *]
+                 │                 ^
+                 │
+                 └─ nofile:3:17\
+             """
     end
 
     test "TokenMissingError (snippet)" do
-      expected = """
-      ** (TokenMissingError) token missing on nofile:1:4:
-          error: syntax error: expression is incomplete
-          │
-        1 │ 1 +
-          │    ^
-          │
-          └─ nofile:1:4\
-      """
-
       output =
         capture_raise(
           """
@@ -330,14 +627,49 @@ defmodule Kernel.DiagnosticsTest do
           TokenMissingError
         )
 
-      assert output == expected
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:1:4:
+                 error: syntax error: expression is incomplete
+                 │
+               1 │ 1 +
+                 │    ^
+                 │
+                 └─ nofile:1:4\
+             """
     end
 
-    test "TokenMissingError (no snippet)" do
+    test "TokenMissingError (snippet) with offset and column" do
+      output =
+        capture_raise(
+          """
+          1 +
+          """,
+          TokenMissingError,
+          line: 3,
+          column: 3
+        )
+
+      assert output == """
+             ** (TokenMissingError) token missing on nofile:3:6:
+                 error: syntax error: expression is incomplete
+                 │
+               3 │   1 +
+                 │      ^
+                 │
+                 └─ nofile:3:6\
+             """
+    end
+
+    test "TokenMissingError (unclosed delimiter)" do
       expected = """
-      ** (TokenMissingError) token missing on nofile:2:1:
-         error: missing terminator: end (for "fn" starting at line 1)
-         └─ nofile:2:1\
+      ** (TokenMissingError) token missing on nofile:1:5:
+          error: missing terminator: end
+          │
+        1 │ fn a
+          │ │   └ missing closing delimiter (expected "end")
+          │ └ unclosed delimiter
+          │
+          └─ nofile:1:5\
       """
 
       output =
@@ -419,7 +751,7 @@ defmodule Kernel.DiagnosticsTest do
           1 -
           """,
           TokenMissingError,
-          fake_stacktrace
+          stacktrace: fake_stacktrace
         )
 
       assert output == expected
@@ -448,7 +780,7 @@ defmodule Kernel.DiagnosticsTest do
           1 -
           """,
           TokenMissingError,
-          fake_stacktrace
+          stacktrace: fake_stacktrace
         )
 
       assert output == expected
@@ -521,7 +853,7 @@ defmodule Kernel.DiagnosticsTest do
           └─ #{path}:3: Sample.a/0
       """
 
-      assert capture_eval(source, false) =~ expected
+      assert capture_eval(source, columns: false) =~ expected
     after
       purge(Sample)
     end
@@ -544,101 +876,65 @@ defmodule Kernel.DiagnosticsTest do
     end
 
     @tag :tmp_dir
-    test "long message (file)", %{tmp_dir: tmp_dir} do
-      path = make_relative_tmp(tmp_dir, "long-warning.ex")
+    test "IO.warn file+line", %{tmp_dir: tmp_dir} do
+      path = make_relative_tmp(tmp_dir, "io-warn-file-line.ex")
 
       source = """
-      defmodule Sample do
-        @file "#{path}"
-
-        def atom_case do
-          v = "bc"
-
-          case v do
-            _ when is_atom(v) -> :ok
-            _ -> :fail
-          end
-        end
-      end
+      IO.warn("oops\\nmulti\\nline", file: __ENV__.file, line: __ENV__.line)
       """
 
       File.write!(path, source)
 
       expected = """
-          warning: incompatible types:
-
-              binary() !~ atom()
-
-          in expression:
-
-              # #{path}:8
-              is_atom(v)
-
-          where "v" was given the type binary() in:
-
-              # #{path}:5
-              v = "bc"
-
-          where "v" was given the type atom() in:
-
-              # #{path}:8
-              is_atom(v)
-
-          Conflict found at
+          warning: oops
+          multi
+          line
           │
-        8 │       _ when is_atom(v) -> :ok
-          │              ~
+        1 │ IO.warn("oops\\nmulti\\nline", file: __ENV__.file, line: __ENV__.line)
+          │ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           │
-          └─ #{path}:8:14: Sample.atom_case/0
+          └─ tmp\
       """
 
-      assert capture_eval(source) =~ expected
-    after
-      purge(Sample)
+      assert capture_io(:stderr, fn -> Code.eval_file(path) end) =~ expected
     end
 
-    test "long message (nofile)" do
-      source = """
-      defmodule Sample do
-        def atom_case do
-          v = "bc"
+    @tag :tmp_dir
+    test "IO.warn file+line+column", %{tmp_dir: tmp_dir} do
+      path = make_relative_tmp(tmp_dir, "io-warn-file-line-column.ex")
 
-          case v do
-            _ when is_atom(v) -> :ok
-            _ -> :fail
-          end
-        end
-      end
+      source = """
+      IO.warn("oops\\nmulti\\nline", file: __ENV__.file, line: __ENV__.line, column: 4)
       """
 
+      File.write!(path, source)
+
       expected = """
-      warning: incompatible types:
-
-          binary() !~ atom()
-
-      in expression:
-
-          # nofile:6
-          is_atom(v)
-
-      where "v" was given the type binary() in:
-
-          # nofile:3
-          v = "bc"
-
-      where "v" was given the type atom() in:
-
-          # nofile:6
-          is_atom(v)
-
-      Conflict found at
-      └─ nofile:6:14: Sample.atom_case/0
-
+          warning: oops
+          multi
+          line
+          │
+        1 │ IO.warn("oops\\nmulti\\nline", file: __ENV__.file, line: __ENV__.line, column: 4)
+          │    ~
+          │
+          └─ tmp\
       """
 
-      assert capture_eval(source) =~ expected
-    after
-      purge(Sample)
+      assert capture_io(:stderr, fn -> Code.eval_file(path) end) =~ expected
+    end
+
+    test "IO.warn with missing data" do
+      assert capture_eval("""
+             IO.warn("oops-bad", file: #{inspect(__ENV__.file)}, line: 3, column: nil)
+             """) =~ "warning: oops-bad"
+
+      assert capture_eval("""
+             IO.warn("oops-bad", file: #{inspect(__ENV__.file)}, line: nil)
+             """) =~ "oops-bad"
+
+      assert capture_eval("""
+             IO.warn("oops-bad", file: nil)
+             """) =~ "oops-bad"
     end
 
     @tag :tmp_dir
@@ -791,7 +1087,7 @@ defmodule Kernel.DiagnosticsTest do
 
       """
 
-      assert capture_eval(source, false) == expected
+      assert capture_eval(source, columns: false) == expected
     after
       purge(Sample)
     end
@@ -822,7 +1118,7 @@ defmodule Kernel.DiagnosticsTest do
 
       """
 
-      assert capture_compile(source, false) == expected
+      assert capture_compile(source, columns: false) == expected
     after
       purge(Sample)
     end
@@ -995,7 +1291,7 @@ defmodule Kernel.DiagnosticsTest do
 
       """
 
-      assert capture_eval(source, false) == expected
+      assert capture_eval(source, columns: false) == expected
     after
       purge(Sample)
     end
@@ -1148,30 +1444,32 @@ defmodule Kernel.DiagnosticsTest do
     |> Path.relative_to_cwd()
   end
 
-  defp capture_eval(source, columns? \\ true) do
+  defp capture_eval(source, opts \\ [columns: true]) do
     capture_io(:stderr, fn ->
-      quoted = Code.string_to_quoted!(source, columns: columns?)
+      quoted = Code.string_to_quoted!(source, opts)
       Code.eval_quoted(quoted)
     end)
   end
 
-  defp capture_compile(source, columns? \\ true) do
+  defp capture_compile(source, opts \\ [columns: true]) do
     capture_io(:stderr, fn ->
       assert_raise CompileError, fn ->
-        ast = Code.string_to_quoted!(source, columns: columns?)
+        ast = Code.string_to_quoted!(source, opts)
         Code.eval_quoted(ast)
       end
     end)
   end
 
-  defp capture_raise(source, exception, mock_stacktrace \\ []) do
+  defp capture_raise(source, exception, opts \\ []) do
+    {stacktrace, opts} = Keyword.pop(opts, :stacktrace, [])
+
     e =
       assert_raise exception, fn ->
-        ast = Code.string_to_quoted!(source, columns: true)
+        ast = Code.string_to_quoted!(source, [columns: true] ++ opts)
         Code.eval_quoted(ast)
       end
 
-    Exception.format(:error, e, mock_stacktrace)
+    Exception.format(:error, e, stacktrace)
   end
 
   defp purge(module) when is_atom(module) do

lib/elixir/test/elixir/kernel/errors_test.exs

@@ -473,7 +473,7 @@ defmodule Kernel.ErrorsTest do
       assert_eval_raise TokenMissingError,
                         [
                           "nofile:1:5:",
-                          ~r/missing terminator: end \(for "fn" starting at line 1\)/
+                          ~r/missing terminator: end/
                         ],
                         ~c"fn 1"
 
@@ -490,16 +490,16 @@ defmodule Kernel.ErrorsTest do
     assert_eval_raise TokenMissingError,
                       [
                         "nofile:1:25:",
-                        "missing terminator: end (for \"do\" starting at line 1)",
+                        "missing terminator: end",
                         "defmodule ShowSnippet do\n",
-                        "^"
+                        "└ unclosed delimiter"
                       ],
                       ~c"defmodule ShowSnippet do"
   end
 
   test "don't show snippet when error line is empty" do
     assert_eval_raise TokenMissingError,
-                      ["nofile:3:1:", "missing terminator: end (for \"do\" starting at line 1)"],
+                      ["nofile:1:25:", "missing terminator: end"],
                       ~c"defmodule ShowSnippet do\n\n"
   end
 

lib/elixir/test/elixir/kernel/parallel_compiler_test.exs

@@ -147,7 +147,7 @@ defmodule Kernel.ParallelCompilerTest do
       expected_msg = "Undef.__struct__/1 is undefined, cannot expand struct Undef"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^fixture, 3, msg}, {^fixture, 0, compile_msg}], []} =
+               assert {:error, [{^fixture, {3, 5}, msg}, {^fixture, 0, compile_msg}], []} =
                         Kernel.ParallelCompiler.compile([fixture])
 
                assert msg =~ expected_msg
@@ -216,7 +216,7 @@ defmodule Kernel.ParallelCompilerTest do
         "ThisModuleWillNeverBeAvailable.__struct__/1 is undefined, cannot expand struct ThisModuleWillNeverBeAvailable"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^fixture, 7, msg}, {^fixture, 0, compile_msg}], []} =
+               assert {:error, [{^fixture, {7, 3}, msg}, {^fixture, 0, compile_msg}], []} =
                         Kernel.ParallelCompiler.compile([fixture])
 
                assert msg =~ expected_msg
@@ -245,7 +245,9 @@ defmodule Kernel.ParallelCompilerTest do
         "ThisModuleWillNeverBeAvailable.__struct__/1 is undefined, cannot expand struct ThisModuleWillNeverBeAvailable"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^missing_struct, 2, msg}, {^missing_struct, 0, compile_msg}], []} =
+               assert {:error,
+                       [{^missing_struct, {2, 3}, msg}, {^missing_struct, 0, compile_msg}],
+                       []} =
                         Kernel.ParallelCompiler.compile([missing_struct, depends_on])
 
                assert msg =~ expected_msg
@@ -272,7 +274,9 @@ defmodule Kernel.ParallelCompilerTest do
       expected_msg = "module Unknown.Module is not loaded and could not be found"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^missing_import, 2, msg}, {^missing_import, 0, compile_msg}], []} =
+               assert {:error,
+                       [{^missing_import, {2, 3}, msg}, {^missing_import, 0, compile_msg}],
+                       []} =
                         Kernel.ParallelCompiler.compile([missing_import, depends_on])
 
                assert msg =~ expected_msg
@@ -476,6 +480,24 @@ defmodule Kernel.ParallelCompilerTest do
       end)
     end
 
+    test "gets correct file+line+column number for SyntaxError" do
+      File.mkdir_p!(tmp_path())
+
+      [fixture] =
+        write_tmp("error",
+          error: """
+          raise SyntaxError, file: "foo/bar.ex", line: 3, column: 10
+          """
+        )
+
+      file = Path.absname("foo/bar.ex")
+
+      capture_io(:stderr, fn ->
+        assert {:error, [%{file: ^file, source: ^fixture, position: {3, 10}}], _} =
+                 Kernel.ParallelCompiler.compile([fixture], return_diagnostics: true)
+      end)
+    end
+
     test "gets proper beam destinations from dynamic modules" do
       fixtures =
         write_tmp(
@@ -509,7 +531,7 @@ defmodule Kernel.ParallelCompilerTest do
       expected_msg = "Undef.__struct__/1 is undefined, cannot expand struct Undef"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^fixture, 3, msg}, {^fixture, 0, compile_msg}], []} =
+               assert {:error, [{^fixture, {3, 5}, msg}, {^fixture, 0, compile_msg}], []} =
                         Kernel.ParallelCompiler.require([fixture])
 
                assert msg =~ expected_msg
@@ -537,7 +559,7 @@ defmodule Kernel.ParallelCompilerTest do
         "ThisModuleWillNeverBeAvailable.__struct__/1 is undefined, cannot expand struct ThisModuleWillNeverBeAvailable"
 
       assert capture_io(:stderr, fn ->
-               assert {:error, [{^fixture, 7, msg}, {^fixture, 0, compile_msg}], []} =
+               assert {:error, [{^fixture, {7, 3}, msg}, {^fixture, 0, compile_msg}], []} =
                         Kernel.ParallelCompiler.require([fixture])
 
                assert msg =~ expected_msg

lib/elixir/test/elixir/kernel/parser_test.exs

@@ -491,83 +491,6 @@ defmodule Kernel.ParserTest do
     end
   end
 
-  describe "token missing errors" do
-    test "missing paren" do
-      assert_token_missing(
-        ["nofile:1:9:", "missing terminator: ) (for \"(\" starting at line 1)"],
-        ~c"case 1 ("
-      )
-    end
-
-    test "dot terminator" do
-      assert_token_missing(
-        ["nofile:1:9:", "missing terminator: \" (for function name starting at line 1)"],
-        ~c"foo.\"bar"
-      )
-    end
-
-    test "sigil terminator" do
-      assert_token_missing(
-        ["nofile:3:1:", "missing terminator: \" (for sigil ~r\" starting at line 1)"],
-        ~c"~r\"foo\n\n"
-      )
-
-      assert_token_missing(
-        ["nofile:3:1:", "missing terminator: } (for sigil ~r{ starting at line 1)"],
-        ~c"~r{foo\n\n"
-      )
-    end
-
-    test "string terminator" do
-      assert_token_missing(
-        ["nofile:1:5:", "missing terminator: \" (for string starting at line 1)"],
-        ~c"\"bar"
-      )
-    end
-
-    test "heredoc with incomplete interpolation" do
-      assert_token_missing(
-        [
-          "nofile:2:1:",
-          ~s/missing interpolation terminator: "}" (for heredoc starting at line 1)/
-        ],
-        ~c"\"\"\"\n\#{\n"
-      )
-    end
-
-    test "heredoc terminator" do
-      assert_token_missing(
-        ["nofile:2:4:", ~s/missing terminator: """ (for heredoc starting at line 1)/],
-        ~c"\"\"\"\nbar"
-      )
-
-      assert_token_missing(
-        ["nofile:2:7:", ~s/missing terminator: """ (for heredoc starting at line 1)/],
-        ~c"\"\"\"\nbar\"\"\""
-      )
-    end
-
-    test "missing end" do
-      assert_token_missing(
-        ["nofile:1:9:", "missing terminator: end \(for \"do\" starting at line 1\)"],
-        ~c"foo do 1"
-      )
-
-      assert_token_missing(
-        ["hint:", ~s/it looks like the "do" on line 2 does not have a matching "end"/],
-        ~c"""
-        defmodule MyApp do
-          def one do
-          # end
-
-          def two do
-          end
-        end
-        """
-      )
-    end
-  end
-
   describe "syntax errors" do
     test "invalid heredoc start" do
       assert_syntax_error(
@@ -1136,18 +1059,18 @@ defmodule Kernel.ParserTest do
 
       assert_syntax_error(
         ["atom length must be less than system limit: "],
-        ~s[:"#{atom}"]
+        ~s{:"#{atom}"}
+      )
+
+      assert_syntax_error(
+        ["atom length must be less than system limit: "],
+        ~s{["#{atom}": 123]}
       )
     end
   end
 
   defp parse!(string), do: Code.string_to_quoted!(string)
 
-  defp assert_token_missing(given_messages, string) do
-    e = assert_raise TokenMissingError, fn -> parse!(string) end
-    assert_exception_msg(e, given_messages)
-  end
-
   defp assert_syntax_error(given_messages, source) do
     e = assert_raise SyntaxError, fn -> parse!(source) end
     assert_exception_msg(e, given_messages)

lib/elixir/test/elixir/kernel/tracers_test.exs

@@ -16,11 +16,9 @@ defmodule Kernel.TracersTest do
 
   setup_all do
     Code.put_compiler_option(:tracers, [__MODULE__])
-    Code.put_compiler_option(:parser_options, columns: true)
 
     on_exit(fn ->
       Code.put_compiler_option(:tracers, [])
-      Code.put_compiler_option(:parser_options, [])
     end)
   end
 

lib/elixir/test/elixir/kernel/warning_test.exs

@@ -1175,18 +1175,6 @@ defmodule Kernel.WarningTest do
     purge(Sample)
   end
 
-  test "parens on nullary remote call" do
-    assert_warn_eval(
-      [
-        "nofile:1:8",
-        "parentheses are required for function calls with no arguments, got: System.version"
-      ],
-      "System.version"
-    )
-  after
-    purge(Sample)
-  end
-
   test "parens with module attribute" do
     assert_warn_eval(
       [
@@ -1652,22 +1640,39 @@ defmodule Kernel.WarningTest do
   end
 
   test "reserved doc metadata keys" do
-    output =
-      capture_eval("""
-      defmodule Sample do
-        @typedoc opaque: false
-        @type t :: binary
+    {output, diagnostics} =
+      Code.with_diagnostics([log: true], fn ->
+        capture_eval("""
+        defmodule Sample do
+          @typedoc opaque: false
+          @type t :: binary
 
-        @doc defaults: 3, since: "1.2.3"
-        def foo(a), do: a
-      end
-      """)
+          @doc defaults: 3, since: "1.2.3"
+          def foo(a), do: a
+        end
+        """)
+      end)
 
     assert output =~ "ignoring reserved documentation metadata key: :opaque"
     assert output =~ "nofile:2: "
     assert output =~ "ignoring reserved documentation metadata key: :defaults"
     assert output =~ "nofile:5: "
     refute output =~ ":since"
+
+    assert [
+             %{
+               message: "ignoring reserved documentation metadata key: :opaque",
+               position: 2,
+               file: "nofile",
+               severity: :warning
+             },
+             %{
+               message: "ignoring reserved documentation metadata key: :defaults",
+               position: 5,
+               file: "nofile",
+               severity: :warning
+             }
+           ] = diagnostics
   after
     purge(Sample)
   end

lib/elixir/test/elixir/module/types/integration_test.exs

@@ -695,6 +695,29 @@ defmodule Module.Types.IntegrationTest do
 
       assert_warnings(files, warnings)
     end
+
+    test "reports unquote functions" do
+      files = %{
+        "a.ex" => """
+        defmodule A do
+          @deprecated "oops"
+          def a, do: :ok
+        end
+        """,
+        "b.ex" => """
+        defmodule B do
+          def b, do: unquote(&A.a/0)
+        end
+        """
+      }
+
+      warnings = [
+        "A.a/0 is deprecated. oops",
+        "b.ex: B.b/0"
+      ]
+
+      assert_warnings(files, warnings)
+    end
   end
 
   defp assert_warnings(files, expected) when is_binary(expected) do

lib/elixir/test/elixir/protocol_test.exs

@@ -154,8 +154,12 @@ defmodule ProtocolTest do
   end
 
   test "protocol defines callbacks" do
-    assert [{:type, 13, :fun, args}] = get_callbacks(@sample_binary, :ok, 1)
-    assert args == [{:type, 13, :product, [{:user_type, 13, :t, []}]}, {:type, 13, :boolean, []}]
+    assert [{:type, {13, 19}, :fun, args}] = get_callbacks(@sample_binary, :ok, 1)
+
+    assert args == [
+             {:type, {13, 19}, :product, [{:user_type, {13, 16}, :t, []}]},
+             {:type, {13, 22}, :boolean, []}
+           ]
 
     assert [{:type, 23, :fun, args}] = get_callbacks(@with_any_binary, :ok, 1)
     assert args == [{:type, 23, :product, [{:user_type, 23, :t, []}]}, {:type, 23, :term, []}]

lib/elixir/test/elixir/string_test.exs

@@ -826,6 +826,33 @@ defmodule StringTest do
     refute String.valid?("asdasdasd" <> <<0xFFFF::16>>, :fast_ascii)
   end
 
+  test "replace_invalid" do
+    assert String.replace_invalid("") === ""
+    assert String.replace_invalid(<<0xFF>>) === "�"
+    assert String.replace_invalid(<<0xFF, 0xFF, 0xFF>>) === "���"
+
+    # Valid ASCII
+    assert String.replace_invalid("hello") === "hello"
+
+    # Valid UTF-8
+    assert String.replace_invalid("こんにちは") === "こんにちは"
+
+    # 2/3 byte truncated "ề"
+    assert String.replace_invalid(<<225, 187>>) === "�"
+    assert String.replace_invalid("nem rán b" <> <<225, 187>> <> " bề") === "nem rán b� bề"
+
+    # 2/4 byte truncated "😔"
+    assert String.replace_invalid(<<240, 159>>) === "�"
+    assert String.replace_invalid("It's so over " <> <<240, 159>>) === "It's so over �"
+
+    # 3/4 byte truncated "😃"
+    assert String.replace_invalid(<<240, 159, 152>>) === "�"
+    assert String.replace_invalid("We're so back " <> <<240, 159, 152>>) === "We're so back �"
+
+    # 3 byte overlong "e"
+    assert String.replace_invalid(<<0b11100000, 0b10000001, 0b10100101>>) === "���"
+  end
+
   test "chunk/2 with :valid trait" do
     assert String.chunk("", :valid) == []
 

lib/elixir/test/elixir/typespec_test.exs

@@ -1582,23 +1582,29 @@ defmodule TypespecTest do
     """)
 
     [type: type] = types(:typespec_test_mod)
-    line = 5
 
     assert Code.Typespec.type_to_quoted(type) ==
-             {:"::", [], [{:t, [], [{:x, [line: line], nil}]}, [{:x, [line: line], nil}]]}
+             {:"::", [],
+              [
+                {:t, [], [{:x, meta(5, 9), nil}]},
+                [{:x, meta(5, 20), nil}]
+              ]}
 
     [{{:f, 1}, [spec]}] = specs(:typespec_test_mod)
-    line = 7
 
     assert Code.Typespec.spec_to_quoted(:f, spec) ==
-             {:when, [line: line],
+             {:when, meta(7, 8),
               [
-                {:"::", [line: line],
-                 [{:f, [line: line], [{:x, [line: line], nil}]}, {:x, [line: line], nil}]},
-                [x: {:var, [line: line], nil}]
+                {:"::", meta(7, 8),
+                 [{:f, meta(7, 8), [{:x, meta(7, 9), nil}]}, {:x, meta(7, 15), nil}]},
+                [x: {:var, meta(7, 8), nil}]
               ]}
   end
 
+  defp meta(line, column) do
+    [line: line, column: column]
+  end
+
   defp erlc(context, module, code) do
     dir = context.tmp_dir
 

lib/ex_unit/lib/ex_unit.ex

@@ -148,6 +148,16 @@ defmodule ExUnit do
   end
 
   defmodule TimeoutError do
+    @moduledoc """
+    Exception raised when a test times out.
+    """
+
+    @typedoc since: "1.16.0"
+    @type t :: %__MODULE__{
+            timeout: non_neg_integer,
+            type: String.t()
+          }
+
     defexception [:timeout, :type]
 
     @impl true

lib/ex_unit/lib/ex_unit/assertions.ex

@@ -1,10 +1,23 @@
 defmodule ExUnit.AssertionError do
   @moduledoc """
   Raised to signal an assertion error.
+
+  This is used by macros such as `ExUnit.Assertions.assert/1`.
   """
 
   @no_value :ex_unit_no_meaningful_value
 
+  @typedoc since: "1.16.0"
+  @type t :: %__MODULE__{
+          left: any,
+          right: any,
+          message: any,
+          expr: any,
+          args: any,
+          doctest: any,
+          context: any
+        }
+
   defexception left: @no_value,
                right: @no_value,
                message: @no_value,
@@ -16,6 +29,7 @@ defmodule ExUnit.AssertionError do
   @doc """
   Indicates no meaningful value for a field.
   """
+  @spec no_value :: atom
   def no_value do
     @no_value
   end
@@ -31,6 +45,11 @@ defmodule ExUnit.MultiError do
   Raised to signal multiple errors happened in a test case.
   """
 
+  @typedoc since: "1.16.0"
+  @type t :: %__MODULE__{
+          errors: [{Exception.kind(), any, Exception.stacktrace()}]
+        }
+
   defexception errors: []
 
   @impl true

lib/ex_unit/lib/ex_unit/case.ex

@@ -1,8 +1,23 @@
 defmodule ExUnit.DuplicateTestError do
+  @moduledoc """
+  Exception raised to indicate two or more tests with the same name.
+  """
+
+  @typedoc since: "1.16.0"
+  @type t :: %__MODULE__{message: String.t()}
+
   defexception [:message]
 end
 
 defmodule ExUnit.DuplicateDescribeError do
+  @moduledoc """
+  Exception raised to indicate two or more `describe` blocks with
+  the same name.
+  """
+
+  @typedoc since: "1.16.0"
+  @type t :: %__MODULE__{message: String.t()}
+
   defexception [:message]
 end
 

lib/ex_unit/lib/ex_unit/doc_test.ex

@@ -152,6 +152,12 @@ defmodule ExUnit.DocTest do
   @opaque_type_regex ~r/#[\w\.]+</
 
   defmodule Error do
+    @moduledoc """
+    Exception raised when there's an error with the syntax or semantics of a doctest.
+    """
+
+    @typedoc since: "1.16.0"
+    @type t :: %__MODULE__{message: String.t()}
     defexception [:message]
 
     @impl true

lib/ex_unit/lib/ex_unit/formatter.ex

@@ -67,6 +67,97 @@ defmodule ExUnit.Formatter do
           load: pos_integer | nil
         }
 
+  @typedoc """
+  Key passed to a formatter callback to format a diff.
+
+  See `t:formatter_callback/0`.
+  """
+  @typedoc since: "1.16.0"
+  @type formatter_callback_diff_key ::
+          :diff_delete
+          | :diff_delete_whitespace
+          | :diff_insert
+          | :diff_insert_whitespace
+
+  @typedoc """
+  Key passed to a formatter callback to format information.
+
+  See `t:formatter_callback/0`.
+  """
+  @typedoc since: "1.16.0"
+  @type formatter_callback_info_key ::
+          :extra_info
+          | :error_info
+          | :test_module_info
+          | :test_info
+          | :location_info
+          | :stacktrace_info
+          | :blame_diff
+
+  @typedoc """
+  A function that this module calls to format various things.
+
+  You can pass this functions to various functions in this module, and use it
+  to customize the formatting of the output. For example, ExUnit's CLI formatter
+  uses this callback to colorize output.
+
+  ## Keys
+
+  The possible keys are:
+
+    * `:diff_enabled?` - whether diffing is enabled. It receives a boolean
+      indicating whether diffing is enabled by default and returns a boolean
+      indicating whether diffing should be enabled for the current test.
+
+    * `:diff_delete` and `:diff_delete_whitespace` - Should format a diff deletion,
+      with or without whitespace respectively.
+
+    * `:diff_insert` and `:diff_insert_whitespace` - Should format a diff insertion,
+      with or without whitespace respectively.
+
+    * `:extra_info` - Should format extra information, such as the `"code: "` label
+      that precedes code to show.
+
+    * `:error_info` - Should format error information.
+
+    * `:error_info` - Should format error information.
+
+    * `:test_module_info` - Should format test module information. The message returned
+    when this key is passed precedes messages such as `"failure on setup_all callback [...]"`.
+
+    * `:test_info` - Should format test information.
+
+    * `:location_info` - Should format test location information.
+
+    * `:stacktrace_info` - Should format stacktrace information.
+
+    * `:blame_diff` - Should format a string of code.
+
+  ## Examples
+
+  For example, to format errors as *red strings* and everything else as is, you could define
+  a formatter callback function like this:
+
+      formatter_callback = fn
+        :error_info, msg -> [:red, msg, :reset] |> IO.ANSI.format() |> IO.iodata_to_binary()
+        _key, value -> value
+      end
+
+  """
+  @typedoc since: "1.16.0"
+  @type formatter_callback ::
+          (:diff_enabled?, boolean -> boolean)
+          | (formatter_callback_diff_key, Inspect.Algebra.t() -> Inspect.Algebra.t())
+          | (formatter_callback_info_key, String.t() -> String.t())
+
+  @typedoc """
+  Width for formatting.
+
+  For example, see `format_assertion_diff/4`.
+  """
+  @typedoc since: "1.16.0"
+  @type width :: non_neg_integer | :infinity
+
   import Exception, only: [format_stacktrace_entry: 1, format_file_line: 3]
 
   alias ExUnit.Diff
@@ -152,9 +243,26 @@ defmodule ExUnit.Formatter do
     end
   end
 
-  @doc """
-  Receives a test and formats its failure.
+  @doc ~S"""
+  Receives a test and formats its failures.
+
+  ## Examples
+
+      iex> failure = {:error, catch_error(raise "oops"), _stacktrace = []}
+      iex> formatter_cb = fn _key, value -> value end
+      iex> test = %ExUnit.Test{name: :"it works", module: MyTest, tags: %{file: "file.ex", line: 7}}
+      iex> format_test_failure(test, [failure], 1, 80, formatter_cb)
+      "  1) it works (MyTest)\n     file.ex:7\n     ** (RuntimeError) oops\n"
+
   """
+  @spec format_test_failure(
+          test,
+          [failure],
+          non_neg_integer,
+          width,
+          formatter_callback
+        ) :: String.t()
+        when failure: {atom, term, Exception.stacktrace()}
   def format_test_failure(test, failures, counter, width, formatter) do
     %ExUnit.Test{name: name, module: module, tags: tags} = test
 
@@ -174,9 +282,26 @@ defmodule ExUnit.Formatter do
     format_test_all_failure(test_case, failures, counter, width, formatter)
   end
 
-  @doc """
+  @doc ~S"""
   Receives a test module and formats its failure.
+
+  ## Examples
+
+      iex> failure = {:error, catch_error(raise "oops"), _stacktrace = []}
+      iex> formatter_cb = fn _key, value -> value end
+      iex> test_module = %ExUnit.TestModule{name: Hello}
+      iex> format_test_all_failure(test_module, [failure], 1, 80, formatter_cb)
+      "  1) Hello: failure on setup_all callback, all tests have been invalidated\n     ** (RuntimeError) oops\n"
+
   """
+  @spec format_test_all_failure(
+          ExUnit.TestModule.t(),
+          [failure],
+          non_neg_integer,
+          width,
+          formatter_callback
+        ) :: String.t()
+        when failure: {atom, term, Exception.stacktrace()}
   def format_test_all_failure(test_module, failures, counter, width, formatter) do
     name = test_module.name
 
@@ -279,7 +404,25 @@ defmodule ExUnit.Formatter do
   It expects the assertion error, the `padding_size`
   for formatted content, the width (may be `:infinity`),
   and the formatter callback function.
+
+  ## Examples
+
+      iex> error = assert_raise ExUnit.AssertionError, fn -> assert [1, 2] == [1, 3] end
+      iex> formatter_cb = fn
+      ...>   :diff_enabled?, _ -> true
+      ...>   _key, value -> value
+      ...> end
+      iex> keyword = format_assertion_diff(error, 5, 80, formatter_cb)
+      iex> for {key, val} <- keyword, do: {key, IO.iodata_to_binary(val)}
+      [left: "[1, 2]", right: "[1, 3]"]
+
   """
+  @spec format_assertion_diff(
+          ExUnit.AssertionError.t(),
+          non_neg_integer,
+          width,
+          formatter_callback
+        ) :: keyword
   def format_assertion_diff(assert_error, padding_size, width, formatter)
 
   def format_assertion_diff(%ExUnit.AssertionError{context: {:mailbox, _pins, []}}, _, _, _) do

lib/ex_unit/test/ex_unit/doc_test_test.exs

@@ -790,15 +790,16 @@ defmodule ExUnit.DocTestTest do
     line = starting_line + 35
 
     assert output =~ """
-             6) doctest ExUnit.DocTestTest.Invalid.misplaced_opaque_type/0 (6) (ExUnit.DocTestTest.InvalidCompiled)
+           6) doctest ExUnit.DocTestTest.Invalid.misplaced_opaque_type/0 (6) (ExUnit.DocTestTest.InvalidCompiled)
                 test/ex_unit/doc_test_test.exs:#{doctest_line}
-                Doctest did not compile, got: (TokenMissingError) token missing on test/ex_unit/doc_test_test.exs:#{line}:7:
-                     error: missing terminator: } (for "{" starting at line #{line})
+                Doctest did not compile, got: (TokenMissingError) token missing on test/ex_unit/doc_test_test.exs:#{line}:20:
+                     error: missing terminator: }
                      │
-                 #{line} │ {:ok, #Inspect<[]>}
-                     │       ^
+                 228 │ {:ok, #Inspect<[]>}
+                     │ │                  └ missing closing delimiter (expected "}")
+                     │ └ unclosed delimiter
                      │
-                     └─ test/ex_unit/doc_test_test.exs:#{line}:7
+                     └─ test/ex_unit/doc_test_test.exs:#{line}:20
                 If you are planning to assert on the result of an iex> expression which contains a value inspected as #Name<...>, please make sure the inspected value is placed at the beginning of the expression; otherwise Elixir will treat it as a comment due to the leading sign #.
                 doctest:
                   iex> {:ok, :oops}

lib/iex/lib/iex/helpers.ex

@@ -1476,7 +1476,8 @@ defmodule IEx.Helpers do
 
   defp get_beam_and_path(module) do
     with {^module, beam, filename} <- :code.get_object_code(module),
-         {:ok, ^module} <- beam |> :beam_lib.info() |> Keyword.fetch(:module) do
+         info_pairs when is_list(info_pairs) <- :beam_lib.info(beam),
+         {:ok, ^module} <- Keyword.fetch(info_pairs, :module) do
       {beam, filename}
     else
       _ -> :error

lib/iex/lib/iex/info.ex

@@ -89,7 +89,7 @@ defimpl IEx.Info, for: Atom do
 
       {^atom, beam, _path} ->
         info = :beam_lib.info(beam)
-        Keyword.fetch(info, :module) == {:ok, atom}
+        is_list(info) and Keyword.fetch(info, :module) == {:ok, atom}
     end
   end
 

lib/iex/lib/iex/pry.ex

@@ -519,7 +519,7 @@ defmodule IEx.Pry do
     next_binding = binding(expr, binding)
     {min_line, max_line} = line_range(expr, line)
 
-    if force? or (min_line > line and min_line != :infinity) do
+    if force? or min_line > line do
       pry_var = next_var(version)
       pry_binding = Map.to_list(binding)
       pry_opts = [line: min_line] ++ opts
@@ -545,7 +545,9 @@ defmodule IEx.Pry do
   end
 
   defp line_range(ast, line) do
-    {_, min_max} =
+    # We want min_line to start from infinity because
+    # if it starts from line it will always just return line.
+    {_, {min, max}} =
       Macro.prewalk(ast, {:infinity, line}, fn
         {_, meta, _} = ast, {min_line, max_line} when is_list(meta) ->
           line = meta[:line]
@@ -560,7 +562,7 @@ defmodule IEx.Pry do
           {ast, acc}
       end)
 
-    min_max
+    if min == :infinity, do: {line, max}, else: {min, max}
   end
 
   defp binding(ast, binding) do

lib/logger/lib/logger.ex

@@ -13,16 +13,16 @@ defmodule Logger do
 
     * Supports both message-based and structural logging.
 
+    * Integrate with Erlang's [`:logger`](`:logger`) and
+      support custom filters and handlers.
+
     * Formats and truncates messages on the client
-      to avoid clogging `Logger` backends.
+      to avoid clogging `Logger` handlers.
 
     * Alternates between sync and async modes to remain
       performant when required but also apply back-pressure
       when under stress.
 
-    * Support for custom filters and handlers as provided by
-      Erlang's `:logger`.
-
     * Allows overriding the logging level for a specific module,
       application or process.
 
@@ -65,7 +65,7 @@ defmodule Logger do
 
   For example, `:info` takes precedence over `:debug`. If your log
   level is set to `:info`, then all `:info`, `:notice` and above will
-  be passed to backends. If your log level is set to `:alert`, only
+  be passed to handlers. If your log level is set to `:alert`, only
   `:alert` and `:emergency` will be printed.
 
   ## Message
@@ -126,8 +126,8 @@ defmodule Logger do
     * `:crash_reason` - a two-element tuple with the throw/error/exit reason
       as first argument and the stacktrace as second. A throw will always be
       `{:nocatch, term}`. An error is always an `Exception` struct. All other
-      entries are exits. The console backend ignores this metadata by default
-      but it can be useful to other backends, such as the ones that report
+      entries are exits. The default formatter ignores this metadata by default
+      but it can be useful to certain handlers, such as the ones that report
       errors to third-party services
 
   There are two special metadata keys, `:module` and `:function`, which
@@ -275,8 +275,8 @@ defmodule Logger do
       Remember that if you want to purge log calls from a dependency, the
       dependency must be recompiled.
 
-  For example, to configure the `:backends` and purge all calls that happen
-  at compile time with level lower than `:info` in a `config/config.exs` file:
+  For example, to purge all calls that happen at compile time with level
+  lower than `:info` in a `config/config.exs` file:
 
       config :logger,
         compile_time_purge_matching: [
@@ -300,7 +300,7 @@ defmodule Logger do
 
     * `:level` - the logging level. Attempting to log any message
       with severity less than the configured level will simply
-      cause the message to be ignored. Keep in mind that each backend
+      cause the message to be ignored. Keep in mind that each handler
       may have its specific level, too. In addition to levels mentioned
       above it also supports 2 "meta-levels":
 
@@ -397,7 +397,7 @@ defmodule Logger do
   Prior to Elixir v1.15, custom logging could be achieved with Logger
   backends. The main API for writing Logger backends have been moved to
   the [`:logger_backends`](https://github.com/elixir-lang/logger_backends)
-  project. However, the backends are still part of Elixir for backwards
+  project. However, the backends API are still part of Elixir for backwards
   compatibility.
 
   Important remarks:
@@ -428,9 +428,12 @@ defmodule Logger do
       Backends, you can still set `backends: [Logger.Backends.Console]` and place
       the configuration under `config :logger, Logger.Backends.Console`. Although
       consider using the [`:logger_backends`](https://github.com/elixir-lang/logger_backends)
-      project in such case, as `Logger.Backends.Console` itself will be deprecated
+      project in such cases, as `Logger.Backends.Console` itself will be deprecated
       in future releases
 
+    * `Logger.Backends` only receive `:debug`, `:info`, `:warning`, and `:error`
+      messages. `:notice` maps to `:info`. `:warn` amps to `:warnings`.
+      All others map to `:error`
   """
 
   @type level ::
@@ -442,6 +445,13 @@ defmodule Logger do
   @levels [:error, :info, :debug] ++ @new_erlang_levels
   @metadata :logger_level
 
+  @doc ~S"""
+  Returns all the available levels.
+  """
+  @doc since: "1.16.0"
+  @spec levels() :: [level(), ...]
+  def levels(), do: @levels
+
   @doc ~S"""
   Returns the default formatter used by Logger.
 
@@ -933,37 +943,21 @@ defmodule Logger do
 
   defp add_elixir_domain(metadata), do: Map.put(metadata, :domain, [:elixir])
 
-  translations = %{
-    emergency: :error,
-    alert: :error,
-    critical: :error,
-    notice: :info
-  }
-
   for level <- @levels do
     report = [something: :reported, this: level]
-
-    extra =
-      if translation = translations[level] do
-        """
-
-
-        This is reported as \"#{translation}\" in Elixir's
-        logger backends for backwards compatibility reasons.
-
-        """
-      end
+    metadata = [user_id: 42, request_id: "xU32kFa"]
+    article = if level in [:info, :error, :alert, :emergency], do: "an", else: "a"
 
     @doc """
-    Logs a #{level} message.
+    Logs #{article} #{level} message.
 
-    Returns `:ok`.#{extra}
+    Returns `:ok`.
 
     ## Examples
 
     Logging a message (string or iodata):
 
-        Logger.#{level}("this is a #{level} message")
+        Logger.#{level}("this is #{article} #{level} message")
 
     Report message (maps or keywords):
 
@@ -973,6 +967,14 @@ defmodule Logger do
         # as map
         Logger.#{level}(#{inspect(Map.new(report))})
 
+    Report message with metadata (maps or keywords):
+
+        # as a keyword list
+        Logger.#{level}("this is #{article} #{level} message", #{inspect(metadata)})
+
+        # as map
+        Logger.#{level}("this is #{article} #{level} message", #{inspect(Map.new(metadata))})
+
     """
 
     # Only macros generated for the "new" Erlang levels are available since 1.11.0. Other

lib/logger/test/logger_test.exs

@@ -16,6 +16,11 @@ defmodule LoggerTest do
     msg("module=LoggerTest #{text}")
   end
 
+  test "levels/0" do
+    assert [_ | _] = Logger.levels()
+    assert :info in Logger.levels()
+  end
+
   test "level/0" do
     assert Logger.level() == :debug
 

lib/mix/lib/mix/compilers/elixir.ex

@@ -1,7 +1,7 @@
 defmodule Mix.Compilers.Elixir do
   @moduledoc false
 
-  @manifest_vsn 22
+  @manifest_vsn 23
   @checkpoint_vsn 2
 
   import Record
@@ -171,7 +171,7 @@ defmodule Mix.Compilers.Elixir do
         state = {%{}, exports, sources, [], modules, removed_modules}
         compiler_loop(stale, stale_modules, dest, timestamp, opts, state)
       else
-        {:ok, info, state} ->
+        {:ok, %{runtime_warnings: runtime_warnings, compile_warnings: compile_warnings}, state} ->
           {modules, _exports, sources, _changed, pending_modules, _stale_exports} = state
 
           previous_warnings =
@@ -179,7 +179,9 @@ defmodule Mix.Compilers.Elixir do
               do: previous_warnings(sources, true),
               else: []
 
-          sources = apply_warnings(sources, info)
+          runtime_warnings = Enum.map(runtime_warnings, &diagnostic/1)
+          compile_warnings = Enum.map(compile_warnings, &diagnostic/1)
+          sources = apply_warnings(sources, runtime_warnings, compile_warnings)
 
           write_manifest(
             manifest,
@@ -193,8 +195,7 @@ defmodule Mix.Compilers.Elixir do
           )
 
           put_compile_env(sources)
-          info_warnings = info.runtime_warnings ++ info.compile_warnings
-          all_warnings = previous_warnings ++ Enum.map(info_warnings, &diagnostic/1)
+          all_warnings = previous_warnings ++ runtime_warnings ++ compile_warnings
           unless_previous_warnings_as_errors(previous_warnings, opts, {:ok, all_warnings})
 
         {:error, errors, %{runtime_warnings: r_warnings, compile_warnings: c_warnings}, state} ->
@@ -717,23 +718,9 @@ defmodule Mix.Compilers.Elixir do
   end
 
   defp previous_warnings(sources, print?) do
-    for {source,
-         source(
-           compile_warnings: compile_warnings,
-           runtime_warnings: runtime_warnings
-         )} <- sources,
-        file = Path.absname(source),
-        {position, message, span} <- compile_warnings ++ runtime_warnings do
-      diagnostic = %Mix.Task.Compiler.Diagnostic{
-        severity: :warning,
-        file: file,
-        position: position,
-        message: message,
-        compiler_name: "Elixir",
-        stacktrace: [],
-        span: span
-      }
-
+    for {_, source(compile_warnings: compile_warnings, runtime_warnings: runtime_warnings)} <-
+          sources,
+        diagnostic <- compile_warnings ++ runtime_warnings do
       if print? do
         Mix.shell().print_app()
         Code.print_diagnostic(diagnostic)
@@ -743,13 +730,13 @@ defmodule Mix.Compilers.Elixir do
     end
   end
 
-  defp apply_warnings(sources, %{runtime_warnings: [], compile_warnings: []}) do
+  defp apply_warnings(sources, [], []) do
     sources
   end
 
-  defp apply_warnings(sources, %{runtime_warnings: r_warnings, compile_warnings: c_warnings}) do
-    runtime_group = Enum.group_by(r_warnings, & &1.file, &{&1.position, &1.message, &1.span})
-    compile_group = Enum.group_by(c_warnings, & &1.file, &{&1.position, &1.message, &1.span})
+  defp apply_warnings(sources, runtime_warnings, compile_warnings) do
+    runtime_group = Enum.group_by(runtime_warnings, & &1.source)
+    compile_group = Enum.group_by(compile_warnings, & &1.source)
 
     for {source_path, source_entry} <- sources, into: %{} do
       key = Path.absname(source_path)
@@ -767,22 +754,27 @@ defmodule Mix.Compilers.Elixir do
     end
   end
 
-  defp diagnostic(%{
-         file: file,
-         position: position,
-         message: message,
-         severity: severity,
-         stacktrace: stacktrace,
-         span: span
-       }) do
+  defp diagnostic(
+         %{
+           file: file,
+           position: position,
+           message: message,
+           severity: severity,
+           stacktrace: stacktrace,
+           span: span,
+           source: source
+         } = diagnostic
+       ) do
     %Mix.Task.Compiler.Diagnostic{
       file: file,
+      source: source,
       position: position,
       message: message,
       severity: severity,
       compiler_name: "Elixir",
       stacktrace: stacktrace,
-      span: span
+      span: span,
+      details: Map.get(diagnostic, :details, nil)
     }
   end
 

lib/mix/lib/mix/compilers/erlang.ex

@@ -288,8 +288,11 @@ defmodule Mix.Compilers.Erlang do
   defp to_diagnostics(warnings_or_errors, severity) do
     for {file, issues} <- warnings_or_errors,
         {location, module, data} <- issues do
+      file = Path.absname(file)
+
       %Mix.Task.Compiler.Diagnostic{
-        file: Path.absname(file),
+        file: file,
+        source: file,
         position: location_normalize(location),
         message: to_string(module.format_error(data)),
         severity: severity,

lib/mix/lib/mix/dep/lock.ex

@@ -11,7 +11,7 @@ defmodule Mix.Dep.Lock do
   """
   @spec read(Path.t()) :: map()
   def read(lockfile \\ lockfile()) do
-    opts = [file: lockfile, warn_on_unnecessary_quotes: false]
+    opts = [file: lockfile, emit_warnings: false]
 
     with {:ok, contents} <- File.read(lockfile),
          assert_no_merge_conflicts_in_lockfile(lockfile, contents),

lib/mix/lib/mix/local/installer.ex

@@ -245,7 +245,7 @@ defmodule Mix.Local.Installer do
   end
 
   defp git_fetcher(url, git_config, opts) do
-    git_opts = git_config ++ [git: url, submodules: opts[:submodules]]
+    git_opts = git_config ++ [git: url, submodules: opts[:submodules], sparse: opts[:sparse]]
 
     app_name =
       if opts[:app] do
@@ -319,8 +319,15 @@ defmodule Mix.Local.Installer do
   """
   @spec fetch(tuple, (atom -> any), (atom -> any)) :: any
   def fetch(dep_spec, in_fetcher \\ &in_fetcher/1, in_package) do
-    with_tmp_dir(fn tmp_path ->
+    tmp_path = tmp_path()
+    previous_env = Mix.env()
+    deps_path = System.get_env("MIX_DEPS_PATH")
+    code_path = :code.get_path()
+
+    try do
       File.mkdir_p!(tmp_path)
+      System.delete_env("MIX_DEPS_PATH")
+      Mix.env(:prod)
 
       File.write!(Path.join(tmp_path, "mix.exs"), """
       defmodule Mix.Local.Installer.MixProject do
@@ -336,64 +343,62 @@ defmodule Mix.Local.Installer do
       end
       """)
 
-      with_reset_prod_env(fn ->
-        Mix.ProjectStack.on_clean_slate(fn ->
-          tmp_path =
-            Mix.Project.in_project(:mix_local_installer, tmp_path, [], fn mix_exs ->
-              in_fetcher.(mix_exs)
-
-              # The tmp_dir may have symlinks in it, so we properly resolve
-              # the directory before customizing deps_path and lockfile.
-              File.cwd!()
-            end)
-
-          package_name = elem(dep_spec, 0)
-          package_name_string = Atom.to_string(package_name)
-          package_path = Path.join([tmp_path, "deps", package_name_string])
-
-          post_config = [
-            deps_path: Path.join(tmp_path, "deps"),
-            lockfile: Path.join(tmp_path, "mix.lock")
-          ]
-
-          Mix.Project.in_project(package_name, package_path, post_config, fn mix_exs ->
+      Mix.ProjectStack.on_clean_slate(fn ->
+        tmp_path =
+          Mix.Project.in_project(:mix_local_installer, tmp_path, [], fn mix_exs ->
             in_fetcher.(mix_exs)
-            in_package.(mix_exs)
+
+            # The tmp_dir may have symlinks in it, so we properly resolve
+            # the directory before customizing deps_path and lockfile.
+            File.cwd!()
           end)
+
+        {package_name, package_path} = package_name_path(dep_spec, tmp_path)
+
+        post_config = [
+          deps_path: Path.join(tmp_path, "deps"),
+          lockfile: Path.join(tmp_path, "mix.lock")
+        ]
+
+        Mix.Project.in_project(package_name, package_path, post_config, fn mix_exs ->
+          in_fetcher.(mix_exs)
+          in_package.(mix_exs)
         end)
       end)
-    end)
-  after
-    :code.purge(Mix.Local.Installer.Fetcher)
-    :code.delete(Mix.Local.Installer.Fetcher)
+    after
+      File.rm_rf(tmp_path)
+      Mix.env(previous_env)
+      deps_path && System.put_env("MIX_DEPS_PATH", deps_path)
+      :code.set_path(code_path)
+      :code.purge(Mix.Local.Installer.Fetcher)
+      :code.delete(Mix.Local.Installer.Fetcher)
+    end
   end
 
+  defp package_name_path(dep_spec, tmp_path) do
+    package_name = elem(dep_spec, 0)
+    package_name_string = Atom.to_string(package_name)
+    package_path = Path.join([tmp_path, "deps", package_name_string, maybe_sparse_dir(dep_spec)])
+
+    {package_name, package_path}
+  end
+
+  defp maybe_sparse_dir({_app, opts}) when is_list(opts) do
+    if opts[:git] do
+      opts[:sparse] || ""
+    else
+      ""
+    end
+  end
+
+  defp maybe_sparse_dir(_dep_spec), do: ""
+
   defp in_fetcher(_mix_exs) do
     Mix.Task.run("deps.get", ["--only", Atom.to_string(Mix.env())])
   end
 
-  defp with_tmp_dir(fun) do
+  defp tmp_path do
     unique = :crypto.strong_rand_bytes(4) |> Base.url_encode64(padding: false)
-    tmp_path = Path.join(System.tmp_dir!(), "mix-local-installer-fetcher-" <> unique)
-
-    try do
-      fun.(tmp_path)
-    after
-      File.rm_rf(tmp_path)
-    end
-  end
-
-  defp with_reset_prod_env(fun) do
-    previous_env = Mix.env()
-    deps_path = System.get_env("MIX_DEPS_PATH")
-
-    try do
-      System.delete_env("MIX_DEPS_PATH")
-      Mix.env(:prod)
-      fun.()
-    after
-      Mix.env(previous_env)
-      deps_path && System.put_env("MIX_DEPS_PATH", deps_path)
-    end
+    Path.join(System.tmp_dir!(), "mix-local-installer-fetcher-" <> unique)
   end
 end

lib/mix/lib/mix/release.ex

@@ -68,7 +68,7 @@ defmodule Mix.Release do
   @safe_modes [:permanent, :temporary, :transient]
   @unsafe_modes [:load, :none]
   @additional_chunks ~w(Attr)c
-  @copy_app_dirs ["priv"]
+  @copy_app_dirs ["priv", "include"]
 
   @doc false
   @spec from_config!(atom, keyword, keyword) :: t