Release v1.17.2

Also note that depth and ref cannot be used in combination.

.formatter.exs

@@ -13,7 +13,10 @@
     assert_same: 2,
 
     # Errors tests
-    assert_eval_raise: 3
+    assert_eval_raise: 3,
+
+    # Float tests
+    float_assert: 1
   ],
   normalize_bitstring_modifiers: false
 ]

.github/workflows/builds.hex.pm.yml

@@ -22,13 +22,13 @@ jobs:
       max-parallel: 1
       matrix:
         include:
-          - otp: 24
-            otp_version: '24.3'
-            upload_generic_version: upload_generic_version
           - otp: 25
-            otp_version: '25.3'
+            otp_version: "25.3"
+            upload_generic_version: upload_generic_version
           - otp: 26
-            otp_version: '26.0'
+            otp_version: "26.0"
+          - otp: 27
+            otp_version: "27.0"
             build_docs: build_docs
     runs-on: ubuntu-22.04
     steps:

.github/workflows/ci-markdown.yml

@@ -27,7 +27,8 @@ jobs:
           fetch-depth: 10
 
       - name: Run markdownlint
-        uses: DavidAnson/markdownlint-cli2-action@v13.0.0
+        uses: DavidAnson/markdownlint-cli2-action@v16.0.0
         with:
           globs: |
             lib/elixir/pages/**/*.md
+            README.md

.github/workflows/ci.yml

@@ -3,10 +3,10 @@ name: CI
 on:
   push:
     paths-ignore:
-      - 'lib/**/*.md'
+      - "lib/**/*.md"
   pull_request:
     paths-ignore:
-      - 'lib/**/*.md'
+      - "lib/**/*.md"
 
 env:
   ELIXIR_ASSERT_TIMEOUT: 2000
@@ -24,12 +24,11 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - otp_version: '26.0'
+          - otp_version: "27.0"
             otp_latest: true
-          - otp_version: '25.3'
-          - otp_version: '25.0'
-          - otp_version: '24.3'
-          - otp_version: '24.0'
+          - otp_version: "26.0"
+          - otp_version: "25.3"
+          - otp_version: "25.0"
           - otp_version: master
             development: true
           - otp_version: maint
@@ -77,7 +76,7 @@ jobs:
     name: Windows Server 2019, Erlang/OTP ${{ matrix.otp_version }}
     strategy:
       matrix:
-        otp_version: ['24', '25', '26']
+        otp_version: ["25.3", "26.0"]
     runs-on: windows-2019
     steps:
       - name: Configure Git

.github/workflows/release.yml

@@ -33,12 +33,12 @@ jobs:
       fail-fast: true
       matrix:
         include:
-          - otp: 24
-            otp_version: '24.3'
           - otp: 25
-            otp_version: '25.3'
+            otp_version: "25.3"
           - otp: 26
-            otp_version: '26.0'
+            otp_version: "26.0"
+          - otp: 27
+            otp_version: "27.0"
             build_docs: build_docs
     runs-on: ubuntu-22.04
     steps:

.markdownlint.jsonc

@@ -17,14 +17,19 @@
   "MD014": false,
   // Multiple headings with the same content. That's fine.
   "MD024": false,
+  // Some headers finish with ! because it refers to a function name
+  "MD026": false,
   // Allow empty line between block quotes. Used by contiguous admonition blocks.
   "MD028": false,
   // Allowed HTML inline elements.
   "MD033": {
     "allowed_elements": [
+      "h1",
       "a",
       "br",
       "img",
+      "picture",
+      "source",
       "noscript",
       "p",
       "script"

CHANGELOG.md

@@ -1,143 +1,215 @@
-# Changelog for Elixir v1.16
+# Changelog for Elixir v1.17
 
-## Code snippets in diagnostics
+This release includes type inference of patterns to provide warnings for an initial set of constructs (binaries, maps, and atoms) within the same function. It also includes a new Duration data type to interact with Calendar types, support for Erlang/OTP 27, and many other improvements.
 
-Elixir v1.15 introduced a new compiler diagnostic format and the ability to print multiple error diagnostics per compilation (in addition to multiple warnings).
+## Warnings from gradual set-theoretic types
 
-With Elixir v1.16, we also include code snippets in exceptions and diagnostics raised by the compiler. For example, a syntax error now includes a pointer to where the error happened:
+This release introduces gradual set-theoretic types to infer types from patterns and use them to type check programs, enabling the Elixir compiler to find faults and bugs in codebases without requiring changes to existing software. The underlying principles, theory, and roadmap of our work have been outlined in ["The Design Principles of the Elixir Type System" by Giuseppe Castagna, Guillaume Duboc, José Valim](https://arxiv.org/abs/2306.06391).
 
-```
-** (SyntaxError) invalid syntax found on lib/my_app.ex:1:17:
-    error: syntax error before: '*'
-    │
-  1 │ [1, 2, 3, 4, 5, *]
-    │                 ^
-    │
-    └─ lib/my_app.ex:1:17
+At the moment, Elixir developers will interact with set-theoretic types only through warnings found by the type system. The current implementation models all data types in the language:
+
+  * `binary()`, `integer()`, `float()`, `pid()`, `port()`, `reference()` - these types are indivisible. This means both `1` and `13` get the same `integer()` type.
+
+  * `atom()` - it represents all atoms and it is divisible. For instance, the atom `:foo` and `:hello_world` are also valid (distinct) types.
+
+  * `map()` and structs - maps can be "closed" or "open". Closed maps only allow the specified keys, such as `%{key: atom(), value: integer()}`. Open maps support any other keys in addition to the ones listed and their definition starts with `...`, such as `%{..., key: atom(), value: integer()}`. Structs are closed maps with the `__struct__` key.
+
+  * `tuple()`, `list()`, and `function()` - currently they are modelled as indivisible types. The next Elixir versions will also introduce fine-grained support to them.
+
+We focused on atoms and maps on this initial release as they are respectively the simplest and the most complex types representations, so we can stress the performance of the type system and quality of error messages. Modelling these types will also provide the most immediate benefits to Elixir developers. Assuming there is a variable named `user`, holding a `%User{}` struct with an `address` field, Elixir v1.17 will emit the following warnings at compile-time:
+
+  * Pattern matching against a map or a struct that does not have the given key, such as `%{adress: ...} = user` (notice `address` vs `adress`)
+
+  * Accessing a key on a map or a struct that does not have the given key, such as `user.adress`
+
+  * Invoking a function on non-modules, such as `user.address()`
+
+  * Capturing a function on non-modules, such as `&user.address/0`
+
+  * Attempting to invoke to call an anonymous function without an actual function, such as `user.()`
+
+  * Performing structural comparisons with structs, such as `my_date < ~D[2010-04-17]`
+
+  * Performing structural comparisons between non-overlapping types, such as `integer >= string`
+
+  * Building and pattern matching on binaries without the relevant specifiers, such as `<<name>>` (this warns because by default it expects an integer, it should have been `<<name::binary>>` instead)
+
+  * Attempting to rescue an undefined exception or a struct that is not an exception
+
+  * Accessing a field that is not defined in a rescued exception
+
+These new warnings help Elixir developers find bugs earlier and give more confidence when refactoring code, especially around maps and structs. While some of these warnings were emitted in the past, they were discovered using syntax analysis. The new warnings are more reliable, precise, and with better error messages. Keep in mind, however, that the Elixir typechecker only infers types from patterns within the same function at the moment. Analysis from guards and across function boundaries will be added in future relases. For more details, see our new [reference document on gradual set-theoretic types](https://hexdocs.pm/elixir/gradual-set-theoretic-types.html).
+
+The type system was made possible thanks to a partnership between [CNRS](https://www.cnrs.fr/) and [Remote](https://remote.com/). The development work is currently sponsored by [Fresha](https://www.fresha.com/), [Starfish*](https://starfish.team/), and [Dashbit](https://dashbit.co/).
+
+## Erlang/OTP support
+
+This release adds support for Erlang/OTP 27 and drops support for Erlang/OTP 24. We recommend Elixir developers to migrate to Erlang/OTP 26 or later, especially on Windows. Support for WERL (a graphical user interface for the Erlang terminal on Windows) will be removed in Elixir v1.18.
+
+## Adding `Duration` and `shift/2` functions
+
+Elixir introduces the `Duration` data type and APIs to shift dates, times, and date times by a given duration, considering different calendars and time zones.
+
+```elixir
+iex> Date.shift(~D[2016-01-31], month: 2)
+~D[2016-03-31]
 ```
 
-For mismatched delimiters, it now shows both delimiters:
+Note the operation is called `shift` (instead of `add`) since working with durations does not obey properties such as associativity. For instance, adding one month and then one month does not give the same result as adding two months:
 
-```
-** (MismatchedDelimiterError) mismatched delimiter found on lib/my_app.ex:1:18:
-    error: unexpected token: )
-    │
-  1 │ [1, 2, 3, 4, 5, 6)
-    │ │                └ mismatched closing delimiter (expected "]")
-    │ └ unclosed delimiter
-    │
-    └─ lib/my_app.ex:1:18
+```elixir
+iex> ~D[2016-01-31] |> Date.shift(month: 1) |> Date.shift(month: 1)
+~D[2016-03-29]
 ```
 
-Errors and warnings diagnostics also include code snippets. When possible, we will show precise spans, such as on undefined variables:
+Still, durations are essential for building intervals, recurring events, and modelling scheduling complexities found in the world around us. For `DateTime`s, Elixir will correctly deal with time zone changes (such as Daylight Saving Time), but provisions are also available in case you want to surface conflicts (for example, you shifted to a wall clock that does not exist, because the clock has been moved forward by one hour). See `DateTime.shift/2` for examples.
 
-```
-  error: undefined variable "unknown_var"
-  │
-5 │     a - unknown_var
-  │         ^^^^^^^^^^^
-  │
-  └─ lib/sample.ex:5:9: Sample.foo/1
+Finally, a new `Kernel.to_timeout/1` function has been added, which helps developers normalize durations and integers to a timeout used by Process APIs. For example, to send a message after one hour, one can now write:
+
+```elixir
+Process.send_after(pid, :wake_up, to_timeout(hour: 1))
 ```
 
-Otherwise the whole line is underlined:
+## v1.17.2 (2024-07-06)
 
-```
-error: function names should start with lowercase characters or underscore, invalid name CamelCase
-  │
-3 │   def CamelCase do
-  │   ^^^^^^^^^^^^^^^^
-  │
-  └─ lib/sample.ex:3
-```
+### 1. Bug fixes
 
-A huge thank you to Vinícius Muller for working on the new diagnostics.
+#### Logger
 
-## Revamped documentation
-
-Elixir's Getting Started guided has been made part of the Elixir repository and incorporated into ExDoc. This was an opportunity to revisit and unify all official guides and references.
-
-We have also incorporated and extended the work on [Understanding Code Smells in Elixir Functional Language](https://github.com/lucasvegi/Elixir-Code-Smells/blob/main/etc/2023-emse-code-smells-elixir.pdf), by Lucas Vegi and Marco Tulio Valente, from [ASERG/DCC/UFMG](http://aserg.labsoft.dcc.ufmg.br/), into the official document in the form of anti-patterns. The anti-patterns are divided into four categories: code-related, design-related, process-related, and meta-programming. Our goal is to give all developers with both positive and negative examples of Elixir code, with context and examples on how to improve their codebases.
-
-Another [ExDoc](https://github.com/elixir-lang/ex_doc) feature we have incorporated in this release is the addition of cheatsheets, starting with [a cheatsheet for the Enum module](https://hexdocs.pm/elixir/main/enum-cheat.html). If you would like to contribute future cheatsheets to Elixir itself, feel free to start a discussion with an issue.
-
-Finally, we have started enriching our documentation with [Mermaid.js](https://mermaid.js.org/) diagrams. You can find examples in the [GenServer](https://hexdocs.pm/elixir/main/GenServer.html) and [Supervisor](https://hexdocs.pm/elixir/main/Supervisor.html) docs.
-
-## v1.16.0-dev
-
-### 1. Enhancements
-
-#### EEx
-
-  * [EEx] Include relative file information in diagnostics
-
-#### Elixir
-
-  * [Code] Automatically include columns in parsing options
-  * [Code] Introduce `MismatchedDelimiterError` for handling mismatched delimiter exceptions
-  * [Code.Fragment] Handle anonymous calls in fragments
-  * [Kernel] Suggest module names based on suffix and casing errors when the module does not exist in `UndefinedFunctionError`
-  * [Kernel.ParallelCompiler] Introduce `Kernel.ParallelCompiler.pmap/2` to compile multiple additional entries in parallel
-  * [Kernel.SpecialForms] Warn if `True`/`False`/`Nil` are used as aliases and there is no such alias
-  * [Macro] Add `Macro.compile_apply/4`
-  * [Module] Add support for `@nifs` annotation from Erlang/OTP 25
-  * [Module] Add support for missing `@dialyzer` configuration
-  * [String] Update to Unicode 15.1.0
-  * [Task] Add `:limit` option to `Task.yield_many/2`
+  * [Logger.Translator] Fix logger crash when `:gen_statem`'s `format_status/2` returns non-tuple
 
 #### Mix
 
-  * [mix] Add `MIX_PROFILE` to profile a list of comma separated tasks
-  * [mix compile.elixir] Optimize scenario where there are thousands of files in `lib/` and one of them is changed
-  * [mix test] Allow testing multiple file:line at once, such as `mix test test/foo_test.exs:13 test/bar_test.exs:27`
+  * [mix deps.get] Fix regression when fetching a git repository with a `:ref`
+  * [mix release] Validate `RELEASE_MODE` and set ERRORLEVEL on `.bat` scripts
+  * [mix release] Fix invalid example in code comment inside the generated vm.args.eex
+
+## v1.17.1 (2024-06-18)
+
+### 1. Enhancements
+
+#### Mix
+
+  * [mix compile.elixir] Do not run fixpoint computation on runtime dependencies. This should considerably improve compilation times for large projects when changing only one or two files
+
+### 2. Bug fixes
+
+#### EEx
+
+  * [EEx] Do not warn for assignment with blocks in EEx
+
+#### Elixir
+
+  * [Kernel] Fix bug when using pinned variables inside `with`'s `else` patterns
+  * [Kernel] Fix Dialyzer error when with else clause is calling a `no_return` function
+
+#### ExUnit
+
+  * [ExUnit] Do not alternative sync/async suites on `--repeat-until-failure`
+
+## v1.17.0 (2024-06-12)
+
+### 1. Enhancements
+
+#### Elixir
+
+  * [Access] Add `Access.find/1` that mirrors `Enum.find/2`
+  * [Code] Support cursor inside fn/rescue/catch/else/after inside `Code.Fragment.container_cursor_to_quoted/2`
+  * [Date] Add `Date.shift/2` to shift dates with duration and calendar-specific semantics
+  * [Date] Allow `Date` to accept years outside of `-9999..9999` range
+  * [DateTime] Add `DateTime.shift/2` to shift datetimes with duration and calendar-specific semantics
+  * [Duration] Add a new `Duration` data type
+  * [GenServer] Add `c:GenServer.format_status/1` callback
+  * [Kernel] Add `Kernel.get_in/1` with safe nil-handling for access and structs
+  * [Kernel] Add `Kernel.is_non_struct_map/1` guard
+  * [Kernel] Add `Kernel.to_timeout/1`
+  * [Kernel] Emit warnings for undefined functions from modules defined within the same context as the caller code
+  * [Kernel] Support integers in uppercase sigils
+  * [Keyword] Add `Keyword.intersect/2-3` to mirror the `Map` API
+  * [Macro] Add `Macro.Env.define_alias/4`, `Macro.Env.define_import/4`, `Macro.Env.define_require/4`, `Macro.Env.expand_alias/4`, `Macro.Env.expand_import/5`, and `Macro.Env.expand_require/6` to aid the implementation of language servers and embedded languages
+  * [NaiveDateTime] Add `NaiveDateTime.shift/2` to shift naive datetimes with duration and calendar-specific semantics
+  * [Process] Add `Process.set_label/1`
+  * [String] Add `String.byte_slice/3` to slice a string to a maximum number of bytes while keeping it UTF-8 encoded
+  * [System] Support `use_stdio: false` in `System.cmd/3` and `System.shell/2`
+  * [Time] Add `Time.shift/2` to shift times with duration and calendar-specific semantics
+
+#### ExUnit
+
+  * [ExUnit] Propagate the test process itself as a caller in `start_supervised`
+  * [ExUnit] Include max cases in ExUnit reports
+
+#### IEx
+
+  * [IEx.Helpers] Warns if `recompile` was called and the current working directory changed
+  * [IEx.Helpers] Add `c/0` as an alias to `continue/0`
+  * [IEx.Pry] Add `IEx.Pry.annotate_quoted/3` to annotate a quoted expression with pry breakpoints
+
+#### Logger
+
+  * [Logger] Format `:gen_statem` reports using Elixir data structures
+  * [Logger] Include process label in logger events
+
+#### Mix
+
+  * [mix deps] Add `:depth` option to `Mix.SCM.Git`, thus supporting shallow clones of Git dependencies
+  * [mix deps] Warn if `:optional` is used in combination with `:in_umbrella`
+  * [mix deps.get] Do not add optional dependency requirements if its parent dep was skipped
+  * [mix deps.tree] Add `--umbrella-only` to `mix deps.tree`
+  * [mix profile.tprof] Add a new profiler, available on Erlang/OTP 27+, which can measure count, time, and heap usage
+  * [mix test] Add `mix test --breakpoints` that sets up a breakpoint before each test that will run
+  * [mix test] Add `mix test --repeat-until-failure` to rerun tests until a failure occurs
+  * [mix test] Add `mix test --slowest-modules` to print slowest modules based on all of the tests they hold
+  * [mix test] Generate cover HTML files in parallel
 
 ### 2. Bug fixes
 
 #### Elixir
 
-  * [Code.Fragment] Fix crash in `Code.Fragment.surround_context/2` when matching on `->`
-  * [IO] Raise when using `IO.binwrite/2` on terminated device (mirroring `IO.write/2`)
-  * [Kernel] Do not expand aliases recursively (the alias stored in Macro.Env is already expanded)
-  * [Kernel] Ensure `dbg` module is a compile-time dependency
-  * [Kernel] Warn when a private function or macro uses `unquote/1` and the function/macro itself is unused
-  * [Kernel] Do not define an alias for nested modules starting with `Elixir.` in their definition
-  * [Kernel.ParallelCompiler] Consider a module has been defined in `@after_compile` callbacks to avoid deadlocks
-  * [Path] Ensure `Path.relative_to/2` returns a relative path when the given argument does not share a common prefix with `cwd`
+  * [bin/elixir.bat] Improve handling of quotes and exclamation marks in flags
+  * [Code] Address a bug where AST nodes for `(a -> b)` were not wrapped as part of the literal encoder
+  * [Kernel] Resolve inconsistencies of how `..` and `...` are handled at the AST level
+  * [Kernel] Fix parsing precedence of ambiguous operators followed by containers
+  * [Kernel] Do not expand code in `quote bind_quoted: ...` twice
+  * [Kernel] Respect `:line` property when `:file` is given as option to `quote`
+  * [Kernel] Do not crash on `Macro.escape/2` when passing a quote triplet without valid meta
+  * [Kernel] Avoid double tracing events when capturing a function
+  * [Kernel] Fix a bug where captured arguments would conflict when a capture included a macro that also used captures
+  * [Module] Return default value in `Module.get_attribute/3` for persisted attributes which have not yet been written to
+  * [String] Properly handle transpositions in `jaro_distance`. This will correct the distance result in certain cases
 
-#### ExUnit
+#### IEx
 
-  * [ExUnit] Raise on incorrectly dedented doctests
+  * [IEx.Helpers] Update the history size whenever history is pruned
 
 #### Mix
 
-  * [Mix] Ensure files with duplicate modules are recompiled whenever any of the files change
+  * [mix deps] Fix error message for diverged SCM definition in sibling
 
 ### 3. Soft deprecations (no warnings emitted)
 
 #### Elixir
 
-  * [File] Deprecate `File.stream!(file, options, line_or_bytes)` in favor of keeping the options as last argument, as in `File.stream!(file, line_or_bytes, options)`
-  * [Kernel.ParallelCompiler] Deprecate `Kernel.ParallelCompiler.async/1` in favor of `Kernel.ParallelCompiler.pmap/2`
-  * [Path] Deprecate `Path.safe_relative_to/2` in favor of `Path.safe_relative/2`
+  * [GenServer] Deprecate `c:GenServer.format_status/2` callback to align with Erlang/OTP 25+
+
+#### Mix
+
+  * [mix profile.cprof] Deprecated in favor of the new `mix profile.tprof`
+  * [mix profile.eprof] Deprecated in favor of the new `mix profile.tprof`
 
 ### 4. Hard deprecations
 
 #### Elixir
 
-  * [Date] Deprecate inferring a range with negative step, call `Date.range/3` with a negative step instead
-  * [Enum] Deprecate passing a range with negative step on `Enum.slice/2`, give `first..last//1` instead
-  * [Kernel] `~R/.../` is deprecated in favor of `~r/.../`. This is because `~R/.../` still allowed escape codes, which did not fit the definition of uppercase sigils
-  * [String] Deprecate passing a range with negative step on `String.slice/2`, give `first..last//1` instead
+  * [IO] Passing `:all` to `IO.read/2` and `IO.binread/2` is deprecated, pass `:eof` instead
+  * [Kernel] Single-quote charlists are deprecated, use `~c` instead
+  * [Kernel] Deprecate escaping closing delimiter in uppercase sigils
+  * [Range] `left..right` without explicit steps inside patterns and guards is deprecated, write `left..right//step` instead
+  * [Range] Decreasing ranges, such as `10..1` without an explicit step is deprecated, write `10..1//-1` instead
 
 #### ExUnit
 
-  * [ExUnit.Formatter] Deprecate `format_time/2`, use `format_times/1` instead
+  * [ExUnit.Case] `register_test/4` is deprecated in favor of `register_test/6` for performance reasons
 
-#### Mix
+## v1.16
 
-  * [mix compile.leex] Require `:leex` to be added as a compiler to run the `leex` compiler
-  * [mix compile.yecc] Require `:yecc` to be added as a compiler to run the `yecc` compiler
-
-## v1.15
-
-The CHANGELOG for v1.15 releases can be found [in the v1.15 branch](https://github.com/elixir-lang/elixir/blob/v1.15/CHANGELOG.md).
+The CHANGELOG for v1.16 releases can be found [in the v1.16 branch](https://github.com/elixir-lang/elixir/blob/v1.16/CHANGELOG.md).

Makefile

@@ -2,7 +2,7 @@ PREFIX ?= /usr/local
 TEST_FILES ?= "*_test.exs"
 SHARE_PREFIX ?= $(PREFIX)/share
 MAN_PREFIX ?= $(SHARE_PREFIX)/man
-CANONICAL := main/
+# CANONICAL := main/
 ELIXIRC := bin/elixirc --ignore-module-conflict $(ELIXIRC_OPTS)
 ERLC := erlc -I lib/elixir/include
 ERL_MAKE := if [ -n "$(ERLC_OPTS)" ]; then ERL_COMPILER_OPTIONS=$(ERLC_OPTS) erl -make; else erl -make; fi
@@ -27,9 +27,9 @@ SOURCE_DATE_EPOCH_FILE = $(SOURCE_DATE_EPOCH_PATH)/SOURCE_DATE_EPOCH
 #==> Functions
 
 define CHECK_ERLANG_RELEASE
-	erl -noshell -eval '{V,_} = string:to_integer(erlang:system_info(otp_release)), io:fwrite("~s", [is_integer(V) and (V >= 24)])' -s erlang halt | grep -q '^true'; \
+	erl -noshell -eval '{V,_} = string:to_integer(erlang:system_info(otp_release)), io:fwrite("~s", [is_integer(V) and (V >= 25)])' -s erlang halt | grep -q '^true'; \
 		if [ $$? != 0 ]; then \
-		  echo "At least Erlang/OTP 24.0 is required to build Elixir"; \
+		  echo "At least Erlang/OTP 25.0 is required to build Elixir"; \
 		  exit 1; \
 		fi
 endef
@@ -177,9 +177,8 @@ clean_residual_files:
 
 #==> Documentation tasks
 
-LOGO_PATH = $(shell test -f ../docs/logo.png && echo "--logo ../docs/logo.png")
 SOURCE_REF = $(shell tag="$(call GIT_TAG)" revision="$(call GIT_REVISION)"; echo "$${tag:-$$revision}")
-DOCS_COMPILE = CANONICAL=$(CANONICAL) bin/elixir ../ex_doc/bin/ex_doc "$(1)" "$(VERSION)" "lib/$(2)/ebin" --main "$(3)" --source-url "https://github.com/elixir-lang/elixir" --source-ref "$(call SOURCE_REF)" $(call LOGO_PATH) --output doc/$(2) --canonical "https://hexdocs.pm/$(2)/$(CANONICAL)" --homepage-url "https://elixir-lang.org/docs.html" $(4)
+DOCS_COMPILE = CANONICAL=$(CANONICAL) bin/elixir ../ex_doc/bin/ex_doc "$(1)" "$(VERSION)" "lib/$(2)/ebin" --main "$(3)" --source-url "https://github.com/elixir-lang/elixir" --source-ref "$(call SOURCE_REF)" --logo lib/elixir/pages/images/logo.png --output doc/$(2) --canonical "https://hexdocs.pm/$(2)/$(CANONICAL)" --homepage-url "https://elixir-lang.org/docs.html" $(4)
 DOCS_CONFIG = bin/elixir lib/elixir/scripts/docs_config.exs "$(1)"
 
 docs: compile ../ex_doc/bin/ex_doc docs_elixir docs_eex docs_mix docs_iex docs_ex_unit docs_logger
@@ -189,7 +188,6 @@ docs_elixir: compile ../ex_doc/bin/ex_doc
 	$(Q) rm -rf doc/elixir
 	$(call DOCS_COMPILE,Elixir,elixir,Kernel,--config "lib/elixir/scripts/elixir_docs.exs")
 	$(call DOCS_CONFIG,elixir)
-	cp -R lib/elixir/pages/images doc/elixir
 
 docs_eex: compile ../ex_doc/bin/ex_doc
 	@ echo "==> ex_doc (eex)"

README.md

@@ -1,7 +1,11 @@
-<img src="https://github.com/elixir-lang/elixir-lang.github.com/raw/main/images/logo/logo.png#gh-light-mode-only" width="200" alt="Elixir">
-<img src="https://github.com/elixir-lang/elixir-lang.github.com/raw/main/images/logo/logo-dark.png#gh-dark-mode-only" width="200" alt="Elixir">
+<h1>
+ <picture>
+  <source media="(prefers-color-scheme: dark)" srcset="https://github.com/elixir-lang/elixir-lang.github.com/raw/main/images/logo/logo-dark.png">
+  <img alt="Elixir logo" src="https://github.com/elixir-lang/elixir-lang.github.com/raw/main/images/logo/logo.png" width="200">
+ </picture>
+</h1>
 
-[![CI](https://github.com/elixir-lang/elixir/workflows/CI/badge.svg?branch=main)](https://github.com/elixir-lang/elixir/actions?query=branch%3Amain+workflow%3ACI)
+[![CI](https://github.com/elixir-lang/elixir/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/elixir-lang/elixir/actions/workflows/ci.yml?query=branch%3Amain)
 
 Elixir is a dynamic, functional language designed for building scalable
 and maintainable applications.
@@ -12,7 +16,7 @@ For more about Elixir, installation and documentation,
 ## Policies
 
 New releases are announced in the [announcement mailing list][8].
-You can subscribe by sending an email to elixir-lang-ann+subscribe@googlegroups.com
+You can subscribe by sending an email to <elixir-lang-ann+subscribe@googlegroups.com>
 and replying to the confirmation email.
 
 All security releases [will be tagged with `[security]`][10]. For more
@@ -25,28 +29,28 @@ All interactions in our official communication channels follow our
 
 For reporting bugs, [visit our issue tracker][2] and follow the steps
 for reporting a new issue. **Please disclose security vulnerabilities
-privately at elixir-security@googlegroups.com**.
+privately at <elixir-security@googlegroups.com>**.
 
 ## Issues tracker management
 
 All currently open bugs related to the Elixir repository are listed
 in the issues tracker. The Elixir team uses the issues tracker to focus
-on *actionable items*, including planned enhancements in the short- and
-medium-term. We also do our best to label entries for clarity and to ease
+on *actionable items*, including planned enhancements in the short and
+medium term. We also do our best to label entries for clarity and to ease
 collaboration.
 
 Our *actionable item policy* has some important consequences, such as:
 
-  * Proposing new features as well as request for support, help, and
+  * Proposing new features as well as requests for support, help, and
     guidance must be done in their own spaces, detailed next.
 
-  * Issues where we have identified to be outside of Elixir scope,
-    such as a bug upstream, will be closed (and requested to be moved
+  * Issues we have identified to be outside of Elixir's scope,
+    such as an upstream bug, will be closed (and requested to be moved
     elsewhere if appropriate).
 
   * We actively close unrelated and non-actionable issues to keep the
-    issues tracker tidy. However, we may get things wrong from time to
-    time, so we are glad to revisit issues and reopen if necessary.
+    issues tracker tidy. We may get things wrong from time to
+    time and will gladly revisit issues, reopening when necessary.
 
 Keep the tone positive and be kind! For more information, see the
 [Code of Conduct][1].
@@ -72,7 +76,7 @@ in the next release are then "closed" and added to the [changelog][7].
 
 ### Discussions, support, and help
 
-For general discussions, support, and help, please use many of the community
+For general discussions, support, and help, please use the community
 spaces [listed on the sidebar of the Elixir website](https://elixir-lang.org/),
 such as forums, chat platforms, etc, where the wider community will be available
 to help you.
@@ -103,29 +107,29 @@ Additionally, you may choose to run the test suite with `make clean test`.
 
 ## Contributing
 
-We welcome everyone to contribute to Elixir. To do so, there are a few
+We invite contributions to Elixir. To contribute, there are a few
 things you need to know about the code. First, Elixir code is divided
-in applications inside the `lib` folder:
+by each application inside the `lib` folder:
 
-* `elixir` - Elixir's kernel and standard library
+  * `elixir` - Elixir's kernel and standard library
 
-* `eex` - EEx is the template engine that allows you to embed Elixir
+  * `eex` - EEx is the template engine that allows you to embed Elixir
 
-* `ex_unit` - ExUnit is a simple test framework that ships with Elixir
+  * `ex_unit` - ExUnit is a simple test framework that ships with Elixir
 
-* `iex` - IEx stands for Interactive Elixir: Elixir's interactive shell
+  * `iex` - IEx stands for Interactive Elixir: Elixir's interactive shell
 
-* `logger` - Logger is the built-in logger
+  * `logger` - Logger is the built-in logger
 
-* `mix` - Mix is Elixir's build tool
+  * `mix` - Mix is Elixir's build tool
 
-You can run all tests in the root directory with `make test` and you can
-also run tests for a specific framework `make test_#{APPLICATION}`, for example,
+You can run all tests in the root directory with `make test`. You can
+also run tests for a specific framework with `make test_#{APPLICATION}`, for example,
 `make test_ex_unit`. If you just changed something in Elixir's standard
 library, you can run only that portion through `make test_stdlib`.
 
-If you are changing just one file, you can choose to compile and run tests only
-for that particular file for fast development cycles. For example, if you
+If you are only changing one file, you can choose to compile and run tests
+for that specific file for faster development cycles. For example, if you
 are changing the String module, you can compile it and run its tests as:
 
 ```sh
@@ -146,7 +150,7 @@ make compile
 ```
 
 After your changes are done, please remember to run `make format` to guarantee
-all files are properly formatted and then run the full suite with
+all files are properly formatted, then run the full suite with
 `make test`.
 
 If your contribution fails during the bootstrapping of the language,
@@ -156,7 +160,7 @@ you can rebuild the language from scratch with:
 make clean_elixir compile
 ```
 
-Similarly, if you can't get Elixir to compile or the tests to pass after
+Similarly, if you can not get Elixir to compile or the tests to pass after
 updating an existing checkout, run `make clean compile`. You can check
 [the official build status](https://github.com/elixir-lang/elixir/actions/workflows/ci.yml).
 More tasks can be found by reading the [Makefile](Makefile).
@@ -166,9 +170,9 @@ With tests running and passing, you are ready to contribute to Elixir and
 We have saved some excellent pull requests we have received in the past in
 case you are looking for some examples:
 
-* [Implement Enum.member? - Pull request](https://github.com/elixir-lang/elixir/pull/992)
-* [Add String.valid? - Pull request](https://github.com/elixir-lang/elixir/pull/1058)
-* [Implement capture_io for ExUnit - Pull request](https://github.com/elixir-lang/elixir/pull/1059)
+  * [Implement Enum.member? - Pull request](https://github.com/elixir-lang/elixir/pull/992)
+  * [Add String.valid? - Pull request](https://github.com/elixir-lang/elixir/pull/1058)
+  * [Implement capture_io for ExUnit - Pull request](https://github.com/elixir-lang/elixir/pull/1059)
 
 ### Reviewing changes
 
@@ -176,7 +180,7 @@ Once a pull request is sent, the Elixir team will review your changes.
 We outline our process below to clarify the roles of everyone involved.
 
 All pull requests must be approved by two committers before being merged into
-the repository. If any changes are necessary, the team will leave appropriate
+the repository. If changes are necessary, the team will leave appropriate
 comments requesting changes to the code. Unfortunately, we cannot guarantee a
 pull request will be merged, even when modifications are requested, as the Elixir
 team will re-evaluate the contribution as it changes.
@@ -196,8 +200,8 @@ a comment.
 
 ## Building documentation
 
-Building the documentation requires [ExDoc](https://github.com/elixir-lang/ex_doc)
-to be installed and built alongside Elixir:
+Building the documentation requires that [ExDoc](https://github.com/elixir-lang/ex_doc)
+is installed and built alongside Elixir:
 
 ```sh
 # After cloning and compiling Elixir, in its parent directory:

RELEASE.md

@@ -8,15 +8,17 @@
 
 3. Update "Compatibility and Deprecations" if a new OTP version is supported
 
-4. Commit changes above with title "Release vVERSION", generate a new tag, and push it
+4. Commit changes above with title "Release vVERSION" and push it
 
-5. Wait until GitHub Actions publish artifacts to the draft release and the CI is green
+6. Once GitHub actions completes, generate a new tag, and push it
 
-6. Copy the relevant bits from /CHANGELOG.md to the GitHub release and publish it
+7. Wait until GitHub Actions publish artifacts to the draft release
 
-7. Update `_data/elixir-versions.yml` (except for RCs) in `elixir-lang/elixir-lang.github.com`
+8. Copy the relevant bits from /CHANGELOG.md to the GitHub release and publish it (link to the announcement if there is one)
 
-## Creating a new vMAJOR.MINOR branch (after first rc)
+9. Update `_data/elixir-versions.yml` (except for RCs) in `elixir-lang/elixir-lang.github.com`
+
+## Creating a new vMAJOR.MINOR branch (before first rc)
 
 ### In the new branch
 

SECURITY.md

@@ -6,12 +6,11 @@ Elixir applies bug fixes only to the latest minor branch. Security patches are a
 
 Elixir version | Support
 :------------- | :-----------------------------
-1.16           | Development
-1.15           | Bug fixes and security patches
+1.17           | Bug fixes and security patches
+1.16           | Security patches only
+1.15           | Security patches only
 1.14           | Security patches only
 1.13           | Security patches only
-1.12           | Security patches only
-1.11           | Security patches only
 
 ## Announcements
 

VERSION

@@ -1 +1 @@
-1.16.0-dev
+1.17.2

bin/elixir

@@ -1,7 +1,7 @@
 #!/bin/sh
 set -e
 
-ELIXIR_VERSION=1.16.0-dev
+ELIXIR_VERSION=1.17.2
 
 if [ $# -eq 0 ] || { [ $# -eq 1 ] && { [ "$1" = "--help" ] || [ "$1" = "-h" ]; }; }; then
   cat <<USAGE >&2
@@ -112,10 +112,10 @@ while [ $I -le $LENGTH ]; do
         C=1
         MODE="iex"
         ;;
-    -v|--no-halt|--dbg)
+    -v|--no-halt)
         C=1
         ;;
-    -e|-r|-pr|-pa|-pz|--eval|--remsh|--dot-iex)
+    -e|-r|-pr|-pa|-pz|--eval|--remsh|--dot-iex|--dbg)
         C=2
         ;;
     --rpc-eval)
@@ -218,6 +218,7 @@ SELF=$(readlink_f "$0")
 SCRIPT_PATH=$(dirname "$SELF")
 
 if [ "$OSTYPE" = "cygwin" ]; then SCRIPT_PATH=$(cygpath -m "$SCRIPT_PATH"); fi
+if [ "$MODE" != "iex" ]; then ERL="-s elixir start_cli $ERL"; fi
 
 if [ "$OS" != "Windows_NT" ] && [ -z "$NO_COLOR" ]; then
   if test -t 1 -a -t 2; then ERL="-elixir ansi_enabled true $ERL"; fi
@@ -228,7 +229,7 @@ fi
 ERTS_BIN=
 ERTS_BIN="$ERTS_BIN"
 
-set -- "$ERTS_BIN$ERL_EXEC" -noshell -elixir_root "$SCRIPT_PATH"/../lib -pa "$SCRIPT_PATH"/../lib/elixir/ebin $ELIXIR_ERL_OPTIONS -s elixir start_$MODE $ERL "$@"
+set -- "$ERTS_BIN$ERL_EXEC" -noshell -elixir_root "$SCRIPT_PATH"/../lib -pa "$SCRIPT_PATH"/../lib/elixir/ebin $ELIXIR_ERL_OPTIONS $ERL "$@"
 
 if [ -n "$RUN_ERL_PIPE" ]; then
   ESCAPED=""

bin/elixir.bat

@@ -1,8 +1,7 @@
-@if defined ELIXIR_CLI_ECHO (@echo on) else (@echo off)
+@echo off
 
-set ELIXIR_VERSION=1.16.0-dev
+set ELIXIR_VERSION=1.17.2
 
-setlocal enabledelayedexpansion
 if    ""%1""==""""                if ""%2""=="""" goto documentation
 if /I ""%1""==""--help""          if ""%2""=="""" goto documentation
 if /I ""%1""==""-h""              if ""%2""=="""" goto documentation
@@ -64,13 +63,11 @@ echo ** Standalone options can't be combined with other options.
 goto end
 
 :shortversion
-echo !ELIXIR_VERSION!
+echo %ELIXIR_VERSION%
 goto end
 
 :parseopts
-
-rem Parameters for Elixir
-set parsElixir=
+setlocal enabledelayedexpansion
 
 rem Parameters for Erlang
 set parsErlang=
@@ -97,89 +94,61 @@ if "!par!"=="" (
 )
 shift
 set par="!par:"=\"!"
-if !endLoop! == 1 (
-  set parsElixir=!parsElixir! !par!
-  goto startloop
-)
 rem ******* EXECUTION OPTIONS **********************
 if !par!=="--werl"   (set useWerl=1 && goto startloop)
-if !par!=="+iex"     (set parsElixir=!parsElixir! +iex && set useIEx=1 && goto startloop)
-if !par!=="+elixirc" (set parsElixir=!parsElixir! +elixirc && goto startloop)
-rem ******* EVAL PARAMETERS ************************
-if ""==!par:-e=! (
-  set "VAR=%~1"
-  if not defined VAR (set VAR= )
-  set parsElixir=!parsElixir! -e "!VAR:"=\"!"
-  shift
-  goto startloop
-)
-if ""==!par:--eval=! (
-  set "VAR=%~1"
-  if not defined VAR (set VAR= )
-  set parsElixir=!parsElixir! --eval "!VAR:"=\"!"
-  shift
-  goto startloop
-)
-if ""==!par:--rpc-eval=! (
-  set "VAR=%~2"
-  if not defined VAR (set VAR= )
-  set parsElixir=!parsElixir! --rpc-eval %1 "!VAR:"=\"!"
-  shift
-  shift
-  goto startloop
-)
+if !par!=="+iex"     (set useIEx=1 && goto startloop)
+if !par!=="+elixirc" (goto startloop)
 rem ******* ELIXIR PARAMETERS **********************
-if ""==!par:-r=!          (set "parsElixir=!parsElixir! -r %~1" && shift && goto startloop)
-if ""==!par:-pr=!         (set "parsElixir=!parsElixir! -pr %~1" && shift && goto startloop)
-if ""==!par:-pa=!         (set "parsElixir=!parsElixir! -pa %~1" && shift && goto startloop)
-if ""==!par:-pz=!         (set "parsElixir=!parsElixir! -pz %~1" && shift && goto startloop)
-if ""==!par:-v=!          (set "parsElixir=!parsElixir! -v" && goto startloop)
-if ""==!par:--version=!   (set "parsElixir=!parsElixir! --version" && goto startloop)
-if ""==!par:--no-halt=!   (set "parsElixir=!parsElixir! --no-halt" && goto startloop)
-if ""==!par:--remsh=!     (set "parsElixir=!parsElixir! --remsh %~1" && shift && goto startloop)
-if ""==!par:--dot-iex=!   (set "parsElixir=!parsElixir! --dot-iex %~1" && shift && goto startloop)
-if ""==!par:--dbg=!       (set "parsElixir=!parsElixir! --dbg %~1" && shift && goto startloop)
+if ""==!par:-e=!          (shift && goto startloop)
+if ""==!par:--eval=!      (shift && goto startloop)
+if ""==!par:--rpc-eval=!  (shift && shift && goto startloop)
+if ""==!par:-r=!          (shift && goto startloop)
+if ""==!par:-pr=!         (shift && goto startloop)
+if ""==!par:-pa=!         (shift && goto startloop)
+if ""==!par:-pz=!         (shift && goto startloop)
+if ""==!par:-v=!          (goto startloop)
+if ""==!par:--version=!   (goto startloop)
+if ""==!par:--no-halt=!   (goto startloop)
+if ""==!par:--remsh=!     (shift && goto startloop)
+if ""==!par:--dot-iex=!   (shift && goto startloop)
+if ""==!par:--dbg=!       (shift && goto startloop)
 rem ******* ERLANG PARAMETERS **********************
-if ""==!par:--boot=!                (set "parsErlang=!parsErlang! -boot %~1" && shift && goto startloop)
-if ""==!par:--boot-var=!            (set "parsErlang=!parsErlang! -boot_var %~1 %~2" && shift && shift && goto startloop)
-if ""==!par:--cookie=!              (set "parsErlang=!parsErlang! -setcookie %~1" && shift && goto startloop)
+if ""==!par:--boot=!                (set "parsErlang=!parsErlang! -boot "%~1"" && shift && goto startloop)
+if ""==!par:--boot-var=!            (set "parsErlang=!parsErlang! -boot_var "%~1" "%~2"" && shift && shift && goto startloop)
+if ""==!par:--cookie=!              (set "parsErlang=!parsErlang! -setcookie "%~1"" && shift && goto startloop)
 if ""==!par:--hidden=!              (set "parsErlang=!parsErlang! -hidden" && goto startloop)
-if ""==!par:--erl-config=!          (set "parsErlang=!parsErlang! -config %~1" && shift && goto startloop)
+if ""==!par:--erl-config=!          (set "parsErlang=!parsErlang! -config "%~1"" && shift && goto startloop)
 if ""==!par:--logger-otp-reports=!  (set "parsErlang=!parsErlang! -logger handle_otp_reports %1" && shift && goto startloop)
 if ""==!par:--logger-sasl-reports=! (set "parsErlang=!parsErlang! -logger handle_sasl_reports %1" && shift && goto startloop)
-if ""==!par:--name=!                (set "parsErlang=!parsErlang! -name %~1" && shift && goto startloop)
-if ""==!par:--sname=!               (set "parsErlang=!parsErlang! -sname %~1" && shift && goto startloop)
-if ""==!par:--vm-args=!             (set "parsErlang=!parsErlang! -args_file %~1" && shift && goto startloop)
+if ""==!par:--name=!                (set "parsErlang=!parsErlang! -name "%~1"" && shift && goto startloop)
+if ""==!par:--sname=!               (set "parsErlang=!parsErlang! -sname "%~1"" && shift && goto startloop)
+if ""==!par:--vm-args=!             (set "parsErlang=!parsErlang! -args_file "%~1"" && shift && goto startloop)
 if ""==!par:--erl=!                 (set "beforeExtra=!beforeExtra! %~1" && shift && goto startloop)
 if ""==!par:--pipe-to=!             (echo --pipe-to : Option is not supported on Windows && goto end)
-set endLoop=1
-set parsElixir=!parsElixir! !par!
-goto startloop
 
 :run
+setlocal disabledelayedexpansion
 reg query HKCU\Console /v VirtualTerminalLevel 2>nul | findstr /e "0x1" >nul 2>nul
 if %errorlevel% == 0 (
-  set beforeExtra=-elixir ansi_enabled true !beforeExtra!
+  set beforeExtra=-elixir ansi_enabled true %beforeExtra%
 )
-if defined useIEx (
-  set beforeExtra=-s elixir start_iex !beforeExtra!
-) else (
-  set beforeExtra=-s elixir start_cli !beforeExtra!
+if not defined useIEx (
+  set beforeExtra=-s elixir start_cli %beforeExtra%
 )
 
-set beforeExtra=-noshell -elixir_root "!SCRIPT_PATH!..\lib" -pa "!SCRIPT_PATH!..\lib\elixir\ebin" !beforeExtra!
+set beforeExtra=-noshell -elixir_root "%SCRIPT_PATH%..\lib" -pa "%SCRIPT_PATH%..\lib\elixir\ebin" %beforeExtra%
 
 if defined ELIXIR_CLI_DRY_RUN (
    if defined useWerl (
-     echo start "" "!ERTS_BIN!werl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+     echo start "" "%ERTS_BIN%werl.exe" %ext_libs% %ELIXIR_ERL_OPTIONS% %parsErlang% %beforeExtra% -extra %*
    ) else (
-     echo "!ERTS_BIN!erl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+     echo "%ERTS_BIN%erl.exe" %ext_libs% %ELIXIR_ERL_OPTIONS% %parsErlang% %beforeExtra% -extra %*
    )
 ) else (
   if defined useWerl (
-    start "" "!ERTS_BIN!werl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+    start "" "%ERTS_BIN%werl.exe" %ext_libs% %ELIXIR_ERL_OPTIONS% %parsErlang% %beforeExtra% -extra %*
   ) else (
-    "!ERTS_BIN!erl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+    "%ERTS_BIN%erl.exe" %ext_libs% %ELIXIR_ERL_OPTIONS% %parsErlang% %beforeExtra% -extra %*
   )
 )
 exit /B %ERRORLEVEL%

bin/elixirc.bat

@@ -1,4 +1,4 @@
-@if defined ELIXIR_CLI_ECHO (@echo on) else (@echo off)
+@echo off
 setlocal
 set argc=0
 for %%A in (%*) do (

bin/iex.bat

@@ -1,4 +1,4 @@
-@if defined ELIXIR_CLI_ECHO (@echo on) else (@echo off)
+@echo off
 setlocal
 if /I ""%1""==""--help"" goto documentation
 if /I ""%1""==""-h""     goto documentation

bin/mix.bat

@@ -1,2 +1,2 @@
-@if defined ELIXIR_CLI_ECHO (@echo on) else (@echo off)
+@echo off
 call "%~dp0\elixir.bat" "%~dp0\mix" %*

lib/eex/lib/eex.ex

@@ -317,8 +317,8 @@ defmodule EEx do
 
   ## Examples
 
-      iex> EEx.tokenize('foo', line: 1, column: 1)
-      {:ok, [{:text, 'foo', %{column: 1, line: 1}}, {:eof, %{column: 4, line: 1}}]}
+      iex> EEx.tokenize(~c"foo", line: 1, column: 1)
+      {:ok, [{:text, ~c"foo", %{column: 1, line: 1}}, {:eof, %{column: 4, line: 1}}]}
 
   ## Result
 

lib/eex/lib/eex/compiler.ex

@@ -71,10 +71,10 @@ defmodule EEx.Compiler do
       {:ok, expr, new_line, new_column, rest} ->
         {key, expr} =
           case :elixir_tokenizer.tokenize(expr, 1, file: "eex", check_terminators: false) do
-            {:ok, _line, _column, _warnings, tokens} ->
+            {:ok, _line, _column, _warnings, rev_tokens, []} ->
               # We ignore warnings because the code will be tokenized
               # again later with the right line+column info
-              token_key(tokens, expr)
+              token_key(rev_tokens, expr)
 
             {:error, _, _, _, _} ->
               {:expr, expr}
@@ -164,8 +164,8 @@ defmodule EEx.Compiler do
   end
 
   # Receives tokens and check if it is a start, middle or an end token.
-  defp token_key(tokens, expr) do
-    case {tokens, tokens |> Enum.reverse() |> drop_eol()} do
+  defp token_key(rev_tokens, expr) do
+    case {Enum.reverse(rev_tokens), drop_eol(rev_tokens)} do
       {[{:end, _} | _], [{:do, _} | _]} ->
         {:middle_expr, expr}
 
@@ -340,13 +340,6 @@ defmodule EEx.Compiler do
          scope,
          state
        ) do
-    if mark == ~c"" do
-      message =
-        "the contents of this expression won't be output unless the EEx block starts with \"<%=\""
-
-      :elixir_errors.erl_warn({meta.line, meta.column}, state.file, message)
-    end
-
     {rest, line, contents} = look_ahead_middle(rest, meta.line, chars) || {rest, meta.line, chars}
     start_line = meta.line
     start_column = column(meta.column, mark)
@@ -359,6 +352,13 @@ defmodule EEx.Compiler do
         %{state | quoted: [], line: line}
       )
 
+    if mark == ~c"" and not match?({:=, _, [_, _]}, contents) do
+      message =
+        "the contents of this expression won't be output unless the EEx block starts with \"<%=\""
+
+      :elixir_errors.erl_warn({meta.line, meta.column}, state.file, message)
+    end
+
     buffer = state.engine.handle_expr(buffer, IO.chardata_to_string(mark), contents)
     generate_buffer(rest, buffer, scope, state)
   end

lib/eex/test/eex/smart_engine_test.exs

@@ -43,15 +43,6 @@ defmodule EEx.SmartEngineTest do
     assert_received :found
   end
 
-  test "error with unused \"do\" block without \"<%=\" modifier" do
-    stderr =
-      ExUnit.CaptureIO.capture_io(:stderr, fn ->
-        assert_eval("", "<% if true do %>I'm invisible!<% end %>", assigns: %{})
-      end)
-
-    assert stderr =~ "the contents of this expression won't be output"
-  end
-
   defp assert_eval(expected, actual, binding \\ []) do
     result = EEx.eval_string(actual, binding, file: __ENV__.file, engine: EEx.SmartEngine)
     assert result == expected

lib/eex/test/eex_test.exs

@@ -543,6 +543,16 @@ defmodule EExTest do
                ~s[unexpected beginning of EEx tag \"<%=\" on \"<%= end %>\"]
     end
 
+    test "unused \"do\" block without \"<%=\" modifier" do
+      assert ExUnit.CaptureIO.capture_io(:stderr, fn ->
+               EEx.compile_string("<% if true do %>I'm invisible!<% end %>")
+             end) =~ "the contents of this expression won't be output"
+
+      # These are fine though
+      EEx.compile_string("<% foo = fn -> %>Hello<% end %>")
+      EEx.compile_string("<% foo = if true do %>Hello<% end %>")
+    end
+
     test "from tokenizer" do
       warning =
         ExUnit.CaptureIO.capture_io(:stderr, fn ->
@@ -761,38 +771,31 @@ defmodule EExTest do
     end
 
     test "line and column meta" do
-      parser_options = Code.get_compiler_option(:parser_options)
-      Code.put_compiler_option(:parser_options, columns: true)
+      indentation = 12
 
-      try do
-        indentation = 12
+      ast =
+        EEx.compile_string(
+          """
+          <%= f() %> <% f() %>
+            <%= f fn -> %>
+              <%= f() %>
+            <% end %>
+          """,
+          indentation: indentation
+        )
 
-        ast =
-          EEx.compile_string(
-            """
-            <%= f() %> <% f() %>
-              <%= f fn -> %>
-                <%= f() %>
-              <% end %>
-            """,
-            indentation: indentation
-          )
+      {_, calls} =
+        Macro.prewalk(ast, [], fn
+          {:f, meta, _args} = expr, acc -> {expr, [meta | acc]}
+          other, acc -> {other, acc}
+        end)
 
-        {_, calls} =
-          Macro.prewalk(ast, [], fn
-            {:f, meta, _args} = expr, acc -> {expr, [meta | acc]}
-            other, acc -> {other, acc}
-          end)
-
-        assert Enum.reverse(calls) == [
-                 [line: 1, column: indentation + 5],
-                 [line: 1, column: indentation + 15],
-                 [line: 2, column: indentation + 7],
-                 [line: 3, column: indentation + 9]
-               ]
-      after
-        Code.put_compiler_option(:parser_options, parser_options)
-      end
+      assert Enum.reverse(calls) == [
+               [line: 1, column: indentation + 5],
+               [line: 1, column: indentation + 15],
+               [line: 2, column: indentation + 7],
+               [line: 3, column: indentation + 9]
+             ]
     end
   end
 

lib/elixir/lib/access.ex

@@ -35,61 +35,74 @@ defmodule Access do
       iex> nil[:a]
       nil
 
-  The access syntax can also be used with the `Kernel.put_in/2`,
-  `Kernel.update_in/2` and `Kernel.get_and_update_in/2` macros
-  to allow values to be set in nested data structures:
-
-      iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
-      iex> put_in(users["john"][:age], 28)
-      %{"john" => %{age: 28}, "meg" => %{age: 23}}
-
   ## Maps and structs
 
   While the access syntax is allowed in maps via `map[key]`,
   if your map is made of predefined atom keys, you should prefer
   to access those atom keys with `map.key` instead of `map[key]`,
   as `map.key` will raise if the key is missing (which is not
-  supposed to happen if the keys are predefined).
+  supposed to happen if the keys are predefined) or if `map` is
+  `nil`.
 
   Similarly, since structs are maps and structs have predefined
   keys, they only allow the `struct.key` syntax and they do not
-  allow the `struct[key]` access syntax. `Access.key/1` can also
-  be used to construct dynamic access to structs and maps.
+  allow the `struct[key]` access syntax.
 
-  In a nutshell, when using `put_in/2` and friends:
+  In other words, the `map[key]` syntax is loose, returning `nil`
+  for missing keys, while the `map.key` syntax is strict, raising
+  for both nil values and missing keys.
 
-      put_in(struct_or_map.key, :value)
-      put_in(keyword_or_map[:key], :value)
+  To bridge this gap, Elixir provides the `get_in/1` and `get_in/2`
+  functions, which are capable of traversing nested data structures,
+  even in the presence of `nil`s:
 
-  When using `put_in/3` and friends:
+      iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
+      iex> get_in(users["john"].age)
+      27
+      iex> get_in(users["unknown"].age)
+      nil
 
-      put_in(struct_or_map, [Access.key!(:key)], :value)
-      put_in(keyword_or_map, [:key], :value)
+  Notice how, even if no user was found, `get_in/1` returned `nil`.
+  Outside of `get_in/1`, trying to access the field `.age` on `nil`
+  would raise.
 
-  This covers the dual nature of maps in Elixir, as they can be
-  either for structured data or as a key-value store. See the `Map`
-  module for more information.
+  The `get_in/2` function takes one step further by allowing
+  different accessors to be mixed in. For example, given a user
+  map with the `:name` and `:languages` keys, here is how to
+  access the name of all programming languages:
 
-  ## Nested data structures
+        iex> languages = [
+        ...>   %{name: "elixir", type: :functional},
+        ...>   %{name: "c", type: :procedural}
+        ...> ]
+        iex> user = %{name: "john", languages: languages}
+        iex> get_in(user, [:languages, Access.all(), :name])
+        ["elixir", "c"]
 
-  Both key-based access syntaxes can be used with the nested update
-  functions and macros in `Kernel`, such as `Kernel.get_in/2`,
-  `Kernel.put_in/3`, `Kernel.update_in/3`, `Kernel.pop_in/2`, and
-  `Kernel.get_and_update_in/3`.
+  This module provides convenience functions for traversing other
+  structures, like tuples and lists. As we will see next, they can
+  even be used to update nested data structures.
 
-  For example, to update a map inside another map:
+  If you want to learn more about the dual nature of maps in Elixir,
+  as they can be either for structured data or as a key-value store,
+  see the `Map` module.
+
+  ## Updating nested data structures
+
+  The access syntax can also be used with the `Kernel.put_in/2`,
+  `Kernel.update_in/2`, `Kernel.get_and_update_in/2`, and `Kernel.pop_in/1`
+  macros to further manipulate values in nested data structures:
 
       iex> users = %{"john" => %{age: 27}, "meg" => %{age: 23}}
       iex> put_in(users["john"].age, 28)
       %{"john" => %{age: 28}, "meg" => %{age: 23}}
 
-  This module provides convenience functions for traversing other
-  structures, like tuples and lists. These functions can be used
-  in all the `Access`-related functions and macros in `Kernel`.
-
-  For instance, given a user map with the `:name` and `:languages`
-  keys, here is how to deeply traverse the map and convert all
-  language names to uppercase:
+  As shown in the previous section, you can also use the
+  `Kernel.put_in/3`, `Kernel.update_in/3`, `Kernel.pop_in/2`, and
+  `Kernel.get_and_update_in/3` functions to provide nested
+  custom accessors. For instance, given a user map with the
+  `:name` and `:languages` keys, here is how to deeply traverse
+  the map and convert all language names to uppercase:
 
       iex> languages = [
       ...>   %{name: "elixir", type: :functional},
@@ -552,7 +565,7 @@ defmodule Access do
       {"john", %{user: %{name: "JOHN"}}}
 
   However, it is not possible to remove fields using the dot notation,
-  as it is implified those fields must also be present. In any case,
+  as it is implied those fields must also be present. In any case,
   `Access.key!/1` is useful when the key is not known in advance
   and must be accessed dynamically.
 
@@ -1013,4 +1026,80 @@ defmodule Access do
   defp get_and_update_slice([], _range, _next, updates, gets, _index) do
     {:lists.reverse(gets), :lists.reverse(updates)}
   end
+
+  @doc ~S"""
+  Returns a function that accesses the first element of a list that matches the provided predicate.
+
+  The returned function is typically passed as an accessor to `Kernel.get_in/2`,
+  `Kernel.get_and_update_in/3`, and friends.
+
+  ## Examples
+
+      iex> list = [%{name: "john", salary: 10}, %{name: "francine", salary: 30}]
+      iex> get_in(list, [Access.find(&(&1.salary > 20)), :name])
+      "francine"
+      iex>  get_and_update_in(list, [Access.find(&(&1.salary <= 40)), :name], fn prev ->
+      ...> {prev, String.upcase(prev)}
+      ...>  end)
+      {"john", [%{name: "JOHN", salary: 10}, %{name: "francine", salary: 30}]}
+
+  `find/1` can also be used to pop the first found element out of a list or
+  a key inside of a list:
+
+      iex> list = [%{name: "john", salary: 10}, %{name: "francine", salary: 30}]
+      iex> pop_in(list, [Access.find(&(&1.salary <= 40))])
+      {%{name: "john", salary: 10}, [%{name: "francine", salary: 30}]}
+
+  When no match is found, nil is returned and the update function is never called
+
+      iex> list = [%{name: "john", salary: 10}, %{name: "francine", salary: 30}]
+      iex> get_in(list, [Access.find(&(&1.salary >= 50)), :name])
+      nil
+      iex> get_and_update_in(list, [Access.find(&(&1.salary >= 50)), :name], fn prev ->
+      ...>   {prev, String.upcase(prev)}
+      ...> end)
+      {nil, [%{name: "john", salary: 10}, %{name: "francine", salary: 30}]}
+
+  An error is raised if the predicate is not a function or is of the incorrect arity:
+
+      iex> get_in([], [Access.find(5)])
+      ** (FunctionClauseError) no function clause matching in Access.find/1
+
+  An error is raised if the accessed structure is not a list:
+
+      iex>  get_in(%{}, [Access.find(fn a -> a == 10 end)])
+      ** (RuntimeError) Access.find/1 expected a list, got: %{}
+  """
+  @doc since: "1.17.0"
+  @spec find((term -> as_boolean(term))) :: access_fun(data :: list, current_value :: term)
+  def find(predicate) when is_function(predicate, 1) do
+    fn op, data, next -> find(op, data, predicate, next) end
+  end
+
+  defp find(:get, data, predicate, next) when is_list(data) do
+    data |> Enum.find(predicate) |> next.()
+  end
+
+  defp find(:get_and_update, data, predicate, next) when is_list(data) do
+    get_and_update_find(data, [], predicate, next)
+  end
+
+  defp find(_op, data, _predicate, _next) do
+    raise "Access.find/1 expected a list, got: #{inspect(data)}"
+  end
+
+  defp get_and_update_find([], updates, _predicate, _next) do
+    {nil, :lists.reverse(updates)}
+  end
+
+  defp get_and_update_find([head | rest], updates, predicate, next) do
+    if predicate.(head) do
+      case next.(head) do
+        {get, update} -> {get, :lists.reverse([update | updates], rest)}
+        :pop -> {head, :lists.reverse(updates, rest)}
+      end
+    else
+      get_and_update_find(rest, [head | updates], predicate, next)
+    end
+  end
 end

lib/elixir/lib/application.ex

@@ -13,7 +13,7 @@ defmodule Application do
 
   Developers typically interact with the application environment and its
   callback module. Therefore those will be the topics we will cover first
-  before jumping into details about the application resource file and life-cycle.
+  before jumping into details about the application resource file and life cycle.
 
   ## The application environment
 
@@ -258,8 +258,8 @@ defmodule Application do
       Application.stop(:ex_unit)
       #=> :ok
 
-  Stopping an application without a callback module is defined, but except for
-  some system tracing, it is in practice a no-op.
+  Stopping an application without a callback module defined, is in practice a
+  no-op, except for some system tracing.
 
   Stopping an application with a callback module has three steps:
 
@@ -277,7 +277,7 @@ defmodule Application do
   invoked only after termination of the whole supervision tree.
 
   Shutting down a live system cleanly can be done by calling `System.stop/1`. It
-  will shut down every application in the opposite order they had been started.
+  will shut down every application in the reverse order they were started.
 
   By default, a SIGTERM from the operating system will automatically translate to
   `System.stop/0`. You can also have more explicit control over operating system
@@ -817,16 +817,29 @@ defmodule Application do
   end
 
   @doc """
-  Puts the environment for multiple apps at the same time.
+  Puts the environment for multiple applications at the same time.
 
   The given config should not:
 
     * have the same application listed more than once
     * have the same key inside the same application listed more than once
 
-  If those conditions are not met, it will raise.
+  If those conditions are not met, this function will raise.
+
+  This function receives the same options as `put_env/4`. Returns `:ok`.
+
+  ## Examples
+
+      Application.put_all_env(
+        my_app: [
+          key: :value,
+          another_key: :another_value
+        ],
+        another_app: [
+          key: :value
+        ]
+      )
 
-  It receives the same options as `put_env/4`. Returns `:ok`.
   """
   @doc since: "1.9.0"
   @spec put_all_env([{app, [{key, value}]}], timeout: timeout, persistent: boolean) :: :ok
@@ -850,7 +863,10 @@ defmodule Application do
 
   # TODO: Remove this deprecation warning on 2.0+ and allow list lookups as in compile_env.
   defp maybe_warn_on_app_env_key(app, key) do
-    message = "passing non-atom as application env key is deprecated, got: #{inspect(key)}"
+    message = fn ->
+      "passing non-atom as application env key is deprecated, got: #{inspect(key)}"
+    end
+
     IO.warn_once({Application, :key, app, key}, message, _stacktrace_drop_levels = 2)
   end
 
@@ -889,11 +905,12 @@ defmodule Application do
 
   ## Options
 
-    * `:type` - if the application should be started in `:permanent`,
-      `:temporary`, or `:transient`. See `t:restart_type/1` for more information.
+    * `:type` - if the application should be started `:temporary` (default),
+      `:permanent`, or `:transient`. See `t:restart_type/1` for more information.
 
     * `:mode` - (since v1.15.0) if the applications should be started serially
-      or concurrently. This option requires Erlang/OTP 26+.
+      (`:serial`, default) or concurrently (`:concurrent`). This option requires
+      Erlang/OTP 26+.
 
   """
   @spec ensure_all_started(app | [app], type: restart_type(), mode: :serial | :concurrent) ::

lib/elixir/lib/atom.ex

@@ -67,7 +67,7 @@ defmodule Atom do
   ## Examples
 
       iex> Atom.to_charlist(:"An atom")
-      'An atom'
+      ~c"An atom"
 
   """
   @spec to_charlist(atom) :: charlist

lib/elixir/lib/bitwise.ex

@@ -2,7 +2,7 @@ defmodule Bitwise do
   @moduledoc """
   A set of functions that perform calculations on bits.
 
-  All bitwise functions work only on integers; otherwise an
+  All bitwise functions work only on integers, otherwise an
   `ArithmeticError` is raised. The functions `band/2`,
   `bor/2`, `bsl/2`, and `bsr/2` also have operators,
   respectively: `&&&/2`, `|||/2`, `<<</2`, and `>>>/2`.

lib/elixir/lib/calendar.ex

@@ -70,14 +70,14 @@ defmodule Calendar do
   @type zone_abbr :: String.t()
 
   @typedoc """
-  The time zone UTC offset in seconds for standard time.
+  The time zone UTC offset in ISO seconds for standard time.
 
   See also `t:std_offset/0`.
   """
   @type utc_offset :: integer
 
   @typedoc """
-  The time zone standard offset in seconds (typically not zero in summer times).
+  The time zone standard offset in ISO seconds (typically not zero in summer times).
 
   It must be added to `t:utc_offset/0` to get the total offset from UTC used for "wall time".
   """
@@ -338,6 +338,34 @@ defmodule Calendar do
   @doc since: "1.15.0"
   @callback iso_days_to_end_of_day(iso_days) :: iso_days
 
+  @doc """
+  Shifts date by given duration according to its calendar.
+  """
+  @doc since: "1.17.0"
+  @callback shift_date(year, month, day, Duration.t()) :: {year, month, day}
+
+  @doc """
+  Shifts naive datetime by given duration according to its calendar.
+  """
+  @doc since: "1.17.0"
+  @callback shift_naive_datetime(
+              year,
+              month,
+              day,
+              hour,
+              minute,
+              second,
+              microsecond,
+              Duration.t()
+            ) :: {year, month, day, hour, minute, second, microsecond}
+
+  @doc """
+  Shifts time by given duration according to its calendar.
+  """
+  @doc since: "1.17.0"
+  @callback shift_time(hour, minute, second, microsecond, Duration.t()) ::
+              {hour, minute, second, microsecond}
+
   # General Helpers
 
   @doc """
@@ -705,7 +733,7 @@ defmodule Calendar do
 
   # Hour using a 12-hour clock
   defp format_modifiers("I" <> rest, width, pad, datetime, format_options, acc) do
-    result = (rem(datetime.hour() + 23, 12) + 1) |> Integer.to_string() |> pad_leading(width, pad)
+    result = (rem(datetime.hour + 23, 12) + 1) |> Integer.to_string() |> pad_leading(width, pad)
     parse(rest, datetime, format_options, [result | acc])
   end
 

lib/elixir/lib/calendar/date.ex

@@ -40,7 +40,7 @@ defmodule Date do
 
   ## Using epochs
 
-  The `add/2` and `diff/2` functions can be used for computing dates
+  The `add/2`, `diff/2` and `shift/2` functions can be used for computing dates
   or retrieving the number of days between instants. For example, if there
   is an interest in computing the number of days from the Unix epoch
   (1970-01-01):
@@ -51,6 +51,9 @@ defmodule Date do
       iex> Date.add(~D[1970-01-01], 14716)
       ~D[2010-04-17]
 
+      iex> Date.shift(~D[1970-01-01], year: 40, month: 3, week: 2, day: 2)
+      ~D[2010-04-17]
+
   Those functions are optimized to deal with common epochs, such
   as the Unix Epoch above or the Gregorian Epoch (0000-01-01).
   """
@@ -687,6 +690,8 @@ defmodule Date do
   The days are counted as Gregorian days. The date is returned in the same
   calendar as it was given in.
 
+  To shift a date by a `Duration` and according to its underlying calendar, use `Date.shift/2`.
+
   ## Examples
 
       iex> Date.add(~D[2000-01-03], -2)
@@ -703,12 +708,7 @@ defmodule Date do
   @spec add(Calendar.date(), integer()) :: t
   def add(%{calendar: Calendar.ISO} = date, days) do
     %{year: year, month: month, day: day} = date
-
-    {year, month, day} =
-      Calendar.ISO.date_to_iso_days(year, month, day)
-      |> Kernel.+(days)
-      |> Calendar.ISO.date_from_iso_days()
-
+    {year, month, day} = Calendar.ISO.shift_days({year, month, day}, days)
     %Date{calendar: Calendar.ISO, year: year, month: month, day: day}
   end
 
@@ -757,6 +757,81 @@ defmodule Date do
     end
   end
 
+  @doc """
+  Shifts given `date` by `duration` according to its calendar.
+
+  Allowed units are: `:year`, `:month`, `:week`, `:day`.
+
+  When using the default ISO calendar, durations are collapsed and
+  applied in the order of months and then days:
+
+  * when shifting by 1 year and 2 months the date is actually shifted by 14 months
+  * when shifting by 2 weeks and 3 days the date is shifted by 17 days
+
+  When shifting by month, days are rounded down to the nearest valid date.
+
+  Raises an `ArgumentError` when called with time scale units.
+
+  ## Examples
+
+      iex> Date.shift(~D[2016-01-03], month: 2)
+      ~D[2016-03-03]
+      iex> Date.shift(~D[2016-01-30], month: -1)
+      ~D[2015-12-30]
+      iex> Date.shift(~D[2016-01-31], year: 4, day: 1)
+      ~D[2020-02-01]
+      iex> Date.shift(~D[2016-01-03], Duration.new!(month: 2))
+      ~D[2016-03-03]
+
+      # leap years
+      iex> Date.shift(~D[2024-02-29], year: 1)
+      ~D[2025-02-28]
+      iex> Date.shift(~D[2024-02-29], year: 4)
+      ~D[2028-02-29]
+
+      # rounding down
+      iex> Date.shift(~D[2015-01-31], month: 1)
+      ~D[2015-02-28]
+
+  """
+  @doc since: "1.17.0"
+  @spec shift(Calendar.date(), Duration.t() | [unit_pair]) :: t
+        when unit_pair: {:year, integer} | {:month, integer} | {:week, integer} | {:day, integer}
+  def shift(%{calendar: calendar} = date, duration) do
+    %{year: year, month: month, day: day} = date
+    {year, month, day} = calendar.shift_date(year, month, day, __duration__!(duration))
+    %Date{calendar: calendar, year: year, month: month, day: day}
+  end
+
+  @doc false
+  def __duration__!(%Duration{} = duration) do
+    duration
+  end
+
+  # This part is inlined by the compiler on constant values
+  def __duration__!(unit_pairs) do
+    Enum.each(unit_pairs, &validate_duration_unit!/1)
+    struct!(Duration, unit_pairs)
+  end
+
+  defp validate_duration_unit!({unit, _value})
+       when unit in [:hour, :minute, :second, :microsecond] do
+    raise ArgumentError, "unsupported unit #{inspect(unit)}. Expected :year, :month, :week, :day"
+  end
+
+  defp validate_duration_unit!({unit, _value}) when unit not in [:year, :month, :week, :day] do
+    raise ArgumentError, "unknown unit #{inspect(unit)}. Expected :year, :month, :week, :day"
+  end
+
+  defp validate_duration_unit!({_unit, value}) when is_integer(value) do
+    :ok
+  end
+
+  defp validate_duration_unit!({unit, value}) do
+    raise ArgumentError,
+          "unsupported value #{inspect(value)} for #{inspect(unit)}. Expected an integer"
+  end
+
   @doc false
   def to_iso_days(%{calendar: Calendar.ISO, year: year, month: month, day: day}) do
     {Calendar.ISO.date_to_iso_days(year, month, day), {0, 86_400_000_000}}
@@ -1083,10 +1158,20 @@ defmodule Date do
   end
 
   defimpl Inspect do
-    def inspect(%{calendar: calendar, year: year, month: month, day: day}, _) do
+    def inspect(%{calendar: calendar, year: year, month: month, day: day}, _)
+        when year in -9999..9999 do
       "~D[" <> calendar.date_to_string(year, month, day) <> suffix(calendar) <> "]"
     end
 
+    def inspect(%{calendar: calendar, year: year, month: month, day: day}, _)
+        when calendar == Calendar.ISO do
+      "Date.new!(#{Integer.to_string(year)}, #{Integer.to_string(month)}, #{Integer.to_string(day)})"
+    end
+
+    def inspect(%{calendar: calendar, year: year, month: month, day: day}, _) do
+      "Date.new!(#{Integer.to_string(year)}, #{Integer.to_string(month)}, #{Integer.to_string(day)}, #{inspect(calendar)})"
+    end
+
     defp suffix(Calendar.ISO), do: ""
     defp suffix(calendar), do: " " <> inspect(calendar)
   end

lib/elixir/lib/calendar/datetime.ex

@@ -60,7 +60,7 @@ defmodule DateTime do
   a date and time at a given time zone". To understand precisely
   what we mean, let's see an example.
 
-  Imagine someone in Poland wants to schedule a meeting with someone
+  Imagine someone in Poland who wants to schedule a meeting with someone
   in Brazil in the next year. The meeting will happen at 2:30 AM
   in the Polish time zone. At what time will the meeting happen in
   Brazil?
@@ -83,14 +83,14 @@ defmodule DateTime do
   not a problem, because time zone rules do not change for past
   events.
 
-  To make matters worse, it may be that the 2:30 AM in Polish time
+  To make matters worse, it may be that 2:30 AM in Polish time
   does not actually even exist or it is ambiguous. If a certain
   time zone observes "Daylight Saving Time", they will move their
   clock forward once a year. When this happens, there is a whole
   hour that does not exist. Then, when they move the clock back,
   there is a certain hour that will happen twice. So if you want to
   schedule a meeting when this shift back happens, you would need to
-  explicitly say which occurence of 2:30 AM you mean: the one in
+  explicitly say which occurrence of 2:30 AM you mean: the one in
   "Summer Time", which occurs before the shift, or the one
   in "Standard Time", which occurs after it. Applications that are
   date and time sensitive need to take these scenarios into account
@@ -170,6 +170,9 @@ defmodule DateTime do
   truncate the resulting datetime. This is available
   since v1.15.0.
 
+  The default unit if none gets passed is `:native`,
+  which results on a default resolution of microseconds.
+
   ## Examples
 
       iex> datetime = DateTime.utc_now()
@@ -894,7 +897,7 @@ defmodule DateTime do
       -17412508655
 
   """
-  @spec to_unix(Calendar.datetime(), System.time_unit()) :: integer
+  @spec to_unix(Calendar.datetime(), :native | System.time_unit()) :: integer
   def to_unix(datetime, unit \\ :second)
 
   def to_unix(%{utc_offset: utc_offset, std_offset: std_offset} = datetime, unit) do
@@ -1494,6 +1497,11 @@ defmodule DateTime do
 
   ## Examples
 
+      iex> DateTime.diff(~U[2024-01-15 10:00:10Z], ~U[2024-01-15 10:00:00Z])
+      10
+
+  This function also considers timezone offsets:
+
       iex> dt1 = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
       ...>                 hour: 23, minute: 0, second: 7, microsecond: {0, 0},
       ...>                 utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
@@ -1605,6 +1613,8 @@ defmodule DateTime do
       iex> result.microsecond
       {21000, 3}
 
+  To shift a datetime by a `Duration` and according to its underlying calendar, use `DateTime.shift/3`.
+
   """
   @doc since: "1.8.0"
   @spec add(
@@ -1633,30 +1643,28 @@ defmodule DateTime do
     add(datetime, amount_to_add * 60, :second, time_zone_database)
   end
 
-  def add(datetime, amount_to_add, unit, time_zone_database) when is_integer(amount_to_add) do
+  def add(%{calendar: calendar} = datetime, amount_to_add, unit, time_zone_database)
+      when is_integer(amount_to_add) do
     %{
+      microsecond: {_, precision},
+      time_zone: time_zone,
       utc_offset: utc_offset,
-      std_offset: std_offset,
-      calendar: calendar,
-      microsecond: {_, precision}
+      std_offset: std_offset
     } = datetime
 
-    if not is_integer(unit) and
-         unit not in ~w(second millisecond microsecond nanosecond)a do
+    if not is_integer(unit) and unit not in ~w(second millisecond microsecond nanosecond)a do
       raise ArgumentError,
             "unsupported time unit. Expected :day, :hour, :minute, :second, :millisecond, :microsecond, :nanosecond, or a positive integer, got #{inspect(unit)}"
     end
 
-    ppd = System.convert_time_unit(86400, :second, unit)
-    total_offset = System.convert_time_unit(utc_offset + std_offset, :second, unit)
     precision = max(Calendar.ISO.time_unit_to_precision(unit), precision)
 
     result =
       datetime
       |> to_iso_days()
-      # Subtract total offset in order to get UTC and add the integer for the addition
-      |> Calendar.ISO.add_day_fraction_to_iso_days(amount_to_add - total_offset, ppd)
-      |> shift_zone_for_iso_days_utc(calendar, precision, datetime.time_zone, time_zone_database)
+      |> Calendar.ISO.shift_time_unit(amount_to_add, unit)
+      |> apply_tz_offset(utc_offset + std_offset)
+      |> shift_zone_for_iso_days_utc(calendar, precision, time_zone, time_zone_database)
 
     case result do
       {:ok, result_datetime} ->
@@ -1669,6 +1677,152 @@ defmodule DateTime do
     end
   end
 
+  @doc """
+  Shifts given `datetime` by `duration` according to its calendar.
+
+  Allowed units are: `:year`, `:month`, `:week`, `:day`, `:hour`, `:minute`, `:second`, `:microsecond`.
+
+  This operation is equivalent to shifting the datetime wall clock
+  (in other words, the value as someone in that timezone would see
+  on their watch), then applying the time zone offset to convert it
+  to UTC, and finally computing the new timezone in case of shifts.
+  This ensures `shift/3` always returns a valid datetime.
+
+  On the other hand, time zones that observe "Daylight Saving Time"
+  or other changes, across summer/winter time will add/remove hours
+  from the resulting datetime:
+
+      dt = DateTime.new!(~D[2019-03-31], ~T[01:00:00], "Europe/Copenhagen")
+      DateTime.shift(dt, hour: 1)
+      #=> #DateTime<2019-03-31 03:00:00+02:00 CEST Europe/Copenhagen>
+
+      dt = DateTime.new!(~D[2018-11-04], ~T[00:00:00], "America/Los_Angeles")
+      DateTime.shift(dt, hour: 2)
+      #=> #DateTime<2018-11-04 01:00:00-08:00 PST America/Los_Angeles>
+
+  In case you don't want these changes to happen automatically or you
+  want to surface time zone conflicts to the user, you can shift
+  the datetime as a naive datetime and then use `from_naive/2`:
+
+      dt |> NaiveDateTime.shift(duration) |> DateTime.from_naive(dt.time_zone)
+
+  When using the default ISO calendar, durations are collapsed and
+  applied in the order of months, then seconds and microseconds:
+
+  * when shifting by 1 year and 2 months the date is actually shifted by 14 months
+  * weeks, days and smaller units are collapsed into seconds and microseconds
+
+  When shifting by month, days are rounded down to the nearest valid date.
+
+  ## Examples
+
+      iex> DateTime.shift(~U[2016-01-01 00:00:00Z], month: 2)
+      ~U[2016-03-01 00:00:00Z]
+      iex> DateTime.shift(~U[2016-01-01 00:00:00Z], year: 1, week: 4)
+      ~U[2017-01-29 00:00:00Z]
+      iex> DateTime.shift(~U[2016-01-01 00:00:00Z], minute: -25)
+      ~U[2015-12-31 23:35:00Z]
+      iex> DateTime.shift(~U[2016-01-01 00:00:00Z], minute: 5, microsecond: {500, 4})
+      ~U[2016-01-01 00:05:00.0005Z]
+
+      # leap years
+      iex> DateTime.shift(~U[2024-02-29 00:00:00Z], year: 1)
+      ~U[2025-02-28 00:00:00Z]
+      iex> DateTime.shift(~U[2024-02-29 00:00:00Z], year: 4)
+      ~U[2028-02-29 00:00:00Z]
+
+      # rounding down
+      iex> DateTime.shift(~U[2015-01-31 00:00:00Z], month: 1)
+      ~U[2015-02-28 00:00:00Z]
+
+  """
+  @doc since: "1.17.0"
+  @spec shift(Calendar.datetime(), Duration.duration(), Calendar.time_zone_database()) :: t
+  def shift(datetime, duration, time_zone_database \\ Calendar.get_time_zone_database())
+
+  def shift(%{calendar: calendar, time_zone: "Etc/UTC"} = datetime, duration, _time_zone_database) do
+    %{
+      year: year,
+      month: month,
+      day: day,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond
+    } = datetime
+
+    {year, month, day, hour, minute, second, microsecond} =
+      calendar.shift_naive_datetime(
+        year,
+        month,
+        day,
+        hour,
+        minute,
+        second,
+        microsecond,
+        __duration__!(duration)
+      )
+
+    %DateTime{
+      year: year,
+      month: month,
+      day: day,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond,
+      time_zone: "Etc/UTC",
+      zone_abbr: "UTC",
+      std_offset: 0,
+      utc_offset: 0
+    }
+  end
+
+  def shift(%{calendar: calendar} = datetime, duration, time_zone_database) do
+    %{
+      year: year,
+      month: month,
+      day: day,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond,
+      std_offset: std_offset,
+      utc_offset: utc_offset,
+      time_zone: time_zone
+    } = datetime
+
+    {year, month, day, hour, minute, second, {_, precision} = microsecond} =
+      calendar.shift_naive_datetime(
+        year,
+        month,
+        day,
+        hour,
+        minute,
+        second,
+        microsecond,
+        __duration__!(duration)
+      )
+
+    result =
+      calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
+      |> apply_tz_offset(utc_offset + std_offset)
+      |> shift_zone_for_iso_days_utc(calendar, precision, time_zone, time_zone_database)
+
+    case result do
+      {:ok, result_datetime} ->
+        result_datetime
+
+      {:error, error} ->
+        raise ArgumentError,
+              "cannot shift #{inspect(datetime)} to #{inspect(duration)} (with time zone " <>
+                "database #{inspect(time_zone_database)}), reason: #{inspect(error)}"
+    end
+  end
+
+  @doc false
+  defdelegate __duration__!(params), to: Duration, as: :new!
+
   @doc """
   Returns the given datetime with the microsecond field truncated to the given
   precision (`:microsecond`, `:millisecond` or `:second`).

lib/elixir/lib/calendar/duration.ex

@@ -0,0 +1,436 @@
+defmodule Duration do
+  @moduledoc """
+  Struct and functions for handling durations.
+
+  A `Duration` struct represents a collection of time scale units,
+  allowing for manipulation and calculation of durations.
+
+  Date and time scale units are represented as integers, allowing for
+  both positive and negative values.
+
+  Microseconds are represented using a tuple `{microsecond, precision}`.
+  This ensures compatibility with other calendar types implementing time,
+  such as `Time`, `DateTime`, and `NaiveDateTime`.
+
+  ## Shifting
+
+  The most common use of durations in Elixir's standard library is to
+  "shift" the calendar types.
+
+      iex> Date.shift(~D[2016-01-03], month: 2)
+      ~D[2016-03-03]
+
+  In the example above, `Date.shift/2` automatically converts the units
+  into a `Duration` struct, although one can also be given directly:
+
+      iex> Date.shift(~D[2016-01-03], Duration.new!(month: 2))
+      ~D[2016-03-03]
+
+  It is important to note that shifting is not an arithmetic operation.
+  For example, adding `date + 1 month + 1 month` does not yield the same
+  result as `date + 2 months`. Let's see an example:
+
+      iex> ~D[2016-01-31] |> Date.shift(month: 1) |> Date.shift(month: 1)
+      ~D[2016-03-29]
+
+      iex> ~D[2016-01-31] |> Date.shift(month: 2)
+      ~D[2016-03-31]
+
+  As you can see above, the results differ, which explains why operations
+  with durations are called "shift" rather than "add". This happens because,
+  once we add one month to `2016-01-31`, we get `2016-02-29`. Then adding
+  one extra month gives us `2016-03-29` instead of `2016-03-31`.
+
+  In particular, when applying durations to `Calendar.ISO` types:
+
+    * larger units (such as years and months) are applied before
+      smaller ones (such as weeks, hours, days, and so on)
+
+    * units are collapsed into months (`:year` and `:month`),
+      seconds (`:week`, `:day`, `:hour`, `:minute`, `:second`)
+      and microseconds (`:microsecond`) before they are applied
+
+    * 1 year is equivalent to 12 months, 1 week is equivalent to 7 days.
+      Therefore, 4 weeks _are not_ equivalent to 1 month
+
+    * in case of non-existing dates, the results are rounded down to the
+      nearest valid date
+
+  As the `shift/2` functions are calendar aware, they are guaranteed to return
+  valid date/times, considering leap years as well as DST in applicable time zones.
+
+  ## Intervals
+
+  Durations in Elixir can be combined with stream operations to build intervals.
+  For example, to retrieve the next three Wednesdays starting from 17th April, 2024:
+
+      iex> ~D[2024-04-17] |> Stream.iterate(&Date.shift(&1, week: 1)) |> Enum.take(3)
+      [~D[2024-04-17], ~D[2024-04-24], ~D[2024-05-01]]
+
+  However, once again, it is important to remember that shifting a duration is not
+  arithmetic, so you may want to use the functions in this module depending on what
+  you to achieve. Compare the results of both examples below:
+
+      # Adding one month after the other
+      iex> date = ~D[2016-01-31]
+      iex> duration = Duration.new!(month: 1)
+      iex> stream = Stream.iterate(date, fn prev_date -> Date.shift(prev_date, duration) end)
+      iex> Enum.take(stream, 3)
+      [~D[2016-01-31], ~D[2016-02-29], ~D[2016-03-29]]
+
+      # Multiplying durations by an index
+      iex> date = ~D[2016-01-31]
+      iex> duration = Duration.new!(month: 1)
+      iex> stream = Stream.from_index(fn i -> Date.shift(date, Duration.multiply(duration, i)) end)
+      iex> Enum.take(stream, 3)
+      [~D[2016-01-31], ~D[2016-02-29], ~D[2016-03-31]]
+
+  The second example consistently points to the last day of the month,
+  as it performs operations on the duration, rather than shifting date
+  after date.
+  """
+
+  @moduledoc since: "1.17.0"
+
+  @derive {Inspect, optional: [:year, :month, :week, :day, :hour, :minute, :second, :microsecond]}
+  defstruct year: 0,
+            month: 0,
+            week: 0,
+            day: 0,
+            hour: 0,
+            minute: 0,
+            second: 0,
+            microsecond: {0, 0}
+
+  @typedoc """
+  The duration struct type.
+  """
+  @type t :: %Duration{
+          year: integer,
+          month: integer,
+          week: integer,
+          day: integer,
+          hour: integer,
+          minute: integer,
+          second: integer,
+          microsecond: {integer, 0..6}
+        }
+
+  @typedoc """
+  The unit pair type specifies a pair of a valid duration unit key and value.
+  """
+  @type unit_pair ::
+          {:year, integer}
+          | {:month, integer}
+          | {:week, integer}
+          | {:day, integer}
+          | {:hour, integer}
+          | {:minute, integer}
+          | {:second, integer}
+          | {:microsecond, {integer, 0..6}}
+
+  @typedoc """
+  The duration type specifies a `%Duration{}` struct or a keyword list of valid duration unit pairs.
+  """
+  @type duration :: t | [unit_pair]
+
+  @microseconds_per_second 1_000_000
+
+  @doc """
+  Creates a new `Duration` struct from given `unit_pairs`.
+
+  Raises an `ArgumentError` when called with invalid unit pairs.
+
+  ## Examples
+
+      iex> Duration.new!(year: 1, week: 3, hour: 4, second: 1)
+      %Duration{year: 1, week: 3, hour: 4, second: 1}
+      iex> Duration.new!(second: 1, microsecond: {1000, 6})
+      %Duration{second: 1, microsecond: {1000, 6}}
+      iex> Duration.new!(month: 2)
+      %Duration{month: 2}
+
+  """
+  @spec new!(duration()) :: t
+  def new!(%Duration{} = duration) do
+    duration
+  end
+
+  def new!(unit_pairs) do
+    Enum.each(unit_pairs, &validate_unit!/1)
+    struct!(Duration, unit_pairs)
+  end
+
+  defp validate_unit!({:microsecond, {ms, precision}})
+       when is_integer(ms) and precision in 0..6 do
+    :ok
+  end
+
+  defp validate_unit!({:microsecond, microsecond}) do
+    raise ArgumentError,
+          "unsupported value #{inspect(microsecond)} for :microsecond. Expected a tuple {ms, precision} where precision is an integer from 0 to 6"
+  end
+
+  defp validate_unit!({unit, _value})
+       when unit not in [:year, :month, :week, :day, :hour, :minute, :second] do
+    raise ArgumentError,
+          "unknown unit #{inspect(unit)}. Expected :year, :month, :week, :day, :hour, :minute, :second, :microsecond"
+  end
+
+  defp validate_unit!({_unit, value}) when is_integer(value) do
+    :ok
+  end
+
+  defp validate_unit!({unit, value}) do
+    raise ArgumentError,
+          "unsupported value #{inspect(value)} for #{inspect(unit)}. Expected an integer"
+  end
+
+  @doc """
+  Adds units of given durations `d1` and `d2`.
+
+  Respects the the highest microsecond precision of the two.
+
+  ## Examples
+
+      iex> Duration.add(Duration.new!(week: 2, day: 1), Duration.new!(day: 2))
+      %Duration{week: 2, day: 3}
+      iex> Duration.add(Duration.new!(microsecond: {400, 3}), Duration.new!(microsecond: {600, 6}))
+      %Duration{microsecond: {1000, 6}}
+
+  """
+  @spec add(t, t) :: t
+  def add(%Duration{} = d1, %Duration{} = d2) do
+    {m1, p1} = d1.microsecond
+    {m2, p2} = d2.microsecond
+
+    %Duration{
+      year: d1.year + d2.year,
+      month: d1.month + d2.month,
+      week: d1.week + d2.week,
+      day: d1.day + d2.day,
+      hour: d1.hour + d2.hour,
+      minute: d1.minute + d2.minute,
+      second: d1.second + d2.second,
+      microsecond: {m1 + m2, max(p1, p2)}
+    }
+  end
+
+  @doc """
+  Subtracts units of given durations `d1` and `d2`.
+
+  Respects the the highest microsecond precision of the two.
+
+  ## Examples
+
+      iex> Duration.subtract(Duration.new!(week: 2, day: 1), Duration.new!(day: 2))
+      %Duration{week: 2, day: -1}
+      iex> Duration.subtract(Duration.new!(microsecond: {400, 6}), Duration.new!(microsecond: {600, 3}))
+      %Duration{microsecond: {-200, 6}}
+
+  """
+  @spec subtract(t, t) :: t
+  def subtract(%Duration{} = d1, %Duration{} = d2) do
+    {m1, p1} = d1.microsecond
+    {m2, p2} = d2.microsecond
+
+    %Duration{
+      year: d1.year - d2.year,
+      month: d1.month - d2.month,
+      week: d1.week - d2.week,
+      day: d1.day - d2.day,
+      hour: d1.hour - d2.hour,
+      minute: d1.minute - d2.minute,
+      second: d1.second - d2.second,
+      microsecond: {m1 - m2, max(p1, p2)}
+    }
+  end
+
+  @doc """
+  Multiplies `duration` units by given `integer`.
+
+  ## Examples
+
+      iex> Duration.multiply(Duration.new!(day: 1, minute: 15, second: -10), 3)
+      %Duration{day: 3, minute: 45, second: -30}
+      iex> Duration.multiply(Duration.new!(microsecond: {200, 4}), 3)
+      %Duration{microsecond: {600, 4}}
+
+  """
+  @spec multiply(t, integer) :: t
+  def multiply(%Duration{microsecond: {ms, p}} = duration, integer) when is_integer(integer) do
+    %Duration{
+      year: duration.year * integer,
+      month: duration.month * integer,
+      week: duration.week * integer,
+      day: duration.day * integer,
+      hour: duration.hour * integer,
+      minute: duration.minute * integer,
+      second: duration.second * integer,
+      microsecond: {ms * integer, p}
+    }
+  end
+
+  @doc """
+  Negates `duration` units.
+
+  ## Examples
+
+      iex> Duration.negate(Duration.new!(day: 1, minute: 15, second: -10))
+      %Duration{day: -1, minute: -15, second: 10}
+      iex> Duration.negate(Duration.new!(microsecond: {500000, 4}))
+      %Duration{microsecond: {-500000, 4}}
+
+  """
+  @spec negate(t) :: t
+  def negate(%Duration{microsecond: {ms, p}} = duration) do
+    %Duration{
+      year: -duration.year,
+      month: -duration.month,
+      week: -duration.week,
+      day: -duration.day,
+      hour: -duration.hour,
+      minute: -duration.minute,
+      second: -duration.second,
+      microsecond: {-ms, p}
+    }
+  end
+
+  @doc """
+  Parses an [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601#Durations) formatted duration string to a `Duration` struct.
+
+  Duration strings, as well as individual units, may be prefixed with plus/minus signs so that:
+
+  - `-PT6H3M` parses as `%Duration{hour: -6, minute: -3}`
+  - `-PT6H-3M` parses as `%Duration{hour: -6, minute: 3}`
+  - `+PT6H3M` parses as `%Duration{hour: 6, minute: 3}`
+  - `+PT6H-3M` parses as `%Duration{hour: 6, minute: -3}`
+
+  Duration designators must be provided in order of magnitude: `P[n]Y[n]M[n]W[n]DT[n]H[n]M[n]S`.
+
+  Only seconds may be specified with a decimal fraction, using either a comma or a full stop: `P1DT4,5S`.
+
+  ## Examples
+
+      iex> Duration.from_iso8601("P1Y2M3DT4H5M6S")
+      {:ok, %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6}}
+      iex> Duration.from_iso8601("P3Y-2MT3H")
+      {:ok, %Duration{year: 3, month: -2, hour: 3}}
+      iex> Duration.from_iso8601("-PT10H-30M")
+      {:ok, %Duration{hour: -10, minute: 30}}
+      iex> Duration.from_iso8601("PT4.650S")
+      {:ok, %Duration{second: 4, microsecond: {650000, 3}}}
+
+  """
+  @spec from_iso8601(String.t()) :: {:ok, t} | {:error, atom}
+  def from_iso8601(string) when is_binary(string) do
+    case Calendar.ISO.parse_duration(string) do
+      {:ok, duration} ->
+        {:ok, new!(duration)}
+
+      error ->
+        error
+    end
+  end
+
+  @doc """
+  Same as `from_iso8601/1` but raises an `ArgumentError`.
+
+  ## Examples
+
+      iex> Duration.from_iso8601!("P1Y2M3DT4H5M6S")
+      %Duration{year: 1, month: 2, day: 3, hour: 4, minute: 5, second: 6}
+      iex> Duration.from_iso8601!("P10D")
+      %Duration{day: 10}
+
+  """
+  @spec from_iso8601!(String.t()) :: t
+  def from_iso8601!(string) when is_binary(string) do
+    case from_iso8601(string) do
+      {:ok, duration} ->
+        duration
+
+      {:error, reason} ->
+        raise ArgumentError, ~s/failed to parse duration "#{string}". reason: #{inspect(reason)}/
+    end
+  end
+
+  @doc """
+  Converts the given `duration` to an [ISO 8601-2:2019](https://en.wikipedia.org/wiki/ISO_8601) formatted string.
+
+  Note this function implements the *extension* of ISO 8601:2019. This extensions allows weeks to
+  appear between months and days: `P3M3W3D`, making it fully compatible with any `Duration` struct.
+
+  ## Examples
+
+      iex> Duration.to_iso8601(Duration.new!(year: 3))
+      "P3Y"
+      iex> Duration.to_iso8601(Duration.new!(day: 40, hour: 12, minute: 42, second: 12))
+      "P40DT12H42M12S"
+      iex> Duration.to_iso8601(Duration.new!(second: 30))
+      "PT30S"
+
+      iex> Duration.to_iso8601(Duration.new!([]))
+      "PT0S"
+
+      iex> Duration.to_iso8601(Duration.new!(second: 1, microsecond: {2_200, 3}))
+      "PT1.002S"
+      iex> Duration.to_iso8601(Duration.new!(second: 1, microsecond: {-1_200_000, 4}))
+      "PT-0.2000S"
+  """
+
+  @spec to_iso8601(t) :: String.t()
+  def to_iso8601(%Duration{} = duration) do
+    case {to_iso8601_duration_date(duration), to_iso8601_duration_time(duration)} do
+      {[], []} -> "PT0S"
+      {date, time} -> IO.iodata_to_binary([?P, date, time])
+    end
+  end
+
+  defp to_iso8601_duration_date(%{year: 0, month: 0, week: 0, day: 0}) do
+    []
+  end
+
+  defp to_iso8601_duration_date(%{year: year, month: month, week: week, day: day}) do
+    [pair(year, ?Y), pair(month, ?M), pair(week, ?W), pair(day, ?D)]
+  end
+
+  defp to_iso8601_duration_time(%{hour: 0, minute: 0, second: 0, microsecond: {0, _}}) do
+    []
+  end
+
+  defp to_iso8601_duration_time(%{hour: hour, minute: minute} = d) do
+    [?T, pair(hour, ?H), pair(minute, ?M), second_component(d)]
+  end
+
+  defp second_component(%{second: 0, microsecond: {0, _}}) do
+    []
+  end
+
+  defp second_component(%{second: 0, microsecond: {_, 0}}) do
+    ~c"0S"
+  end
+
+  defp second_component(%{second: second, microsecond: {_, 0}}) do
+    [Integer.to_string(second), ?S]
+  end
+
+  defp second_component(%{second: second, microsecond: {ms, p}}) do
+    total_ms = second * @microseconds_per_second + ms
+    second = total_ms |> div(@microseconds_per_second) |> abs()
+    ms = total_ms |> rem(@microseconds_per_second) |> abs()
+    sign = if total_ms < 0, do: ?-, else: []
+
+    [
+      sign,
+      Integer.to_string(second),
+      ?.,
+      ms |> Integer.to_string() |> String.pad_leading(6, "0") |> binary_part(0, p),
+      ?S
+    ]
+  end
+
+  @compile {:inline, pair: 2}
+  defp pair(0, _key), do: []
+  defp pair(num, key), do: [Integer.to_string(num), key]
+end

lib/elixir/lib/calendar/iso.ex

@@ -18,7 +18,8 @@ defmodule Calendar.ISO do
 
   The standard library supports a minimal set of possible ISO 8601 features.
   Specifically, the parser only supports calendar dates and does not support
-  ordinal and week formats.
+  ordinal and week formats. Additionally, it supports parsing ISO 8601
+  formatted durations, including negative time units and fractional seconds.
 
   By default Elixir only parses extended-formatted date/times. You can opt-in
   to parse basic-formatted date/times.
@@ -29,7 +30,7 @@ defmodule Calendar.ISO do
 
   Elixir does not support reduced accuracy formats (for example, a date without
   the day component) nor decimal precisions in the lowest component (such as
-  `10:01:25,5`). No functions exist to parse ISO 8601 durations or time intervals.
+  `10:01:25,5`).
 
   #### Examples
 
@@ -230,9 +231,9 @@ defmodule Calendar.ISO do
       ]
     end
 
-  defguardp is_year(year) when year in -9999..9999
-  defguardp is_year_BCE(year) when year in -9999..0
-  defguardp is_year_CE(year) when year in 1..9999
+  defguardp is_year(year) when is_integer(year)
+  defguardp is_year_BCE(year) when year <= 0
+  defguardp is_year_CE(year) when year >= 1
   defguardp is_month(month) when month in 1..12
   defguardp is_day(day) when day in 1..31
   defguardp is_hour(hour) when hour in 0..23
@@ -663,6 +664,79 @@ defmodule Calendar.ISO do
     end
   end
 
+  @doc """
+  Parses an ISO 8601 formatted duration string to a list of `Duration` compabitble unit pairs.
+
+  See `Duration.from_iso8601/1`.
+  """
+  @doc since: "1.17.0"
+  @spec parse_duration(String.t()) :: {:ok, [Duration.unit_pair()]} | {:error, atom}
+  def parse_duration("P" <> string) when byte_size(string) > 0 do
+    parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D)
+  end
+
+  def parse_duration("+P" <> string) when byte_size(string) > 0 do
+    parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D)
+  end
+
+  def parse_duration("-P" <> string) when byte_size(string) > 0 do
+    with {:ok, fields} <- parse_duration_date(string, [], year: ?Y, month: ?M, week: ?W, day: ?D) do
+      {:ok,
+       Enum.map(fields, fn
+         {:microsecond, {value, precision}} -> {:microsecond, {-value, precision}}
+         {unit, value} -> {unit, -value}
+       end)}
+    end
+  end
+
+  def parse_duration(_) do
+    {:error, :invalid_duration}
+  end
+
+  defp parse_duration_date("", acc, _allowed), do: {:ok, acc}
+
+  defp parse_duration_date("T" <> string, acc, _allowed) when byte_size(string) > 0 do
+    parse_duration_time(string, acc, hour: ?H, minute: ?M, second: ?S)
+  end
+
+  defp parse_duration_date(string, acc, allowed) do
+    with {integer, <<next, rest::binary>>} <- Integer.parse(string),
+         {key, allowed} <- find_unit(allowed, next) do
+      parse_duration_date(rest, [{key, integer} | acc], allowed)
+    else
+      _ -> {:error, :invalid_date_component}
+    end
+  end
+
+  defp parse_duration_time("", acc, _allowed), do: {:ok, acc}
+
+  defp parse_duration_time(string, acc, allowed) do
+    case Integer.parse(string) do
+      {second, <<delimiter, _::binary>> = rest} when delimiter in [?., ?,] ->
+        case parse_microsecond(rest) do
+          {{ms, precision}, "S"} ->
+            ms = if second > 0, do: ms, else: -ms
+            {:ok, [second: second, microsecond: {ms, precision}] ++ acc}
+
+          _ ->
+            {:error, :invalid_time_component}
+        end
+
+      {integer, <<next, rest::binary>>} ->
+        case find_unit(allowed, next) do
+          {key, allowed} -> parse_duration_time(rest, [{key, integer} | acc], allowed)
+          false -> {:error, :invalid_time_component}
+        end
+
+      _ ->
+        {:error, :invalid_time_component}
+    end
+  end
+
+  defp find_unit([{key, unit} | rest], unit), do: {key, rest}
+  defp find_unit([_ | rest], unit), do: find_unit(rest, unit)
+  defp find_unit([], _unit), do: false
+
   @doc """
   Returns the `t:Calendar.iso_days/0` format of the specified date.
 
@@ -809,7 +883,7 @@ defmodule Calendar.ISO do
 
   # Converts count of days since 0000-01-01 to {year, month, day} tuple.
   @doc false
-  def date_from_iso_days(days) when days in -3_652_059..3_652_424 do
+  def date_from_iso_days(days) do
     {year, day_of_year} = days_to_year(days)
     extra_day = if leap_year?(year), do: 1, else: 0
     {month, day_in_month} = year_day_to_year_date(extra_day, day_of_year)
@@ -1364,7 +1438,7 @@ defmodule Calendar.ISO do
   @spec valid_date?(year, month, day) :: boolean
   def valid_date?(year, month, day)
       when is_integer(year) and is_integer(month) and is_integer(day) do
-    is_year(year) and is_month(month) and day in 1..days_in_month(year, month)
+    is_month(month) and day in 1..days_in_month(year, month)
   end
 
   @doc """
@@ -1455,6 +1529,230 @@ defmodule Calendar.ISO do
     {days, {@parts_per_day - 1, @parts_per_day}}
   end
 
+  @doc """
+  Shifts Date by Duration according to its calendar.
+
+  ## Examples
+
+      iex> Calendar.ISO.shift_date(2016, 1, 3, Duration.new!(month: 2))
+      {2016, 3, 3}
+      iex> Calendar.ISO.shift_date(2016, 2, 29, Duration.new!(month: 1))
+      {2016, 3, 29}
+      iex> Calendar.ISO.shift_date(2016, 1, 31, Duration.new!(month: 1))
+      {2016, 2, 29}
+      iex> Calendar.ISO.shift_date(2016, 1, 31, Duration.new!(year: 4, day: 1))
+      {2020, 2, 1}
+  """
+  @impl true
+  @spec shift_date(year, month, day, Duration.t()) :: {year, month, day}
+  def shift_date(year, month, day, duration) do
+    shift_options = shift_date_options(duration)
+
+    Enum.reduce(shift_options, {year, month, day}, fn
+      {_, 0}, date ->
+        date
+
+      {:month, value}, date ->
+        shift_months(date, value)
+
+      {:day, value}, date ->
+        shift_days(date, value)
+    end)
+  end
+
+  @doc """
+  Shifts NaiveDateTime by Duration according to its calendar.
+
+  ## Examples
+
+      iex> Calendar.ISO.shift_naive_datetime(2016, 1, 3, 0, 0, 0, {0, 0}, Duration.new!(hour: 1))
+      {2016, 1, 3, 1, 0, 0, {0, 0}}
+      iex> Calendar.ISO.shift_naive_datetime(2016, 1, 3, 0, 0, 0, {0, 0}, Duration.new!(hour: 30))
+      {2016, 1, 4, 6, 0, 0, {0, 0}}
+      iex> Calendar.ISO.shift_naive_datetime(2016, 1, 3, 0, 0, 0, {0, 0}, Duration.new!(microsecond: {100, 6}))
+      {2016, 1, 3, 0, 0, 0, {100, 6}}
+  """
+  @impl true
+  @spec shift_naive_datetime(
+          year,
+          month,
+          day,
+          hour,
+          minute,
+          second,
+          microsecond,
+          Duration.t()
+        ) :: {year, month, day, hour, minute, second, microsecond}
+  def shift_naive_datetime(year, month, day, hour, minute, second, microsecond, duration) do
+    shift_options = shift_datetime_options(duration)
+
+    Enum.reduce(shift_options, {year, month, day, hour, minute, second, microsecond}, fn
+      {_, 0}, naive_datetime ->
+        naive_datetime
+
+      {:month, value}, {year, month, day, hour, minute, second, microsecond} ->
+        {new_year, new_month, new_day} = shift_months({year, month, day}, value)
+        {new_year, new_month, new_day, hour, minute, second, microsecond}
+
+      {time_unit, value}, naive_datetime ->
+        shift_time_unit(naive_datetime, value, time_unit)
+    end)
+  end
+
+  @doc """
+  Shifts Time by Duration units according to its calendar.
+
+  ## Examples
+
+      iex> Calendar.ISO.shift_time(13, 0, 0, {0, 0}, Duration.new!(hour: 2))
+      {15, 0, 0, {0, 0}}
+      iex> Calendar.ISO.shift_time(13, 0, 0, {0, 0}, Duration.new!(microsecond: {100, 6}))
+      {13, 0, 0, {100, 6}}
+  """
+  @impl true
+  @spec shift_time(hour, minute, second, microsecond, Duration.t()) ::
+          {hour, minute, second, microsecond}
+  def shift_time(hour, minute, second, microsecond, duration) do
+    shift_options = shift_time_options(duration)
+
+    Enum.reduce(shift_options, {hour, minute, second, microsecond}, fn
+      {_, 0}, time ->
+        time
+
+      {time_unit, value}, time ->
+        shift_time_unit(time, value, time_unit)
+    end)
+  end
+
+  @doc false
+  def shift_days({year, month, day}, days) do
+    {year, month, day} =
+      date_to_iso_days(year, month, day)
+      |> Kernel.+(days)
+      |> date_from_iso_days()
+
+    {year, month, day}
+  end
+
+  defp shift_months({year, month, day}, months) do
+    months_in_year = 12
+    total_months = year * months_in_year + month + months - 1
+
+    new_year = Integer.floor_div(total_months, months_in_year)
+
+    new_month =
+      case rem(total_months, months_in_year) + 1 do
+        new_month when new_month < 1 -> new_month + months_in_year
+        new_month -> new_month
+      end
+
+    new_day = min(day, days_in_month(new_year, new_month))
+
+    {new_year, new_month, new_day}
+  end
+
+  @doc false
+  def shift_time_unit({year, month, day, hour, minute, second, microsecond}, value, unit)
+      when unit in [:second, :millisecond, :microsecond, :nanosecond] or is_integer(unit) do
+    {value, precision} = shift_time_unit_values(value, microsecond)
+
+    {year, month, day, hour, minute, second, {ms_value, _}} =
+      naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
+      |> shift_time_unit(value, unit)
+      |> naive_datetime_from_iso_days()
+
+    {year, month, day, hour, minute, second, {ms_value, precision}}
+  end
+
+  def shift_time_unit({hour, minute, second, microsecond}, value, unit)
+      when unit in [:second, :millisecond, :microsecond, :nanosecond] or is_integer(unit) do
+    {value, precision} = shift_time_unit_values(value, microsecond)
+
+    {_days, day_fraction} =
+      shift_time_unit({0, time_to_day_fraction(hour, minute, second, microsecond)}, value, unit)
+
+    {hour, minute, second, {microsecond, _}} = time_from_day_fraction(day_fraction)
+
+    {hour, minute, second, {microsecond, precision}}
+  end
+
+  def shift_time_unit({_days, _day_fraction} = iso_days, value, unit)
+      when unit in [:second, :millisecond, :microsecond, :nanosecond] or is_integer(unit) do
+    ppd = System.convert_time_unit(86400, :second, unit)
+    add_day_fraction_to_iso_days(iso_days, value, ppd)
+  end
+
+  defp shift_time_unit_values({0, _}, {_, original_precision}) do
+    {0, original_precision}
+  end
+
+  defp shift_time_unit_values({ms_value, ms_precision}, {_, _}) do
+    {ms_value, ms_precision}
+  end
+
+  defp shift_time_unit_values(value, {_, original_precision}) do
+    {value, original_precision}
+  end
+
+  defp shift_date_options(%Duration{
+         year: year,
+         month: month,
+         week: week,
+         day: day,
+         hour: 0,
+         minute: 0,
+         second: 0,
+         microsecond: {0, 0}
+       }) do
+    [
+      month: year * 12 + month,
+      day: week * 7 + day
+    ]
+  end
+
+  defp shift_date_options(_duration) do
+    raise ArgumentError,
+          "cannot shift date by time scale unit. Expected :year, :month, :week, :day"
+  end
+
+  defp shift_datetime_options(%Duration{
+         year: year,
+         month: month,
+         week: week,
+         day: day,
+         hour: hour,
+         minute: minute,
+         second: second,
+         microsecond: microsecond
+       }) do
+    [
+      month: year * 12 + month,
+      second: week * 7 * 86400 + day * 86400 + hour * 3600 + minute * 60 + second,
+      microsecond: microsecond
+    ]
+  end
+
+  defp shift_time_options(%Duration{
+         year: 0,
+         month: 0,
+         week: 0,
+         day: 0,
+         hour: hour,
+         minute: minute,
+         second: second,
+         microsecond: microsecond
+       }) do
+    [
+      second: hour * 3600 + minute * 60 + second,
+      microsecond: microsecond
+    ]
+  end
+
+  defp shift_time_options(_duration) do
+    raise ArgumentError,
+          "cannot shift time by date scale unit. Expected :hour, :minute, :second, :microsecond"
+  end
+
   ## Helpers
 
   @doc false
@@ -1642,11 +1940,22 @@ defmodule Calendar.ISO do
 
   defp days_in_previous_years(0), do: 0
 
-  defp days_in_previous_years(year) do
+  # A concise version of the algorithm would use floor_div instead of div.
+  # However, floor_div would check the operands on every operation.
+  # We optimize this by providing a positive and negative version of each algorithm.
+  defp days_in_previous_years(year) when year > 0 do
     previous_year = year - 1
 
-    Integer.floor_div(previous_year, 4) - Integer.floor_div(previous_year, 100) +
-      Integer.floor_div(previous_year, 400) + previous_year * @days_per_nonleap_year +
+    div(previous_year, 4) - div(previous_year, 100) +
+      div(previous_year, 400) + previous_year * @days_per_nonleap_year +
+      @days_per_leap_year
+  end
+
+  defp days_in_previous_years(year) when year < 0 do
+    previous_year = year - 1
+
+    div(year, 4) - div(year, 100) +
+      div(year, 400) - 1 + previous_year * @days_per_nonleap_year +
       @days_per_leap_year
   end
 

lib/elixir/lib/calendar/naive_datetime.ex

@@ -448,6 +448,8 @@ defmodule NaiveDateTime do
       iex> NaiveDateTime.add(dt, 21, :second)
       ~N[2000-02-29 23:00:28]
 
+  To shift a naive datetime by a `Duration` and according to its underlying calendar, use `NaiveDateTime.shift/2`.
+
   """
   @doc since: "1.4.0"
   @spec add(Calendar.naive_datetime(), integer, :day | :hour | :minute | System.time_unit()) :: t
@@ -466,23 +468,21 @@ defmodule NaiveDateTime do
   end
 
   def add(
-        %{microsecond: {_, precision}, calendar: calendar} = naive_datetime,
+        %{calendar: calendar, microsecond: {_, precision}} = naive_datetime,
         amount_to_add,
         unit
       )
       when is_integer(amount_to_add) do
-    if not is_integer(unit) and
-         unit not in ~w(second millisecond microsecond nanosecond)a do
+    if not is_integer(unit) and unit not in ~w(second millisecond microsecond nanosecond)a do
       raise ArgumentError,
             "unsupported time unit. Expected :day, :hour, :minute, :second, :millisecond, :microsecond, :nanosecond, or a positive integer, got #{inspect(unit)}"
     end
 
-    ppd = System.convert_time_unit(86400, :second, unit)
     precision = max(Calendar.ISO.time_unit_to_precision(unit), precision)
 
     naive_datetime
     |> to_iso_days()
-    |> Calendar.ISO.add_day_fraction_to_iso_days(amount_to_add, ppd)
+    |> Calendar.ISO.shift_time_unit(amount_to_add, unit)
     |> from_iso_days(calendar, precision)
   end
 
@@ -571,6 +571,83 @@ defmodule NaiveDateTime do
     units1 - units2
   end
 
+  @doc """
+  Shifts given `naive_datetime` by `duration` according to its calendar.
+
+  Allowed units are: `:year`, `:month`, `:week`, `:day`, `:hour`, `:minute`, `:second`, `:microsecond`.
+
+  When using the default ISO calendar, durations are collapsed and
+  applied in the order of months, then seconds and microseconds:
+
+  * when shifting by 1 year and 2 months the date is actually shifted by 14 months
+  * weeks, days and smaller units are collapsed into seconds and microseconds
+
+  When shifting by month, days are rounded down to the nearest valid date.
+
+  ## Examples
+
+      iex> NaiveDateTime.shift(~N[2016-01-31 00:00:00], month: 1)
+      ~N[2016-02-29 00:00:00]
+      iex> NaiveDateTime.shift(~N[2016-01-31 00:00:00], year: 4, day: 1)
+      ~N[2020-02-01 00:00:00]
+      iex> NaiveDateTime.shift(~N[2016-01-31 00:00:00], year: -2, day: 1)
+      ~N[2014-02-01 00:00:00]
+      iex> NaiveDateTime.shift(~N[2016-01-31 00:00:00], second: 45)
+      ~N[2016-01-31 00:00:45]
+      iex> NaiveDateTime.shift(~N[2016-01-31 00:00:00], microsecond: {100, 6})
+      ~N[2016-01-31 00:00:00.000100]
+
+      # leap years
+      iex> NaiveDateTime.shift(~N[2024-02-29 00:00:00], year: 1)
+      ~N[2025-02-28 00:00:00]
+      iex> NaiveDateTime.shift(~N[2024-02-29 00:00:00], year: 4)
+      ~N[2028-02-29 00:00:00]
+
+      # rounding down
+      iex> NaiveDateTime.shift(~N[2015-01-31 00:00:00], month: 1)
+      ~N[2015-02-28 00:00:00]
+
+  """
+  @doc since: "1.17.0"
+  @spec shift(Calendar.naive_datetime(), Duration.duration()) :: t
+  def shift(%{calendar: calendar} = naive_datetime, duration) do
+    %{
+      year: year,
+      month: month,
+      day: day,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond
+    } = naive_datetime
+
+    {year, month, day, hour, minute, second, microsecond} =
+      calendar.shift_naive_datetime(
+        year,
+        month,
+        day,
+        hour,
+        minute,
+        second,
+        microsecond,
+        __duration__!(duration)
+      )
+
+    %NaiveDateTime{
+      calendar: calendar,
+      year: year,
+      month: month,
+      day: day,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond
+    }
+  end
+
+  @doc false
+  defdelegate __duration__!(params), to: Duration, as: :new!
+
   @doc """
   Returns the given naive datetime with the microsecond field truncated to the
   given precision (`:microsecond`, `:millisecond` or `:second`).
@@ -976,11 +1053,28 @@ defmodule NaiveDateTime do
   """
   @doc since: "1.11.0"
   @spec from_gregorian_seconds(integer(), Calendar.microsecond(), Calendar.calendar()) :: t
-  def from_gregorian_seconds(
-        seconds,
-        {microsecond, precision} \\ {0, 0},
-        calendar \\ Calendar.ISO
-      )
+  def from_gregorian_seconds(seconds, microsecond_precision \\ {0, 0}, calendar \\ Calendar.ISO)
+
+  def from_gregorian_seconds(seconds, {microsecond, precision}, Calendar.ISO)
+      when is_integer(seconds) do
+    {days, seconds} = div_rem(seconds, 24 * 60 * 60)
+    {hours, seconds} = div_rem(seconds, 60 * 60)
+    {minutes, seconds} = div_rem(seconds, 60)
+    {year, month, day} = Calendar.ISO.date_from_iso_days(days)
+
+    %NaiveDateTime{
+      calendar: Calendar.ISO,
+      year: year,
+      month: month,
+      day: day,
+      hour: hours,
+      minute: minutes,
+      second: seconds,
+      microsecond: {microsecond, precision}
+    }
+  end
+
+  def from_gregorian_seconds(seconds, {microsecond, precision}, calendar)
       when is_integer(seconds) do
     iso_days = Calendar.ISO.gregorian_seconds_to_iso_days(seconds, microsecond)
 
@@ -999,6 +1093,17 @@ defmodule NaiveDateTime do
     }
   end
 
+  defp div_rem(int1, int2) do
+    div = div(int1, int2)
+    rem = int1 - div * int2
+
+    if rem >= 0 do
+      {div, rem}
+    else
+      {div - 1, rem + int2}
+    end
+  end
+
   @doc """
   Converts a `NaiveDateTime` struct to a number of gregorian seconds and microseconds.
 

lib/elixir/lib/calendar/time.ex

@@ -50,7 +50,6 @@ defmodule Time do
           calendar: Calendar.calendar()
         }
 
-  @parts_per_day 86_400_000_000
   @seconds_per_day 24 * 60 * 60
 
   @doc """
@@ -500,6 +499,8 @@ defmodule Time do
       iex> result.microsecond
       {21000, 3}
 
+  To shift a time by a `Duration` and according to its underlying calendar, use `Time.shift/2`.
+
   """
   @doc since: "1.6.0"
   @spec add(Calendar.time(), integer, :hour | :minute | System.time_unit()) :: t
@@ -515,13 +516,26 @@ defmodule Time do
 
   def add(%{calendar: calendar, microsecond: {_, precision}} = time, amount_to_add, unit)
       when is_integer(amount_to_add) do
-    amount_to_add = System.convert_time_unit(amount_to_add, unit, :microsecond)
-    total = time_to_microseconds(time) + amount_to_add
-    parts = Integer.mod(total, @parts_per_day)
+    valid? =
+      if is_integer(unit),
+        do: unit > 0,
+        else: unit in ~w(second millisecond microsecond nanosecond)a
+
+    unless valid? do
+      raise ArgumentError,
+            "unsupported time unit. Expected :hour, :minute, :second, :millisecond, :microsecond, :nanosecond, or a positive integer, got #{inspect(unit)}"
+    end
+
+    %{hour: hour, minute: minute, second: second, microsecond: microsecond} = time
+
     precision = max(Calendar.ISO.time_unit_to_precision(unit), precision)
 
-    {hour, minute, second, {microsecond, _}} =
-      calendar.time_from_day_fraction({parts, @parts_per_day})
+    {hour, minute, second, {microsecond, _precision}} =
+      Calendar.ISO.shift_time_unit(
+        {hour, minute, second, microsecond},
+        amount_to_add,
+        unit
+      )
 
     %Time{
       hour: hour,
@@ -532,19 +546,91 @@ defmodule Time do
     }
   end
 
-  defp time_to_microseconds(%{
-         calendar: Calendar.ISO,
-         hour: 0,
-         minute: 0,
-         second: 0,
-         microsecond: {0, _}
-       }) do
-    0
+  @doc """
+  Shifts given `time` by `duration` according to its calendar.
+
+  Available duration units are: `:hour`, `:minute`, `:second`, `:microsecond`.
+
+  When using the default ISO calendar, durations are collapsed to seconds and
+  microseconds before they are applied.
+
+  Raises an `ArgumentError` when called with date scale units.
+
+  ## Examples
+
+      iex> Time.shift(~T[01:00:15], hour: 12)
+      ~T[13:00:15]
+      iex> Time.shift(~T[01:35:00], hour: 6, minute: -15)
+      ~T[07:20:00]
+      iex> Time.shift(~T[01:15:00], second: 125)
+      ~T[01:17:05]
+      iex> Time.shift(~T[01:00:15], microsecond: {100, 6})
+      ~T[01:00:15.000100]
+      iex> Time.shift(~T[01:15:00], Duration.new!(second: 65))
+      ~T[01:16:05]
+
+  """
+  @doc since: "1.17.0"
+  @spec shift(Calendar.time(), Duration.t() | [unit_pair]) :: t
+        when unit_pair:
+               {:hour, integer}
+               | {:minute, integer}
+               | {:second, integer}
+               | {:microsecond, {integer, 0..6}}
+  def shift(%{calendar: calendar} = time, duration) do
+    %{hour: hour, minute: minute, second: second, microsecond: microsecond} = time
+
+    {hour, minute, second, microsecond} =
+      calendar.shift_time(hour, minute, second, microsecond, __duration__!(duration))
+
+    %Time{
+      calendar: calendar,
+      hour: hour,
+      minute: minute,
+      second: second,
+      microsecond: microsecond
+    }
   end
 
-  defp time_to_microseconds(time) do
-    iso_days = {0, to_day_fraction(time)}
-    Calendar.ISO.iso_days_to_unit(iso_days, :microsecond)
+  @doc false
+  def __duration__!(%Duration{} = duration) do
+    duration
+  end
+
+  # This part is inlined by the compiler on constant values
+  def __duration__!(unit_pairs) do
+    Enum.each(unit_pairs, &validate_duration_unit!/1)
+    struct!(Duration, unit_pairs)
+  end
+
+  defp validate_duration_unit!({:microsecond, {ms, precision}})
+       when is_integer(ms) and precision in 0..6 do
+    :ok
+  end
+
+  defp validate_duration_unit!({:microsecond, microsecond}) do
+    raise ArgumentError,
+          "unsupported value #{inspect(microsecond)} for :microsecond. Expected a tuple {ms, precision} where precision is an integer from 0 to 6"
+  end
+
+  defp validate_duration_unit!({unit, _value}) when unit in [:year, :month, :week, :day] do
+    raise ArgumentError,
+          "unsupported unit #{inspect(unit)}. Expected :hour, :minute, :second, :microsecond"
+  end
+
+  defp validate_duration_unit!({unit, _value})
+       when unit not in [:hour, :minute, :second, :microsecond] do
+    raise ArgumentError,
+          "unknown unit #{inspect(unit)}. Expected :hour, :minute, :second, :microsecond"
+  end
+
+  defp validate_duration_unit!({_unit, value}) when is_integer(value) do
+    :ok
+  end
+
+  defp validate_duration_unit!({unit, value}) do
+    raise ArgumentError,
+          "unsupported value #{inspect(value)} for #{inspect(unit)}. Expected an integer"
   end
 
   @doc """

lib/elixir/lib/code.ex

@@ -3,14 +3,14 @@ defmodule Code do
   Utilities for managing code compilation, code evaluation, and code loading.
 
   This module complements Erlang's [`:code` module](`:code`)
-  to add behaviour which is specific to Elixir. For functions to
+  to add behavior which is specific to Elixir. For functions to
   manipulate Elixir's AST (rather than evaluating it), see the
   `Macro` module.
 
   ## Working with files
 
   This module contains three functions for compiling and evaluating files.
-  Here is a summary of them and their behaviour:
+  Here is a summary of them and their behavior:
 
     * `require_file/2` - compiles a file and tracks its name. It does not
       compile the file again if it has been previously required.
@@ -158,6 +158,10 @@ defmodule Code do
       of keys to traverse in the application environment and `return` is either
       `{:ok, value}` or `:error`.
 
+    * `:defmodule` - (since v1.16.2) traced as soon as the definition of a module
+      starts. This is invoked early on in the module life cycle, `Module.open?/1`
+      still returns `false` for such traces
+
     * `{:on_module, bytecode, _ignore}` - (since v1.13.0) traced whenever a module
       is defined. This is equivalent to the `@after_compile` callback and invoked
       after any `@after_compile` in the given module. The third element is currently
@@ -196,20 +200,44 @@ defmodule Code do
 
   @typedoc """
   Diagnostics returned by the compiler and code evaluation.
+
+  The file and position relate to where the diagnostic should be shown.
+  If there is a file and position, then the diagnostic is precise
+  and you can use the given file and position for generating snippets,
+  IDEs annotations, and so on. An optional span is available with
+  the line and column the diagnostic ends.
+
+  Otherwise, a stacktrace may be given, which you can place your own
+  heuristics to provide better reporting.
+
+  The source field points to the source file the compiler tracked
+  the error to. For example, a file `lib/foo.ex` may embed `.eex`
+  templates from `lib/foo/bar.eex`. A syntax error on the EEx template
+  will point to file `lib/foo/bar.eex` but the source is `lib/foo.ex`.
   """
   @type diagnostic(severity) :: %{
-          required(:file) => Path.t(),
+          required(:source) => Path.t() | nil,
+          required(:file) => Path.t() | nil,
           required(:severity) => severity,
           required(:message) => String.t(),
-          required(:position) => position,
+          required(:position) => position(),
           required(:stacktrace) => Exception.stacktrace(),
-          required(:span) => {non_neg_integer, non_neg_integer} | nil,
+          required(:span) => {line :: pos_integer(), column :: pos_integer()} | nil,
+          optional(:details) => term(),
           optional(any()) => any()
         }
 
   @typedoc "The line. 0 indicates no line."
   @type line() :: non_neg_integer()
-  @type position() :: line() | {pos_integer(), column :: non_neg_integer}
+
+  @typedoc """
+  The position of the diagnostic.
+
+  Can be either a line number or a `{line, column}`.
+  Line and columns numbers are one-based.
+  A position of `0` represents unknown.
+  """
+  @type position() :: line() | {line :: pos_integer(), column :: pos_integer()}
 
   @boolean_compiler_options [
     :docs,
@@ -637,9 +665,9 @@ defmodule Code do
     * `:line` - the line the string starts, used for error reporting
 
     * `:line_length` - the line length to aim for when formatting
-      the document. Defaults to 98. Note this value is used as
-      guideline but there are situations where it is not enforced.
-      See the "Line length" section below for more information
+      the document. Defaults to 98. This value indicates when an expression
+      should be broken over multiple lines but it is not guaranteed
+      to do so. See the "Line length" section below for more information
 
     * `:locals_without_parens` - a keyword list of name and arity
       pairs that should be kept without parens whenever possible.
@@ -686,8 +714,8 @@ defmodule Code do
   specially because a function is named `defmodule`, `def`, or the like. This
   principle mirrors Elixir's goal of being an extensible language where
   developers can extend the language with new constructs as if they were
-  part of the language. When it is absolutely necessary to change behaviour
-  based on the name, this behaviour should be configurable, such as the
+  part of the language. When it is absolutely necessary to change behavior
+  based on the name, this behavior should be configurable, such as the
   `:locals_without_parens` option.
 
   ## Running the formatter
@@ -770,9 +798,10 @@ defmodule Code do
   ## Line length
 
   Another point about the formatter is that the `:line_length` configuration
-  is a guideline. In many cases, it is not possible for the formatter to break
-  your code apart, which means it will go over the line length. For example,
-  if you have a long string:
+  indicates when an expression should be broken over multiple lines but it is
+  not guaranteed to do so. In many cases, it is not possible for the formatter
+  to break your code apart, which means it will go over the line length.
+  For example, if you have a long string:
 
       "this is a very long string that will go over the line length"
 
@@ -785,15 +814,15 @@ defmodule Code do
   The string concatenation makes the code fit on a single line and also
   gives more options to the formatter.
 
-  This may also appear in do/end blocks, where the `do` keyword (or `->`)
-  may go over the line length because there is no opportunity for the
-  formatter to introduce a line break in a readable way. For example,
-  if you do:
+  This may also appear in keywords such as do/end blocks and operators,
+  where the `do` keyword may go over the line length because there is no
+  opportunity for the formatter to introduce a line break in a readable way.
+  For example, if you do:
 
       case very_long_expression() do
       end
 
-  And only the `do` keyword is above the line length, Elixir **will not**
+  And only the `do` keyword is beyond the line length, Elixir **will not**
   emit this:
 
       case very_long_expression()
@@ -831,7 +860,7 @@ defmodule Code do
     * Newlines before certain operators (such as the pipeline operators)
       and before other operators (such as comparison operators)
 
-  The behaviours above are not guaranteed. We may remove or add new
+  The behaviors above are not guaranteed. We may remove or add new
   rules in the future. The goal of documenting them is to provide better
   understanding on what to expect from the formatter.
 
@@ -962,10 +991,9 @@ defmodule Code do
     to_quoted_opts =
       [
         unescape: false,
-        warn_on_unnecessary_quotes: false,
         literal_encoder: &{:ok, {:__block__, &2, [&1]}},
         token_metadata: true,
-        warnings: false
+        emit_warnings: false
       ] ++ opts
 
     {forms, comments} = string_to_quoted_with_comments!(string, to_quoted_opts)
@@ -1101,7 +1129,7 @@ defmodule Code do
       For example, `"null byte\\t\\x00"` will be kept as is instead of being
       converted to a bitstring literal. Note if you set this option to false, the
       resulting AST is no longer valid, but it can be useful to analyze/transform
-      source code, typically in in combination with `quoted_to_algebra/2`.
+      source code, typically in combination with `quoted_to_algebra/2`.
       Defaults to `true`.
 
     * `:existing_atoms_only` - when `true`, raises an error
@@ -1122,13 +1150,12 @@ defmodule Code do
 
     * `:static_atoms_encoder` - the static atom encoder function, see
       "The `:static_atoms_encoder` function" section below. Note this
-      option overrides the `:existing_atoms_only` behaviour for static
+      option overrides the `:existing_atoms_only` behavior for static
       atoms but `:existing_atoms_only` is still used for dynamic atoms,
       such as atoms with interpolations.
 
-    * `:warn_on_unnecessary_quotes` - when `false`, does not warn
-      when atoms, keywords or calls have unnecessary quotes on
-      them. Defaults to `true`.
+    * `:emit_warnings` (since v1.16.0) - when `false`, does not emit
+      tokenizing/parsing related warnings. Defaults to `true`.
 
   ## `Macro.to_string/2`
 
@@ -1192,6 +1219,7 @@ defmodule Code do
   It returns the AST if it succeeds,
   raises an exception otherwise. The exception is a `TokenMissingError`
   in case a token is missing (usually because the expression is incomplete),
+  `MismatchedDelimiterError` (in case of mismatched opening and closing delimiters) and
   `SyntaxError` otherwise.
 
   Check `string_to_quoted/2` for options information.
@@ -1215,7 +1243,7 @@ defmodule Code do
 
   Comments are maps with the following fields:
 
-    * `:line` - The line number the source code
+    * `:line` - The line number of the source code
 
     * `:text` - The full text of the comment, including the leading `#`
 
@@ -1605,8 +1633,9 @@ defmodule Code do
       error. You may be set it to `:warn` if you want undefined variables to
       emit a warning and expand as to a local call to the zero-arity function
       of the same name (for example, `node` would be expanded as `node()`).
-      This `:warn` behaviour only exists for compatibility reasons when working
-      with old dependencies.
+      This `:warn` behavior only exists for compatibility reasons when working
+      with old dependencies, its usage is discouraged and it will be removed
+      in future releases.
 
   It always returns `:ok`. Raises an error for invalid options.
 
@@ -1657,6 +1686,7 @@ defmodule Code do
   end
 
   # TODO: Make this option have no effect on Elixir v2.0
+  # TODO: Warn if mode is :warn on Elixir v1.19
   def put_compiler_option(:on_undefined_variable, value) when value in [:raise, :warn] do
     :elixir_config.put(:on_undefined_variable, value)
     :ok
@@ -1695,7 +1725,7 @@ defmodule Code do
 
   Returns a list of tuples where the first element is the module name
   and the second one is its bytecode (as a binary). A `file` can be
-  given as second argument which will be used for reporting warnings
+  given as a second argument which will be used for reporting warnings
   and errors.
 
   **Warning**: `string` can be any Elixir code and code can be executed with
@@ -1939,7 +1969,7 @@ defmodule Code do
   @doc """
   Returns `true` if the module is loaded.
 
-  This function doesn't attempt to load the module. For such behaviour,
+  This function doesn't attempt to load the module. For such behavior,
   `ensure_loaded?/1` can be used.
 
   ## Examples
@@ -2006,7 +2036,11 @@ defmodule Code do
   @spec fetch_docs(module | String.t()) ::
           {:docs_v1, annotation, beam_language, format, module_doc :: doc_content, metadata,
            docs :: [doc_element]}
-          | {:error, :module_not_found | :chunk_not_found | {:invalid_chunk, binary}}
+          | {:error,
+             :module_not_found
+             | :chunk_not_found
+             | {:invalid_chunk, binary}
+             | :invalid_beam}
         when annotation: :erl_anno.anno(),
              beam_language: :elixir | :erlang | atom(),
              doc_content: %{optional(binary) => binary} | :none | :hidden,
@@ -2057,7 +2091,8 @@ defmodule Code do
 
   defp get_beam_and_path(module) do
     with {^module, beam, filename} <- :code.get_object_code(module),
-         {:ok, ^module} <- beam |> :beam_lib.info() |> Keyword.fetch(:module) do
+         info_pairs when is_list(info_pairs) <- :beam_lib.info(beam),
+         {:ok, ^module} <- Keyword.fetch(info_pairs, :module) do
       {beam, filename}
     else
       _ -> :error
@@ -2076,6 +2111,9 @@ defmodule Code do
 
       {:error, :beam_lib, {:file_error, _, :enoent}} ->
         {:error, :module_not_found}
+
+      {:error, :beam_lib, _} ->
+        {:error, :invalid_beam}
     end
   end
 

lib/elixir/lib/code/formatter.ex

@@ -6,7 +6,8 @@ defmodule Code.Formatter do
   @double_heredoc "\"\"\""
   @single_quote "'"
   @single_heredoc "'''"
-  @sigil_c "~c\""
+  @sigil_c_double "~c\""
+  @sigil_c_single "~c'"
   @sigil_c_heredoc "~c\"\"\""
   @newlines 2
   @min_line 0
@@ -289,6 +290,7 @@ defmodule Code.Formatter do
     end
   end
 
+  # TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed
   defp quoted_to_algebra(
          {{:., _, [List, :to_charlist]}, meta, [entries]} = quoted,
          context,
@@ -299,7 +301,7 @@ defmodule Code.Formatter do
         remote_to_algebra(quoted, context, state)
 
       meta[:delimiter] == ~s['''] ->
-        {opener, quotes} = get_charlist_quotes(true, state)
+        {opener, quotes} = get_charlist_quotes(:heredoc, state)
 
         {doc, state} =
           entries
@@ -309,7 +311,7 @@ defmodule Code.Formatter do
         {force_unfit(doc), state}
 
       true ->
-        {opener, quotes} = get_charlist_quotes(false, state)
+        {opener, quotes} = get_charlist_quotes({:regular, entries}, state)
         list_interpolation_to_algebra(entries, quotes, state, opener, quotes)
     end
   end
@@ -365,16 +367,22 @@ defmodule Code.Formatter do
     tuple_to_algebra(meta, [left, right], :flex_break, state)
   end
 
+  # (left -> right)
+  defp quoted_to_algebra({:__block__, _, [[{:->, _, _} | _] = clauses]}, _context, state) do
+    paren_fun_to_algebra(clauses, @max_line, @min_line, state)
+  end
+
   defp quoted_to_algebra({:__block__, meta, [list]}, _context, state) when is_list(list) do
     case meta[:delimiter] do
       ~s['''] ->
-        {opener, quotes} = get_charlist_quotes(true, state)
+        {opener, quotes} = get_charlist_quotes(:heredoc, state)
         string = list |> List.to_string() |> escape_heredoc(quotes)
         {opener |> concat(string) |> concat(quotes) |> force_unfit(), state}
 
       ~s['] ->
-        {opener, quotes} = get_charlist_quotes(false, state)
-        string = list |> List.to_string() |> escape_string(quotes)
+        string = list |> List.to_string()
+        {opener, quotes} = get_charlist_quotes({:regular, [string]}, state)
+        string = escape_string(string, quotes)
         {opener |> concat(string) |> concat(quotes), state}
 
       _other ->
@@ -414,6 +422,7 @@ defmodule Code.Formatter do
     {Keyword.fetch!(meta, :token) |> float_to_algebra(state.inspect_opts), state}
   end
 
+  # (unquote_splicing(...))
   defp quoted_to_algebra(
          {:__block__, _meta, [{:unquote_splicing, meta, [_] = args}]},
          context,
@@ -477,6 +486,11 @@ defmodule Code.Formatter do
     end
   end
 
+  # ...
+  defp quoted_to_algebra({:..., _meta, []}, _context, state) do
+    {"...", state}
+  end
+
   # 1..2//3
   defp quoted_to_algebra({:"..//", meta, [left, middle, right]}, context, state) do
     quoted_to_algebra({:"//", meta, [{:.., meta, [left, middle]}, right]}, context, state)
@@ -497,11 +511,6 @@ defmodule Code.Formatter do
     remote_to_algebra(quoted, context, state)
   end
 
-  # (left -> right)
-  defp quoted_to_algebra([{:->, _, _} | _] = clauses, _context, state) do
-    paren_fun_to_algebra(clauses, @max_line, @min_line, state)
-  end
-
   # [keyword: :list] (inner part)
   # %{:foo => :bar} (inner part)
   defp quoted_to_algebra(list, context, state) when is_list(list) do
@@ -520,14 +529,15 @@ defmodule Code.Formatter do
               {string(~S{"..//":}), state}
 
             {:__block__, _, [atom]} when is_atom(atom) ->
-              key =
+              iodata =
                 if Macro.classify_atom(atom) in [:identifier, :unquoted] do
-                  IO.iodata_to_binary([Atom.to_string(atom), ?:])
+                  [Atom.to_string(atom), ?:]
                 else
-                  IO.iodata_to_binary([?", Atom.to_string(atom), ?", ?:])
+                  [?", atom |> Atom.to_string() |> String.replace("\"", "\\\""), ?", ?:]
                 end
 
-              {string(key) |> color(:atom, state.inspect_opts), state}
+              {iodata |> IO.iodata_to_binary() |> string() |> color(:atom, state.inspect_opts),
+               state}
 
             {{:., _, [:erlang, :binary_to_atom]}, _, [{:<<>>, _, entries}, :utf8]} ->
               interpolation_to_algebra(entries, @double_quote, state, "\"", "\":")
@@ -627,7 +637,7 @@ defmodule Code.Formatter do
 
   defp maybe_binary_op_to_algebra(fun, meta, args, context, state) do
     with [left, right] <- args,
-         {_, _} <- Code.Identifier.binary_op(fun) do
+         {_, _} <- augmented_binary_op(fun) do
       binary_op_to_algebra(fun, Atom.to_string(fun), meta, left, right, context, state)
     else
       _ -> :error
@@ -654,7 +664,7 @@ defmodule Code.Formatter do
 
   defp binary_op_to_algebra(op, op_string, meta, left_arg, right_arg, context, state, _nesting)
        when op in @right_new_line_before_binary_operators do
-    op_info = Code.Identifier.binary_op(op)
+    op_info = augmented_binary_op(op)
     op_string = op_string <> " "
     left_context = left_op_context(context)
     right_context = right_op_context(context)
@@ -691,7 +701,7 @@ defmodule Code.Formatter do
 
   defp binary_op_to_algebra(op, _, meta, left_arg, right_arg, context, state, _nesting)
        when op in @pipeline_operators do
-    op_info = Code.Identifier.binary_op(op)
+    op_info = augmented_binary_op(op)
     left_context = left_op_context(context)
     right_context = right_op_context(context)
     max_line = line(meta)
@@ -705,7 +715,7 @@ defmodule Code.Formatter do
         {{doc, @empty, 1}, state}
 
       {{op, context}, arg}, _args, state ->
-        op_info = Code.Identifier.binary_op(op)
+        op_info = augmented_binary_op(op)
         op_string = Atom.to_string(op) <> " "
         {doc, state} = binary_operand_to_algebra(arg, context, state, op, op_info, :right, 0)
         {{concat(op_string, doc), @empty, 1}, state}
@@ -715,7 +725,7 @@ defmodule Code.Formatter do
   end
 
   defp binary_op_to_algebra(op, op_string, meta, left_arg, right_arg, context, state, nesting) do
-    op_info = Code.Identifier.binary_op(op)
+    op_info = augmented_binary_op(op)
     left_context = left_op_context(context)
     right_context = right_op_context(context)
 
@@ -774,7 +784,7 @@ defmodule Code.Formatter do
     {parent_assoc, parent_prec} = parent_info
 
     with {op, meta, [left, right]} <- operand,
-         op_info = Code.Identifier.binary_op(op),
+         op_info = augmented_binary_op(op),
          {_assoc, prec} <- op_info do
       op_string = Atom.to_string(op)
 
@@ -1956,6 +1966,14 @@ defmodule Code.Formatter do
   # fn a, b, c when d -> e end
   defp clause_args_to_algebra([{:when, meta, args}], state) do
     {args, right} = split_last(args)
+
+    # If there are any keywords, wrap them in lists
+    args =
+      Enum.map(args, fn
+        [_ | _] = keyword -> {:__block__, [], [keyword]}
+        other -> other
+      end)
+
     left = {{:special, :clause_args}, meta, [args]}
     binary_op_to_algebra(:when, "when", meta, left, right, :no_parens_arg, state)
   end
@@ -2185,10 +2203,15 @@ defmodule Code.Formatter do
     unary_operator?(quoted) or binary_operator?(quoted)
   end
 
+  # We convert ..// into two operators for simplicity,
+  # so we need to augment the binary table.
+  defp augmented_binary_op(:"//"), do: {:right, 190}
+  defp augmented_binary_op(op), do: Code.Identifier.binary_op(op)
+
   defp binary_operator?(quoted) do
     case quoted do
       {op, _, [_, _, _]} when op in @multi_binary_operators -> true
-      {op, _, [_, _]} when is_atom(op) -> Code.Identifier.binary_op(op) != :error
+      {op, _, [_, _]} when is_atom(op) -> augmented_binary_op(op) != :error
       _ -> false
     end
   end
@@ -2224,6 +2247,7 @@ defmodule Code.Formatter do
       (not interpolated?(entries) and eol_or_comments?(meta, state))
   end
 
+  # TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed
   defp next_break_fits?({{:., _, [List, :to_charlist]}, meta, [[_ | _]]}, _state) do
     meta[:delimiter] == ~s[''']
   end
@@ -2409,19 +2433,23 @@ defmodule Code.Formatter do
     {left, right}
   end
 
-  defp get_charlist_quotes(_heredoc = false, state) do
-    if state.normalize_charlists_as_sigils do
-      {@sigil_c, @double_quote}
-    else
-      {@single_quote, @single_quote}
-    end
-  end
-
-  defp get_charlist_quotes(_heredoc = true, state) do
+  defp get_charlist_quotes(:heredoc, state) do
     if state.normalize_charlists_as_sigils do
       {@sigil_c_heredoc, @double_heredoc}
     else
       {@single_heredoc, @single_heredoc}
     end
   end
+
+  defp get_charlist_quotes({:regular, chunks}, state) do
+    cond do
+      !state.normalize_charlists_as_sigils -> {@single_quote, @single_quote}
+      Enum.any?(chunks, &has_double_quote?/1) -> {@sigil_c_single, @single_quote}
+      true -> {@sigil_c_double, @double_quote}
+    end
+  end
+
+  defp has_double_quote?(chunk) do
+    is_binary(chunk) and chunk =~ @double_quote
+  end
 end

lib/elixir/lib/code/fragment.ex

@@ -31,7 +31,7 @@ defmodule Code.Fragment do
       :expr
 
       iex> Code.Fragment.cursor_context("hello_wor")
-      {:local_or_var, 'hello_wor'}
+      {:local_or_var, ~c"hello_wor"}
 
   ## Return values
 
@@ -481,10 +481,10 @@ defmodule Code.Fragment do
 
   defp operator(rest, count, acc, _call_op?) do
     case :elixir_tokenizer.tokenize(acc, 1, 1, []) do
-      {:ok, _, _, _, [{:atom, _, _}]} ->
+      {:ok, _, _, _, [{:atom, _, _}], []} ->
         {{:unquoted_atom, tl(acc)}, count}
 
-      {:ok, _, _, _, [{_, _, op}]} ->
+      {:ok, _, _, _, [{_, _, op}], []} ->
         {rest, dot_count} = strip_spaces(rest, count)
 
         cond do
@@ -547,7 +547,7 @@ defmodule Code.Fragment do
   ## Examples
 
       iex> Code.Fragment.surround_context("foo", {1, 1})
-      %{begin: {1, 1}, context: {:local_or_var, 'foo'}, end: {1, 4}}
+      %{begin: {1, 1}, context: {:local_or_var, ~c"foo"}, end: {1, 4}}
 
   ## Differences to `cursor_context/2`
 
@@ -636,7 +636,7 @@ defmodule Code.Fragment do
     {reversed_pre, post} = adjust_position(reversed_pre, post)
 
     case take_identifier(post, []) do
-      :none ->
+      {_, [], _} ->
         maybe_operator(reversed_pre, post, line, opts)
 
       {:identifier, reversed_post, rest} ->
@@ -644,7 +644,7 @@ defmodule Code.Fragment do
         reversed = reversed_post ++ reversed_pre
 
         case codepoint_cursor_context(reversed, opts) do
-          {{:struct, acc}, offset} when acc != [] ->
+          {{:struct, acc}, offset} ->
             build_surround({:struct, acc}, reversed, line, offset)
 
           {{:alias, acc}, offset} ->
@@ -749,27 +749,11 @@ defmodule Code.Fragment do
     do: take_identifier(t, [h | acc])
 
   defp take_identifier(rest, acc) do
-    {stripped, _} = strip_spaces(rest, 0)
-
-    with [?. | t] <- stripped,
+    with {[?. | t], _} <- strip_spaces(rest, 0),
          {[h | _], _} when h in ?A..?Z <- strip_spaces(t, 0) do
       take_alias(rest, acc)
     else
-      # Consider it an identifier if we are at the end of line
-      # or if we have spaces not followed by . (call) or / (arity)
-      _ when acc == [] and (rest == [] or (hd(rest) in @space and hd(stripped) not in ~c"/.")) ->
-        {:identifier, acc, rest}
-
-      # If we are immediately followed by a container, we are still part of the identifier.
-      # We don't consider << as it _may_ be an operator.
-      _ when acc == [] and hd(stripped) in ~c"({[" ->
-        {:identifier, acc, rest}
-
-      _ when acc == [] ->
-        :none
-
-      _ ->
-        {:identifier, acc, rest}
+      _ -> {:identifier, acc, rest}
     end
   end
 
@@ -1104,9 +1088,41 @@ defmodule Code.Fragment do
   @spec container_cursor_to_quoted(List.Chars.t(), keyword()) ::
           {:ok, Macro.t()} | {:error, {location :: keyword, binary | {binary, binary}, binary}}
   def container_cursor_to_quoted(fragment, opts \\ []) do
-    opts =
-      Keyword.take(opts, [:file, :line, :column, :columns, :token_metadata, :literal_encoder])
+    opts = Keyword.take(opts, [:columns, :token_metadata, :literal_encoder])
+    opts = [cursor_completion: true, emit_warnings: false] ++ opts
 
-    Code.string_to_quoted(fragment, [cursor_completion: true, warnings: false] ++ opts)
+    file = Keyword.get(opts, :file, "nofile")
+    line = Keyword.get(opts, :line, 1)
+    column = Keyword.get(opts, :column, 1)
+
+    case :elixir_tokenizer.tokenize(to_charlist(fragment), line, column, opts) do
+      {:ok, line, column, _warnings, rev_tokens, rev_terminators} ->
+        tokens = :lists.reverse(rev_tokens, rev_terminators)
+
+        case :elixir.tokens_to_quoted(tokens, file, opts) do
+          {:ok, ast} ->
+            {:ok, ast}
+
+          {:error, error} ->
+            # In case parsing fails, we give it another shot but handling fn/do/else/catch/rescue/after.
+            tokens =
+              :lists.reverse(
+                rev_tokens,
+                [{:stab_op, {line, column, nil}, :->}, {nil, {line, column + 2, nil}}] ++
+                  Enum.map(rev_terminators, fn tuple ->
+                    {line, column, info} = elem(tuple, 1)
+                    put_elem(tuple, 1, {line, column + 5, info})
+                  end)
+              )
+
+            case :elixir.tokens_to_quoted(tokens, file, opts) do
+              {:ok, ast} -> {:ok, ast}
+              {:error, _} -> {:error, error}
+            end
+        end
+
+      {:error, info, _rest, _warnings, _so_far} ->
+        {:error, :elixir.format_token_error(info)}
+    end
   end
 end

lib/elixir/lib/code/identifier.ex

@@ -13,7 +13,7 @@ defmodule Code.Identifier do
   @spec unary_op(atom) :: {:non_associative, precedence :: pos_integer} | :error
   def unary_op(op) do
     cond do
-      op in [:&] -> {:non_associative, 90}
+      op in [:&, :...] -> {:non_associative, 90}
       op in [:!, :^, :not, :+, :-, :"~~~"] -> {:non_associative, 300}
       op in [:@] -> {:non_associative, 320}
       true -> :error
@@ -44,7 +44,6 @@ defmodule Code.Identifier do
       op in [:|>, :<<<, :>>>, :<~, :~>, :<<~, :~>>, :<~>, :"<|>"] -> {:left, 160}
       op in [:in] -> {:left, 170}
       op in [:"^^^"] -> {:left, 180}
-      op in [:"//"] -> {:right, 190}
       op in [:++, :--, :.., :<>, :+++, :---] -> {:right, 200}
       op in [:+, :-] -> {:left, 210}
       op in [:*, :/] -> {:left, 220}

lib/elixir/lib/code/normalizer.ex

@@ -96,6 +96,7 @@ defmodule Code.Normalizer do
   end
 
   # Charlists with interpolations
+  # TODO: Remove this clause on Elixir v2.0 once single-quoted charlists are removed
   defp do_normalize({{:., dot_meta, [List, :to_charlist]}, call_meta, [parts]} = quoted, state) do
     if list_interpolated?(parts) do
       parts =
@@ -136,11 +137,6 @@ defmodule Code.Normalizer do
     {:., meta, [left, right]}
   end
 
-  # A list of left to right arrows is not considered as a list literal, so it's not wrapped
-  defp do_normalize([{:->, _, [_ | _]} | _] = quoted, state) do
-    normalize_args(quoted, state)
-  end
-
   # left -> right
   defp do_normalize({:->, meta, [left, right]}, state) do
     meta = patch_meta_line(meta, state.parent_meta)
@@ -169,9 +165,6 @@ defmodule Code.Normalizer do
           right = normalize_map_args(right, state)
           [{:|, pipe_meta, [left, right]}]
 
-        [{_, _, _} = call] ->
-          [do_normalize(call, state)]
-
         args ->
           normalize_map_args(args, state)
       end
@@ -349,19 +342,25 @@ defmodule Code.Normalizer do
         meta
       end
 
+    last = List.last(args)
+
     cond do
-      Keyword.has_key?(meta, :do) or match?([{{:__block__, _, [:do]}, _} | _], List.last(args)) ->
+      not allow_keyword?(form, arity) ->
+        args = normalize_args(args, %{state | parent_meta: meta})
+        {form, meta, args}
+
+      Keyword.has_key?(meta, :do) or match?([{{:__block__, _, [:do]}, _} | _], last) ->
         # def foo do :ok end
         # def foo, do: :ok
         normalize_kw_blocks(form, meta, args, state)
 
-      match?([{:do, _} | _], List.last(args)) ->
+      match?([{:do, _} | _], last) and Keyword.keyword?(last) ->
         # Non normalized kw blocks
         line = state.parent_meta[:line]
         meta = meta ++ [do: [line: line], end: [line: line]]
         normalize_kw_blocks(form, meta, args, state)
 
-      allow_keyword?(form, arity) ->
+      true ->
         args = normalize_args(args, %{state | parent_meta: meta})
         {last_arg, leading_args} = List.pop_at(args, -1, [])
 
@@ -382,10 +381,6 @@ defmodule Code.Normalizer do
           end
 
         {form, meta, leading_args ++ last_args}
-
-      true ->
-        args = normalize_args(args, %{state | parent_meta: meta})
-        {form, meta, args}
     end
   end
 

lib/elixir/lib/code/typespec.ex

@@ -175,7 +175,8 @@ defmodule Code.Typespec do
 
   defp get_module_and_beam(module) when is_atom(module) do
     with {^module, beam, _filename} <- :code.get_object_code(module),
-         {:ok, ^module} <- beam |> :beam_lib.info() |> Keyword.fetch(:module) do
+         info_pairs when is_list(info_pairs) <- :beam_lib.info(beam),
+         {:ok, ^module} <- Keyword.fetch(info_pairs, :module) do
       {module, beam}
     else
       _ -> :error
@@ -419,5 +420,13 @@ defmodule Code.Typespec do
     :error
   end
 
-  defp meta(anno), do: [line: :erl_anno.line(anno)]
+  defp meta(anno) do
+    case :erl_anno.location(anno) do
+      {line, column} ->
+        [line: line, column: column]
+
+      line when is_integer(line) ->
+        [line: line]
+    end
+  end
 end

lib/elixir/lib/collectable.ex

@@ -94,10 +94,10 @@ end
 
 defimpl Collectable, for: List do
   def into(list) do
-    # TODO: Change the behaviour so the into always comes last on Elixir v2.0
+    # TODO: Change the behavior so the into always comes last on Elixir v2.0
     if list != [] do
       IO.warn(
-        "the Collectable protocol is deprecated for non-empty lists. The behaviour of " <>
+        "the Collectable protocol is deprecated for non-empty lists. The behavior of " <>
           "Enum.into/2 and \"for\" comprehensions with an :into option is incorrect " <>
           "when collecting into non-empty lists. If you're collecting into a non-empty keyword " <>
           "list, consider using Keyword.merge/2 instead. If you're collecting into a non-empty " <>
@@ -140,6 +140,10 @@ defimpl Collectable, for: BitString do
 
       __acc, :halt ->
         :ok
+
+      _acc, {:cont, other} ->
+        raise ArgumentError,
+              "collecting into a binary requires a bitstring, got: #{inspect(other)}"
     end
 
     {[binary], fun}
@@ -155,6 +159,10 @@ defimpl Collectable, for: BitString do
 
       _acc, :halt ->
         :ok
+
+      _acc, {:cont, other} ->
+        raise ArgumentError,
+              "collecting into a bitstring requires a bitstring, got: #{inspect(other)}"
     end
 
     {bitstring, fun}
@@ -172,6 +180,10 @@ defimpl Collectable, for: Map do
 
       _map_acc, :halt ->
         :ok
+
+      _map_acc, {:cont, other} ->
+        raise ArgumentError,
+              "collecting into a map requires {key, value} tuples, got: #{inspect(other)}"
     end
 
     {map, fun}

lib/elixir/lib/config.ex

@@ -46,9 +46,9 @@ defmodule Config do
   ## Migrating from `use Mix.Config`
 
   The `Config` module in Elixir was introduced in v1.9 as a replacement to
-  `Mix.Config`, which was specific to Mix and has been deprecated.
+  `use Mix.Config`, which was specific to Mix and has been deprecated.
 
-  You can leverage `Config` instead of `Mix.Config` in three steps. The first
+  You can leverage `Config` instead of `use Mix.Config` in three steps. The first
   step is to replace `use Mix.Config` at the top of your config files by
   `import Config`.
 
@@ -82,9 +82,9 @@ defmodule Config do
         ...
       end
 
-  The only files where you may access functions from the `Mix` module are
-  the `mix.exs` file and inside custom Mix tasks, which always within the
-  `Mix.Tasks` namespace.
+  The only places where you may access functions from the `Mix` module are
+  the `mix.exs` file and inside custom Mix tasks, which are always within
+  the `Mix.Tasks` namespace.
 
   ## `config/runtime.exs`
 

lib/elixir/lib/config/provider.ex

@@ -312,7 +312,7 @@ defmodule Config.Provider do
            """
            the application #{inspect(app)} has a different value set #{path(key, path)} \
            during runtime compared to compile time. Since this application environment entry was \
-           marked as compile time, this difference can lead to different behaviour than expected:
+           marked as compile time, this difference can lead to different behavior than expected:
 
              * Compile time value #{return_to_text(compile_return)}
              * Runtime value #{return_to_text(runtime_return)}

lib/elixir/lib/dynamic_supervisor.ex

@@ -1136,15 +1136,6 @@ defmodule DynamicSupervisor do
     ]
   end
 
-  @impl true
-  def format_status(:terminate, [_pdict, state]) do
-    state
-  end
-
-  def format_status(_, [_pdict, %{mod: mod} = state]) do
-    [data: [{~c"State", state}], supervisor: [{~c"Callback", mod}]]
-  end
-
   ## Helpers
 
   @compile {:inline, call: 2}

lib/elixir/lib/enum.ex

@@ -80,7 +80,7 @@ defprotocol Enumerable do
 
   Returns the accumulator for the next enumeration step.
   """
-  @type reducer :: (element :: term, current_acc :: acc -> updated_acc :: acc)
+  @type reducer :: (element :: term, element_acc :: term -> acc)
 
   @typedoc """
   The result of the reduce operation.
@@ -261,7 +261,7 @@ defmodule Enum do
   traversed as if it was an enumerable.
 
   For a general overview of all functions in the `Enum` module, see
-  [the `Enum` cheatsheet](enum-cheat.html).
+  [the `Enum` cheatsheet](enum-cheat.cheatmd).
 
   The functions in this module work in linear time. This means that, the
   time it takes to perform an operation grows at the same rate as the length
@@ -269,12 +269,12 @@ defmodule Enum do
   After all, if we want to traverse every element on a list, the longer the
   list, the more elements we need to traverse, and the longer it will take.
 
-  This linear behaviour should also be expected on operations like `count/1`,
+  This linear behavior should also be expected on operations like `count/1`,
   `member?/2`, `at/2` and similar. While Elixir does allow data types to
   provide performant variants for such operations, you should not expect it
   to always be available, since the `Enum` module is meant to work with a
   large variety of data types and not all data types can provide optimized
-  behaviour.
+  behavior.
 
   Finally, note the functions in the `Enum` module are eager: they will
   traverse the enumerable as soon as they are invoked. This is particularly
@@ -1096,6 +1096,10 @@ defmodule Enum do
 
       iex> Enum.filter([1, 2, 3], fn x -> rem(x, 2) == 0 end)
       [2]
+      iex> Enum.filter(["apple", "pear", "banana"], fn fruit -> String.contains?(fruit, "a") end)
+      ["apple", "pear", "banana"]
+      iex> Enum.filter([4, 21, 24, 904], fn seconds -> seconds > 1000 end)
+      []
 
   Keep in mind that `filter` is not capable of filtering and
   transforming an element at the same time. If you would like
@@ -1428,7 +1432,7 @@ defmodule Enum do
     )
 
     # Avoid warnings about Dict
-    dict_module = Dict
+    dict_module = String.to_atom("Dict")
 
     reduce(reverse(enumerable), dict, fn entry, categories ->
       dict_module.update(categories, fun.(entry), [entry], &[entry | &1])
@@ -1528,8 +1532,7 @@ defmodule Enum do
   defp into_map(enumerable) when is_list(enumerable), do: :maps.from_list(enumerable)
   defp into_map(enumerable), do: enumerable |> Enum.to_list() |> :maps.from_list()
 
-  defp into_map(%{} = enumerable, collectable),
-    do: Map.merge(collectable, enumerable)
+  defp into_map(%{} = enumerable, collectable), do: Map.merge(collectable, enumerable)
 
   defp into_map(enumerable, collectable) when is_list(enumerable),
     do: Map.merge(collectable, :maps.from_list(enumerable))
@@ -1636,8 +1639,8 @@ defmodule Enum do
 
   If `joiner` is not passed at all, it defaults to an empty string.
 
-  All elements in the `enumerable` must be convertible to a string,
-  otherwise an error is raised.
+  All elements in the `enumerable` must be convertible to a string
+  or be a binary, otherwise an error is raised.
 
   ## Examples
 
@@ -1651,7 +1654,7 @@ defmodule Enum do
       "ab cdefg h i"
 
   """
-  @spec join(t, String.t()) :: String.t()
+  @spec join(t, binary()) :: binary()
   def join(enumerable, joiner \\ "")
 
   def join(enumerable, "") do
@@ -2362,12 +2365,6 @@ defmodule Enum do
   the random value. Check its documentation for setting a
   different random algorithm or a different seed.
 
-  The implementation is based on the
-  [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling#Relation_to_Fisher-Yates_shuffle)
-  algorithm.
-  It assumes that the sample being returned can fit into memory;
-  the input `enumerable` doesn't have to, as it is traversed just once.
-
   If a range is passed into the function, this function will pick a
   random value between the range limits, without traversing the whole
   range (thus executing in constant time and constant memory).
@@ -2386,6 +2383,12 @@ defmodule Enum do
       iex> Enum.random(1..1_000)
       309
 
+  ## Implementation
+
+  The random functions in this module implement reservoir sampling,
+  which allows them to sample infinite collections. In particular,
+  we implement Algorithm L, as described in by Kim-Hung Li in
+  "Reservoir-Sampling Algorithms of Time Complexity O(n(1+log(N/n)))".
   """
   @spec random(t) :: element
   def random(enumerable)
@@ -2733,7 +2736,7 @@ defmodule Enum do
   end
 
   # Normalize negative input ranges like Enum.slice/2
-  def slide(enumerable, first..last, insertion_index)
+  def slide(enumerable, first..last//_, insertion_index)
       when first < 0 or last < 0 or insertion_index < 0 do
     count = Enum.count(enumerable)
     normalized_first = if first >= 0, do: first, else: Kernel.max(first + count, 0)
@@ -2750,23 +2753,23 @@ defmodule Enum do
     end
   end
 
-  def slide(enumerable, insertion_index.._, insertion_index) do
+  def slide(enumerable, insertion_index.._//_, insertion_index) do
     Enum.to_list(enumerable)
   end
 
-  def slide(_, first..last, insertion_index)
+  def slide(_, first..last//_, insertion_index)
       when insertion_index > first and insertion_index <= last do
     raise ArgumentError,
           "insertion index for slide must be outside the range being moved " <>
             "(tried to insert #{first}..#{last} at #{insertion_index})"
   end
 
-  def slide(enumerable, first..last, _insertion_index) when first > last do
+  def slide(enumerable, first..last//_, _insertion_index) when first > last do
     Enum.to_list(enumerable)
   end
 
   # Guarantees at this point: step size == 1 and first <= last and (insertion_index < first or insertion_index > last)
-  def slide(enumerable, first..last, insertion_index) do
+  def slide(enumerable, first..last//_, insertion_index) do
     impl = if is_list(enumerable), do: &slide_list_start/4, else: &slide_any/4
 
     cond do
@@ -2902,11 +2905,11 @@ defmodule Enum do
   the default from Erlang/OTP 22:
 
       # Although not necessary, let's seed the random algorithm
-      iex> :rand.seed(:exsss, {1, 2, 3})
-      iex> Enum.shuffle([1, 2, 3])
-      [3, 2, 1]
+      iex> :rand.seed(:exsss, {11, 22, 33})
       iex> Enum.shuffle([1, 2, 3])
       [2, 1, 3]
+      iex> Enum.shuffle([1, 2, 3])
+      [2, 3, 1]
 
   """
   @spec shuffle(t) :: list
@@ -2916,9 +2919,12 @@ defmodule Enum do
         [{:rand.uniform(), x} | acc]
       end)
 
-    shuffle_unwrap(:lists.keysort(1, randomized), [])
+    shuffle_unwrap(:lists.keysort(1, randomized))
   end
 
+  defp shuffle_unwrap([{_, h} | rest]), do: [h | shuffle_unwrap(rest)]
+  defp shuffle_unwrap([]), do: []
+
   @doc """
   Returns a subset list of the given `enumerable` by `index_range`.
 
@@ -3588,101 +3594,115 @@ defmodule Enum do
       # Although not necessary, let's seed the random algorithm
       iex> :rand.seed(:exsss, {1, 2, 3})
       iex> Enum.take_random(1..10, 2)
-      [3, 1]
+      [6, 1]
       iex> Enum.take_random(?a..?z, 5)
-      'mikel'
+      ~c"bkzmt"
 
   """
   @spec take_random(t, non_neg_integer) :: list
   def take_random(enumerable, count)
   def take_random(_enumerable, 0), do: []
-
   def take_random([], _), do: []
-  def take_random([h | t], 1), do: take_random_list_one(t, h, 1)
 
   def take_random(enumerable, 1) do
     enumerable
-    |> reduce([], fn
-      x, [current | index] ->
-        if :rand.uniform(index + 1) == 1 do
-          [x | index + 1]
-        else
-          [current | index + 1]
-        end
+    |> reduce({0, 0, 1.0, nil}, fn
+      elem, {idx, idx, w, _current} ->
+        {jdx, w} = take_jdx_w(idx, w, 1)
+        {idx + 1, jdx, w, elem}
 
-      x, [] ->
-        [x | 1]
+      _elem, {idx, jdx, w, current} ->
+        {idx + 1, jdx, w, current}
     end)
     |> case do
-      [] -> []
-      [current | _index] -> [current]
+      {0, 0, 1.0, nil} -> []
+      {_idx, _jdx, _w, current} -> [current]
     end
   end
 
-  def take_random(enumerable, count) when is_integer(count) and count in 0..128 do
+  def take_random(enumerable, count) when count in 0..128 do
     sample = Tuple.duplicate(nil, count)
 
-    reducer = fn elem, {idx, sample} ->
-      jdx = random_index(idx)
+    reducer = fn
+      elem, {idx, jdx, w, sample} when idx < count ->
+        rand = take_index(idx)
+        sample = sample |> put_elem(idx, elem(sample, rand)) |> put_elem(rand, elem)
 
-      cond do
-        idx < count ->
-          value = elem(sample, jdx)
-          {idx + 1, put_elem(sample, idx, value) |> put_elem(jdx, elem)}
+        if idx == jdx do
+          {jdx, w} = take_jdx_w(idx, w, count)
+          {idx + 1, jdx, w, sample}
+        else
+          {idx + 1, jdx, w, sample}
+        end
 
-        jdx < count ->
-          {idx + 1, put_elem(sample, jdx, elem)}
+      elem, {idx, idx, w, sample} ->
+        pos = :rand.uniform(count) - 1
+        {jdx, w} = take_jdx_w(idx, w, count)
+        {idx + 1, jdx, w, put_elem(sample, pos, elem)}
 
-        true ->
-          {idx + 1, sample}
-      end
+      _elem, {idx, jdx, w, sample} ->
+        {idx + 1, jdx, w, sample}
     end
 
-    {size, sample} = reduce(enumerable, {0, sample}, reducer)
-    sample |> Tuple.to_list() |> take(Kernel.min(count, size))
+    {size, _, _, sample} = reduce(enumerable, {0, count - 1, 1.0, sample}, reducer)
+
+    if count < size do
+      Tuple.to_list(sample)
+    else
+      take_tupled(sample, size, [])
+    end
   end
 
   def take_random(enumerable, count) when is_integer(count) and count >= 0 do
-    reducer = fn elem, {idx, sample} ->
-      jdx = random_index(idx)
+    reducer = fn
+      elem, {idx, jdx, w, sample} when idx < count ->
+        rand = take_index(idx)
+        sample = sample |> Map.put(idx, Map.get(sample, rand)) |> Map.put(rand, elem)
 
-      cond do
-        idx < count ->
-          value = Map.get(sample, jdx)
-          {idx + 1, Map.put(sample, idx, value) |> Map.put(jdx, elem)}
+        if idx == jdx do
+          {jdx, w} = take_jdx_w(idx, w, count)
+          {idx + 1, jdx, w, sample}
+        else
+          {idx + 1, jdx, w, sample}
+        end
 
-        jdx < count ->
-          {idx + 1, Map.put(sample, jdx, elem)}
+      elem, {idx, idx, w, sample} ->
+        pos = :rand.uniform(count) - 1
+        {jdx, w} = take_jdx_w(idx, w, count)
+        {idx + 1, jdx, w, %{sample | pos => elem}}
 
-        true ->
-          {idx + 1, sample}
-      end
+      _elem, {idx, jdx, w, sample} ->
+        {idx + 1, jdx, w, sample}
     end
 
-    {size, sample} = reduce(enumerable, {0, %{}}, reducer)
-    take_random(sample, Kernel.min(count, size), [])
+    {size, _, _, sample} = reduce(enumerable, {0, count - 1, 1.0, %{}}, reducer)
+    take_mapped(sample, Kernel.min(count, size), [])
   end
 
-  defp take_random(_sample, 0, acc), do: acc
+  @compile {:inline, take_jdx_w: 3, take_index: 1}
+  defp take_jdx_w(idx, w, count) do
+    w = w * :math.exp(:math.log(:rand.uniform()) / count)
+    jdx = idx + floor(:math.log(:rand.uniform()) / :math.log(1 - w)) + 1
+    {jdx, w}
+  end
 
-  defp take_random(sample, position, acc) do
+  defp take_index(0), do: 0
+  defp take_index(idx), do: :rand.uniform(idx + 1) - 1
+
+  defp take_tupled(_sample, 0, acc), do: acc
+
+  defp take_tupled(sample, position, acc) do
     position = position - 1
-    take_random(sample, position, [Map.get(sample, position) | acc])
+    take_tupled(sample, position, [elem(sample, position) | acc])
   end
 
-  defp take_random_list_one([h | t], current, index) do
-    if :rand.uniform(index + 1) == 1 do
-      take_random_list_one(t, h, index + 1)
-    else
-      take_random_list_one(t, current, index + 1)
-    end
+  defp take_mapped(_sample, 0, acc), do: acc
+
+  defp take_mapped(sample, position, acc) do
+    position = position - 1
+    take_mapped(sample, position, [Map.fetch!(sample, position) | acc])
   end
 
-  defp take_random_list_one([], current, _), do: [current]
-
-  defp random_index(0), do: 0
-  defp random_index(idx), do: :rand.uniform(idx + 1) - 1
-
   @doc """
   Takes the elements from the beginning of the `enumerable` while `fun` returns
   a truthy value.
@@ -3793,9 +3813,6 @@ defmodule Enum do
       iex> Enum.unzip([{:a, 1}, {:b, 2}, {:c, 3}])
       {[:a, :b, :c], [1, 2, 3]}
 
-      iex> Enum.unzip(%{a: 1, b: 2})
-      {[:a, :b], [1, 2]}
-
   """
   @spec unzip(t) :: {[element], [element]}
 
@@ -3826,7 +3843,7 @@ defmodule Enum do
 
   @doc """
   Returns the `enumerable` with each element wrapped in a tuple
-  alongside its index.
+  alongside its index or according to a given function.
 
   May receive a function or an integer offset.
 
@@ -3876,6 +3893,10 @@ defmodule Enum do
   Zips corresponding elements from two enumerables into a list
   of tuples.
 
+  Because a list of two-element tuples with atoms as the first
+  tuple element is a keyword list (`Keyword`), zipping a first list
+  of atoms with a second list of any kind creates a keyword list.
+
   The zipping finishes as soon as either enumerable completes.
 
   ## Examples
@@ -3883,6 +3904,9 @@ defmodule Enum do
       iex> Enum.zip([1, 2, 3], [:a, :b, :c])
       [{1, :a}, {2, :b}, {3, :c}]
 
+      iex> Enum.zip([:a, :b, :c], [1, 2, 3])
+      [a: 1, b: 2, c: 3]
+
       iex> Enum.zip([1, 2, 3, 4, 5], [:a, :b, :c])
       [{1, :a}, {2, :b}, {3, :c}]
 
@@ -4003,7 +4027,7 @@ defmodule Enum do
   @doc """
   Reduces over two enumerables halting as soon as either enumerable is empty.
 
-  In practice, the behaviour provided by this function can be achieved with:
+  In practice, the behavior provided by this function can be achieved with:
 
       Enum.reduce(Stream.zip(left, right), acc, reducer)
 
@@ -4037,7 +4061,7 @@ defmodule Enum do
   The reducer will receive 2 args: a list of elements (one from each enum) and the
   accumulator.
 
-  In practice, the behaviour provided by this function can be achieved with:
+  In practice, the behavior provided by this function can be achieved with:
 
       Enum.reduce(Stream.zip(enums), acc, reducer)
 
@@ -4051,7 +4075,7 @@ defmodule Enum do
       ...> end)
       [{1, 2, 3}, {1, 2, 3}]
 
-      iex> enums = [[1, 2], %{a: 3, b: 4}, [5, 6]]
+      iex> enums = [[1, 2], [a: 3, b: 4], [5, 6]]
       ...> Enum.zip_reduce(enums, [], fn elements, acc ->
       ...>   [List.to_tuple(elements) | acc]
       ...> end)
@@ -4439,14 +4463,6 @@ defmodule Enum do
     [acc | scan_list(rest, acc, fun)]
   end
 
-  ## shuffle
-
-  defp shuffle_unwrap([{_, h} | enumerable], t) do
-    shuffle_unwrap(enumerable, [h | t])
-  end
-
-  defp shuffle_unwrap([], t), do: t
-
   ## slice
 
   defp slice_forward(enumerable, start, amount, step) when start < 0 do

lib/elixir/lib/exception.ex

@@ -1,11 +1,18 @@
 defmodule Exception do
   @moduledoc """
-  Functions to format throw/catch/exit and exceptions.
+  Functions for dealing with throw/catch/exit and exceptions.
 
-  Note that stacktraces in Elixir are only available inside
-  catch and rescue by using the `__STACKTRACE__/0` variable.
+  This module also defines the behaviour required by custom
+  exceptions. To define your own, see `defexception/1`.
 
-  Do not rely on the particular format returned by the `format*`
+  ## Formatting functions
+
+  Several functions in this module help format exceptions.
+  Some of these functions expect the stacktrace as argument.
+  The stacktrace is typically available inside catch and
+  rescue by using the `__STACKTRACE__/0` variable.
+
+  Do not rely on the particular format returned by the
   functions in this module. They may be changed in future releases
   in order to better suit Elixir's tool chain. In other words,
   by using the functions in this module it is guaranteed you will
@@ -31,7 +38,21 @@ defmodule Exception do
   @type arity_or_args :: non_neg_integer | list
   @type location :: keyword
 
+  @doc """
+  Receives the arguments given to `raise/2` and returns the exception struct.
+
+  The default implementation accepts either a set of keyword arguments
+  that is merged into the struct or a string to be used as the exception's message.
+  """
   @callback exception(term) :: t
+
+  @doc """
+  Receives the exception struct and must return its message.
+
+  Most commonly exceptions have a message field which by default is accessed
+  by this function. However, if an exception does not have a message field,
+  this function must be explicitly implemented.
+  """
   @callback message(t) :: String.t()
 
   @doc """
@@ -793,6 +814,194 @@ defmodule Exception do
       col -> format_file_line_column(Keyword.get(opts, :file), Keyword.get(opts, :line), col, " ")
     end
   end
+
+  @doc false
+  def format_delimiter(delimiter) do
+    if delimiter |> Atom.to_string() |> String.contains?(["\"", "'"]),
+      do: delimiter,
+      else: ~s("#{delimiter}")
+  end
+
+  @doc false
+  def format_snippet(
+        {start_line, _start_column} = start_pos,
+        {end_line, end_column} = end_pos,
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
+      when start_line < end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+
+    relevant_lines =
+      if end_line - start_line < 5 do
+        line_range(lines, start_pos, end_pos, padding, max_digits, start_message, end_message)
+      else
+        trimmed_inbetween_lines(
+          lines,
+          start_pos,
+          end_pos,
+          padding,
+          max_digits,
+          start_message,
+          end_message
+        )
+      end
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{relevant_lines}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  def format_snippet(
+        {start_line, start_column},
+        {end_line, end_column},
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
+      when start_line == end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+    formatted_line = [line_padding(end_line, max_digits), to_string(end_line), " │ ", hd(lines)]
+
+    mismatched_closing_line =
+      [
+        n_spaces(start_column - 1),
+        red("│"),
+        format_end_message(end_column - start_column, end_message)
+      ]
+
+    unclosed_delimiter_line =
+      [padding, " │ ", format_start_message(start_column, start_message)]
+
+    below_line = [padding, " │ ", mismatched_closing_line, "\n", unclosed_delimiter_line]
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{formatted_line}
+    #{below_line}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  defp line_padding(line_number, max_digits) do
+    line_digits = digits(line_number)
+
+    spacing =
+      if line_digits == 1 do
+        max(2, max_digits)
+      else
+        max_digits - line_digits + 1
+      end
+
+    n_spaces(spacing)
+  end
+
+  defp n_spaces(n), do: String.duplicate(" ", n)
+
+  defp digits(number, acc \\ 1)
+  defp digits(number, acc) when number < 10, do: acc
+  defp digits(number, acc), do: digits(div(number, 10), acc + 1)
+
+  defp trimmed_inbetween_lines(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         start_message,
+         end_message
+       ) do
+    start_padding = line_padding(start_line, max_digits)
+    end_padding = line_padding(end_line, max_digits)
+    first_line = hd(lines)
+    last_line = List.last(lines)
+
+    """
+    #{start_padding}#{start_line} │ #{first_line}
+     #{padding}│ #{format_start_message(start_column, start_message)}
+     ...
+    #{end_padding}#{end_line} │ #{last_line}
+     #{padding}│ #{format_end_message(end_column, end_message)}\
+    """
+  end
+
+  defp line_range(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         start_message,
+         end_message
+       ) do
+    Enum.zip_with(lines, start_line..end_line, fn line, line_number ->
+      line_padding = line_padding(line_number, max_digits)
+
+      cond do
+        line_number == start_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            format_start_message(start_column, start_message)
+          ]
+
+        line_number == end_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            format_end_message(end_column, end_message)
+          ]
+
+        true ->
+          [line_padding, to_string(line_number), " │ ", line]
+      end
+    end)
+    |> Enum.intersperse("\n")
+  end
+
+  defp format_end_message(end_column, message),
+    do: [
+      n_spaces(end_column - 1),
+      red(message)
+    ]
+
+  defp format_start_message(start_column, message),
+    do: [n_spaces(start_column - 1), red(message)]
+
+  defp pad_message(message, padding), do: String.replace(message, "\n", "\n #{padding}")
+
+  defp red(string) do
+    if IO.ANSI.enabled?() do
+      [IO.ANSI.red(), string, IO.ANSI.reset()]
+    else
+      string
+    end
+  end
 end
 
 # Some exceptions implement "message/1" instead of "exception/1" mostly
@@ -839,44 +1048,6 @@ defmodule ArgumentError do
   """
 
   defexception message: "argument error"
-
-  @impl true
-  def blame(
-        exception,
-        [{:erlang, :apply, [module, function, args], _} | _] = stacktrace
-      ) do
-    message =
-      cond do
-        not proper_list?(args) ->
-          "you attempted to apply a function named #{inspect(function)} on module #{inspect(module)} " <>
-            "with arguments #{inspect(args)}. Arguments (the third argument of apply) must always be a proper list"
-
-        # Note that args may be an empty list even if they were supplied
-        not is_atom(module) and is_atom(function) and args == [] ->
-          "you attempted to apply a function named #{inspect(function)} on #{inspect(module)}. " <>
-            "If you are using Kernel.apply/3, make sure the module is an atom. " <>
-            "If you are using the dot syntax, such as module.function(), " <>
-            "make sure the left-hand side of the dot is a module atom"
-
-        not is_atom(module) ->
-          "you attempted to apply a function on #{inspect(module)}. " <>
-            "Modules (the first argument of apply) must always be an atom"
-
-        not is_atom(function) ->
-          "you attempted to apply a function named #{inspect(function)} on module #{inspect(module)}. " <>
-            "However, #{inspect(function)} is not a valid function name. Function names (the second argument " <>
-            "of apply) must always be an atom"
-      end
-
-    {%{exception | message: message}, stacktrace}
-  end
-
-  def blame(exception, stacktrace) do
-    {exception, stacktrace}
-  end
-
-  defp proper_list?(list) when length(list) >= 0, do: true
-  defp proper_list?(_), do: false
 end
 
 defmodule ArithmeticError do
@@ -944,11 +1115,24 @@ defmodule MismatchedDelimiterError do
   An exception raised when a mismatched delimiter is found when parsing code.
 
   For example:
-  - `[1, 2, 3}`
-  - `fn a -> )`
-  """
 
-  @max_lines_shown 5
+    * `[1, 2, 3}`
+    * `fn a -> )`
+
+  The following fields of this exceptions are public and can be accessed freely:
+
+    * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
+      the error occurred in code that did not come from a file
+    * `:line` - the line for the opening delimiter
+    * `:column` - the column for the opening delimiter
+    * `:end_line` - the line for the mismatched closing delimiter
+    * `:end_column` - the column for the mismatched closing delimiter
+    * `:opening_delimiter` - an atom representing the opening delimiter
+    * `:closing_delimiter` - an atom representing the mismatched closing delimiter
+    * `:expected_delimiter` - an atom representing the closing delimiter
+    * `:description` - a description of the mismatched delimiter error
+
+  """
 
   defexception [
     :file,
@@ -958,6 +1142,7 @@ defmodule MismatchedDelimiterError do
     :end_column,
     :opening_delimiter,
     :closing_delimiter,
+    :expected_delimiter,
     :snippet,
     description: "mismatched delimiter error"
   ]
@@ -969,206 +1154,32 @@ defmodule MismatchedDelimiterError do
         end_line: end_line,
         end_column: end_column,
         description: description,
-        opening_delimiter: opening_delimiter,
-        closing_delimiter: _closing_delimiter,
+        expected_delimiter: expected_delimiter,
         file: file,
         snippet: snippet
       }) do
     start_pos = {start_line, start_column}
     end_pos = {end_line, end_column}
     lines = String.split(snippet, "\n")
-    expected_delimiter = :elixir_tokenizer.terminator(opening_delimiter)
+    expected_delimiter = Exception.format_delimiter(expected_delimiter)
+
+    start_message = "└ unclosed delimiter"
+    end_message = ~s/└ mismatched closing delimiter (expected #{expected_delimiter})/
+
+    snippet =
+      Exception.format_snippet(
+        start_pos,
+        end_pos,
+        description,
+        file,
+        lines,
+        start_message,
+        end_message
+      )
 
-    snippet = format_snippet(start_pos, end_pos, description, file, lines, expected_delimiter)
     format_message(file, end_line, end_column, snippet)
   end
 
-  defp format_snippet(
-         {start_line, _start_column} = start_pos,
-         {end_line, end_column} = end_pos,
-         description,
-         file,
-         lines,
-         expected_delimiter
-       )
-       when start_line < end_line do
-    max_digits = digits(end_line)
-    general_padding = max(2, max_digits) + 1
-    padding = n_spaces(general_padding)
-
-    relevant_lines =
-      if end_line - start_line < @max_lines_shown do
-        line_range(lines, start_pos, end_pos, padding, max_digits, expected_delimiter)
-      else
-        trimmed_inbetween_lines(
-          lines,
-          start_pos,
-          end_pos,
-          padding,
-          max_digits,
-          expected_delimiter
-        )
-      end
-
-    """
-     #{padding}#{red("error:")} #{pad_message(description, padding)}
-     #{padding}│
-    #{relevant_lines}
-     #{padding}│
-     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
-    """
-  end
-
-  defp format_snippet(
-         {start_line, start_column},
-         {end_line, end_column},
-         description,
-         file,
-         lines,
-         expected_delimiter
-       )
-       when start_line == end_line do
-    max_digits = digits(end_line)
-    general_padding = max(2, max_digits) + 1
-    padding = n_spaces(general_padding)
-
-    line = Enum.fetch!(lines, end_line - 1)
-    formatted_line = [line_padding(end_line, max_digits), to_string(end_line), " │ ", line]
-
-    mismatched_closing_line =
-      [
-        n_spaces(start_column - 1),
-        red("│"),
-        mismatched_closing_delimiter(end_column - start_column, expected_delimiter)
-      ]
-
-    unclosed_delimiter_line =
-      [padding, " │ ", unclosed_delimiter(start_column)]
-
-    below_line = [padding, " │ ", mismatched_closing_line, "\n", unclosed_delimiter_line]
-
-    """
-     #{padding}#{red("error:")} #{pad_message(description, padding)}
-     #{padding}│
-    #{formatted_line}
-    #{below_line}
-     #{padding}│
-     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
-    """
-  end
-
-  defp line_padding(line_number, max_digits) do
-    line_digits = digits(line_number)
-
-    spacing =
-      if line_digits == 1 do
-        max(2, max_digits)
-      else
-        max_digits - line_digits + 1
-      end
-
-    n_spaces(spacing)
-  end
-
-  defp n_spaces(n), do: String.duplicate(" ", n)
-
-  defp digits(number, acc \\ 1)
-  defp digits(number, acc) when number < 10, do: acc
-  defp digits(number, acc), do: digits(div(number, 10), acc + 1)
-
-  defp trimmed_inbetween_lines(
-         lines,
-         {start_line, start_column},
-         {end_line, end_column},
-         padding,
-         max_digits,
-         expected_delimiter
-       ) do
-    start_padding = line_padding(start_line, max_digits)
-    end_padding = line_padding(end_line, max_digits)
-    first_line = Enum.fetch!(lines, start_line - 1)
-    last_line = Enum.fetch!(lines, end_line - 1)
-
-    """
-    #{start_padding}#{start_line} │ #{first_line}
-     #{padding}│ #{unclosed_delimiter(start_column)}
-     ...
-    #{end_padding}#{end_line} │ #{last_line}
-     #{padding}│ #{mismatched_closing_delimiter(end_column, expected_delimiter)}\
-    """
-  end
-
-  defp line_range(
-         lines,
-         {start_line, start_column},
-         {end_line, end_column},
-         padding,
-         max_digits,
-         expected_delimiter
-       ) do
-    start_line = start_line - 1
-    end_line = end_line - 1
-
-    lines
-    |> Enum.slice(start_line..end_line)
-    |> Enum.zip_with(start_line..end_line, fn line, line_number ->
-      line_number = line_number + 1
-      start_line = start_line + 1
-      end_line = end_line + 1
-
-      line_padding = line_padding(line_number, max_digits)
-
-      cond do
-        line_number == start_line ->
-          [
-            line_padding,
-            to_string(line_number),
-            " │ ",
-            line,
-            "\n",
-            padding,
-            " │ ",
-            unclosed_delimiter(start_column)
-          ]
-
-        line_number == end_line ->
-          [
-            line_padding,
-            to_string(line_number),
-            " │ ",
-            line,
-            "\n",
-            padding,
-            " │ ",
-            mismatched_closing_delimiter(end_column, expected_delimiter)
-          ]
-
-        true ->
-          [line_padding, to_string(line_number), " │ ", line]
-      end
-    end)
-    |> Enum.intersperse("\n")
-  end
-
-  defp mismatched_closing_delimiter(end_column, expected_closing_delimiter),
-    do: [
-      n_spaces(end_column - 1),
-      red(~s/└ mismatched closing delimiter (expected "#{expected_closing_delimiter}")/)
-    ]
-
-  defp unclosed_delimiter(start_column),
-    do: [n_spaces(start_column - 1), red("└ unclosed delimiter")]
-
-  defp pad_message(message, padding), do: String.replace(message, "\n", "\n #{padding}")
-
-  defp red(string) do
-    if IO.ANSI.enabled?() do
-      [IO.ANSI.red(), string, IO.ANSI.reset()]
-    else
-      string
-    end
-  end
-
   defp format_message(file, line, column, message) do
     location = Exception.format_file_line_column(Path.relative_to_cwd(file), line, column)
     "mismatched delimiter found on " <> location <> "\n" <> message
@@ -1183,8 +1194,9 @@ defmodule SyntaxError do
 
     * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
       the error occurred in code that did not come from a file
-    * `:line` (`t:non_neg_integer/0`) - the line where the error occurred
-    * `:column` (`t:non_neg_integer/0`) - the column where the error occurred
+    * `:line` - the line where the error occurred
+    * `:column` - the column where the error occurred
+    * `:description` - a description of the syntax error
 
   """
 
@@ -1200,7 +1212,7 @@ defmodule SyntaxError do
       })
       when not is_nil(snippet) and not is_nil(column) do
     snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil)
+      :elixir_errors.format_snippet(:error, {line, column}, file, description, snippet, %{})
 
     format_message(file, line, column, snippet)
   end
@@ -1213,7 +1225,7 @@ defmodule SyntaxError do
         description: description
       }) do
     snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, nil, :error, [], nil)
+      :elixir_errors.format_snippet(:error, {line, column}, file, description, nil, %{})
 
     padded = "   " <> String.replace(snippet, "\n", "\n   ")
     format_message(file, line, column, padded)
@@ -1233,17 +1245,24 @@ defmodule TokenMissingError do
 
     * `:file` (`t:Path.t/0` or `nil`) - the file where the error occurred, or `nil` if
       the error occurred in code that did not come from a file
-    * `:line` (`t:non_neg_integer/0`) - the line where the error occurred
-    * `:column` (`t:non_neg_integer/0`) - the column where the error occurred
-
+    * `:line` - the line for the opening delimiter
+    * `:column` - the column for the opening delimiter
+    * `:end_line` - the line for the end of the string
+    * `:end_column` - the column for the end of the string
+    * `:opening_delimiter` - an atom representing the opening delimiter
+    * `:expected_delimiter` - an atom representing the expected delimiter
+    * `:description` - a description of the missing token error
   """
 
   defexception [
     :file,
     :line,
-    :snippet,
     :column,
+    :end_line,
+    :end_column,
+    :snippet,
     :opening_delimiter,
+    :expected_delimiter,
     description: "expression is incomplete"
   ]
 
@@ -1252,14 +1271,40 @@ defmodule TokenMissingError do
         file: file,
         line: line,
         column: column,
+        end_line: end_line,
         description: description,
+        expected_delimiter: expected_delimiter,
         snippet: snippet
       })
-      when not is_nil(snippet) and not is_nil(column) do
-    snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, snippet, :error, [], nil)
+      when not is_nil(snippet) and not is_nil(column) and not is_nil(end_line) do
+    {trimmed, [last_line | _] = reversed_lines} =
+      snippet
+      |> String.split("\n")
+      |> Enum.reverse()
+      |> Enum.split_while(&(&1 == ""))
 
-    format_message(file, line, column, snippet)
+    end_line = end_line - length(trimmed)
+    end_column = String.length(last_line) + 1
+
+    start_pos = {line, column}
+    end_pos = {end_line, end_column}
+    expected_delimiter = Exception.format_delimiter(expected_delimiter)
+
+    start_message = ~s/└ unclosed delimiter/
+    end_message = ~s/└ missing closing delimiter (expected #{expected_delimiter})/
+
+    snippet =
+      Exception.format_snippet(
+        start_pos,
+        end_pos,
+        description,
+        file,
+        Enum.reverse(reversed_lines),
+        start_message,
+        end_message
+      )
+
+    format_message(file, end_line, end_column, snippet)
   end
 
   @impl true
@@ -1267,13 +1312,13 @@ defmodule TokenMissingError do
         file: file,
         line: line,
         column: column,
+        snippet: snippet,
         description: description
       }) do
     snippet =
-      :elixir_errors.format_snippet({line, column}, file, description, nil, :error, [], nil)
+      :elixir_errors.format_snippet(:error, {line, column}, file, description, snippet, %{})
 
-    padded = "   " <> String.replace(snippet, "\n", "\n   ")
-    format_message(file, line, column, padded)
+    format_message(file, line, column, snippet)
   end
 
   defp format_message(file, line, column, message) do
@@ -1998,10 +2043,17 @@ defmodule KeyError do
   defp message(key, term) do
     message = "key #{inspect(key)} not found"
 
-    if term != nil do
-      message <> " in: #{inspect(term, pretty: true, limit: :infinity)}"
-    else
-      message
+    cond do
+      term == nil ->
+        message
+
+      is_atom(term) and is_atom(key) ->
+        message <>
+          " in: #{inspect(term)} (if instead you want to invoke #{inspect(term)}.#{key}(), " <>
+          "make sure to add parentheses after the function name)"
+
+      true ->
+        message <> " in: #{inspect(term, pretty: true, limit: :infinity)}"
     end
   end
 
@@ -2010,11 +2062,6 @@ defmodule KeyError do
     {exception, stacktrace}
   end
 
-  def blame(exception = %{term: nil}, stacktrace) do
-    message = message(exception.key, exception.term)
-    {%{exception | message: message}, stacktrace}
-  end
-
   def blame(exception, stacktrace) do
     %{term: term, key: key} = exception
     message = message(key, term)
@@ -2231,9 +2278,39 @@ defmodule ErlangError do
 
   @doc false
   def normalize(:badarg, stacktrace) do
-    case error_info(:badarg, stacktrace, "errors were found at the given arguments") do
-      {:ok, reason, details} -> %ArgumentError{message: reason <> details}
-      :error -> %ArgumentError{}
+    case stacktrace do
+      [{:erlang, :apply, [module, function, args], _} | _] when not is_atom(module) ->
+        message =
+          cond do
+            is_map(module) and is_atom(function) and is_map_key(module, function) ->
+              "you attempted to apply a function named #{inspect(function)} on a map/struct. " <>
+                "If you are using Kernel.apply/3, make sure the module is an atom. " <>
+                if is_function(module[function]) do
+                  "If you are trying to invoke an anonymous function in a map/struct, " <>
+                    "add a dot between the function name and the parenthesis: map.#{function}.()"
+                else
+                  "If you are using the dot syntax, ensure there are no parentheses " <>
+                    "after the field name, such as map.#{function}"
+                end
+
+            is_atom(function) and args == [] ->
+              "you attempted to apply a function named #{inspect(function)} on #{inspect(module)}. " <>
+                "If you are using Kernel.apply/3, make sure the module is an atom. " <>
+                "If you are using the dot syntax, such as module.function(), " <>
+                "make sure the left-hand side of the dot is an atom representing a module"
+
+            true ->
+              "you attempted to apply a function on #{inspect(module)}. " <>
+                "Modules (the first argument of apply) must always be an atom"
+          end
+
+        %ArgumentError{message: message}
+
+      _ ->
+        case error_info(:badarg, stacktrace, "errors were found at the given arguments") do
+          {:ok, reason, details} -> %ArgumentError{message: reason <> details}
+          :error -> %ArgumentError{}
+        end
     end
   end
 

lib/elixir/lib/file.ex

@@ -101,6 +101,7 @@ defmodule File do
 
   @type posix :: :file.posix()
   @type io_device :: :file.io_device()
+  @type file_descriptor :: :file.fd()
   @type stat_options :: [time: :local | :universal | :posix]
   @type mode ::
           :append
@@ -133,6 +134,7 @@ defmodule File do
 
   @type stream_mode ::
           encoding_mode()
+          | read_offset_mode()
           | :append
           | :compressed
           | :delayed_write
@@ -140,6 +142,8 @@ defmodule File do
           | {:read_ahead, pos_integer | false}
           | {:delayed_write, non_neg_integer, non_neg_integer}
 
+  @type read_offset_mode :: {:read_offset, non_neg_integer()}
+
   @type erlang_time ::
           {{year :: non_neg_integer(), month :: 1..12, day :: 1..31},
            {hour :: 0..23, minute :: 0..59, second :: 0..59}}
@@ -755,7 +759,7 @@ defmodule File do
 
   Note: The command `mv` in Unix-like systems behaves differently depending on
   whether `source` is a file and the `destination` is an existing directory.
-  We have chosen to explicitly disallow this behaviour.
+  We have chosen to explicitly disallow this behavior.
 
   ## Examples
 
@@ -819,7 +823,7 @@ defmodule File do
       The function receives arguments for `source_file` and `destination_file`. It should
       return `true` if the existing file should be overwritten, `false` if otherwise.
       The default callback returns `true`. On earlier versions, this callback could be
-      given as third argument, but such behaviour is now deprecated.
+      given as third argument, but such behavior is now deprecated.
 
   """
   @spec cp(Path.t(), Path.t(), on_conflict: on_conflict_callback) :: :ok | {:error, posix}
@@ -892,7 +896,7 @@ defmodule File do
 
   Note: The command `cp` in Unix-like systems behaves differently depending on
   whether `destination` is an existing directory or not. We have chosen to
-  explicitly disallow this behaviour. If `source` is a `file` and `destination`
+  explicitly disallow this behavior. If `source` is a `file` and `destination`
   is a directory, `{:error, :eisdir}` will be returned.
 
   ## Options
@@ -901,7 +905,7 @@ defmodule File do
       The function receives arguments for `source` and `destination`. It should return
       `true` if the existing file should be overwritten, `false` if otherwise. The default
       callback returns `true`. On earlier versions, this callback could be given as third
-      argument, but such behaviour is now deprecated.
+      argument, but such behavior is now deprecated.
 
     * `:dereference_symlinks` - (since v1.14.0) By default, this function will copy symlinks
       by creating symlinks that point to the same location. This option forces symlinks to be
@@ -1360,6 +1364,11 @@ defmodule File do
       {:ok, _} ->
         {:ok, :regular}
 
+      {:error, :eio} when major == :win32 ->
+        # unix domain socket returns `{:error, :eio}`
+        # on other platforms the result is `{:ok, :regular}`
+        {:ok, :regular}
+
       {:error, reason} ->
         {:error, reason}
     end
@@ -1386,13 +1395,6 @@ defmodule File do
   @doc ~S"""
   Opens the given `path`.
 
-  In order to write and read files, one must use the functions
-  in the `IO` module. By default, a file is opened in `:binary` mode,
-  which requires the functions `IO.binread/2` and `IO.binwrite/2`
-  to interact with the file. A developer may pass `:utf8` as an
-  option when opening the file and then all other functions from
-  `IO` are available, since they work directly with Unicode data.
-
   `modes_or_function` can either be a list of modes or a function. If it's a
   list, it's considered to be a list of modes (that are documented below). If
   it's a function, then it's equivalent to calling `open(path, [],
@@ -1401,8 +1403,8 @@ defmodule File do
 
   The allowed modes:
 
-    * `:binary` - opens the file in binary mode, disabling special handling of Unicode sequences
-      (default mode).
+    * `:binary` - opens the file in binary mode, disabling special handling of
+      Unicode sequences (default mode).
 
     * `:read` - the file, which must exist, is opened for reading.
 
@@ -1443,19 +1445,33 @@ defmodule File do
 
   This function returns:
 
-    * `{:ok, io_device}` - the file has been opened in the requested mode.
+    * `{:ok, io_device | file_descriptor}` - the file has been opened in
+      the requested mode. We explore the differences between these two results
+      in the following section
 
-      `io_device` is actually the PID of the process which handles the file.
-      This process monitors the process that originally opened the file (the
-      owner process). If the owner process terminates, the file is closed and
-      the process itself terminates too. If any process to which the `io_device`
-      is linked terminates, the file will be closed and the process itself will
-      be terminated.
+    * `{:error, reason}` - the file could not be opened due to `reason`.
 
-      An `io_device` returned from this call can be used as an argument to the
-      `IO` module functions.
+  ## IO devices
 
-    * `{:error, reason}` - the file could not be opened.
+  By default, this function returns an IO device. An `io_device` is
+  a process which handles the file and you can interact with it using
+  the functions in the `IO` module. By default, a file is opened in
+  `:binary` mode, which requires the functions `IO.binread/2` and
+  `IO.binwrite/2` to interact with the file. A developer may pass `:utf8`
+  as a mode when opening the file and then all other functions from
+  `IO` are available, since they work directly with Unicode data.
+
+  Given the IO device is a file, if the owner process terminates,
+  the file is closed and the process itself terminates too. If any
+  process to which the `io_device` is linked terminates, the file
+  will be closed and the process itself will be terminated.
+
+  ## File descriptors
+
+  When the `:raw` or `:ram` modes are given, this function returns
+  a low-level file descriptors. This avoids creating a process but
+  requires using the functions in the [`:file`](`:file`) module to
+  interact with it.
 
   ## Examples
 
@@ -1464,8 +1480,9 @@ defmodule File do
       File.close(file)
 
   """
-  @spec open(Path.t(), [mode | :ram]) :: {:ok, io_device} | {:error, posix}
-  @spec open(Path.t(), (io_device -> res)) :: {:ok, res} | {:error, posix} when res: var
+  @spec open(Path.t(), [mode | :ram]) :: {:ok, io_device | file_descriptor} | {:error, posix}
+  @spec open(Path.t(), (io_device | file_descriptor -> res)) :: {:ok, res} | {:error, posix}
+        when res: var
   def open(path, modes_or_function \\ [])
 
   def open(path, modes) when is_list(modes) do
@@ -1499,7 +1516,8 @@ defmodule File do
 
   See `open/2` for the list of available `modes`.
   """
-  @spec open(Path.t(), [mode | :ram], (io_device -> res)) :: {:ok, res} | {:error, posix}
+  @spec open(Path.t(), [mode | :ram], (io_device | file_descriptor -> res)) ::
+          {:ok, res} | {:error, posix}
         when res: var
   def open(path, modes, function) when is_list(modes) and is_function(function, 1) do
     case open(path, modes) do
@@ -1521,8 +1539,8 @@ defmodule File do
 
   See `open/2` for the list of available modes.
   """
-  @spec open!(Path.t(), [mode | :ram]) :: io_device
-  @spec open!(Path.t(), (io_device -> res)) :: res when res: var
+  @spec open!(Path.t(), [mode | :ram]) :: io_device | file_descriptor
+  @spec open!(Path.t(), (io_device | file_descriptor -> res)) :: res when res: var
   def open!(path, modes_or_function \\ []) do
     case open(path, modes_or_function) do
       {:ok, io_device_or_function_result} ->
@@ -1541,7 +1559,7 @@ defmodule File do
 
   See `open/2` for the list of available `modes`.
   """
-  @spec open!(Path.t(), [mode | :ram], (io_device -> res)) :: res when res: var
+  @spec open!(Path.t(), [mode | :ram], (io_device | file_descriptor -> res)) :: res when res: var
   def open!(path, modes, function) do
     case open(path, modes, function) do
       {:ok, function_result} ->
@@ -1748,18 +1766,20 @@ defmodule File do
   One may also consider passing the `:delayed_write` option if the stream
   is meant to be written to under a tight loop.
 
-  ## Byte order marks
+  ## Byte order marks and read offset
 
   If you pass `:trim_bom` in the modes parameter, the stream will
   trim UTF-8, UTF-16 and UTF-32 byte order marks when reading from file.
 
   Note that this function does not try to discover the file encoding
-  based on BOM.
+  based on BOM. From Elixir v1.16.0, you may also pass a `:read_offset`
+  that is skipped whenever enumerating the stream (if both `:read_offset`
+  and `:trim_bom` are given, the offset is skipped after the BOM).
 
   ## Examples
 
       # Read a utf8 text file which may include BOM
-      File.stream!("./test/test.txt", encoding: :utf8, trim_bom: true)
+      File.stream!("./test/test.txt", [:trim_bom, encoding: :utf8])
 
       # Read in 2048 byte chunks rather than lines
       File.stream!("./test/test.data", 2048)

lib/elixir/lib/file/stat.ex

@@ -60,19 +60,19 @@ defmodule File.Stat do
   defstruct keys
 
   @type t :: %__MODULE__{
-          size: non_neg_integer(),
-          type: :device | :directory | :regular | :other | :symlink,
-          access: :read | :write | :read_write | :none,
-          atime: :calendar.datetime() | integer(),
-          mtime: :calendar.datetime() | integer(),
-          ctime: :calendar.datetime() | integer(),
-          mode: non_neg_integer(),
-          links: non_neg_integer(),
-          major_device: non_neg_integer(),
-          minor_device: non_neg_integer(),
-          inode: non_neg_integer(),
-          uid: non_neg_integer(),
-          gid: non_neg_integer()
+          size: non_neg_integer() | :undefined,
+          type: :device | :directory | :regular | :other | :symlink | :undefined,
+          access: :read | :write | :read_write | :none | :undefined,
+          atime: :calendar.datetime() | integer() | :undefined,
+          mtime: :calendar.datetime() | integer() | :undefined,
+          ctime: :calendar.datetime() | integer() | :undefined,
+          mode: non_neg_integer() | :undefined,
+          links: non_neg_integer() | :undefined,
+          major_device: non_neg_integer() | :undefined,
+          minor_device: non_neg_integer() | :undefined,
+          inode: non_neg_integer() | :undefined,
+          uid: non_neg_integer() | :undefined,
+          gid: non_neg_integer() | :undefined
         }
 
   @doc """

lib/elixir/lib/file/stream.ex

@@ -18,6 +18,12 @@ defmodule File.Stream do
 
   @doc false
   def __build__(path, line_or_bytes, modes) do
+    with {:read_offset, offset} <- :lists.keyfind(:read_offset, 1, modes),
+         false <- is_integer(offset) and offset >= 0 do
+      raise ArgumentError,
+            "expected :read_offset to be a non-negative integer, got: #{inspect(offset)}"
+    end
+
     raw = :lists.keyfind(:encoding, 1, modes) == false
 
     modes =
@@ -88,7 +94,7 @@ defmodule File.Stream do
       start_fun = fn ->
         case File.Stream.__open__(stream, read_modes(modes)) do
           {:ok, device} ->
-            if :trim_bom in modes, do: trim_bom(device, raw) |> elem(0), else: device
+            skip_bom_and_offset(device, raw, modes)
 
           {:error, reason} ->
             raise File.Error, reason: reason, action: "stream", path: stream.path
@@ -104,9 +110,14 @@ defmodule File.Stream do
       Stream.resource(start_fun, next_fun, &:file.close/1).(acc, fun)
     end
 
-    def count(%{modes: modes, line_or_bytes: :line, path: path} = stream) do
+    def count(%{modes: modes, line_or_bytes: :line, path: path, raw: raw} = stream) do
       pattern = :binary.compile_pattern("\n")
-      counter = &count_lines(&1, path, pattern, read_function(stream), 0)
+
+      counter = fn device ->
+        device = skip_bom_and_offset(device, raw, modes)
+        count_lines(device, path, pattern, read_function(stream), 0)
+      end
+
       {:ok, open!(stream, modes, counter)}
     end
 
@@ -116,8 +127,11 @@ defmodule File.Stream do
           {:error, __MODULE__}
 
         {:ok, %{size: size}} ->
+          bom_offset = count_raw_bom(stream, modes)
+          offset = get_read_offset(modes)
+          size = max(size - bom_offset - offset, 0)
           remainder = if rem(size, bytes) == 0, do: 0, else: 1
-          {:ok, div(size, bytes) + remainder - count_raw_bom(stream, modes)}
+          {:ok, div(size, bytes) + remainder}
 
         {:error, reason} ->
           raise File.Error, reason: reason, action: "stream", path: path
@@ -158,6 +172,23 @@ defmodule File.Stream do
       end
     end
 
+    defp skip_bom_and_offset(device, raw, modes) do
+      device =
+        if :trim_bom in modes do
+          device |> trim_bom(raw) |> elem(0)
+        else
+          device
+        end
+
+      offset = get_read_offset(modes)
+
+      if offset > 0 do
+        {:ok, _} = :file.position(device, {:cur, offset})
+      end
+
+      device
+    end
+
     defp trim_bom(device, true) do
       bom_length = device |> IO.binread(4) |> bom_length()
       {:ok, new_pos} = :file.position(device, bom_length)
@@ -183,6 +214,13 @@ defmodule File.Stream do
     defp bom_length(<<254, 255, 0, 0, _rest::binary>>), do: 4
     defp bom_length(_binary), do: 0
 
+    def get_read_offset(modes) do
+      case :lists.keyfind(:read_offset, 1, modes) do
+        {:read_offset, offset} -> offset
+        false -> 0
+      end
+    end
+
     defp read_modes(modes) do
       for mode <- modes, mode not in [:write, :append, :trim_bom], do: mode
     end

lib/elixir/lib/float.ex

@@ -198,7 +198,7 @@ defmodule Float do
   defp add_dot(acc, false), do: acc <> ".0"
 
   @doc """
-  Rounds a float to the largest number less than or equal to `num`.
+  Rounds a float to the largest float less than or equal to `number`.
 
   `floor/2` also accepts a precision to round a floating-point value down
   to an arbitrary number of fractional digits (between 0 and 15).
@@ -210,7 +210,7 @@ defmodule Float do
 
   ## Known issues
 
-  The behaviour of `floor/2` for floats can be surprising. For example:
+  The behavior of `floor/2` for floats can be surprising. For example:
 
       iex> Float.floor(12.52, 2)
       12.51
@@ -218,7 +218,7 @@ defmodule Float do
   One may have expected it to floor to 12.52. This is not a bug.
   Most decimal fractions cannot be represented as a binary floating point
   and therefore the number above is internally represented as 12.51999999,
-  which explains the behaviour above.
+  which explains the behavior above.
 
   ## Examples
 
@@ -246,7 +246,7 @@ defmodule Float do
   end
 
   @doc """
-  Rounds a float to the smallest integer greater than or equal to `num`.
+  Rounds a float to the smallest float greater than or equal to `number`.
 
   `ceil/2` also accepts a precision to round a floating-point value down
   to an arbitrary number of fractional digits (between 0 and 15).
@@ -254,7 +254,7 @@ defmodule Float do
   The operation is performed on the binary floating point, without a
   conversion to decimal.
 
-  The behaviour of `ceil/2` for floats can be surprising. For example:
+  The behavior of `ceil/2` for floats can be surprising. For example:
 
       iex> Float.ceil(-12.52, 2)
       -12.51
@@ -262,7 +262,7 @@ defmodule Float do
   One may have expected it to ceil to -12.52. This is not a bug.
   Most decimal fractions cannot be represented as a binary floating point
   and therefore the number above is internally represented as -12.51999999,
-  which explains the behaviour above.
+  which explains the behavior above.
 
   This function always returns floats. `Kernel.trunc/1` may be used instead to
   truncate the result to an integer afterwards.
@@ -275,6 +275,8 @@ defmodule Float do
       -56.0
       iex> Float.ceil(34.251, 2)
       34.26
+      iex> Float.ceil(-0.01)
+      -0.0
 
   """
   @spec ceil(float, precision_range) :: float
@@ -305,7 +307,7 @@ defmodule Float do
 
   ## Known issues
 
-  The behaviour of `round/2` for floats can be surprising. For example:
+  The behavior of `round/2` for floats can be surprising. For example:
 
       iex> Float.round(5.5675, 3)
       5.567
@@ -313,8 +315,8 @@ defmodule Float do
   One may have expected it to round to the half up 5.568. This is not a bug.
   Most decimal fractions cannot be represented as a binary floating point
   and therefore the number above is internally represented as 5.567499999,
-  which explains the behaviour above. If you want exact rounding for decimals,
-  you must use a decimal library. The behaviour above is also in accordance
+  which explains the behavior above. If you want exact rounding for decimals,
+  you must use a decimal library. The behavior above is also in accordance
   to reference implementations, such as "Correctly Rounded Binary-Decimal and
   Decimal-Binary Conversions" by David M. Gay.
 
@@ -332,6 +334,8 @@ defmodule Float do
       -6.0
       iex> Float.round(12.341444444444441, 15)
       12.341444444444441
+      iex> Float.round(-0.01)
+      -0.0
 
   """
   @spec round(float, precision_range) :: float
@@ -340,8 +344,13 @@ defmodule Float do
   # and could be implemented in the future.
   def round(float, precision \\ 0)
 
+  def round(float, 0) when float == 0.0, do: float
+
   def round(float, 0) when is_float(float) do
-    float |> :erlang.round() |> :erlang.float()
+    case float |> :erlang.round() |> :erlang.float() do
+      zero when zero == 0.0 and float < 0.0 -> -0.0
+      rounded -> rounded
+    end
   end
 
   def round(float, precision) when is_float(float) and precision in @precision_range do
@@ -365,6 +374,8 @@ defmodule Float do
         case rounding do
           :ceil when sign === 0 -> 1 / power_of_10(precision)
           :floor when sign === 1 -> -1 / power_of_10(precision)
+          :ceil when sign === 1 -> minus_zero()
+          :half_up when sign === 1 -> minus_zero()
           _ -> 0.0
         end
 
@@ -394,6 +405,9 @@ defmodule Float do
         boundary = den <<< 52
 
         cond do
+          num == 0 and sign == 1 ->
+            minus_zero()
+
           num == 0 ->
             0.0
 
@@ -408,6 +422,11 @@ defmodule Float do
     end
   end
 
+  # TODO remove once we require Erlang/OTP 27+
+  # This function tricks the compiler to avoid this bug in previous versions:
+  # https://github.com/elixir-lang/elixir/blob/main/lib/elixir/lib/float.ex#L408-L412
+  defp minus_zero, do: -0.0
+
   defp decompose(significant, initial) do
     decompose(significant, 1, 0, initial)
   end
@@ -571,12 +590,12 @@ defmodule Float do
   ## Examples
 
       iex> Float.to_charlist(7.0)
-      '7.0'
+      ~c"7.0"
 
   """
   @spec to_charlist(float) :: charlist
   def to_charlist(float) when is_float(float) do
-    :io_lib_format.fwrite_g(float)
+    :erlang.float_to_list(float, [:short])
   end
 
   @doc """
@@ -603,7 +622,7 @@ defmodule Float do
   """
   @spec to_string(float) :: String.t()
   def to_string(float) when is_float(float) do
-    IO.iodata_to_binary(:io_lib_format.fwrite_g(float))
+    :erlang.float_to_binary(float, [:short])
   end
 
   @doc false

lib/elixir/lib/gen_event.ex

@@ -22,7 +22,7 @@ defmodule GenEvent do
   This approach has some shortcomings (it provides no back-pressure for example)
   but can still replace GenEvent for low-profile usages of it. [This blog post
   by José
-  Valim](http://blog.plataformatec.com.br/2016/11/replacing-genevent-by-a-supervisor-genserver/)
+  Valim](https://dashbit.co/blog/replacing-genevent-by-a-supervisor-plus-genserver)
   has more detailed information on this approach.
 
   ### GenStage
@@ -507,6 +507,8 @@ defmodule GenEvent do
     {:ok, states, [name, handlers, hib]}
   end
 
+  # Keeping deprecated format_status/2 since the current implementation is not
+  # compatible with format_status/1 and GenEvent is deprecated anyway
   @doc false
   def format_status(opt, status_data) do
     [pdict, sys_state, parent, _debug, [name, handlers, _hib]] = status_data

lib/elixir/lib/gen_server.ex

@@ -9,14 +9,10 @@ defmodule GenServer do
   tracing and error reporting. It will also fit into a supervision tree.
 
   ```mermaid
-  graph TD
-      GenServer
-      GenServer -. reply -.-> A
-      GenServer -. reply -.-> B
-      GenServer -. reply -.-> C
-      A(Client #1) -- request --> GenServer
-      B(Client #2) -- request --> GenServer
-      C(Client #3) -- request --> GenServer
+  graph BT
+      C(Client #3) ~~~ B(Client #2) ~~~ A(Client #1)
+      A & B & C -->|request| GenServer
+      GenServer -.->|reply| A & B & C
   ```
 
   ## Example
@@ -404,7 +400,7 @@ defmodule GenServer do
       or is suspended, the parent PID, the debugger state, and the state of
       the behaviour module, which includes the callback module state
       (as returned by `:sys.get_state/2`). It's possible to change how this
-      status is represented by defining the optional `c:GenServer.format_status/2`
+      status is represented by defining the optional `c:GenServer.format_status/1`
       callback.
     * `:sys.trace/3` - prints all the system events to `:stdio`.
     * `:sys.statistics/3` - manages collection of process statistics.
@@ -475,7 +471,7 @@ defmodule GenServer do
   guide provides a tutorial-like introduction. The documentation and links
   in Erlang can also provide extra insight.
 
-    * [GenServer - Elixir's Getting Started Guide](https://elixir-lang.org/getting-started/mix-otp/genserver.html)
+    * [GenServer - Elixir's Getting Started Guide](genservers.md)
     * [`:gen_server` module documentation](`:gen_server`)
     * [gen_server Behaviour - OTP Design Principles](https://www.erlang.org/doc/design_principles/gen_server_concepts.html)
     * [Clients and Servers - Learn You Some Erlang for Great Good!](http://learnyousomeerlang.com/clients-and-servers)
@@ -763,22 +759,39 @@ defmodule GenServer do
             when old_vsn: term | {:down, term}
 
   @doc """
-  Invoked in some cases to retrieve a formatted version of the `GenServer` status:
+  This function is called by a `GenServer` process in the following situations:
 
-    * one of `:sys.get_status/1` or `:sys.get_status/2` is invoked to get the
-      status of the `GenServer`; in such cases, `reason` is `:normal`
+    * [`:sys.get_status/1,2`](`:sys.get_status/1`) is invoked to get the `GenServer` status.
+    * The `GenServer` process terminates abnormally and logs an error.
 
-    * the `GenServer` terminates abnormally and logs an error; in such cases,
-      `reason` is `:terminate`
+  This callback is used to limit the status of the process returned by
+  [`:sys.get_status/1,2`](`:sys.get_status/1`) or sent to logger.
 
-  This callback can be useful to control the *appearance* of the status of the
-  `GenServer`. For example, it can be used to return a compact representation of
-  the `GenServer`'s state to avoid having large state terms printed.
+  The callback gets a map `status` describing the current status and shall return
+  a map `new_status` with the same keys, but it may transform some values.
+
+  Two possible use cases for this callback is to remove sensitive information
+  from the state to prevent it from being printed in log files, or to compact
+  large irrelevant status items that would only clutter the logs.
+
+  ## Example
+
+      @impl GenServer
+      def format_status(status) do
+        Map.new(status, fn
+          {:state, state} -> {:state, Map.delete(state, :private_key)}
+          {:message, {:password, _}} -> {:message, {:password, "redacted"}}
+          key_value -> key_value
+        end)
+      end
 
-  `pdict_and_state` is a two-elements list `[pdict, state]` where `pdict` is a
-  list of `{key, value}` tuples representing the current process dictionary of
-  the `GenServer` and `state` is the current state of the `GenServer`.
   """
+  @doc since: "1.17.0"
+  @callback format_status(status :: :gen_server.format_status()) ::
+              new_status :: :gen_server.format_status()
+
+  # TODO: Remove this on v2.0
+  @doc deprecated: "Use format_status/1 callback instead"
   @callback format_status(reason, pdict_and_state :: list) :: term
             when reason: :normal | :terminate
 
@@ -787,6 +800,7 @@ defmodule GenServer do
                       handle_info: 2,
                       handle_cast: 2,
                       handle_call: 3,
+                      format_status: 1,
                       format_status: 2,
                       handle_continue: 2
 
@@ -1106,10 +1120,6 @@ defmodule GenServer do
       nil ->
         exit({:noproc, {__MODULE__, :call, [server, request, timeout]}})
 
-      # TODO: remove this clause when we require Erlang/OTP 25+
-      pid when pid == self() ->
-        exit({:calling_self, {__MODULE__, :call, [server, request, timeout]}})
-
       pid ->
         try do
           :gen.call(pid, :"$gen_call", request, timeout)

lib/elixir/lib/hash_dict.ex

@@ -258,13 +258,13 @@ end
 defimpl Enumerable, for: HashDict do
   def reduce(dict, acc, fun) do
     # Avoid warnings about HashDict being deprecated.
-    module = HashDict
+    module = String.to_atom("HashDict")
     module.reduce(dict, acc, fun)
   end
 
   def member?(dict, {key, value}) do
     # Avoid warnings about HashDict being deprecated.
-    module = HashDict
+    module = String.to_atom("HashDict")
     {:ok, match?({:ok, ^value}, module.fetch(dict, key))}
   end
 
@@ -274,7 +274,7 @@ defimpl Enumerable, for: HashDict do
 
   def count(dict) do
     # Avoid warnings about HashDict being deprecated.
-    module = HashDict
+    module = String.to_atom("HashDict")
     {:ok, module.size(dict)}
   end
 
@@ -286,7 +286,7 @@ end
 defimpl Collectable, for: HashDict do
   def into(original) do
     # Avoid warnings about HashDict being deprecated.
-    module = HashDict
+    module = String.to_atom("HashDict")
 
     collector_fun = fn
       dict, {:cont, {key, value}} -> module.put(dict, key, value)
@@ -303,7 +303,7 @@ defimpl Inspect, for: HashDict do
 
   def inspect(dict, opts) do
     # Avoid warnings about HashDict being deprecated.
-    module = HashDict
+    module = String.to_atom("HashDict")
     concat(["#HashDict<", Inspect.List.inspect(module.to_list(dict), opts), ">"])
   end
 end

lib/elixir/lib/hash_set.ex

@@ -272,19 +272,19 @@ end
 defimpl Enumerable, for: HashSet do
   def reduce(set, acc, fun) do
     # Avoid warnings about HashSet being deprecated.
-    module = HashSet
+    module = String.to_atom("HashSet")
     module.reduce(set, acc, fun)
   end
 
   def member?(set, term) do
     # Avoid warnings about HashSet being deprecated.
-    module = HashSet
+    module = String.to_atom("HashSet")
     {:ok, module.member?(set, term)}
   end
 
   def count(set) do
     # Avoid warnings about HashSet being deprecated.
-    module = HashSet
+    module = String.to_atom("HashSet")
     {:ok, module.size(set)}
   end
 
@@ -296,7 +296,7 @@ end
 defimpl Collectable, for: HashSet do
   def into(original) do
     # Avoid warnings about HashSet being deprecated.
-    module = HashSet
+    module = String.to_atom("HashSet")
 
     collector_fun = fn
       set, {:cont, term} -> module.put(set, term)
@@ -313,7 +313,7 @@ defimpl Inspect, for: HashSet do
 
   def inspect(set, opts) do
     # Avoid warnings about HashSet being deprecated.
-    module = HashSet
+    module = String.to_atom("HashSet")
     concat(["#HashSet<", Inspect.List.inspect(module.to_list(set), opts), ">"])
   end
 end

lib/elixir/lib/inspect.ex

@@ -387,7 +387,7 @@ defimpl Inspect, for: Float do
       if abs >= 1.0 and abs < 1.0e16 and trunc(float) == float do
         [Integer.to_string(trunc(float)), ?., ?0]
       else
-        :io_lib_format.fwrite_g(float)
+        :erlang.float_to_list(float, [:short])
       end
 
     color(IO.iodata_to_binary(formatted), :number, opts)
@@ -396,24 +396,38 @@ end
 
 defimpl Inspect, for: Regex do
   def inspect(regex = %{opts: regex_opts}, opts) when is_list(regex_opts) do
-    concat([
-      "Regex.compile!(",
-      Inspect.BitString.inspect(regex.source, opts),
-      ", ",
-      Inspect.List.inspect(regex_opts, opts),
-      ")"
-    ])
+    case translate_options(regex_opts, []) do
+      :error ->
+        concat([
+          "Regex.compile!(",
+          Inspect.BitString.inspect(regex.source, opts),
+          ", ",
+          Inspect.List.inspect(regex_opts, opts),
+          ")"
+        ])
+
+      translated_opts ->
+        {escaped, _} =
+          regex.source
+          |> normalize(<<>>)
+          |> Identifier.escape(?/, :infinity, &escape_map/1)
+
+        source = IO.iodata_to_binary([?~, ?r, ?/, escaped, ?/, translated_opts])
+        color(source, :regex, opts)
+    end
   end
 
-  def inspect(regex, opts) do
-    {escaped, _} =
-      regex.source
-      |> normalize(<<>>)
-      |> Identifier.escape(?/, :infinity, &escape_map/1)
+  defp translate_options([:dotall, {:newline, :anycrlf} | t], acc),
+    do: translate_options(t, [?s | acc])
 
-    source = IO.iodata_to_binary([?~, ?r, ?/, escaped, ?/, regex.opts])
-    color(source, :regex, opts)
-  end
+  defp translate_options([:unicode, :ucp | t], acc), do: translate_options(t, [?u | acc])
+  defp translate_options([:caseless | t], acc), do: translate_options(t, [?i | acc])
+  defp translate_options([:extended | t], acc), do: translate_options(t, [?x | acc])
+  defp translate_options([:firstline | t], acc), do: translate_options(t, [?f | acc])
+  defp translate_options([:ungreedy | t], acc), do: translate_options(t, [?U | acc])
+  defp translate_options([:multiline | t], acc), do: translate_options(t, [?m | acc])
+  defp translate_options([], acc), do: acc
+  defp translate_options(_t, _acc), do: :error
 
   defp normalize(<<?\\, ?\\, rest::binary>>, acc), do: normalize(rest, <<acc::binary, ?\\, ?\\>>)
   defp normalize(<<?\\, ?/, rest::binary>>, acc), do: normalize(rest, <<acc::binary, ?/>>)
@@ -449,7 +463,7 @@ defimpl Inspect, for: Function do
 
       match?(@elixir_compiler ++ _, Atom.to_charlist(mod)) ->
         if function_exported?(mod, :__RELATIVE__, 0) do
-          "#Function<#{uniq(fun_info)} in file:#{mod.__RELATIVE__}>"
+          "#Function<#{uniq(fun_info)} in file:#{mod.__RELATIVE__()}>"
         else
           default_inspect(mod, fun_info)
         end
@@ -535,7 +549,8 @@ end
 
 defimpl Inspect, for: Any do
   defmacro __deriving__(module, struct, options) do
-    fields = Map.keys(struct) -- [:__exception__, :__struct__]
+    fields = Enum.sort(Map.keys(struct) -- [:__exception__, :__struct__])
+
     only = Keyword.get(options, :only, fields)
     except = Keyword.get(options, :except, [])
     optional = Keyword.get(options, :optional, [])
@@ -545,7 +560,7 @@ defimpl Inspect, for: Any do
     :ok = validate_option(:optional, optional, fields, module)
 
     inspect_module =
-      if fields == only and except == [] do
+      if fields == Enum.sort(only) and except == [] do
         Inspect.Map
       else
         Inspect.Any

lib/elixir/lib/inspect/algebra.ex

@@ -143,9 +143,8 @@ defmodule Inspect.Opts do
   function as this must be controlled by applications. Libraries
   should instead define their own structs with custom inspect
   implementations. If a library must change the default inspect
-  function, then it is best to define to ask users of your library
-  to explicitly call `default_inspect_fun/1` with your function of
-  choice.
+  function, then it is best to ask users of your library to explicitly
+  call `default_inspect_fun/1` with your function of choice.
 
   The default is `Inspect.inspect/2`.
 

lib/elixir/lib/integer.ex

@@ -161,7 +161,7 @@ defmodule Integer do
   @doc since: "1.4.0"
   @spec floor_div(integer, neg_integer | pos_integer) :: integer
   def floor_div(dividend, divisor) do
-    if dividend * divisor < 0 and rem(dividend, divisor) != 0 do
+    if :erlang.xor(dividend < 0, divisor < 0) and rem(dividend, divisor) != 0 do
       div(dividend, divisor) - 1
     else
       div(dividend, divisor)
@@ -362,25 +362,25 @@ defmodule Integer do
   ## Examples
 
       iex> Integer.to_charlist(123)
-      '123'
+      ~c"123"
 
       iex> Integer.to_charlist(+456)
-      '456'
+      ~c"456"
 
       iex> Integer.to_charlist(-789)
-      '-789'
+      ~c"-789"
 
       iex> Integer.to_charlist(0123)
-      '123'
+      ~c"123"
 
       iex> Integer.to_charlist(100, 16)
-      '64'
+      ~c"64"
 
       iex> Integer.to_charlist(-100, 16)
-      '-64'
+      ~c"-64"
 
       iex> Integer.to_charlist(882_681_651, 36)
-      'ELIXIR'
+      ~c"ELIXIR"
 
   """
   @spec to_charlist(integer, 2..36) :: charlist

lib/elixir/lib/io.ex

@@ -128,8 +128,15 @@ defmodule IO do
   @doc """
   Reads from the IO `device`.
 
-  The `device` is iterated by the given number of characters, line by line if
-  `:line` is given, or until `:eof`.
+  The `device` is iterated as specified by the `line_or_chars` argument:
+
+    * if `line_or_chars` is an integer, it represents a number of bytes. The device is
+      iterated by that number of bytes.
+
+    * if `line_or_chars` is `:line`, the device is iterated line by line.
+
+    * if `line_or_chars` is `:eof` (since v1.13), the device is iterated until `:eof`.
+      If the device is already at the end, it returns `:eof` itself.
 
   It returns:
 
@@ -145,8 +152,10 @@ defmodule IO do
   @spec read(device, :eof | :line | non_neg_integer) :: chardata | nodata
   def read(device \\ :stdio, line_or_chars)
 
-  # TODO: Deprecate me on v1.17
+  # TODO: Remove me on v2.0
   def read(device, :all) do
+    IO.warn("IO.read(device, :all) is deprecated, use IO.read(device, :eof) instead")
+
     with :eof <- read(device, :eof) do
       with [_ | _] = opts <- :io.getopts(device),
            false <- Keyword.get(opts, :binary, true) do
@@ -179,10 +188,8 @@ defmodule IO do
 
     * if `line_or_chars` is `:line`, the device is iterated line by line.
 
-    * if `line_or_chars` is `:eof`, the device is iterated until `:eof`. `line_or_chars`
-      can only be `:eof` since Elixir 1.13.0. `:eof` replaces the deprecated `:all`,
-      with the difference that `:all` returns `""` on end of file, while `:eof` returns
-      `:eof` itself.
+    * if `line_or_chars` is `:eof` (since v1.13), the device is iterated until `:eof`.
+      If the device is already at the end, it returns `:eof` itself.
 
   It returns:
 
@@ -200,8 +207,9 @@ defmodule IO do
   @spec binread(device, :eof | :line | non_neg_integer) :: iodata | nodata
   def binread(device \\ :stdio, line_or_chars)
 
-  # TODO: Deprecate me on v1.17
+  # TODO: Remove me on v2.0
   def binread(device, :all) do
+    IO.warn("IO.binread(device, :all) is deprecated, use IO.binread(device, :eof) instead")
     with :eof <- binread(device, :eof), do: ""
   end
 
@@ -308,17 +316,20 @@ defmodule IO do
       entry from the compilation environment will be used
 
     * a keyword list with at least the `:file` option representing
-      a single stacktrace entry (since v1.14.0). The `:line`, `:module`,
-      `:function` options are also supported
+      a single stacktrace entry (since v1.14.0). The `:line`, `:column`,
+      `:module`, and `:function` options are also supported
 
-  This function also notifies the compiler a warning was printed
-  (in case --warnings-as-errors was enabled). It returns `:ok`
-  if it succeeds.
+  This function notifies the compiler a warning was printed
+  and emits a compiler diagnostic (`t:Code.diagnostic/1`).
+  The diagnostic will include precise file and location information
+  if a `Macro.Env` is given or those values have been passed as
+  keyword list, but not for stacktraces, as they are often imprecise.
+
+  It returns `:ok` if it succeeds.
 
   ## Examples
 
-      stacktrace = [{MyApp, :main, 1, [file: 'my_app.ex', line: 4]}]
-      IO.warn("variable bar is unused", stacktrace)
+      IO.warn("variable bar is unused", module: MyApp, function: {:main, 1}, line: 4, file: "my_app.ex")
       #=> warning: variable bar is unused
       #=>   my_app.ex:4: MyApp.main/1
 
@@ -337,15 +348,22 @@ defmodule IO do
 
   def warn(message, [{_, _} | _] = keyword) do
     if file = keyword[:file] do
-      warn(
-        message,
-        %{
+      line = keyword[:line]
+      column = keyword[:column]
+      position = if line && column, do: {line, column}, else: line
+      message = to_chardata(message)
+
+      stacktrace =
+        Macro.Env.stacktrace(%{
           __ENV__
           | module: keyword[:module],
             function: keyword[:function],
-            line: keyword[:line],
+            line: line,
             file: file
-        }
+        })
+
+      :elixir_errors.emit_diagnostic(:warning, position, file, message, stacktrace,
+        read_snippet: true
       )
     else
       warn(message, [])
@@ -368,7 +386,7 @@ defmodule IO do
     stacktrace = Enum.drop(stacktrace, stacktrace_drop_levels)
 
     if :elixir_config.warn(key, stacktrace) do
-      warn(message, stacktrace)
+      warn(message.(), stacktrace)
     else
       :ok
     end
@@ -398,7 +416,7 @@ defmodule IO do
   end
 
   @doc """
-  Inspects and writes the given `item` to the device.
+  Inspects and writes the given `item` to the standard output.
 
   It's important to note that it returns the given `item` unchanged.
   This makes it possible to "spy" on values by inserting an
@@ -414,6 +432,7 @@ defmodule IO do
   The label will be printed before the inspected `item`.
 
   See `Inspect.Opts` for a full list of remaining formatting options.
+  To print to other IO devices, see `IO.inspect/3`
 
   ## Examples
 

lib/elixir/lib/io/ansi.ex

@@ -268,7 +268,7 @@ defmodule IO.ANSI do
   The named sequences are represented by atoms.
 
   It will also append an `IO.ANSI.reset/0` to the chardata when a conversion is
-  performed. If you don't want this behaviour, use `format_fragment/2`.
+  performed. If you don't want this behavior, use `format_fragment/2`.
 
   An optional boolean parameter can be passed to enable or disable
   emitting actual ANSI codes. When `false`, no ANSI codes will be emitted.
@@ -299,7 +299,7 @@ defmodule IO.ANSI do
 
   ## Examples
 
-      iex> IO.ANSI.format_fragment([:bright, 'Word'], true)
+      iex> IO.ANSI.format_fragment([:bright, ~c"Word"], true)
       [[[[[[] | "\e[1m"], 87], 111], 114], 100]
 
   """

lib/elixir/lib/io/ansi/docs.ex

@@ -176,12 +176,17 @@ defmodule IO.ANSI.Docs do
     process_code(rest, [line], indent, options)
   end
 
-  defp process(["```" <> _line | rest], text, indent, options) do
-    process_fenced_code_block(rest, text, indent, options, _delimiter = "```")
+  defp process(["```mermaid" <> _line | rest], text, indent, options) do
+    write_text(text, indent, options)
+
+    rest
+    |> Enum.drop_while(&(&1 != "```"))
+    |> Enum.drop(1)
+    |> process([], indent, options)
   end
 
-  defp process(["~~~" <> _line | rest], text, indent, options) do
-    process_fenced_code_block(rest, text, indent, options, _delimiter = "~~~")
+  defp process(["```" <> _line | rest], text, indent, options) do
+    process_fenced_code_block(rest, text, indent, options, _delimiter = "```")
   end
 
   defp process(["<!--" <> line | rest], text, indent, options) do
@@ -216,10 +221,6 @@ defmodule IO.ANSI.Docs do
 
   ### Quotes
 
-  defp process_quote([], lines, indent, options) do
-    write_quote(lines, indent, options, false)
-  end
-
   defp process_quote([">", ">" <> line | rest], lines, indent, options) do
     write_quote(lines, indent, options, true)
     write_empty_quote_line(options)
@@ -360,10 +361,6 @@ defmodule IO.ANSI.Docs do
 
   ### Code blocks
 
-  defp process_code([], code, indent, options) do
-    write_code(code, indent, options)
-  end
-
   # Blank line between code blocks
   defp process_code(["", "    " <> line | rest], code, indent, options) do
     process_code(rest, [line, "" | code], indent, options)
@@ -388,7 +385,7 @@ defmodule IO.ANSI.Docs do
   end
 
   defp process_fenced_code([line | rest], code, indent, options, delimiter) do
-    if line === delimiter do
+    if line == delimiter do
       process_code(rest, code, indent, options)
     else
       process_fenced_code(rest, [line | code], indent, options, delimiter)

lib/elixir/lib/io/stream.ex

@@ -1,11 +1,9 @@
 defmodule IO.StreamError do
-  defexception [:reason, :message]
+  defexception [:reason]
 
   @impl true
-  def exception(opts) do
-    reason = opts[:reason]
-    formatted = IO.iodata_to_binary(:file.format_error(reason))
-    %IO.StreamError{message: "error during streaming: #{formatted}", reason: reason}
+  def message(%{reason: reason}) do
+    "error during streaming: #{inspect(reason)}"
   end
 end
 

lib/elixir/lib/kernel/cli.ex

@@ -327,6 +327,8 @@ defmodule Kernel.CLI do
     parse_argv(t, %{config | profile: :time})
   end
 
+  ## IEx
+
   defp parse_argv([~c"--dbg", backend | t], %{mode: :iex} = config) do
     case backend do
       ~c"pry" ->
@@ -344,6 +346,21 @@ defmodule Kernel.CLI do
   defp parse_argv([~c"--dot-iex", _ | t], %{mode: :iex} = config), do: parse_argv(t, config)
   defp parse_argv([~c"--remsh", _ | t], %{mode: :iex} = config), do: parse_argv(t, config)
 
+  ## Erlang flags
+
+  defp parse_argv([~c"--boot", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--boot-var", _, _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--cookie", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--hidden" | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--erl-config", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--logger-otp-reports", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--logger-sasl-reports", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--name", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--sname", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--vm-args", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--erl", _ | t], config), do: parse_argv(t, config)
+  defp parse_argv([~c"--pipe-to", _, _ | t], config), do: parse_argv(t, config)
+
   ## Fallback
 
   defp parse_argv([h | t], %{mode: :elixirc} = config) do
@@ -462,8 +479,7 @@ defmodule Kernel.CLI do
 
   defp process_command({:compile, patterns}, config) do
     # If ensuring the dir returns an error no files will be found.
-    # TODO: Use :filelib.ensure_path/1 once we require Erlang/OTP 25+
-    _ = :filelib.ensure_dir(:filename.join(config.output, "."))
+    _ = :filelib.ensure_path(config.output)
 
     case filter_multiple_patterns(patterns) do
       {:ok, []} ->

lib/elixir/lib/kernel/parallel_compiler.ex

@@ -18,23 +18,36 @@ defmodule Kernel.ParallelCompiler do
   # TODO: Deprecate this on Elixir v1.20.
   @doc deprecated: "Use `pmap/2` instead"
   def async(fun) when is_function(fun, 0) do
+    {ref, task} = inner_async(fun)
+    send(task.pid, ref)
+    task
+  end
+
+  defp inner_async(fun) do
     case :erlang.get(:elixir_compiler_info) do
       {compiler_pid, file_pid} ->
+        ref = make_ref()
         file = :erlang.get(:elixir_compiler_file)
         dest = :erlang.get(:elixir_compiler_dest)
 
         {:error_handler, error_handler} = :erlang.process_info(self(), :error_handler)
         {_parent, checker} = Module.ParallelChecker.get()
 
-        Task.async(fn ->
-          send(compiler_pid, {:async, self()})
-          Module.ParallelChecker.put(compiler_pid, checker)
-          :erlang.put(:elixir_compiler_info, {compiler_pid, file_pid})
-          :erlang.put(:elixir_compiler_file, file)
-          dest != :undefined and :erlang.put(:elixir_compiler_dest, dest)
-          :erlang.process_flag(:error_handler, error_handler)
-          fun.()
-        end)
+        task =
+          Task.async(fn ->
+            Module.ParallelChecker.put(compiler_pid, checker)
+            :erlang.put(:elixir_compiler_info, {compiler_pid, file_pid})
+            :erlang.put(:elixir_compiler_file, file)
+            dest != :undefined and :erlang.put(:elixir_compiler_dest, dest)
+            :erlang.process_flag(:error_handler, error_handler)
+
+            receive do
+              ^ref -> fun.()
+            end
+          end)
+
+        send(compiler_pid, {:async, task.pid})
+        {ref, task}
 
       :undefined ->
         raise ArgumentError,
@@ -52,41 +65,29 @@ defmodule Kernel.ParallelCompiler do
   """
   @doc since: "1.16.0"
   def pmap(collection, fun) when is_function(fun, 1) do
-    parent = self()
     ref = make_ref()
 
     # We spawn a series of tasks for parallel processing.
-    # The tasks notify themselves to the compiler.
-    tasks =
+    # The tasks are waiting until we give the go ahead.
+    refs_tasks =
       Enum.map(collection, fn item ->
-        async(fn ->
-          send(parent, {ref, self()})
-
-          receive do
-            ^ref -> fun.(item)
-          end
-        end)
+        inner_async(fn -> fun.(item) end)
       end)
 
-    # Then the tasks notify us. This is important because if
-    # we wait before the tasks notify the compiler, we may be
-    # released as there is nothing else running.
-    on =
-      for %{pid: pid} <- tasks do
-        receive do
-          {^ref, ^pid} -> pid
-        end
-      end
-
     # Notify the compiler we are waiting on the tasks.
     {compiler_pid, file_pid} = :erlang.get(:elixir_compiler_info)
     defining = :elixir_module.compiler_modules()
+    on = Enum.map(refs_tasks, fn {_ref, %{pid: pid}} -> pid end)
     send(compiler_pid, {:waiting, :pmap, self(), ref, file_pid, on, defining, :raise})
 
     # Now we allow the tasks to run. This step is not strictly
     # necessary but it makes compilation more deterministic by
     # only allowing tasks to run once we are waiting.
-    Enum.each(on, &send(&1, ref))
+    tasks =
+      Enum.map(refs_tasks, fn {ref, task} ->
+        send(task.pid, ref)
+        task
+      end)
 
     # Await tasks and notify the compiler they are done. We could
     # have the tasks report directly to the compiler, which in turn
@@ -455,16 +456,22 @@ defmodule Kernel.ParallelCompiler do
   # No more queue, nothing waiting, this cycle is done
   defp spawn_workers([], spawned, waiting, files, result, warnings, errors, state)
        when map_size(spawned) == 0 and map_size(waiting) == 0 do
-    [] = errors
+    # Print any spurious error that we may have found
+    Enum.map(errors, fn {diagnostic, read_snippet} ->
+      :elixir_errors.print_diagnostic(diagnostic, read_snippet)
+    end)
+
     [] = files
     cycle_return = each_cycle_return(state.each_cycle.())
     state = cycle_timing(result, state)
 
     case cycle_return do
       {:runtime, dependent_modules, extra_warnings} ->
+        :elixir_code_server.cast(:purge_compiler_modules)
         verify_modules(result, extra_warnings ++ warnings, dependent_modules, state)
 
       {:compile, [], extra_warnings} ->
+        :elixir_code_server.cast(:purge_compiler_modules)
         verify_modules(result, extra_warnings ++ warnings, [], state)
 
       {:compile, more, extra_warnings} ->
@@ -510,8 +517,9 @@ defmodule Kernel.ParallelCompiler do
     if deadlocked do
       spawn_workers(deadlocked, spawned, waiting, files, result, warnings, errors, state)
     else
-      deadlock_errors = handle_deadlock(waiting, files)
-      {return_error(deadlock_errors ++ errors, warnings), state}
+      return_error(warnings, errors, state, fn ->
+        handle_deadlock(waiting, files)
+      end)
     end
   end
 
@@ -681,12 +689,13 @@ defmodule Kernel.ParallelCompiler do
         state = %{state | timer_ref: timer_ref}
         spawn_workers(queue, spawned, waiting, files, result, warnings, errors, state)
 
-      {:diagnostic, %{severity: :warning} = diagnostic} ->
-        warnings = [Module.ParallelChecker.format_diagnostic_file(diagnostic) | warnings]
+      {:diagnostic, %{severity: :warning, file: file} = diagnostic, read_snippet} ->
+        :elixir_errors.print_diagnostic(diagnostic, read_snippet)
+        warnings = [%{diagnostic | file: file && Path.absname(file)} | warnings]
         wait_for_messages(queue, spawned, waiting, files, result, warnings, errors, state)
 
-      {:diagnostic, %{severity: :error} = diagnostic} ->
-        errors = [Module.ParallelChecker.format_diagnostic_file(diagnostic) | errors]
+      {:diagnostic, %{severity: :error} = diagnostic, read_snippet} ->
+        errors = [{diagnostic, read_snippet} | errors]
         wait_for_messages(queue, spawned, waiting, files, result, warnings, errors, state)
 
       {:file_ok, child_pid, ref, file, lexical} ->
@@ -706,10 +715,13 @@ defmodule Kernel.ParallelCompiler do
         spawn_workers(queue, new_spawned, waiting, new_files, result, warnings, errors, state)
 
       {:file_error, child_pid, file, {kind, reason, stack}} ->
-        print_error(file, kind, reason, stack)
         {_file, _new_spawned, new_files} = discard_file_pid(spawned, files, child_pid)
         terminate(new_files)
-        {return_error([to_error(file, kind, reason, stack) | errors], warnings), state}
+
+        return_error(warnings, errors, state, fn ->
+          print_error(file, kind, reason, stack)
+          [to_error(file, kind, reason, stack)]
+        end)
 
       {:DOWN, ref, :process, pid, reason} when is_map_key(spawned, ref) ->
         # async spawned processes have no file, so we always have to delete the ref directly
@@ -718,18 +730,30 @@ defmodule Kernel.ParallelCompiler do
         {file, spawned, files} = discard_file_pid(spawned, files, pid)
 
         if file do
-          print_error(file.file, :exit, reason, [])
           terminate(files)
-          {return_error([to_error(file.file, :exit, reason, []) | errors], warnings), state}
+
+          return_error(warnings, errors, state, fn ->
+            print_error(file.file, :exit, reason, [])
+            [to_error(file.file, :exit, reason, [])]
+          end)
         else
           wait_for_messages(queue, spawned, waiting, files, result, warnings, errors, state)
         end
     end
   end
 
-  defp return_error(errors, warnings) do
+  defp return_error(warnings, errors, state, fun) do
+    # Also prune compiler modules in case of errors
+    :elixir_code_server.cast(:purge_compiler_modules)
+
+    errors =
+      Enum.map(errors, fn {%{file: file} = diagnostic, read_snippet} ->
+        :elixir_errors.print_diagnostic(diagnostic, read_snippet)
+        %{diagnostic | file: file && Path.absname(file)}
+      end)
+
     info = %{compile_warnings: Enum.reverse(warnings), runtime_warnings: []}
-    {:error, Enum.reverse(errors), info}
+    {{:error, Enum.reverse(errors, fun.()), info}, state}
   end
 
   defp update_result(result, kind, module, value) do
@@ -859,9 +883,12 @@ defmodule Kernel.ParallelCompiler do
     )
 
     for {file, _, description, stacktrace} <- deadlock do
+      file = Path.absname(file)
+
       %{
         severity: :error,
-        file: Path.absname(file),
+        file: file,
+        source: file,
         position: nil,
         message: description,
         stacktrace: stacktrace,
@@ -889,50 +916,64 @@ defmodule Kernel.ParallelCompiler do
     ])
   end
 
-  defp to_error(file, kind, reason, stack) do
-    line = get_line(file, reason, stack)
-    file = Path.absname(file)
+  defp to_error(source, kind, reason, stack) do
+    {file, line, span} = get_snippet_info(source, reason, stack)
+    source = Path.absname(source)
     message = :unicode.characters_to_binary(Kernel.CLI.format_error(kind, reason, stack))
 
     %{
-      file: file,
+      file: file || source,
+      source: source,
       position: line || 0,
       message: message,
       severity: :error,
       stacktrace: stack,
-      span: nil
+      span: span,
+      details: {kind, reason}
     }
   end
 
-  defp get_line(_file, %{line: line, column: column}, _stack)
+  defp get_snippet_info(
+         _file,
+         %{file: file, line: line, column: column, end_line: end_line, end_column: end_column},
+         _stack
+       )
+       when is_integer(line) and line > 0 and is_integer(column) and column >= 0 and
+              is_integer(end_line) and end_line > 0 and is_integer(end_column) and end_column >= 0 do
+    {Path.absname(file), {line, column}, {end_line, end_column}}
+  end
+
+  defp get_snippet_info(_file, %{file: file, line: line, column: column}, _stack)
        when is_integer(line) and line > 0 and is_integer(column) and column >= 0 do
-    {line, column}
+    {Path.absname(file), {line, column}, nil}
   end
 
-  defp get_line(_file, %{line: line}, _stack) when is_integer(line) and line > 0 do
-    line
+  defp get_snippet_info(_file, %{line: line}, _stack) when is_integer(line) and line > 0 do
+    {nil, line, nil}
   end
 
-  defp get_line(file, :undef, [{_, _, _, []}, {_, _, _, info} | _]) do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+  defp get_snippet_info(file, :undef, [{_, _, _, []}, {_, _, _, info} | _]) do
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(file, _reason, [{_, _, _, [file: expanding]}, {_, _, _, info} | _])
+  defp get_snippet_info(file, _reason, [{_, _, _, [file: expanding]}, {_, _, _, info} | _])
        when expanding in [~c"expanding macro", ~c"expanding struct"] do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(file, _reason, [{_, _, _, info} | _]) do
-    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
-      Keyword.get(info, :line)
-    end
+  defp get_snippet_info(file, _reason, [{_, _, _, info} | _]) do
+    get_snippet_info_from_stacktrace_info(info, file)
   end
 
-  defp get_line(_, _, _) do
-    nil
+  defp get_snippet_info(_, _, _) do
+    {nil, nil, nil}
+  end
+
+  defp get_snippet_info_from_stacktrace_info(info, file) do
+    if Keyword.get(info, :file) == to_charlist(Path.relative_to_cwd(file)) do
+      {nil, Keyword.get(info, :line), nil}
+    else
+      {nil, nil, nil}
+    end
   end
 end

lib/elixir/lib/kernel/special_forms.ex

@@ -550,7 +550,7 @@ defmodule Kernel.SpecialForms do
   Elixir won't emit any warnings though, since the alias
   was not explicitly defined.
 
-  Both warning behaviours could be changed by explicitly
+  Both warning behaviors could be changed by explicitly
   setting the `:warn` option to `true` or `false`.
 
   """
@@ -674,7 +674,7 @@ defmodule Kernel.SpecialForms do
   Elixir won't emit any warnings though, since the import
   was not explicitly defined.
 
-  Both warning behaviours could be changed by explicitly
+  Both warning behaviors could be changed by explicitly
   setting the `:warn` option to `true` or `false`.
 
   ## Ambiguous function/macro names
@@ -888,7 +888,7 @@ defmodule Kernel.SpecialForms do
   At first, there is nothing in this example that actually reveals it is a
   macro. But what is happening is that, at compilation time, `squared(5)`
   becomes `5 * 5`. The argument `5` is duplicated in the produced code, we
-  can see this behaviour in practice though because our macro actually has
+  can see this behavior in practice though because our macro actually has
   a bug:
 
       import Math
@@ -915,7 +915,7 @@ defmodule Kernel.SpecialForms do
       my_number.() * my_number.()
 
   Which invokes the function twice, explaining why we get the printed value
-  twice! In the majority of the cases, this is actually unexpected behaviour,
+  twice! In the majority of the cases, this is actually unexpected behavior,
   and that's why one of the first things you need to keep in mind when it
   comes to macros is to **not unquote the same value more than once**.
 
@@ -1444,7 +1444,7 @@ defmodule Kernel.SpecialForms do
       [elixir: :prolog]
 
   Given the grandparents of Erlang and Prolog were nil, those values were
-  filtered out. If you don't want this behaviour, a simple option is to
+  filtered out. If you don't want this behavior, a simple option is to
   move the filter inside the do-block:
 
       iex> languages = [elixir: :erlang, erlang: :prolog, prolog: nil]
@@ -1494,7 +1494,9 @@ defmodule Kernel.SpecialForms do
 
   ## The `:reduce` option
 
-  While the `:into` option allows us to customize the comprehension behaviour
+  *Available since Elixir v1.8*.
+
+  While the `:into` option allows us to customize the comprehension behavior
   to a given data type, such as putting all of the values inside a map or inside
   a binary, it is not always enough.
 
@@ -1622,7 +1624,7 @@ defmodule Kernel.SpecialForms do
       iex> width
       nil
 
-  The behaviour of any expression in a clause is the same as if it was
+  The behavior of any expression in a clause is the same as if it was
   written outside of `with`. For example, `=` will raise a `MatchError`
   instead of returning the non-matched value:
 
@@ -1754,6 +1756,15 @@ defmodule Kernel.SpecialForms do
   """
   defmacro unquote(:__block__)(args), do: error!([args])
 
+  @doc """
+  Internal special form for cursor position.
+
+  This is the special form used whenever we need to represent
+  the cursor position in Elixir's AST. See `Code.Fragment` for
+  more information.
+  """
+  defmacro unquote(:__cursor__)(args), do: error!([args])
+
   @doc """
   Capture operator. Captures or creates an anonymous function.
 
@@ -1876,6 +1887,11 @@ defmodule Kernel.SpecialForms do
   @doc ~S"""
   Matches the given expression against the given clauses.
 
+  `case/2` relies on pattern matching and guards to choose
+  which clause to execute. If your logic cannot be expressed
+  within patterns and guards, consider using `if/2` or `cond/1`
+  instead.
+
   ## Examples
 
       case File.read(file) do
@@ -1906,6 +1922,9 @@ defmodule Kernel.SpecialForms do
       end
       #=> "This clause would match any value (x = 10)"
 
+  If you find yourself nesting `case` expressions inside
+  `case` expressions, consider using `with/1`.
+
   ## Variable handling
 
   Note that variables bound in a clause do not leak to the outer context:
@@ -1965,17 +1984,20 @@ defmodule Kernel.SpecialForms do
   Evaluates the expression corresponding to the first clause that
   evaluates to a truthy value.
 
+  ## Examples
+
+  The following example has a single clause that always evaluates
+  to true:
+
       cond do
         hd([1, 2, 3]) ->
           "1 is considered as true"
       end
       #=> "1 is considered as true"
 
-  Raises an error if all conditions evaluate to `nil` or `false`.
+  If all clauses evaluate to `nil` or `false`, `cond` raises an error.
   For this reason, it may be necessary to add a final always-truthy condition
-  (anything non-`false` and non-`nil`), which will always match.
-
-  ## Examples
+  (anything non-`false` and non-`nil`), which will always match:
 
       cond do
         1 + 1 == 1 ->
@@ -1987,6 +2009,9 @@ defmodule Kernel.SpecialForms do
       end
       #=> "This will"
 
+
+  If your `cond` has two clauses, and the last one falls back to
+  `true`, you may consider using `if/2` instead.
   """
   defmacro cond(clauses), do: error!([clauses])
 
@@ -2300,11 +2325,13 @@ defmodule Kernel.SpecialForms do
   defmacro try(args), do: error!([args])
 
   @doc """
-  Checks if there is a message matching the given clauses
-  in the current process mailbox.
+  Checks if there is a message matching any of the given clauses in the current
+  process mailbox.
 
-  In case there is no such message, the current process hangs
-  until a message arrives or waits until a given timeout value.
+  If there is no matching message, the current process waits until a matching
+  message arrives or until after a given timeout value.
+
+  Any new and existing messages that do not match will remain in the mailbox.
 
   ## Examples
 
@@ -2317,8 +2344,8 @@ defmodule Kernel.SpecialForms do
           IO.puts(:stderr, "Unexpected message received")
       end
 
-  An optional `after` clause can be given in case the message was not
-  received after the given timeout period, specified in milliseconds:
+  An optional `after` clause can be given in case no matching message is
+  received during the given timeout period, specified in milliseconds:
 
       receive do
         {:selector, number, name} when is_integer(number) ->

lib/elixir/lib/kernel/typespec.ex

@@ -385,17 +385,17 @@ defmodule Kernel.Typespec do
       compile_error(caller, error)
     end
 
-    line = line(meta)
+    location = location(meta)
     vars = Keyword.keys(guard)
 
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
     {return, state} = typespec(return, vars, caller, state)
-    spec = {:type, line, :fun, [{:type, line, :product, args}, return]}
+    spec = {:type, location, :fun, [{:type, location, :product, args}, return]}
 
     {spec, state} =
       case guard_to_constraints(guard, vars, meta, caller, state) do
         {[], state} -> {spec, state}
-        {constraints, state} -> {{:type, line, :bounded_fun, [spec, constraints]}, state}
+        {constraints, state} -> {{:type, location, :bounded_fun, [spec, constraints]}, state}
       end
 
     ensure_no_unused_local_vars!(caller, state.local_vars)
@@ -437,7 +437,7 @@ defmodule Kernel.Typespec do
   defp ensure_not_default(_), do: :ok
 
   defp guard_to_constraints(guard, vars, meta, caller, state) do
-    line = line(meta)
+    location = location(meta)
 
     fun = fn
       {_name, {:var, _, context}}, {constraints, state} when is_atom(context) ->
@@ -445,9 +445,9 @@ defmodule Kernel.Typespec do
 
       {name, type}, {constraints, state} ->
         {spec, state} = typespec(type, vars, caller, state)
-        constraint = [{:atom, line, :is_subtype}, [{:var, line, name}, spec]]
+        constraint = [{:atom, location, :is_subtype}, [{:var, location, name}, spec]]
         state = update_local_vars(state, name)
-        {[{:type, line, :constraint, constraint} | constraints], state}
+        {[{:type, location, :constraint, constraint} | constraints], state}
     end
 
     {constraints, state} = :lists.foldl(fun, {[], state}, guard)
@@ -456,21 +456,27 @@ defmodule Kernel.Typespec do
 
   ## To typespec conversion
 
-  defp line(meta) do
-    Keyword.get(meta, :line, 0)
+  defp location(meta) do
+    line = Keyword.get(meta, :line, 0)
+
+    if column = Keyword.get(meta, :column) do
+      {line, column}
+    else
+      line
+    end
   end
 
   # Handle unions
   defp typespec({:|, meta, [_, _]} = exprs, vars, caller, state) do
     exprs = collect_union(exprs)
     {union, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, exprs)
-    {{:type, line(meta), :union, union}, state}
+    {{:type, location(meta), :union, union}, state}
   end
 
   # Handle binaries
   defp typespec({:<<>>, meta, []}, _, _, state) do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line, 0}, {:integer, line, 0}]}, state}
+    location = location(meta)
+    {{:type, location, :binary, [{:integer, location, 0}, {:integer, location, 0}]}, state}
   end
 
   defp typespec(
@@ -480,14 +486,18 @@ defmodule Kernel.Typespec do
          state
        )
        when is_atom(ctx1) and is_atom(ctx2) and unit in 1..256 do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line, 0}, {:integer, line(unit_meta), unit}]}, state}
+    location = location(meta)
+
+    {{:type, location, :binary, [{:integer, location, 0}, {:integer, location(unit_meta), unit}]},
+     state}
   end
 
   defp typespec({:<<>>, meta, [{:"::", size_meta, [{:_, _, ctx}, size]}]}, _, _, state)
        when is_atom(ctx) and is_integer(size) and size >= 0 do
-    line = line(meta)
-    {{:type, line, :binary, [{:integer, line(size_meta), size}, {:integer, line, 0}]}, state}
+    location = location(meta)
+
+    {{:type, location, :binary, [{:integer, location(size_meta), size}, {:integer, location, 0}]},
+     state}
   end
 
   defp typespec(
@@ -505,8 +515,8 @@ defmodule Kernel.Typespec do
        )
        when is_atom(ctx1) and is_atom(ctx2) and is_atom(ctx3) and is_integer(size) and
               size >= 0 and unit in 1..256 do
-    args = [{:integer, line(size_meta), size}, {:integer, line(unit_meta), unit}]
-    {{:type, line(meta), :binary, args}, state}
+    args = [{:integer, location(size_meta), size}, {:integer, location(unit_meta), unit}]
+    {{:type, location(meta), :binary, args}, state}
   end
 
   defp typespec({:<<>>, _meta, _args}, _vars, caller, _state) do
@@ -519,7 +529,7 @@ defmodule Kernel.Typespec do
 
   ## Handle maps and structs
   defp typespec({:map, meta, args}, _vars, _caller, state) when args == [] or is_atom(args) do
-    {{:type, line(meta), :map, :any}, state}
+    {{:type, location(meta), :map, :any}, state}
   end
 
   defp typespec({:%{}, meta, fields} = map, vars, caller, state) do
@@ -527,17 +537,17 @@ defmodule Kernel.Typespec do
       {{:required, meta2, [k]}, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta2), :map_field_exact, [arg1, arg2]}, state}
+        {{:type, location(meta2), :map_field_exact, [arg1, arg2]}, state}
 
       {{:optional, meta2, [k]}, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta2), :map_field_assoc, [arg1, arg2]}, state}
+        {{:type, location(meta2), :map_field_assoc, [arg1, arg2]}, state}
 
       {k, v}, state ->
         {arg1, state} = typespec(k, vars, caller, state)
         {arg2, state} = typespec(v, vars, caller, state)
-        {{:type, line(meta), :map_field_exact, [arg1, arg2]}, state}
+        {{:type, location(meta), :map_field_exact, [arg1, arg2]}, state}
 
       {:|, _, [_, _]}, _state ->
         error =
@@ -551,7 +561,7 @@ defmodule Kernel.Typespec do
     end
 
     {fields, state} = :lists.mapfoldl(fun, state, fields)
-    {{:type, line(meta), :map, fields}, state}
+    {{:type, location(meta), :map, fields}, state}
   end
 
   defp typespec({:%, _, [name, {:%{}, meta, fields}]} = node, vars, caller, state) do
@@ -644,7 +654,7 @@ defmodule Kernel.Typespec do
     {right, state} = typespec(right, vars, caller, state)
     :ok = validate_range(left, right, caller)
 
-    {{:type, line(meta), :range, [left, right]}, state}
+    {{:type, location(meta), :range, [left, right]}, state}
   end
 
   # Handle special forms
@@ -668,7 +678,7 @@ defmodule Kernel.Typespec do
         pair -> pair
       end
 
-    {{:type, line(meta), :fun, fun_args}, state}
+    {{:type, location(meta), :fun, fun_args}, state}
   end
 
   # Handle type operator
@@ -691,10 +701,10 @@ defmodule Kernel.Typespec do
         # This may be generating an invalid typespec but we need to generate it
         # to avoid breaking existing code that was valid but only broke Dialyzer
         {right, state} = typespec(expr, vars, caller, state)
-        {{:ann_type, line(meta), [{:var, line(var_meta), var_name}, right]}, state}
+        {{:ann_type, location(meta), [{:var, location(var_meta), var_name}, right]}, state}
 
       {right, state} ->
-        {{:ann_type, line(meta), [{:var, line(var_meta), var_name}, right]}, state}
+        {{:ann_type, location(meta), [{:var, location(var_meta), var_name}, right]}, state}
     end
   end
 
@@ -723,13 +733,13 @@ defmodule Kernel.Typespec do
     {left, state} = typespec(left, vars, caller, state)
     state = %{state | undefined_type_error_enabled?: true}
     {right, state} = typespec(right, vars, caller, state)
-    {{:ann_type, line(meta), [left, right]}, state}
+    {{:ann_type, location(meta), [left, right]}, state}
   end
 
   # Handle unary ops
   defp typespec({op, meta, [integer]}, _, _, state) when op in [:+, :-] and is_integer(integer) do
-    line = line(meta)
-    {{:op, line, op, {:integer, line, integer}}, state}
+    location = location(meta)
+    {{:op, location, op, {:integer, location, integer}}, state}
   end
 
   # Handle remote calls in the form of @module_attribute.type.
@@ -778,12 +788,12 @@ defmodule Kernel.Typespec do
 
   # Handle tuples
   defp typespec({:tuple, meta, []}, _vars, _caller, state) do
-    {{:type, line(meta), :tuple, :any}, state}
+    {{:type, location(meta), :tuple, :any}, state}
   end
 
   defp typespec({:{}, meta, t}, vars, caller, state) when is_list(t) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, t)
-    {{:type, line(meta), :tuple, args}, state}
+    {{:type, location(meta), :tuple, args}, state}
   end
 
   defp typespec({left, right}, vars, caller, state) do
@@ -799,7 +809,7 @@ defmodule Kernel.Typespec do
   defp typespec({name, meta, atom}, vars, caller, state) when is_atom(atom) do
     if :lists.member(name, vars) do
       state = update_local_vars(state, name)
-      {{:var, line(meta), name}, state}
+      {{:var, location(meta), name}, state}
     else
       typespec({name, meta, []}, vars, caller, state)
     end
@@ -814,7 +824,7 @@ defmodule Kernel.Typespec do
 
     IO.warn(warning, caller)
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :string, args}, state}
+    {{:type, location(meta), :string, args}, state}
   end
 
   defp typespec({:nonempty_string, meta, args}, vars, caller, state) do
@@ -825,7 +835,7 @@ defmodule Kernel.Typespec do
 
     IO.warn(warning, caller)
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :nonempty_string, args}, state}
+    {{:type, location(meta), :nonempty_string, args}, state}
   end
 
   defp typespec({type, _meta, []}, vars, caller, state) when type in [:charlist, :char_list] do
@@ -855,7 +865,7 @@ defmodule Kernel.Typespec do
 
   defp typespec({:fun, meta, args}, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :fun, args}, state}
+    {{:type, location(meta), :fun, args}, state}
   end
 
   defp typespec({:..., _meta, _args}, _vars, caller, _state) do
@@ -872,7 +882,7 @@ defmodule Kernel.Typespec do
 
     case :erl_internal.is_type(name, arity) do
       true ->
-        {{:type, line(meta), name, args}, state}
+        {{:type, location(meta), name, args}, state}
 
       false ->
         if state.undefined_type_error_enabled? and
@@ -890,7 +900,7 @@ defmodule Kernel.Typespec do
             %{state | used_type_pairs: [{name, arity} | state.used_type_pairs]}
           end
 
-        {{:user_type, line(meta), name, args}, state}
+        {{:user_type, location(meta), name, args}, state}
     end
   end
 
@@ -907,11 +917,11 @@ defmodule Kernel.Typespec do
     typespec({nil, [], []}, vars, caller, state)
   end
 
-  defp typespec([{:..., _, atom}], vars, caller, state) when is_atom(atom) do
+  defp typespec([{:..., _, _}], vars, caller, state) do
     typespec({:nonempty_list, [], []}, vars, caller, state)
   end
 
-  defp typespec([spec, {:..., _, atom}], vars, caller, state) when is_atom(atom) do
+  defp typespec([spec, {:..., _, _}], vars, caller, state) do
     typespec({:nonempty_list, [], [spec]}, vars, caller, state)
   end
 
@@ -940,17 +950,15 @@ defmodule Kernel.Typespec do
 
   # This is a backport of Macro.expand/2 because we want to expand
   # aliases but we don't them to become compile-time references.
-  defp expand_remote({:__aliases__, _, _} = alias, env) do
-    case :elixir_aliases.expand(alias, env) do
+  defp expand_remote({:__aliases__, meta, list} = alias, env) do
+    case :elixir_aliases.expand_or_concat(meta, list, env, true) do
       receiver when is_atom(receiver) ->
         receiver
 
-      aliases ->
-        aliases = :lists.map(&Macro.expand_once(&1, env), aliases)
-
-        case :lists.all(&is_atom/1, aliases) do
-          true -> :elixir_aliases.concat(aliases)
-          false -> alias
+      [head | tail] ->
+        case Macro.expand_once(head, env) do
+          head when is_atom(head) -> :elixir_aliases.concat([head | tail])
+          _ -> alias
         end
     end
   end
@@ -963,7 +971,7 @@ defmodule Kernel.Typespec do
 
   defp remote_type({remote, meta, name, args}, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:remote_type, line(meta), [remote, name, args]}, state}
+    {{:remote_type, location(meta), [remote, name, args]}, state}
   end
 
   defp collect_union({:|, _, [a, b]}), do: [a | collect_union(b)]
@@ -996,16 +1004,16 @@ defmodule Kernel.Typespec do
   end
 
   defp fn_args(meta, [{:..., _, _}], _vars, _caller, state) do
-    {{:type, line(meta), :any}, state}
+    {{:type, location(meta), :any}, state}
   end
 
   defp fn_args(meta, args, vars, caller, state) do
     {args, state} = :lists.mapfoldl(&typespec(&1, vars, caller, &2), state, args)
-    {{:type, line(meta), :product, args}, state}
+    {{:type, location(meta), :product, args}, state}
   end
 
   defp variable({name, meta, args}) when is_atom(name) and is_atom(args) do
-    {:var, line(meta), name}
+    {:var, location(meta), name}
   end
 
   defp variable(expr), do: expr

lib/elixir/lib/kernel/utils.ex

@@ -36,14 +36,14 @@ defmodule Kernel.Utils do
     if is_list(funs) do
       IO.warn(
         "passing a list to Kernel.defdelegate/2 is deprecated, please define each delegate separately",
-        Macro.Env.stacktrace(env)
+        env
       )
     end
 
     if Keyword.has_key?(opts, :append_first) do
       IO.warn(
         "Kernel.defdelegate/2 :append_first option is deprecated",
-        Macro.Env.stacktrace(env)
+        env
       )
     end
 
@@ -120,7 +120,7 @@ defmodule Kernel.Utils do
     mapper = fn
       {key, val} when is_atom(key) ->
         try do
-          :elixir_quote.escape(val, false, :none)
+          :elixir_quote.escape(val, :none, false)
         rescue
           e in [ArgumentError] ->
             raise ArgumentError, "invalid value for struct field #{key}, " <> Exception.message(e)
@@ -208,10 +208,10 @@ defmodule Kernel.Utils do
 
     case enforce_keys -- :maps.keys(struct) do
       [] ->
-        # The __struct__ field is used for expansion and for loading remote structs
+        # The __struct__ attribute is during expansion and for loading remote structs
         :ets.insert(set, {:__struct__, struct, nil, []})
 
-        # Store all field metadata to go into __info__(:struct)
+        # The complete metadata goes into __info__(:struct)
         mapper = fn {key, val} ->
           %{field: key, default: val, required: :lists.member(key, enforce_keys)}
         end
@@ -289,7 +289,7 @@ defmodule Kernel.Utils do
   macro.
 
   Secondly, if the expression is being used outside of a guard, we want to unquote
-  `value`, but only once, and then re-use the unquoted form throughout the expression.
+  `value`, but only once, and then reuse the unquoted form throughout the expression.
 
   This helper does exactly that: takes the AST for an expression and a list of
   variable references it should be aware of, and rewrites it into a new expression

lib/elixir/lib/keyword.ex

@@ -102,6 +102,9 @@ defmodule Keyword do
   @type key :: atom
   @type value :: any
 
+  @typedoc since: "1.17.0"
+  @type default :: any
+
   @type t :: [{key, value}]
   @type t(value) :: [{key, value}]
 
@@ -388,7 +391,7 @@ defmodule Keyword do
       3
 
   """
-  @spec get(t, key, value) :: value
+  @spec get(t, key, default) :: value | default
   def get(keywords, key, default \\ nil) when is_list(keywords) and is_atom(key) do
     case :lists.keyfind(key, 1, keywords) do
       {^key, value} -> value
@@ -957,6 +960,36 @@ defmodule Keyword do
     :lists.sort(left) === :lists.sort(right)
   end
 
+  @doc """
+  Intersects two keyword lists, returning a keyword with the common keys.
+
+  By default, it returns the values of the intersected keys in `keyword2`.
+  The keys are returned in the order found in `keyword1`.
+
+  ## Examples
+
+      iex> Keyword.intersect([a: 1, b: 2], [b: "b", c: "c"])
+      [b: "b"]
+
+      iex> Keyword.intersect([a: 1, b: 2], [b: 2, c: 3], fn _k, v1, v2 ->
+      ...>   v1 + v2
+      ...> end)
+      [b: 4]
+
+  """
+  @doc since: "1.17.0"
+  @spec intersect(keyword, keyword, (key, value, value -> value)) :: keyword
+  def intersect(keyword1, keyword2, fun \\ fn _key, _v1, v2 -> v2 end)
+
+  def intersect([{k, v1} | keyword1], keyword2, fun) do
+    case :lists.keyfind(k, 1, keyword2) do
+      {_, v2} -> [{k, fun.(k, v1, v2)} | intersect(keyword1, keyword2, fun)]
+      false -> intersect(keyword1, keyword2, fun)
+    end
+  end
+
+  def intersect([], _keyword2, _fun), do: []
+
   @doc """
   Merges two keyword lists into one.
 
@@ -1289,7 +1322,7 @@ defmodule Keyword do
       {1, []}
 
   """
-  @spec pop(t, key, value) :: {value, t}
+  @spec pop(t, key, default) :: {value | default, t}
   def pop(keywords, key, default \\ nil) when is_list(keywords) and is_atom(key) do
     case fetch(keywords, key) do
       {:ok, value} -> {value, delete(keywords, key)}
@@ -1405,7 +1438,7 @@ defmodule Keyword do
       {1, [a: 2]}
 
   """
-  @spec pop_first(t, key, value) :: {value, t}
+  @spec pop_first(t, key, default) :: {value | default, t}
   def pop_first(keywords, key, default \\ nil) when is_list(keywords) and is_atom(key) do
     case :lists.keytake(key, 1, keywords) do
       {:value, {^key, value}, rest} -> {value, rest}

lib/elixir/lib/list.ex

@@ -107,7 +107,7 @@ defmodule List do
   charlists in IEx when you encounter them, which shows you the type, description
   and also the raw representation in one single summary.
 
-  The rationale behind this behaviour is to better support
+  The rationale behind this behavior is to better support
   Erlang libraries which may return text as charlists
   instead of Elixir strings. In Erlang, charlists are the default
   way of handling strings, while in Elixir it's binaries. One
@@ -895,7 +895,7 @@ defmodule List do
   end
 
   @doc """
-  Returns `true` if `list` starts with the given `prefix` list; otherwise returns `false`.
+  Returns `true` if `list` starts with the given `prefix` list, otherwise returns `false`.
 
   If `prefix` is an empty list, it returns `true`.
 

lib/elixir/lib/list/chars.ex

@@ -58,6 +58,6 @@ end
 
 defimpl List.Chars, for: Float do
   def to_charlist(term) do
-    :io_lib_format.fwrite_g(term)
+    :erlang.float_to_list(term, [:short])
   end
 end

lib/elixir/lib/macro.ex

@@ -115,7 +115,7 @@ defmodule Macro do
     * `:from_brackets` - Used to determine whether a call to `Access.get/3` is from
       bracket syntax.
 
-    * `:from_interpolation` - Used to determine whether a call to `Access.get/3` is
+    * `:from_interpolation` - Used to determine whether a call to `Kernel.to_string/1` is
       from interpolation.
 
     * `:generated` - Whether the code should be considered as generated by
@@ -136,8 +136,9 @@ defmodule Macro do
 
     * `:closing` - contains metadata about the closing pair, such as a `}`
       in a tuple or in a map, or such as the closing `)` in a function call
-      with parens. The `:closing` does not delimit the end of expression if
-      there are `:do` and `:end` metadata  (when `:token_metadata` is true)
+      with parens (when `:token_metadata` is true). If the function call
+      has a do-end block attached to it, its metadata is found under the
+      `:do` and `:end` metadata
 
     * `:column` - the column number of the AST node (when `:columns` is true).
       Note column information is always discarded from quoted code.
@@ -154,8 +155,11 @@ defmodule Macro do
       `do`-`end` blocks (when `:token_metadata` is true)
 
     * `:end_of_expression` - denotes when the end of expression effectively
-      happens. Available for all expressions except the last one inside a
-      `__block__` (when `:token_metadata` is true)
+      happens (when `:token_metadata` is true). This is only available for
+      expressions inside "blocks of code", which are either direct children
+      of a `__block__` or the right side of `->`. The last expression of the
+      block does not have metadata if it is not followed by an end of line
+      character (either a newline or `;`)
 
     * `:indentation` - indentation of a sigil heredoc
 
@@ -367,7 +371,7 @@ defmodule Macro do
   Note the arguments are not unique. If you later on want
   to access the same variables, you can invoke this function
   with the same inputs. Use `generate_unique_arguments/2` to
-  generate a unique arguments that can't be overridden.
+  generate unique arguments that can't be overridden.
 
   ## Examples
 
@@ -491,10 +495,8 @@ defmodule Macro do
   """
   @doc since: "1.11.3"
   @spec generate_unique_arguments(0, context :: atom) :: []
-  @spec generate_unique_arguments(pos_integer, context) :: [
-          {atom, [counter: integer], context},
-          ...
-        ]
+  @spec generate_unique_arguments(pos_integer, context) ::
+          [{atom, [counter: integer], context}, ...]
         when context: atom
   def generate_unique_arguments(amount, context),
     do: generate_arguments(amount, context, &unique_var/2)
@@ -1130,7 +1132,6 @@ defmodule Macro do
 
   """
   @spec to_string(t()) :: String.t()
-  # TODO: Allow line_length to be configurable on v1.17
   def to_string(tree) do
     doc = Inspect.Algebra.format(Code.quoted_to_algebra(tree), 98)
     IO.iodata_to_binary(doc)
@@ -1710,7 +1711,7 @@ defmodule Macro do
   @doc """
   Applies a `mod`, `function`, and `args` at compile-time in `caller`.
 
-  This is used when you want to programatically invoke a macro at
+  This is used when you want to programmatically invoke a macro at
   compile-time.
   """
   @doc since: "1.16.0"
@@ -1731,7 +1732,9 @@ defmodule Macro do
 
   If the expression cannot be expanded, it returns the expression
   itself. This function does not traverse the AST, only the root
-  node is expanded.
+  node is expanded. The expansion happens as if it was expanded by
+  the Elixir compiler and therefore compilation tracers will be invoked
+  and deprecation warnings will be emitted during the expansion.
 
   `expand_once/2` performs the expansion just once. Check `expand/2`
   to perform expansion until the node can no longer be expanded.
@@ -1802,23 +1805,23 @@ defmodule Macro do
     elem(do_expand_once(ast, env), 0)
   end
 
-  defp do_expand_once({:__aliases__, meta, _} = original, env) do
-    case :elixir_aliases.expand_or_concat(original, env) do
+  defp do_expand_once({:__aliases__, meta, list} = alias, env) do
+    case :elixir_aliases.expand_or_concat(meta, list, env, true) do
       receiver when is_atom(receiver) ->
         :elixir_env.trace({:alias_reference, meta, receiver}, env)
         {receiver, true}
 
-      aliases ->
-        aliases = :lists.map(&elem(do_expand_once(&1, env), 0), aliases)
+      [head | tail] ->
+        {head, _} = do_expand_once(head, env)
 
-        case :lists.all(&is_atom/1, aliases) do
+        case is_atom(head) do
           true ->
-            receiver = :elixir_aliases.concat(aliases)
+            receiver = :elixir_aliases.concat([head | tail])
             :elixir_env.trace({:alias_reference, meta, receiver}, env)
             {receiver, true}
 
           false ->
-            {original, false}
+            {alias, false}
         end
     end
   end
@@ -1843,56 +1846,39 @@ defmodule Macro do
     end
   end
 
-  defp do_expand_once({atom, meta, context} = original, _env)
-       when is_atom(atom) and is_list(meta) and is_atom(context) do
+  defp do_expand_once({name, meta, context} = original, _env)
+       when is_atom(name) and is_list(meta) and is_atom(context) do
     {original, false}
   end
 
-  defp do_expand_once({atom, meta, args} = original, env)
-       when is_atom(atom) and is_list(args) and is_list(meta) do
+  defp do_expand_once({name, meta, args} = original, env)
+       when is_atom(name) and is_list(args) and is_list(meta) do
     arity = length(args)
 
-    if special_form?(atom, arity) do
-      {original, false}
-    else
-      module = env.module
+    case Macro.Env.expand_import(env, meta, name, arity) do
+      {:macro, _receiver, expander} ->
+        # We don't want the line to propagate yet, but generated might!
+        {expander.(Keyword.take(meta, [:generated]), args), true}
 
-      extra =
-        if function_exported?(module, :__info__, 1) do
-          [{module, module.__info__(:macros)}]
-        else
-          []
+      {:function, Kernel, op} when op in [:+, :-] and arity == 1 ->
+        case expand_once(hd(args), env) do
+          integer when is_integer(integer) -> {apply(Kernel, op, [integer]), true}
+          _ -> {original, false}
         end
 
-      s = :elixir_env.env_to_ex(env)
+      {:function, _receiver, _name} ->
+        {original, false}
 
-      expand =
-        :elixir_dispatch.expand_import(meta, {atom, length(args)}, args, s, env, extra, true)
+      {:error, :not_found} ->
+        {original, false}
 
-      case expand do
-        {:ok, receiver, quoted} ->
-          next = :elixir_module.next_counter(module)
-          # We don't want the line to propagate yet, but generated might!
-          meta = Keyword.take(meta, [:generated])
-          {:elixir_quote.linify_with_context_counter(meta, {receiver, next}, quoted), true}
-
-        {:ok, Kernel, op, [arg]} when op in [:+, :-] ->
-          case expand_once(arg, env) do
-            integer when is_integer(integer) -> {apply(Kernel, op, [integer]), true}
-            _ -> {original, false}
-          end
-
-        {:ok, _receiver, _name, _args} ->
-          {original, false}
-
-        :error ->
-          {original, false}
-      end
+      {:error, other} ->
+        :elixir_errors.file_error(meta, env, :elixir_dispatch, {:import, other, name, arity})
     end
   end
 
   # Expand possible macro require invocation
-  defp do_expand_once({{:., _, [left, right]}, meta, args} = original, env) when is_atom(right) do
+  defp do_expand_once({{:., _, [left, name]}, meta, args} = original, env) when is_atom(name) do
     {receiver, _} = do_expand_once(left, env)
 
     case is_atom(receiver) do
@@ -1900,16 +1886,10 @@ defmodule Macro do
         {original, false}
 
       true ->
-        s = :elixir_env.env_to_ex(env)
-        name_arity = {right, length(args)}
-        expand = :elixir_dispatch.expand_require(meta, receiver, name_arity, args, s, env)
-
-        case expand do
-          {:ok, receiver, quoted} ->
-            next = :elixir_module.next_counter(env.module)
+        case Macro.Env.expand_require(env, meta, receiver, name, length(args)) do
+          {:macro, _receiver, expander} ->
             # We don't want the line to propagate yet, but generated might!
-            meta = Keyword.take(meta, [:generated])
-            {:elixir_quote.linify_with_context_counter(meta, {receiver, next}, quoted), true}
+            {expander.(Keyword.take(meta, [:generated]), args), true}
 
           :error ->
             {original, false}
@@ -1964,8 +1944,8 @@ defmodule Macro do
   def operator?(name, 1) when is_atom(name),
     do: Identifier.unary_op(name) != :error
 
-  def operator?(:.., 0),
-    do: true
+  def operator?(:.., 0), do: true
+  def operator?(:..., 0), do: true
 
   def operator?(name, arity) when is_atom(name) and is_integer(arity), do: false
 

lib/elixir/lib/macro/env.ex

@@ -6,9 +6,15 @@ defmodule Macro.Env do
   `__ENV__/0`. Inside macros, the caller environment can be
   accessed as `__CALLER__/0`.
 
-  An instance of `Macro.Env` must not be modified by hand. If you need to
-  create a custom environment to pass to `Code.eval_quoted/3`, use the
-  following trick:
+  The majority of the functions in this module are provided
+  for low-level tools, which need to integrate with the Elixir
+  compiler, such as language servers and embedded languages.
+  For regular usage in Elixir code and macros, you must use
+  the `Macro` module instead. In particular, avoid modifying
+  the `Macro.Env` struct directly and prefer to use high-level
+  constructs, such as a `import`, `aliases`, and so forth to
+  build your own environment. For example, to build a custom
+  environment, you can define a function such as:
 
       def make_custom_env do
         import SomeModule, only: [some_function: 2], warn: false
@@ -16,10 +22,9 @@ defmodule Macro.Env do
         __ENV__
       end
 
-  You may then call `make_custom_env()` to get a struct with the desired
-  imports and aliases included.
+  ## Struct fields
 
-  It contains the following fields:
+  The `Macro.Env` struct contains the following fields:
 
     * `context` - the context of the environment; it can be `nil`
       (default context), `:guard` (inside a guard) or `:match` (inside a match)
@@ -98,7 +103,7 @@ defmodule Macro.Env do
 
   # Define the __struct__ callbacks by hand for bootstrap reasons.
   {struct, [], kv, body} = Kernel.Utils.defstruct(__MODULE__, fields, false, __ENV__)
-  def __struct__(), do: unquote(:elixir_quote.escape(struct, false, :none))
+  def __struct__(), do: unquote(:elixir_quote.escape(struct, :none, false))
   def __struct__(unquote(kv)), do: unquote(body)
 
   @doc """
@@ -163,43 +168,13 @@ defmodule Macro.Env do
     [file: file, line: line]
   end
 
-  @doc """
-  Fetches the alias for the given atom.
-
-  Returns `{:ok, alias}` if the alias exists, `:error`
-  otherwise.
-
-  ## Examples
-
-      iex> alias Foo.Bar, as: Baz
-      iex> Baz
-      Foo.Bar
-      iex> Macro.Env.fetch_alias(__ENV__, :Baz)
-      {:ok, Foo.Bar}
-      iex> Macro.Env.fetch_alias(__ENV__, :Unknown)
-      :error
-
-  """
-  @doc since: "1.13.0"
-  @spec fetch_alias(t, atom) :: {:ok, atom} | :error
-  def fetch_alias(env, atom)
-
+  # TODO: Deprecate on Elixir 1.21 in favor of expand_alias/4
+  @doc false
   def fetch_alias(%{__struct__: Macro.Env, aliases: aliases}, atom) when is_atom(atom),
     do: Keyword.fetch(aliases, :"Elixir.#{atom}")
 
-  @doc """
-  Fetches the macro alias for the given atom.
-
-  Returns `{:ok, macro_alias}` if the alias exists, `:error`
-  otherwise.
-
-  A macro alias is only used inside quoted expansion. See
-  `fetch_alias/2` for a more general example.
-  """
-  @doc since: "1.13.0"
-  @spec fetch_macro_alias(t, atom) :: {:ok, atom} | :error
-  def fetch_macro_alias(env, atom)
-
+  # TODO: Deprecate on Elixir 1.21 in favor of expand_alias/4
+  @doc false
   def fetch_macro_alias(%{__struct__: Macro.Env, macro_aliases: aliases}, atom)
       when is_atom(atom),
       do: Keyword.fetch(aliases, :"Elixir.#{atom}")
@@ -212,6 +187,14 @@ defmodule Macro.Env do
   `{:function | :macro, module}`. The elements in the list
   are in no particular order and the order is not guaranteed.
 
+  > #### Use only for introspection {: .warning}
+  >
+  > This function does not emit compiler tracing events,
+  > which may block the compiler from correctly tracking
+  > dependencies. Use this function for reflection purposes
+  > but to do not use it to expand imports into qualified
+  > calls. Instead, use `expand_import/5`.
+
   ## Examples
 
       iex> Macro.Env.lookup_import(__ENV__, {:duplicate, 2})
@@ -303,8 +286,337 @@ defmodule Macro.Env do
     %{env | tracers: [tracer | tracers]}
   end
 
+  trace_option = """
+  `:trace` - when set to `false`, it disables compilation tracers and
+  lexical tracker. This option must only be used by language servers and
+  other tools that need to introspect code without affecting how it is compiled.
+  Disabling tracer inside macros or regular code expansion is extremely
+  discouraged as it blocks the compiler from accurately tracking dependencies\
+  """
+
   @doc """
-  Returns a `Macro.Env` in the match context.
+  Defines the given `module` as required in the environment.
+
+  It does not check or assert the module is available.
+  This is used by tools which need to mimic the Elixir compiler.
+  The appropriate `:require` compiler tracing event will be emitted.
+
+  ## Additional options
+
+  It accepts the same options as `Kernel.SpecialForm.require/2` plus:
+
+    * #{trace_option}
+
+  ## Examples
+
+      iex> env = __ENV__
+      iex> Macro.Env.required?(env, Integer)
+      false
+      iex> {:ok, env} = Macro.Env.define_require(env, [line: 10], Integer)
+      iex> Macro.Env.required?(env, Integer)
+      true
+
+  If the `:as` option is given, it will also define an alias:
+
+      iex> env = __ENV__
+      iex> {:ok, env} = Macro.Env.define_require(env, [line: 10], Foo.Bar, as: Baz)
+      iex> Macro.Env.expand_alias(env, [], [:Baz])
+      {:alias, Foo.Bar}
+
+  """
+  @doc since: "1.17.0"
+  @spec define_require(t, Macro.metadata(), module) :: {:ok, t}
+  def define_require(env, meta, module, opts \\ [])
+      when is_list(meta) and is_atom(module) and is_list(opts) do
+    {trace, opts} = Keyword.pop(opts, :trace, true)
+    env = :elixir_aliases.require(meta, module, opts, env, trace)
+    result = :elixir_aliases.alias(meta, module, false, opts, env, trace)
+    maybe_define_error(result, :elixir_aliases)
+  end
+
+  @doc """
+  Defines the given `module` as imported in the environment.
+
+  It assumes `module` is available. This is used by tools which
+  need to mimic the Elixir compiler. The appropriate `:import`
+  compiler tracing event will be emitted.
+
+  ## Additional options
+
+  It accepts the same options as `Kernel.SpecialForm.import/2` plus:
+
+    * `:emit_warnings` - emit warnings found when defining imports
+
+    * #{trace_option}
+
+    * `:info_callback` - a function to use instead of `c:Module.__info__/1`.
+      The function will be invoked with `:functions` or `:macros` argument.
+      It has to return a list of `{function, arity}` key value pairs.
+      If it fails, it defaults to using module metadata based on `module_info/1`.
+
+  ## Examples
+
+      iex> env = __ENV__
+      iex> Macro.Env.lookup_import(env, {:flatten, 1})
+      []
+      iex> {:ok, env} = Macro.Env.define_import(env, [line: 10], List)
+      iex> Macro.Env.lookup_import(env, {:flatten, 1})
+      [{:function, List}]
+
+  It accepts the same options as `Kernel.SpecialForm.import/2`:
+
+      iex> env = __ENV__
+      iex> Macro.Env.lookup_import(env, {:is_odd, 1})
+      []
+      iex> {:ok, env} = Macro.Env.define_import(env, [line: 10], Integer, only: :macros)
+      iex> Macro.Env.lookup_import(env, {:is_odd, 1})
+      [{:macro, Integer}]
+
+  ## Info callback override
+
+      iex> env = __ENV__
+      iex> Macro.Env.lookup_import(env, {:flatten, 1})
+      []
+      iex> {:ok, env} = Macro.Env.define_import(env, [line: 10], SomeModule, [info_callback: fn :functions -> [{:flatten, 1}]; :macros -> [{:some, 2}]; end])
+      iex> Macro.Env.lookup_import(env, {:flatten, 1})
+      [{:function, SomeModule}]
+      iex> Macro.Env.lookup_import(env, {:some, 2})
+      [{:macro, SomeModule}]
+
+  """
+  @doc since: "1.17.0"
+  @spec define_import(t, Macro.metadata(), module, keyword) :: {:ok, t} | {:error, String.t()}
+  def define_import(env, meta, module, opts \\ [])
+      when is_list(meta) and is_atom(module) and is_list(opts) do
+    {trace, opts} = Keyword.pop(opts, :trace, true)
+    {warnings, opts} = Keyword.pop(opts, :emit_warnings, true)
+    {info_callback, opts} = Keyword.pop(opts, :info_callback, &module.__info__/1)
+
+    result = :elixir_import.import(meta, module, opts, env, warnings, trace, info_callback)
+    maybe_define_error(result, :elixir_import)
+  end
+
+  @doc """
+  Defines the given `as` an alias to `module` in the environment.
+
+  This is used by tools which need to mimic the Elixir compiler.
+  The appropriate `:alias` compiler tracing event will be emitted.
+
+  ## Additional options
+
+  It accepts the same options as `Kernel.SpecialForm.alias/2` plus:
+
+    * #{trace_option}
+
+  ## Examples
+
+      iex> env = __ENV__
+      iex> Macro.Env.expand_alias(env, [], [:Baz])
+      :error
+      iex> {:ok, env} = Macro.Env.define_alias(env, [line: 10], Foo.Bar, as: Baz)
+      iex> Macro.Env.expand_alias(env, [], [:Baz])
+      {:alias, Foo.Bar}
+      iex> Macro.Env.expand_alias(env, [], [:Baz, :Bat])
+      {:alias, Foo.Bar.Bat}
+
+  If no `:as` option is given, the alias will be inferred from the module:
+
+      iex> env = __ENV__
+      iex> {:ok, env} = Macro.Env.define_alias(env, [line: 10], Foo.Bar)
+      iex> Macro.Env.expand_alias(env, [], [:Bar])
+      {:alias, Foo.Bar}
+
+  If it is not possible to infer one, an error is returned:
+
+      iex> Macro.Env.define_alias(__ENV__, [line: 10], :an_atom)
+      {:error,
+       "alias cannot be inferred automatically for module: :an_atom, " <>
+         "please use the :as option. Implicit aliasing is only supported with Elixir modules"}
+
+  """
+  @doc since: "1.17.0"
+  @spec define_alias(t, Macro.metadata(), module, keyword) :: {:ok, t} | {:error, String.t()}
+  def define_alias(env, meta, module, opts \\ [])
+      when is_list(meta) and is_atom(module) and is_list(opts) do
+    {trace, opts} = Keyword.pop(opts, :trace, true)
+    result = :elixir_aliases.alias(meta, module, true, opts, env, trace)
+    maybe_define_error(result, :elixir_aliases)
+  end
+
+  defp maybe_define_error({:ok, env}, _mod),
+    do: {:ok, env}
+
+  defp maybe_define_error({:error, reason}, mod),
+    do: {:error, Kernel.to_string(mod.format_error(reason))}
+
+  @doc """
+  Expands an alias given by the alias segments.
+
+  It returns `{:alias, alias}` if the segments is a list
+  of atoms and an alias was found. Returns `:error` otherwise.
+
+  This expansion may emit the `:alias_expansion` trace event
+  but it does not emit the `:alias_reference` one.
+
+  ## Options
+
+    * #{trace_option}
+
+  ## Examples
+
+      iex> alias List, as: MyList
+      iex> Macro.Env.expand_alias(__ENV__, [], [:MyList])
+      {:alias, List}
+      iex> Macro.Env.expand_alias(__ENV__, [], [:MyList, :Nested])
+      {:alias, List.Nested}
+
+  If there is no alias or the alias starts with `Elixir.`
+  (which disables aliasing), then `:error` is returned:
+
+      iex> alias List, as: MyList
+      iex> Macro.Env.expand_alias(__ENV__, [], [:Elixir, MyList])
+      :error
+      iex> Macro.Env.expand_alias(__ENV__, [], [:AnotherList])
+      :error
+
+  """
+  @doc since: "1.17.0"
+  @spec expand_alias(t, keyword, [atom()], keyword) ::
+          {:alias, atom()} | :error
+  def expand_alias(env, meta, list, opts \\ [])
+      when is_list(meta) and is_list(list) and is_list(opts) do
+    trace = Keyword.get(opts, :trace, true)
+
+    case :elixir_aliases.expand(meta, list, env, trace) do
+      atom when is_atom(atom) -> {:alias, atom}
+      [_ | _] -> :error
+    end
+  end
+
+  @doc """
+  Expands an import given by `name` and `arity`.
+
+  If the import points to a macro, it returns a tuple
+  with the module and a function that expands the macro.
+  The function expects the metadata to be attached to the
+  expansion and the arguments of the macro.
+
+  If the import points to a function, it returns a tuple
+  with the module and the function name.
+
+  If any import is found, the appropriate compiler tracing
+  event will be emitted.
+
+  Otherwise returns `{:error, reason}`.
+
+  ## Options
+
+    * `:allow_locals` - when set to `false`, it does not attempt to capture
+      local macros defined in the current module in `env`
+
+    * `:check_deprecations` - when set to `false`, does not check for deprecations
+      when expanding macros
+
+    * #{trace_option}
+
+  """
+  @doc since: "1.17.0"
+  @spec expand_import(t, keyword, atom(), arity(), keyword) ::
+          {:macro, module(), (Macro.metadata(), args :: [Macro.t()] -> Macro.t())}
+          | {:function, module(), atom()}
+          | {:error, :not_found | {:conflict, module()} | {:ambiguous, [module()]}}
+  def expand_import(env, meta, name, arity, opts \\ [])
+      when is_list(meta) and is_atom(name) and is_integer(arity) and is_list(opts) do
+    case :elixir_import.special_form(name, arity) do
+      true ->
+        {:error, :not_found}
+
+      false ->
+        allow_locals = Keyword.get(opts, :allow_locals, true)
+        trace = Keyword.get(opts, :trace, true)
+        module = env.module
+
+        extra =
+          case allow_locals and function_exported?(module, :__info__, 1) do
+            true -> [{module, module.__info__(:macros)}]
+            false -> []
+          end
+
+        case :elixir_dispatch.expand_import(meta, name, arity, env, extra, allow_locals, trace) do
+          {:macro, receiver, expander} ->
+            {:macro, receiver, wrap_expansion(receiver, expander, meta, name, arity, env, opts)}
+
+          {:function, receiver, name} ->
+            {:function, receiver, name}
+
+          error ->
+            {:error, error}
+        end
+    end
+  end
+
+  @doc """
+  Expands a require given by `module`, `name`, and `arity`.
+
+  If the require points to a macro and the module has been
+  required, it returns a tuple with the module and a function
+  that expands the macro. The function expects the metadata
+  to be attached to the expansion and the arguments of the macro.
+  The appropriate `:remote_macro` compiler tracing event will
+  be emitted if a macro is found (note a `:remote_function`
+  event is not emitted in `:error` cases).
+
+  Otherwise returns `:error`.
+
+  ## Options
+
+    * `:check_deprecations` - when set to `false`, does not check for deprecations
+      when expanding macros
+
+    * #{trace_option}
+
+  """
+  @doc since: "1.17.0"
+  @spec expand_require(t, keyword, module(), atom(), arity(), keyword) ::
+          {:macro, module(), (Macro.metadata(), args :: [Macro.t()] -> Macro.t())}
+          | :error
+  def expand_require(env, meta, module, name, arity, opts \\ [])
+      when is_list(meta) and is_atom(module) and is_atom(name) and is_integer(arity) and
+             is_list(opts) do
+    trace = Keyword.get(opts, :trace, true)
+
+    case :elixir_dispatch.expand_require(meta, module, name, arity, env, trace) do
+      {:macro, receiver, expander} ->
+        {:macro, receiver, wrap_expansion(receiver, expander, meta, name, arity, env, opts)}
+
+      :error ->
+        :error
+    end
+  end
+
+  defp wrap_expansion(receiver, expander, meta, name, arity, env, opts) do
+    fn expansion_meta, args ->
+      if Keyword.get(opts, :check_deprecations, true) do
+        :elixir_dispatch.check_deprecated(:macro, meta, receiver, name, arity, env)
+      end
+
+      quoted = expander.(args, env)
+      next = :elixir_module.next_counter(env.module)
+      :elixir_quote.linify_with_context_counter(expansion_meta, {receiver, next}, quoted)
+    end
+  end
+
+  @doc """
+  Returns an environment in the guard context.
+  """
+  @doc since: "1.17.0"
+  @spec to_guard(t) :: t
+  def to_guard(%{__struct__: Macro.Env} = env) do
+    %{env | context: :guard}
+  end
+
+  @doc """
+  Returns an environment in the match context.
   """
   @spec to_match(t) :: t
   def to_match(%{__struct__: Macro.Env} = env) do

lib/elixir/lib/map.ex

@@ -147,7 +147,7 @@ defmodule Map do
 
   ## Examples
 
-      iex> Map.keys(%{a: 1, b: 2})
+      Map.keys(%{a: 1, b: 2})
       [:a, :b]
 
   """
@@ -161,7 +161,7 @@ defmodule Map do
 
   ## Examples
 
-      iex> Map.values(%{a: 1, b: 2})
+      Map.values(%{a: 1, b: 2})
       [1, 2]
 
   """

lib/elixir/lib/module.ex

@@ -326,8 +326,17 @@ defmodule Module do
   ### `@nifs` (since v1.16.0)
 
   A list of functions and their arities which will be overridden
-  by a native implementation (NIF). See the Erlang documentation
-  for more information: https://www.erlang.org/doc/man/erl_nif
+  by a native implementation (NIF).
+
+      defmodule MyLibrary.MyModule do
+        @nifs [foo: 1, bar: 2]
+
+        def foo(arg1), do: :erlang.nif_error(:not_loaded)
+        def bar(arg1, arg2), do: :erlang.nif_error(:not_loaded)
+      end
+
+  See the Erlang documentation for more information:
+  https://www.erlang.org/doc/man/erl_nif
 
   ### `@on_definition`
 
@@ -529,7 +538,9 @@ defmodule Module do
   undefined functions, deprecations, etc. A module is always verified after
   it is compiled. In Mix projects, a module is also verified when any of its
   runtime dependencies change. Therefore this is useful to perform verification
-  of the current module while avoiding compile-time dependencies.
+  of the current module while avoiding compile-time dependencies. Given the
+  callback is invoked under different scenarios, Elixir provides no guarantees
+  of when in the compilation cycle nor in which process the callback runs.
 
   Accepts a module or a `{module, function_name}` tuple. The function
   must take one argument: the module name. When just a module is provided,
@@ -1432,7 +1443,7 @@ defmodule Module do
                 "to defoverridable/1 because #{error_explanation}"
     end
 
-    behaviour_callbacks = Module.Types.Behaviour.callbacks(behaviour)
+    behaviour_callbacks = Module.Behaviour.callbacks(behaviour)
 
     tuples =
       for definition <- definitions_in(module),
@@ -1949,6 +1960,9 @@ defmodule Module do
     {set, bag} = data_tables_for(module)
 
     case :ets.lookup(set, key) do
+      [{_, _, :unset, _}] ->
+        default
+
       [{_, _, :accumulate, traces}] ->
         trace_attribute(trace?, module, traces, set, key, [])
         lookup_accumulate_attribute(bag, key, default, last_accumulated?)
@@ -2124,8 +2138,7 @@ defmodule Module do
   end
 
   defp attribute_stack(module, line) do
-    file = String.to_charlist(Path.relative_to_cwd(:elixir_module.file(module)))
-    [{module, :__MODULE__, 0, file: file, line: line}]
+    struct!(Macro.Env, module: module, file: :elixir_module.file(module), line: line)
   end
 
   ## Helpers

lib/elixir/lib/module/behaviour.ex

@@ -1,4 +1,4 @@
-defmodule Module.Types.Behaviour do
+defmodule Module.Behaviour do
   # Checking functionality for @behaviours and @impl
   @moduledoc false
 
@@ -284,7 +284,11 @@ defmodule Module.Types.Behaviour do
     end
   end
 
-  def format_warning({:undefined_behaviour, module, behaviour}) do
+  def format_diagnostic(warning) do
+    %{message: IO.iodata_to_binary(format_warning(warning))}
+  end
+
+  defp format_warning({:undefined_behaviour, module, behaviour}) do
     [
       "@behaviour ",
       inspect(behaviour),
@@ -294,12 +298,12 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:module_does_not_define_behaviour, module, behaviour}) do
+  defp format_warning({:module_does_not_define_behaviour, module, behaviour}) do
     ["module ", inspect(behaviour), " is not a behaviour (in module ", inspect(module), ")"]
   end
 
-  def format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback})
-      when conflict == behaviour do
+  defp format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback})
+       when conflict == behaviour do
     [
       "the behaviour ",
       inspect(behaviour),
@@ -311,7 +315,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) do
+  defp format_warning({:duplicate_behaviour, module, behaviour, conflict, kind, callback}) do
     [
       "conflicting behaviours found. Callback ",
       format_definition(kind, callback),
@@ -325,7 +329,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:missing_callback, module, callback, kind, behaviour}) do
+  defp format_warning({:missing_callback, module, callback, kind, behaviour}) do
     [
       format_callback(callback, kind, behaviour),
       " is not implemented (in module ",
@@ -334,7 +338,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:callback_mismatch, module, callback, kind, wrong_kind, behaviour}) do
+  defp format_warning({:callback_mismatch, module, callback, kind, wrong_kind, behaviour}) do
     [
       format_callback(callback, kind, behaviour),
       " was implemented as \"",
@@ -347,14 +351,14 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:private_function, callback, kind}) do
+  defp format_warning({:private_function, callback, kind}) do
     [
       format_definition(kind, callback),
       " is private, @impl attribute is always discarded for private functions/macros"
     ]
   end
 
-  def format_warning({:no_behaviours, callback, kind, value}) do
+  defp format_warning({:no_behaviours, callback, kind, value}) do
     [
       "got \"@impl ",
       inspect(value),
@@ -364,7 +368,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:impl_not_defined, callback, kind, {_fa, behaviour}}) do
+  defp format_warning({:impl_not_defined, callback, kind, {_fa, behaviour}}) do
     [
       "got \"@impl false\" for ",
       format_definition(kind, callback),
@@ -373,7 +377,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:impl_defined, callback, kind, callbacks}) do
+  defp format_warning({:impl_defined, callback, kind, callbacks}) do
     [
       "got \"@impl true\" for ",
       format_definition(kind, callback),
@@ -382,7 +386,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:behaviour_not_declared, callback, kind, behaviour}) do
+  defp format_warning({:behaviour_not_declared, callback, kind, behaviour}) do
     [
       "got \"@impl ",
       inspect(behaviour),
@@ -392,7 +396,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:behaviour_not_defined, callback, kind, behaviour, callbacks}) do
+  defp format_warning({:behaviour_not_defined, callback, kind, behaviour, callbacks}) do
     [
       "got \"@impl ",
       inspect(behaviour),
@@ -403,7 +407,7 @@ defmodule Module.Types.Behaviour do
     ]
   end
 
-  def format_warning({:missing_impl, callback, kind, behaviour}) do
+  defp format_warning({:missing_impl, callback, kind, behaviour}) do
     [
       "module attribute @impl was not set for ",
       format_definition(kind, callback),

lib/elixir/lib/module/parallel_checker.ex

@@ -167,8 +167,9 @@ defmodule Module.ParallelChecker do
 
   defp collect_results(count, diagnostics) do
     receive do
-      {:diagnostic, diagnostic} ->
-        diagnostic = format_diagnostic_file(diagnostic)
+      {:diagnostic, %{file: file} = diagnostic, read_snippet} ->
+        :elixir_errors.print_diagnostic(diagnostic, read_snippet)
+        diagnostic = %{diagnostic | file: file && Path.absname(file)}
         collect_results(count, [diagnostic | diagnostics])
 
       {__MODULE__, _module, new_diagnostics} ->
@@ -249,7 +250,7 @@ defmodule Module.ParallelChecker do
       |> merge_compiler_no_warn_undefined()
 
     behaviour_warnings =
-      Module.Types.Behaviour.check_behaviours_and_impls(
+      Module.Behaviour.check_behaviours_and_impls(
         module,
         file,
         line,
@@ -258,7 +259,7 @@ defmodule Module.ParallelChecker do
         definitions
       )
 
-    warnings =
+    diagnostics =
       module
       |> Module.Types.warnings(file, definitions, no_warn_undefined, cache)
       |> Kernel.++(behaviour_warnings)
@@ -269,7 +270,7 @@ defmodule Module.ParallelChecker do
     |> Map.get(:after_verify, [])
     |> Enum.each(fn {verify_mod, verify_fun} -> apply(verify_mod, verify_fun, [module]) end)
 
-    warnings
+    diagnostics
   end
 
   defp extract_no_warn_undefined(compile_opts) do
@@ -287,11 +288,6 @@ defmodule Module.ParallelChecker do
     end
   end
 
-  @doc false
-  def format_diagnostic_file(%{file: file} = diagnostic) do
-    %{diagnostic | file: file && Path.absname(file)}
-  end
-
   ## Warning helpers
 
   defp group_warnings(warnings) do
@@ -306,35 +302,36 @@ defmodule Module.ParallelChecker do
 
   defp emit_warnings(warnings, log?) do
     Enum.flat_map(warnings, fn {module, warning, locations} ->
-      message = module.format_warning(warning)
-      diagnostics = Enum.map(locations, &to_diagnostic(message, &1))
-      log? and print_warning(message, diagnostics)
+      %{message: _} = diagnostic = module.format_diagnostic(warning)
+      diagnostics = Enum.map(locations, &to_diagnostic(diagnostic, &1))
+      log? and print_diagnostics(diagnostics)
       diagnostics
     end)
   end
 
-  defp print_warning(message, [diagnostic]) do
-    :elixir_errors.print_warning(message, diagnostic)
+  defp print_diagnostics([diagnostic]) do
+    :elixir_errors.print_diagnostic(diagnostic, true)
   end
 
-  defp print_warning(message, grouped_warnings) do
-    :elixir_errors.print_warning_group(message, grouped_warnings)
+  defp print_diagnostics(diagnostics) do
+    :elixir_errors.print_diagnostics(diagnostics)
   end
 
-  defp to_diagnostic(message, {file, position, mfa}) when is_list(position) do
+  defp to_diagnostic(diagnostic, {file, position, mfa}) when is_list(position) do
     %{
       severity: :warning,
+      source: file,
       file: file,
       position: position_to_tuple(position),
-      message: IO.iodata_to_binary(message),
       stacktrace: [to_stacktrace(file, position, mfa)],
       span: nil
     }
+    |> Map.merge(diagnostic)
   end
 
   defp position_to_tuple(position) do
     case position[:column] do
-      nil -> position[:line]
+      nil -> position[:line] || 0
       col -> {position[:line], col}
     end
   end

lib/elixir/lib/module/types.ex

@@ -1,31 +1,24 @@
 defmodule Module.Types do
   @moduledoc false
 
-  defmodule Error do
-    defexception [:message]
-  end
-
-  import Module.Types.Helpers
-  alias Module.Types.{Expr, Pattern, Unify}
+  alias Module.Types.{Expr, Pattern}
 
   @doc false
   def warnings(module, file, defs, no_warn_undefined, cache) do
-    stack = stack()
+    context = context()
 
     Enum.flat_map(defs, fn {{fun, arity} = function, kind, meta, clauses} ->
-      context = context(with_file_meta(meta, file), module, function, no_warn_undefined, cache)
-
-      Enum.flat_map(clauses, fn {_meta, args, guards, body} ->
-        def_expr = {kind, meta, [guards_to_expr(guards, {fun, [], args})]}
+      stack = stack(with_file_meta(meta, file), module, function, no_warn_undefined, cache)
 
+      Enum.flat_map(clauses, fn {meta, args, guards, body} ->
         try do
-          warnings_from_clause(args, guards, body, def_expr, stack, context)
+          warnings_from_clause(meta, args, guards, body, stack, context)
         rescue
           e ->
             def_expr = {kind, meta, [guards_to_expr(guards, {fun, [], args}), [do: body]]}
 
             error =
-              Error.exception("""
+              RuntimeError.exception("""
               found error while checking types for #{Exception.format_mfa(module, fun, arity)}:
 
               #{Exception.format_banner(:error, e, __STACKTRACE__)}\
@@ -34,7 +27,7 @@ defmodule Module.Types do
 
               #{Macro.to_string(def_expr)}
 
-              In case it is a bug, please report it at: https://github.com/elixir-lang/elixir/issues
+              Please report this bug at: https://github.com/elixir-lang/elixir/issues
               """)
 
             reraise error, __STACKTRACE__
@@ -58,21 +51,20 @@ defmodule Module.Types do
     guards_to_expr(guards, {:when, [], [left, guard]})
   end
 
-  defp warnings_from_clause(args, guards, body, def_expr, stack, context) do
-    head_stack = Unify.push_expr_stack(def_expr, stack)
-
-    with {:ok, _types, context} <- Pattern.of_head(args, guards, head_stack, context),
-         {:ok, _type, context} <- Expr.of_expr(body, :dynamic, stack, context) do
+  defp warnings_from_clause(meta, args, guards, body, stack, context) do
+    with {:ok, _types, context} <- Pattern.of_head(args, guards, meta, stack, context),
+         {:ok, _type, context} <- Expr.of_expr(body, stack, context) do
       context.warnings
     else
-      {:error, {type, error, context}} ->
-        [error_to_warning(type, error, context) | context.warnings]
+      {:error, context} -> context.warnings
     end
   end
 
   @doc false
-  def context(file, module, function, no_warn_undefined, cache) do
+  def stack(file, module, function, no_warn_undefined, cache) do
     %{
+      # The fallback meta used for literals in patterns and guards
+      meta: [],
       # File of module
       file: file,
       # Module of definitions
@@ -83,450 +75,18 @@ defmodule Module.Types do
       no_warn_undefined: no_warn_undefined,
       # A list of cached modules received from the parallel compiler
       cache: cache,
-      # Expression variable to type variable
-      vars: %{},
-      # Type variable to expression variable
-      types_to_vars: %{},
-      # Type variable to type
-      types: %{},
-      # Trace of all variables that have been refined to a type,
-      # including the type they were refined to, why, and where
-      traces: %{},
-      # Counter to give type variables unique names
-      counter: 0,
-      # Track if a variable was inferred from a type guard function such is_tuple/1
-      # or a guard function that fails such as elem/2, possible values are:
-      # `:guarded` when `is_tuple(x)`
-      # `:guarded` when `is_tuple and elem(x, 0)`
-      # `:fail` when `elem(x, 0)`
-      guard_sources: %{},
-      # A list with all warnings from the running the code
-      warnings: []
+      # If variable refinements is enabled or not
+      refine: true
     }
   end
 
   @doc false
-  def stack() do
+  def context() do
     %{
-      # Stack of variables we have refined during unification,
-      # used for creating relevant traces
-      unify_stack: [],
-      # Last expression we have recursed through during inference,
-      # used for tracing
-      last_expr: nil,
-      # When false do not add a trace when a type variable is refined,
-      # useful when merging contexts where the variables already have traces
-      trace: true,
-      # There are two factors that control how we track guards.
-      #
-      # * consider_type_guards?: if type guards should be considered.
-      #   This applies only at the root and root-based "and" and "or" nodes.
-      #
-      # * keep_guarded? - if a guarded clause should remain as guarded
-      #   even on failure. Used on the right side of and.
-      #
-      type_guards: {_consider_type_guards? = true, _keep_guarded? = false},
-      # Context used to determine if unification is bi-directional, :expr
-      # is directional, :pattern is bi-directional
-      context: nil
+      # A list of all warnings found so far
+      warnings: [],
+      # Information about all vars and their types
+      vars: %{}
     }
   end
-
-  ## ERROR TO WARNING
-
-  # Collect relevant information from context and traces to report error
-  def error_to_warning(:unable_apply, {mfa, args, expected, signature, stack}, context) do
-    {fun, arity} = context.function
-    location = {context.file, get_position(stack), {context.module, fun, arity}}
-
-    traces = type_traces(stack, context)
-    {[signature | args], traces} = lift_all_types([signature | args], traces, context)
-    error = {:unable_apply, mfa, args, expected, signature, {location, stack.last_expr, traces}}
-    {Module.Types, error, location}
-  end
-
-  def error_to_warning(:unable_unify, {left, right, stack}, context) do
-    {fun, arity} = context.function
-    location = {context.file, get_position(stack), {context.module, fun, arity}}
-
-    traces = type_traces(stack, context)
-    {[left, right], traces} = lift_all_types([left, right], traces, context)
-    error = {:unable_unify, left, right, {location, stack.last_expr, traces}}
-    {Module.Types, error, location}
-  end
-
-  defp get_position(stack) do
-    get_meta(stack.last_expr)
-  end
-
-  # Collect relevant traces from context.traces using stack.unify_stack
-  defp type_traces(stack, context) do
-    # TODO: Do we need the unify_stack or is enough to only get the last variable
-    #       in the stack since we get related variables anyway?
-    stack =
-      stack.unify_stack
-      |> Enum.flat_map(&[&1 | related_variables(&1, context.types)])
-      |> Enum.uniq()
-
-    Enum.flat_map(stack, fn var_index ->
-      with %{^var_index => traces} <- context.traces,
-           %{^var_index => expr_var} <- context.types_to_vars do
-        Enum.map(traces, &tag_trace(expr_var, &1, context))
-      else
-        _other -> []
-      end
-    end)
-  end
-
-  defp related_variables(var, types) do
-    Enum.flat_map(types, fn
-      {related_var, {:var, ^var}} ->
-        [related_var | related_variables(related_var, types)]
-
-      _ ->
-        []
-    end)
-  end
-
-  # Tag if trace is for a concrete type or type variable
-  defp tag_trace(var, {type, expr, location}, context) do
-    with {:var, var_index} <- type,
-         %{^var_index => expr_var} <- context.types_to_vars do
-      {:var, var, expr_var, expr, location}
-    else
-      _ -> {:type, var, type, expr, location}
-    end
-  end
-
-  defp lift_all_types(types, traces, context) do
-    trace_types = for({:type, _, type, _, _} <- traces, do: type)
-    {types, lift_context} = Unify.lift_types(types, context)
-    {trace_types, _lift_context} = Unify.lift_types(trace_types, lift_context)
-
-    {traces, []} =
-      Enum.map_reduce(traces, trace_types, fn
-        {:type, var, _, expr, location}, [type | acc] -> {{:type, var, type, expr, location}, acc}
-        other, acc -> {other, acc}
-      end)
-
-    {types, traces}
-  end
-
-  ## FORMAT WARNINGS
-
-  def format_warning({:unable_apply, mfa, args, expected, signature, {location, expr, traces}}) do
-    {original_module, original_function, arity} = mfa
-    {_, _, args} = mfa_or_fa = erl_to_ex(original_module, original_function, args, [])
-    {module, function, ^arity} = call_to_mfa(mfa_or_fa)
-    format_mfa = Exception.format_mfa(module, function, arity)
-    {traces, [] = _hints} = format_traces(traces, [], false)
-
-    clauses =
-      Enum.map(signature, fn {ins, out} ->
-        {_, _, ins} = erl_to_ex(original_module, original_function, ins, [])
-
-        {:fun, [{ins, out}]}
-        |> Unify.format_type(false)
-        |> IO.iodata_to_binary()
-        |> binary_slice(1..-2//1)
-      end)
-
-    [
-      "expected #{format_mfa} to have signature:\n\n    ",
-      Enum.map_join(args, ", ", &Unify.format_type(&1, false)),
-      " -> #{Unify.format_type(expected, false)}",
-      "\n\nbut it has signature:\n\n    ",
-      indent(Enum.join(clauses, "\n")),
-      "\n\n",
-      format_expr(expr, location),
-      traces,
-      "Conflict found at"
-    ]
-  end
-
-  def format_warning({:unable_unify, left, right, {location, expr, traces}}) do
-    if map_type?(left) and map_type?(right) and match?({:ok, _, _}, missing_field(left, right)) do
-      {:ok, atom, known_atoms} = missing_field(left, right)
-
-      # Drop the last trace which is the expression map.foo
-      traces = Enum.drop(traces, 1)
-      {traces, hints} = format_traces(traces, [left, right], true)
-
-      [
-        "undefined field \"#{atom}\" ",
-        format_expr(expr, location),
-        "expected one of the following fields: ",
-        Enum.join(Enum.sort(known_atoms), ", "),
-        "\n\n",
-        traces,
-        format_message_hints(hints),
-        "Conflict found at"
-      ]
-    else
-      simplify_left? = simplify_type?(left, right)
-      simplify_right? = simplify_type?(right, left)
-
-      {traces, hints} = format_traces(traces, [left, right], simplify_left? or simplify_right?)
-
-      [
-        "incompatible types:\n\n    ",
-        Unify.format_type(left, simplify_left?),
-        " !~ ",
-        Unify.format_type(right, simplify_right?),
-        "\n\n",
-        format_expr(expr, location),
-        traces,
-        format_message_hints(hints),
-        "Conflict found at"
-      ]
-    end
-  end
-
-  defp missing_field(
-         {:map, [{:required, {:atom, atom} = type, _}, {:optional, :dynamic, :dynamic}]},
-         {:map, fields}
-       ) do
-    matched_missing_field(fields, type, atom)
-  end
-
-  defp missing_field(
-         {:map, fields},
-         {:map, [{:required, {:atom, atom} = type, _}, {:optional, :dynamic, :dynamic}]}
-       ) do
-    matched_missing_field(fields, type, atom)
-  end
-
-  defp missing_field(_, _), do: :error
-
-  defp matched_missing_field(fields, type, atom) do
-    if List.keymember?(fields, type, 1) do
-      :error
-    else
-      known_atoms = for {_, {:atom, atom}, _} <- fields, do: atom
-      {:ok, atom, known_atoms}
-    end
-  end
-
-  defp format_traces([], _types, _simplify?) do
-    {[], []}
-  end
-
-  defp format_traces(traces, types, simplify?) do
-    traces
-    |> Enum.uniq()
-    |> Enum.reverse()
-    |> Enum.map_reduce([], fn
-      {:type, var, type, expr, location}, hints ->
-        {hint, hints} = format_type_hint(type, types, expr, hints)
-
-        trace = [
-          "where \"",
-          Macro.to_string(var),
-          "\" was given the type ",
-          Unify.format_type(type, simplify?),
-          hint,
-          " in:\n\n    # ",
-          format_location(location),
-          "    ",
-          indent(expr_to_string(expr)),
-          "\n\n"
-        ]
-
-        {trace, hints}
-
-      {:var, var1, var2, expr, location}, hints ->
-        trace = [
-          "where \"",
-          Macro.to_string(var1),
-          "\" was given the same type as \"",
-          Macro.to_string(var2),
-          "\" in:\n\n    # ",
-          format_location(location),
-          "    ",
-          indent(expr_to_string(expr)),
-          "\n\n"
-        ]
-
-        {trace, hints}
-    end)
-  end
-
-  defp format_location({file, position, _mfa}) do
-    format_location({file, position[:line]})
-  end
-
-  defp format_location({file, line}) do
-    file = Path.relative_to_cwd(file)
-    line = if line, do: [Integer.to_string(line)], else: []
-    [file, ?:, line, ?\n]
-  end
-
-  defp simplify_type?(type, other) do
-    map_like_type?(type) and not map_like_type?(other)
-  end
-
-  ## EXPRESSION FORMATTING
-
-  defp format_expr(nil, _location) do
-    []
-  end
-
-  defp format_expr(expr, location) do
-    [
-      "in expression:\n\n    # ",
-      format_location(location),
-      "    ",
-      indent(expr_to_string(expr)),
-      "\n\n"
-    ]
-  end
-
-  @doc false
-  def expr_to_string(expr) do
-    expr
-    |> reverse_rewrite()
-    |> Macro.to_string()
-  end
-
-  defp reverse_rewrite(guard) do
-    Macro.prewalk(guard, fn
-      {{:., _, [mod, fun]}, meta, args} -> erl_to_ex(mod, fun, args, meta)
-      other -> other
-    end)
-  end
-
-  defp erl_to_ex(mod, fun, args, meta) do
-    case :elixir_rewrite.erl_to_ex(mod, fun, args) do
-      {Kernel, fun, args} -> {fun, meta, args}
-      {mod, fun, args} -> {{:., [], [mod, fun]}, meta, args}
-    end
-  end
-
-  ## Hints
-
-  defp format_message_hints(hints) do
-    hints
-    |> Enum.uniq()
-    |> Enum.reverse()
-    |> Enum.map(&[format_message_hint(&1), "\n"])
-  end
-
-  defp format_message_hint(:inferred_dot) do
-    """
-    #{hint()} "var.field" (without parentheses) implies "var" is a map() while \
-    "var.fun()" (with parentheses) implies "var" is an atom()
-    """
-  end
-
-  defp format_message_hint(:inferred_bitstring_spec) do
-    """
-    #{hint()} all expressions given to binaries are assumed to be of type \
-    integer() unless said otherwise. For example, <<expr>> assumes "expr" \
-    is an integer. Pass a modifier, such as <<expr::float>> or <<expr::binary>>, \
-    to change the default behaviour.
-    """
-  end
-
-  defp format_message_hint({:sized_and_unsize_tuples, {size, var}}) do
-    """
-    #{hint()} use pattern matching or "is_tuple(#{Macro.to_string(var)}) and \
-    tuple_size(#{Macro.to_string(var)}) == #{size}" to guard a sized tuple.
-    """
-  end
-
-  defp hint, do: :elixir_errors.prefix(:hint)
-
-  defp format_type_hint(type, types, expr, hints) do
-    case format_type_hint(type, types, expr) do
-      {message, hint} -> {message, [hint | hints]}
-      :error -> {[], hints}
-    end
-  end
-
-  defp format_type_hint(type, types, expr) do
-    cond do
-      dynamic_map_dot?(type, expr) ->
-        {" (due to calling var.field)", :inferred_dot}
-
-      dynamic_remote_call?(type, expr) ->
-        {" (due to calling var.fun())", :inferred_dot}
-
-      inferred_bitstring_spec?(type, expr) ->
-        {[], :inferred_bitstring_spec}
-
-      message = sized_and_unsize_tuples(expr, types) ->
-        {[], {:sized_and_unsize_tuples, message}}
-
-      true ->
-        :error
-    end
-  end
-
-  defp dynamic_map_dot?(type, expr) do
-    with true <- map_type?(type),
-         {{:., _meta1, [_map, _field]}, meta2, []} <- expr,
-         true <- Keyword.get(meta2, :no_parens, false) do
-      true
-    else
-      _ -> false
-    end
-  end
-
-  defp dynamic_remote_call?(type, expr) do
-    with true <- atom_type?(type),
-         {{:., _meta1, [_module, _field]}, meta2, []} <- expr,
-         false <- Keyword.get(meta2, :no_parens, false) do
-      true
-    else
-      _ -> false
-    end
-  end
-
-  defp inferred_bitstring_spec?(type, expr) do
-    with true <- integer_type?(type),
-         {:<<>>, _, args} <- expr,
-         true <- Enum.any?(args, &match?({:"::", [{:inferred_bitstring_spec, true} | _], _}, &1)) do
-      true
-    else
-      _ -> false
-    end
-  end
-
-  defp sized_and_unsize_tuples({{:., _, [:erlang, :is_tuple]}, _, [var]}, types) do
-    case Enum.find(types, &match?({:tuple, _, _}, &1)) do
-      {:tuple, size, _} ->
-        {size, var}
-
-      nil ->
-        nil
-    end
-  end
-
-  defp sized_and_unsize_tuples(_expr, _types) do
-    nil
-  end
-
-  ## Formatting helpers
-
-  defp indent(string) do
-    String.replace(string, "\n", "\n    ")
-  end
-
-  defp map_type?({:map, _}), do: true
-  defp map_type?(_other), do: false
-
-  defp map_like_type?({:map, _}), do: true
-  defp map_like_type?({:union, union}), do: Enum.any?(union, &map_like_type?/1)
-  defp map_like_type?(_other), do: false
-
-  defp atom_type?(:atom), do: true
-  defp atom_type?({:atom, _}), do: false
-  defp atom_type?({:union, union}), do: Enum.all?(union, &atom_type?/1)
-  defp atom_type?(_other), do: false
-
-  defp integer_type?(:integer), do: true
-  defp integer_type?(_other), do: false
-
-  defp call_to_mfa({{:., _, [mod, fun]}, _, args}), do: {mod, fun, length(args)}
-  defp call_to_mfa({fun, _, args}) when is_atom(fun), do: {Kernel, fun, length(args)}
 end

lib/elixir/lib/module/types/expr.ex

@@ -2,312 +2,257 @@ defmodule Module.Types.Expr do
   @moduledoc false
 
   alias Module.Types.{Of, Pattern}
-  import Module.Types.{Helpers, Unify}
+  import Module.Types.{Helpers, Descr}
 
-  def of_expr(expr, expected, %{context: stack_context} = stack, context)
-      when stack_context != :expr do
-    of_expr(expr, expected, %{stack | context: :expr}, context)
+  14 = length(Macro.Env.__info__(:struct))
+
+  @caller closed_map(
+            __struct__: atom([Macro.Env]),
+            aliases: list(),
+            context: atom([:match, :guard, nil]),
+            context_modules: list(),
+            file: binary(),
+            function: union(tuple(), atom([nil])),
+            functions: list(),
+            lexical_tracker: union(pid(), atom([nil])),
+            line: integer(),
+            macro_aliases: list(),
+            macros: list(),
+            module: atom(),
+            requires: list(),
+            tracers: list(),
+            versioned_vars: open_map()
+          )
+
+  @atom_true atom([true])
+  @exception open_map(__struct__: atom(), __exception__: @atom_true)
+
+  defp of_expr(expr, expected_expr, stack, context) do
+    with {:ok, actual, context} <- of_expr(expr, stack, context) do
+      Of.intersect(actual, expected_expr, stack, context)
+    end
   end
 
   # :atom
-  def of_expr(atom, _expected, _stack, context) when is_atom(atom) do
-    {:ok, {:atom, atom}, context}
+  def of_expr(atom, _stack, context) when is_atom(atom) do
+    {:ok, atom([atom]), context}
   end
 
   # 12
-  def of_expr(literal, _expected, _stack, context) when is_integer(literal) do
-    {:ok, :integer, context}
+  def of_expr(literal, _stack, context) when is_integer(literal) do
+    {:ok, integer(), context}
   end
 
   # 1.2
-  def of_expr(literal, _expected, _stack, context) when is_float(literal) do
-    {:ok, :float, context}
+  def of_expr(literal, _stack, context) when is_float(literal) do
+    {:ok, float(), context}
   end
 
   # "..."
-  def of_expr(literal, _expected, _stack, context) when is_binary(literal) do
-    {:ok, :binary, context}
+  def of_expr(literal, _stack, context) when is_binary(literal) do
+    {:ok, binary(), context}
   end
 
   # #PID<...>
-  def of_expr(literal, _expected, _stack, context) when is_pid(literal) do
-    {:ok, :dynamic, context}
-  end
-
-  # <<...>>>
-  def of_expr({:<<>>, _meta, args}, _expected, stack, context) do
-    case Of.binary(args, stack, context, &of_expr/4) do
-      {:ok, context} -> {:ok, :binary, context}
-      {:error, reason} -> {:error, reason}
-    end
-  end
-
-  # left | []
-  def of_expr({:|, _meta, [left_expr, []]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-    of_expr(left_expr, :dynamic, stack, context)
-  end
-
-  # left | right
-  def of_expr({:|, _meta, [left_expr, right_expr]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
-    case of_expr(left_expr, :dynamic, stack, context) do
-      {:ok, left, context} ->
-        case of_expr(right_expr, :dynamic, stack, context) do
-          {:ok, {:list, right}, context} ->
-            {:ok, to_union([left, right], context), context}
-
-          {:ok, right, context} ->
-            {:ok, to_union([left, right], context), context}
-
-          {:error, reason} ->
-            {:error, reason}
-        end
-
-      {:error, reason} ->
-        {:error, reason}
-    end
+  def of_expr(literal, _stack, context) when is_pid(literal) do
+    {:ok, pid(), context}
   end
 
   # []
-  def of_expr([], _expected, _stack, context) do
-    {:ok, {:list, :dynamic}, context}
+  def of_expr([], _stack, context) do
+    {:ok, empty_list(), context}
   end
 
-  # [expr, ...]
-  def of_expr(exprs, _expected, stack, context) when is_list(exprs) do
-    stack = push_expr_stack(exprs, stack)
-
-    case map_reduce_ok(exprs, context, &of_expr(&1, :dynamic, stack, &2)) do
-      {:ok, types, context} -> {:ok, {:list, to_union(types, context)}, context}
-      {:error, reason} -> {:error, reason}
+  # TODO: [expr, ...]
+  def of_expr(exprs, stack, context) when is_list(exprs) do
+    case map_reduce_ok(exprs, context, &of_expr(&1, stack, &2)) do
+      {:ok, _types, context} -> {:ok, non_empty_list(), context}
+      {:error, context} -> {:error, context}
     end
   end
 
-  # __CALLER__
-  def of_expr({:__CALLER__, _meta, var_context}, _expected, _stack, context)
-      when is_atom(var_context) do
-    struct_pair = {:required, {:atom, :__struct__}, {:atom, Macro.Env}}
-
-    pairs =
-      Enum.map(Map.from_struct(Macro.Env.__struct__()), fn {key, _value} ->
-        {:required, {:atom, key}, :dynamic}
-      end)
-
-    {:ok, {:map, [struct_pair | pairs]}, context}
-  end
-
-  # __STACKTRACE__
-  def of_expr({:__STACKTRACE__, _meta, var_context}, _expected, _stack, context)
-      when is_atom(var_context) do
-    file = {:tuple, 2, [{:atom, :file}, {:list, :integer}]}
-    line = {:tuple, 2, [{:atom, :line}, :integer]}
-    file_line = {:list, {:union, [file, line]}}
-    type = {:list, {:tuple, 4, [:atom, :atom, :integer, file_line]}}
-    {:ok, type, context}
-  end
-
-  # var
-  def of_expr(var, _expected, _stack, context) when is_var(var) do
-    {:ok, get_var!(var, context), context}
-  end
-
   # {left, right}
-  def of_expr({left, right}, expected, stack, context) do
-    of_expr({:{}, [], [left, right]}, expected, stack, context)
+  def of_expr({left, right}, stack, context) do
+    of_expr({:{}, [], [left, right]}, stack, context)
   end
 
-  # {...}
-  def of_expr({:{}, _meta, exprs} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
-    case map_reduce_ok(exprs, context, &of_expr(&1, :dynamic, stack, &2)) do
-      {:ok, types, context} -> {:ok, {:tuple, length(types), types}, context}
-      {:error, reason} -> {:error, reason}
+  # <<...>>>
+  def of_expr({:<<>>, _meta, args}, stack, context) do
+    case Of.binary(args, :expr, stack, context, &of_expr/4) do
+      {:ok, context} -> {:ok, binary(), context}
+      # It is safe to discard errors from binary inside expressions
+      {:error, context} -> {:ok, binary(), context}
     end
   end
 
-  # left = right
-  def of_expr({:=, _meta, [left_expr, right_expr]} = expr, _expected, stack, context) do
-    # TODO: We might want to bring the expected type forward in case the type of this
-    #       pattern is not useful. For example: 1 = _ = expr
+  # TODO: left | []
+  def of_expr({:|, _meta, [left_expr, []]}, stack, context) do
+    of_expr(left_expr, stack, context)
+  end
 
-    stack = push_expr_stack(expr, stack)
+  # TODO: left | right
+  def of_expr({:|, _meta, [left_expr, right_expr]}, stack, context) do
+    case of_expr(left_expr, stack, context) do
+      {:ok, _left, context} ->
+        of_expr(right_expr, stack, context)
 
-    with {:ok, left_type, context} <-
-           Pattern.of_pattern(left_expr, stack, context),
-         {:ok, right_type, context} <- of_expr(right_expr, left_type, stack, context),
-         do: unify(right_type, left_type, stack, context)
+      {:error, context} ->
+        {:error, context}
+    end
+  end
+
+  def of_expr({:__CALLER__, _meta, var_context}, _stack, context)
+      when is_atom(var_context) do
+    {:ok, @caller, context}
+  end
+
+  # TODO: __STACKTRACE__
+  def of_expr({:__STACKTRACE__, _meta, var_context}, _stack, context)
+      when is_atom(var_context) do
+    {:ok, list(), context}
+  end
+
+  # TODO: {...}
+  def of_expr({:{}, _meta, exprs}, stack, context) do
+    case map_reduce_ok(exprs, context, &of_expr(&1, stack, &2)) do
+      {:ok, _types, context} -> {:ok, tuple(), context}
+      {:error, context} -> {:error, context}
+    end
+  end
+
+  # TODO: left = right
+  def of_expr({:=, _meta, [left_expr, right_expr]} = expr, stack, context) do
+    with {:ok, right_type, context} <- of_expr(right_expr, stack, context) do
+      Pattern.of_pattern(left_expr, {right_type, expr}, stack, context)
+    end
   end
 
   # %{map | ...}
-  def of_expr({:%{}, _, [{:|, _, [map, args]}]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-    map_type = {:map, [{:optional, :dynamic, :dynamic}]}
-
-    with {:ok, map_type, context} <- of_expr(map, map_type, stack, context),
-         {:ok, {:map, arg_pairs}, context} <- Of.closed_map(args, stack, context, &of_expr/4),
-         dynamic_value_pairs =
-           Enum.map(arg_pairs, fn {:required, key, _value} -> {:required, key, :dynamic} end),
-         args_type = {:map, dynamic_value_pairs ++ [{:optional, :dynamic, :dynamic}]},
-         {:ok, type, context} <- unify(map_type, args_type, stack, context) do
-      # Retrieve map type and overwrite with the new value types from the map update
-      case resolve_var(type, context) do
-        {:map, pairs} ->
-          updated_pairs =
-            Enum.reduce(arg_pairs, pairs, fn {:required, key, value}, pairs ->
-              List.keyreplace(pairs, key, 1, {:required, key, value})
-            end)
-
-          {:ok, {:map, updated_pairs}, context}
-
-        _ ->
-          {:ok, :dynamic, context}
-      end
+  def of_expr({:%{}, _, [{:|, _, [map, args]}]}, stack, context) do
+    with {:ok, _args_type, context} <- Of.closed_map(args, stack, context, &of_expr/3),
+         {:ok, _map_type, context} <- of_expr(map, stack, context) do
+      # TODO: intersect map with keys of terms for args
+      # TODO: Merge args_type into map_type with dynamic/static key requirement
+      {:ok, dynamic(open_map()), context}
     end
   end
 
   # %Struct{map | ...}
   def of_expr(
-        {:%, meta, [module, {:%{}, _, [{:|, _, [_, _]}]} = update]} = expr,
-        _expected,
+        {:%, struct_meta, [module, {:%{}, _, [{:|, update_meta, [map, args]}]}]} = expr,
         stack,
         context
       ) do
-    stack = push_expr_stack(expr, stack)
-    map_type = {:map, [{:optional, :dynamic, :dynamic}]}
-
-    with {:ok, struct, context} <- Of.struct(module, meta, context),
-         {:ok, update, context} <- of_expr(update, map_type, stack, context) do
-      unify(update, struct, stack, context)
+    with {:ok, args_types, context} <-
+           map_reduce_ok(args, context, fn {key, value}, context when is_atom(key) ->
+             with {:ok, type, context} <- of_expr(value, stack, context) do
+               {:ok, {key, type}, context}
+             end
+           end),
+         {:ok, struct_type, context} <-
+           Of.struct(module, args_types, :only_defaults, struct_meta, stack, context),
+         {:ok, map_type, context} <- of_expr(map, stack, context) do
+      if empty?(intersection(struct_type, map_type)) do
+        warning = {:badupdate, :struct, expr, struct_type, map_type, context}
+        {:ok, dynamic(), warn(__MODULE__, warning, update_meta, stack, context)}
+      else
+        # TODO: Merge args_type into map_type with dynamic/static key requirement
+        Of.struct(module, args_types, :merge_defaults, struct_meta, stack, context)
+      end
     end
   end
 
   # %{...}
-  def of_expr({:%{}, _meta, args} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-    Of.closed_map(args, stack, context, &of_expr/4)
+  def of_expr({:%{}, _meta, args}, stack, context) do
+    Of.closed_map(args, stack, context, &of_expr/3)
   end
 
-  # %Struct{...}
-  def of_expr({:%, meta1, [module, {:%{}, _meta2, args}]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
-    with {:ok, struct, context} <- Of.struct(module, meta1, context),
-         {:ok, map, context} <- Of.open_map(args, stack, context, &of_expr/4) do
-      unify(map, struct, stack, context)
-    end
+  # %Struct{}
+  def of_expr({:%, _, [module, {:%{}, _, args}]} = expr, stack, context) do
+    Of.struct(expr, module, args, :skip_defaults, stack, context, &of_expr/3)
   end
 
   # ()
-  def of_expr({:__block__, _meta, []}, _expected, _stack, context) do
-    {:ok, {:atom, nil}, context}
+  def of_expr({:__block__, _meta, []}, _stack, context) do
+    {:ok, atom([nil]), context}
   end
 
   # (expr; expr)
-  def of_expr({:__block__, _meta, exprs}, expected, stack, context) do
-    expected_types = List.duplicate(:dynamic, length(exprs) - 1) ++ [expected]
+  def of_expr({:__block__, _meta, exprs}, stack, context) do
+    {pre, [post]} = Enum.split(exprs, -1)
 
     result =
-      map_reduce_ok(Enum.zip(exprs, expected_types), context, fn {expr, expected}, context ->
-        of_expr(expr, expected, stack, context)
+      map_reduce_ok(pre, context, fn expr, context ->
+        of_expr(expr, stack, context)
       end)
 
     case result do
-      {:ok, expr_types, context} -> {:ok, Enum.at(expr_types, -1), context}
-      {:error, reason} -> {:error, reason}
+      {:ok, _, context} -> of_expr(post, stack, context)
+      {:error, context} -> {:error, context}
     end
   end
 
-  # cond do pat -> expr end
-  def of_expr({:cond, _meta, [[{:do, clauses}]]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
+  # TODO: cond do pat -> expr end
+  def of_expr({:cond, _meta, [[{:do, clauses}]]}, stack, context) do
     {result, context} =
-      reduce_ok(clauses, context, fn {:->, meta, [head, body]}, context = acc ->
-        case of_expr(head, :dynamic, stack, context) do
-          {:ok, _, context} ->
-            with {:ok, _expr_type, context} <- of_expr(body, :dynamic, stack, context) do
-              {:ok, keep_warnings(acc, context)}
-            end
-
-          error ->
-            # Skip the clause if it the head has an error
-            if meta[:generated], do: {:ok, acc}, else: error
-        end
+      reduce_ok(clauses, context, fn {:->, _meta, [head, body]}, context ->
+        with {:ok, _, context} <- of_expr(head, stack, context),
+             {:ok, _, context} <- of_expr(body, stack, context),
+             do: {:ok, context}
       end)
 
     case result do
-      :ok -> {:ok, :dynamic, context}
+      :ok -> {:ok, dynamic(), context}
       :error -> {:error, context}
     end
   end
 
-  # case expr do pat -> expr end
-  def of_expr({:case, _meta, [case_expr, [{:do, clauses}]]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
-    with {:ok, _expr_type, context} <- of_expr(case_expr, :dynamic, stack, context),
+  # TODO: case expr do pat -> expr end
+  def of_expr({:case, _meta, [case_expr, [{:do, clauses}]]}, stack, context) do
+    with {:ok, _expr_type, context} <- of_expr(case_expr, stack, context),
          {:ok, context} <- of_clauses(clauses, stack, context),
-         do: {:ok, :dynamic, context}
+         do: {:ok, dynamic(), context}
   end
 
-  # fn pat -> expr end
-  def of_expr({:fn, _meta, clauses} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
+  # TODO: fn pat -> expr end
+  def of_expr({:fn, _meta, clauses}, stack, context) do
     case of_clauses(clauses, stack, context) do
-      {:ok, context} -> {:ok, :dynamic, context}
-      {:error, reason} -> {:error, reason}
+      {:ok, context} -> {:ok, fun(), context}
+      {:error, context} -> {:error, context}
     end
   end
 
   @try_blocks [:do, :after]
-  @try_clause_blocks [:catch, :else, :after]
-
-  # try do expr end
-  def of_expr({:try, _meta, [blocks]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
+  @try_clause_blocks [:catch, :else]
 
+  # TODO: try do expr end
+  def of_expr({:try, _meta, [blocks]}, stack, context) do
     {result, context} =
       reduce_ok(blocks, context, fn
         {:rescue, clauses}, context ->
           reduce_ok(clauses, context, fn
-            {:->, _, [[{:in, _, [var, _exceptions]}], body]}, context = acc ->
-              {_type, context} = new_pattern_var(var, context)
+            {:->, _, [[{:in, meta, [var, exceptions]} = expr], body]}, context ->
+              of_rescue(var, exceptions, body, expr, [], meta, stack, context)
 
-              with {:ok, context} <- of_expr_context(body, :dynamic, stack, context) do
-                {:ok, keep_warnings(acc, context)}
-              end
-
-            {:->, _, [[var], body]}, context = acc ->
-              {_type, context} = new_pattern_var(var, context)
-
-              with {:ok, context} <- of_expr_context(body, :dynamic, stack, context) do
-                {:ok, keep_warnings(acc, context)}
-              end
+            {:->, meta, [[var], body]}, context ->
+              of_rescue(var, [], body, var, [:anonymous_rescue], meta, stack, context)
           end)
 
-        {block, body}, context = acc when block in @try_blocks ->
-          with {:ok, context} <- of_expr_context(body, :dynamic, stack, context) do
-            {:ok, keep_warnings(acc, context)}
-          end
+        {block, body}, context when block in @try_blocks ->
+          of_expr_context(body, stack, context)
 
         {block, clauses}, context when block in @try_clause_blocks ->
           of_clauses(clauses, stack, context)
       end)
 
     case result do
-      :ok -> {:ok, :dynamic, context}
+      :ok -> {:ok, dynamic(), context}
       :error -> {:error, context}
     end
   end
 
-  # receive do pat -> expr end
-  def of_expr({:receive, _meta, [blocks]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
+  # TODO: receive do pat -> expr end
+  def of_expr({:receive, _meta, [blocks]}, stack, context) do
     {result, context} =
       reduce_ok(blocks, context, fn
         {:do, {:__block__, _, []}}, context ->
@@ -316,139 +261,171 @@ defmodule Module.Types.Expr do
         {:do, clauses}, context ->
           of_clauses(clauses, stack, context)
 
-        {:after, [{:->, _meta, [head, body]}]}, context = acc ->
-          with {:ok, _type, context} <- of_expr(head, :dynamic, stack, context),
-               {:ok, _type, context} <- of_expr(body, :dynamic, stack, context),
-               do: {:ok, keep_warnings(acc, context)}
+        {:after, [{:->, _meta, [head, body]}]}, context ->
+          with {:ok, _type, context} <- of_expr(head, stack, context),
+               {:ok, _type, context} <- of_expr(body, stack, context),
+               do: {:ok, context}
       end)
 
     case result do
-      :ok -> {:ok, :dynamic, context}
+      :ok -> {:ok, dynamic(), context}
       :error -> {:error, context}
     end
   end
 
-  # for pat <- expr do expr end
-  def of_expr({:for, _meta, [_ | _] = args} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
+  # TODO: for pat <- expr do expr end
+  def of_expr({:for, _meta, [_ | _] = args}, stack, context) do
     {clauses, [[{:do, block} | opts]]} = Enum.split(args, -1)
 
     with {:ok, context} <- reduce_ok(clauses, context, &for_clause(&1, stack, &2)),
          {:ok, context} <- reduce_ok(opts, context, &for_option(&1, stack, &2)) do
       if Keyword.has_key?(opts, :reduce) do
         with {:ok, context} <- of_clauses(block, stack, context) do
-          {:ok, :dynamic, context}
+          {:ok, dynamic(), context}
         end
       else
-        with {:ok, _type, context} <- of_expr(block, :dynamic, stack, context) do
-          {:ok, :dynamic, context}
+        with {:ok, _type, context} <- of_expr(block, stack, context) do
+          {:ok, dynamic(), context}
         end
       end
     end
   end
 
-  # with pat <- expr do expr end
-  def of_expr({:with, _meta, [_ | _] = clauses} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
+  # TODO: with pat <- expr do expr end
+  def of_expr({:with, _meta, [_ | _] = clauses}, stack, context) do
     case reduce_ok(clauses, context, &with_clause(&1, stack, &2)) do
-      {:ok, context} -> {:ok, :dynamic, context}
-      {:error, reason} -> {:error, reason}
+      {:ok, context} -> {:ok, dynamic(), context}
+      {:error, context} -> {:error, context}
     end
   end
 
-  # fun.(args)
-  def of_expr({{:., _meta1, [fun]}, _meta2, args} = expr, _expected, stack, context) do
-    # TODO: Use expected type to infer intersection return type
-    stack = push_expr_stack(expr, stack)
+  # TODO: fun.(args)
+  def of_expr({{:., _meta1, [fun]}, _meta2, args}, stack, context) do
+    with {:ok, fun_type, context} <- of_expr(fun, stack, context),
+         {:ok, _args_types, context} <-
+           map_reduce_ok(args, context, &of_expr(&1, stack, &2)) do
+      context =
+        case fun_fetch(fun_type, length(args)) do
+          :ok -> context
+          :error -> Of.incompatible_warn(fun, fun(), fun_type, stack, context)
+        end
 
-    with {:ok, _fun_type, context} <- of_expr(fun, :dynamic, stack, context),
-         {:ok, _arg_types, context} <-
-           map_reduce_ok(args, context, &of_expr(&1, :dynamic, stack, &2)) do
-      {:ok, :dynamic, context}
+      {:ok, dynamic(), context}
     end
   end
 
-  # expr.key_or_fun
-  def of_expr({{:., _meta1, [expr1, key_or_fun]}, meta2, []} = expr2, _expected, stack, context)
-      when not is_atom(expr1) do
-    stack = push_expr_stack(expr2, stack)
-
-    if Keyword.get(meta2, :no_parens, false) do
-      with {:ok, expr_type, context} <- of_expr(expr1, :dynamic, stack, context),
-           {value_var, context} = add_var(context),
-           pair_type = {:required, {:atom, key_or_fun}, value_var},
-           optional_type = {:optional, :dynamic, :dynamic},
-           map_field_type = {:map, [pair_type, optional_type]},
-           {:ok, _map_type, context} <- unify(map_field_type, expr_type, stack, context),
-           do: {:ok, value_var, context}
-    else
-      # TODO: Use expected type to infer intersection return type
-      with {:ok, expr_type, context} <- of_expr(expr1, :dynamic, stack, context),
-           {:ok, _map_type, context} <- unify(expr_type, :atom, stack, context),
-           do: {:ok, :dynamic, context}
+  def of_expr({{:., _, [callee, key_or_fun]}, meta, []} = expr, stack, context)
+      when not is_atom(callee) and is_atom(key_or_fun) do
+    with {:ok, type, context} <- of_expr(callee, stack, context) do
+      if Keyword.get(meta, :no_parens, false) do
+        Of.map_fetch(expr, type, key_or_fun, stack, context)
+      else
+        {mods, context} = Of.remote(type, key_or_fun, 0, [:dot], expr, meta, stack, context)
+        apply_many(mods, key_or_fun, [], expr, stack, context)
+      end
     end
   end
 
-  # expr.fun(arg)
-  def of_expr({{:., _meta1, [expr1, fun]}, meta2, args} = expr2, _expected, stack, context) do
-    # TODO: Use expected type to infer intersection return type
-
-    context = Of.remote(expr1, fun, length(args), meta2, context)
-    stack = push_expr_stack(expr2, stack)
-
-    with {:ok, _expr_type, context} <- of_expr(expr1, :dynamic, stack, context),
-         {:ok, _fun_type, context} <- of_expr(fun, :dynamic, stack, context),
-         {:ok, _arg_types, context} <-
-           map_reduce_ok(args, context, &of_expr(&1, :dynamic, stack, &2)) do
-      {:ok, :dynamic, context}
+  # TODO: expr.fun(arg)
+  def of_expr({{:., _, [remote, name]}, meta, args} = expr, stack, context) do
+    with {:ok, remote_type, context} <- of_expr(remote, stack, context),
+         {:ok, args_types, context} <- map_reduce_ok(args, context, &of_expr(&1, stack, &2)) do
+      {mods, context} = Of.remote(remote_type, name, length(args), expr, meta, stack, context)
+      apply_many(mods, name, args_types, expr, stack, context)
     end
   end
 
-  # &Foo.bar/1
+  # TODO: &Foo.bar/1
   def of_expr(
-        {:&, _, [{:/, _, [{{:., _, [module, fun]}, meta, []}, arity]}]},
-        _expected,
-        _stack,
+        {:&, _, [{:/, _, [{{:., _, [remote, name]}, meta, []}, arity]}]} = expr,
+        stack,
         context
       )
-      when is_atom(module) and is_atom(fun) do
-    context = Of.remote(module, fun, arity, meta, context)
-    {:ok, :dynamic, context}
+      when is_atom(name) and is_integer(arity) do
+    with {:ok, remote_type, context} <- of_expr(remote, stack, context) do
+      # TODO: We cannot return the unions of functions. Do we forbid this?
+      # Do we check it is always the same return type? Do we simply say it is a function?
+      {_mods, context} = Of.remote(remote_type, name, arity, expr, meta, stack, context)
+      {:ok, fun(), context}
+    end
   end
 
   # &foo/1
-  # & &1
-  def of_expr({:&, _meta, _arg}, _expected, _stack, context) do
-    # TODO: Function type
-    {:ok, :dynamic, context}
+  # TODO: & &1
+  def of_expr({:&, _meta, _arg}, _stack, context) do
+    {:ok, fun(), context}
   end
 
-  # fun(arg)
-  def of_expr({fun, _meta, args} = expr, _expected, stack, context)
+  # TODO: call(arg)
+  def of_expr({fun, _meta, args}, stack, context)
       when is_atom(fun) and is_list(args) do
-    # TODO: Use expected type to infer intersection return type
-
-    stack = push_expr_stack(expr, stack)
-
-    case map_reduce_ok(args, context, &of_expr(&1, :dynamic, stack, &2)) do
-      {:ok, _arg_types, context} -> {:ok, :dynamic, context}
-      {:error, reason} -> {:error, reason}
+    with {:ok, _arg_types, context} <-
+           map_reduce_ok(args, context, &of_expr(&1, stack, &2)) do
+      {:ok, dynamic(), context}
     end
   end
 
-  defp for_clause({:<-, _, [left, expr]}, stack, context) do
+  # var
+  def of_expr(var, _stack, context) when is_var(var) do
+    {:ok, Of.var(var, context), context}
+  end
+
+  ## Try
+
+  defp of_rescue(var, exceptions, body, expr, hints, meta, stack, context) do
+    args = [__exception__: @atom_true]
+
+    with {:ok, structs, context} <-
+           map_reduce_ok(exceptions, context, fn exception, context ->
+             # Exceptions are not validated in the compiler,
+             # to avoid export dependencies. So we do it here.
+             if Code.ensure_loaded?(exception) and function_exported?(exception, :__struct__, 0) do
+               Of.struct(exception, args, :merge_defaults, meta, stack, context)
+             else
+               # Whenever there is a failure (such as undefined function),
+               # we return dynamic to avoid cascading errors. In this case,
+               # we can return something a bit more precise than dynamic,
+               # but we still want an open map to avoid cascading.
+               context = Of.remote(exception, :__struct__, 0, meta, stack, context)
+               {:ok, dynamic(open_map([__struct__: atom([exception])] ++ args)), context}
+             end
+           end) do
+      context =
+        case var do
+          {:_, _, _} ->
+            context
+
+          _ ->
+            expected = if structs == [], do: @exception, else: Enum.reduce(structs, &union/2)
+
+            formatter = fn expr ->
+              {"rescue #{expr_to_string(expr)} ->", hints}
+            end
+
+            {:ok, _type, context} = Of.refine_var(var, expected, expr, formatter, stack, context)
+            context
+        end
+
+      of_expr_context(body, stack, context)
+    end
+  end
+
+  ## Comprehensions
+
+  defp for_clause({:<-, meta, [left, expr]}, stack, context) do
     {pattern, guards} = extract_head([left])
 
-    with {:ok, _pattern_type, context} <- Pattern.of_head([pattern], guards, stack, context),
-         {:ok, _expr_type, context} <- of_expr(expr, :dynamic, stack, context),
+    with {:ok, _pattern_type, context} <-
+           Pattern.of_head([pattern], guards, meta, stack, context),
+         {:ok, _expr_type, context} <- of_expr(expr, stack, context),
          do: {:ok, context}
   end
 
   defp for_clause({:<<>>, _, [{:<-, _, [pattern, expr]}]}, stack, context) do
     # TODO: the compiler guarantees pattern is a binary but we need to check expr is a binary
-    with {:ok, _pattern_type, context} <- Pattern.of_pattern(pattern, stack, context),
-         {:ok, _expr_type, context} <- of_expr(expr, :dynamic, stack, context),
+    with {:ok, _pattern_type, context} <-
+           Pattern.of_pattern(pattern, stack, context),
+         {:ok, _expr_type, context} <- of_expr(expr, stack, context),
          do: {:ok, context}
   end
 
@@ -457,26 +434,29 @@ defmodule Module.Types.Expr do
   end
 
   defp for_clause(expr, stack, context) do
-    of_expr_context(expr, :dynamic, stack, context)
+    of_expr_context(expr, stack, context)
   end
 
   defp for_option({:into, expr}, stack, context) do
-    of_expr_context(expr, :dynamic, stack, context)
+    of_expr_context(expr, stack, context)
   end
 
   defp for_option({:reduce, expr}, stack, context) do
-    of_expr_context(expr, :dynamic, stack, context)
+    of_expr_context(expr, stack, context)
   end
 
   defp for_option({:uniq, _}, _stack, context) do
     {:ok, context}
   end
 
-  defp with_clause({:<-, _, [left, expr]}, stack, context) do
+  ## With
+
+  defp with_clause({:<-, meta, [left, expr]}, stack, context) do
     {pattern, guards} = extract_head([left])
 
-    with {:ok, _pattern_type, context} <- Pattern.of_head([pattern], guards, stack, context),
-         {:ok, _expr_type, context} <- of_expr(expr, :dynamic, stack, context),
+    with {:ok, _pattern_type, context} <-
+           Pattern.of_head([pattern], guards, meta, stack, context),
+         {:ok, _expr_type, context} <- of_expr(expr, stack, context),
          do: {:ok, context}
   end
 
@@ -485,36 +465,44 @@ defmodule Module.Types.Expr do
   end
 
   defp with_clause(expr, stack, context) do
-    of_expr_context(expr, :dynamic, stack, context)
+    of_expr_context(expr, stack, context)
   end
 
   defp with_option({:do, body}, stack, context) do
-    of_expr_context(body, :dynamic, stack, context)
+    of_expr_context(body, stack, context)
   end
 
   defp with_option({:else, clauses}, stack, context) do
     of_clauses(clauses, stack, context)
   end
 
-  defp of_clauses(clauses, stack, context) do
-    reduce_ok(clauses, context, fn {:->, meta, [head, body]}, context = acc ->
-      {patterns, guards} = extract_head(head)
+  ## General helpers
 
-      case Pattern.of_head(patterns, guards, stack, context) do
-        {:ok, _, context} ->
-          with {:ok, _expr_type, context} <- of_expr(body, :dynamic, stack, context) do
-            {:ok, keep_warnings(acc, context)}
-          end
-
-        error ->
-          # Skip the clause if it the head has an error
-          if meta[:generated], do: {:ok, acc}, else: error
-      end
-    end)
+  defp apply_many([], _function, _args_types, _expr, _stack, context) do
+    {:ok, dynamic(), context}
   end
 
-  defp keep_warnings(context, %{warnings: warnings}) do
-    %{context | warnings: warnings}
+  defp apply_many([mod], function, args_types, expr, stack, context) do
+    Of.apply(mod, function, args_types, expr, stack, context)
+  end
+
+  defp apply_many(mods, function, args_types, expr, stack, context) do
+    with {:ok, returns, context} <-
+           map_reduce_ok(mods, context, fn mod, context ->
+             Of.apply(mod, function, args_types, expr, stack, context)
+           end) do
+      {:ok, Enum.reduce(returns, &union/2), context}
+    end
+  end
+
+  defp of_clauses(clauses, stack, context) do
+    reduce_ok(clauses, context, fn {:->, meta, [head, body]}, context ->
+      {patterns, guards} = extract_head(head)
+
+      with {:ok, _, context} <- Pattern.of_head(patterns, guards, meta, stack, context),
+           {:ok, _, context} <- of_expr(body, stack, context),
+           do: {:ok, context}
+    end)
   end
 
   defp extract_head([{:when, _meta, args}]) do
@@ -536,18 +524,37 @@ defmodule Module.Types.Expr do
     [other]
   end
 
-  defp of_expr_context(expr, expected, stack, context) do
-    case of_expr(expr, expected, stack, context) do
+  defp of_expr_context(expr, stack, context) do
+    case of_expr(expr, stack, context) do
       {:ok, _type, context} -> {:ok, context}
-      {:error, reason} -> {:error, reason}
+      {:error, context} -> {:error, context}
     end
   end
 
-  defp new_pattern_var({:_, _meta, var_context}, context) when is_atom(var_context) do
-    {:dynamic, context}
-  end
+  ## Warning formatting
 
-  defp new_pattern_var(var, context) do
-    new_var(var, context)
+  def format_diagnostic({:badupdate, type, expr, expected_type, actual_type, context}) do
+    traces = Of.collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          incompatible types in #{type} update:
+
+              #{expr_to_string(expr) |> indent(4)}
+
+          expected type:
+
+              #{to_quoted_string(expected_type) |> indent(4)}
+
+          but got type:
+
+              #{to_quoted_string(actual_type) |> indent(4)}
+          """,
+          Of.format_traces(traces)
+        ])
+    }
   end
 end

lib/elixir/lib/module/types/helpers.ex

@@ -15,9 +15,64 @@ defmodule Module.Types.Helpers do
   end
 
   @doc """
-  Returns unique identifier for the current assignment of the variable.
+  Formatted hints in typing errors.
   """
-  def var_name({_name, meta, _context}), do: Keyword.fetch!(meta, :version)
+  def format_hints(hints) do
+    hints
+    |> Enum.uniq()
+    |> Enum.map(fn
+      :inferred_bitstring_spec ->
+        """
+
+        #{hint()} all expressions given to binaries are assumed to be of type \
+        integer() unless said otherwise. For example, <<expr>> assumes "expr" \
+        is an integer. Pass a modifier, such as <<expr::float>> or <<expr::binary>>, \
+        to change the default behavior.
+        """
+
+      :dot ->
+        """
+
+        #{hint()} "var.field" (without parentheses) means "var" is a map() while \
+        "var.fun()" (with parentheses) means "var" is an atom()
+        """
+
+      :anonymous_rescue ->
+        """
+
+        #{hint()} when you rescue without specifying exception names, \
+        the variable is assigned a type of a struct but all of its fields are unknown. \
+        If you are trying to access an exception's :message key, either specify the \
+        exception names or use `Exception.message/1`.
+        """
+    end)
+  end
+
+  defp hint, do: :elixir_errors.prefix(:hint)
+
+  @doc """
+  Converts the given expression to a string,
+  translating inlined Erlang calls back to Elixir.
+  """
+  def expr_to_string(expr) do
+    expr
+    |> reverse_rewrite()
+    |> Macro.to_string()
+  end
+
+  defp reverse_rewrite(guard) do
+    Macro.prewalk(guard, fn
+      {{:., _, [mod, fun]}, meta, args} -> erl_to_ex(mod, fun, args, meta)
+      other -> other
+    end)
+  end
+
+  defp erl_to_ex(mod, fun, args, meta) do
+    case :elixir_rewrite.erl_to_ex(mod, fun, args) do
+      {Kernel, fun, args} -> {fun, meta, args}
+      {mod, fun, args} -> {{:., [], [mod, fun]}, meta, args}
+    end
+  end
 
   @doc """
   Returns the AST metadata.
@@ -25,6 +80,22 @@ defmodule Module.Types.Helpers do
   def get_meta({_, meta, _}), do: meta
   def get_meta(_other), do: []
 
+  @doc """
+  Indents new lines.
+  """
+  def indent(content, count) do
+    String.replace(content, "\n", "\n" <> String.duplicate(" ", count))
+  end
+
+  @doc """
+  Emits a warnings.
+  """
+  def warn(module, warning, meta, stack, context) do
+    {fun, arity} = stack.function
+    location = {stack.file, meta, {stack.module, fun, arity}}
+    %{context | warnings: [{module, warning, location} | context.warnings]}
+  end
+
   @doc """
   Like `Enum.reduce/3` but only continues while `fun` returns `{:ok, acc}`
   and stops on `{:error, reason}`.
@@ -45,42 +116,6 @@ defmodule Module.Types.Helpers do
 
   defp do_reduce_ok([], acc, _fun), do: {:ok, acc}
 
-  @doc """
-  Like `Enum.unzip/1` but only continues while `fun` returns `{:ok, elem1, elem2}`
-  and stops on `{:error, reason}`.
-  """
-  def unzip_ok(list) do
-    do_unzip_ok(list, [], [])
-  end
-
-  defp do_unzip_ok([{:ok, head1, head2} | tail], acc1, acc2) do
-    do_unzip_ok(tail, [head1 | acc1], [head2 | acc2])
-  end
-
-  defp do_unzip_ok([{:error, reason} | _tail], _acc1, _acc2), do: {:error, reason}
-
-  defp do_unzip_ok([], acc1, acc2), do: {:ok, Enum.reverse(acc1), Enum.reverse(acc2)}
-
-  @doc """
-  Like `Enum.map/2` but only continues while `fun` returns `{:ok, elem}`
-  and stops on `{:error, reason}`.
-  """
-  def map_ok(list, fun) do
-    do_map_ok(list, [], fun)
-  end
-
-  defp do_map_ok([head | tail], acc, fun) do
-    case fun.(head) do
-      {:ok, elem} ->
-        do_map_ok(tail, [elem | acc], fun)
-
-      {:error, reason} ->
-        {:error, reason}
-    end
-  end
-
-  defp do_map_ok([], acc, _fun), do: {:ok, Enum.reverse(acc)}
-
   @doc """
   Like `Enum.map_reduce/3` but only continues while `fun` returns `{:ok, elem, acc}`
   and stops on `{:error, reason}`.
@@ -100,97 +135,4 @@ defmodule Module.Types.Helpers do
   end
 
   defp do_map_reduce_ok([], {list, acc}, _fun), do: {:ok, Enum.reverse(list), acc}
-
-  @doc """
-  Like `Enum.flat_map/2` but only continues while `fun` returns `{:ok, list}`
-  and stops on `{:error, reason}`.
-  """
-  def flat_map_ok(list, fun) do
-    do_flat_map_ok(list, [], fun)
-  end
-
-  defp do_flat_map_ok([head | tail], acc, fun) do
-    case fun.(head) do
-      {:ok, elem} ->
-        do_flat_map_ok(tail, [elem | acc], fun)
-
-      {:error, reason} ->
-        {:error, reason}
-    end
-  end
-
-  defp do_flat_map_ok([], acc, _fun), do: {:ok, Enum.reverse(Enum.concat(acc))}
-
-  @doc """
-  Like `Enum.flat_map_reduce/3` but only continues while `fun` returns `{:ok, list, acc}`
-  and stops on `{:error, reason}`.
-  """
-  def flat_map_reduce_ok(list, acc, fun) do
-    do_flat_map_reduce_ok(list, {[], acc}, fun)
-  end
-
-  defp do_flat_map_reduce_ok([head | tail], {list, acc}, fun) do
-    case fun.(head, acc) do
-      {:ok, elems, acc} ->
-        do_flat_map_reduce_ok(tail, {[elems | list], acc}, fun)
-
-      {:error, reason} ->
-        {:error, reason}
-    end
-  end
-
-  defp do_flat_map_reduce_ok([], {list, acc}, _fun),
-    do: {:ok, Enum.reverse(Enum.concat(list)), acc}
-
-  @doc """
-  Given a list of `[{:ok, term()} | {:error, term()}]` it returns a list of
-  errors `{:error, [term()]}` in case of at least one error or `{:ok, [term()]}`
-  if there are no errors.
-  """
-  def oks_or_errors(list) do
-    case Enum.split_with(list, &match?({:ok, _}, &1)) do
-      {oks, []} -> {:ok, Enum.map(oks, fn {:ok, ok} -> ok end)}
-      {_oks, errors} -> {:error, Enum.map(errors, fn {:error, error} -> error end)}
-    end
-  end
-
-  @doc """
-  Combines a list of guard expressions `when x when y when z` to an expression
-  combined with `or`, `x or y or z`.
-  """
-  # TODO: Remove this and let multiple when be treated as multiple clauses,
-  #       meaning they will be intersection types
-  def guards_to_or([]) do
-    []
-  end
-
-  def guards_to_or(guards) do
-    Enum.reduce(guards, fn guard, acc -> {{:., [], [:erlang, :orelse]}, [], [guard, acc]} end)
-  end
-
-  @doc """
-  Like `Enum.zip/1` but will zip multiple lists together instead of only two.
-  """
-  def zip_many(lists) do
-    zip_many(lists, [], [[]])
-  end
-
-  defp zip_many([], [], [[] | acc]) do
-    map_reverse(acc, [], &Enum.reverse/1)
-  end
-
-  defp zip_many([], remain, [last | acc]) do
-    zip_many(Enum.reverse(remain), [], [[] | [last | acc]])
-  end
-
-  defp zip_many([[] | _], remain, [last | acc]) do
-    zip_many(Enum.reverse(remain), [], [last | acc])
-  end
-
-  defp zip_many([[elem | list1] | list2], remain, [last | acc]) do
-    zip_many(list2, [list1 | remain], [[elem | last] | acc])
-  end
-
-  defp map_reverse([], acc, _fun), do: acc
-  defp map_reverse([head | tail], acc, fun), do: map_reverse(tail, [fun.(head) | acc], fun)
 end

lib/elixir/lib/module/types/of.ex

@@ -3,136 +3,194 @@ defmodule Module.Types.Of do
   # Generic AST and Enum helpers go to Module.Types.Helpers.
   @moduledoc false
 
+  alias Module.ParallelChecker
+  import Module.Types.{Helpers, Descr}
+
   @prefix quote(do: ...)
   @suffix quote(do: ...)
 
-  alias Module.ParallelChecker
+  @integer_or_float union(integer(), float())
+  @integer_or_binary union(integer(), binary())
+  @integer integer()
+  @float float()
+  @binary binary()
 
-  import Module.Types.Helpers
-  import Module.Types.Unify
-
-  # There are important assumptions on how we work with maps.
-  #
-  # First, the keys in the map must be ordered by subtyping.
-  #
-  # Second, optional keys must be a superset of the required
-  # keys, i.e. %{required(atom) => integer, optional(:foo) => :bar}
-  # is forbidden.
-  #
-  # Third, in order to preserve co/contra-variance, a supertype
-  # must satisfy its subtypes. I.e. %{foo: :bar, atom() => :baz}
-  # is forbidden, it must be %{foo: :bar, atom() => :baz | :bar}.
-  #
-  # Once we support user declared maps, we need to validate these
-  # assumptions.
+  ## Variables
 
   @doc """
-  Handles open maps (with dynamic => dynamic).
+  Fetches the type of a defined variable.
   """
-  def open_map(args, stack, context, of_fun) do
-    with {:ok, pairs, context} <- map_pairs(args, stack, context, of_fun) do
-      # If we match on a map such as %{"foo" => "bar"}, we cannot
-      # assert that %{binary() => binary()}, since we are matching
-      # only a single binary of infinite possible values. Therefore,
-      # the correct would be to match it to %{binary() => binary() | var}.
-      #
-      # We can skip this in two cases:
-      #
-      #   1. If the key is a singleton, then we know that it has no
-      #      other value than the current one
-      #
-      #   2. If the value is a variable, then there is no benefit in
-      #      creating another variable, so we can skip it
-      #
-      # For now, we skip generating the var itself and introduce
-      # :dynamic instead.
-      pairs =
-        for {key, value} <- pairs, not has_unbound_var?(key, context) do
-          if singleton?(key, context) or match?({:var, _}, value) do
-            {key, value}
-          else
-            {key, to_union([value, :dynamic], context)}
-          end
+  def var({_name, meta, _context}, context) do
+    version = Keyword.fetch!(meta, :version)
+    %{vars: %{^version => %{type: type}}} = context
+    type
+  end
+
+  @doc """
+  Refines the type of a variable.
+  """
+  def refine_var(var, type, expr, formatter \\ :default, stack, context) do
+    {var_name, meta, var_context} = var
+    version = Keyword.fetch!(meta, :version)
+
+    case context.vars do
+      %{^version => %{type: old_type, off_traces: off_traces} = data} ->
+        new_type = intersection(type, old_type)
+
+        data = %{
+          data
+          | type: new_type,
+            off_traces: new_trace(expr, type, formatter, stack, off_traces)
+        }
+
+        context = put_in(context.vars[version], data)
+
+        # We need to return error otherwise it leads to cascading errors
+        if empty?(new_type) do
+          {:error, warn({:refine_var, old_type, type, var, context}, meta, stack, context)}
+        else
+          {:ok, new_type, context}
         end
 
-      triplets = pairs_to_unions(pairs, [], context) ++ [{:optional, :dynamic, :dynamic}]
-      {:ok, {:map, triplets}, context}
+      %{} ->
+        data = %{
+          type: type,
+          name: var_name,
+          context: var_context,
+          off_traces: new_trace(expr, type, formatter, stack, [])
+        }
+
+        context = put_in(context.vars[version], data)
+        {:ok, type, context}
+    end
+  end
+
+  defp new_trace(nil, _type, _formatter, _stack, traces),
+    do: traces
+
+  defp new_trace(expr, type, formatter, stack, traces),
+    do: [{expr, stack.file, type, formatter} | traces]
+
+  ## Map/structs
+
+  @doc """
+  Handles fetching a map key.
+  """
+  def map_fetch(expr, type, field, stack, context) when is_atom(field) do
+    case map_fetch(type, field) do
+      {_optional?, value_type} ->
+        {:ok, value_type, context}
+
+      reason ->
+        {:ok, dynamic(),
+         warn({reason, expr, type, field, context}, elem(expr, 1), stack, context)}
     end
   end
 
   @doc """
-  Handles closed maps (without dynamic => dynamic).
+  Builds a closed map.
   """
-  def closed_map(args, stack, context, of_fun) do
-    with {:ok, pairs, context} <- map_pairs(args, stack, context, of_fun) do
-      {:ok, {:map, closed_to_unions(pairs, context)}, context}
+  def closed_map(pairs, extra \\ [], stack, context, of_fun) do
+    result =
+      reduce_ok(pairs, {true, extra, [], context}, fn
+        {key, value}, {closed?, single, multiple, context} ->
+          with {:ok, keys, context} <- of_finite_key_type(key, stack, context, of_fun),
+               {:ok, value_type, context} <- of_fun.(value, stack, context) do
+            case keys do
+              :none ->
+                {:ok, {false, single, multiple, context}}
+
+              [key] when multiple == [] ->
+                {:ok, {closed?, [{key, value_type} | single], multiple, context}}
+
+              keys ->
+                {:ok, {closed?, single, [{keys, value_type} | multiple], context}}
+            end
+          end
+      end)
+
+    with {:ok, {closed?, single, multiple, context}} <- result do
+      map =
+        case Enum.reverse(multiple) do
+          [] ->
+            pairs = Enum.reverse(single)
+            if closed?, do: closed_map(pairs), else: open_map(pairs)
+
+          [{keys, type} | tail] ->
+            for key <- keys, t <- cartesian_map(tail) do
+              pairs = Enum.reverse(single, [{key, type} | t])
+              if closed?, do: closed_map(pairs), else: open_map(pairs)
+            end
+            |> Enum.reduce(&union/2)
+        end
+
+      {:ok, map, context}
     end
   end
 
-  defp map_pairs(pairs, stack, context, of_fun) do
-    map_reduce_ok(pairs, context, fn {key, value}, context ->
-      with {:ok, key_type, context} <- of_fun.(key, :dynamic, stack, context),
-           {:ok, value_type, context} <- of_fun.(value, :dynamic, stack, context),
-           do: {:ok, {key_type, value_type}, context}
-    end)
+  defp of_finite_key_type(key, _stack, context, _of_fun) when is_atom(key) do
+    {:ok, [key], context}
   end
 
-  defp closed_to_unions([{key, value}], _context), do: [{:required, key, value}]
-
-  defp closed_to_unions(pairs, context) do
-    case Enum.split_with(pairs, fn {key, _value} -> has_unbound_var?(key, context) end) do
-      {[], pairs} -> pairs_to_unions(pairs, [], context)
-      {[_ | _], pairs} -> pairs_to_unions([{:dynamic, :dynamic} | pairs], [], context)
+  defp of_finite_key_type(key, stack, context, of_fun) do
+    with {:ok, key_type, context} <- of_fun.(key, stack, context) do
+      case atom_fetch(key_type) do
+        {:finite, list} -> {:ok, list, context}
+        _ -> {:ok, :none, context}
+      end
     end
   end
 
-  defp pairs_to_unions([{key, value} | ahead], behind, context) do
-    {matched_ahead, values} = find_matching_values(ahead, key, [], [])
+  defp cartesian_map(lists) do
+    case lists do
+      [] ->
+        [[]]
 
-    # In case nothing matches, use the original ahead
-    ahead = matched_ahead || ahead
-
-    all_values =
-      [value | values] ++
-        find_subtype_values(ahead, key, context) ++
-        find_subtype_values(behind, key, context)
-
-    pairs_to_unions(ahead, [{key, to_union(all_values, context)} | behind], context)
+      [{keys, type} | tail] ->
+        for key <- keys, t <- cartesian_map(tail), do: [{key, type} | t]
+    end
   end
 
-  defp pairs_to_unions([], acc, context) do
-    acc
-    |> Enum.sort(&subtype?(elem(&1, 0), elem(&2, 0), context))
-    |> Enum.map(fn {key, value} -> {:required, key, value} end)
-  end
-
-  defp find_subtype_values(pairs, key, context) do
-    for {pair_key, pair_value} <- pairs, subtype?(pair_key, key, context), do: pair_value
-  end
-
-  defp find_matching_values([{key, value} | ahead], key, acc, values) do
-    find_matching_values(ahead, key, acc, [value | values])
-  end
-
-  defp find_matching_values([{_, _} = pair | ahead], key, acc, values) do
-    find_matching_values(ahead, key, [pair | acc], values)
-  end
-
-  defp find_matching_values([], _key, acc, [_ | _] = values), do: {Enum.reverse(acc), values}
-  defp find_matching_values([], _key, _acc, []), do: {nil, []}
-
   @doc """
-  Handles structs.
+  Handles structs creation.
   """
-  def struct(struct, meta, context) do
-    context = remote(struct, :__struct__, 0, meta, context)
+  def struct({:%, meta, _}, struct, args, default_handling, stack, context, of_fun)
+      when is_atom(struct) do
+    # The compiler has already checked the keys are atoms and which ones are required.
+    with {:ok, args_types, context} <-
+           map_reduce_ok(args, context, fn {key, value}, context when is_atom(key) ->
+             with {:ok, type, context} <- of_fun.(value, stack, context) do
+               {:ok, {key, type}, context}
+             end
+           end) do
+      struct(struct, args_types, default_handling, meta, stack, context)
+    end
+  end
 
-    entries =
-      for key <- Map.keys(struct.__struct__()), key != :__struct__ do
-        {:required, {:atom, key}, :dynamic}
+  @doc """
+  Struct handling assuming the args have already been converted.
+  """
+  # TODO: Allow structs fields to be defined. If the fields are defined,
+  # then the struct is no longer dynamic. And we need to validate args
+  # against the struct types.
+  # TODO: Use the struct default values to define the default types.
+  def struct(struct, args_types, default_handling, meta, stack, context) do
+    context = remote(struct, :__struct__, 0, meta, stack, context)
+    term = term()
+
+    defaults =
+      for %{field: field} <- struct.__info__(:struct), field != :__struct__ do
+        {field, term}
       end
 
-    {:ok, {:map, [{:required, {:atom, :__struct__}, {:atom, struct}} | entries]}, context}
+    pairs =
+      case default_handling do
+        :merge_defaults -> [{:__struct__, atom([struct])} | defaults] ++ args_types
+        :skip_defaults -> [{:__struct__, atom([struct])} | args_types]
+        :only_defaults -> [{:__struct__, atom([struct])} | defaults]
+      end
+
+    {:ok, dynamic(closed_map(pairs)), context}
   end
 
   ## Binary
@@ -143,154 +201,169 @@ defmodule Module.Types.Of do
   In the stack, we add nodes such as <<expr>>, <<..., expr>>, etc,
   based on the position of the expression within the binary.
   """
-  def binary([], _stack, context, _of_fun) do
+  def binary([], _kind, _stack, context, _of_fun) do
     {:ok, context}
   end
 
-  def binary([head], stack, context, of_fun) do
-    head_stack = push_expr_stack({:<<>>, get_meta(head), [head]}, stack)
-    binary_segment(head, head_stack, context, of_fun)
+  def binary([head], kind, stack, context, of_fun) do
+    binary_segment(head, kind, [head], stack, context, of_fun)
   end
 
-  def binary([head | tail], stack, context, of_fun) do
-    head_stack = push_expr_stack({:<<>>, get_meta(head), [head, @suffix]}, stack)
-
-    case binary_segment(head, head_stack, context, of_fun) do
-      {:ok, context} -> binary_many(tail, stack, context, of_fun)
+  def binary([head | tail], kind, stack, context, of_fun) do
+    case binary_segment(head, kind, [head, @suffix], stack, context, of_fun) do
+      {:ok, context} -> binary_many(tail, kind, stack, context, of_fun)
       {:error, reason} -> {:error, reason}
     end
   end
 
-  defp binary_many([last], stack, context, of_fun) do
-    last_stack = push_expr_stack({:<<>>, get_meta(last), [@prefix, last]}, stack)
-    binary_segment(last, last_stack, context, of_fun)
+  defp binary_many([last], kind, stack, context, of_fun) do
+    binary_segment(last, kind, [@prefix, last], stack, context, of_fun)
   end
 
-  defp binary_many([head | tail], stack, context, of_fun) do
-    head_stack = push_expr_stack({:<<>>, get_meta(head), [@prefix, head, @suffix]}, stack)
-
-    case binary_segment(head, head_stack, context, of_fun) do
-      {:ok, context} -> binary_many(tail, stack, context, of_fun)
+  defp binary_many([head | tail], kind, stack, context, of_fun) do
+    case binary_segment(head, kind, [@prefix, head, @suffix], stack, context, of_fun) do
+      {:ok, context} -> binary_many(tail, kind, stack, context, of_fun)
       {:error, reason} -> {:error, reason}
     end
   end
 
-  defp binary_segment({:"::", _meta, [expr, specifiers]}, stack, context, of_fun) do
-    expected_type =
-      collect_binary_specifier(specifiers, &binary_type(stack.context, &1)) || :integer
+  # If the segment is a literal, the compiler has already checked its validity,
+  # so we just skip it.
+  defp binary_segment({:"::", _meta, [left, _right]}, _kind, _args, _stack, context, _of_fun)
+       when is_binary(left) or is_number(left) do
+    {:ok, context}
+  end
 
-    utf? = collect_binary_specifier(specifiers, &utf_type?/1)
-    float? = collect_binary_specifier(specifiers, &float_type?/1)
+  defp binary_segment({:"::", meta, [left, right]}, kind, args, stack, context, of_fun) do
+    expected_type = specifier_info(kind, right)
+    expr = {:<<>>, meta, args}
+
+    with {:ok, _type, context} <- of_fun.(left, {expected_type, expr}, stack, context) do
+      {:ok, context}
+    end
+  end
+
+  defp specifier_info(kind, {:-, _, [left, _right]}), do: specifier_info(kind, left)
+  defp specifier_info(:expr, {:float, _, _}), do: @integer_or_float
+  defp specifier_info(:expr, {:utf8, _, _}), do: @integer_or_binary
+  defp specifier_info(:expr, {:utf16, _, _}), do: @integer_or_binary
+  defp specifier_info(:expr, {:utf32, _, _}), do: @integer_or_binary
+  defp specifier_info(:pattern, {:utf8, _, _}), do: @integer
+  defp specifier_info(:pattern, {:utf16, _, _}), do: @integer
+  defp specifier_info(:pattern, {:utf32, _, _}), do: @integer
+  defp specifier_info(:pattern, {:float, _, _}), do: @float
+  defp specifier_info(_kind, {:integer, _, _}), do: @integer
+  defp specifier_info(_kind, {:bits, _, _}), do: @binary
+  defp specifier_info(_kind, {:bitstring, _, _}), do: @binary
+  defp specifier_info(_kind, {:bytes, _, _}), do: @binary
+  defp specifier_info(_kind, {:binary, _, _}), do: @binary
+  defp specifier_info(_kind, _specifier), do: @integer
+
+  ## Apply
+
+  def apply(:erlang, name, [left, right], expr, stack, context)
+      when name in [:>=, :"=<", :>, :<, :min, :max] do
+    result = if name in [:min, :max], do: union(left, right), else: boolean()
 
-    # Special case utf and float specifiers because they can be two types as literals
-    # but only a specific type as a variable in a pattern
     cond do
-      stack.context == :pattern and utf? and is_binary(expr) ->
-        {:ok, context}
+      match?({false, _}, map_fetch(left, :__struct__)) or
+          match?({false, _}, map_fetch(right, :__struct__)) ->
+        warning = {:struct_comparison, expr, context}
+        {:ok, result, warn(warning, elem(expr, 1), stack, context)}
 
-      stack.context == :pattern and float? and is_integer(expr) ->
-        {:ok, context}
+      number_type?(left) and number_type?(right) ->
+        {:ok, result, context}
+
+      empty?(intersection(left, right)) ->
+        warning = {:mismatched_comparison, expr, context}
+        {:ok, result, warn(warning, elem(expr, 1), stack, context)}
 
       true ->
-        with {:ok, type, context} <- of_fun.(expr, expected_type, stack, context),
-             {:ok, _type, context} <- unify(type, expected_type, stack, context),
-             do: {:ok, context}
+        {:ok, result, context}
     end
   end
 
-  # Collect binary type specifiers,
-  # from `<<pattern::integer-size(10)>>` collect `integer`
-  defp collect_binary_specifier({:-, _meta, [left, right]}, fun) do
-    collect_binary_specifier(left, fun) || collect_binary_specifier(right, fun)
+  def apply(mod, name, args, expr, stack, context) do
+    case :elixir_rewrite.inline(mod, name, length(args)) do
+      {mod, name} -> apply(mod, name, args, expr, stack, context)
+      false -> {:ok, dynamic(), context}
+    end
   end
 
-  defp collect_binary_specifier(other, fun) do
-    fun.(other)
-  end
-
-  defp binary_type(:expr, {:float, _, _}), do: {:union, [:integer, :float]}
-  defp binary_type(:expr, {:utf8, _, _}), do: {:union, [:integer, :binary]}
-  defp binary_type(:expr, {:utf16, _, _}), do: {:union, [:integer, :binary]}
-  defp binary_type(:expr, {:utf32, _, _}), do: {:union, [:integer, :binary]}
-  defp binary_type(:pattern, {:utf8, _, _}), do: :integer
-  defp binary_type(:pattern, {:utf16, _, _}), do: :integer
-  defp binary_type(:pattern, {:utf32, _, _}), do: :integer
-  defp binary_type(:pattern, {:float, _, _}), do: :float
-  defp binary_type(_context, {:integer, _, _}), do: :integer
-  defp binary_type(_context, {:bits, _, _}), do: :binary
-  defp binary_type(_context, {:bitstring, _, _}), do: :binary
-  defp binary_type(_context, {:bytes, _, _}), do: :binary
-  defp binary_type(_context, {:binary, _, _}), do: :binary
-  defp binary_type(_context, _specifier), do: nil
-
-  defp utf_type?({specifier, _, _}), do: specifier in [:utf8, :utf16, :utf32]
-  defp utf_type?(_), do: false
-
-  defp float_type?({:float, _, _}), do: true
-  defp float_type?(_), do: false
-
   ## Remote
 
-  @doc """
-  Handles remote calls.
-  """
-  def remote(module, fun, arity, meta, context) when is_atom(module) do
-    # TODO: In the future we may want to warn for modules defined
-    # in the local context
-    if Keyword.get(meta, :context_module, false) do
-      context
-    else
-      ParallelChecker.preload_module(context.cache, module)
-      check_export(module, fun, arity, meta, context)
+  def remote(type, fun, arity, hints \\ [], expr, meta, stack, context) do
+    case atom_fetch(type) do
+      {_, mods} ->
+        context =
+          Enum.reduce(mods, context, fn mod, context ->
+            remote(mod, fun, arity, meta, stack, context)
+          end)
+
+        {mods, context}
+
+      :error ->
+        warning = {:badmodule, expr, type, fun, arity, hints, context}
+        {[], warn(warning, meta, stack, context)}
     end
   end
 
-  def remote(_module, _fun, _arity, _meta, context), do: context
+  @doc """
+  Checks a module is a valid remote.
+  """
+  def remote(module, fun, arity, meta, stack, context) when is_atom(module) do
+    if Keyword.get(meta, :runtime_module, false) do
+      context
+    else
+      ParallelChecker.preload_module(stack.cache, module)
+      check_export(module, fun, arity, meta, stack, context)
+    end
+  end
 
-  defp check_export(module, fun, arity, meta, context) do
-    case ParallelChecker.fetch_export(context.cache, module, fun, arity) do
+  defp check_export(module, fun, arity, meta, stack, context) do
+    case ParallelChecker.fetch_export(stack.cache, module, fun, arity) do
       {:ok, mode, :def, reason} ->
-        check_deprecated(mode, module, fun, arity, reason, meta, context)
+        check_deprecated(mode, module, fun, arity, reason, meta, stack, context)
 
       {:ok, mode, :defmacro, reason} ->
-        context = warn(meta, context, {:unrequired_module, module, fun, arity})
-        check_deprecated(mode, module, fun, arity, reason, meta, context)
+        context = warn({:unrequired_module, module, fun, arity}, meta, stack, context)
+        check_deprecated(mode, module, fun, arity, reason, meta, stack, context)
 
       {:error, :module} ->
-        if warn_undefined?(module, fun, arity, context) do
-          warn(meta, context, {:undefined_module, module, fun, arity})
+        if warn_undefined?(module, fun, arity, stack) do
+          warn({:undefined_module, module, fun, arity}, meta, stack, context)
         else
           context
         end
 
       {:error, :function} ->
-        if warn_undefined?(module, fun, arity, context) do
-          exports = ParallelChecker.all_exports(context.cache, module)
-          warn(meta, context, {:undefined_function, module, fun, arity, exports})
+        if warn_undefined?(module, fun, arity, stack) do
+          exports = ParallelChecker.all_exports(stack.cache, module)
+          warn({:undefined_function, module, fun, arity, exports}, meta, stack, context)
         else
           context
         end
     end
   end
 
-  defp check_deprecated(:elixir, module, fun, arity, reason, meta, context) do
+  defp check_deprecated(:elixir, module, fun, arity, reason, meta, stack, context) do
     if reason do
-      warn(meta, context, {:deprecated, module, fun, arity, reason})
+      warn({:deprecated, module, fun, arity, reason}, meta, stack, context)
     else
       context
     end
   end
 
-  defp check_deprecated(:erlang, module, fun, arity, _reason, meta, context) do
+  defp check_deprecated(:erlang, module, fun, arity, _reason, meta, stack, context) do
     case :otp_internal.obsolete(module, fun, arity) do
       {:deprecated, string} when is_list(string) ->
         reason = string |> List.to_string() |> :string.titlecase()
-        warn(meta, context, {:deprecated, module, fun, arity, reason})
+        warn({:deprecated, module, fun, arity, reason}, meta, stack, context)
 
       {:deprecated, string, removal} when is_list(string) and is_list(removal) ->
         reason = string |> List.to_string() |> :string.titlecase()
         reason = "It will be removed in #{removal}. #{reason}"
-        warn(meta, context, {:deprecated, module, fun, arity, reason})
+        warn({:deprecated, module, fun, arity, reason}, meta, stack, context)
 
       _ ->
         context
@@ -307,59 +380,376 @@ defmodule Module.Types.Of do
   #
   # But for protocols we don't want to traverse the protocol code anyway.
   # TODO: remove this clause once we no longer traverse the protocol code.
-  defp warn_undefined?(_module, :__impl__, 1, _context), do: false
-  defp warn_undefined?(_module, :module_info, 0, _context), do: false
-  defp warn_undefined?(_module, :module_info, 1, _context), do: false
-  defp warn_undefined?(:erlang, :orelse, 2, _context), do: false
-  defp warn_undefined?(:erlang, :andalso, 2, _context), do: false
+  defp warn_undefined?(_module, :__impl__, 1, _stack), do: false
+  defp warn_undefined?(_module, :module_info, 0, _stack), do: false
+  defp warn_undefined?(_module, :module_info, 1, _stack), do: false
+  defp warn_undefined?(:erlang, :orelse, 2, _stack), do: false
+  defp warn_undefined?(:erlang, :andalso, 2, _stack), do: false
 
   defp warn_undefined?(_, _, _, %{no_warn_undefined: :all}) do
     false
   end
 
-  defp warn_undefined?(module, fun, arity, context) do
-    not Enum.any?(context.no_warn_undefined, &(&1 == module or &1 == {module, fun, arity}))
+  defp warn_undefined?(module, fun, arity, stack) do
+    not Enum.any?(stack.no_warn_undefined, &(&1 == module or &1 == {module, fun, arity}))
   end
 
-  defp warn(meta, context, warning) do
-    {fun, arity} = context.function
-    location = {context.file, meta, {context.module, fun, arity}}
-    %{context | warnings: [{__MODULE__, warning, location} | context.warnings]}
+  ## Warning helpers
+
+  @doc """
+  Intersects two types and emit an incompatible warning if empty.
+  """
+  def intersect(actual, {expected, expr}, stack, context) do
+    type = intersection(actual, expected)
+
+    if empty?(type) do
+      {:error, incompatible_warn(expr, expected, actual, stack, context)}
+    else
+      {:ok, type, context}
+    end
   end
 
+  @doc """
+  Emits incompatible types warning for the given expression.
+
+  This is a generic warning for when the expected/actual types
+  themselves may come from several different circumstances.
+  """
+  def incompatible_warn(expr, expected_type, actual_type, stack, context) do
+    meta = get_meta(expr) || stack.meta
+    hints = if meta[:inferred_bitstring_spec], do: [:inferred_bitstring_spec], else: []
+    warning = {:incompatible, expr, expected_type, actual_type, hints, context}
+    warn(warning, meta, stack, context)
+  end
+
+  defp warn(warning, meta, stack, context) do
+    warn(__MODULE__, warning, meta, stack, context)
+  end
+
+  ## Traces
+
+  def collect_traces(expr, %{vars: vars}) do
+    {_, versions} =
+      Macro.prewalk(expr, %{}, fn
+        {var_name, meta, var_context}, versions when is_atom(var_name) and is_atom(var_context) ->
+          version = meta[:version]
+
+          case vars do
+            %{^version => %{off_traces: [_ | _] = off_traces, name: name, context: context}} ->
+              {:ok,
+               Map.put(versions, version, %{
+                 type: :variable,
+                 name: name,
+                 context: context,
+                 traces: collect_var_traces(off_traces)
+               })}
+
+            _ ->
+              {:ok, versions}
+          end
+
+        node, versions ->
+          {node, versions}
+      end)
+
+    versions
+    |> Map.values()
+    |> Enum.sort_by(& &1.name)
+  end
+
+  defp collect_var_traces(traces) do
+    traces
+    |> Enum.reverse()
+    |> Enum.map(fn {expr, file, type, formatter} ->
+      meta = get_meta(expr)
+
+      {formatted_expr, formatter_hints} =
+        case formatter do
+          :default -> {expr_to_string(expr), []}
+          formatter -> formatter.(expr)
+        end
+
+      %{
+        file: file,
+        meta: meta,
+        formatted_expr: formatted_expr,
+        formatted_hints: format_hints(formatter_hints ++ expr_hints(expr)),
+        formatted_type: to_quoted_string(type)
+      }
+    end)
+  end
+
+  def format_traces(traces) do
+    Enum.map(traces, &format_trace/1)
+  end
+
+  defp format_trace(%{type: :variable, name: name, context: context, traces: traces}) do
+    traces =
+      for trace <- traces do
+        location =
+          trace.file
+          |> Path.relative_to_cwd()
+          |> Exception.format_file_line(trace.meta[:line])
+          |> String.replace_suffix(":", "")
+
+        [
+          """
+
+              # type: #{indent(trace.formatted_type, 4)}
+              # from: #{location}
+              \
+          """,
+          indent(trace.formatted_expr, 4),
+          ?\n,
+          trace.formatted_hints
+        ]
+      end
+
+    type_or_types = pluralize(traces, "type", "types")
+    ["\nwhere #{format_var(name, context)} was given the #{type_or_types}:\n" | traces]
+  end
+
+  defp format_var({var, _, context}), do: format_var(var, context)
+  defp format_var(var, nil), do: "\"#{var}\""
+  defp format_var(var, context), do: "\"#{var}\" (context #{inspect(context)})"
+
+  defp pluralize([_], singular, _plural), do: singular
+  defp pluralize(_, _singular, plural), do: plural
+
+  defp expr_hints({:<<>>, [inferred_bitstring_spec: true] ++ _meta, _}),
+    do: [:inferred_bitstring_spec]
+
+  defp expr_hints(_), do: []
+
   ## Warning formatting
 
-  def format_warning({:undefined_module, module, fun, arity}) do
-    [
-      Exception.format_mfa(module, fun, arity),
-      " is undefined (module ",
-      inspect(module),
-      " is not available or is yet to be defined)"
-    ]
+  def format_diagnostic({:refine_var, old_type, new_type, var, context}) do
+    traces = collect_traces(var, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          incompatible types assigned to #{format_var(var)}:
+
+              #{to_quoted_string(old_type)} !~ #{to_quoted_string(new_type)}
+          """,
+          format_traces(traces)
+        ])
+    }
   end
 
-  def format_warning({:undefined_function, module, fun, arity, exports}) do
-    [
-      Exception.format_mfa(module, fun, arity),
-      " is undefined or private",
-      UndefinedFunctionError.hint_for_loaded_module(module, fun, arity, exports)
-    ]
+  def format_diagnostic({:incompatible, expr, expected_type, actual_type, hints, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          incompatible types in expression:
+
+              #{expr_to_string(expr) |> indent(4)}
+
+          expected type:
+
+              #{to_quoted_string(expected_type) |> indent(4)}
+
+          but got type:
+
+              #{to_quoted_string(actual_type) |> indent(4)}
+          """,
+          format_traces(traces),
+          format_hints(hints)
+        ])
+    }
   end
 
-  def format_warning({:deprecated, module, fun, arity, reason}) do
-    [
-      Exception.format_mfa(module, fun, arity),
-      " is deprecated. ",
-      reason
-    ]
+  def format_diagnostic({:badmap, expr, type, key, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          expected a map or struct when accessing .#{key} in expression:
+
+              #{expr_to_string(expr) |> indent(4)}
+          """,
+          empty_if(dot_var?(expr), """
+
+          but got type:
+
+              #{to_quoted_string(type) |> indent(4)}
+          """),
+          format_traces(traces),
+          format_hints([:dot])
+        ])
+    }
   end
 
-  def format_warning({:unrequired_module, module, fun, arity}) do
-    [
-      "you must require ",
-      inspect(module),
-      " before invoking the macro ",
-      Exception.format_mfa(module, fun, arity)
-    ]
+  def format_diagnostic({:badkey, expr, type, key, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      span: expr |> get_meta() |> :elixir_env.calculate_span(key) |> Keyword.get(:span),
+      message:
+        IO.iodata_to_binary([
+          """
+          unknown key .#{key} in expression:
+
+              #{expr_to_string(expr) |> indent(4)}
+          """,
+          empty_if(dot_var?(expr), """
+
+          the given type does not have the given key:
+
+              #{to_quoted_string(type) |> indent(4)}
+          """),
+          format_traces(traces)
+        ])
+    }
+  end
+
+  def format_diagnostic({:badmodule, expr, type, fun, arity, hints, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          expected a module (an atom) when invoking #{fun}/#{arity} in expression:
+
+              #{expr_to_string(expr) |> indent(4)}
+          """,
+          empty_if(dot_var?(expr), """
+
+          but got type:
+
+              #{to_quoted_string(type) |> indent(4)}
+          """),
+          format_traces(traces),
+          format_hints(hints)
+        ])
+    }
+  end
+
+  def format_diagnostic({:mismatched_comparison, expr, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          comparison between incompatible types found:
+
+              #{expr_to_string(expr) |> indent(4)}
+          """,
+          format_traces(traces),
+          """
+
+          While Elixir can compare across all types, you are comparing \
+          across types which are always distinct, and the result is either \
+          always true or always false
+          """
+        ])
+    }
+  end
+
+  def format_diagnostic({:struct_comparison, expr, context}) do
+    traces = collect_traces(expr, context)
+
+    %{
+      details: %{typing_traces: traces},
+      message:
+        IO.iodata_to_binary([
+          """
+          comparison with structs found:
+
+              #{expr_to_string(expr) |> indent(4)}
+          """,
+          format_traces(traces),
+          """
+
+          Comparison operators (>, <, >=, <=, min, and max) perform structural \
+          and not semantic comparison. Comparing with a struct won't give meaningful \
+          results. Structs that can be compared typically define a compare/2 function \
+          within their modules that can be used for semantic comparison.
+          """
+        ])
+    }
+  end
+
+  def format_diagnostic({:undefined_module, module, fun, arity}) do
+    top =
+      if fun == :__struct__ and arity == 0 do
+        "struct #{inspect(module)}"
+      else
+        Exception.format_mfa(module, fun, arity)
+      end
+
+    %{
+      message:
+        IO.iodata_to_binary([
+          top,
+          " is undefined (module ",
+          inspect(module),
+          " is not available or is yet to be defined)"
+        ])
+    }
+  end
+
+  def format_diagnostic({:undefined_function, module, :__struct__, 0, _exports}) do
+    %{
+      message:
+        "struct #{inspect(module)} is undefined (there is such module but it does not define a struct)"
+    }
+  end
+
+  def format_diagnostic({:undefined_function, module, fun, arity, exports}) do
+    %{
+      message:
+        IO.iodata_to_binary([
+          Exception.format_mfa(module, fun, arity),
+          " is undefined or private",
+          UndefinedFunctionError.hint_for_loaded_module(module, fun, arity, exports)
+        ])
+    }
+  end
+
+  def format_diagnostic({:deprecated, module, fun, arity, reason}) do
+    %{
+      message:
+        IO.iodata_to_binary([
+          Exception.format_mfa(module, fun, arity),
+          " is deprecated. ",
+          reason
+        ])
+    }
+  end
+
+  def format_diagnostic({:unrequired_module, module, fun, arity}) do
+    %{
+      message:
+        IO.iodata_to_binary([
+          "you must require ",
+          inspect(module),
+          " before invoking the macro ",
+          Exception.format_mfa(module, fun, arity)
+        ])
+    }
+  end
+
+  defp dot_var?(expr) do
+    match?({{:., _, [var, _fun]}, _, _args} when is_var(var), expr)
+  end
+
+  defp empty_if(condition, content) do
+    if condition, do: "", else: content
   end
 end

lib/elixir/lib/module/types/pattern.ex

@@ -2,774 +2,305 @@ defmodule Module.Types.Pattern do
   @moduledoc false
 
   alias Module.Types.Of
-  import Module.Types.{Helpers, Unify}
+  import Module.Types.{Helpers, Descr}
+
+  @guard atom([true, false, :fail])
 
   @doc """
   Handles patterns and guards at once.
   """
-  def of_head(patterns, guards, stack, context) do
+  # TODO: The expected types for patterns/guards must always given as arguments.
+  # Meanwhile, it is hardcoded to dynamic.
+  def of_head(patterns, guards, meta, stack, context) do
+    pattern_stack = %{stack | meta: meta}
+
     with {:ok, types, context} <-
-           map_reduce_ok(patterns, context, &of_pattern(&1, stack, &2)),
-         # TODO: Check that of_guard/4 returns boolean() | :fail
-         {:ok, _, context} <- of_guard(guards_to_or(guards), :dynamic, stack, context),
+           map_reduce_ok(patterns, context, &of_pattern(&1, pattern_stack, &2)),
+         {:ok, _, context} <-
+           map_reduce_ok(guards, context, &of_guard(&1, {@guard, &1}, stack, &2)),
          do: {:ok, types, context}
   end
 
+  ## Patterns
+
   @doc """
-  Return the type and typing context of a pattern expression or an error
-  in case of a typing conflict.
+  Return the type and typing context of a pattern expression
+  with no {expected, expr} pair. of_pattern/4 must be preferred
+  whenever possible as it adds more context to errors.
   """
-  def of_pattern(pattern, %{context: stack_context} = stack, context)
-      when stack_context != :pattern do
-    of_pattern(pattern, %{stack | context: :pattern}, context)
+  def of_pattern(expr, stack, context) do
+    # TODO: Remove the hardcoding of dynamic
+    of_pattern(expr, {dynamic(), expr}, stack, context)
   end
 
-  # _
-  def of_pattern({:_, _meta, atom}, _stack, context) when is_atom(atom) do
-    {:ok, :dynamic, context}
-  end
+  @doc """
+  Return the type and typing context of a pattern expression with
+  the given {expected, expr} pair or an error in case of a typing conflict.
+  """
 
   # ^var
-  def of_pattern({:^, _meta, [var]}, _stack, context) do
-    {:ok, get_var!(var, context), context}
-  end
-
-  # var
-  def of_pattern(var, _stack, context) when is_var(var) do
-    {type, context} = new_var(var, context)
-    {:ok, type, context}
+  def of_pattern({:^, _meta, [var]}, expected_expr, stack, context) do
+    Of.intersect(Of.var(var, context), expected_expr, stack, context)
   end
 
   # left = right
-  def of_pattern({:=, _meta, [left_expr, right_expr]} = expr, stack, context) do
-    stack = push_expr_stack(expr, stack)
+  # TODO: Track variables and handle nesting
+  def of_pattern({:=, _meta, [left_expr, right_expr]}, {expected, expr}, stack, context) do
+    case {is_var(left_expr), is_var(right_expr)} do
+      {true, false} ->
+        with {:ok, type, context} <- of_pattern(right_expr, {expected, expr}, stack, context) do
+          of_pattern(left_expr, {type, expr}, stack, context)
+        end
 
-    with {:ok, left_type, context} <- of_pattern(left_expr, stack, context),
-         {:ok, right_type, context} <- of_pattern(right_expr, stack, context),
-         do: unify(left_type, right_type, stack, context)
-  end
+      {false, true} ->
+        with {:ok, type, context} <- of_pattern(left_expr, {expected, expr}, stack, context) do
+          of_pattern(right_expr, {type, expr}, stack, context)
+        end
 
-  # %_{...}
-  def of_pattern(
-        {:%, _meta1, [{:_, _meta2, var_context}, {:%{}, _meta3, args}]} = expr,
-        stack,
-        context
-      )
-      when is_atom(var_context) do
-    stack = push_expr_stack(expr, stack)
-    expected_fun = fn arg, _expected, stack, context -> of_pattern(arg, stack, context) end
-
-    with {:ok, {:map, pairs}, context} <- Of.open_map(args, stack, context, expected_fun) do
-      {:ok, {:map, [{:required, {:atom, :__struct__}, :atom} | pairs]}, context}
+      {_, _} ->
+        with {:ok, _, context} <- of_pattern(left_expr, {expected, expr}, stack, context),
+             {:ok, _, context} <- of_pattern(right_expr, {expected, expr}, stack, context),
+             do: {:ok, dynamic(), context}
     end
   end
 
   # %var{...} and %^var{...}
-  def of_pattern({:%, _meta1, [var, {:%{}, _meta2, args}]} = expr, stack, context)
-      when not is_atom(var) do
-    stack = push_expr_stack(expr, stack)
-    expected_fun = fn arg, _expected, stack, context -> of_pattern(arg, stack, context) end
-
-    with {:ok, var_type, context} = of_pattern(var, stack, context),
-         {:ok, _, context} <- unify(var_type, :atom, stack, context),
-         {:ok, {:map, pairs}, context} <- Of.open_map(args, stack, context, expected_fun) do
-      {:ok, {:map, [{:required, {:atom, :__struct__}, var_type} | pairs]}, context}
+  def of_pattern(
+        {:%, _meta, [struct_var, {:%{}, _meta2, args}]} = expr,
+        expected_expr,
+        stack,
+        context
+      )
+      when not is_atom(struct_var) do
+    with {:ok, struct_type, context} <-
+           of_pattern(struct_var, {atom(), expr}, %{stack | refine: false}, context),
+         {:ok, map_type, context} <-
+           of_open_map(args, [__struct__: struct_type], expected_expr, stack, context),
+         {_, struct_type} = map_fetch(map_type, :__struct__),
+         {:ok, _struct_type, context} <-
+           of_pattern(struct_var, {struct_type, expr}, stack, context) do
+      {:ok, map_type, context}
     end
   end
 
-  def of_pattern(expr, stack, context) do
-    of_shared(expr, stack, context, &of_pattern/3)
+  # %Struct{...}
+  def of_pattern({:%, _meta, [module, {:%{}, _, args}]} = expr, expected_expr, stack, context)
+      when is_atom(module) do
+    with {:ok, actual, context} <-
+           Of.struct(expr, module, args, :merge_defaults, stack, context, &of_pattern/3) do
+      Of.intersect(actual, expected_expr, stack, context)
+    end
   end
 
-  ## GUARDS
+  # %{...}
+  def of_pattern({:%{}, _meta, args}, expected_expr, stack, context) do
+    of_open_map(args, [], expected_expr, stack, context)
+  end
 
-  # TODO: Some guards can be changed to intersection types or higher order types
-  @boolean {:union, [{:atom, true}, {:atom, false}]}
-  @number {:union, [:integer, :float]}
-  @unary_number_fun [{[:integer], :integer}, {[@number], :float}]
-  @binary_number_fun [
-    {[:integer, :integer], :integer},
-    {[:float, @number], :float},
-    {[@number, :float], :float}
-  ]
+  # <<...>>>
+  def of_pattern({:<<>>, _meta, args}, _expected_expr, stack, context) do
+    case Of.binary(args, :pattern, stack, context, &of_pattern/4) do
+      {:ok, context} -> {:ok, binary(), context}
+      {:error, context} -> {:error, context}
+    end
+  end
 
-  @guard_functions %{
-    {:is_atom, 1} => [{[:atom], @boolean}],
-    {:is_binary, 1} => [{[:binary], @boolean}],
-    {:is_bitstring, 1} => [{[:binary], @boolean}],
-    {:is_boolean, 1} => [{[@boolean], @boolean}],
-    {:is_float, 1} => [{[:float], @boolean}],
-    {:is_function, 1} => [{[:fun], @boolean}],
-    {:is_function, 2} => [{[:fun, :integer], @boolean}],
-    {:is_integer, 1} => [{[:integer], @boolean}],
-    {:is_list, 1} => [{[{:list, :dynamic}], @boolean}],
-    {:is_map, 1} => [{[{:map, [{:optional, :dynamic, :dynamic}]}], @boolean}],
-    {:is_map_key, 2} => [{[:dynamic, {:map, [{:optional, :dynamic, :dynamic}]}], :dynamic}],
-    {:is_number, 1} => [{[@number], @boolean}],
-    {:is_pid, 1} => [{[:pid], @boolean}],
-    {:is_port, 1} => [{[:port], @boolean}],
-    {:is_reference, 1} => [{[:reference], @boolean}],
-    {:is_tuple, 1} => [{[:tuple], @boolean}],
-    {:<, 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:"=<", 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:>, 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:>=, 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:"/=", 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:"=/=", 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:==, 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:"=:=", 2} => [{[:dynamic, :dynamic], @boolean}],
-    {:*, 2} => @binary_number_fun,
-    {:+, 1} => @unary_number_fun,
-    {:+, 2} => @binary_number_fun,
-    {:-, 1} => @unary_number_fun,
-    {:-, 2} => @binary_number_fun,
-    {:/, 2} => @binary_number_fun,
-    {:abs, 1} => @unary_number_fun,
-    {:ceil, 1} => [{[@number], :integer}],
-    {:floor, 1} => [{[@number], :integer}],
-    {:round, 1} => [{[@number], :integer}],
-    {:trunc, 1} => [{[@number], :integer}],
-    {:element, 2} => [{[:integer, :tuple], :dynamic}],
-    {:hd, 1} => [{[{:list, :dynamic}], :dynamic}],
-    {:length, 1} => [{[{:list, :dynamic}], :integer}],
-    {:map_get, 2} => [{[:dynamic, {:map, [{:optional, :dynamic, :dynamic}]}], :dynamic}],
-    {:map_size, 1} => [{[{:map, [{:optional, :dynamic, :dynamic}]}], :integer}],
-    {:tl, 1} => [{[{:list, :dynamic}], :dynamic}],
-    {:tuple_size, 1} => [{[:tuple], :integer}],
-    {:node, 1} => [{[{:union, [:pid, :reference, :port]}], :atom}],
-    {:binary_part, 3} => [{[:binary, :integer, :integer], :binary}],
-    {:bit_size, 1} => [{[:binary], :integer}],
-    {:byte_size, 1} => [{[:binary], :integer}],
-    {:size, 1} => [{[{:union, [:binary, :tuple]}], @boolean}],
-    {:div, 2} => [{[:integer, :integer], :integer}],
-    {:rem, 2} => [{[:integer, :integer], :integer}],
-    {:node, 0} => [{[], :atom}],
-    {:self, 0} => [{[], :pid}],
-    {:bnot, 1} => [{[:integer], :integer}],
-    {:band, 2} => [{[:integer, :integer], :integer}],
-    {:bor, 2} => [{[:integer, :integer], :integer}],
-    {:bxor, 2} => [{[:integer, :integer], :integer}],
-    {:bsl, 2} => [{[:integer, :integer], :integer}],
-    {:bsr, 2} => [{[:integer, :integer], :integer}],
-    {:or, 2} => [{[@boolean, @boolean], @boolean}],
-    {:and, 2} => [{[@boolean, @boolean], @boolean}],
-    {:xor, 2} => [{[@boolean, @boolean], @boolean}],
-    {:not, 1} => [{[@boolean], @boolean}]
+  # _
+  def of_pattern({:_, _meta, _var_context}, {expected, _expr}, _stack, context) do
+    {:ok, expected, context}
+  end
 
-    # Following guards are matched explicitly to handle
-    # type guard functions such as is_atom/1
-    # {:andalso, 2} => {[@boolean, @boolean], @boolean}
-    # {:orelse, 2} => {[@boolean, @boolean], @boolean}
-  }
+  # var
+  def of_pattern({name, meta, ctx} = var, {expected, expr}, stack, context)
+      when is_atom(name) and is_atom(ctx) do
+    case stack do
+      %{refine: true} ->
+        Of.refine_var(var, expected, expr, stack, context)
 
-  @type_guards [
-    :is_atom,
-    :is_binary,
-    :is_bitstring,
-    :is_boolean,
-    :is_float,
-    :is_function,
-    :is_integer,
-    :is_list,
-    :is_map,
-    :is_number,
-    :is_pid,
-    :is_port,
-    :is_reference,
-    :is_tuple
-  ]
+      %{refine: false} ->
+        version = Keyword.fetch!(meta, :version)
+
+        case context do
+          %{vars: %{^version => %{type: type}}} ->
+            Of.intersect(type, {expected, expr}, stack, context)
+
+          %{} ->
+            {:ok, expected, context}
+        end
+    end
+  end
+
+  def of_pattern(expr, expected_expr, stack, context) do
+    of_shared(expr, expected_expr, stack, context, &of_pattern/4)
+  end
+
+  # TODO: Track variables inside the map (mirror it with %var{} handling)
+  defp of_open_map(args, extra, expected_expr, stack, context) do
+    result =
+      reduce_ok(args, {[], context}, fn {key, value}, {fields, context} ->
+        with {:ok, value_type, context} <- of_pattern(value, stack, context) do
+          if is_atom(key) do
+            {:ok, {[{key, value_type} | fields], context}}
+          else
+            {:ok, {fields, context}}
+          end
+        end
+      end)
+
+    with {:ok, {fields, context}} <- result do
+      Of.intersect(open_map(extra ++ fields), expected_expr, stack, context)
+    end
+  end
 
   @doc """
   Refines the type variables in the typing context using type check guards
   such as `is_integer/1`.
   """
-  def of_guard(expr, expected, %{context: stack_context} = stack, context)
-      when stack_context != :pattern do
-    of_guard(expr, expected, %{stack | context: :pattern}, context)
+
+  def of_guard(expr, stack, context) do
+    of_guard(expr, {dynamic(), expr}, stack, context)
   end
 
-  def of_guard({{:., _, [:erlang, :andalso]}, _, [left, right]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-
-    with {:ok, left_type, context} <- of_guard(left, @boolean, stack, context),
-         {:ok, _, context} <- unify(left_type, @boolean, stack, context),
-         {:ok, right_type, context} <- of_guard(right, :dynamic, keep_guarded(stack), context),
-         do: {:ok, to_union([@boolean, right_type], context), context}
+  # %Struct{...}
+  def of_guard({:%, _, [module, {:%{}, _, args}]} = expr, _expected_expr, stack, context)
+      when is_atom(module) do
+    Of.struct(expr, module, args, :skip_defaults, stack, context, &of_guard/3)
   end
 
-  def of_guard({{:., _, [:erlang, :orelse]}, _, [left, right]} = expr, _expected, stack, context) do
-    stack = push_expr_stack(expr, stack)
-    left_indexes = collect_var_indexes_from_expr(left, context)
-    right_indexes = collect_var_indexes_from_expr(right, context)
-
-    with {:ok, left_type, left_context} <- of_guard(left, @boolean, stack, context),
-         {:ok, _right_type, right_context} <- of_guard(right, :dynamic, stack, context),
-         context =
-           merge_context_or(
-             left_indexes,
-             right_indexes,
-             context,
-             stack,
-             left_context,
-             right_context
-           ),
-         {:ok, _, context} <- unify(left_type, @boolean, stack, context),
-         do: {:ok, @boolean, context}
+  # %{...}
+  def of_guard({:%{}, _meta, args}, _expected_expr, stack, context) do
+    Of.closed_map(args, stack, context, &of_guard/3)
   end
 
-  # The unary operators + and - are special cased to avoid common warnings until
-  # we add support for intersection types for the guard functions
-  # -integer / +integer
-  def of_guard({{:., _, [:erlang, guard]}, _, [integer]}, _expected, _stack, context)
-      when guard in [:+, :-] and is_integer(integer) do
-    {:ok, :integer, context}
-  end
-
-  # -float / +float
-  def of_guard({{:., _, [:erlang, guard]}, _, [float]}, _expected, _stack, context)
-      when guard in [:+, :-] and is_float(float) do
-    {:ok, :float, context}
-  end
-
-  # tuple_size(arg) == integer
-  def of_guard(
-        {{:., _, [:erlang, :==]}, _, [{{:., _, [:erlang, :tuple_size]}, _, [var]}, size]} = expr,
-        expected,
-        stack,
-        context
-      )
-      when is_var(var) and is_integer(size) do
-    of_tuple_size(var, size, expr, expected, stack, context)
-  end
-
-  # integer == tuple_size(arg)
-  def of_guard(
-        {{:., _, [:erlang, :==]}, _, [size, {{:., _, [:erlang, :tuple_size]}, _, [var]}]} = expr,
-        expected,
-        stack,
-        context
-      )
-      when is_var(var) and is_integer(size) do
-    of_tuple_size(var, size, expr, expected, stack, context)
-  end
-
-  # fun(args)
-  def of_guard({{:., _, [:erlang, guard]}, _, args} = expr, expected, stack, context) do
-    type_guard? = type_guard?(guard)
-    {consider_type_guards?, keep_guarded?} = stack.type_guards
-    signature = guard_signature(guard, length(args))
-
-    # Only check type guards in the context of and/or/not,
-    # a type guard in the context of is_tuple(x) > :foo
-    # should not affect the inference of x
-    if not type_guard? or consider_type_guards? do
-      stack = push_expr_stack(expr, stack)
-      expected_clauses = filter_clauses(signature, expected, stack, context)
-      param_unions = signature_to_param_unions(expected_clauses, context)
-      arg_stack = %{stack | type_guards: {false, keep_guarded?}}
-      mfa = {:erlang, guard, length(args)}
-
-      with {:ok, arg_types, context} <-
-             map_reduce_ok(Enum.zip(args, param_unions), context, fn {arg, param}, context ->
-               of_guard(arg, param, arg_stack, context)
-             end),
-           {:ok, return_type, context} <-
-             unify_call(
-               arg_types,
-               expected_clauses,
-               expected,
-               mfa,
-               signature,
-               stack,
-               context,
-               type_guard?
-             ) do
-        guard_sources = guard_sources(arg_types, type_guard?, keep_guarded?, context)
-        {:ok, return_type, %{context | guard_sources: guard_sources}}
-      end
-    else
-      # Assume that type guards always return boolean
-      boolean = {:union, [atom: true, atom: false]}
-      [{_params, ^boolean}] = signature
-      {:ok, boolean, context}
+  # <<>>
+  def of_guard({:<<>>, _meta, args}, _expected_expr, stack, context) do
+    case Of.binary(args, :expr, stack, context, &of_guard/4) do
+      {:ok, context} -> {:ok, binary(), context}
+      # It is safe to discard errors from binary inside expressions
+      {:error, context} -> {:ok, binary(), context}
     end
   end
 
-  # map.field
-  def of_guard({{:., meta1, [map, field]}, meta2, []}, expected, stack, context) do
-    of_guard({{:., meta1, [:erlang, :map_get]}, meta2, [field, map]}, expected, stack, context)
+  # var.field
+  def of_guard({{:., _, [callee, key]}, _, []} = expr, _expected_expr, stack, context)
+      when not is_atom(callee) do
+    with {:ok, type, context} <- of_guard(callee, stack, context) do
+      Of.map_fetch(expr, type, key, stack, context)
+    end
+  end
+
+  # Remote
+  def of_guard({{:., _, [:erlang, function]}, _, args} = expr, _expected_expr, stack, context)
+      when is_atom(function) do
+    with {:ok, args_type, context} <-
+           map_reduce_ok(args, context, &of_guard(&1, stack, &2)) do
+      Of.apply(:erlang, function, args_type, expr, stack, context)
+    end
   end
 
   # var
-  def of_guard(var, _expected, _stack, context) when is_var(var) do
-    {:ok, get_var!(var, context), context}
+  def of_guard(var, _expected_expr, _stack, context) when is_var(var) do
+    {:ok, Of.var(var, context), context}
   end
 
-  def of_guard(expr, _expected, stack, context) do
-    of_shared(expr, stack, context, &of_guard(&1, :dynamic, &2, &3))
+  def of_guard(expr, expected_expr, stack, context) do
+    of_shared(expr, expected_expr, stack, context, &of_guard/4)
   end
 
-  defp of_tuple_size(var, size, expr, _expected, stack, context) do
-    {consider_type_guards?, _keep_guarded?} = stack.type_guards
-
-    result =
-      if consider_type_guards? do
-        stack = push_expr_stack(expr, stack)
-        tuple_elems = Enum.map(1..size//1, fn _ -> :dynamic end)
-
-        with {:ok, type, context} <- of_guard(var, :dynamic, stack, context),
-             {:ok, _type, context} <- unify({:tuple, size, tuple_elems}, type, stack, context),
-             do: {:ok, context}
-      else
-        {:ok, context}
-      end
-
-    case result do
-      {:ok, context} ->
-        boolean = {:union, [atom: true, atom: false]}
-        {:ok, boolean, context}
-
-      {:error, reason} ->
-        {:error, reason}
-    end
-  end
-
-  defp signature_to_param_unions(signature, context) do
-    signature
-    |> Enum.map(fn {params, _return} -> params end)
-    |> zip_many()
-    |> Enum.map(&to_union(&1, context))
-  end
-
-  # Collect guard sources from argument types, see type context documentation
-  # for more information
-  defp guard_sources(arg_types, type_guard?, keep_guarded?, context) do
-    {arg_types, guard_sources} =
-      case arg_types do
-        [{:var, index} | rest_arg_types] when type_guard? ->
-          guard_sources = Map.put_new(context.guard_sources, index, :guarded)
-          {rest_arg_types, guard_sources}
-
-        _ ->
-          {arg_types, context.guard_sources}
-      end
-
-    Enum.reduce(arg_types, guard_sources, fn
-      {:var, index}, guard_sources ->
-        Map.update(guard_sources, index, :fail, &guarded_if_keep_guarded(&1, keep_guarded?))
-
-      _, guard_sources ->
-        guard_sources
-    end)
-  end
-
-  defp collect_var_indexes_from_expr(expr, context) do
-    {_, vars} =
-      Macro.prewalk(expr, %{}, fn
-        {:"::", _, [left, right]}, acc ->
-          # Do not mistake binary modifiers as variables
-          {collect_exprs_from_modifiers(right, [left]), acc}
-
-        var, acc when is_var(var) ->
-          var_name = var_name(var)
-          %{^var_name => type} = context.vars
-          {var, collect_var_indexes(type, context, acc)}
-
-        other, acc ->
-          {other, acc}
-      end)
-
-    Map.keys(vars)
-  end
-
-  defp collect_exprs_from_modifiers({:-, _, [left, right]}, acc) do
-    collect_exprs_from_modifiers(left, collect_expr_from_modifier(right, acc))
-  end
-
-  defp collect_exprs_from_modifiers(modifier, acc) do
-    collect_expr_from_modifier(modifier, acc)
-  end
-
-  defp collect_expr_from_modifier({:unit, _, [arg]}, acc), do: [arg | acc]
-  defp collect_expr_from_modifier({:size, _, [arg]}, acc), do: [arg | acc]
-  defp collect_expr_from_modifier({var, _, ctx}, acc) when is_atom(var) and is_atom(ctx), do: acc
-
-  defp unify_call(args, clauses, _expected, _mfa, _signature, stack, context, true = _type_guard?) do
-    unify_type_guard_call(args, clauses, stack, context)
-  end
-
-  defp unify_call(args, clauses, expected, mfa, signature, stack, context, false = _type_guard?) do
-    unify_call(args, clauses, expected, mfa, signature, stack, context)
-  end
-
-  defp unify_call([], [{[], return}], _expected, _mfa, _signature, _stack, context) do
-    {:ok, return, context}
-  end
-
-  defp unify_call(args, clauses, expected, mfa, signature, stack, context) do
-    # Given the arguments:
-    # foo | bar, {:ok, baz | bat}
-
-    # Expand unions in arguments:
-    # foo | bar, {:ok, baz} | {:ok, bat}
-
-    # Permute arguments:
-    # foo, {:ok, baz}
-    # foo, {:ok, bat}
-    # bar, {:ok, baz}
-    # bar, {:ok, bat}
-
-    flatten_args = Enum.map(args, &flatten_union(&1, context))
-    cartesian_args = cartesian_product(flatten_args)
-
-    # Remove clauses that do not match the expected type
-    # Ignore type variables in parameters by changing them to dynamic
-
-    clauses =
-      clauses
-      |> filter_clauses(expected, stack, context)
-      |> Enum.map(fn {params, return} ->
-        {Enum.map(params, &var_to_dynamic/1), return}
-      end)
-
-    # For each permuted argument find the clauses they match
-    # All arguments must match at least one clause, but all clauses
-    # do not need to match
-    # Collect the return values from clauses that matched and collect
-    # the type contexts from unifying argument and parameter to
-    # infer type variables in arguments
-    result =
-      flat_map_ok(cartesian_args, fn cartesian_args ->
-        result =
-          Enum.flat_map(clauses, fn {params, return} ->
-            result =
-              map_ok(Enum.zip(cartesian_args, params), fn {arg, param} ->
-                case unify(arg, param, stack, context) do
-                  {:ok, _type, context} -> {:ok, context}
-                  {:error, reason} -> {:error, reason}
-                end
-              end)
-
-            case result do
-              {:ok, contexts} -> [{return, contexts}]
-              {:error, _reason} -> []
-            end
-          end)
-
-        if result != [] do
-          {:ok, result}
-        else
-          {:error, args}
-        end
-      end)
-
-    case result do
-      {:ok, returns_contexts} ->
-        {success_returns, contexts} = Enum.unzip(returns_contexts)
-        contexts = Enum.concat(contexts)
-
-        indexes =
-          for types <- flatten_args,
-              type <- types,
-              index <- collect_var_indexes_from_type(type),
-              do: index,
-              uniq: true
-
-        # Build unions from collected type contexts to unify with
-        # type variables from arguments
-        result =
-          map_reduce_ok(indexes, context, fn index, context ->
-            union =
-              contexts
-              |> Enum.map(&Map.fetch!(&1.types, index))
-              |> Enum.reject(&(&1 == :unbound))
-
-            if union == [] do
-              {:ok, {:var, index}, context}
-            else
-              unify({:var, index}, to_union(union, context), stack, context)
-            end
-          end)
-
-        case result do
-          {:ok, _types, context} -> {:ok, to_union(success_returns, context), context}
-          {:error, reason} -> {:error, reason}
-        end
-
-      {:error, args} ->
-        error(:unable_apply, {mfa, args, expected, signature, stack}, context)
-    end
-  end
-
-  defp unify_type_guard_call(args, [{params, return}], stack, context) do
-    result =
-      reduce_ok(Enum.zip(args, params), context, fn {arg, param}, context ->
-        case unify(arg, param, stack, context) do
-          {:ok, _, context} -> {:ok, context}
-          {:error, reason} -> {:error, reason}
-        end
-      end)
-
-    case result do
-      {:ok, context} -> {:ok, return, context}
-      {:error, reason} -> {:error, reason}
-    end
-  end
-
-  defp cartesian_product(lists) do
-    List.foldr(lists, [[]], fn list, acc ->
-      for elem_list <- list,
-          list_acc <- acc,
-          do: [elem_list | list_acc]
-    end)
-  end
-
-  defp var_to_dynamic(type) do
-    {type, _acc} =
-      walk(type, :ok, fn
-        {:var, _index}, :ok ->
-          {:dynamic, :ok}
-
-        other, :ok ->
-          {other, :ok}
-      end)
-
-    type
-  end
-
-  defp collect_var_indexes_from_type(type) do
-    {_type, indexes} =
-      walk(type, [], fn
-        {:var, index}, indexes ->
-          {{:var, index}, [index | indexes]}
-
-        other, indexes ->
-          {other, indexes}
-      end)
-
-    indexes
-  end
-
-  defp merge_context_or(left_indexes, right_indexes, context, stack, left, right) do
-    left_different = filter_different_indexes(left_indexes, left, right)
-    right_different = filter_different_indexes(right_indexes, left, right)
-
-    case {left_different, right_different} do
-      {[index], [index]} -> merge_context_or_equal(index, stack, left, right)
-      {_, _} -> merge_context_or_diff(left_different, context, left)
-    end
-  end
-
-  defp filter_different_indexes(indexes, left, right) do
-    Enum.filter(indexes, fn index ->
-      %{^index => left_type} = left.types
-      %{^index => right_type} = right.types
-      left_type != right_type
-    end)
-  end
-
-  defp merge_context_or_equal(index, stack, left, right) do
-    %{^index => left_type} = left.types
-    %{^index => right_type} = right.types
-
-    cond do
-      left_type == :unbound ->
-        refine_var!(index, right_type, stack, left)
-
-      right_type == :unbound ->
-        left
-
-      true ->
-        # Only include right side if left side is from type guard such as is_list(x),
-        # do not refine in case of length(x)
-        if left.guard_sources[index] == :fail do
-          guard_sources = Map.put(left.guard_sources, index, :fail)
-          left = %{left | guard_sources: guard_sources}
-          refine_var!(index, left_type, stack, left)
-        else
-          guard_sources = merge_guard_sources([left.guard_sources, right.guard_sources])
-          left = %{left | guard_sources: guard_sources}
-          refine_var!(index, to_union([left_type, right_type], left), stack, left)
-        end
-    end
-  end
-
-  # If the variable failed, we can keep them from the left side as is.
-  # If they didn't fail, then we need to restore them to their original value.
-  defp merge_context_or_diff(indexes, old_context, new_context) do
-    Enum.reduce(indexes, new_context, fn index, context ->
-      if new_context.guard_sources[index] == :fail do
-        context
-      else
-        restore_var!(index, new_context, old_context)
-      end
-    end)
-  end
-
-  defp merge_guard_sources(sources) do
-    Enum.reduce(sources, fn left, right ->
-      Map.merge(left, right, fn
-        _index, :guarded, :guarded -> :guarded
-        _index, _, _ -> :fail
-      end)
-    end)
-  end
-
-  defp guarded_if_keep_guarded(:guarded, true), do: :guarded
-  defp guarded_if_keep_guarded(_, _), do: :fail
-
-  defp keep_guarded(%{type_guards: {consider?, _}} = stack),
-    do: %{stack | type_guards: {consider?, true}}
-
-  defp filter_clauses(signature, expected, stack, context) do
-    Enum.filter(signature, fn {_params, return} ->
-      match?({:ok, _type, _context}, unify(return, expected, stack, context))
-    end)
-  end
-
-  Enum.each(@guard_functions, fn {{name, arity}, signature} ->
-    defp guard_signature(unquote(name), unquote(arity)), do: unquote(Macro.escape(signature))
-  end)
-
-  Enum.each(@type_guards, fn name ->
-    defp type_guard?(unquote(name)), do: true
-  end)
-
-  defp type_guard?(name) when is_atom(name), do: false
-
   ## Shared
 
   # :atom
-  defp of_shared(atom, _stack, context, _fun) when is_atom(atom) do
-    {:ok, {:atom, atom}, context}
-  end
-
-  # 12
-  defp of_shared(literal, _stack, context, _fun) when is_integer(literal) do
-    {:ok, :integer, context}
-  end
-
-  # 1.2
-  defp of_shared(literal, _stack, context, _fun) when is_float(literal) do
-    {:ok, :float, context}
-  end
-
-  # "..."
-  defp of_shared(literal, _stack, context, _fun) when is_binary(literal) do
-    {:ok, :binary, context}
-  end
-
-  # <<...>>>
-  defp of_shared({:<<>>, _meta, args}, stack, context, fun) do
-    expected_fun = fn arg, _expected, stack, context -> fun.(arg, stack, context) end
-
-    case Of.binary(args, stack, context, expected_fun) do
-      {:ok, context} -> {:ok, :binary, context}
-      {:error, reason} -> {:error, reason}
+  defp of_shared(atom, {expected, expr}, stack, context, _fun) when is_atom(atom) do
+    if atom_type?(expected, atom) do
+      {:ok, atom([atom]), context}
+    else
+      {:error, Of.incompatible_warn(expr, expected, atom([atom]), stack, context)}
     end
   end
 
-  # left | []
-  defp of_shared({:|, _meta, [left_expr, []]} = expr, stack, context, fun) do
-    stack = push_expr_stack(expr, stack)
-    fun.(left_expr, stack, context)
+  # 12
+  defp of_shared(literal, {expected, expr}, stack, context, _fun) when is_integer(literal) do
+    if integer_type?(expected) do
+      {:ok, integer(), context}
+    else
+      {:error, Of.incompatible_warn(expr, expected, integer(), stack, context)}
+    end
   end
 
-  # left | right
-  defp of_shared({:|, _meta, [left_expr, right_expr]} = expr, stack, context, fun) do
-    stack = push_expr_stack(expr, stack)
+  # 1.2
+  defp of_shared(literal, {expected, expr}, stack, context, _fun) when is_float(literal) do
+    if float_type?(expected) do
+      {:ok, float(), context}
+    else
+      {:error, Of.incompatible_warn(expr, expected, float(), stack, context)}
+    end
+  end
 
-    case fun.(left_expr, stack, context) do
-      {:ok, left, context} ->
-        case fun.(right_expr, stack, context) do
-          {:ok, {:list, right}, context} ->
-            {:ok, to_union([left, right], context), context}
-
-          {:ok, right, context} ->
-            {:ok, to_union([left, right], context), context}
-
-          {:error, reason} ->
-            {:error, reason}
-        end
-
-      {:error, reason} ->
-        {:error, reason}
+  # "..."
+  defp of_shared(literal, {expected, expr}, stack, context, _fun) when is_binary(literal) do
+    if binary_type?(expected) do
+      {:ok, binary(), context}
+    else
+      {:error, Of.incompatible_warn(expr, expected, binary(), stack, context)}
     end
   end
 
   # []
-  defp of_shared([], _stack, context, _fun) do
-    {:ok, {:list, :dynamic}, context}
+  defp of_shared([], _expected_expr, _stack, context, _fun) do
+    {:ok, empty_list(), context}
   end
 
   # [expr, ...]
-  defp of_shared(exprs, stack, context, fun) when is_list(exprs) do
-    stack = push_expr_stack(exprs, stack)
-
-    case map_reduce_ok(exprs, context, &fun.(&1, stack, &2)) do
-      {:ok, types, context} -> {:ok, {:list, to_union(types, context)}, context}
+  defp of_shared(exprs, _expected_expr, stack, context, fun) when is_list(exprs) do
+    case map_reduce_ok(exprs, context, &fun.(&1, {dynamic(), &1}, stack, &2)) do
+      {:ok, _types, context} -> {:ok, non_empty_list(), context}
       {:error, reason} -> {:error, reason}
     end
   end
 
+  # {left, right}
+  defp of_shared({left, right}, expected_expr, stack, context, fun) do
+    of_shared({:{}, [], [left, right]}, expected_expr, stack, context, fun)
+  end
+
+  # left | []
+  defp of_shared({:|, _meta, [left_expr, []]}, _expected_expr, stack, context, fun) do
+    fun.(left_expr, {dynamic(), left_expr}, stack, context)
+  end
+
+  # left | right
+  defp of_shared({:|, _meta, [left_expr, right_expr]}, _expected_expr, stack, context, fun) do
+    case fun.(left_expr, {dynamic(), left_expr}, stack, context) do
+      {:ok, _, context} ->
+        fun.(right_expr, {dynamic(), right_expr}, stack, context)
+
+      {:error, reason} ->
+        {:error, reason}
+    end
+  end
+
   # left ++ right
   defp of_shared(
-         {{:., _meta1, [:erlang, :++]}, _meta2, [left_expr, right_expr]} = expr,
+         {{:., _meta1, [:erlang, :++]}, _meta2, [left_expr, right_expr]},
+         _expected_expr,
          stack,
          context,
          fun
        ) do
-    stack = push_expr_stack(expr, stack)
-
-    case fun.(left_expr, stack, context) do
-      {:ok, {:list, left}, context} ->
-        case fun.(right_expr, stack, context) do
-          {:ok, {:list, right}, context} ->
-            {:ok, {:list, to_union([left, right], context)}, context}
-
-          {:ok, right, context} ->
-            {:ok, {:list, to_union([left, right], context)}, context}
-
-          {:error, reason} ->
-            {:error, reason}
-        end
-
-      {:error, reason} ->
-        {:error, reason}
+    # The left side is always a list
+    with {:ok, _, context} <- fun.(left_expr, {dynamic(), left_expr}, stack, context),
+         {:ok, _, context} <- fun.(right_expr, {dynamic(), right_expr}, stack, context) do
+      # TODO: Both lists can be empty, so this may be an empty list,
+      # so we return dynamic for now.
+      {:ok, dynamic(), context}
     end
   end
 
-  # {left, right}
-  defp of_shared({left, right}, stack, context, fun) do
-    of_shared({:{}, [], [left, right]}, stack, context, fun)
-  end
-
   # {...}
-  defp of_shared({:{}, _meta, exprs} = expr, stack, context, fun) do
-    stack = push_expr_stack(expr, stack)
-
-    case map_reduce_ok(exprs, context, &fun.(&1, stack, &2)) do
-      {:ok, types, context} -> {:ok, {:tuple, length(types), types}, context}
+  defp of_shared({:{}, _meta, exprs}, _expected_expr, stack, context, fun) do
+    case map_reduce_ok(exprs, context, &fun.(&1, {dynamic(), &1}, stack, &2)) do
+      {:ok, _, context} -> {:ok, tuple(), context}
       {:error, reason} -> {:error, reason}
     end
   end
-
-  # %{...}
-  defp of_shared({:%{}, _meta, args} = expr, stack, context, fun) do
-    stack = push_expr_stack(expr, stack)
-    expected_fun = fn arg, _expected, stack, context -> fun.(arg, stack, context) end
-    Of.open_map(args, stack, context, expected_fun)
-  end
-
-  # %Struct{...}
-  defp of_shared({:%, meta1, [module, {:%{}, _meta2, args}]} = expr, stack, context, fun)
-       when is_atom(module) do
-    stack = push_expr_stack(expr, stack)
-    expected_fun = fn arg, _expected, stack, context -> fun.(arg, stack, context) end
-
-    with {:ok, struct, context} <- Of.struct(module, meta1, context),
-         {:ok, map, context} <- Of.open_map(args, stack, context, expected_fun) do
-      unify(map, struct, stack, context)
-    end
-  end
 end

lib/elixir/lib/module/types/unify.ex

@@ -1,992 +0,0 @@
-defmodule Module.Types.Unify do
-  @moduledoc false
-
-  import Module.Types.Helpers
-
-  # Those are the simple types known to the system:
-  #
-  #   :dynamic
-  #   {:var, var}
-  #   {:atom, atom} < :atom
-  #   :integer
-  #   :float
-  #   :binary
-  #   :pid
-  #   :port
-  #   :reference
-  #
-  # Those are the composite types:
-  #
-  #   {:list, type}
-  #   {:tuple, size, [type]} < :tuple
-  #   {:union, [type]}
-  #   {:map, [{:required | :optional, key_type, value_type}]}
-  #   {:fun, [{params, return}]}
-  #
-  # Once new types are added, they should be considered in:
-  #
-  #   * unify (all)
-  #   * format_type (all)
-  #   * subtype? (subtypes only)
-  #   * recursive_type? (composite only)
-  #   * collect_var_indexes (composite only)
-  #   * lift_types (composite only)
-  #   * flatten_union (composite only)
-  #   * walk (composite only)
-  #
-
-  @doc """
-  Unifies two types and returns the unified type and an updated typing context
-  or an error in case of a typing conflict.
-  """
-  def unify(same, same, _stack, context) do
-    {:ok, same, context}
-  end
-
-  def unify({:var, var}, type, stack, context) do
-    unify_var(var, type, stack, context, _var_source = true)
-  end
-
-  def unify(type, {:var, var}, stack, context) do
-    unify_var(var, type, stack, context, _var_source = false)
-  end
-
-  def unify({:tuple, n, sources}, {:tuple, n, targets}, stack, context) do
-    result =
-      map_reduce_ok(Enum.zip(sources, targets), context, fn {source, target}, context ->
-        unify(source, target, stack, context)
-      end)
-
-    case result do
-      {:ok, types, context} -> {:ok, {:tuple, n, types}, context}
-      {:error, reason} -> {:error, reason}
-    end
-  end
-
-  def unify({:list, source}, {:list, target}, stack, context) do
-    case unify(source, target, stack, context) do
-      {:ok, type, context} -> {:ok, {:list, type}, context}
-      {:error, reason} -> {:error, reason}
-    end
-  end
-
-  def unify({:map, source_pairs}, {:map, target_pairs}, stack, context) do
-    unify_maps(source_pairs, target_pairs, stack, context)
-  end
-
-  def unify(source, :dynamic, _stack, context) do
-    {:ok, source, context}
-  end
-
-  def unify(:dynamic, target, _stack, context) do
-    {:ok, target, context}
-  end
-
-  def unify({:union, types}, target, stack, context) do
-    unify_result =
-      map_reduce_ok(types, context, fn type, context ->
-        unify(type, target, stack, context)
-      end)
-
-    case unify_result do
-      {:ok, types, context} -> {:ok, to_union(types, context), context}
-      {:error, context} -> {:error, context}
-    end
-  end
-
-  def unify(source, target, stack, context) do
-    cond do
-      # TODO: This condition exists to handle unions with unbound vars.
-      match?({:union, _}, target) and has_unbound_var?(target, context) ->
-        {:ok, source, context}
-
-      subtype?(source, target, context) ->
-        {:ok, source, context}
-
-      true ->
-        error(:unable_unify, {source, target, stack}, context)
-    end
-  end
-
-  def unify_var(var, :dynamic, _stack, context, _var_source?) do
-    {:ok, {:var, var}, context}
-  end
-
-  def unify_var(var, type, stack, context, var_source?) do
-    case context.types do
-      %{^var => :unbound} ->
-        context = refine_var!(var, type, stack, context)
-        stack = push_unify_stack(var, stack)
-
-        if recursive_type?(type, [], context) do
-          if var_source? do
-            error(:unable_unify, {{:var, var}, type, stack}, context)
-          else
-            error(:unable_unify, {type, {:var, var}, stack}, context)
-          end
-        else
-          {:ok, {:var, var}, context}
-        end
-
-      %{^var => {:var, _} = var_type} ->
-        # Do not recursively traverse type vars for now
-        # to avoid pathological cases related to performance.
-        {:ok, var_type, context}
-
-      %{^var => var_type} ->
-        # Only add trace if the variable wasn't already "expanded"
-        context =
-          if variable_expanded?(var, stack, context) do
-            context
-          else
-            trace_var(var, type, stack, context)
-          end
-
-        stack = push_unify_stack(var, stack)
-
-        unify_result =
-          if var_source? do
-            unify(var_type, type, stack, context)
-          else
-            unify(type, var_type, stack, context)
-          end
-
-        case unify_result do
-          {:ok, {:var, ^var}, context} ->
-            {:ok, {:var, var}, context}
-
-          {:ok, res_type, context} ->
-            context = refine_var!(var, res_type, stack, context)
-            {:ok, {:var, var}, context}
-
-          {:error, reason} ->
-            {:error, reason}
-        end
-    end
-  end
-
-  # * All required keys on each side need to match to the other side.
-  # * All optional keys on each side that do not match must be discarded.
-
-  def unify_maps(source_pairs, target_pairs, stack, context) do
-    {source_required, source_optional} = split_pairs(source_pairs)
-    {target_required, target_optional} = split_pairs(target_pairs)
-
-    with {:ok, source_required_pairs, context} <-
-           unify_source_required(source_required, target_pairs, stack, context),
-         {:ok, target_required_pairs, context} <-
-           unify_target_required(target_required, source_pairs, stack, context),
-         {:ok, source_optional_pairs, context} <-
-           unify_source_optional(source_optional, target_optional, stack, context),
-         {:ok, target_optional_pairs, context} <-
-           unify_target_optional(target_optional, source_optional, stack, context) do
-      # Remove duplicate pairs from matching in both left and right directions
-      pairs =
-        Enum.uniq(
-          source_required_pairs ++
-            target_required_pairs ++
-            source_optional_pairs ++
-            target_optional_pairs
-        )
-
-      {:ok, {:map, pairs}, context}
-    else
-      {:error, :unify} ->
-        error(:unable_unify, {{:map, source_pairs}, {:map, target_pairs}, stack}, context)
-
-      {:error, context} ->
-        {:error, context}
-    end
-  end
-
-  def unify_source_required(source_required, target_pairs, stack, context) do
-    map_reduce_ok(source_required, context, fn {source_key, source_value}, context ->
-      Enum.find_value(target_pairs, fn {target_kind, target_key, target_value} ->
-        with {:ok, key, context} <- unify(source_key, target_key, stack, context) do
-          case unify(source_value, target_value, stack, context) do
-            {:ok, value, context} ->
-              {:ok, {:required, key, value}, context}
-
-            {:error, _reason} ->
-              source_map = {:map, [{:required, source_key, source_value}]}
-              target_map = {:map, [{target_kind, target_key, target_value}]}
-              error(:unable_unify, {source_map, target_map, stack}, context)
-          end
-        else
-          {:error, _reason} -> nil
-        end
-      end) || {:error, :unify}
-    end)
-  end
-
-  def unify_target_required(target_required, source_pairs, stack, context) do
-    map_reduce_ok(target_required, context, fn {target_key, target_value}, context ->
-      Enum.find_value(source_pairs, fn {source_kind, source_key, source_value} ->
-        with {:ok, key, context} <- unify(source_key, target_key, stack, context) do
-          case unify(source_value, target_value, stack, context) do
-            {:ok, value, context} ->
-              {:ok, {:required, key, value}, context}
-
-            {:error, _reason} ->
-              source_map = {:map, [{source_kind, source_key, source_value}]}
-              target_map = {:map, [{:required, target_key, target_value}]}
-              error(:unable_unify, {source_map, target_map, stack}, context)
-          end
-        else
-          {:error, _reason} -> nil
-        end
-      end) || {:error, :unify}
-    end)
-  end
-
-  def unify_source_optional(source_optional, target_optional, stack, context) do
-    flat_map_reduce_ok(source_optional, context, fn {source_key, source_value}, context ->
-      Enum.find_value(target_optional, fn {target_key, target_value} ->
-        with {:ok, key, context} <- unify(source_key, target_key, stack, context) do
-          case unify(source_value, target_value, stack, context) do
-            {:ok, value, context} ->
-              {:ok, [{:optional, key, value}], context}
-
-            {:error, _reason} ->
-              source_map = {:map, [{:optional, source_key, source_value}]}
-              target_map = {:map, [{:optional, target_key, target_value}]}
-              error(:unable_unify, {source_map, target_map, stack}, context)
-          end
-        else
-          _ -> nil
-        end
-      end) || {:ok, [], context}
-    end)
-  end
-
-  def unify_target_optional(target_optional, source_optional, stack, context) do
-    flat_map_reduce_ok(target_optional, context, fn {target_key, target_value}, context ->
-      Enum.find_value(source_optional, fn {source_key, source_value} ->
-        with {:ok, key, context} <- unify(source_key, target_key, stack, context) do
-          case unify(source_value, target_value, stack, context) do
-            {:ok, value, context} ->
-              {:ok, [{:optional, key, value}], context}
-
-            {:error, _reason} ->
-              source_map = {:map, [{:optional, source_key, source_value}]}
-              target_map = {:map, [{:optional, target_key, target_value}]}
-              error(:unable_unify, {source_map, target_map, stack}, context)
-          end
-        else
-          _ -> nil
-        end
-      end) || {:ok, [], context}
-    end)
-  end
-
-  defp split_pairs(pairs) do
-    {required, optional} =
-      Enum.split_with(pairs, fn {kind, _key, _value} -> kind == :required end)
-
-    required = Enum.map(required, fn {_kind, key, value} -> {key, value} end)
-    optional = Enum.map(optional, fn {_kind, key, value} -> {key, value} end)
-    {required, optional}
-  end
-
-  def error(type, reason, context), do: {:error, {type, reason, context}}
-
-  @doc """
-  Push expression to stack.
-
-  The expression stack is used to give the context where a type variable
-  was refined when show a type conflict error.
-  """
-  def push_expr_stack(expr, stack) do
-    %{stack | last_expr: expr}
-  end
-
-  @doc """
-  Gets a variable.
-  """
-  def get_var!(var, context) do
-    Map.fetch!(context.vars, var_name(var))
-  end
-
-  @doc """
-  Adds a variable to the typing context and returns its type variable.
-  If the variable has already been added, return the existing type variable.
-  """
-  def new_var(var, context) do
-    var_name = var_name(var)
-
-    case context.vars do
-      %{^var_name => type} ->
-        {type, context}
-
-      %{} ->
-        type = {:var, context.counter}
-        vars = Map.put(context.vars, var_name, type)
-        types_to_vars = Map.put(context.types_to_vars, context.counter, var)
-        types = Map.put(context.types, context.counter, :unbound)
-        traces = Map.put(context.traces, context.counter, [])
-
-        context = %{
-          context
-          | vars: vars,
-            types_to_vars: types_to_vars,
-            types: types,
-            traces: traces,
-            counter: context.counter + 1
-        }
-
-        {type, context}
-    end
-  end
-
-  @doc """
-  Adds an internal variable to the typing context and returns its type variable.
-  An internal variable is used to help unify complex expressions,
-  it does not belong to a specific AST expression.
-  """
-  def add_var(context) do
-    type = {:var, context.counter}
-    types = Map.put(context.types, context.counter, :unbound)
-    traces = Map.put(context.traces, context.counter, [])
-
-    context = %{
-      context
-      | types: types,
-        traces: traces,
-        counter: context.counter + 1
-    }
-
-    {type, context}
-  end
-
-  @doc """
-  Maybe resolves a variable.
-  """
-  def resolve_var({:var, var}, context) do
-    case context.types do
-      %{^var => :unbound} -> {:var, var}
-      %{^var => type} -> resolve_var(type, context)
-    end
-  end
-
-  def resolve_var(other, _context), do: other
-
-  # Check unify stack to see if variable was already expanded
-  defp variable_expanded?(var, stack, context) do
-    Enum.any?(stack.unify_stack, &variable_same?(var, &1, context))
-  end
-
-  defp variable_same?(left, right, context) do
-    case context.types do
-      %{^left => {:var, new_left}} ->
-        variable_same?(new_left, right, context)
-
-      %{^right => {:var, new_right}} ->
-        variable_same?(left, new_right, context)
-
-      %{} ->
-        false
-    end
-  end
-
-  defp push_unify_stack(var, stack) do
-    %{stack | unify_stack: [var | stack.unify_stack]}
-  end
-
-  @doc """
-  Restores the variable information from the old context into new context.
-  """
-  def restore_var!(var, new_context, old_context) do
-    %{^var => type} = old_context.types
-    %{^var => trace} = old_context.traces
-    types = Map.put(new_context.types, var, type)
-    traces = Map.put(new_context.traces, var, trace)
-    %{new_context | types: types, traces: traces}
-  end
-
-  @doc """
-  Set the type for a variable and add trace.
-  """
-  def refine_var!(var, type, stack, context) do
-    types = Map.put(context.types, var, type)
-    context = %{context | types: types}
-    trace_var(var, type, stack, context)
-  end
-
-  @doc """
-  Remove type variable and all its traces.
-  """
-  def remove_var(var, context) do
-    types = Map.delete(context.types, var)
-    traces = Map.delete(context.traces, var)
-    %{context | types: types, traces: traces}
-  end
-
-  defp trace_var(var, type, %{trace: true, last_expr: last_expr} = _stack, context) do
-    line = get_meta(last_expr)[:line]
-    trace = {type, last_expr, {context.file, line}}
-    traces = Map.update!(context.traces, var, &[trace | &1])
-    %{context | traces: traces}
-  end
-
-  defp trace_var(_var, _type, %{trace: false} = _stack, context) do
-    context
-  end
-
-  # Check if a variable is recursive and incompatible with itself
-  # Bad: `{var} = var`
-  # Good: `x = y; y = z; z = x`
-  defp recursive_type?({:var, var} = parent, parents, context) do
-    case context.types do
-      %{^var => :unbound} ->
-        false
-
-      %{^var => type} ->
-        if type in parents do
-          not Enum.all?(parents, &match?({:var, _}, &1))
-        else
-          recursive_type?(type, [parent | parents], context)
-        end
-    end
-  end
-
-  defp recursive_type?({:list, type} = parent, parents, context) do
-    recursive_type?(type, [parent | parents], context)
-  end
-
-  defp recursive_type?({:union, types} = parent, parents, context) do
-    Enum.any?(types, &recursive_type?(&1, [parent | parents], context))
-  end
-
-  defp recursive_type?({:tuple, _, types} = parent, parents, context) do
-    Enum.any?(types, &recursive_type?(&1, [parent | parents], context))
-  end
-
-  defp recursive_type?({:map, pairs} = parent, parents, context) do
-    Enum.any?(pairs, fn {_kind, key, value} ->
-      recursive_type?(key, [parent | parents], context) or
-        recursive_type?(value, [parent | parents], context)
-    end)
-  end
-
-  defp recursive_type?({:fun, clauses}, parents, context) do
-    Enum.any?(clauses, fn {args, return} ->
-      Enum.any?([return | args], &recursive_type?(&1, [clauses | parents], context))
-    end)
-  end
-
-  defp recursive_type?(_other, _parents, _context) do
-    false
-  end
-
-  @doc """
-  Collects all type vars recursively.
-  """
-  def collect_var_indexes(type, context, acc \\ %{}) do
-    {_type, indexes} =
-      walk(type, acc, fn
-        {:var, var}, acc ->
-          case acc do
-            %{^var => _} ->
-              {{:var, var}, acc}
-
-            %{} ->
-              case context.types do
-                %{^var => :unbound} ->
-                  {{:var, var}, Map.put(acc, var, true)}
-
-                %{^var => type} ->
-                  {{:var, var}, collect_var_indexes(type, context, Map.put(acc, var, true))}
-              end
-          end
-
-        other, acc ->
-          {other, acc}
-      end)
-
-    indexes
-  end
-
-  @doc """
-  Checks if the type has a type var.
-  """
-  def has_unbound_var?(type, context) do
-    walk(type, :ok, fn
-      {:var, var}, acc ->
-        case context.types do
-          %{^var => :unbound} ->
-            throw(:has_unbound_var?)
-
-          %{^var => type} ->
-            has_unbound_var?(type, context)
-            {{:var, var}, acc}
-        end
-
-      other, acc ->
-        {other, acc}
-    end)
-
-    false
-  catch
-    :throw, :has_unbound_var? -> true
-  end
-
-  @doc """
-  Returns `true` if it is a singleton type.
-
-  Only atoms are singleton types. Unbound vars are not
-  considered singleton types.
-  """
-  def singleton?({:var, var}, context) do
-    case context.types do
-      %{^var => :unbound} -> false
-      %{^var => type} -> singleton?(type, context)
-    end
-  end
-
-  def singleton?({:atom, _}, _context), do: true
-  def singleton?(_type, _context), do: false
-
-  @doc """
-  Checks if the first argument is a subtype of the second argument.
-
-  This function assumes that:
-
-    * unbound variables are not subtype of anything
-
-    * dynamic is not considered a subtype of all other types but the top type.
-      This allows this function can be used for ordering, in other cases, you
-      may need to check for both sides
-
-  """
-  def subtype?(type, type, _context), do: true
-
-  def subtype?({:var, var}, other, context) do
-    case context.types do
-      %{^var => :unbound} -> false
-      %{^var => type} -> subtype?(type, other, context)
-    end
-  end
-
-  def subtype?(other, {:var, var}, context) do
-    case context.types do
-      %{^var => :unbound} -> false
-      %{^var => type} -> subtype?(other, type, context)
-    end
-  end
-
-  def subtype?(_, :dynamic, _context), do: true
-  def subtype?({:atom, atom}, :atom, _context) when is_atom(atom), do: true
-
-  # Composite
-
-  def subtype?({:tuple, _, _}, :tuple, _context), do: true
-
-  def subtype?({:tuple, n, left_types}, {:tuple, n, right_types}, context) do
-    left_types
-    |> Enum.zip(right_types)
-    |> Enum.all?(fn {left, right} -> subtype?(left, right, context) end)
-  end
-
-  def subtype?({:map, left_pairs}, {:map, right_pairs}, context) do
-    Enum.all?(left_pairs, fn
-      {:required, left_key, left_value} ->
-        Enum.any?(right_pairs, fn {_, right_key, right_value} ->
-          subtype?(left_key, right_key, context) and subtype?(left_value, right_value, context)
-        end)
-
-      {:optional, _, _} ->
-        true
-    end)
-  end
-
-  def subtype?({:list, left}, {:list, right}, context) do
-    subtype?(left, right, context)
-  end
-
-  def subtype?({:union, left_types}, {:union, _} = right_union, context) do
-    Enum.all?(left_types, &subtype?(&1, right_union, context))
-  end
-
-  def subtype?(left, {:union, right_types}, context) do
-    Enum.any?(right_types, &subtype?(left, &1, context))
-  end
-
-  def subtype?({:union, left_types}, right, context) do
-    Enum.all?(left_types, &subtype?(&1, right, context))
-  end
-
-  def subtype?(_left, _right, _context), do: false
-
-  @doc """
-  Returns a "simplified" union using `subtype?/3` to remove redundant types.
-
-  Due to limitations in `subtype?/3` some overlapping types may still be
-  included. For example unions with overlapping non-concrete types such as
-  `{boolean()} | {atom()}` will not be merged or types with variables that
-  are distinct but equivalent such as `a | b when a ~ b`.
-  """
-  def to_union([type], _context), do: type
-
-  def to_union(types, context) when types != [] do
-    case unique_super_types(unnest_unions(types), context) do
-      [type] -> type
-      types -> {:union, types}
-    end
-  end
-
-  defp unnest_unions(types) do
-    Enum.flat_map(types, fn
-      {:union, types} -> unnest_unions(types)
-      type -> [type]
-    end)
-  end
-
-  # Filter subtypes
-  #
-  # `boolean() | atom()` => `atom()`
-  # `:foo | atom()` => `atom()`
-  #
-  # Does not merge `true | false` => `boolean()`
-  defp unique_super_types([type | types], context) do
-    types = Enum.reject(types, &subtype?(&1, type, context))
-
-    if Enum.any?(types, &subtype?(type, &1, context)) do
-      unique_super_types(types, context)
-    else
-      [type | unique_super_types(types, context)]
-    end
-  end
-
-  defp unique_super_types([], _context) do
-    []
-  end
-
-  ## Type lifting
-
-  @doc """
-  Lifts type variables to their inferred types from the context.
-  """
-  def lift_types(types, %{lifted_types: _} = context) do
-    Enum.map_reduce(types, context, &lift_type/2)
-  end
-
-  def lift_types(types, context) do
-    context = %{
-      types: context.types,
-      lifted_types: %{},
-      lifted_counter: 0
-    }
-
-    Enum.map_reduce(types, context, &lift_type/2)
-  end
-
-  # Lift type variable to its inferred (hopefully concrete) types from the context
-  defp lift_type({:var, var}, context) do
-    case context.lifted_types do
-      %{^var => lifted_var} ->
-        {{:var, lifted_var}, context}
-
-      %{} ->
-        case context.types do
-          %{^var => :unbound} ->
-            new_lifted_var(var, context)
-
-          %{^var => type} ->
-            if recursive_type?(type, [], context) do
-              new_lifted_var(var, context)
-            else
-              # Remove visited types to avoid infinite loops
-              # then restore after we are done recursing on vars
-              types = context.types
-              context = put_in(context.types[var], :unbound)
-              {type, context} = lift_type(type, context)
-              {type, %{context | types: types}}
-            end
-
-          %{} ->
-            new_lifted_var(var, context)
-        end
-    end
-  end
-
-  defp lift_type({:union, types}, context) do
-    {types, context} = Enum.map_reduce(types, context, &lift_type/2)
-    {{:union, types}, context}
-  end
-
-  defp lift_type({:tuple, n, types}, context) do
-    {types, context} = Enum.map_reduce(types, context, &lift_type/2)
-    {{:tuple, n, types}, context}
-  end
-
-  defp lift_type({:map, pairs}, context) do
-    {pairs, context} =
-      Enum.map_reduce(pairs, context, fn {kind, key, value}, context ->
-        {key, context} = lift_type(key, context)
-        {value, context} = lift_type(value, context)
-        {{kind, key, value}, context}
-      end)
-
-    {{:map, pairs}, context}
-  end
-
-  defp lift_type({:list, type}, context) do
-    {type, context} = lift_type(type, context)
-    {{:list, type}, context}
-  end
-
-  defp lift_type({:fun, clauses}, context) do
-    clauses =
-      Enum.map_reduce(clauses, context, fn {args, return}, context ->
-        {[return | args], context} = Enum.map_reduce([return | args], context, &lift_type/2)
-        {{args, return}, context}
-      end)
-
-    {{:fun, clauses}, context}
-  end
-
-  defp lift_type(other, context) do
-    {other, context}
-  end
-
-  defp new_lifted_var(original_var, context) do
-    types = Map.put(context.lifted_types, original_var, context.lifted_counter)
-    counter = context.lifted_counter + 1
-
-    type = {:var, context.lifted_counter}
-    context = %{context | lifted_types: types, lifted_counter: counter}
-    {type, context}
-  end
-
-  # TODO: Figure out function expansion
-
-  @doc """
-  Expand unions so that all unions are at the top level.
-
-      {integer() | float()} => {integer()} | {float()}
-  """
-  def flatten_union({:union, types}, context) do
-    Enum.flat_map(types, &flatten_union(&1, context))
-  end
-
-  def flatten_union(type, context) do
-    List.wrap(do_flatten_union(type, context))
-  end
-
-  def do_flatten_union({:tuple, num, types}, context) do
-    flatten_union_tuple(types, num, context, [])
-  end
-
-  def do_flatten_union({:list, type}, context) do
-    case do_flatten_union(type, context) do
-      {:union, union_types} -> Enum.map(union_types, &{:list, &1})
-      _type -> [{:list, type}]
-    end
-  end
-
-  def do_flatten_union({:map, pairs}, context) do
-    flatten_union_map(pairs, context, [])
-  end
-
-  def do_flatten_union({:var, var}, context) do
-    if looping_var?(var, context, []) do
-      {:var, var}
-    else
-      case context.types do
-        %{^var => :unbound} -> {:var, var}
-        %{^var => {:union, types}} -> Enum.map(types, &do_flatten_union(&1, context))
-        %{^var => type} -> do_flatten_union(type, context)
-      end
-    end
-  end
-
-  def do_flatten_union(type, _context) do
-    type
-  end
-
-  defp flatten_union_tuple([type | types], num, context, acc) do
-    case do_flatten_union(type, context) do
-      {:union, union_types} ->
-        Enum.flat_map(union_types, &flatten_union_tuple(types, num, context, [&1 | acc]))
-
-      type ->
-        flatten_union_tuple(types, num, context, [type | acc])
-    end
-  end
-
-  defp flatten_union_tuple([], num, _context, acc) do
-    [{:tuple, num, Enum.reverse(acc)}]
-  end
-
-  defp flatten_union_map([{kind, key, value} | pairs], context, acc) do
-    case do_flatten_union(key, context) do
-      {:union, union_types} ->
-        Enum.flat_map(union_types, &flatten_union_map_value(kind, &1, value, pairs, context, acc))
-
-      type ->
-        flatten_union_map_value(kind, type, value, pairs, context, acc)
-    end
-  end
-
-  defp flatten_union_map([], _context, acc) do
-    [{:map, Enum.reverse(acc)}]
-  end
-
-  defp flatten_union_map_value(kind, key, value, pairs, context, acc) do
-    case do_flatten_union(value, context) do
-      {:union, union_types} ->
-        Enum.flat_map(union_types, &flatten_union_map(pairs, context, [{kind, key, &1} | acc]))
-
-      value ->
-        flatten_union_map(pairs, context, [{kind, key, value} | acc])
-    end
-  end
-
-  defp looping_var?(var, context, parents) do
-    case context.types do
-      %{^var => :unbound} ->
-        false
-
-      %{^var => {:var, type}} ->
-        if var in parents do
-          true
-        else
-          looping_var?(type, context, [var | parents])
-        end
-
-      %{^var => _type} ->
-        false
-    end
-  end
-
-  @doc """
-  Formats types.
-
-  The second argument says when complex types such as maps and
-  structs should be simplified and not shown.
-  """
-  def format_type({:map, pairs}, true) do
-    case List.keyfind(pairs, {:atom, :__struct__}, 1) do
-      {:required, {:atom, :__struct__}, {:atom, struct}} ->
-        ["%", inspect(struct), "{}"]
-
-      _ ->
-        "map()"
-    end
-  end
-
-  def format_type({:union, types}, simplify?) do
-    types
-    |> Enum.map(&format_type(&1, simplify?))
-    |> Enum.intersperse(" | ")
-  end
-
-  def format_type({:tuple, _, types}, simplify?) do
-    format =
-      types
-      |> Enum.map(&format_type(&1, simplify?))
-      |> Enum.intersperse(", ")
-
-    ["{", format, "}"]
-  end
-
-  def format_type({:list, type}, simplify?) do
-    ["[", format_type(type, simplify?), "]"]
-  end
-
-  def format_type({:map, pairs}, false) do
-    case List.keytake(pairs, {:atom, :__struct__}, 1) do
-      {{:required, {:atom, :__struct__}, {:atom, struct}}, pairs} ->
-        ["%", inspect(struct), "{", format_map_pairs(pairs), "}"]
-
-      _ ->
-        ["%{", format_map_pairs(pairs), "}"]
-    end
-  end
-
-  def format_type({:atom, literal}, _simplify?) do
-    inspect(literal)
-  end
-
-  def format_type({:var, index}, _simplify?) do
-    ["var", Integer.to_string(index + 1)]
-  end
-
-  def format_type({:fun, clauses}, simplify?) do
-    format =
-      Enum.map(clauses, fn {params, return} ->
-        params = Enum.intersperse(Enum.map(params, &format_type(&1, simplify?)), ", ")
-        params = if params == [], do: params, else: [params, " "]
-        return = format_type(return, simplify?)
-        [params, "-> ", return]
-      end)
-
-    ["(", Enum.intersperse(format, "; "), ")"]
-  end
-
-  def format_type(atom, _simplify?) when is_atom(atom) do
-    [Atom.to_string(atom), "()"]
-  end
-
-  defp format_map_pairs(pairs) do
-    {atoms, others} = Enum.split_with(pairs, &match?({:required, {:atom, _}, _}, &1))
-    {required, optional} = Enum.split_with(others, &match?({:required, _, _}, &1))
-
-    (atoms ++ required ++ optional)
-    |> Enum.map(fn
-      {:required, {:atom, atom}, right} ->
-        [Atom.to_string(atom), ": ", format_type(right, false)]
-
-      {:required, left, right} ->
-        [format_type(left, false), " => ", format_type(right, false)]
-
-      {:optional, left, right} ->
-        ["optional(", format_type(left, false), ") => ", format_type(right, false)]
-    end)
-    |> Enum.intersperse(", ")
-  end
-
-  @doc """
-  Performs a depth-first, pre-order traversal of the type tree using an accumulator.
-  """
-  def walk({:map, pairs}, acc, fun) do
-    {pairs, acc} =
-      Enum.map_reduce(pairs, acc, fn {kind, key, value}, acc ->
-        {key, acc} = walk(key, acc, fun)
-        {value, acc} = walk(value, acc, fun)
-        {{kind, key, value}, acc}
-      end)
-
-    fun.({:map, pairs}, acc)
-  end
-
-  def walk({:union, types}, acc, fun) do
-    {types, acc} = Enum.map_reduce(types, acc, &walk(&1, &2, fun))
-    fun.({:union, types}, acc)
-  end
-
-  def walk({:tuple, num, types}, acc, fun) do
-    {types, acc} = Enum.map_reduce(types, acc, &walk(&1, &2, fun))
-    fun.({:tuple, num, types}, acc)
-  end
-
-  def walk({:list, type}, acc, fun) do
-    {type, acc} = walk(type, acc, fun)
-    fun.({:list, type}, acc)
-  end
-
-  def walk({:fun, clauses}, acc, fun) do
-    {clauses, acc} =
-      Enum.map_reduce(clauses, acc, fn {params, return}, acc ->
-        {params, acc} = Enum.map_reduce(params, acc, &walk(&1, &2, fun))
-        {return, acc} = walk(return, acc, fun)
-        {{params, return}, acc}
-      end)
-
-    fun.({:fun, clauses}, acc)
-  end
-
-  def walk(type, acc, fun) do
-    fun.(type, acc)
-  end
-end

lib/elixir/lib/node.ex

@@ -219,7 +219,7 @@ defmodule Node do
 
   If `node` does not exist, a useless PID is returned.
 
-  For the list of available options, see `:erlang.spawn/4`.
+  For the list of available options, see `:erlang.spawn_opt/5`.
 
   Inlined by the compiler.
   """

lib/elixir/lib/option_parser.ex

@@ -152,44 +152,47 @@ defmodule OptionParser do
   ### Parsing unknown switches
 
   When the `:switches` option is given, `OptionParser` will attempt to parse
-  unknown switches:
+  unknown switches.
+
+  Switches without an argument will be set to `true`:
 
       iex> OptionParser.parse(["--debug"], switches: [key: :string])
       {[debug: true], [], []}
 
   Even though we haven't specified `--debug` in the list of switches, it is part
-  of the returned options. This would also work:
+  of the returned options. The same happens for switches followed by another switch:
+
+      iex> OptionParser.parse(["--debug", "--ok"], switches: [])
+      {[debug: true, ok: true], [], []}
+
+  Switches followed by a value will be assigned the value, as a string:
 
       iex> OptionParser.parse(["--debug", "value"], switches: [key: :string])
       {[debug: "value"], [], []}
 
-  Switches followed by a value will be assigned the value, as a string. Switches
-  without an argument will be set automatically to `true`. Since we cannot assert
-  the type of the switch value, it is preferred to use the `:strict` option that
-  accepts only known switches and always verify their types.
+  Since we cannot assert the type of the switch value, it is preferred to use the
+  `:strict` option that accepts only known switches and always verify their types.
 
   If you do want to parse unknown switches, remember that Elixir converts switches
-  to atoms. Since atoms are not garbage-collected, OptionParser will only parse
-  switches that translate to atoms used by the runtime to avoid leaking atoms.
-  For instance, the code below will discard the `--option-parser-example` switch
-  because the `:option_parser_example` atom is never used anywhere:
+  to atoms. Since atoms are not garbage-collected, to avoid creating new ones,
+  OptionParser by default only parses switches that translate to existing atoms.
+  The code below discards the `--option-parser-example` switch because the
+  `:option_parser_example` atom is never used anywhere:
 
-      OptionParser.parse(["--option-parser-example"], switches: [debug: :boolean])
-      # The :option_parser_example atom is not used anywhere below
+      iex> OptionParser.parse(["--option-parser-example"], switches: [])
+      {[], [], []}
 
-  However, the code below would work as long as `:option_parser_example` atom is
-  used at some point later (or earlier) **in the same module**. For example:
+  If a switch corresponds to an existing Elixir atom, whether from your
+  code, a dependency or from Elixir itself, it will be accepted. However,
+  it is best to not rely on external code, and always define the atoms
+  you want to parse in the same module that calls `OptionParser` itself,
+  as direct arguments to the `:switches` or `:strict` options.
 
-      {opts, _, _} = OptionParser.parse(["--option-parser-example"], switches: [debug: :boolean])
-      # ... then somewhere in the same module you access it ...
-      opts[:option_parser_example]
-
-  In other words, Elixir will only parse options that are used by the runtime,
-  ignoring all others. If you would like to parse all switches, regardless if
-  they exist or not, you can force creation of atoms by passing
-  `allow_nonexistent_atoms: true` as option. Use this option with care. It is
-  only useful when you are building command-line applications that receive
-  dynamically-named arguments and must be avoided in long-running systems.
+  If you would like to parse all switches, regardless if they exist or not,
+  you can force creation of atoms by passing `allow_nonexistent_atoms: true`
+  as option. Use this option with care. It is only useful when you are building
+  command-line applications that receive dynamically-named arguments and must
+  be avoided in long-running systems.
 
   ## Aliases
 

lib/elixir/lib/path.ex

@@ -44,17 +44,17 @@ defmodule Path do
   """
   @spec absname(t) :: binary
   def absname(path) do
-    absname(path, File.cwd!())
+    absname(path, &File.cwd!/0)
   end
 
   @doc """
   Builds a path from `relative_to` to `path`.
 
   If `path` is already an absolute path, `relative_to` is ignored. See also
-  `relative_to/3`.
+  `relative_to/3`. `relative_to` is either a path or an anonymous function,
+  which is invoked only when necessary, that returns a path.
 
-  Unlike `expand/2`, no attempt is made to
-  resolve `..`, `.` or `~`.
+  Unlike `expand/2`, no attempt is made to resolve `..`, `.` or `~`.
 
   ## Examples
 
@@ -64,20 +64,37 @@ defmodule Path do
       iex> Path.absname("../x", "bar")
       "bar/../x"
 
+      iex> Path.absname("foo", fn -> "lazy" end)
+      "lazy/foo"
+
   """
-  @spec absname(t, t) :: binary
+  @spec absname(t, t | (-> t)) :: binary
   def absname(path, relative_to) do
     path = IO.chardata_to_string(path)
 
     case type(path) do
       :relative ->
+        relative_to =
+          if is_function(relative_to, 0) do
+            relative_to.()
+          else
+            relative_to
+          end
+
         absname_join([relative_to, path])
 
       :absolute ->
         absname_join([path])
 
       :volumerelative ->
-        relative_to = IO.chardata_to_string(relative_to)
+        relative_to =
+          if is_function(relative_to, 0) do
+            relative_to.()
+          else
+            relative_to
+          end
+          |> IO.chardata_to_string()
+
         absname_vr(split(path), split(relative_to), relative_to)
     end
   end
@@ -155,15 +172,21 @@ defmodule Path do
   Converts the path to an absolute one, expanding
   any `.` and `..` components and a leading `~`.
 
+  If a relative path is provided it is expanded relatively to
+  the current working directory.
+
   ## Examples
 
       Path.expand("/foo/bar/../baz")
       #=> "/foo/baz"
 
+      Path.expand("foo/bar/../baz")
+      #=> "$PWD/foo/baz"
+
   """
   @spec expand(t) :: binary
   def expand(path) do
-    expand_dot(absname(expand_home(path), File.cwd!()))
+    expand_dot(absname(expand_home(path), &File.cwd!/0))
   end
 
   @doc """
@@ -192,7 +215,7 @@ defmodule Path do
   """
   @spec expand(t, t) :: binary
   def expand(path, relative_to) do
-    expand_dot(absname(absname(expand_home(path), expand_home(relative_to)), File.cwd!()))
+    expand_dot(absname(absname(expand_home(path), expand_home(relative_to)), &File.cwd!/0))
   end
 
   @doc """
@@ -244,7 +267,7 @@ defmodule Path do
       Path.relative("/bar/foo.ex")      #=> "bar/foo.ex"
 
   """
-  # Note this function does not expand paths because the behaviour
+  # Note this function does not expand paths because the behavior
   # is ambiguous. If we expand it before converting to relative, then
   # "/usr/../../foo" means "/foo". If we expand it after, it means "../foo".
   # We could expand only relative paths but it is best to say it never
@@ -410,6 +433,8 @@ defmodule Path do
   defp relative_to_unforced(_, _, original), do: join(original)
 
   defp relative_to_forced(path, path, _original), do: "."
+  defp relative_to_forced(["."], _path, _original), do: "."
+  defp relative_to_forced(path, ["."], _original), do: join(path)
   defp relative_to_forced([h | t1], [h | t2], original), do: relative_to_forced(t1, t2, original)
 
   # this should only happen if we have two paths on different drives on windows
@@ -745,9 +770,9 @@ defmodule Path do
   You may call `Path.expand/1` to normalize the path before invoking
   this function.
 
-  A character preceded by \ loses its special meaning.
-  Note that \ must be written as \\ in a string literal.
-  For example, "\\?*" will match any filename starting with ?.
+  A character preceded by `\\` loses its special meaning.
+  Note that `\\` must be written as `\\\\` in a string literal.
+  For example, `"\\\\?*"` will match any filename starting with `?.`.
 
   By default, the patterns `*` and `?` do not match files starting
   with a dot `.`. See the `:match_dot` option in the "Options" section

lib/elixir/lib/port.ex

@@ -16,7 +16,7 @@ defmodule Port do
       iex> send(port, {self(), :close})
       :ok
       iex> flush()
-      {#Port<0.1464>, :closed}
+      {#Port<0.1444>, :closed}
       :ok
 
   In the example above, we have created a new port that executes the
@@ -79,6 +79,27 @@ defmodule Port do
   are for advanced usage within the VM. Also consider using `System.cmd/3`
   if all you want is to execute a program and retrieve its return value.
 
+  > #### Windows argument splitting and untrusted arguments {: .warning}
+  >
+  > On Unix systems, arguments are passed to a new operating system
+  > process as an array of strings but on Windows it is up to the child
+  > process to parse them and some Windows programs may apply their own
+  > rules, which are inconsistent with the standard C runtime `argv` parsing
+  >
+  > This is particularly troublesome when invoking `.bat` or `.com` files
+  > as these run implicitly through `cmd.exe`, whose argument parsing is
+  > vulnerable to malicious input and can be used to run arbitrary shell
+  > commands.
+  >
+  > Therefore, if you are running on Windows and you execute batch
+  > files or `.com` applications, you must not pass untrusted input as
+  > arguments to the program. You may avoid accidentally executing them
+  > by explicitly passing the extension of the program you want to run,
+  > such as `.exe`, and double check the program is indeed not a batch
+  > file or `.com` application.
+  >
+  > This affects both `spawn` and `spawn_executable`.
+
   ### spawn
 
   The `:spawn` tuple receives a binary that is going to be executed as a

lib/elixir/lib/process.ex

@@ -195,7 +195,7 @@ defmodule Process do
   @doc """
   Sends an exit signal with the given `reason` to `pid`.
 
-  The following behaviour applies if `reason` is any term except `:normal`
+  The following behavior applies if `reason` is any term except `:normal`
   or `:kill`:
 
     1. If `pid` is not trapping exits, `pid` will exit with the given
@@ -504,7 +504,7 @@ defmodule Process do
   If the process is already dead when calling `Process.monitor/1`, a
   `:DOWN` message is delivered immediately.
 
-  See ["The need for monitoring"](https://elixir-lang.org/getting-started/mix-otp/genserver.html#the-need-for-monitoring)
+  See ["The need for monitoring"](genservers.md#the-need-for-monitoring)
   for an example. See `:erlang.monitor/2` for more information.
 
   Inlined by the compiler.
@@ -624,7 +624,7 @@ defmodule Process do
   exits with a reason other than `:normal` (which is also the exit reason used
   when a process finishes its job) and `pid1` is not trapping exits (see
   `flag/2`), then `pid1` will exit with the same reason as `pid2` and in turn
-  emit an exit signal to all its other linked processes. The behaviour when
+  emit an exit signal to all its other linked processes. The behavior when
   `pid1` is trapping exits is described in `exit/2`.
 
   See `:erlang.link/1` for more information.
@@ -839,13 +839,17 @@ defmodule Process do
     nilify(:erlang.process_info(pid))
   end
 
+  @type process_info_item :: atom | {:dictionary, term}
+  @type process_info_result_item :: {process_info_item, term}
+
   @doc """
   Returns information about the process identified by `pid`,
   or returns `nil` if the process is not alive.
 
   See `:erlang.process_info/2` for more information.
   """
-  @spec info(pid, atom | [atom]) :: {atom, term} | [{atom, term}] | nil
+  @spec info(pid, process_info_item) :: process_info_result_item | nil
+  @spec info(pid, [process_info_item]) :: [process_info_result_item] | nil
   def info(pid, spec)
 
   def info(pid, :registered_name) do
@@ -856,7 +860,7 @@ defmodule Process do
     end
   end
 
-  def info(pid, spec) when is_atom(spec) or is_list(spec) do
+  def info(pid, spec) do
     nilify(:erlang.process_info(pid, spec))
   end
 
@@ -940,6 +944,31 @@ defmodule Process do
   @spec unalias(alias) :: boolean
   defdelegate unalias(alias), to: :erlang
 
+  @doc """
+  Add a descriptive term to the current process.
+
+  The term does not need to be unique, and in Erlang/OTP 27+ will be shown in
+  Observer and in crash logs.
+  This label may be useful for identifying a process as one of multiple in a
+  given role, such as `:queue_worker` or `{:live_chat, user_id}`.
+
+  ## Examples
+
+      Process.set_label(:worker)
+      #=> :ok
+
+      Process.set_label({:any, "term"})
+      #=> :ok
+  """
+  @doc since: "1.17.0"
+  @spec set_label(term()) :: :ok
+  def set_label(label) do
+    # TODO: switch to `:proc_lib.set_label/2` when we require Erlang/OTP 27+
+    Process.put(:"$process_label", label)
+    # mimic return value of `:proc_lib.set_label/2`
+    :ok
+  end
+
   @compile {:inline, nilify: 1}
   defp nilify(:undefined), do: nil
   defp nilify(other), do: other

lib/elixir/lib/protocol.ex

@@ -126,8 +126,8 @@ defmodule Protocol do
   Although the implementation above is arguably not a reasonable
   one. For example, it makes no sense to say a PID or an integer
   have a size of `0`. That's one of the reasons why `@fallback_to_any`
-  is an opt-in behaviour. For the majority of protocols, raising
-  an error when a protocol is not implemented is the proper behaviour.
+  is an opt-in behavior. For the majority of protocols, raising
+  an error when a protocol is not implemented is the proper behavior.
 
   ## Multiple implementations
 
@@ -448,7 +448,7 @@ defmodule Protocol do
   ## Examples
 
       # Get Elixir's ebin directory path and retrieve all protocols
-      iex> path = :code.lib_dir(:elixir, :ebin)
+      iex> path = Application.app_dir(:elixir, "ebin")
       iex> mods = Protocol.extract_protocols([path])
       iex> Enumerable in mods
       true
@@ -477,7 +477,7 @@ defmodule Protocol do
   ## Examples
 
       # Get Elixir's ebin directory path and retrieve all protocols
-      iex> path = :code.lib_dir(:elixir, :ebin)
+      iex> path = Application.app_dir(:elixir, "ebin")
       iex> mods = Protocol.extract_impls(Enumerable, [path])
       iex> List in mods
       true
@@ -972,6 +972,7 @@ defmodule Protocol do
       Protocol.__ensure_defimpl__(protocol, for, __ENV__)
 
       defmodule name do
+        @moduledoc false
         @behaviour protocol
         @protocol protocol
         @for for

lib/elixir/lib/range.ex

@@ -172,7 +172,7 @@ defmodule Range do
   one element, which is the number itself.
 
   If `first` is greater than `last`, the range will be decreasing from `first`
-  to `last`, albeit this behaviour is deprecated. Therefore, it is advised to
+  to `last`, albeit this behavior is deprecated. Therefore, it is advised to
   explicitly list the step with `new/3`.
 
   ## Examples
@@ -184,7 +184,7 @@ defmodule Range do
 
   @spec new(limit, limit) :: t
   def new(first, last) when is_integer(first) and is_integer(last) do
-    # TODO: Deprecate inferring a range with a step of -1 on Elixir v1.17
+    # TODO: Deprecate inferring a range with a step of -1 on Elixir v1.18
     step = if first <= last, do: 1, else: -1
     %Range{first: first, last: last, step: step}
   end
@@ -231,8 +231,6 @@ defmodule Range do
       iex> Range.size(1..10//-1)
       0
 
-      iex> Range.size(10..1)
-      10
       iex> Range.size(10..1//-1)
       10
       iex> Range.size(10..1//-2)
@@ -428,7 +426,7 @@ defmodule Range do
 
       iex> Range.disjoint?(1..5, 6..9)
       true
-      iex> Range.disjoint?(5..1, 6..9)
+      iex> Range.disjoint?(5..1//-1, 6..9)
       true
       iex> Range.disjoint?(1..5, 5..9)
       false
@@ -508,8 +506,10 @@ defmodule Range do
 
   @doc false
   @deprecated "Pattern match on first..last//step instead"
-  def range?(term)
-  def range?(first..last) when is_integer(first) and is_integer(last), do: true
+  def range?(%{__struct__: Range, first: first, last: last})
+      when is_integer(first) and is_integer(last),
+      do: true
+
   def range?(_), do: false
 end
 

lib/elixir/lib/regex.ex

@@ -84,7 +84,7 @@ defmodule Regex do
       each line; use `\A` and `\z` to match the end or beginning of the string
 
     * `:extended` (x) - whitespace characters are ignored except when escaped
-      and allow `#` to delimit comments
+      or within `[..]`, and allow `#` to delimit comments
 
     * `:firstline` (f) - forces the unanchored pattern to match before or at the
       first newline, though the matched text may continue over the newline
@@ -122,7 +122,7 @@ defmodule Regex do
     * `:all_names` - captures all named subpattern matches in the Regex as a list
       ordered **alphabetically** by the names of the subpatterns
 
-    * `list(binary)` - a list of named captures to capture
+    * `list(binary | atom)` - a list of named captures to capture
 
   ## Character classes
 
@@ -152,10 +152,10 @@ defmodule Regex do
 
   There is another character class, `ascii`, that erroneously matches
   Latin-1 characters instead of the 0-127 range specified by POSIX. This
-  cannot be fixed without altering the behaviour of other classes, so we
+  cannot be fixed without altering the behavior of other classes, so we
   recommend matching the range with `[\\0-\x7f]` instead.
 
-  Note the behaviour of those classes may change according to the Unicode
+  Note the behavior of those classes may change according to the Unicode
   and other modifiers:
 
       iex> String.match?("josé", ~r/^[[:lower:]]+$/)
@@ -184,7 +184,7 @@ defmodule Regex do
   check and recompile the regex if necessary.
   """
 
-  defstruct re_pattern: nil, source: "", opts: "", re_version: ""
+  defstruct re_pattern: nil, source: "", opts: [], re_version: ""
 
   @type t :: %__MODULE__{re_pattern: term, source: binary, opts: binary | [term]}
 
@@ -213,7 +213,7 @@ defmodule Regex do
       {:ok, ~r/foo/}
 
       iex> Regex.compile("*foo")
-      {:error, {'nothing to repeat', 0}}
+      {:error, {~c"nothing to repeat", 0}}
 
       iex> Regex.compile("foo", "i")
       {:ok, ~r/foo/i}
@@ -223,39 +223,30 @@ defmodule Regex do
 
   """
   @spec compile(binary, binary | [term]) :: {:ok, t} | {:error, any}
-  def compile(source, options \\ "") when is_binary(source) do
-    compile(source, options, version())
+  def compile(source, opts \\ "") when is_binary(source) do
+    compile(source, opts, version())
   end
 
-  defp compile(source, options, version) when is_binary(options) do
-    case translate_options(options, []) do
+  defp compile(source, opts, version) when is_binary(opts) do
+    case translate_options(opts, []) do
       {:error, rest} ->
         {:error, {:invalid_option, rest}}
 
-      translated_options ->
-        compile(source, translated_options, options, version)
+      translated_opts ->
+        compile(source, translated_opts, version)
     end
   end
 
-  defp compile(source, options, version) when is_list(options) do
-    compile(source, options, "", version)
-  end
-
-  defp compile(source, opts, doc_opts, version) do
+  defp compile(source, opts, version) when is_list(opts) do
     case :re.compile(source, opts) do
       {:ok, re_pattern} ->
-        doc_opts = format_doc_opts(doc_opts, opts)
-        {:ok, %Regex{re_pattern: re_pattern, re_version: version, source: source, opts: doc_opts}}
+        {:ok, %Regex{re_pattern: re_pattern, re_version: version, source: source, opts: opts}}
 
       error ->
         error
     end
   end
 
-  defp format_doc_opts(_doc_opts = "", _opts = []), do: ""
-  defp format_doc_opts(_doc_opts = "", opts), do: opts
-  defp format_doc_opts(doc_opts, _opts), do: doc_opts
-
   @doc """
   Compiles the regular expression and raises `Regex.CompileError` in case of errors.
   """
@@ -343,7 +334,7 @@ defmodule Regex do
 
     * `:return` - when set to `:index`, returns byte index and match length.
       Defaults to `:binary`.
-    * `:capture` - what to capture in the result. Check the moduledoc for `Regex`
+    * `:capture` - what to capture in the result. See the ["Captures" section](#module-captures)
       to see the possible capture values.
     * `:offset` - (since v1.12.0) specifies the starting offset to match in the given string.
       Defaults to zero.
@@ -359,6 +350,12 @@ defmodule Regex do
       iex> Regex.run(~r/c(d)/, "abcd", return: :index)
       [{2, 2}, {3, 1}]
 
+      iex> Regex.run(~r/c(d)/, "abcd", capture: :first)
+      ["cd"]
+
+      iex> Regex.run(~r/c(?<foo>d)/, "abcd", capture: ["foo", "bar"])
+      ["d", ""]
+
   """
   @spec run(t, binary, [term]) :: nil | [binary] | [{integer, integer}]
   def run(regex, string, options \\ [])
@@ -426,21 +423,20 @@ defmodule Regex do
   end
 
   @doc """
-  Returns the regex options, as a string or list depending on how
-  it was compiled.
+  Returns the regex options.
 
   See the documentation of `Regex.compile/2` for more information.
 
   ## Examples
 
       iex> Regex.opts(~r/foo/m)
-      "m"
+      [:multiline]
 
       iex> Regex.opts(Regex.compile!("foo", [:caseless]))
       [:caseless]
 
   """
-  @spec opts(t) :: String.t() | [term]
+  @spec opts(t) :: [term]
   def opts(%Regex{opts: opts}) do
     opts
   end
@@ -471,8 +467,7 @@ defmodule Regex do
   end
 
   @doc ~S"""
-  Same as `run/3`, but scans the target several times collecting all
-  matches of the regular expression.
+  Same as `run/3` but returns all non-overlapping matches of the regular expression.
 
   A list of lists is returned, where each entry in the primary list represents a
   match and each entry in the secondary list represents the captured contents.
@@ -481,7 +476,7 @@ defmodule Regex do
 
     * `:return` - when set to `:index`, returns byte index and match length.
       Defaults to `:binary`.
-    * `:capture` - what to capture in the result. Check the moduledoc for `Regex`
+    * `:capture` - what to capture in the result. See the ["Captures" section](#module-captures)
       to see the possible capture values.
     * `:offset` - (since v1.12.0) specifies the starting offset to match in the given string.
       Defaults to zero.
@@ -497,12 +492,21 @@ defmodule Regex do
       iex> Regex.scan(~r/e/, "abcd")
       []
 
+      iex> Regex.scan(~r/ab|bc|cd/, "abcd")
+      [["ab"], ["cd"]]
+
+      iex> Regex.scan(~r/ab|bc|cd/, "abbccd")
+      [["ab"], ["bc"], ["cd"]]
+
       iex> Regex.scan(~r/\p{Sc}/u, "$, £, and €")
       [["$"], ["£"], ["€"]]
 
       iex> Regex.scan(~r/=+/, "=ü†ƒ8===", return: :index)
       [[{0, 1}], [{9, 3}]]
 
+      iex> Regex.scan(~r/c(d|e)/, "abcd abce", capture: :first)
+      [["cd"], ["ce"]]
+
   """
   @spec scan(t(), String.t(), [term()]) :: [[String.t()]] | [[{integer(), integer()}]]
   def scan(regex, string, options \\ [])
@@ -526,8 +530,15 @@ defmodule Regex do
          options
        ) do
     case version() do
-      ^version -> :re.run(string, compiled, options)
-      _ -> :re.run(string, source, translate_options(compile_opts, options))
+      ^version ->
+        :re.run(string, compiled, options)
+
+      _ when is_list(compile_opts) ->
+        :re.run(string, source, compile_opts ++ options)
+
+      # TODO: This clause is kept for compatibility with previous Elixir versions. Remove on v2.0+.
+      _ when is_binary(compile_opts) ->
+        :re.run(string, source, translate_options(compile_opts, options))
     end
   end
 
@@ -547,7 +558,7 @@ defmodule Regex do
 
     * `:on` - specifies which captures to split the string on, and in what
       order. Defaults to `:first` which means captures inside the regex do not
-      affect the splitting process. Check the moduledoc for `Regex`
+      affect the splitting process. See the ["Captures" section](#module-captures)
       to see the possible capture values.
 
     * `:include_captures` - when `true`, includes in the result the matches of
@@ -580,6 +591,9 @@ defmodule Regex do
       iex> Regex.split(~r{a(?<second>b)c}, "abc", on: [:second], include_captures: true)
       ["a", "b", "c"]
 
+      iex> Regex.split(~r{-}, "-a-b--c", trim: true)
+      ["a", "b", "c"]
+
   """
   @spec split(t, String.t(), [term]) :: [String.t()]
   def split(regex, string, options \\ [])
@@ -709,7 +723,8 @@ defmodule Regex do
       "Abcadc"
 
   """
-  @spec replace(t, String.t(), String.t() | (... -> String.t()), [term]) :: String.t()
+  @spec replace(t, String.t(), String.t() | (... -> String.t()), global: boolean()) ::
+          String.t()
   def replace(%Regex{} = regex, string, replacement, options \\ [])
       when is_binary(string) and is_list(options) do
     opts = if Keyword.get(options, :global) != false, do: [:global], else: []
@@ -883,15 +898,14 @@ defmodule Regex do
 
   # Helpers
 
+  defp translate_options(<<?s, t::binary>>, acc),
+    do: translate_options(t, [:dotall, {:newline, :anycrlf} | acc])
+
   defp translate_options(<<?u, t::binary>>, acc), do: translate_options(t, [:unicode, :ucp | acc])
   defp translate_options(<<?i, t::binary>>, acc), do: translate_options(t, [:caseless | acc])
   defp translate_options(<<?x, t::binary>>, acc), do: translate_options(t, [:extended | acc])
   defp translate_options(<<?f, t::binary>>, acc), do: translate_options(t, [:firstline | acc])
   defp translate_options(<<?U, t::binary>>, acc), do: translate_options(t, [:ungreedy | acc])
-
-  defp translate_options(<<?s, t::binary>>, acc),
-    do: translate_options(t, [:dotall, {:newline, :anycrlf} | acc])
-
   defp translate_options(<<?m, t::binary>>, acc), do: translate_options(t, [:multiline | acc])
 
   defp translate_options(<<?r, t::binary>>, acc) do
@@ -900,5 +914,5 @@ defmodule Regex do
   end
 
   defp translate_options(<<>>, acc), do: acc
-  defp translate_options(rest, _acc), do: {:error, rest}
+  defp translate_options(t, _acc), do: {:error, t}
 end

lib/elixir/lib/registry.ex

@@ -18,7 +18,7 @@ defmodule Registry do
   implementation. We explore some of those use cases below.
 
   The registry may also be transparently partitioned, which provides
-  more scalable behaviour for running registries on highly concurrent
+  more scalable behavior for running registries on highly concurrent
   environments with thousands or millions of entries.
 
   ## Using in `:via`
@@ -27,8 +27,8 @@ defmodule Registry do
   `Registry.start_link/1`, it can be used to register and access named
   processes using the `{:via, Registry, {registry, key}}` tuple:
 
-      {:ok, _} = Registry.start_link(keys: :unique, name: Registry.ViaTest)
-      name = {:via, Registry, {Registry.ViaTest, "agent"}}
+      {:ok, _} = Registry.start_link(keys: :unique, name: MyApp.Registry)
+      name = {:via, Registry, {MyApp.Registry, "agent"}}
       {:ok, _} = Agent.start_link(fn -> 0 end, name: name)
       Agent.get(name, & &1)
       #=> 0
@@ -39,22 +39,22 @@ defmodule Registry do
   In the previous example, we were not interested in associating a value to the
   process:
 
-      Registry.lookup(Registry.ViaTest, "agent")
+      Registry.lookup(MyApp.Registry, "agent")
       #=> [{self(), nil}]
 
   However, in some cases it may be desired to associate a value to the process
   using the alternate `{:via, Registry, {registry, key, value}}` tuple:
 
-      {:ok, _} = Registry.start_link(keys: :unique, name: Registry.ViaTest)
-      name = {:via, Registry, {Registry.ViaTest, "agent", :hello}}
+      {:ok, _} = Registry.start_link(keys: :unique, name: MyApp.Registry)
+      name = {:via, Registry, {MyApp.Registry, "agent", :hello}}
       {:ok, agent_pid} = Agent.start_link(fn -> 0 end, name: name)
-      Registry.lookup(Registry.ViaTest, "agent")
+      Registry.lookup(MyApp.Registry, "agent")
       #=> [{agent_pid, :hello}]
 
   To this point, we have been starting `Registry` using `start_link/1`.
   Typically the registry is started as part of a supervision tree though:
 
-      {Registry, keys: :unique, name: Registry.ViaTest}
+      {Registry, keys: :unique, name: MyApp.Registry}
 
   Only registries with unique keys can be used in `:via`. If the name is
   already taken, the case-specific `start_link` function (`Agent.start_link/2`
@@ -757,28 +757,28 @@ defmodule Registry do
   In the example below we register the current process and look it up
   both from itself and other processes:
 
-      iex> Registry.start_link(keys: :unique, name: Registry.UniqueLookupTest)
-      iex> Registry.values(Registry.UniqueLookupTest, "hello", self())
+      iex> Registry.start_link(keys: :unique, name: Registry.UniqueValuesTest)
+      iex> Registry.values(Registry.UniqueValuesTest, "hello", self())
       []
-      iex> {:ok, _} = Registry.register(Registry.UniqueLookupTest, "hello", :world)
-      iex> Registry.values(Registry.UniqueLookupTest, "hello", self())
+      iex> {:ok, _} = Registry.register(Registry.UniqueValuesTest, "hello", :world)
+      iex> Registry.values(Registry.UniqueValuesTest, "hello", self())
       [:world]
-      iex> Task.async(fn -> Registry.values(Registry.UniqueLookupTest, "hello", self()) end) |> Task.await()
+      iex> Task.async(fn -> Registry.values(Registry.UniqueValuesTest, "hello", self()) end) |> Task.await()
       []
       iex> parent = self()
-      iex> Task.async(fn -> Registry.values(Registry.UniqueLookupTest, "hello", parent) end) |> Task.await()
+      iex> Task.async(fn -> Registry.values(Registry.UniqueValuesTest, "hello", parent) end) |> Task.await()
       [:world]
 
   The same applies to duplicate registries:
 
-      iex> Registry.start_link(keys: :duplicate, name: Registry.DuplicateLookupTest)
-      iex> Registry.values(Registry.DuplicateLookupTest, "hello", self())
+      iex> Registry.start_link(keys: :duplicate, name: Registry.DuplicateValuesTest)
+      iex> Registry.values(Registry.DuplicateValuesTest, "hello", self())
       []
-      iex> {:ok, _} = Registry.register(Registry.DuplicateLookupTest, "hello", :world)
-      iex> Registry.values(Registry.DuplicateLookupTest, "hello", self())
+      iex> {:ok, _} = Registry.register(Registry.DuplicateValuesTest, "hello", :world)
+      iex> Registry.values(Registry.DuplicateValuesTest, "hello", self())
       [:world]
-      iex> {:ok, _} = Registry.register(Registry.DuplicateLookupTest, "hello", :another)
-      iex> Enum.sort(Registry.values(Registry.DuplicateLookupTest, "hello", self()))
+      iex> {:ok, _} = Registry.register(Registry.DuplicateValuesTest, "hello", :another)
+      iex> Enum.sort(Registry.values(Registry.DuplicateValuesTest, "hello", self()))
       [:another, :world]
 
   """
@@ -1301,16 +1301,16 @@ defmodule Registry do
       iex> Registry.start_link(keys: :unique, name: Registry.SelectAllTest)
       iex> {:ok, _} = Registry.register(Registry.SelectAllTest, "hello", :value)
       iex> {:ok, _} = Registry.register(Registry.SelectAllTest, "world", :value)
-      iex> Registry.select(Registry.SelectAllTest, [{{:"$1", :"$2", :"$3"}, [], [{{:"$1", :"$2", :"$3"}}]}])
-      [{"world", self(), :value}, {"hello", self(), :value}]
+      iex> Registry.select(Registry.SelectAllTest, [{{:"$1", :"$2", :"$3"}, [], [{{:"$1", :"$2", :"$3"}}]}]) |> Enum.sort()
+      [{"hello", self(), :value}, {"world", self(), :value}]
 
   Get all keys in the registry:
 
       iex> Registry.start_link(keys: :unique, name: Registry.SelectAllTest)
       iex> {:ok, _} = Registry.register(Registry.SelectAllTest, "hello", :value)
       iex> {:ok, _} = Registry.register(Registry.SelectAllTest, "world", :value)
-      iex> Registry.select(Registry.SelectAllTest, [{{:"$1", :_, :_}, [], [:"$1"]}])
-      ["world", "hello"]
+      iex> Registry.select(Registry.SelectAllTest, [{{:"$1", :_, :_}, [], [:"$1"]}]) |> Enum.sort()
+      ["hello", "world"]
 
   """
   @doc since: "1.9.0"

lib/elixir/lib/stream.ex

@@ -1138,10 +1138,52 @@ defmodule Stream do
   end
 
   @doc """
-  Creates a stream where each element in the enumerable will
-  be wrapped in a tuple alongside its index.
+  Builds a stream from an index, either starting from offset, or given by function.
 
-  If an `offset` is given, we will index from the given offset instead of from zero.
+  May receive a function or an integer offset.
+
+  If an `offset` is given, it will emit elements from offset.
+
+  If a `function` is given, it will invoke the function with
+  elements from offset.
+
+  ## Examples
+
+      iex> Stream.from_index() |> Enum.take(3)
+      [0, 1, 2]
+
+      iex> Stream.from_index(1) |> Enum.take(3)
+      [1, 2, 3]
+
+      iex> Stream.from_index(fn x -> x * 10 end) |> Enum.take(3)
+      [0, 10, 20]
+
+  """
+  @doc since: "1.17.0"
+  @spec from_index(integer) :: Enumerable.t(integer)
+  @spec from_index((integer -> return_value)) :: Enumerable.t(return_value)
+        when return_value: term
+  def from_index(fun_or_offset \\ 0)
+
+  def from_index(offset) when is_integer(offset) do
+    unfold(offset, &{&1, &1 + 1})
+  end
+
+  def from_index(fun) when is_function(fun) do
+    unfold(0, &{fun.(&1), &1 + 1})
+  end
+
+  @doc """
+  Creates a stream where each element in the enumerable will
+  be wrapped in a tuple alongside its index or according to a given function.
+
+  May receive a function or an integer offset.
+
+  If an `offset` is given, it will index from the given offset instead of from
+  zero.
+
+  If a `function` is given, it will index by invoking the function for each
+  element and index (zero-based) of the enumerable.
 
   ## Examples
 
@@ -1153,12 +1195,24 @@ defmodule Stream do
       iex> Enum.to_list(stream)
       [{1, 3}, {2, 4}, {3, 5}]
 
+      iex> stream = Stream.with_index([1, 2, 3], fn x, index -> x + index end)
+      iex> Enum.to_list(stream)
+      [1, 3, 5]
+
   """
-  @spec with_index(Enumerable.t(), integer) :: Enumerable.t()
-  def with_index(enum, offset \\ 0) when is_integer(offset) do
+  @spec with_index(Enumerable.t(), integer) :: Enumerable.t({element, integer})
+  @spec with_index(Enumerable.t(), (element, index -> return_value)) :: Enumerable.t(return_value)
+        when return_value: term
+  def with_index(enum, fun_or_offset \\ 0)
+
+  def with_index(enum, offset) when is_integer(offset) do
     lazy(enum, offset, fn f1 -> R.with_index(f1) end)
   end
 
+  def with_index(enum, fun) when is_function(fun, 2) do
+    lazy(enum, 0, fn f1 -> R.with_index(fun, f1) end)
+  end
+
   ## Combiners
 
   @doc """
@@ -1200,6 +1254,11 @@ defmodule Stream do
   @doc """
   Zips two enumerables together, lazily.
 
+  Because a list of two-element tuples with atoms as the first
+  tuple element is a keyword list (`Keyword`), zipping a first `Stream`
+  of atoms with a second `Stream` of any kind creates a `Stream`
+  that generates a keyword list.
+
   The zipping finishes as soon as either enumerable completes.
 
   ## Examples
@@ -1208,6 +1267,8 @@ defmodule Stream do
       iex> cycle = Stream.cycle([:a, :b, :c])
       iex> Stream.zip(concat, cycle) |> Enum.to_list()
       [{1, :a}, {2, :b}, {3, :c}, {4, :a}, {5, :b}, {6, :c}]
+      iex> Stream.zip(cycle, concat) |> Enum.to_list()
+      [a: 1, b: 2, c: 3, a: 4, b: 5, c: 6]
 
   """
   @spec zip(Enumerable.t(), Enumerable.t()) :: Enumerable.t()
@@ -1511,13 +1572,15 @@ defmodule Stream do
   end
 
   @doc """
-  Emits a sequence of values, starting with `start_value`. Successive
-  values are generated by calling `next_fun` on the previous value.
+  Emits a sequence of values, starting with `start_value`.
+
+  Successive values are generated by calling `next_fun`
+  on the previous value.
 
   ## Examples
 
-      iex> Stream.iterate(0, &(&1 + 1)) |> Enum.take(5)
-      [0, 1, 2, 3, 4]
+      iex> Stream.iterate(1, &(&1 * 2)) |> Enum.take(5)
+      [1, 2, 4, 8, 16]
 
   """
   @spec iterate(element, (element -> element)) :: Enumerable.t()

lib/elixir/lib/stream/reducers.ex

@@ -238,11 +238,19 @@ defmodule Stream.Reducers do
     end
   end
 
-  defmacro with_index(fun \\ nil) do
+  defmacro with_index(fun) do
     quote do
       fn entry, acc(head, counter, tail) ->
         next_with_acc(unquote(fun), {entry, counter}, head, counter + 1, tail)
       end
     end
   end
+
+  defmacro with_index(callback, fun) do
+    quote do
+      fn entry, acc(head, counter, tail) ->
+        next_with_acc(unquote(fun), unquote(callback).(entry, counter), head, counter + 1, tail)
+      end
+    end
+  end
 end

lib/elixir/lib/string.ex

@@ -167,7 +167,7 @@ defmodule String do
   the locale, it is not taken into account by this module.
 
   In general, the functions in this module rely on the Unicode
-  Standard, but do not contain any of the locale specific behaviour.
+  Standard, but do not contain any of the locale specific behavior.
   More information about graphemes can be found in the [Unicode
   Standard Annex #29](https://www.unicode.org/reports/tr29/).
 
@@ -201,6 +201,9 @@ defmodule String do
       iex> eacute
       233
 
+  See the [*Patterns and Guards* guide](patterns-and-guards.md) and the documentation for
+  [`<<>>`](`<<>>/1`) for more information on binary pattern matching.
+
   You can also fully convert a string into a list of integer code points,
   known as "charlists" in Elixir, by calling `String.to_charlist/1`:
 
@@ -227,7 +230,7 @@ defmodule String do
   to the definition of the encoding) is encountered, only one
   code point needs to be rejected.
 
-  This module relies on this behaviour to ignore such invalid
+  This module relies on this behavior to ignore such invalid
   characters. For example, `length/1` will return
   a correct result even if an invalid code point is fed into it.
 
@@ -393,6 +396,9 @@ defmodule String do
   When the pattern used is a regular expression, the string is
   split using `Regex.split/3`.
 
+  If the pattern cannot be found, a list containing the original
+  string will be returned.
+
   ## Options
 
     * `:parts` (positive integer or `:infinity`) - the string
@@ -458,6 +464,11 @@ defmodule String do
       iex> String.split("abc", "", parts: 3)
       ["", "a", "bc"]
 
+  Splitting on an non-existing pattern returns the original string:
+
+      iex> String.split("abc", ",")
+      ["abc"]
+
   Be aware that this function can split within or across grapheme boundaries.
   For example, take the grapheme "é" which is made of the characters
   "e" and the acute accent. The following will split the string into two parts:
@@ -966,6 +977,7 @@ defmodule String do
 
   def capitalize(string, mode) when is_binary(string) do
     case :unicode_util.gc(string) do
+      [gc] -> grapheme_to_binary(:string.titlecase([gc]))
       [gc, rest] -> grapheme_to_binary(:string.titlecase([gc])) <> downcase(rest, mode)
       [gc | rest] -> grapheme_to_binary(:string.titlecase([gc])) <> downcase(rest, mode)
       [] -> ""
@@ -1472,7 +1484,7 @@ defmodule String do
   The `replacement` may be a string or a function that receives the matched
   pattern and must return the replacement as a string or iodata.
 
-  By default it replaces all occurrences but this behaviour can be controlled
+  By default it replaces all occurrences but this behavior can be controlled
   through the `:global` option; see the "Options" section below.
 
   ## Options
@@ -1843,7 +1855,7 @@ defmodule String do
       ** (FunctionClauseError) no function clause matching in String.valid?/2
 
   """
-  @spec valid?(t) :: boolean
+  @spec valid?(t, :default | :fast_ascii) :: boolean
   def valid?(string, algorithm \\ :default)
 
   def valid?(<<string::binary>>, :default), do: valid_utf8?(string)
@@ -1871,6 +1883,109 @@ defmodule String do
     end
   end
 
+  defguardp replace_invalid_ii_of_iii(i, ii)
+            when Bitwise.bor(Bitwise.bsl(i, 6), ii) in 32..863 or
+                   Bitwise.bor(Bitwise.bsl(i, 6), ii) in 896..1023
+
+  defguardp replace_invalid_ii_of_iv(i, ii)
+            when Bitwise.bor(Bitwise.bsl(i, 6), ii) in 16..271
+
+  defguardp replace_invalid_iii_of_iv(i, ii, iii)
+            when Bitwise.bor(Bitwise.bor(Bitwise.bsl(i, 12), Bitwise.bsl(ii, 6)), iii) in 1024..17407
+
+  defguardp replace_invalid_is_next(next) when Bitwise.bsr(next, 6) !== 0b10
+
+  @doc ~S"""
+  Returns a new string created by replacing all invalid bytes with `replacement` (`"�"` by default).
+
+  ## Examples
+
+      iex> String.replace_invalid("asd" <> <<0xFF::8>>)
+      "asd�"
+
+      iex> String.replace_invalid("nem rán bề bề")
+      "nem rán bề bề"
+
+      iex> String.replace_invalid("nem rán b" <> <<225, 187>> <> " bề")
+      "nem rán b� bề"
+
+      iex> String.replace_invalid("nem rán b" <> <<225, 187>> <> " bề", "ERROR!")
+      "nem rán bERROR! bề"
+  """
+  @doc since: "1.16.0"
+  @spec replace_invalid(binary, t) :: t
+  def replace_invalid(bytes, replacement \\ "�")
+      when is_binary(bytes) and is_binary(replacement) do
+    do_replace_invalid(bytes, replacement, <<>>)
+  end
+
+  # Valid ASCII (for better average speed)
+  defp do_replace_invalid(<<ascii::8, next::8, _::binary>> = rest, rep, acc)
+       when ascii in 0..127 and replace_invalid_is_next(next) do
+    <<_::8, rest::binary>> = rest
+    do_replace_invalid(rest, rep, acc <> <<ascii::8>>)
+  end
+
+  # Valid UTF-8
+  defp do_replace_invalid(<<grapheme::utf8, rest::binary>>, rep, acc) do
+    do_replace_invalid(rest, rep, acc <> <<grapheme::utf8>>)
+  end
+
+  # 2/3 truncated sequence
+  defp do_replace_invalid(<<0b1110::4, i::4, 0b10::2, ii::6>>, rep, acc)
+       when replace_invalid_ii_of_iii(i, ii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(
+         <<0b1110::4, i::4, 0b10::2, ii::6, next::8, _::binary>> = rest,
+         rep,
+         acc
+       )
+       when replace_invalid_ii_of_iii(i, ii) and replace_invalid_is_next(next) do
+    <<_::16, rest::binary>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # 2/4
+  defp do_replace_invalid(<<0b11110::5, i::3, 0b10::2, ii::6>>, rep, acc)
+       when replace_invalid_ii_of_iv(i, ii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(
+         <<0b11110::5, i::3, 0b10::2, ii::6, next::8, _::binary>> = rest,
+         rep,
+         acc
+       )
+       when replace_invalid_ii_of_iv(i, ii) and replace_invalid_is_next(next) do
+    <<_::16, rest::binary>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # 3/4
+  defp do_replace_invalid(<<0b11110::5, i::3, 0b10::2, ii::6, 0b10::2, iii::6>>, rep, acc)
+       when replace_invalid_iii_of_iv(i, ii, iii) do
+    acc <> rep
+  end
+
+  defp do_replace_invalid(
+         <<0b11110::5, i::3, 0b10::2, ii::6, 0b10::2, iii::6, next::8, _::binary>> = rest,
+         rep,
+         acc
+       )
+       when replace_invalid_iii_of_iv(i, ii, iii) and replace_invalid_is_next(next) do
+    <<_::24, rest::binary>> = rest
+    do_replace_invalid(rest, rep, acc <> rep)
+  end
+
+  # Everything else
+  defp do_replace_invalid(<<_, rest::binary>>, rep, acc),
+    do: do_replace_invalid(rest, rep, acc <> rep)
+
+  # Final
+  defp do_replace_invalid(<<>>, _, acc), do: acc
+
   @doc ~S"""
   Splits the string into chunks of characters that share a common trait.
 
@@ -1991,12 +2106,27 @@ defmodule String do
     end
   end
 
-  @doc false
-  @deprecated "Use String.next_grapheme/1 instead"
+  @doc """
+  Returns the size (in bytes) of the next grapheme.
+
+  The result is a tuple with the next grapheme size in bytes and
+  the remainder of the string or `nil` in case the string
+  reached its end.
+
+  ## Examples
+
+      iex> String.next_grapheme_size("olá")
+      {1, "lá"}
+
+      iex> String.next_grapheme_size("")
+      nil
+
+  """
   @spec next_grapheme_size(t) :: {pos_integer, t} | nil
   def next_grapheme_size(string) when is_binary(string) do
     case :unicode_util.gc(string) do
       [gc] -> {grapheme_byte_size(gc), <<>>}
+      [gc, rest] -> {grapheme_byte_size(gc), rest}
       [gc | rest] -> {grapheme_byte_size(gc), rest}
       [] -> nil
       {:error, <<_, rest::bits>>} -> {1, rest}
@@ -2139,12 +2269,14 @@ defmodule String do
   @doc """
   Returns a substring starting at the offset `start`, and of the given `length`.
 
-  If the offset is greater than string length, then it returns `""`.
+  This function works on Unicode graphemes. For example, slicing the first
+  three characters of the string "héllo" will return "hél", which internally
+  is represented by more than three bytes. Use `String.byte_slice/3` if you
+  want to slice by a given number of bytes, while respecting the codepoint
+  boundaries. If you want to work on raw bytes, check `Kernel.binary_part/3`
+  or `Kernel.binary_slice/3` instead.
 
-  Remember this function works with Unicode graphemes and considers
-  the slices to represent grapheme offsets. If you want to split
-  on raw bytes, check `Kernel.binary_part/3` or `Kernel.binary_slice/3`
-  instead.
+  If the offset is greater than string length, then it returns `""`.
 
   ## Examples
 
@@ -2204,6 +2336,13 @@ defmodule String do
   Returns a substring from the offset given by the start of the
   range to the offset given by the end of the range.
 
+  This function works on Unicode graphemes. For example, slicing the first
+  three characters of the string "héllo" will return "hél", which internally
+  is represented by more than three bytes. Use `String.byte_slice/3` if you
+  want to slice by a given number of bytes, while respecting the codepoint
+  boundaries. If you want to work on raw bytes, check `Kernel.binary_part/3`
+  or `Kernel.binary_slice/3` instead.
+
   If the start of the range is not a valid offset for the given
   string or if the range is in reverse order, returns `""`.
 
@@ -2211,11 +2350,6 @@ defmodule String do
   is traversed first in order to convert the negative indices into
   positive ones.
 
-  Remember this function works with Unicode graphemes and considers
-  the slices to represent grapheme offsets. If you want to split
-  on raw bytes, check `Kernel.binary_part/3` or
-  `Kernel.binary_slice/2` instead
-
   ## Examples
 
       iex> String.slice("elixir", 1..3)
@@ -2383,6 +2517,116 @@ defmodule String do
   defp split_bytes([], acc, _), do: {acc, 0}
   defp split_bytes([head | tail], acc, count), do: split_bytes(tail, head + acc, count - 1)
 
+  @doc """
+  Returns a substring starting at (or after) `start_bytes` and of at most
+  the given `size_bytes`.
+
+  This function works on bytes and then adjusts the string to eliminate
+  truncated codepoints. This is useful when you have a string and you need
+  to guarantee it does not exceed a certain amount of bytes.
+
+  If the offset is greater than the number of bytes in the string, then it
+  returns `""`. Similar to `String.slice/2`, a negative `start_bytes`
+  will be adjusted to the end of the string (but in bytes).
+
+  This function does not guarantee the string won't have invalid codepoints,
+  it only guarantees to remove truncated codepoints immediately at the beginning
+  or the end of the slice.
+
+  ## Examples
+
+  Consider the string "héllo". Let's see its representation:
+
+      iex> inspect("héllo", binaries: :as_binaries)
+      "<<104, 195, 169, 108, 108, 111>>"
+
+  Although the string has 5 characters, it is made of 6 bytes. Now imagine
+  we want to get only the first two bytes. To do so, let's use `binary_slice/3`,
+  which is unaware of codepoints:
+
+      iex> binary_slice("héllo", 0, 2)
+      <<104, 195>>
+
+  As you can see, this operation is unsafe and returns an invalid string.
+  That's because we cut the string in the middle of the bytes representing
+  "é". On the other hand, we could use `String.slice/3`:
+
+      iex> String.slice("héllo", 0, 2)
+      "hé"
+
+  While the above is correct, it has 3 bytes. If you have a requirement where
+  you need *at most* 2 bytes, the result would also be invalid. In such scenarios,
+  you can use this function, which will slice the given bytes, but clean up
+  the truncated codepoints:
+
+      iex> String.byte_slice("héllo", 0, 2)
+      "h"
+
+  Truncated codepoints at the beginning are also cleaned up:
+
+      iex> String.byte_slice("héllo", 2, 3)
+      "llo"
+
+  Note that, if you want to work on raw bytes, then you must use `binary_slice/3`
+  instead.
+  """
+  @doc since: "1.17.0"
+  @spec byte_slice(t, integer, non_neg_integer) :: t
+  def byte_slice(string, start_bytes, size_bytes)
+      when is_binary(string) and is_integer(start_bytes) and is_integer(size_bytes) and
+             size_bytes >= 0 do
+    total = byte_size(string)
+    start_bytes = if start_bytes < 0, do: max(total + start_bytes, 0), else: start_bytes
+
+    if start_bytes < total do
+      :erlang.binary_part(string, start_bytes, total - start_bytes)
+      |> invalid_prefix()
+      |> invalid_suffix(size_bytes)
+    else
+      ""
+    end
+  end
+
+  defp invalid_prefix(<<0b10::2, _::6, rest::binary>>), do: invalid_prefix(rest)
+  defp invalid_prefix(rest), do: rest
+
+  defp invalid_suffix(string, size) do
+    last = invalid_suffix(string, min(size, byte_size(string)) - 1, 0)
+    :erlang.binary_part(string, 0, last)
+  end
+
+  defp invalid_suffix(string, last, truncated) when last >= 0 do
+    byte = :binary.at(string, last)
+
+    cond do
+      # ASCII byte, discard all truncated entries
+      byte <= 127 ->
+        last + 1
+
+      # In the middle of a codepoint
+      byte <= 191 ->
+        invalid_suffix(string, last - 1, truncated + 1)
+
+      # 2 bytes codepoint start
+      byte <= 223 ->
+        if truncated == 1, do: last + truncated + 1, else: last
+
+      # 3 bytes codepoint start
+      byte <= 239 ->
+        if truncated == 2, do: last + truncated + 1, else: last
+
+      # 4 bytes codepoint start
+      byte <= 247 ->
+        if truncated == 3, do: last + truncated + 1, else: last
+
+      # Invalid codepoint, discard it, stop checking
+      true ->
+        last + 1
+    end
+  end
+
+  defp invalid_suffix(_string, _last, _truncated), do: 0
+
   @doc """
   Returns `true` if `string` starts with any of the prefixes given.
 
@@ -2834,74 +3078,13 @@ defmodule String do
   @spec jaro_distance(t, t) :: float
   def jaro_distance(string1, string2)
 
-  def jaro_distance(string, string), do: 1.0
+  def jaro_distance(string, string) when is_binary(string), do: 1.0
   def jaro_distance(_string, ""), do: 0.0
   def jaro_distance("", _string), do: 0.0
 
   def jaro_distance(string1, string2) when is_binary(string1) and is_binary(string2) do
-    {chars1, len1} = graphemes_and_length(string1)
-    {chars2, len2} = graphemes_and_length(string2)
-
-    case match(chars1, len1, chars2, len2) do
-      {0, _trans} ->
-        0.0
-
-      {comm, trans} ->
-        (comm / len1 + comm / len2 + (comm - trans) / comm) / 3
-    end
-  end
-
-  defp match(chars1, len1, chars2, len2) do
-    if len1 < len2 do
-      match(chars1, chars2, div(len2, 2) - 1)
-    else
-      match(chars2, chars1, div(len1, 2) - 1)
-    end
-  end
-
-  defp match(chars1, chars2, lim) do
-    match(chars1, chars2, {0, lim}, {0, 0, -1}, 0)
-  end
-
-  defp match([char | rest], chars, range, state, idx) do
-    {chars, state} = submatch(char, chars, range, state, idx)
-
-    case range do
-      {lim, lim} -> match(rest, tl(chars), range, state, idx + 1)
-      {pre, lim} -> match(rest, chars, {pre + 1, lim}, state, idx + 1)
-    end
-  end
-
-  defp match([], _, _, {comm, trans, _}, _), do: {comm, trans}
-
-  defp submatch(char, chars, {pre, _} = range, state, idx) do
-    case detect(char, chars, range) do
-      nil ->
-        {chars, state}
-
-      {subidx, chars} ->
-        {chars, proceed(state, idx - pre + subidx)}
-    end
-  end
-
-  defp detect(char, chars, {pre, lim}) do
-    detect(char, chars, pre + 1 + lim, 0, [])
-  end
-
-  defp detect(_char, _chars, 0, _idx, _acc), do: nil
-  defp detect(_char, [], _lim, _idx, _acc), do: nil
-
-  defp detect(char, [char | rest], _lim, idx, acc), do: {idx, Enum.reverse(acc, [nil | rest])}
-
-  defp detect(char, [other | rest], lim, idx, acc),
-    do: detect(char, rest, lim - 1, idx + 1, [other | acc])
-
-  defp proceed({comm, trans, former}, current) do
-    if current < former do
-      {comm + 1, trans + 1, current}
-    else
-      {comm + 1, trans, current}
-    end
+    # TODO: Replace by :string.jaro_similarity/2 when we require Erlang/OTP 27+
+    :elixir_utils.jaro_similarity(string1, string2)
   end
 
   @doc """
@@ -2936,7 +3119,6 @@ defmodule String do
             codepoint_byte_size: 1,
             grapheme_byte_size: 1,
             grapheme_to_binary: 1,
-            graphemes_and_length: 1,
             reverse_characters_to_binary: 1}
 
   defp byte_size_unicode(binary) when is_binary(binary), do: byte_size(binary)
@@ -2973,22 +3155,6 @@ defmodule String do
   defp grapheme_byte_size([], acc),
     do: acc
 
-  defp graphemes_and_length(string),
-    do: graphemes_and_length(string, [], 0)
-
-  defp graphemes_and_length(string, acc, length) do
-    case :unicode_util.gc(string) do
-      [gc | rest] ->
-        graphemes_and_length(rest, [gc | acc], length + 1)
-
-      [] ->
-        {:lists.reverse(acc), length}
-
-      {:error, <<byte, rest::bits>>} ->
-        graphemes_and_length(rest, [<<byte>> | acc], length + 1)
-    end
-  end
-
   defp reverse_characters_to_binary(acc),
     do: acc |> :lists.reverse() |> :unicode.characters_to_binary()
 end

lib/elixir/lib/string/chars.ex

@@ -57,6 +57,6 @@ end
 
 defimpl String.Chars, for: Float do
   def to_string(term) do
-    IO.iodata_to_binary(:io_lib_format.fwrite_g(term))
+    :erlang.float_to_binary(term, [:short])
   end
 end

lib/elixir/lib/supervisor.ex

@@ -413,10 +413,10 @@ defmodule Supervisor do
   The difference between the two approaches is that a module-based
   supervisor gives you more direct control over how the supervisor
   is initialized. Instead of calling `Supervisor.start_link/2` with
-  a list of child specifications that are automatically initialized, we manually
-  initialize the children by calling `Supervisor.init/2` inside its
-  `c:init/1` callback. `Supervisor.init/2` accepts the same `:strategy`,
-  `:max_restarts`, and `:max_seconds` options as `start_link/2`.
+  a list of child specifications that are implicitly initialized for us,
+  we must explicitly initialize the children by calling `Supervisor.init/2`
+  inside its `c:init/1` callback. `Supervisor.init/2` accepts the same
+  `:strategy`, `:max_restarts`, and `:max_seconds` options as `start_link/2`.
 
   > #### `use Supervisor` {: .info}
   >
@@ -998,7 +998,9 @@ defmodule Supervisor do
   def start_child(supervisor, args) when is_list(args) do
     IO.warn_once(
       {__MODULE__, :start_child},
-      "Supervisor.start_child/2 with a list of args is deprecated, please use DynamicSupervisor instead",
+      fn ->
+        "Supervisor.start_child/2 with a list of args is deprecated, please use DynamicSupervisor instead"
+      end,
       _stacktrace_drop_levels = 2
     )
 

lib/elixir/lib/system.ex

@@ -42,7 +42,7 @@ defmodule System do
     * `system_time/0` - the VM view of the `os_time/0`. The system time and operating
       system time may not match in case of time warps although the VM works towards
       aligning them. This time is not monotonic (i.e., it may decrease)
-      as its behaviour is configured [by the VM time warp
+      as its behavior is configured [by the VM time warp
       mode](https://www.erlang.org/doc/apps/erts/time_correction.html#Time_Warp_Modes);
 
     * `monotonic_time/0` - a monotonically increasing time provided
@@ -305,22 +305,6 @@ defmodule System do
     :elixir_config.get(:no_halt)
   end
 
-  @doc """
-  Waits until the system boots.
-
-  Calling this function blocks until all of ARGV is processed.
-  Inside a release, this means the boot script and then ARGV
-  have been processed. This is only useful for those implementing
-  custom shells/consoles on top of Elixir.
-
-  However, be careful to not invoke this command from within
-  the process that is processing the command line arguments,
-  as doing so would lead to a deadlock.
-  """
-  @doc since: "1.15.0"
-  @spec wait_until_booted() :: :ok
-  defdelegate wait_until_booted(), to: :elixir_config
-
   @doc """
   Current working directory.
 
@@ -522,7 +506,7 @@ defmodule System do
   in case trapping exists is not supported by the current OS.
 
   The first time a signal is trapped, it will override the
-  default behaviour from the operating system. If the same
+  default behavior from the operating system. If the same
   signal is trapped multiple times, subsequent functions
   given to `trap_signal` will execute *first*. In other
   words, you can consider each function is prepended to
@@ -536,7 +520,7 @@ defmodule System do
     * `:sigusr1` - halts the VM via status code of 1
 
   Therefore, if you add traps to the signals above, the
-  default behaviour above will be executed after all user
+  default behavior above will be executed after all user
   signals.
 
   ## Implementation notes
@@ -955,6 +939,12 @@ defmodule System do
   @doc since: "1.12.0"
   @spec shell(binary, keyword) :: {Collectable.t(), exit_status :: non_neg_integer}
   def shell(command, opts \\ []) when is_binary(command) do
+    command |> String.trim() |> do_shell(opts)
+  end
+
+  defp do_shell("", _opts), do: {"", 0}
+
+  defp do_shell(command, opts) do
     assert_no_null_byte!(command, "System.shell/2")
     {close_stdin?, opts} = Keyword.pop(opts, :close_stdin, false)
 
@@ -1005,6 +995,25 @@ defmodule System do
   `Port` module describes this problem and possible solutions under
   the "Zombie processes" section.
 
+  > #### Windows argument splitting and untrusted arguments {: .warning}
+  >
+  > On Unix systems, arguments are passed to a new operating system
+  > process as an array of strings but on Windows it is up to the child
+  > process to parse them and some Windows programs may apply their own
+  > rules, which are inconsistent with the standard C runtime `argv` parsing
+  >
+  > This is particularly troublesome when invoking `.bat` or `.com` files
+  > as these run implicitly through `cmd.exe`, whose argument parsing is
+  > vulnerable to malicious input and can be used to run arbitrary shell
+  > commands.
+  >
+  > Therefore, if you are running on Windows and you execute batch
+  > files or `.com` applications, you must not pass untrusted input as
+  > arguments to the program. You may avoid accidentally executing them
+  > by explicitly passing the extension of the program you want to run,
+  > such as `.exe`, and double check the program is indeed not a batch
+  > file or `.com` application.
+
   ## Examples
 
       iex> System.cmd("echo", ["hello"])
@@ -1044,7 +1053,11 @@ defmodule System do
 
     * `:arg0` - sets the command arg0
 
-    * `:stderr_to_stdout` - redirects stderr to stdout when `true`
+    * `:stderr_to_stdout` - redirects stderr to stdout when `true`, no effect
+      if `use_stdio` is `false`.
+
+    * `:use_stdio` - `true` by default, setting it to false allows direct
+      interaction with the terminal from the callee
 
     * `:parallelism` - when `true`, the VM will schedule port tasks to improve
       parallelism in the system. If set to `false`, the VM will try to perform
@@ -1105,8 +1118,10 @@ defmodule System do
   end
 
   defp do_cmd(port_init, base_opts, opts) do
+    {use_stdio?, opts} = Keyword.pop(opts, :use_stdio, true)
+
     {into, line, opts} =
-      cmd_opts(opts, [:use_stdio, :exit_status, :binary, :hide] ++ base_opts, "", false)
+      cmd_opts(opts, [:exit_status, :binary, :hide] ++ base_opts, "", false, use_stdio?)
 
     {initial, fun} = Collectable.into(into)
 
@@ -1155,36 +1170,41 @@ defmodule System do
     end
   end
 
-  defp cmd_opts([{:into, any} | t], opts, _into, line),
-    do: cmd_opts(t, opts, any, line)
+  defp cmd_opts([{:into, any} | t], opts, _into, line, stdio?),
+    do: cmd_opts(t, opts, any, line, stdio?)
 
-  defp cmd_opts([{:cd, bin} | t], opts, into, line) when is_binary(bin),
-    do: cmd_opts(t, [{:cd, bin} | opts], into, line)
+  defp cmd_opts([{:cd, bin} | t], opts, into, line, stdio?) when is_binary(bin),
+    do: cmd_opts(t, [{:cd, bin} | opts], into, line, stdio?)
 
-  defp cmd_opts([{:arg0, bin} | t], opts, into, line) when is_binary(bin),
-    do: cmd_opts(t, [{:arg0, bin} | opts], into, line)
+  defp cmd_opts([{:arg0, bin} | t], opts, into, line, stdio?) when is_binary(bin),
+    do: cmd_opts(t, [{:arg0, bin} | opts], into, line, stdio?)
 
-  defp cmd_opts([{:stderr_to_stdout, true} | t], opts, into, line),
-    do: cmd_opts(t, [:stderr_to_stdout | opts], into, line)
+  defp cmd_opts([{:stderr_to_stdout, true} | t], opts, into, line, true),
+    do: cmd_opts(t, [:stderr_to_stdout | opts], into, line, true)
 
-  defp cmd_opts([{:stderr_to_stdout, false} | t], opts, into, line),
-    do: cmd_opts(t, opts, into, line)
+  defp cmd_opts([{:stderr_to_stdout, true} | _], _opts, _into, _line, false),
+    do: raise(ArgumentError, "cannot use \"stderr_to_stdout: true\" and \"use_stdio: false\"")
 
-  defp cmd_opts([{:parallelism, bool} | t], opts, into, line) when is_boolean(bool),
-    do: cmd_opts(t, [{:parallelism, bool} | opts], into, line)
+  defp cmd_opts([{:stderr_to_stdout, false} | t], opts, into, line, stdio?),
+    do: cmd_opts(t, opts, into, line, stdio?)
 
-  defp cmd_opts([{:env, enum} | t], opts, into, line),
-    do: cmd_opts(t, [{:env, validate_env(enum)} | opts], into, line)
+  defp cmd_opts([{:parallelism, bool} | t], opts, into, line, stdio?) when is_boolean(bool),
+    do: cmd_opts(t, [{:parallelism, bool} | opts], into, line, stdio?)
 
-  defp cmd_opts([{:lines, max_line_length} | t], opts, into, _line)
+  defp cmd_opts([{:env, enum} | t], opts, into, line, stdio?),
+    do: cmd_opts(t, [{:env, validate_env(enum)} | opts], into, line, stdio?)
+
+  defp cmd_opts([{:lines, max_line_length} | t], opts, into, _line, stdio?)
        when is_integer(max_line_length) and max_line_length > 0,
-       do: cmd_opts(t, [{:line, max_line_length} | opts], into, true)
+       do: cmd_opts(t, [{:line, max_line_length} | opts], into, true, stdio?)
 
-  defp cmd_opts([{key, val} | _], _opts, _into, _line),
+  defp cmd_opts([{key, val} | _], _opts, _into, _line, _stdio?),
     do: raise(ArgumentError, "invalid option #{inspect(key)} with value #{inspect(val)}")
 
-  defp cmd_opts([], opts, into, line),
-    do: {into, line, opts}
+  defp cmd_opts([], opts, into, line, stdio?) do
+    opt = if stdio?, do: :use_stdio, else: :nouse_stdio
+    {into, line, [opt | opts]}
+  end
 
   defp validate_env(enum) do
     Enum.map(enum, fn
@@ -1422,8 +1442,10 @@ defmodule System do
   defp warn(unit, replacement_unit) do
     IO.warn_once(
       {__MODULE__, unit},
-      "deprecated time unit: #{inspect(unit)}. A time unit should be " <>
-        ":second, :millisecond, :microsecond, :nanosecond, or a positive integer",
+      fn ->
+        "deprecated time unit: #{inspect(unit)}. A time unit should be " <>
+          ":second, :millisecond, :microsecond, :nanosecond, or a positive integer"
+      end,
       _stacktrace_drop_levels = 4
     )
 

lib/elixir/lib/task.ex

@@ -40,14 +40,42 @@ defmodule Task do
        as they are *always* sent. If you are not expecting a reply,
        consider using `Task.start_link/1` as detailed below.
 
-    2. async tasks link the caller and the spawned process. This
+    2. Async tasks link the caller and the spawned process. This
        means that, if the caller crashes, the task will crash
        too and vice-versa. This is on purpose: if the process
        meant to receive the result no longer exists, there is
-       no purpose in completing the computation.
+       no purpose in completing the computation. If this is not
+       desired, you will want to use supervised tasks, described
+       in a subsequent section.
 
-       If this is not desired, you will want to use supervised
-       tasks, described next.
+  ## Tasks are processes
+
+  Tasks are processes and so data will need to be completely copied
+  to them. Take the following code as an example:
+
+      large_data = fetch_large_data()
+      task = Task.async(fn -> do_some_work(large_data) end)
+      res = do_some_other_work()
+      res + Task.await(task)
+
+  The code above copies over all of `large_data`, which can be
+  resource intensive depending on the size of the data.
+  There are two ways to address this.
+
+  First, if you need to access only part of `large_data`,
+  consider extracting it before the task:
+
+      large_data = fetch_large_data()
+      subset_data = large_data.some_field
+      task = Task.async(fn -> do_some_work(subset_data) end)
+
+  Alternatively, if you can move the data loading altogether
+  to the task, it may be even better:
+
+      task = Task.async(fn ->
+        large_data = fetch_large_data()
+        do_some_work(large_data)
+      end)
 
   ## Dynamically supervised tasks
 
@@ -107,14 +135,14 @@ defmodule Task do
 
   With `Task.Supervisor`, it is easy to dynamically start tasks across nodes:
 
-      # On the remote node named :remote@local
+      # First on the remote node named :remote@local
       Task.Supervisor.start_link(name: MyApp.DistSupervisor)
 
-      # On the client
+      # Then on the local client node
       supervisor = {MyApp.DistSupervisor, :remote@local}
       Task.Supervisor.async(supervisor, MyMod, :my_fun, [arg1, arg2, arg3])
 
-  Note that, when working with distributed tasks, one should use the
+  Note that, as above, when working with distributed tasks, one should use the
   `Task.Supervisor.async/5` function that expects explicit module, function,
   and arguments, instead of `Task.Supervisor.async/3` that works with anonymous
   functions. That's because anonymous functions expect the same module version
@@ -265,6 +293,17 @@ defmodule Task do
   """
   @opaque ref :: reference()
 
+  @typedoc """
+  Options given to `async_stream` functions.
+  """
+  @typedoc since: "1.17.0"
+  @type async_stream_option ::
+          {:max_concurrency, pos_integer()}
+          | {:ordered, boolean()}
+          | {:timeout, timeout()}
+          | {:on_timeout, :exit | :kill_task}
+          | {:zip_input_on_exit, boolean()}
+
   defguardp is_timeout(timeout)
             when timeout == :infinity or (is_integer(timeout) and timeout >= 0)
 
@@ -653,7 +692,8 @@ defmodule Task do
   example above.
   """
   @doc since: "1.4.0"
-  @spec async_stream(Enumerable.t(), module, atom, [term], keyword) :: Enumerable.t()
+  @spec async_stream(Enumerable.t(), module, atom, [term], [async_stream_option]) ::
+          Enumerable.t()
   def async_stream(enumerable, module, function_name, args, options \\ [])
       when is_atom(module) and is_atom(function_name) and is_list(args) do
     build_stream(enumerable, {module, function_name, args}, options)
@@ -682,13 +722,15 @@ defmodule Task do
   See `async_stream/5` for discussion, options, and more examples.
   """
   @doc since: "1.4.0"
-  @spec async_stream(Enumerable.t(), (term -> term), keyword) :: Enumerable.t()
+  @spec async_stream(Enumerable.t(), (term -> term), [async_stream_option]) :: Enumerable.t()
   def async_stream(enumerable, fun, options \\ [])
       when is_function(fun, 1) and is_list(options) do
     build_stream(enumerable, fun, options)
   end
 
   defp build_stream(enumerable, fun, options) do
+    options = Task.Supervised.validate_stream_options(options)
+
     fn acc, acc_fun ->
       owner = get_owner(self())
 
@@ -1153,7 +1195,7 @@ defmodule Task do
   given time. All other tasks will have been shut down using
   the `Task.shutdown/2` call.
 
-  As a convenience, you can achieve a similar behaviour to above
+  As a convenience, you can achieve a similar behavior to above
   by specifying the `:on_timeout` option to be `:kill_task` (or
   `:ignore`). See `Task.await_many/2` if you would rather exit
   the caller process on timeout.

lib/elixir/lib/task/supervised.ex

@@ -117,7 +117,9 @@ defmodule Task.Supervised do
               starter: get_from(owner),
               function: fun,
               args: args,
-              reason: {log_value(kind, value), __STACKTRACE__}
+              reason: {log_value(kind, value), __STACKTRACE__},
+              # TODO use Process.get_label/0 when we require Erlang/OTP 27+
+              process_label: Process.get(:"$process_label", :undefined)
             }
           },
           %{
@@ -147,16 +149,27 @@ defmodule Task.Supervised do
           starter: starter,
           function: fun,
           args: args,
-          reason: reason
+          reason: reason,
+          process_label: process_label
         }
       }) do
     message =
-      ~c"** Task ~p terminating~n" ++
-        ~c"** Started from ~p~n" ++
+      ~c"** Started from ~p~n" ++
         ~c"** When function  == ~p~n" ++
         ~c"**      arguments == ~p~n" ++ ~c"** Reason for termination == ~n" ++ ~c"** ~p~n"
 
-    {message, [starter, name, fun, args, get_reason(reason)]}
+    terms = [name, fun, args, get_reason(reason)]
+
+    {message, terms} =
+      case process_label do
+        :undefined -> {message, terms}
+        _ -> {~c"** Process Label == ~p~n" ++ message, [process_label | terms]}
+      end
+
+    message =
+      ~c"** Task ~p terminating~n" ++ message
+
+    {message, [starter | terms]}
   end
 
   defp get_from({node, pid_or_name, _pid}) when node == node(), do: pid_or_name
@@ -189,18 +202,36 @@ defmodule Task.Supervised do
 
   ## Stream
 
-  def stream(enumerable, acc, reducer, callers, mfa, options, spawn) do
-    next = &Enumerable.reduce(enumerable, &1, fn x, acc -> {:suspend, [x | acc]} end)
-    max_concurrency = Keyword.get(options, :max_concurrency, System.schedulers_online())
+  def validate_stream_options(options) do
+    max_concurrency = Keyword.get_lazy(options, :max_concurrency, &System.schedulers_online/0)
+    on_timeout = Keyword.get(options, :on_timeout, :exit)
+    timeout = Keyword.get(options, :timeout, 5000)
+    ordered = Keyword.get(options, :ordered, true)
+    zip_input_on_exit = Keyword.get(options, :zip_input_on_exit, false)
 
     unless is_integer(max_concurrency) and max_concurrency > 0 do
       raise ArgumentError, ":max_concurrency must be an integer greater than zero"
     end
 
-    ordered? = Keyword.get(options, :ordered, true)
-    timeout = Keyword.get(options, :timeout, 5000)
-    on_timeout = Keyword.get(options, :on_timeout, :exit)
-    zip_input_on_exit? = Keyword.get(options, :zip_input_on_exit, false)
+    unless on_timeout in [:exit, :kill_task] do
+      raise ArgumentError, ":on_timeout must be either :exit or :kill_task"
+    end
+
+    unless (is_integer(timeout) and timeout >= 0) or timeout == :infinity do
+      raise ArgumentError, ":timeout must be either a positive integer or :infinity"
+    end
+
+    %{
+      max_concurrency: max_concurrency,
+      on_timeout: on_timeout,
+      timeout: timeout,
+      ordered: ordered,
+      zip_input_on_exit: zip_input_on_exit
+    }
+  end
+
+  def stream(enumerable, acc, reducer, callers, mfa, options, spawn) when is_map(options) do
+    next = &Enumerable.reduce(enumerable, &1, fn x, acc -> {:suspend, [x | acc]} end)
     parent = self()
 
     {:trap_exit, trap_exit?} = Process.info(self(), :trap_exit)
@@ -212,7 +243,7 @@ defmodule Task.Supervised do
 
     {monitor_pid, monitor_ref} =
       Process.spawn(
-        fn -> stream_monitor(parent, spawn, trap_exit?, timeout) end,
+        fn -> stream_monitor(parent, spawn, trap_exit?, options.timeout) end,
         spawn_opts
       )
 
@@ -221,21 +252,21 @@ defmodule Task.Supervised do
     # about our reference to it.
     send(monitor_pid, {parent, monitor_ref})
 
-    config = %{
-      reducer: reducer,
-      monitor_pid: monitor_pid,
-      monitor_ref: monitor_ref,
-      ordered: ordered?,
-      timeout: timeout,
-      on_timeout: on_timeout,
-      zip_input_on_exit: zip_input_on_exit?,
-      callers: callers,
-      mfa: mfa
-    }
+    config =
+      Map.merge(
+        options,
+        %{
+          reducer: reducer,
+          monitor_pid: monitor_pid,
+          monitor_ref: monitor_ref,
+          callers: callers,
+          mfa: mfa
+        }
+      )
 
     stream_reduce(
       acc,
-      max_concurrency,
+      options.max_concurrency,
       _spawned = 0,
       _delivered = 0,
       _waiting = %{},

lib/elixir/lib/task/supervisor.ex

@@ -77,6 +77,12 @@ defmodule Task.Supervisor do
           DynamicSupervisor.option()
           | DynamicSupervisor.init_option()
 
+  @typedoc """
+  Options given to `async_stream` and `async_stream_nolink` functions.
+  """
+  @typedoc since: "1.17.0"
+  @type async_stream_option :: Task.async_stream_option() | {:shutdown, Supervisor.shutdown()}
+
   @doc false
   def child_spec(opts) when is_list(opts) do
     id =
@@ -152,7 +158,7 @@ defmodule Task.Supervisor do
   Starts a task that can be awaited on.
 
   The `supervisor` must be a reference as defined in `Supervisor`.
-  The task will still be linked to the caller, see `Task.async/3` for
+  The task will still be linked to the caller, see `Task.async/1` for
   more information and `async_nolink/3` for a non-linked variant.
 
   Raises an error if `supervisor` has reached the maximum number of
@@ -174,7 +180,7 @@ defmodule Task.Supervisor do
   Starts a task that can be awaited on.
 
   The `supervisor` must be a reference as defined in `Supervisor`.
-  The task will still be linked to the caller, see `Task.async/3` for
+  The task will still be linked to the caller, see `Task.async/1` for
   more information and `async_nolink/3` for a non-linked variant.
 
   Raises an error if `supervisor` has reached the maximum number of
@@ -196,7 +202,7 @@ defmodule Task.Supervisor do
   Starts a task that can be awaited on.
 
   The `supervisor` must be a reference as defined in `Supervisor`.
-  The task won't be linked to the caller, see `Task.async/3` for
+  The task won't be linked to the caller, see `Task.async/1` for
   more information.
 
   Raises an error if `supervisor` has reached the maximum number of
@@ -283,7 +289,7 @@ defmodule Task.Supervisor do
   Starts a task that can be awaited on.
 
   The `supervisor` must be a reference as defined in `Supervisor`.
-  The task won't be linked to the caller, see `Task.async/3` for
+  The task won't be linked to the caller, see `Task.async/1` for
   more information.
 
   Raises an error if `supervisor` has reached the maximum number of
@@ -356,8 +362,14 @@ defmodule Task.Supervisor do
 
   """
   @doc since: "1.4.0"
-  @spec async_stream(Supervisor.supervisor(), Enumerable.t(), module, atom, [term], keyword) ::
-          Enumerable.t()
+  @spec async_stream(
+          Supervisor.supervisor(),
+          Enumerable.t(),
+          module,
+          atom,
+          [term],
+          [async_stream_option]
+        ) :: Enumerable.t()
   def async_stream(supervisor, enumerable, module, function, args, options \\ [])
       when is_atom(module) and is_atom(function) and is_list(args) do
     build_stream(supervisor, :link, enumerable, {module, function, args}, options)
@@ -374,8 +386,12 @@ defmodule Task.Supervisor do
   See `async_stream/6` for discussion, options, and examples.
   """
   @doc since: "1.4.0"
-  @spec async_stream(Supervisor.supervisor(), Enumerable.t(), (term -> term), keyword) ::
-          Enumerable.t()
+  @spec async_stream(
+          Supervisor.supervisor(),
+          Enumerable.t(),
+          (term -> term),
+          [async_stream_option]
+        ) :: Enumerable.t()
   def async_stream(supervisor, enumerable, fun, options \\ []) when is_function(fun, 1) do
     build_stream(supervisor, :link, enumerable, fun, options)
   end
@@ -397,14 +413,14 @@ defmodule Task.Supervisor do
           module,
           atom,
           [term],
-          keyword
+          [async_stream_option]
         ) :: Enumerable.t()
   def async_stream_nolink(supervisor, enumerable, module, function, args, options \\ [])
       when is_atom(module) and is_atom(function) and is_list(args) do
     build_stream(supervisor, :nolink, enumerable, {module, function, args}, options)
   end
 
-  @doc """
+  @doc ~S"""
   Returns a stream that runs the given `function` concurrently on each
   element in `enumerable`.
 
@@ -414,10 +430,46 @@ defmodule Task.Supervisor do
   to `async_nolink/3`.
 
   See `async_stream/6` for discussion and examples.
+
+  ## Error handling and cleanup
+
+  Even if tasks are not linked to the caller, there is no risk of leaving dangling tasks
+  running after the stream halts.
+
+  Consider the following example:
+
+      Task.Supervisor.async_stream_nolink(MySupervisor, collection, fun, on_timeout: :kill_task, ordered: false)
+      |> Enum.each(fn
+        {:ok, _} -> :ok
+        {:exit, reason} -> raise "Task exited: #{Exception.format_exit(reason)}"
+      end)
+
+  If one task raises or times out:
+
+    1. the second clause gets called
+    2. an exception is raised
+    3. the stream halts
+    4. all ongoing tasks will be shut down
+
+  Here is another example:
+
+      Task.Supervisor.async_stream_nolink(MySupervisor, collection, fun, on_timeout: :kill_task, ordered: false)
+      |> Stream.filter(&match?({:ok, _}, &1))
+      |> Enum.take(3)
+
+  This will return the three first tasks to succeed, ignoring timeouts and errors, and shut down
+  every ongoing task.
+
+  Just running the stream with `Stream.run/1` on the other hand would ignore errors and process the whole stream.
+
   """
   @doc since: "1.4.0"
-  @spec async_stream_nolink(Supervisor.supervisor(), Enumerable.t(), (term -> term), keyword) ::
-          Enumerable.t()
+  @spec async_stream_nolink(
+          Supervisor.supervisor(),
+          Enumerable.t(),
+          (term -> term),
+          [async_stream_option]
+        ) :: Enumerable.t()
   def async_stream_nolink(supervisor, enumerable, fun, options \\ []) when is_function(fun, 1) do
     build_stream(supervisor, :nolink, enumerable, fun, options)
   end
@@ -560,8 +612,15 @@ defmodule Task.Supervisor do
   end
 
   defp build_stream(supervisor, link_type, enumerable, fun, options) do
+    shutdown = Keyword.get(options, :shutdown, 5000)
+
+    unless (is_integer(shutdown) and shutdown >= 0) or shutdown == :brutal_kill do
+      raise ArgumentError, ":shutdown must be either a positive integer or :brutal_kill"
+    end
+
+    options = Task.Supervised.validate_stream_options(options)
+
     fn acc, acc_fun ->
-      shutdown = options[:shutdown]
       owner = get_owner(self())
 
       Task.Supervised.stream(enumerable, acc, acc_fun, get_callers(self()), fun, options, fn ->

lib/elixir/lib/uri.ex

@@ -5,6 +5,9 @@ defmodule URI do
   This module provides functions for working with URIs (for example, parsing
   URIs or encoding query strings). The functions in this module are implemented
   according to [RFC 3986](https://tools.ietf.org/html/rfc3986).
+
+  Additionally, the Erlang [`:uri_string` module](`:uri_string`) provides certain functionalities,
+  such as RFC 3986 compliant URI normalization.
   """
 
   @doc """
@@ -235,7 +238,7 @@ defmodule URI do
 
       {{key, value}, rest} ->
         # Avoid warnings about Dict being deprecated
-        dict_module = Dict
+        dict_module = String.to_atom("Dict")
         decode_query_into_dict(rest, dict_module.put(dict, key, value), encoding)
     end
   end
@@ -362,22 +365,24 @@ defmodule URI do
   end
 
   @doc """
-  Percent-escapes all characters that require escaping in `string`.
+  Percent-encodes all characters that require escaping in `string`.
 
-  This means reserved characters, such as `:` and `/`, and the
-  so-called unreserved characters, which have the same meaning both
-  escaped and unescaped, won't be escaped by default.
+  By default, this function is meant to escape the whole URI, and
+  therefore it will only escape characters which are foreign in
+  all parts of a URI. Reserved characters (such as `:` and `/`)
+  or unreserved (such as letters and numbers) are not escaped.
+
+  Because different components of a URI require different escaping
+  rules, this function also accepts a `predicate` function as an optional
+  argument. If passed, this function will be called with each byte
+  in `string` as its argument and should return a truthy value (anything other
+  than `false` or `nil`) if the given byte should be left as is, or
+  return a falsy value (`false` or `nil`) if the character should be
+  escaped. Defaults to `URI.char_unescaped?/1`.
 
   See `encode_www_form/1` if you are interested in escaping reserved
   characters too.
 
-  This function also accepts a `predicate` function as an optional
-  argument. If passed, this function will be called with each byte
-  in `string` as its argument and should return a truthy value (anything other
-  than `false` or `nil`) if the given byte should be left as is, or return a
-  falsy value (`false` or `nil`) if the character should be escaped. Defaults
-  to `URI.char_unescaped?/1`.
-
   ## Examples
 
       iex> URI.encode("ftp://s-ite.tld/?value=put it+й")
@@ -654,16 +659,16 @@ defmodule URI do
         scheme = String.downcase(scheme, :ascii)
 
         case map do
-          %{port: port} when port != :undefined ->
+          %{port: port} when is_integer(port) ->
             %{uri | scheme: scheme}
 
           %{} ->
-            case default_port(scheme) do
-              nil -> %{uri | scheme: scheme}
-              port -> %{uri | scheme: scheme, port: port}
-            end
+            %{uri | scheme: scheme, port: default_port(scheme)}
         end
 
+      %{port: :undefined} ->
+        %{uri | port: nil}
+
       %{} ->
         uri
     end