Release v1.5.3

Signed-off-by: José Valim <jose.valim@plataformatec.com.br>

.travis.yml

@@ -23,7 +23,14 @@ matrix:
     - os: linux
       otp_release: 20.0
 
-script: "make compile && rm -rf .git && make test"
+env:
+  - ELIXIR_ASSERT_TIMEOUT=2000
+
+script:
+  - make compile
+  - rm -rf .git
+  - make test
+  - dialyzer -pa lib/elixir/ebin --build_plt --output_plt elixir.plt --apps lib/elixir/ebin/elixir.beam lib/elixir/ebin/Elixir.Kernel.beam
 
 notifications:
   recipients:

CHANGELOG.md

@@ -14,10 +14,42 @@ Or:
 
     saudação = "Bom dia!"
 
-Elixir follows the recommendations in [Unicode Annex #31](http://unicode.org/reports/tr31/) to make the language more accessible to other languages and communities. Identifiers must still be a sequence of letters, followed by digits and combining marks. This means symbols, such as mathematical notations and emoji, are not allowed identifiers.
+Elixir follows the recommendations in [Unicode Annex #31](http://unicode.org/reports/tr31/) to make the language more accessible to other languages and communities. Identifiers must start with a letter or underscore, optionally followed by letters, digits, and underscores. Here letter means any UTF-8 letter-character (optionally with a combining mark) and digit means a UTF-8 decimal-digit-character. If you're using ASCII, this does what you'd expect.
+
+Examples of valid variables are:
+
+    name    josé   _age   まつもと  _42  адрес
+    
+Examples of invalid variables include:
+
+    name•   a±2  42
+
+Symbols, such as mathematical notations and emoji, are not allowed identifiers.
 
 For a complete reference on Elixir syntax, see the [Syntax Reference](https://hexdocs.pm/elixir/syntax-reference.html). For technical details on Unicode support, see [Unicode Syntax](https://hexdocs.pm/elixir/unicode-syntax.html).
 
+## IEx improvements
+
+IEx got many improvements. The autocompletion system is now capable of autocompleting variables and user imports. New helpers have also been added:
+
+  * `exports/1` lists all exports (functions and macros) in a given module
+  * `open/1` opens up the source of a module or function directly in your editor. For example, `open MyApp.Module`
+  * `runtime_info/0` prints general information about the running system, such as number of cores, runtime version, allocation of memory in the VM and more
+
+IEx also features a breakpoint system for code debugging. The following functions have been added to aid debugging:
+
+  * `break!/2` - sets up a breakpoint for a given `Mod.fun/arity`
+  * `break!/4` - sets up a breakpoint for the given module, function, arity
+  * `breaks/0` - prints all breakpoints and their ids
+  * `continue/0` - continues until the next breakpoint in the same process
+  * `open/0` - opens editor on the current breakpoint
+  * `remove_breaks/0` - removes all breakpoints in all modules
+  * `remove_breaks/1` - removes all breakpoints in a given module
+  * `reset_break/1` - sets the number of stops on the given id to zero
+  * `reset_break/3` - sets the number of stops on the given module, function, arity to zer
+  * `respawn/0` - starts a new shell (breakpoints will ask for permission once more)
+  * `whereami/1` - shows the current location
+
 ## Exception.blame
 
 `Exception.blame/3` is a new function in Elixir that is capable of attaching debug information to certain exceptions. Currently this is used to augment `FunctionClauseError`s with a summary of all clauses and which parts of clause match and which ones didn't. For example:
@@ -134,38 +166,128 @@ Overall, using `@impl` has the following advantages:
 
 This release brings further improvements to Calendar types. It adds arithmetic and others functions to `Time`, `Date`, `NaiveDateTime` and `Datetime` as well as conversion between different calendars.
 
-## v1.5.0-dev
+## v1.5.3 (2017-12-19)
+
+### 1. Bug fixes
+
+#### Elixir
+
+  * [Calendar] Consider microseconds in maximum possible Calendar.ISO datetime
+  * [Enum] Fix `chunk_every/4` when `step > count`
+  * [Kernel] Warn duplicate definitions in macros
+  * [Kernel] Remove dialyzer warnings from else in with clauses
+  * [Kernel] Do not warning on upcoming `@deprecated` and `@since` attributes for v1.6
+  * [MapSet] Return valid MapSet when union-ing a legacy MapSet
+
+#### ExUnit
+
+  * [ExUnit] Fix compiler warnings in `assert_receive/3`
+
+## v1.5.2 (2017-09-29)
+
+### 1. Enhacements
+
+#### Elixir
+
+  * [Kernel] Optimize function definition with multiple clauses by not traversing the internal clauses table
+  * [Kernel] Warn if unary operators are followed by new lines
+  * [Registry] Use the name of the Registry as its `:id` in the `child_spec/1` function
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [DateTime] Fix negative microsecond result when passing negative Unix epochs to `from_unix/2`
+  * [Kernel] Improve error message for oversized atoms
+  * [Kernel] Ensure `@impl` attribute also propagates to clauses from default arguments
+  * [Kernel] Emit proper error for unknown vars inside binary pattern in match
+
+#### IEx
+
+  * [IEx] Do not crash IEx unexpectedly on `System.stop/0`
+  * [IEx.Helpers] Ensure exiting a breakpoint set inside a breakpoint does not terminate the shell unexpectedly
+
+#### Mix
+
+  * [mix local.hex] Ensure `--if-missing` flag works as advertised
+  * [mix test] Do not trigger additional error reports when there is a failure when loading test files
+  * [Mix.SCM.Git] Ensure errors when invoking `git` propagate correctly
+
+## v1.5.1 (2017-08-01)
+
+### 1. Enhancements
+
+#### EEx
+
+  * [EEx.Engine] Add `handle_begin` and `handle_end` to EEx
+
+#### Elixir
+
+  * [Kernel] Do not use references on function/macro definitions - this provides large improvements in compilation times in some rare corner cases
+  * [Supervisor] Support mixing old and new typespecs in `Supervisor.init/2` and `Supevisor.start_link/2`
+
+#### Mix
+
+  * [mix profile] Allow profile tasks to run without a project
+
+### 2. Bug fixes
+
+#### EEx
+
+  * [EEx.Engine] Do not re-use the value of the `init/1` callback throughout the compilation stack
+
+#### Elixir
+
+  * [Kernel] Ensure dialyzer does not emit warnings in some uses of `with`
+  * [Kernel] Fix dialyzer warnings when `defmacrop` is used in modules
+  * [Kernel] Ensure Elixir modules can be dialyzed without starting the Elixir application
+  * [Kernel] Do not serialize references in quoted expressions
+  * [Kernel] Make sure structs expansion use the latest definition available when struct modules are recompiled
+  * [Task] Support `:infinity` timeout on Task streams
+  * [Typespec] Ensure typespecs allow `tuple` to be used as variable names
+
+## v1.5.0 (2017-07-25)
 
 ### 1. Enhancements
 
 #### Elixir
 
-  * [Access] Optimize Access.get/2
-  * [Calendar] Limit `Calendar.ISO` up to year 10000
-  * [Calendar] Add Rata Die format for conversions between Calendars and `Date.convert/2`, `Time.convert/2`, `NaiveDateTime.convert/2` and `DateTime.convert/2` (as well as bang variants)
+  * [Access] Optimize `Access.get/2`
+  * [Base] Optimise Base encode/decode
+  * [Calendar] Implement Inspect for DateTime with Calendar.ISO
+  * [Calendar] Add "ISO days" format for conversions between Calendars and `Date.convert/2`, `Time.convert/2`, `NaiveDateTime.convert/2` and `DateTime.convert/2` (as well as bang variants)
   * [Calendar] Add `:calendar` field to `Time` struct
   * [Calendar] Add `Time.diff/3`, `Date.add/2`, `Date.diff/2`, `DateTime.diff/3`
   * [Calendar] Add `Date.range/2`
+  * [Calendar] Add `Date.new/4`, `DateTime.utc_now/1`, `NaiveDateTime.new/8` and `Time.new/5` that allow specifing calendar
   * [Enum] Add `Enum.chunk_by/4` and `Stream.chunk_by/4`
+  * [Enum] Add `Enum.chunk_every/2` and `Enum.chunk_every/4` with a more explicit API than `Enum.chunk/2` and `Enum.chunk/4`
   * [Exception] Add `Exception.blame/3` that adds metadata to exceptions
   * [File] Add `File.read_link/1` and `File.read_link!/1`
   * [File] Introduce `:trim_bom` option for `File.stream!/2`
   * [Inspect] Add `:printable_limit` to control the limit of printable structures
   * [Integer] Add `Integer.gcd/2`
+  * [Kernel] Add `left not in right` to check that the left side is not in the enumerable on the right
   * [Kernel] Use the new `debug_info` chunk in OTP 20. This provides a mechanism for tools to retrieve the Elixir AST from beam files
   * [Kernel] `defoverridable/1` accepts a module name as argument and marks all callbacks as overridable
   * [Kernel] Allow non-quoted Unicode atoms and variables according to Unicode Annex #31 (see Unicode Syntax document)
   * [Kernel] Warn when a `:__struct__` key is used when building/updating structs
+  * [Kernel] Cache the AST on definitions. This speeds up the compilation time from 10% to 15% measured across different projects
+  * [Kernel] Improve compiler error message on invalid patterns and guards
   * [Keyword] Add `replace/3` and `replace!/3` for replacing an existing key
   * [List] `List.starts_with?/2`
   * [Macro] Introduce `Macro.generate_arguments/2`
   * [Map] Optimize `Map.merge/3` by choosing merging direction
   * [Map] Add `replace/3` and `replace!/3` for replacing an existing key
+  * [Map] Raise `BadMapError` in `Map.equal?/2` when either of the two arguments is not a map
   * [MapSet] Reduce `MapSet` size when serialized to approximately half
   * [Process] Add `Process.cancel_timer/2`
   * [Protocol] Show available implementations on `Protocol.UndefinedError` if the protocol has been consolidated
   * [Registry] Support ETS guard conditions in `Registry.match/3`
   * [Registry] Support `parallel: true` in `Registry.dispatch/3`
+  * [Registry] Introduce `Registry.unregister_match/4`
+  * [Stream] Add `Stream.chunk_every/2` and `Stream.chunk_every/4` with a more explicit API than `Stream.chunk/2` and `Stream.chunk/4`
+  * [String] Optimise binary pattern matching in `String.split/1` and `String.trim_*/1`
   * [Supervisor] Add `Supervisor.init/2` and `Supervisor.child_spec/2`
   * [Supervisor] Allow `module` and `{module, arg}` to be given to `Supervisor.start_link/2` and invoke `module.child_spec(arg)` on each argument
   * [Task] Support `:on_timeout` in `Task.async_stream` to control how tasks are terminated
@@ -174,15 +296,20 @@ This release brings further improvements to Calendar types. It adds arithmetic a
 #### ExUnit
 
   * [ExUnit] Show code snippet from test source file in case of test errors
-  * [ExUnit] Show the value of variables used in an assertion
   * [ExUnit] Use `Exception.blame/3` when formatting test errors
   * [ExUnit] Make `assert_raise/2` fail if the underlying exception has a broken `message/1` implementation
   * [ExUnit] Add `start_supervised/2` and `stop_supervised/1` to ExUnit. Processes started by this function are automatically shut down when the test exits
 
 #### IEx
 
+  * [IEx.Autocomplete] Support autocompletion of variable names
+  * [IEx.Autocomplete] Support autocompletion of functions imported using `import Mod, only: [...]`
   * [IEx.Evaluator] Use `Exception.blame/3` when showing errors in the terminal
-  * [IEx.Helpers] Add `e/1` IEx helper to list all exports in a module
+  * [IEx.Helpers] Add `exports/1` IEx helper to list all exports in a module
+  * [IEx.Helpers] Add `break!/2`, `break!/4`, `breaks/0`, `continue/0`, `open/0`, `remove_breaks/0`, `remove_breaks/1`, `reset_break/1`, `reset_break/3` and `whereami/1` for code debugging
+  * [IEx.Helpers] No longer emit warnings for IEx commands without parentheses
+  * [IEx.Helpers] Add `runtime_info/0` for printing runtime system information
+  * [IEx.Helpers] Add `open/1` to open the source of a given module/function in your editor
   * [IEx.Info] Implement `IEx.Info` protocol for calendar types
 
 #### Logger
@@ -196,20 +323,37 @@ This release brings further improvements to Calendar types. It adds arithmetic a
   * [mix loadpaths] Ensure `--no-deps-check` do not trigger SCM callbacks (such as `git`)
   * [mix local.hex] Add `--if-missing` flag to `local.hex` mix task
   * [mix profile.cprof] Add `Mix.Tasks.Profile.Cprof` for count-based profiling
+  * [mix new] New styling for generated applications
 
 ### 2. Bug fixes
 
 #### Elixir
 
+  * [Calendar] Ensure `Calendar.ISO` raises a readable error when reaching up the year 10000 restriction
+  * [Calendar] Return `{:error, :invalid_time}` for wrong precision instead of crashing when parsing ISO dates
+  * [Enumerable] Raise `Protocol.UndefinedError` on bad functions in Enumerable implementation
+  * [File] Ensure recursive file operations raise on paths with null bytes (*security issue reported by Griffin Byatt*)
   * [File] Support `:ram`/`:raw` files in `File.copy/2`
+  * [Inspect] Do not use colors when inspecting error messages
   * [Kernel] Support guards on anonymous functions of zero arity
   * [Kernel] Fix compilation of maps used as maps keys inside matches
+  * [Kernel] Ensure `do` clause in `with` is tail call optimizable
+  * [Module] `on_definition/6` callback receives body wrapped in a keyword list, such as `[do: body]`. This solves a bug where it was impossible to distinguish between a bodyless clause and a function that returns `nil`.
+  * [Path] Ensure recursive path operations raise on paths with null bytes (*security issue reported by Griffin Byatt*)
+  * [Protocol] Do not lose source compile info on protocol consolidation
   * [Record] Properly escape quoted expressions passed to `defrecord`
+  * [Regex] Fix `inspect/2` for regexes with `/` terminator in them
+  * [Registry] Ensure `Registry.match/4` works with `:_` as key
+  * [Stream] Fix stream cycle over empty enumerable
   * [String] Consider Unicode non-characters valid according to the specification in `String.valid?/1`
+  * [StringIO] Fix encoding and performance issues in `StringIO.get_until`
+  * [System] Raise on paths with null bytes in `System.cmd/2` and in `System.find_executable/1` (*security issue reported by Griffin Byatt*)
+  * [System] Raise on ill-formed environment variables (*security issue reported by Griffin Byatt*)
 
 #### ExUnit
 
   * [ExUnit] Properly account failed tests when `setup_all` fails
+  * [ExUnit] Having two or more `describe` blocks with the same name will now raise an error.
 
 #### IEx
 
@@ -221,7 +365,9 @@ This release brings further improvements to Calendar types. It adds arithmetic a
 
   * [mix compile.elixir] Store multiple sources in case of module conflicts. This solves an issue where `_build` would get corrupted when compiling Elixir projects with module conflicts
   * [mix compile.erlang] Do not silently discard Erlang compile errors
+  * [mix compile.erlang] Properly track `-compile` module attribute when specified as a list
   * [mix compile.protocols] Ensure protocol implementations do not "disappear" when switching between applications in umbrella projects by having separate consolidation paths per project
+  * [mix compile.protocols] Do not raise when consolidating a protocol that was converted into a module
 
 ### 3. Soft deprecations (no warnings emitted)
 
@@ -247,6 +393,10 @@ This release brings further improvements to Calendar types. It adds arithmetic a
   * [String] `String.lstrip/2` and `String.rstrip/2` are deprecated in favor of `String.trim_leading/2` and `String.trim_trailing/2` with a binary as second argument
   * [Typespec] `char_list/0` type is deprecated in favor of `charlist/0`
 
+#### EEx
+
+  * [EEx] Deprecate `<%= ` in "middle" and "end" expressions, e.g.: `<%= else %>` and `<%= end %>`
+
 ## v1.4
 
 The CHANGELOG for v1.4 releases can be found [in the v1.4 branch](https://github.com/elixir-lang/elixir/blob/v1.4/CHANGELOG.md).

Makefile

@@ -1,7 +1,7 @@
 REBAR ?= "$(CURDIR)/rebar"
 PREFIX ?= /usr/local
 SHARE_PREFIX ?= $(PREFIX)/share
-CANONICAL := master/
+CANONICAL := v1.5/
 ELIXIRC := bin/elixirc --verbose --ignore-module-conflict
 ERLC := erlc -I lib/elixir/include
 ERL := erl -I lib/elixir/include -noshell -pa lib/elixir/ebin

RELEASE.md

@@ -10,18 +10,17 @@ This document simply outlines the release process:
 
 3. Ensure CHANGELOG is updated and add current date
 
-4. If a new `vMAJOR.MINOR`, update in [Deprecations](lib/elixir/pages/Deprecations.md) page the link
-   to [vVERSION]'s changelog (located at the end of the file) by replacing "master" with "vVERSION"
+4. If a new `vMAJOR.MINOR`, replace "master" with "vVERSION" in the "Deprecations" page and commit
 
-5. Commit changes above with title "Release vVERSION" and generate new tag
+5. If a new `vMAJOR.MINOR`, create a new branch "vMAJOR.MINOR" and set `CANONICAL=` in Makefile
 
-6. Run `make clean test` to ensure all tests pass from scratch and the CI is green
+6. Commit changes above with title "Release vVERSION" and generate new tag
 
-7. Recompile an existing project (for example, Ecto) to ensure manifests can be upgraded
+7. Run `make clean test` to ensure all tests pass from scratch and the CI is green
 
-8. Push branch and the new tag
+8. Recompile an existing project (for example, Ecto) to ensure manifests can be upgraded
 
-9. If a new `vMAJOR.MINOR`, create a new branch "vMAJOR.MINOR" and set `CANONICAL=` in Makefile before building docs
+9. Push branch and the new tag
 
 10. Publish new zips with `make zips`, upload `Precompiled.zip` and `Docs.zip` to GitHub Releases
 

VERSION

@@ -1 +1 @@
-1.5.0-dev
+1.5.3

lib/eex/lib/eex/compiler.ex

@@ -16,10 +16,10 @@ defmodule EEx.Compiler do
     trim = opts[:trim] || false
     case EEx.Tokenizer.tokenize(source, line, trim: trim) do
       {:ok, tokens} ->
-        state = %{engine: opts[:engine] || @default_engine, init: nil,
+        state = %{engine: opts[:engine] || @default_engine,
                   file: file, line: line, quoted: [], start_line: nil}
         init = state.engine.init(opts)
-        generate_buffer(tokens, init, [], %{state | init: init})
+        generate_buffer(tokens, init, [], state)
       {:error, line, message} ->
         raise EEx.SyntaxError, line: line, file: file, message: message
     end
@@ -42,7 +42,7 @@ defmodule EEx.Compiler do
   defp generate_buffer([{:start_expr, start_line, mark, chars} | rest], buffer, scope, state) do
     {contents, line, rest} = look_ahead_text(rest, start_line, chars)
     {contents, rest} =
-      generate_buffer(rest, state.init, [contents | scope],
+      generate_buffer(rest, state.engine.handle_begin(buffer), [contents | scope],
                       %{state | quoted: [], line: line, start_line: start_line})
     buffer = state.engine.handle_expr(buffer, IO.chardata_to_string(mark), contents)
     generate_buffer(rest, buffer, scope, state)
@@ -50,7 +50,7 @@ defmodule EEx.Compiler do
 
   defp generate_buffer([{:middle_expr, line, '', chars} | rest], buffer, [current | scope], state) do
     {wrapped, state} = wrap_expr(current, line, buffer, chars, state)
-    generate_buffer(rest, state.init, [wrapped | scope], %{state | line: line})
+    generate_buffer(rest, state.engine.handle_begin(buffer), [wrapped | scope], %{state | line: line})
   end
 
   defp generate_buffer([{:middle_expr, line, modifier, chars} | t], buffer, scope, state) do
@@ -99,7 +99,7 @@ defmodule EEx.Compiler do
     key = length(state.quoted)
     placeholder = '__EEX__(' ++ Integer.to_charlist(key) ++ ');'
     {current ++ placeholder ++ new_lines ++ chars,
-     %{state | quoted: [{key, buffer} | state.quoted]}}
+     %{state | quoted: [{key, state.engine.handle_end(buffer)} | state.quoted]}}
   end
 
   # Look text ahead on expressions

lib/eex/lib/eex/engine.ex

@@ -2,19 +2,26 @@ defmodule EEx.Engine do
   @moduledoc ~S"""
   Basic EEx engine that ships with Elixir.
 
-  An engine needs to implement four functions:
+  An engine needs to implement six functions:
 
-    * `init(opts)` - returns the initial buffer
+    * `init(opts)` - called at the beginning of every text
+      and it must return the initial state.
 
-    * `handle_body(quoted)` - receives the final built quoted
-      expression, should do final post-processing and return a
-      quoted expression.
+    * `handle_body(state)` - receives the state of the document
+      and it must return a quoted expression.
 
-    * `handle_text(buffer, text)` - it receives the buffer,
+    * `handle_text(state, text)` - it receives the state,
       the text and must return a new quoted expression.
 
-    * `handle_expr(buffer, marker, expr)` - it receives the buffer,
-      the marker, the expr and must return a new quoted expression.
+    * `handle_expr(state, marker, expr)` - it receives the state,
+      the marker, the expr and must return a new state.
+
+    * `handle_begin(state)` - called every time there a new state
+      is needed with an empty buffer. Typically called for do/end
+      blocks, case expressions, anonymous functions, etc
+
+    * `handle_end(state)` - opposite of `handle_begin(state)` and
+      it must return quoted expression
 
       The marker is what follows exactly after `<%`. For example,
       `<% foo %>` has an empty marker, but `<%= foo %>` has `"="`
@@ -27,10 +34,14 @@ defmodule EEx.Engine do
   default implementations for the functions above.
   """
 
-  @callback init(opts :: keyword) :: Macro.t
-  @callback handle_body(quoted :: Macro.t) :: Macro.t
-  @callback handle_text(buffer :: Macro.t, text :: String.t) :: Macro.t
-  @callback handle_expr(buffer :: Macro.t, marker :: String.t, expr :: Macro.t) :: Macro.t
+  @type state :: term
+
+  @callback init(opts :: keyword) :: state
+  @callback handle_body(state) :: Macro.t
+  @callback handle_text(state, text :: String.t) :: state
+  @callback handle_expr(state, marker :: String.t, expr :: Macro.t) :: state
+  @callback handle_begin(state) :: state
+  @callback handle_end(state) :: Macro.t
 
   @doc false
   defmacro __using__(_) do
@@ -45,6 +56,14 @@ defmodule EEx.Engine do
         EEx.Engine.handle_body(quoted)
       end
 
+      def handle_begin(quoted) do
+        EEx.Engine.handle_begin(quoted)
+      end
+
+      def handle_end(quoted) do
+        EEx.Engine.handle_end(quoted)
+      end
+
       def handle_text(buffer, text) do
         EEx.Engine.handle_text(buffer, text)
       end
@@ -104,6 +123,20 @@ defmodule EEx.Engine do
     ""
   end
 
+  @doc """
+  Returns an empty string as the new buffer.
+  """
+  def handle_begin(_previous) do
+    ""
+  end
+
+  @doc """
+  End of the new buffer.
+  """
+  def handle_end(quoted) do
+    quoted
+  end
+
   @doc """
   The default implementation simply returns the given expression.
   """

lib/eex/test/eex_test.exs

@@ -436,25 +436,38 @@ defmodule EExTest do
     @behaviour EEx.Engine
 
     def init(_opts) do
-      ""
+      "INIT"
     end
 
     def handle_body(body) do
-      {:wrapped, body}
+      "BODY(#{body})"
+    end
+
+    def handle_begin(_) do
+      "BEGIN"
+    end
+
+    def handle_end(buffer) do
+      buffer <> ":END"
     end
 
     def handle_text(buffer, text) do
-      EEx.Engine.handle_text(buffer, text)
+      buffer <> ":TEXT(#{String.trim(text)})"
     end
 
-    def handle_expr(buffer, mark, expr) do
-      EEx.Engine.handle_expr(buffer, mark, expr)
+    def handle_expr(buffer, "=", expr) do
+      buffer <> ":EQUAL(#{Macro.to_string(expr)})"
     end
   end
 
   describe "custom engines" do
-    test "calls handle_body" do
-      assert {:wrapped, "foo"} = EEx.eval_string("foo", [], engine: TestEngine)
+    test "text" do
+      assert_eval "BODY(INIT:TEXT(foo))", "foo", [], engine: TestEngine
+    end
+
+    test "begin/end" do
+      assert_eval ~s[BODY(INIT:TEXT(foo):EQUAL(if() do\n  "BEGIN:TEXT(this):END"\nelse\n  "BEGIN:TEXT(that):END"\nend))],
+                  "foo <%= if do %>this<% else %>that<% end %>", [], engine: TestEngine
     end
   end
 

lib/elixir/lib/application.ex

@@ -9,17 +9,57 @@ defmodule Application do
   Applications are defined with an application file named `APP.app` where
   `APP` is the application name, usually in `underscore_case`. The application
   file must reside in the same `ebin` directory as the compiled modules of the
-  application.
+  application. In Elixir, the Mix build tool is responsible for compiling your
+  source code and generating your application `.app` file. You can learn more
+  about the generation of `.app` files by typing `mix help compile.app`.
 
-  In Elixir, Mix is responsible for compiling your source code and
-  generating your application `.app` file. Furthermore, Mix is also
-  responsible for configuring, starting and stopping your application
-  and its dependencies. For this reason, this documentation will focus
-  on the remaining aspects of your application: the application environment
-  and the application callback module.
+  Once your application is compiled, running your system is a matter of starting
+  your current application and its dependencies. Differently from other languages,
+  Elixir does not have a `main` procedure that is responsible for starting your
+  system. Instead, you start one or more applications, each with their own
+  initialization and termination logic.
 
-  You can learn more about Mix generation of `.app` files by typing
-  `mix help compile.app`.
+  Starting an application is done via the "application module callback", which
+  is a module that defines the `start/2` function. The `start/2` function should
+  then start a supervisor, which is often called as the top-level supervisor, since
+  it sits at the root of a potentially long supervision tree. When the system is
+  shutting down, all applications shut down their top-level supervisor, which
+  terminates children in the opposite order they are started.
+
+  We have mentioned the Mix build tool is responsible for compiling applications,
+  but it is also capable of running applications. For example, `mix test`
+  automatically starts your application dependencies and your application itself
+  before your test runs. `mix run --no-halt` also boots your current project and
+  can be used to start a long running system. See `mix help run`.
+
+  Developers can also use tools like [Distillery](https://github.com/bitwalker/distillery)
+  that build **releases**. Releases are able to package all of your source code
+  as well as the Erlang VM into a single directory. Releases also give you explicit
+  control over how each application is started and in which order. They also provide
+  a more streamlined mechanism for starting and stopping systems, debugging, logging,
+  as well as system monitoring.
+
+  Finally, Elixir provides tools such as escripts and archives, which are
+  different mechanisms for packaging your application. Those are typically used
+  when tools must be shared between developers and not as deployment options.
+  See `mix help archive.build` and `mix help escript.build` for more detail.
+
+  Shutting down a live system cleanly can be done by calling `System.stop/1`.
+  It will shut down all applications in the opposite order they are started.
+  Each application will then shutdown its top-level supervisor, if one is
+  available, which then shuts down its children.
+
+  From Erlang/OTP 19.1, a SIGTERM from the operating system will automatically
+  translate to `System.stop/0`. Erlang/OTP 20 gives user more explicit control
+  over OS signals via the `:os.set_signal/2` function.
+
+  Applications also provide an "application environment", which is how
+  applications are configured. The application environment can either be set
+  statically, via a configuration file, or dynamically via `put_env/3` and
+  friends.
+
+  Over the next sections, we will cover the "application environment" and
+  the "application module callback" in more detail.
 
   ## Application environment
 
@@ -40,9 +80,13 @@ defmodule Application do
       Application.get_env(:APP_NAME, :hello)
       #=> :world
 
-  It is also possible to put and delete values from the application value,
-  including new values that are not defined in the environment file (although
-  this should be avoided).
+  Applications and dependencies in Mix projects are typically configured
+  via the `config/config.exs` file. For example, someone using your
+  application can configure the `:hello` key as follows:
+
+      config :APP_NAME, hello: :brand_new_world
+
+  It is also possible to configure applications dynamically via `put_env/3`.
 
   Keep in mind that each application is responsible for its environment.
   Do not use the functions in this module for directly accessing or modifying
@@ -78,8 +122,8 @@ defmodule Application do
 
   The `type` argument passed to `start/2` is usually `:normal` unless in a
   distributed setup where application takeovers and failovers are configured.
-  This particular aspect of applications is explained in more detail in the
-  OTP documentation:
+  Distributed applications is beyond the scope of this documentation. For those
+  interested on the topic, please access the OTP documentation:
 
     * [`:application` module](http://www.erlang.org/doc/man/application.html)
     * [Applications – OTP Design Principles](http://www.erlang.org/doc/design_principles/applications.html)

lib/elixir/lib/base.ex

@@ -92,58 +92,145 @@ defmodule Base do
 
   """
 
-  b16_alphabet    = Enum.with_index '0123456789ABCDEF'
-  b64_alphabet    = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
-  b64url_alphabet = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
-  b32_alphabet    = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
-  b32hex_alphabet = Enum.with_index '0123456789ABCDEFGHIJKLMNOPQRSTUV'
+  b16_alphabet    = '0123456789ABCDEF'
+  b64_alphabet    = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
+  b64url_alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
+  b32_alphabet    = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
+  b32hex_alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUV'
 
-  Enum.each [{:enc16,    :dec16,    b16_alphabet},
-             {:enc32,    :dec32,    b32_alphabet},
-             {:enc64,    :dec64,    b64_alphabet},
-             {:enc64url, :dec64url, b64url_alphabet},
-             {:enc32hex, :dec32hex, b32hex_alphabet}], fn({enc, dec, alphabet}) ->
-    for {encoding, value} <- alphabet do
-      defp unquote(enc)(unquote(value)), do: unquote(encoding)
-      defp unquote(dec)(unquote(encoding)), do: unquote(value)
-    end
-    defp unquote(dec)(c) do
-      raise ArgumentError, "non-alphabet digit found: #{inspect <<c>>, binaries: :as_strings} (byte #{c})"
+  defmacrop encode_pair(alphabet, case, value) do
+    quote do
+      case unquote(value) do
+        unquote(encode_pair_clauses(alphabet, case))
+      end
     end
   end
 
-  @compile {:inline, from_upper: 1, from_lower: 1, from_mixed: 1,
-                     to_lower: 1, to_upper: 1, enc16: 1, dec16: 1,
-                     enc32: 1, dec32: 1, enc32hex: 1, dec32hex: 1,
-                     enc64: 1, dec64: 1, enc64url: 1, dec64url: 1}
+  defp encode_pair_clauses(alphabet, case) when case in [:sensitive, :upper] do
+    shift = shift(alphabet)
+    alphabet
+    |> Enum.with_index()
+    |> encode_clauses(shift)
+  end
 
-  defp to_lower(char) when char in ?A..?Z,
-    do: char + (?a - ?A)
-  defp to_lower(char),
-    do: char
+  defp encode_pair_clauses(alphabet, :lower) do
+    shift = shift(alphabet)
+    alphabet
+    |> Stream.map(fn c -> (if c in ?A..?Z, do: c - ?A + ?a, else: c) end)
+    |> Enum.with_index()
+    |> encode_clauses(shift)
+  end
 
-  defp to_upper(char), do: char
+  defp shift(alphabet) do
+    alphabet
+    |> length()
+    |> :math.log2()
+    |> round()
+  end
 
-  defp from_upper(char), do: char
+  defp encode_clauses(alphabet, shift) do
+    for {encoding1, value1} <- alphabet, {encoding2, value2} <- alphabet do
+      encoding = bsl(encoding1, 8) + encoding2
+      value = bsl(value1, shift) + value2
+      [clause] = quote do: (unquote(value) -> unquote(encoding))
+      clause
+    end
+  end
 
-  defp from_lower(char) when char in ?a..?z,
-    do: char - (?a - ?A)
-  defp from_lower(char) when char not in ?A..?Z,
-    do: char
-  defp from_lower(char),
-    do: raise(ArgumentError, "non-alphabet digit found: \"#{<<char>>}\" (byte #{char})")
+  defmacrop decode_char(alphabet, case, encoding) do
+    quote do
+      case unquote(encoding) do
+        unquote(decode_char_clauses(alphabet, case))
+      end
+    end
+  end
 
-  defp from_mixed(char) when char in ?a..?z,
-    do: char - (?a - ?A)
-  defp from_mixed(char),
-    do: char
+  defp decode_char_clauses(alphabet, case) when case in [:sensitive, :upper] do
+    clauses =
+      alphabet
+      |> Enum.with_index()
+      |> decode_clauses()
+    clauses ++ bad_digit_clause()
+  end
 
-  defp maybe_pad(subject, false, _, _),
-    do: subject
-  defp maybe_pad(subject, _, group_size, pad) do
-    case rem(byte_size(subject), group_size) do
-      0 -> subject
-      x -> subject <> String.duplicate(pad, group_size - x)
+  defp decode_char_clauses(alphabet, :lower) do
+    {uppers, rest} =
+      alphabet
+      |> Stream.with_index()
+      |> Enum.split_with(fn {encoding, _} -> encoding in ?A..?Z end)
+    lowers =
+      Enum.map(uppers, fn {encoding, value} -> {encoding - ?A + ?a, value} end)
+
+    if length(uppers) > length(rest) do
+      decode_mixed_clauses(lowers, rest)
+    else
+      decode_mixed_clauses(rest, lowers)
+    end
+  end
+
+  defp decode_char_clauses(alphabet, :mixed) when length(alphabet) == 16 do
+    alphabet = Enum.with_index(alphabet)
+    lowers =
+      alphabet
+      |> Stream.filter(fn {encoding, _} -> encoding in ?A..?Z end)
+      |> Enum.map(fn {encoding, value} -> {encoding - ?A + ?a, value} end)
+    decode_mixed_clauses(alphabet, lowers)
+  end
+
+  defp decode_char_clauses(alphabet, :mixed) when length(alphabet) == 32 do
+    alphabet
+    |> Stream.with_index()
+    |> Enum.flat_map(fn {encoding, value} = pair ->
+         if encoding in ?A..?Z do
+           [pair, {encoding - ?A + ?a, value}]
+         else
+           [pair]
+         end
+       end)
+    |> decode_clauses()
+  end
+
+  defp decode_mixed_clauses(first, second) do
+    first_clauses = decode_clauses(first)
+    second_clauses = decode_clauses(second) ++ bad_digit_clause()
+    join_clause =
+      quote do
+        encoding ->
+          case encoding do
+            unquote(second_clauses)
+          end
+      end
+    first_clauses ++ join_clause
+  end
+
+  defp decode_clauses(alphabet) do
+    for {encoding, value} <- alphabet do
+      [clause] = quote do: (unquote(encoding) -> unquote(value))
+      clause
+    end
+  end
+
+  defp bad_digit_clause() do
+    quote do
+      c ->
+        raise ArgumentError,
+              "non-alphabet digit found: #{inspect <<c>>, binaries: :as_strings} (byte #{c})"
+    end
+  end
+
+  defp maybe_pad(body, "", _, _),
+    do: body
+  defp maybe_pad(body, tail, false, _),
+    do: body <> tail
+  defp maybe_pad(body, tail, _, group_size) do
+    case group_size - rem(byte_size(tail), group_size) do
+      ^group_size -> body <> tail
+      6 -> body <> tail <> "======"
+      5 -> body <> tail <> "====="
+      4 -> body <> tail <> "===="
+      3 -> body <> tail <> "==="
+      2 -> body <> tail <> "=="
+      1 -> body <> tail <> "="
     end
   end
 
@@ -587,7 +674,7 @@ defmodule Base do
   def hex_encode32(data, opts \\ []) when is_binary(data) do
     case = Keyword.get(opts, :case, :upper)
     pad? = Keyword.get(opts, :padding, true)
-    do_hex_encode32(case, data, pad?)
+    do_encode32hex(case, data, pad?)
   end
 
   @doc """
@@ -678,7 +765,7 @@ defmodule Base do
   def hex_decode32!(string, opts \\ []) when is_binary(string) do
     case = Keyword.get(opts, :case, :upper)
     pad? = Keyword.get(opts, :padding, true)
-    do_hex_decode32(case, string, pad?)
+    do_decode32hex(case, string, pad?)
   end
 
   defp remove_ignored(string, nil), do: string
@@ -686,244 +773,298 @@ defmodule Base do
     for <<char::8 <- string>>, char not in '\s\t\r\n', into: <<>>, do: <<char::8>>
   end
 
-  defp do_encode16(_, <<>>), do: <<>>
-  defp do_encode16(:upper, data) do
-    for <<c::4 <- data>>, into: <<>>, do: <<enc16(c)::8>>
+  enc16 = [upper: :enc16_upper, lower: :enc16_lower]
+
+  for {case, fun} <- enc16 do
+    defp unquote(fun)(char) do
+      encode_pair(unquote(b16_alphabet), unquote(case), char)
+    end
   end
-  defp do_encode16(:lower, data) do
-    for <<c::4 <- data>>, into: <<>>, do: <<to_lower(enc16(c))::8>>
+
+  defp do_encode16(_, <<>>), do: <<>>
+
+  for {case, fun} <- enc16 do
+    defp do_encode16(unquote(case), data) do
+      split =  8 * div(byte_size(data), 8)
+      <<main::size(split)-binary, rest::binary>> = data
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16,
+            unquote(fun)(c7)::16, unquote(fun)(c8)::16>>
+        end
+      case rest do
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16,
+            unquote(fun)(c7)::16>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16>>
+        <<c1::8, c2::8, c3::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16>>
+        <<c1::8, c2::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16>>
+        <<c1::8>> ->
+          <<main::binary, unquote(fun)(c1)::16>>
+        <<>> ->
+          main
+      end
+    end
+  end
+
+  dec16 = [upper: :dec16_upper, lower: :dec16_lower, mixed: :dec16_mixed]
+
+  for {case, fun} <- dec16 do
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(b16_alphabet), unquote(case), encoding)
+    end
   end
 
   defp do_decode16(_, <<>>), do: <<>>
-  defp do_decode16(:upper, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(c1)::4, dec16(c2)::4>>
-    end
-  end
-  defp do_decode16(:lower, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(from_lower(c1))::4, dec16(from_lower(c2))::4>>
-    end
-  end
-  defp do_decode16(:mixed, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(from_mixed(c1))::4, dec16(from_mixed(c2))::4>>
+
+  for {case, fun} <- dec16 do
+    defp do_decode16(unquote(case), string) do
+      split =  8 * div(byte_size(string), 8)
+      <<main::size(split)-binary, rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4,
+            unquote(fun)(c5)::4, unquote(fun)(c6)::4,
+            unquote(fun)(c7)::4, unquote(fun)(c8)::4>>
+        end
+      case rest do
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4,
+            unquote(fun)(c5)::4, unquote(fun)(c6)::4>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4>>
+        <<c1::8, c2::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4>>
+        <<_::8>> ->
+          raise ArgumentError, "odd-length string"
+        <<>> ->
+          main
+      end
     end
   end
 
-  defp do_encode64(<<>>, _), do: <<>>
-  defp do_encode64(data, pad?) do
-    split =  3 * div(byte_size(data), 3)
-    <<main::size(split)-binary, rest::binary>> = data
-    main = for <<c::6 <- main>>, into: <<>>, do: <<enc64(c)::8>>
-    tail = case rest do
-      <<c1::6, c2::6, c3::4>> ->
-        <<enc64(c1)::8, enc64(c2)::8, enc64(bsl(c3, 2))::8>>
-      <<c1::6, c2::2>> ->
-        <<enc64(c1)::8, enc64(bsl(c2, 4))::8>>
-      <<>> ->
-        <<>>
+  for {base, alphabet} <- ["64": b64_alphabet, "64url": b64url_alphabet] do
+    pair = :"enc#{base}_pair"
+    char = :"enc#{base}_char"
+    do_encode = :"do_encode#{base}"
+
+    defp unquote(pair)(value) do
+      encode_pair(unquote(alphabet), :sensitive, value)
     end
-    main <> maybe_pad(tail, pad?, 4, "=")
-  end
 
-  defp do_decode64(<<>>, _), do: <<>>
-  defp do_decode64(string, false) do
-    maybe_pad(string, true, 4, "=") |> do_decode64(true)
-  end
-  defp do_decode64(string, _pad?) when rem(byte_size(string), 4) == 0 do
-    split = byte_size(string) - 4
-    <<main::size(split)-binary, rest::binary>> = string
-    main = for <<c::8 <- main>>, into: <<>>, do: <<dec64(c)::6>>
-    tail = case rest do
-      <<c1::8, c2::8, ?=, ?=>> ->
-        <<dec64(c1)::6, bsr(dec64(c2), 4)::2>>
-      <<c1::8, c2::8, c3::8, ?=>> ->
-        <<dec64(c1)::6, dec64(c2)::6, bsr(dec64(c3), 2)::4>>
-      <<c1::8, c2::8, c3::8, c4::8>> ->
-        <<dec64(c1)::6, dec64(c2)::6, dec64(c3)::6, dec64(c4)::6>>
-      <<>> ->
-        <<>>
+    defp unquote(char)(value) do
+      value
+      |> unquote(pair)()
+      |> band(0x00FF)
     end
-    main <> tail
-  end
-  defp do_decode64(_, _) do
-    raise ArgumentError, "incorrect padding"
-  end
 
-  defp do_encode64url(<<>>, _), do: <<>>
-  defp do_encode64url(data, pad?) do
-    split =  3 * div(byte_size(data), 3)
-    <<main::size(split)-binary, rest::binary>> = data
-    main = for <<c::6 <- main>>, into: <<>>, do: <<enc64url(c)::8>>
-    tail = case rest do
-      <<c1::6, c2::6, c3::4>> ->
-        <<enc64url(c1)::8, enc64url(c2)::8, enc64url(bsl(c3, 2))::8>>
-      <<c1::6, c2::2>> ->
-        <<enc64url(c1)::8, enc64url(bsl(c2, 4))::8>>
-      <<>> ->
-        <<>>
+    defp unquote(do_encode)(<<>>, _), do: <<>>
+    defp unquote(do_encode)(data, pad?) do
+      split =  6 * div(byte_size(data), 6)
+      <<main::size(split)-binary, rest::binary>> = data
+      main = for <<c1::12, c2::12, c3::12, c4::12 <- main>>, into: <<>> do
+        <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+          unquote(pair)(c3)::16, unquote(pair)(c4)::16>>
+      end
+      tail = case rest do
+        <<c1::12, c2::12, c3::12, c::4>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16, unquote(pair)(c3)::16,
+            unquote(char)(bsl(c, 2))::8>>
+        <<c1::12, c2::12, c3::8>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(bsl(c3, 4))::16>>
+        <<c1::12, c2::12>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16>>
+        <<c1::12, c2::4>> ->
+          <<unquote(pair)(c1)::16, unquote(char)(bsl(c2, 2))::8>>
+        <<c1::8>> ->
+          <<unquote(pair)(bsl(c1, 4))::16>>
+        <<>> ->
+          <<>>
+      end
+      maybe_pad(main, tail, pad?, 4)
     end
-    main <> maybe_pad(tail, pad?, 4, "=")
   end
 
-  defp do_decode64url(<<>>, _), do: <<>>
-  defp do_decode64url(string, false) do
-    maybe_pad(string, true, 4, "=") |> do_decode64url(true)
-  end
-  defp do_decode64url(string, _pad?) when rem(byte_size(string), 4) == 0 do
-    split = byte_size(string) - 4
-    <<main::size(split)-binary, rest::binary>> = string
-    main = for <<c::8 <- main>>, into: <<>>, do: <<dec64url(c)::6>>
-    tail = case rest do
-      <<c1::8, c2::8, ?=, ?=>> ->
-        <<dec64url(c1)::6, bsr(dec64url(c2), 4)::2>>
-      <<c1::8, c2::8, c3::8, ?=>> ->
-        <<dec64url(c1)::6, dec64url(c2)::6, bsr(dec64url(c3), 2)::4>>
-      <<c1::8, c2::8, c3::8, c4::8>> ->
-        <<dec64url(c1)::6, dec64url(c2)::6, dec64url(c3)::6, dec64url(c4)::6>>
-      <<>> ->
-        <<>>
+  for {base, alphabet} <- ["64": b64_alphabet, "64url": b64url_alphabet] do
+    fun = :"dec#{base}"
+    do_decode = :"do_decode#{base}"
+
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(alphabet), :sensitive, encoding)
+    end
+
+    defp unquote(do_decode)(<<>>, _), do: <<>>
+    defp unquote(do_decode)(string, pad?) do
+      segs = div(byte_size(string) + 7, 8) - 1
+      <<main::size(segs)-binary-unit(64), rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::6, unquote(fun)(c2)::6, unquote(fun)(c3)::6,
+            unquote(fun)(c4)::6, unquote(fun)(c5)::6, unquote(fun)(c6)::6,
+            unquote(fun)(c7)::6, unquote(fun)(c8)::6>>
+        end
+      case rest do
+        <<c1::8, c2::8, ?=, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, bsr(unquote(fun)(c2), 4)::2>>
+        <<c1::8, c2::8, c3::8, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            bsr(unquote(fun)(c3), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, ?=, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            bsr(unquote(fun)(c6), 4)::2>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, bsr(unquote(fun)(c7), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, unquote(fun)(c7)::6, unquote(fun)(c8)::6>>
+        <<c1::8, c2::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, bsr(unquote(fun)(c2), 4)::2>>
+        <<c1::8, c2::8, c3::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            bsr(unquote(fun)(c3), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            bsr(unquote(fun)(c6), 4)::2>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, bsr(unquote(fun)(c7), 2)::4>>
+        _ ->
+          raise ArgumentError, "incorrect padding"
+      end
     end
-    main <> tail
-  end
-  defp do_decode64url(_, _) do
-    raise ArgumentError, "incorrect padding"
   end
 
-  defp do_encode32(_, <<>>, _), do: <<>>
+  for {base, alphabet} <- ["32": b32_alphabet, "32hex": b32hex_alphabet],
+      case <- [:upper, :lower] do
+    pair = :"enc#{base}_#{case}_pair"
+    char = :"enc#{base}_#{case}_char"
+    do_encode = :"do_encode#{base}"
 
-  for {case, fun} <- [upper: :to_upper, lower: :to_lower] do
-    defp do_encode32(unquote(case), data, pad?) do
+    defp unquote(pair)(value) do
+      encode_pair(unquote(alphabet), unquote(case), value)
+    end
+
+    defp unquote(char)(value) do
+      value
+      |> unquote(pair)()
+      |> band(0x00FF)
+    end
+
+    defp unquote(do_encode)(_, <<>>, _), do: <<>>
+    defp unquote(do_encode)(unquote(case), data, pad?) do
       split =  5 * div(byte_size(data), 5)
       <<main::size(split)-binary, rest::binary>> = data
-      main = for <<c::5 <- main>>, into: <<>>, do: <<unquote(fun)(enc32(c))::8>>
+      main =
+        for <<c1::10, c2::10, c3::10, c4::10 <- main>>, into: <<>> do
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(c3)::16, unquote(pair)(c4)::16>>
+        end
       tail = case rest do
-        <<c1::5, c2::5, c3::5, c4::5, c5::5, c6::5, c7::2>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(c4))::8,
-            unquote(fun)(enc32(c5))::8, unquote(fun)(enc32(c6))::8,
-            unquote(fun)(enc32(bsl(c7, 3)))::8>>
-        <<c1::5, c2::5, c3::5, c4::5, c5::4>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(c4))::8,
-            unquote(fun)(enc32(bsl(c5, 1)))::8>>
-        <<c1::5, c2::5, c3::5, c4::1>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(bsl(c4, 4)))::8>>
-        <<c1::5, c2::3>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(bsl(c2, 2)))::8>>
+        <<c1::10, c2::10, c3::10, c4::2>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(c3)::16, unquote(char)(bsl(c4, 3))::8>>
+        <<c1::10, c2::10, c3::4>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(char)(bsl(c3, 1))::8>>
+        <<c1::10, c2::6>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(bsl(c2, 4))::16>>
+        <<c1::8>> ->
+          <<unquote(pair)(bsl(c1, 2))::16>>
         <<>> ->
           <<>>
       end
-      main <> maybe_pad(tail, pad?, 8, "=")
+      maybe_pad(main, tail, pad?, 8)
     end
   end
 
-  defp do_decode32(_, <<>>, _), do: <<>>
-  defp do_decode32(case, string, false),
-    do: do_decode32(case, maybe_pad(string, true, 8, "="), true)
+  for {base, alphabet} <- ["32": b32_alphabet, "32hex": b32hex_alphabet],
+      case <- [:upper, :lower, :mixed] do
+    fun = :"dec#{base}_#{case}"
+    do_decode = :"do_decode#{base}"
 
-  for {case, fun} <- [upper: :from_upper, lower: :from_lower, mixed: :from_mixed] do
-    defp do_decode32(unquote(case), string, _pad?) when rem(byte_size(string), 8) == 0 do
-      split = byte_size(string) - 8
-      <<main::size(split)-binary, rest::binary>> = string
-      main = for <<c::8 <- main>>, into: <<>>, do: <<dec32(unquote(fun)(c))::5>>
-      tail = case rest do
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(alphabet), unquote(case), encoding)
+    end
+
+    defp unquote(do_decode)(_, <<>>, _), do: <<>>
+
+    defp unquote(do_decode)(unquote(case), string, pad?) do
+      segs = div(byte_size(string) + 7, 8) - 1
+      <<main::size(segs)-binary-unit(64), rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            unquote(fun)(c7)::5, unquote(fun)(c8)::5>>
+        end
+      case rest do
         <<c1::8, c2::8, ?=, ?=, ?=, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, bsr(dec32(unquote(fun)(c2)), 2)::3>>
+          <<main::bits, unquote(fun)(c1)::5, bsr(unquote(fun)(c2), 2)::3>>
         <<c1::8, c2::8, c3::8, c4::8, ?=, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, bsr(dec32(unquote(fun)(c4)), 4)::1>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, bsr(unquote(fun)(c4), 4)::1>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            bsr(dec32(unquote(fun)(c5)), 1)::4>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            bsr(unquote(fun)(c5), 1)::4>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            dec32(unquote(fun)(c5))::5, dec32(unquote(fun)(c6))::5,
-            bsr(dec32(unquote(fun)(c7)), 3)::2>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            bsr(unquote(fun)(c7), 3)::2>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            dec32(unquote(fun)(c5))::5, dec32(unquote(fun)(c6))::5,
-            dec32(unquote(fun)(c7))::5, dec32(unquote(fun)(c8))::5>>
-        <<>> ->
-          <<>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            unquote(fun)(c7)::5, unquote(fun)(c8)::5>>
+        <<c1::8, c2::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, bsr(unquote(fun)(c2), 2)::3>>
+        <<c1::8, c2::8, c3::8, c4::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, bsr(unquote(fun)(c4), 4)::1>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            bsr(unquote(fun)(c5), 1)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            bsr(unquote(fun)(c7), 3)::2>>
+        _ ->
+          raise ArgumentError, "incorrect padding"
       end
-      main <> tail
     end
   end
-
-  defp do_decode32(_, _, _),
-    do: raise ArgumentError, "incorrect padding"
-
-  defp do_hex_encode32(_, <<>>, _), do: <<>>
-
-  for {case, fun} <- [upper: :to_upper, lower: :to_lower] do
-    defp do_hex_encode32(unquote(case), data, pad?) do
-      split =  5 * div(byte_size(data), 5)
-      <<main::size(split)-binary, rest::binary>> = data
-      main = for <<c::5 <- main>>, into: <<>>, do: <<unquote(fun)(enc32hex(c))::8>>
-      tail = case rest do
-        <<c1::5, c2::5, c3::5, c4::5, c5::5, c6::5, c7::2>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(c4))::8,
-            unquote(fun)(enc32hex(c5))::8, unquote(fun)(enc32hex(c6))::8,
-            unquote(fun)(enc32hex(bsl(c7, 3)))::8>>
-        <<c1::5, c2::5, c3::5, c4::5, c5::4>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(c4))::8,
-            unquote(fun)(enc32hex(bsl(c5, 1)))::8>>
-        <<c1::5, c2::5, c3::5, c4::1>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(bsl(c4, 4)))::8>>
-        <<c1::5, c2::3>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(bsl(c2, 2)))::8>>
-        <<>> ->
-          <<>>
-      end
-      main <> maybe_pad(tail, pad?, 8, "=")
-    end
-  end
-
-  defp do_hex_decode32(_, <<>>, _), do: <<>>
-  defp do_hex_decode32(case, string, false),
-    do: do_hex_decode32(case, maybe_pad(string, true, 8, "="), true)
-
-  for {case, fun} <- [upper: :from_upper, lower: :from_lower, mixed: :from_mixed] do
-    defp do_hex_decode32(unquote(case), string, _pad?) when rem(byte_size(string), 8) == 0 do
-      split = byte_size(string) - 8
-      <<main::size(split)-binary, rest::binary>> = string
-      main = for <<c::8 <- main>>, into: <<>>, do: <<dec32hex(unquote(fun)(c))::5>>
-      tail = case rest do
-        <<c1::8, c2::8, ?=, ?=, ?=, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, bsr(dec32hex(unquote(fun)(c2)), 2)::3>>
-        <<c1::8, c2::8, c3::8, c4::8, ?=, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, bsr(dec32hex(unquote(fun)(c4)), 4)::1>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            bsr(dec32hex(unquote(fun)(c5)), 1)::4>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            dec32hex(unquote(fun)(c5))::5, dec32hex(unquote(fun)(c6))::5,
-            bsr(dec32hex(unquote(fun)(c7)), 3)::2>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            dec32hex(unquote(fun)(c5))::5, dec32hex(unquote(fun)(c6))::5,
-            dec32hex(unquote(fun)(c7))::5, dec32hex(unquote(fun)(c8))::5>>
-        <<>> ->
-          <<>>
-      end
-      main <> tail
-    end
-  end
-
-  defp do_hex_decode32(_, _, _),
-    do: raise ArgumentError, "incorrect padding"
 end

lib/elixir/lib/calendar.ex

@@ -35,17 +35,16 @@ defmodule Calendar do
   @typedoc """
   The internal date format that is used when converting between calendars.
 
-  This is the amount of days including the fractional part that has passed of the last day,
-  since midnight 1 January AD 1 of the Proleptic Gregorian Calendar
-  (0000-01-01+00:00T00:00.00000 in ISO 8601 notation).
+  This is the amount of days including the fractional part that has passed of
+  the last day since 0000-01-01+00:00T00:00.00000 in ISO 8601 notation (also
+  known as midnight 1 January BC 1 of the Proleptic Gregorian Calendar).
 
   The `parts_per_day` represent how many subparts the current day is subdivided in
   (for different calendars, picking a different `parts_per_day` might make sense).
-  The `parts_in_day` represents how many of these `parts_per_day` have passed in the last day.
-
-  Thus, a Rata Die like `{1234, {1, 2}}` should be read as `1234½`.
+  The `parts_in_day` represents how many of these `parts_per_day` have passed in the
+  last day.
   """
-  @type rata_die :: {days :: integer, day_fraction}
+  @type iso_days :: {days :: integer, day_fraction}
 
   @typedoc """
   Microseconds with stored precision.
@@ -127,14 +126,14 @@ defmodule Calendar do
   @callback time_to_string(hour, minute, second, microsecond) :: String.t
 
   @doc """
-  Converts the given datetime (with time zone) into the `t:rata_die` format.
+  Converts the given datetime (with time zone) into the `t:iso_days` format.
   """
-  @callback naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) :: rata_die
+  @callback naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) :: iso_days
 
   @doc """
-  Converts `t:rata_die` to the Calendar's datetime format.
+  Converts `t:iso_days` to the Calendar's datetime format.
   """
-  @callback naive_datetime_from_rata_die(rata_die) :: {year, month, day, hour, minute, second, microsecond}
+  @callback naive_datetime_from_iso_days(iso_days) :: {year, month, day, hour, minute, second, microsecond}
 
   @doc """
   Converts the given time to the `t:day_fraction` format.

lib/elixir/lib/calendar/date.ex

@@ -67,8 +67,11 @@ defmodule Date do
 
   ## Examples
 
-      iex> Date.range(~D[2000-01-01], ~D[2001-01-01])
-      #DateRange<~D[2000-01-01], ~D[2001-01-01]>
+      iex> Date.range(~D[1999-01-01], ~D[2000-01-01])
+      #DateRange<~D[1999-01-01], ~D[2000-01-01]>
+
+      iex> Date.range(~N[2000-01-01 09:00:00], ~D[1999-01-01])
+      #DateRange<~N[2000-01-01 09:00:00], ~D[1999-01-01]>
 
   A range of dates implements the `Enumerable` protocol, which means
   functions in the `Enum` module can be used to work with
@@ -79,24 +82,25 @@ defmodule Date do
       366
       iex> Enum.member?(range, ~D[2001-02-01])
       true
-      iex> Enum.reduce(range, 0, fn(_date, acc) -> acc - 1 end)
+      iex> Enum.reduce(range, 0, fn _date, acc -> acc - 1 end)
       -366
   """
 
-  @spec range(Date.t, Date.t) :: Date.Range.t
+  @spec range(Calendar.date, Calendar.date) :: Date.Range.t
   def range(%{calendar: calendar} = first, %{calendar: calendar} = last) do
-    {first_days, _} = to_rata_die(first)
-    {last_days, _} = to_rata_die(last)
+    {first_days, _} = to_iso_days(first)
+    {last_days, _} = to_iso_days(last)
 
     %Date.Range{
       first: first,
       last: last,
-      first_rata_die: first_days,
-      last_rata_die: last_days,
+      first_in_iso_days: first_days,
+      last_in_iso_days: last_days,
     }
   end
 
-  def range(%Date{}, %Date{}) do
+  def range(%{calendar: _, year: _, month: _, day: _},
+            %{calendar: _, year: _, month: _, day: _}) do
     raise ArgumentError, "both dates must have matching calendars"
   end
 
@@ -125,7 +129,7 @@ defmodule Date do
   end
 
   @doc """
-  Returns true if the year in `date` is a leap year.
+  Returns true if the year in the given `date` is a leap year.
 
   ## Examples
 
@@ -149,7 +153,7 @@ defmodule Date do
   end
 
   @doc """
-  Returns the number of days in the given date month.
+  Returns the number of days in the given `date` month.
 
   ## Examples
 
@@ -217,15 +221,9 @@ defmodule Date do
   end
 
   @doc """
-  Parses the extended "Date and time of day" format described by
+  Parses the extended "Dates" format described by
   [ISO 8601:2004](https://en.wikipedia.org/wiki/ISO_8601).
 
-  Timezone offset may be included in the string but they will be
-  simply discarded as such information is not included in naive date
-  times.
-
-  Time representations with reduced accuracy are not supported.
-
   ## Examples
 
       iex> Date.from_iso8601("2015-01-23")
@@ -257,7 +255,7 @@ defmodule Date do
   end
 
   @doc """
-  Parses the extended "Date and time of day" format described by
+  Parses the extended "Dates" format described by
   [ISO 8601:2004](https://en.wikipedia.org/wiki/ISO_8601).
 
   Raises if the format is invalid.
@@ -309,7 +307,7 @@ defmodule Date do
   end
 
   @doc """
-  Converts a `Date` struct to an Erlang date tuple.
+  Converts the given `date` to an Erlang date tuple.
 
   Only supports converting dates which are in the ISO calendar,
   or other calendars in which the days also start at midnight.
@@ -375,7 +373,7 @@ defmodule Date do
   end
 
   @doc """
-  Compares two `Date` structs.
+  Compares two date structs.
 
   Returns `:gt` if first date is later than the second
   and `:lt` for vice versa. If the two dates are equal
@@ -408,7 +406,7 @@ defmodule Date do
   end
   def compare(date1, date2) do
     if Calendar.compatible_calendars?(date1.calendar, date2.calendar) do
-      case {to_rata_die(date1), to_rata_die(date2)} do
+      case {to_iso_days(date1), to_iso_days(date2)} do
         {first, second} when first > second -> :gt
         {first, second} when first < second -> :lt
         _ -> :eq
@@ -424,21 +422,33 @@ defmodule Date do
   end
 
   @doc """
-  Converts a date from one calendar to another.
+  Converts the given `date` from it's calendar to the given `calendar`.
 
   Returns `{:ok, date}` if the calendars are compatible,
   or `{:error, :incompatible_calendars}` if they are not.
 
   See also `Calendar.compatible_calendars?/2`.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Date.convert(~D[2000-01-01], Calendar.Holocene)
+      {:ok, %Date{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1}}
+
   """
-  @spec convert(Calendar.date(), Calendar.calendar) :: {:ok, Date.t} | {:error, :incompatible_calendars}
-  def convert(%Date{calendar: calendar} = date, calendar), do: {:ok, date}
+  @spec convert(Calendar.date, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day}, calendar) do
+    {:ok, %Date{calendar: calendar, year: year, month: month, day: day}}
+  end
   def convert(%{calendar: calendar} = date, target_calendar) do
     if Calendar.compatible_calendars?(calendar, target_calendar) do
       result_date =
         date
-        |> to_rata_die()
-        |> from_rata_die(target_calendar)
+        |> to_iso_days()
+        |> from_iso_days(target_calendar)
       {:ok, result_date}
     else
       {:error, :incompatible_calendars}
@@ -448,8 +458,18 @@ defmodule Date do
   @doc """
   Similar to `Date.convert/2`, but raises an `ArgumentError`
   if the conversion between the two calendars is not possible.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Date.convert!(~D[2000-01-01], Calendar.Holocene)
+      %Date{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1}
+
   """
-  @spec convert!(Date.t, Calendar.calendar) :: Date.t
+  @spec convert!(Calendar.date, Calendar.calendar) :: t
   def convert!(date, calendar) do
     case convert(date, calendar) do
       {:ok, value} ->
@@ -460,7 +480,7 @@ defmodule Date do
   end
 
   @doc """
-  Adds the number of days to the given date.
+  Adds the number of days to the given `date`.
 
   The days are counted as gregorian days. The date is returned in the same
   calendar as it was given in.
@@ -472,11 +492,14 @@ defmodule Date do
       iex> Date.add(~D[2000-01-01], 2)
       ~D[2000-01-03]
 
+      iex> Date.add(~N[2000-01-01 09:00:00], 2)
+      ~D[2000-01-03]
+
   """
-  @spec add(Date.t, integer()) :: Date.t
-  def add(%Date{calendar: calendar} = date, days) do
-    {rata_days, fraction} = to_rata_die(date)
-    from_rata_die({rata_days + days, fraction}, calendar)
+  @spec add(Calendar.date, integer()) :: t
+  def add(%{calendar: calendar} = date, days) do
+    {iso_days_days, fraction} = to_iso_days(date)
+    from_iso_days({iso_days_days + days, fraction}, calendar)
   end
 
   @doc """
@@ -493,42 +516,45 @@ defmodule Date do
       iex> Date.diff(~D[2000-01-01], ~D[2000-01-03])
       -2
 
+      iex> Date.diff(~D[2000-01-01], ~N[2000-01-03 09:00:00])
+      -2
+
   """
-  @spec diff(Date.t, Date.t) :: integer
-  def diff(%Date{calendar: Calendar.ISO, year: year1, month: month1, day: day1},
-           %Date{calendar: Calendar.ISO, year: year2, month: month2, day: day2}) do
-    Calendar.ISO.date_to_rata_die_days(year1, month1, day1) -
-      Calendar.ISO.date_to_rata_die_days(year2, month2, day2)
+  @spec diff(Calendar.date, Calendar.date) :: integer
+  def diff(%{calendar: Calendar.ISO, year: year1, month: month1, day: day1},
+           %{calendar: Calendar.ISO, year: year2, month: month2, day: day2}) do
+    Calendar.ISO.date_to_iso_days_days(year1, month1, day1) -
+      Calendar.ISO.date_to_iso_days_days(year2, month2, day2)
   end
 
-  def diff(%Date{} = date1, %Date{} = date2) do
-    if Calendar.compatible_calendars?(date1.calendar, date2.calendar) do
-      {days1, _} = to_rata_die(date1)
-      {days2, _} = to_rata_die(date2)
+  def diff(%{calendar: calendar1} = date1, %{calendar: calendar2} = date2) do
+    if Calendar.compatible_calendars?(calendar1, calendar2) do
+      {days1, _} = to_iso_days(date1)
+      {days2, _} = to_iso_days(date2)
       days1 - days2
     else
       raise ArgumentError, "cannot calculate the difference between #{inspect date1} and #{inspect date2} because their calendars are not compatible and thus the result would be ambiguous"
     end
   end
 
-  defp to_rata_die(%{calendar: Calendar.ISO, year: year, month: month, day: day}) do
-    {Calendar.ISO.date_to_rata_die_days(year, month, day), {0, 86400000000}}
+  defp to_iso_days(%{calendar: Calendar.ISO, year: year, month: month, day: day}) do
+    {Calendar.ISO.date_to_iso_days_days(year, month, day), {0, 86400000000}}
   end
-  defp to_rata_die(%{calendar: calendar, year: year, month: month, day: day}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, 0, 0, 0, {0, 0})
+  defp to_iso_days(%{calendar: calendar, year: year, month: month, day: day}) do
+    calendar.naive_datetime_to_iso_days(year, month, day, 0, 0, 0, {0, 0})
   end
 
-  defp from_rata_die({days, _}, Calendar.ISO) do
-    {year, month, day} = Calendar.ISO.date_from_rata_die_days(days)
+  defp from_iso_days({days, _}, Calendar.ISO) do
+    {year, month, day} = Calendar.ISO.date_from_iso_days_days(days)
     %Date{year: year, month: month, day: day, calendar: Calendar.ISO}
   end
-  defp from_rata_die(rata_die, target_calendar) do
-    {year, month, day, _, _, _, _} = target_calendar.naive_datetime_from_rata_die(rata_die)
+  defp from_iso_days(iso_days, target_calendar) do
+    {year, month, day, _, _, _, _} = target_calendar.naive_datetime_from_iso_days(iso_days)
     %Date{year: year, month: month, day: day, calendar: target_calendar}
   end
 
   @doc """
-  Calculates the day of the week of a given `Date` struct.
+  Calculates the day of the week of a given `date`.
 
   Returns the day of the week as an integer. For the ISO 8601
   calendar (the default), it is an integer from 1 to 7, where

lib/elixir/lib/calendar/date_range.ex

@@ -13,22 +13,21 @@ defmodule Date.Range do
   """
 
   @type t :: %__MODULE__{first: Date.t, last: Date.t,
-                         first_rata_die: rata_die_days, last_rata_die: rata_die_days}
+                         first_in_iso_days: Calendar.days,
+                         last_in_iso_days: Calendar.days}
 
-  @opaque rata_die_days :: Calendar.days
-
-  defstruct [:first, :last, :first_rata_die, :last_rata_die]
+  defstruct [:first, :last, :first_in_iso_days, :last_in_iso_days]
 
   defimpl Enumerable do
     def member?(%{first: %{calendar: calendar, year: first_year, month: first_month, day: first_day},
                   last: %{calendar: calendar, year: last_year, month: last_month, day: last_day},
-                  first_rata_die: first_rata_die, last_rata_die: last_rata_die},
+                  first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days},
                 %Date{calendar: calendar, year: year, month: month, day: day}) do
       first = {first_year, first_month, first_day}
       last = {last_year, last_month, last_day}
       date = {year, month, day}
 
-      if first_rata_die <= last_rata_die do
+      if first_in_iso_days <= last_in_iso_days do
         {:ok, date >= first and date <= last}
       else
         {:ok, date >= last and date <= first}
@@ -39,13 +38,13 @@ defmodule Date.Range do
       {:ok, false}
     end
 
-    def count(%Date.Range{first_rata_die: first_rata_die, last_rata_die: last_rata_die}) do
-      {:ok, abs(first_rata_die - last_rata_die) + 1}
+    def count(%Date.Range{first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days}) do
+      {:ok, abs(first_in_iso_days - last_in_iso_days) + 1}
     end
 
-    def reduce(%Date.Range{first_rata_die: first_rata_die, last_rata_die: last_rata_die,
+    def reduce(%Date.Range{first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days,
                            first: %{calendar: calendar}}, acc, fun) do
-      reduce(first_rata_die, last_rata_die, acc, fun, calendar, first_rata_die <= last_rata_die)
+      reduce(first_in_iso_days, last_in_iso_days, acc, fun, calendar, first_in_iso_days <= last_in_iso_days)
     end
 
     defp reduce(_x, _y, {:halt, acc}, _fun, _calendar, _up?) do
@@ -57,24 +56,24 @@ defmodule Date.Range do
     end
 
     defp reduce(x, y, {:cont, acc}, fun, calendar, up? = true) when x <= y do
-      reduce(x + 1, y, fun.(date_from_rata_days(x, calendar), acc), fun, calendar, up?)
+      reduce(x + 1, y, fun.(date_from_iso_days_days(x, calendar), acc), fun, calendar, up?)
     end
 
     defp reduce(x, y, {:cont, acc}, fun, calendar, up? = false) when x >= y do
-      reduce(x - 1, y, fun.(date_from_rata_days(x, calendar), acc), fun, calendar, up?)
+      reduce(x - 1, y, fun.(date_from_iso_days_days(x, calendar), acc), fun, calendar, up?)
     end
 
     defp reduce(_, _, {:cont, acc}, _fun, _calendar, _up) do
       {:done, acc}
     end
 
-    defp date_from_rata_days(days, Calendar.ISO) do
-      {year, month, day} = Calendar.ISO.date_from_rata_die_days(days)
+    defp date_from_iso_days_days(days, Calendar.ISO) do
+      {year, month, day} = Calendar.ISO.date_from_iso_days_days(days)
       %Date{year: year, month: month, day: day, calendar: Calendar.ISO}
     end
 
-    defp date_from_rata_days(days, calendar) do
-      {year, month, day, _, _, _, _} = calendar.naive_datetime_from_rata_die({days, {0, 86400000000}})
+    defp date_from_iso_days_days(days, calendar) do
+      {year, month, day, _, _, _, _} = calendar.naive_datetime_from_iso_days({days, {0, 86400000000}})
       %Date{year: year, month: month, day: day, calendar: calendar}
     end
   end

lib/elixir/lib/calendar/datetime.ex

@@ -12,6 +12,11 @@ defmodule DateTime do
   are structural and based on the DateTime struct fields. For proper
   comparison between datetimes, use the `compare/2` function.
 
+  The functions on this module work with the `DateTime` struct as well
+  as any struct that contains the same fields as the `DateTime` struct.
+  Such functions expect `t:Calendar.datetime/0` in their typespecs
+  (instead of `t:t/0`).
+
   Developers should avoid creating the DateTime struct directly
   and instead rely on the functions provided by this module as
   well as the ones in 3rd party calendar libraries.
@@ -40,7 +45,7 @@ defmodule DateTime do
                          time_zone: Calendar.time_zone, zone_abbr: Calendar.zone_abbr,
                          utc_offset: Calendar.utc_offset, std_offset: Calendar.std_offset}
 
-  @unix_days :calendar.date_to_gregorian_days({1970, 1, 1}) - 365
+  @unix_days :calendar.date_to_gregorian_days({1970, 1, 1})
 
   @doc """
   Returns the current datetime in UTC.
@@ -52,13 +57,13 @@ defmodule DateTime do
       "Etc/UTC"
 
   """
-  @spec utc_now(Calendar.calendar) :: DateTime.t
+  @spec utc_now(Calendar.calendar) :: t
   def utc_now(calendar \\ Calendar.ISO) do
     System.os_time |> from_unix!(:native, calendar)
   end
 
   @doc """
-  Converts the given Unix time to DateTime.
+  Converts the given Unix time to `DateTime`.
 
   The integer can be given in different unit
   according to `System.convert_time_unit/3` and it will
@@ -69,27 +74,24 @@ defmodule DateTime do
 
   ## Examples
 
-      iex> DateTime.from_unix(1464096368)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {0, 0}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2016, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(1464096368)
+      iex> datetime
+      #DateTime<2016-05-24 13:26:08Z>
 
-      iex> DateTime.from_unix(1432560368868569, :microsecond)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 25, hour: 13, microsecond: {868569, 6}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2015, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(1432560368868569, :microsecond)
+      iex> datetime
+      #DateTime<2015-05-25 13:26:08.868569Z>
 
   The unit can also be an integer as in `t:System.time_unit/0`:
 
-      iex> DateTime.from_unix(143256036886856, 1024)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 17, hour: 7, microsecond: {320312, 3},
-        minute: 5, month: 3, second: 22, std_offset: 0, time_zone: "Etc/UTC",
-        utc_offset: 0, year: 6403, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(143256036886856, 1024)
+      iex> datetime
+      #DateTime<6403-03-17 07:05:22.320Z>
 
   Negative Unix times are supported, up to -62167219200 seconds,
   which is equivalent to "0000-01-01T00:00:00Z" or 0 Gregorian seconds.
   """
-  @spec from_unix(integer, :native | System.time_unit, Calendar.calendar) :: {:ok, DateTime.t} | {:error, atom}
+  @spec from_unix(integer, :native | System.time_unit, Calendar.calendar) :: {:ok, t} | {:error, atom}
   def from_unix(integer, unit \\ :second, calendar \\ Calendar.ISO) when is_integer(integer) do
     case Calendar.ISO.from_unix(integer, unit) do
       {:ok, {year, month, day}, {hour, minute, second}, microsecond} ->
@@ -103,7 +105,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given Unix time to DateTime.
+  Converts the given Unix time to `DateTime`.
 
   The integer can be given in different unit
   according to `System.convert_time_unit/3` and it will
@@ -116,22 +118,16 @@ defmodule DateTime do
 
       # An easy way to get the Unix epoch is passing 0 to this function
       iex> DateTime.from_unix!(0)
-      %DateTime{calendar: Calendar.ISO, day: 1, hour: 0, microsecond: {0, 0},
-                minute: 0, month: 1, second: 0, std_offset: 0, time_zone: "Etc/UTC",
-                utc_offset: 0, year: 1970, zone_abbr: "UTC"}
+      #DateTime<1970-01-01 00:00:00Z>
 
       iex> DateTime.from_unix!(1464096368)
-      %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {0, 0}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2016, zone_abbr: "UTC"}
+      #DateTime<2016-05-24 13:26:08Z>
 
       iex> DateTime.from_unix!(1432560368868569, :microsecond)
-      %DateTime{calendar: Calendar.ISO, day: 25, hour: 13, microsecond: {868569, 6}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2015, zone_abbr: "UTC"}
+      #DateTime<2015-05-25 13:26:08.868569Z>
 
   """
-  @spec from_unix!(integer, :native | System.time_unit, Calendar.calendar) :: DateTime.t
+  @spec from_unix!(integer, :native | System.time_unit, Calendar.calendar) :: t
   def from_unix!(integer, unit \\ :second, calendar \\ Calendar.ISO) when is_atom(unit) do
     case from_unix(integer, unit, calendar) do
       {:ok, datetime} ->
@@ -142,19 +138,19 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given NaiveDateTime to DateTime.
+  Converts the given `NaiveDateTime` to `DateTime`.
 
   It expects a time zone to put the NaiveDateTime in.
   Currently it only supports "Etc/UTC" as time zone.
 
   ## Examples
 
-      iex> DateTime.from_naive(~N[2016-05-24 13:26:08.003], "Etc/UTC")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {3000, 3}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2016, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_naive(~N[2016-05-24 13:26:08.003], "Etc/UTC")
+      iex> datetime
+      #DateTime<2016-05-24 13:26:08.003Z>
+
   """
-  @spec from_naive(NaiveDateTime.t, Calendar.time_zone) :: {:ok, DateTime.t}
+  @spec from_naive(NaiveDateTime.t, Calendar.time_zone) :: {:ok, t}
   def from_naive(naive_datetime, time_zone)
 
   def from_naive(%NaiveDateTime{calendar: calendar,
@@ -166,7 +162,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given NaiveDateTime to DateTime.
+  Converts the given `NaiveDateTime` to `DateTime`.
 
   It expects a time zone to put the NaiveDateTime in.
   Currently it only supports "Etc/UTC" as time zone.
@@ -174,12 +170,10 @@ defmodule DateTime do
   ## Examples
 
       iex> DateTime.from_naive!(~N[2016-05-24 13:26:08.003], "Etc/UTC")
-      %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {3000, 3}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2016, zone_abbr: "UTC"}
+      #DateTime<2016-05-24 13:26:08.003Z>
 
   """
-  @spec from_naive!(non_neg_integer, :native | System.time_unit) :: DateTime.t
+  @spec from_naive!(NaiveDateTime.t, Calendar.time_zone) :: t
   def from_naive!(naive_datetime, time_zone) do
     case from_naive(naive_datetime, time_zone) do
       {:ok, datetime} ->
@@ -190,9 +184,9 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given DateTime to Unix time.
+  Converts the given `datetime` to Unix time.
 
-  The DateTime is expected to be using the ISO calendar
+  The `datetime` is expected to be using the ISO calendar
   with a year greater than or equal to 0.
 
   It will return the integer with the given unit,
@@ -216,18 +210,18 @@ defmodule DateTime do
       -17412508655
 
   """
-  @spec to_unix(DateTime.t, System.time_unit) :: non_neg_integer
+  @spec to_unix(Calendar.datetime, System.time_unit) :: integer
   def to_unix(datetime, unit \\ :second)
 
-  def to_unix(%DateTime{utc_offset: utc_offset, std_offset: std_offset} = datetime, unit) do
-    {days, fraction} = to_rata_die(datetime)
-    unix_units = Calendar.ISO.rata_die_to_unit({days - @unix_days, fraction}, unit)
+  def to_unix(%{utc_offset: utc_offset, std_offset: std_offset} = datetime, unit) do
+    {days, fraction} = to_iso_days(datetime)
+    unix_units = Calendar.ISO.iso_days_to_unit({days - @unix_days, fraction}, unit)
     offset_units = System.convert_time_unit(utc_offset + std_offset, :second, unit)
     unix_units - offset_units
   end
 
   @doc """
-  Converts a `DateTime` into a `NaiveDateTime`.
+  Converts the given `datetime` into a `NaiveDateTime`.
 
   Because `NaiveDateTime` does not hold time zone information,
   any time zone related data will be lost during the conversion.
@@ -241,7 +235,8 @@ defmodule DateTime do
       ~N[2000-02-29 23:00:07.0]
 
   """
-  def to_naive(%DateTime{year: year, month: month, day: day, calendar: calendar,
+  @spec to_naive(t) :: NaiveDateTime.t
+  def to_naive(%DateTime{calendar: calendar, year: year, month: month, day: day,
                          hour: hour, minute: minute, second: second, microsecond: microsecond}) do
     %NaiveDateTime{year: year, month: month, day: day, calendar: calendar,
                    hour: hour, minute: minute, second: second, microsecond: microsecond}
@@ -262,6 +257,7 @@ defmodule DateTime do
       ~D[2000-02-29]
 
   """
+  @spec to_date(t) :: Date.t
   def to_date(%DateTime{year: year, month: month, day: day, calendar: calendar}) do
     %Date{year: year, month: month, day: day, calendar: calendar}
   end
@@ -281,6 +277,7 @@ defmodule DateTime do
       ~T[23:00:07.0]
 
   """
+  @spec to_time(t) :: Time.t
   def to_time(%DateTime{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}) do
     %Time{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}
   end
@@ -329,25 +326,25 @@ defmodule DateTime do
   @spec to_iso8601(Calendar.datetime, :extended | :basic ) :: String.t
   def to_iso8601(datetime, format \\ :extended)
 
+  def to_iso8601(_, format) when format not in [:extended, :basic] do
+    raise ArgumentError, "DateTime.to_iso8601/2 expects format to be :extended or :basic, got: #{inspect format}"
+  end
+
   def to_iso8601(%{calendar: Calendar.ISO, year: year, month: month, day: day,
                   hour: hour, minute: minute, second: second, microsecond: microsecond,
-                  time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, format) when format in [:extended, :basic] do
+                  time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, format) do
     Calendar.ISO.datetime_to_iso8601(year, month, day, hour, minute, second, microsecond,
                                      time_zone, zone_abbr, utc_offset, std_offset, format)
   end
 
   def to_iso8601(%{calendar: _, year: _, month: _, day: _,
                    hour: _, minute: _, second: _, microsecond: _,
-                   time_zone: _, zone_abbr: _, utc_offset: _, std_offset: _} = datetime, format) when format in [:extended, :basic] do
+                   time_zone: _, zone_abbr: _, utc_offset: _, std_offset: _} = datetime, format) do
     datetime
     |> convert!(Calendar.ISO)
     |> to_iso8601(format)
   end
 
-  def to_iso8601(_, format) do
-    raise ArgumentError, "DateTime.to_iso8601/2 expects format to be :extended or :basic, got: #{inspect format}"
-  end
-
   @doc """
   Parses the extended "Date and time of day" format described by
   [ISO 8601:2004](https://en.wikipedia.org/wiki/ISO_8601).
@@ -367,15 +364,17 @@ defmodule DateTime do
 
   ## Examples
 
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07Z")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 23, microsecond: {0, 0}, minute: 50, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 0}
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07.123+02:30")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 21, microsecond: {123000, 3}, minute: 20, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 9000}
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07,123+02:30")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 21, microsecond: {123000, 3}, minute: 20, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 9000}
+      iex> {:ok, datetime, 0} = DateTime.from_iso8601("2015-01-23T23:50:07Z")
+      iex> datetime
+      #DateTime<2015-01-23 23:50:07Z>
+
+      iex> {:ok, datetime, 9000} = DateTime.from_iso8601("2015-01-23T23:50:07.123+02:30")
+      iex> datetime
+      #DateTime<2015-01-23 21:20:07.123Z>
+
+      iex> {:ok, datetime, 9000} = DateTime.from_iso8601("2015-01-23T23:50:07,123+02:30")
+      iex> datetime
+      #DateTime<2015-01-23 21:20:07.123Z>
 
       iex> DateTime.from_iso8601("2015-01-23P23:50:07")
       {:error, :invalid_format}
@@ -411,11 +410,12 @@ defmodule DateTime do
          {:ok, offset} <- parse_offset(rest) do
       %{year: year, month: month, day: day} = date
       %{hour: hour, minute: minute, second: second, microsecond: microsecond} = time
+      {_, precision} = microsecond
 
       datetime =
-        Calendar.ISO.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+        Calendar.ISO.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
         |> apply_tz_offset(offset)
-        |> from_rata_die("Etc/UTC", "UTC", 0, 0, calendar)
+        |> from_iso_days("Etc/UTC", "UTC", 0, 0, calendar, precision)
 
       {:ok, %{datetime | microsecond: microsecond}, offset}
     else
@@ -437,7 +437,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given datetime to a string according to its calendar.
+  Converts the given `datetime` to a string according to its calendar.
 
   ### Examples
 
@@ -479,8 +479,21 @@ defmodule DateTime do
     end
   end
 
+  defimpl Inspect do
+    def inspect(%{calendar: Calendar.ISO, year: year, month: month, day: day,
+                  hour: hour, minute: minute, second: second, microsecond: microsecond,
+                  time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, _) do
+      "#DateTime<" <> Calendar.ISO.datetime_to_string(year, month, day, hour, minute, second, microsecond,
+                                                      time_zone, zone_abbr, utc_offset, std_offset) <> ">"
+    end
+
+    def inspect(datetime, opts) do
+      Inspect.Any.inspect(datetime, opts)
+    end
+  end
+
   @doc """
-  Compares two `DateTime` structs.
+  Compares two datetime structs.
 
   Returns `:gt` if first datetime is later than the second
   and `:lt` for vice versa. If the two datetimes are equal
@@ -501,24 +514,24 @@ defmodule DateTime do
       :gt
 
   """
-  @spec compare(DateTime.t, DateTime.t) :: :lt | :eq | :gt
+  @spec compare(Calendar.datetime, Calendar.datetime) :: :lt | :eq | :gt
   def compare(%DateTime{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
               %DateTime{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2) do
     {days1, {parts1, ppd1}} =
       datetime1
-      |> to_rata_die()
+      |> to_iso_days()
       |> apply_tz_offset(utc_offset1 + std_offset1)
 
     {days2, {parts2, ppd2}} =
       datetime2
-      |> to_rata_die()
+      |> to_iso_days()
       |> apply_tz_offset(utc_offset2 + std_offset2)
 
     # Ensure fraction tuples have same denominator.
-    rata_die1 = {days1, parts1 * ppd2}
-    rata_die2 = {days2, parts2 * ppd1}
+    iso_days1 = {days1, parts1 * ppd2}
+    iso_days2 = {days2, parts2 * ppd1}
 
-    case {rata_die1, rata_die2}  do
+    case {iso_days1, iso_days2}  do
       {first, second} when first > second -> :gt
       {first, second} when first < second -> :lt
       _ -> :eq
@@ -547,90 +560,111 @@ defmodule DateTime do
       -18000
 
   """
-  @spec diff(DateTime.t, DateTime.t) :: integer()
-  def diff(%DateTime{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
-           %DateTime{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2, unit \\ :second) do
+  @spec diff(Calendar.datetime, Calendar.datetime) :: integer()
+  def diff(%{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
+           %{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2, unit \\ :second) do
     naive_diff =
-      (datetime1 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)) -
-      (datetime2 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit))
+      (datetime1 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)) -
+      (datetime2 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit))
     offset_diff =
       (utc_offset2 + std_offset2) - (utc_offset1 + std_offset1)
     naive_diff + System.convert_time_unit(offset_diff, :second, unit)
   end
 
   @doc """
-  Converts a DateTime from one calendar to another.
+  Converts a given `datetime` from one calendar to another.
 
-  If this conversion fails for some reason, an `{:error, reason}` tuple is returned.
+  If it is not possible to convert unambiguously between the calendars
+  (see `Calendar.compatible_calendars?/2`), an `{:error, :incompatible_calendars}` tuple
+  is returned.
 
   ## Examples
 
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
       iex> dt1 = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
       ...>                 hour: 23, minute: 0, second: 7, microsecond: {0, 0},
       ...>                 utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
-      iex> DateTime.convert(dt1, Calendar.ISO)
-      {:ok, %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
-                      hour: 23, minute: 0, second: 7, microsecond: {0, 0},
-                      utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}}
+      iex> DateTime.convert(dt1, Calendar.Holocene)
+      {:ok, %DateTime{calendar: Calendar.Holocene, day: 29, hour: 23,
+                      microsecond: {0, 0}, minute: 0, month: 2, second: 7, std_offset: 0,
+                      time_zone: "America/Manaus", utc_offset: -14400, year: 12000,
+                      zone_abbr: "AMT"}}
 
   """
-  @spec convert(DateTime.t, Calendar.calendar) :: {:ok, DateTime.t} | {:error, atom}
-  def convert(%DateTime{calendar: calendar} = datetime, calendar) do
-    {:ok, datetime}
+  @spec convert(Calendar.datetime, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day,
+                hour: hour, minute: minute, second: second, microsecond: microsecond,
+                time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, calendar) do
+    {:ok, %DateTime{calendar: calendar, year: year, month: month, day: day,
+                    hour: hour, minute: minute, second: second, microsecond: microsecond,
+                    time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}}
   end
 
-  def convert(%DateTime{} = datetime, calendar) do
-    result_datetime =
-      datetime
-      |> to_rata_die
-      |> from_rata_die(datetime, calendar)
-    {:ok, result_datetime}
+  def convert(%{calendar: dt_calendar, microsecond: {_, precision}} = datetime, calendar) do
+    if Calendar.compatible_calendars?(dt_calendar, calendar) do
+      result_datetime =
+        datetime
+        |> to_iso_days
+        |> from_iso_days(datetime, calendar, precision)
+      {:ok, result_datetime}
+    else
+      {:error, :incompatible_calendars}
+    end
   end
 
   @doc """
-  Converts a `DateTime` struct from one calendar to another.
+  Converts a given `datetime` from one calendar to another.
 
-  If this conversion fails for some reason, an `ArgumentError` is raised.
+  If it is not possible to convert unambiguously between the calendars
+  (see `Calendar.compatible_calendars?/2`), an ArgumentError is raised.
 
   ## Examples
 
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
       iex> dt1 = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
       ...>                 hour: 23, minute: 0, second: 7, microsecond: {0, 0},
       ...>                 utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
-      iex> DateTime.convert!(dt1, Calendar.ISO)
-      %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
-                hour: 23, minute: 0, second: 7, microsecond: {0, 0},
-                utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
+      iex> DateTime.convert!(dt1, Calendar.Holocene)
+      %DateTime{calendar: Calendar.Holocene, day: 29, hour: 23,
+                microsecond: {0, 0}, minute: 0, month: 2, second: 7, std_offset: 0,
+                time_zone: "America/Manaus", utc_offset: -14400, year: 12000,
+                zone_abbr: "AMT"}
 
   """
-  @spec convert!(DateTime.t, Calendar.calendar) :: DateTime.t
+  @spec convert!(Calendar.datetime, Calendar.calendar) :: t | no_return
   def convert!(datetime, calendar) do
     case convert(datetime, calendar) do
       {:ok, value} ->
         value
-      {:error, reason} ->
-        raise ArgumentError, "cannot convert #{inspect datetime} to target calendar #{inspect calendar}, reason: #{inspect reason}"
+      {:error, :incompatible_calendars} ->
+        raise ArgumentError, "cannot convert #{inspect datetime} to target calendar #{inspect calendar}, reason: #{inspect datetime.calendar} and #{inspect calendar} have different day rollover moments, making this conversion ambiguous"
     end
   end
 
-  defp to_rata_die(%DateTime{calendar: calendar,year: year, month: month, day: day,
-                             hour: hour, minute: minute, second: second, microsecond: microsecond}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+  defp to_iso_days(%{calendar: calendar,year: year, month: month, day: day,
+                     hour: hour, minute: minute, second: second, microsecond: microsecond}) do
+    calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
   end
 
-  defp from_rata_die(rata_die, datetime, calendar) do
+  defp from_iso_days(iso_days, datetime, calendar, precision) do
     %{time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset} = datetime
-    from_rata_die(rata_die, time_zone, zone_abbr, utc_offset, std_offset, calendar)
+    from_iso_days(iso_days, time_zone, zone_abbr, utc_offset, std_offset, calendar, precision)
   end
 
-  defp from_rata_die(rata_die, time_zone, zone_abbr, utc_offset, std_offset, calendar) do
-    {year, month, day, hour, minute, second, microsecond} = calendar.naive_datetime_from_rata_die(rata_die)
-    %DateTime{year: year, month: month, day: day,
-              hour: hour, minute: minute, second: second, microsecond: microsecond,
+  defp from_iso_days(iso_days, time_zone, zone_abbr, utc_offset, std_offset, calendar, precision) do
+    {year, month, day, hour, minute, second, {microsecond, _}} = calendar.naive_datetime_from_iso_days(iso_days)
+    %DateTime{calendar: calendar, year: year, month: month, day: day,
+              hour: hour, minute: minute, second: second, microsecond: {microsecond, precision},
               time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}
   end
 
-  defp apply_tz_offset(rata_die, offset) do
-    Calendar.ISO.add_day_fraction_to_rata_die(rata_die, -offset, 86400)
+  defp apply_tz_offset(iso_days, offset) do
+    Calendar.ISO.add_day_fraction_to_iso_days(iso_days, -offset, 86400)
   end
 end

lib/elixir/lib/calendar/iso.ex

@@ -17,7 +17,7 @@ defmodule Calendar.ISO do
 
   @unix_epoch 62167219200
   @unix_start 1_000_000 * -@unix_epoch
-  @unix_end 1_000_000 * (315569519999 - @unix_epoch)
+  @unix_end 315569519999999999 - @unix_epoch * 1_000_000
   @unix_range_microseconds @unix_start..@unix_end
 
   @type year :: 0..9999
@@ -30,46 +30,44 @@ defmodule Calendar.ISO do
   @microseconds_per_second 1_000_000
 
   @doc """
-  Returns the normalized Rata Die representation of the specified date.
+  Returns the `t:Calendar.iso_days` format of the specified date.
 
   ## Examples
 
-      iex> Calendar.ISO.naive_datetime_to_rata_die(1, 1, 1, 0, 0, 0, {0, 6})
-      {1, {0, 86400000000}}
-      iex> Calendar.ISO.naive_datetime_to_rata_die(2000, 1, 1, 12, 0, 0, {0, 6})
-      {730120, {43200000000, 86400000000}}
-      iex> Calendar.ISO.naive_datetime_to_rata_die(2000, 1, 1, 13, 0, 0, {0, 6})
-      {730120, {46800000000, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(0, 1, 1, 0, 0, 0, {0, 6})
+      {0, {0, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(2000, 1, 1, 12, 0, 0, {0, 6})
+      {730485, {43200000000, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(2000, 1, 1, 13, 0, 0, {0, 6})
+      {730485, {46800000000, 86400000000}}
 
   """
-  @spec naive_datetime_to_rata_die(Calendar.year, Calendar.month, Calendar.day,
+  @spec naive_datetime_to_iso_days(Calendar.year, Calendar.month, Calendar.day,
                                    Calendar.hour, Calendar.minute, Calendar.second,
-                                   Calendar.microsecond) :: Calendar.rata_die
-  def naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) do
-    {date_to_rata_die_days(year, month, day),
+                                   Calendar.microsecond) :: Calendar.iso_days
+  def naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) do
+    {date_to_iso_days_days(year, month, day),
      time_to_day_fraction(hour, minute, second, microsecond)}
   end
 
   @doc """
-  Converts a Rata Die to the datetime format specified by this calendar.
+  Converts the `t:Calendar.iso_days` format to the datetime format specified by this calendar.
 
   ## Examples
 
-      iex> Calendar.ISO.naive_datetime_from_rata_die({1, {0, 86400}})
-      {1, 1, 1, 0, 0, 0, {0, 6}}
-
-      iex> Calendar.ISO.naive_datetime_from_rata_die({730120, {0, 86400}})
+      iex> Calendar.ISO.naive_datetime_from_iso_days({0, {0, 86400}})
+      {0, 1, 1, 0, 0, 0, {0, 6}}
+      iex> Calendar.ISO.naive_datetime_from_iso_days({730485, {0, 86400}})
       {2000, 1, 1, 0, 0, 0, {0, 6}}
-
-      iex> Calendar.ISO.naive_datetime_from_rata_die({730120, {43200, 86400}})
+      iex> Calendar.ISO.naive_datetime_from_iso_days({730485, {43200, 86400}})
       {2000, 1, 1, 12, 0, 0, {0, 6}}
 
   """
-  @spec naive_datetime_from_rata_die(Calendar.rata_die) ::
+  @spec naive_datetime_from_iso_days(Calendar.iso_days) ::
         {Calendar.year, Calendar.month, Calendar.day,
          Calendar.hour, Calendar.minute, Calendar.second, Calendar.microsecond}
-  def naive_datetime_from_rata_die({days, day_fraction}) do
-    {year, month, day} = date_from_rata_die_days(days)
+  def naive_datetime_from_iso_days({days, day_fraction}) do
+    {year, month, day} = date_from_iso_days_days(days)
     {hour, minute, second, microsecond} = time_from_day_fraction(day_fraction)
     {year, month, day, hour, minute, second, microsecond}
   end
@@ -116,23 +114,22 @@ defmodule Calendar.ISO do
     {hours, minutes, seconds, {microseconds, 6}}
   end
 
-  # Converts a year, month, day in only a count of days since the Rata Die epoch.
+  # Converts year, month, day to count of days since 0000-01-01.
   @doc false
-  def date_to_rata_die_days(0, 1, 1) do
-    -365
+  def date_to_iso_days_days(0, 1, 1) do
+    0
   end
-  def date_to_rata_die_days(1970, 1, 1) do
-    719163
+  def date_to_iso_days_days(1970, 1, 1) do
+    719528
   end
-  def date_to_rata_die_days(year, month, day) do
-    # Rata Die starts at year 1, rather than at year 0.
-    :calendar.date_to_gregorian_days(year, month, day) - 365
+  def date_to_iso_days_days(year, month, day) when year <= 9999 do
+    :calendar.date_to_gregorian_days(year, month, day)
   end
 
-  # Calculates {year, month, day} from the count of days since the Rata Die epoch.
+  # Converts count of days since 0000-01-01 to {year, month, day} tuple.
   @doc false
-  def date_from_rata_die_days(days) do
-    :calendar.gregorian_days_to_date(days + 365)
+  def date_from_iso_days_days(days) when days <= 3652424 do
+    :calendar.gregorian_days_to_date(days)
   end
 
   defp div_mod(int1, int2) do
@@ -273,8 +270,9 @@ defmodule Calendar.ISO do
     year <= 9999 and :calendar.valid_date(year, month, day)
   end
 
-  def valid_time?(hour, minute, second, {microsecond, _}) do
-    hour in 0..23 and minute in 0..59 and second in 0..60 and microsecond in 0..999_999
+  def valid_time?(hour, minute, second, {microsecond, precision}) do
+    hour in 0..23 and minute in 0..59 and second in 0..60 and
+      microsecond in 0..999_999 and precision in 0..6
   end
 
   def day_rollover_relative_to_midnight_utc() do
@@ -316,10 +314,11 @@ defmodule Calendar.ISO do
   def from_unix(integer, unit) when is_integer(integer) do
     total = System.convert_time_unit(integer, unit, :microsecond)
     if total in @unix_range_microseconds do
-      microsecond = rem(total, 1_000_000)
+      microseconds = Integer.mod(total, @microseconds_per_second)
+      seconds = @unix_epoch + Integer.floor_div(total, @microseconds_per_second)
       precision = precision_for_unit(unit)
-      {date, time} = :calendar.gregorian_seconds_to_datetime(@unix_epoch + div(total, 1_000_000))
-      {:ok, date, time, {microsecond, precision}}
+      {date, time} = :calendar.gregorian_seconds_to_datetime(seconds)
+      {:ok, date, time, {microseconds, precision}}
     else
       {:error, :invalid_unix_time}
     end
@@ -418,26 +417,26 @@ defmodule Calendar.ISO do
   end
 
   @doc false
-  def rata_die_to_unit({days, {parts, ppd}}, unit) do
+  def iso_days_to_unit({days, {parts, ppd}}, unit) do
     day_microseconds = days * @seconds_per_day * @microseconds_per_second
     microseconds = div(parts * @seconds_per_day * @microseconds_per_second, ppd)
     System.convert_time_unit(day_microseconds + microseconds, :microsecond, unit)
   end
 
   @doc false
-  def add_day_fraction_to_rata_die({days, {parts, ppd}}, add, ppd) do
-    normalize_rata_die(days, parts + add, ppd)
+  def add_day_fraction_to_iso_days({days, {parts, ppd}}, add, ppd) do
+    normalize_iso_days(days, parts + add, ppd)
   end
-  def add_day_fraction_to_rata_die({days, {parts, ppd}}, add, add_ppd) do
+  def add_day_fraction_to_iso_days({days, {parts, ppd}}, add, add_ppd) do
     parts = parts * add_ppd
     add = add * ppd
     gcd = Integer.gcd(ppd, add_ppd)
     result_parts = div(parts + add, gcd)
     result_ppd = div(ppd * add_ppd, gcd)
-    normalize_rata_die(days, result_parts, result_ppd)
+    normalize_iso_days(days, result_parts, result_ppd)
   end
 
-  defp normalize_rata_die(days, parts, ppd) do
+  defp normalize_iso_days(days, parts, ppd) do
     days_offset = div(parts, ppd)
     parts = rem(parts, ppd)
     if parts < 0 do

lib/elixir/lib/calendar/naive_datetime.ex

@@ -27,6 +27,11 @@ defmodule NaiveDateTime do
   `NaiveDateTime` is not validated against a time zone, such errors
   would go unnoticed.
 
+  The functions on this module work with the `NaiveDateTime` struct as well
+  as any struct that contains the same fields as the `NaiveDateTime` struct,
+  such as `DateTime`. Such functions expect
+  `t:Calendar.naive_datetime/0` in their typespecs (instead of `t:t/0`).
+
   Developers should avoid creating the NaiveDateTime structs directly
   and instead rely on the functions provided by this module as well
   as the ones in 3rd party calendar libraries.
@@ -149,7 +154,7 @@ defmodule NaiveDateTime do
   def new(date, time)
 
   def new(%Date{calendar: calendar, year: year, month: month, day: day},
-          %Time{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}) do
+          %Time{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}) do
     {:ok, %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
                          hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
@@ -195,9 +200,9 @@ defmodule NaiveDateTime do
           integer, unit \\ :second) when is_integer(integer) do
     ppd = System.convert_time_unit(86400, :second, unit)
     naive_datetime
-    |> to_rata_die()
-    |> Calendar.ISO.add_day_fraction_to_rata_die(integer, ppd)
-    |> from_rata_die(calendar, precision)
+    |> to_iso_days()
+    |> Calendar.ISO.add_day_fraction_to_iso_days(integer, ppd)
+    |> from_iso_days(calendar, precision)
   end
 
   @doc """
@@ -232,8 +237,8 @@ defmodule NaiveDateTime do
       raise ArgumentError, "cannot calculate the difference between #{inspect naive_datetime1} and #{inspect naive_datetime2} because their calendars are not compatible and thus the result would be ambiguous"
     end
 
-    units1 = naive_datetime1 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)
-    units2 = naive_datetime2 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)
+    units1 = naive_datetime1 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)
+    units2 = naive_datetime2 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)
     units1 - units2
   end
 
@@ -509,6 +514,7 @@ defmodule NaiveDateTime do
       {:error, :invalid_date}
       iex> NaiveDateTime.from_erl({{2000, 13, 1},{13, 30, 15}})
       {:error, :invalid_date}
+
   """
   @spec from_erl(:calendar.datetime, Calendar.microsecond) :: {:ok, t} | {:error, atom}
   def from_erl(tuple, microsecond \\ {0, 0}, calendar \\ Calendar.ISO)
@@ -532,6 +538,7 @@ defmodule NaiveDateTime do
       ~N[2000-01-01 13:30:15.005]
       iex> NaiveDateTime.from_erl!({{2000, 13, 1}, {13, 30, 15}})
       ** (ArgumentError) cannot convert {{2000, 13, 1}, {13, 30, 15}} to naive datetime, reason: :invalid_date
+
   """
   @spec from_erl!(:calendar.datetime, Calendar.microsecond) :: t | no_return
   def from_erl!(tuple, microsecond \\ {0, 0}) do
@@ -574,7 +581,7 @@ defmodule NaiveDateTime do
   @spec compare(Calendar.naive_datetime, Calendar.naive_datetime) :: :lt | :eq | :gt
   def compare(%{calendar: calendar1} = naive_datetime1, %{calendar: calendar2} = naive_datetime2) do
     if Calendar.compatible_calendars?(calendar1, calendar2) do
-      case {to_rata_die(naive_datetime1), to_rata_die(naive_datetime2)} do
+      case {to_iso_days(naive_datetime1), to_iso_days(naive_datetime2)} do
         {first, second} when first > second -> :gt
         {first, second} when first < second -> :lt
         _ -> :eq
@@ -590,23 +597,36 @@ defmodule NaiveDateTime do
   end
 
   @doc """
-  Converts a `NaiveDateTime` struct from one calendar to another.
+  Converts the given `naive_datetime` from one calendar to another.
 
   If it is not possible to convert unambiguously between the calendars
   (see `Calendar.compatible_calendars?/2`), an `{:error, :incompatible_calendars}` tuple
   is returned.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> NaiveDateTime.convert(~N[2000-01-01 13:30:15], Calendar.Holocene)
+      {:ok, %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
+                           hour: 13, minute: 30, second: 15, microsecond: {0, 0}}}
+
   """
-  @spec convert(Calendar.naive_datetime, Calendar.calendar) :: {:ok, NaiveDateTime.t} | {:error, :incompatible_calendars}
-  def convert(%NaiveDateTime{calendar: calendar} = naive_datetime, calendar) do
-    {:ok, naive_datetime}
+  @spec convert(Calendar.naive_datetime, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day,
+                hour: hour, minute: minute, second: second, microsecond: microsecond}, calendar) do
+    {:ok, %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
+                         hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
 
   def convert(%{calendar: ndt_calendar, microsecond: {_, precision}} = naive_datetime, calendar) do
     if Calendar.compatible_calendars?(ndt_calendar, calendar) do
       result_naive_datetime =
         naive_datetime
-        |> to_rata_die
-        |> from_rata_die(calendar, precision)
+        |> to_iso_days
+        |> from_iso_days(calendar, precision)
       {:ok, result_naive_datetime}
     else
       {:error, :incompatible_calendars}
@@ -614,40 +634,49 @@ defmodule NaiveDateTime do
   end
 
   @doc """
-  Converts a NaiveDateTime from one calendar to another.
+  Converts the given `naive_datetime` from one calendar to another.
 
   If it is not possible to convert unambiguously between the calendars
   (see `Calendar.compatible_calendars?/2`), an ArgumentError is raised.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> NaiveDateTime.convert!(~N[2000-01-01 13:30:15], Calendar.Holocene)
+      %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
+                     hour: 13, minute: 30, second: 15, microsecond: {0, 0}}
+
   """
-  @spec convert!(NaiveDateTime.t, Calendar.calendar) :: NaiveDateTime.t
+  @spec convert!(Calendar.naive_datetime, Calendar.calendar) :: t
   def convert!(naive_datetime, calendar) do
     case convert(naive_datetime, calendar) do
       {:ok, value} ->
         value
       {:error, :incompatible_calendars} ->
         raise ArgumentError, "cannot convert #{inspect naive_datetime} to target calendar #{inspect calendar}, reason: #{inspect naive_datetime.calendar} and #{inspect calendar} have different day rollover moments, making this conversion ambiguous"
-      {:error, reason} ->
-        raise ArgumentError, "cannot convert #{inspect naive_datetime} to target calendar #{inspect calendar}, reason: #{inspect reason}"
     end
   end
 
   ## Helpers
 
-  defp to_rata_die(%{calendar: calendar, year: year, month: month, day: day,
+  defp to_iso_days(%{calendar: calendar, year: year, month: month, day: day,
                      hour: hour, minute: minute, second: second, microsecond: microsecond}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+    calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
   end
 
-  defp from_rata_die(rata_die, calendar, precision) do
+  defp from_iso_days(iso_days, calendar, precision) do
     {year, month, day, hour, minute, second, {microsecond, _}} =
-      calendar.naive_datetime_from_rata_die(rata_die)
-    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second,
-                   microsecond: {microsecond, precision}, calendar: calendar}
+      calendar.naive_datetime_from_iso_days(iso_days)
+    %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
+                   hour: hour, minute: minute, second: second, microsecond: {microsecond, precision}}
   end
 
   defimpl String.Chars do
     def to_string(%{calendar: calendar, year: year, month: month, day: day,
-                     hour: hour, minute: minute, second: second, microsecond: microsecond}) do
+                    hour: hour, minute: minute, second: second, microsecond: microsecond}) do
       calendar.naive_datetime_to_string(year, month, day, hour, minute, second, microsecond)
     end
   end

lib/elixir/lib/calendar/time.ex

@@ -63,7 +63,9 @@ defmodule Time do
   Expects all values to be integers. Returns `{:ok, time}` if each
   entry fits its appropriate range, returns `{:error, reason}` otherwise.
 
-  Note a time may have 60 seconds in case of leap seconds.
+  Note a time may have 60 seconds in case of leap seconds. Microseconds
+  can also be given with a precision, which must be an integer between
+  0 and 6.
 
   ## Examples
 
@@ -87,9 +89,13 @@ defmodule Time do
       iex> Time.new(23, 59, 59, 1_000_000)
       {:error, :invalid_time}
 
+      # Invalid precision
+      Time.new(23, 59, 59, {999_999, 10})
+      {:error, :invalid_time}
+
   """
   @spec new(Calendar.hour, Calendar.minute, Calendar.second, Calendar.microsecond, Calendar.calendar) ::
-        {:ok, Time.t} | {:error, atom}
+        {:ok, t} | {:error, atom}
   def new(hour, minute, second, microsecond \\ {0, 0}, calendar \\ Calendar.ISO)
 
   def new(hour, minute, second, microsecond, calendar) when is_integer(microsecond) do
@@ -108,7 +114,7 @@ defmodule Time do
   end
 
   @doc """
-  Converts the given time to a string.
+  Converts the given `time` to a string.
 
   ### Examples
 
@@ -252,7 +258,7 @@ defmodule Time do
   end
 
   @doc """
-  Converts a `Time` struct to an Erlang time tuple.
+  Converts given `time` to an Erlang time tuple.
 
   WARNING: Loss of precision may occur, as Erlang time tuples
   only contain hours/minutes/seconds.
@@ -315,7 +321,7 @@ defmodule Time do
   end
 
   @doc """
-  Compares two `Time` structs.
+  Compares two time structs.
 
   Returns `:gt` if first time is later than the second
   and `:lt` for vice versa. If the two times are equal
@@ -325,12 +331,16 @@ defmodule Time do
 
       iex> Time.compare(~T[16:04:16], ~T[16:04:28])
       :lt
+      iex> Time.compare(~T[16:04:16], ~T[16:04:16])
+      :eq
       iex> Time.compare(~T[16:04:16.01], ~T[16:04:16.001])
       :gt
 
   This function can also be used to compare across more
   complex calendar types by considering only the time fields:
 
+      iex> Time.compare(~N[1900-01-01 16:04:16], ~N[2015-01-01 16:04:16])
+      :eq
       iex> Time.compare(~N[2015-01-01 16:04:16], ~N[2015-01-01 16:04:28])
       :lt
       iex> Time.compare(~N[2015-01-01 16:04:16.01], ~N[2000-01-01 16:04:16.001])
@@ -359,14 +369,24 @@ defmodule Time do
   end
 
   @doc """
-  Converts the `Time` struct to a different calendar.
+  Converts given `time` to a different calendar.
 
   Returns `{:ok, time}` if the conversion was successful,
   or `{:error, reason}` if it was not, for some reason.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Time.convert(~T[13:30:15], Calendar.Holocene)
+      {:ok, %Time{calendar: Calendar.Holocene, hour: 13, minute: 30, second: 15, microsecond: {0, 0}}}
+
   """
-  @spec convert(Calendar.time, Calendar.calendar) :: {:ok, Time.t} | {:error, atom}
-  def convert(%Time{calendar: calendar} = time, calendar) do
-    {:ok, time}
+  @spec convert(Calendar.time, Calendar.calendar) :: {:ok, t} | {:error, atom}
+  def convert(%{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}, calendar) do
+    {:ok, %Time{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
 
   def convert(%{microsecond: {_, precision}} = time, calendar) do
@@ -381,8 +401,18 @@ defmodule Time do
   @doc """
   Similar to `Time.convert/2`, but raises an `ArgumentError`
   if the conversion between the two calendars is not possible.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Time.convert!(~T[13:30:15], Calendar.Holocene)
+      %Time{calendar: Calendar.Holocene, hour: 13, minute: 30, second: 15, microsecond: {0, 0}}
+
   """
-  @spec convert!(Calendar.time, Calendar.calendar) :: Time.t
+  @spec convert!(Calendar.time, Calendar.calendar) :: t
   def convert!(time, calendar) do
     case convert(time, calendar) do
       {:ok, value} ->
@@ -393,7 +423,14 @@ defmodule Time do
   end
 
   @doc """
-  Returns the difference between two `Time` structs.
+  Returns the difference between two times, considering only the hour, minute
+  second and microsecond.
+
+  As with the `compare/2` function both `Time` structs and other structures
+  containing time can be used. If for instance a `NaiveDateTime` or `DateTime`
+  is passed, only the hour, month, second, and microsecond is considered. Any
+  additional information about a date or time zone is ignored when calculating
+  the difference.
 
   The answer can be returned in any `unit` available from
   `t:System.time_unit/0`. If the first unit is smaller than
@@ -406,18 +443,28 @@ defmodule Time do
 
       iex> Time.diff(~T[00:29:12], ~T[00:29:10])
       2
+
+      # When passing a `NaiveDateTime` the date part is ignored.
+      iex> Time.diff(~N[2017-01-01 00:29:12], ~T[00:29:10])
+      2
+
+      # Two `NaiveDateTime` structs could have big differences in the date
+      # but only the time part is considered.
+      iex> Time.diff(~N[2017-01-01 00:29:12], (~N[1900-02-03 00:29:10]))
+      2
+
       iex> Time.diff(~T[00:29:12], ~T[00:29:10], :microsecond)
       2_000_000
       iex> Time.diff(~T[00:29:10], ~T[00:29:12], :microsecond)
       -2_000_000
 
   """
-  @spec diff(Time.t, Time.t, System.time_unit) :: integer
-  def diff(%Time{} = time1, %Time{} = time2, unit \\ :second) do
+  @spec diff(Calendar.time, Calendar.time, System.time_unit) :: integer
+  def diff(time1, time2, unit \\ :second) do
     fraction1 = to_day_fraction(time1)
     fraction2 = to_day_fraction(time2)
-    Calendar.ISO.rata_die_to_unit({0, fraction1}, unit) -
-      Calendar.ISO.rata_die_to_unit({0, fraction2}, unit)
+    Calendar.ISO.iso_days_to_unit({0, fraction1}, unit) -
+      Calendar.ISO.iso_days_to_unit({0, fraction2}, unit)
   end
 
   ## Helpers

lib/elixir/lib/enum.ex

@@ -314,12 +314,22 @@ defmodule Enum do
     end
   end
 
-  @doc """
-  Shortcut to `chunk(enumerable, count, count)`.
-  """
-  @spec chunk(t, pos_integer) :: [list]
+  # Deprecate on v1.7
+  @doc false
   def chunk(enumerable, count), do: chunk(enumerable, count, count, nil)
 
+  # Deprecate on v1.7
+  @doc false
+  def chunk(enumerable, count, step, leftover \\ nil) do
+    chunk_every(enumerable, count, step, leftover || :discard)
+  end
+
+  @doc """
+  Shortcut to `chunk_every(enumerable, count, count)`.
+  """
+  @spec chunk_every(t, pos_integer) :: [list]
+  def chunk_every(enumerable, count), do: chunk_every(enumerable, count, count, [])
+
   @doc """
   Returns list of lists containing `count` items each, where
   each new chunk starts `step` elements into the enumerable.
@@ -327,55 +337,51 @@ defmodule Enum do
   `step` is optional and, if not passed, defaults to `count`, i.e.
   chunks do not overlap.
 
-  If the final chunk does not have `count` elements to fill the chunk,
-  the final chunk is dropped unless `leftover` is given.
+  If the last chunk does not have `count` elements to fill the chunk,
+  elements are taken from `leftover` to fill in the chunk. If `leftover`
+  does not have enough elements to fill the chunk, then a partial chunk
+  is returned with less than `count` elements.
 
-  If `leftover` is given, elements are taken from `leftover` to fill in
-  the chunk. If `leftover` is passed and does not have enough elements
-  to fill the chunk, then a partial chunk is returned with less than
-  `count` elements. Therefore, an empty list can be given to `leftover`
-  when one simply desires for the last chunk to not be discarded.
+  If `:discard` is given in `leftover`, the last chunk is discarded
+  unless it has exactly `count` elements.
 
   ## Examples
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 2)
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 2)
       [[1, 2], [3, 4], [5, 6]]
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 3, 2)
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard)
       [[1, 2, 3], [3, 4, 5]]
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 3, 2, [7])
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, [7])
       [[1, 2, 3], [3, 4, 5], [5, 6, 7]]
 
-      iex> Enum.chunk([1, 2, 3, 4], 3, 3, [])
+      iex> Enum.chunk_every([1, 2, 3, 4], 3, 3, [])
       [[1, 2, 3], [4]]
 
-      iex> Enum.chunk([1, 2, 3, 4], 10)
-      []
-
-      iex> Enum.chunk([1, 2, 3, 4], 10, 10, [])
+      iex> Enum.chunk_every([1, 2, 3, 4], 10)
       [[1, 2, 3, 4]]
 
+      iex> Enum.chunk_every([1, 2, 3, 4, 5], 2, 3, [])
+      [[1, 2], [4, 5]]
+
   """
-  @spec chunk(t, pos_integer, pos_integer, t | nil) :: [list]
-  def chunk(enumerable, count, step, leftover \\ nil)
+  @spec chunk_every(t, pos_integer, pos_integer, t | :discard) :: [list]
+  def chunk_every(enumerable, count, step, leftover \\ [])
       when is_integer(count) and count > 0 and is_integer(step) and step > 0 do
-    limit = :erlang.max(count, step)
-
-    {acc, {buffer, i}} =
-      reduce(enumerable, {[], {[], 0}}, R.chunk(count, step, limit))
-
-    if is_nil(leftover) || i == 0 do
-      :lists.reverse(acc)
-    else
-      buffer = :lists.reverse(buffer, take(leftover, count - i))
-      :lists.reverse([buffer | acc])
-    end
+    R.chunk_every(&chunk_while/4, enumerable, count, step, leftover)
   end
 
   @doc """
-  Splits enumerable on every element for which `fun` returns a new
-  value.
+  Chunks the `enum` with fine grained control when every chunk is emitted.
+
+  `chunk_fun` receives the current element and the accumulator and
+  must return `{:cont, element, acc}` to emit the given chunk and
+  continue with accumulator or `{:cont, acc}` to not emit any chunk
+  and continue with the return accumulator.
+
+  `after_fun` is invoked when iteration is done and must also return
+  `{:cont, element, acc}` or `{:cont, acc}`.
 
   Returns a list of lists.
 
@@ -392,15 +398,22 @@ defmodule Enum do
       ...>   [] -> {:cont, []}
       ...>   acc -> {:cont, Enum.reverse(acc), []}
       ...> end
-      iex> Enum.chunk_by(1..10, [], chunk_fun, after_fun)
+      iex> Enum.chunk_while(1..10, [], chunk_fun, after_fun)
       [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
 
   """
-  @spec chunk_by(t, acc,
-                 (element, acc -> {:cont, chunk, acc} | {:cont, acc}),
-                 (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
-  def chunk_by(enum, acc, chunk_fun, after_fun) do
-    {res, acc} = reduce(enum, {[], acc}, R.chunk_by(chunk_fun))
+  @spec chunk_while(t, acc,
+                    (element, acc -> {:cont, chunk, acc} | {:cont, acc} | {:halt, acc}),
+                    (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
+  def chunk_while(enum, acc, chunk_fun, after_fun) do
+    {_, {res, acc}} =
+      Enumerable.reduce(enum, {:cont, {[], acc}}, fn entry, {buffer, acc} ->
+        case chunk_fun.(entry, acc) do
+          {:cont, emit, acc} -> {:cont, {[emit | buffer], acc}}
+          {:cont, acc} -> {:cont, {buffer, acc}}
+          {:halt, acc} -> {:halt, {buffer, acc}}
+        end
+      end)
 
     case after_fun.(acc) do
       {:cont, _acc} -> :lists.reverse(res)
@@ -422,18 +435,7 @@ defmodule Enum do
   """
   @spec chunk_by(t, (element -> any)) :: [list]
   def chunk_by(enumerable, fun) do
-    chunk_by(enumerable, nil, fn
-      entry, nil ->
-        {:cont, {[entry], fun.(entry)}}
-      entry, {acc, value} ->
-        case fun.(entry) do
-          ^value -> {:cont, {[entry | acc], value}}
-          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
-        end
-    end, fn
-      nil -> {:cont, :done}
-      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
-    end)
+    R.chunk_by(&chunk_while/4, enumerable, fun)
   end
 
   @doc """
@@ -833,13 +835,28 @@ defmodule Enum do
   Filters the enumerable, i.e. returns only those elements
   for which `fun` returns a truthy value.
 
-  See also `reject/2`.
+  See also `reject/2` which discards all elements where the
+  function returns true.
 
   ## Examples
 
       iex> Enum.filter([1, 2, 3], fn(x) -> rem(x, 2) == 0 end)
       [2]
 
+  Keep in mind that `filter` is not capable of filtering and
+  transforming an element at the same time. If you would like
+  to do so, consider using `flat_map/2`. For example, if you
+  want to convert all strings that represent an integer and
+  discard the invalid one in one pass:
+
+      strings = ["1234", "abc", "12ab"]
+      Enum.flat_map(strings, fn string ->
+        case Integer.parse(string) do
+          {int, _rest} -> [int] # transform to integer
+          :error -> [] # skip the value
+        end
+      end)
+
   """
   @spec filter(t, (element -> as_boolean(term))) :: list
   def filter(enumerable, fun) when is_list(enumerable) do
@@ -3192,6 +3209,13 @@ defimpl Enumerable, for: Function do
   def member?(_function, _value),
     do: {:error, __MODULE__}
 
-  def reduce(function, acc, fun),
+  def reduce(function, acc, fun) when is_function(function, 2),
     do: function.(acc, fun)
+
+  def reduce(function, _acc, _fun) do
+    raise Protocol.UndefinedError,
+      protocol: @protocol,
+      value: function,
+      description: "only anonymous functions of arity 2 are enumerable"
+  end
 end

lib/elixir/lib/exception.ex

@@ -230,7 +230,7 @@ defmodule Exception do
          {_, kind, _, clauses} <- List.keyfind(defs, {function, arity}, 0) do
       clauses =
         for {meta, ex_args, guards, _block} <- clauses do
-          scope = :elixir_erl.definition_scope(meta, kind, function, arity, "nofile")
+          scope = :elixir_erl.definition_scope(meta, "nofile")
           {erl_args, scope} =
             :elixir_erl_clauses.match(&:elixir_erl_pass.translate_args/2, ex_args, scope)
           {args, binding} =
@@ -862,8 +862,7 @@ defmodule UndefinedFunctionError do
   end
 
   defp format_fa({_dist, fun, arity}) do
-    fun = with ":" <> fun <- inspect(fun), do: fun
-    ["      * ", fun, ?/, Integer.to_string(arity), ?\n]
+    ["      * ", Inspect.Function.escape_name(fun), ?/, Integer.to_string(arity), ?\n]
   end
 
   defp exports_for(module) do

lib/elixir/lib/file.ex

@@ -670,15 +670,21 @@ defmodule File do
       File.cp_r "samples", "tmp"
 
       # Same as before, but asks the user how to proceed in case of conflicts
-      File.cp_r "samples", "tmp", fn(source, destination) ->
+      File.cp_r "samples", "tmp", fn source, destination ->
         IO.gets("Overwriting #{destination} by #{source}. Type y to confirm. ") == "y\n"
       end
 
   """
   @spec cp_r(Path.t, Path.t, (Path.t, Path.t -> boolean)) :: {:ok, [binary]} | {:error, posix, binary}
-  def cp_r(source, destination, callback \\ fn(_, _) -> true end) when is_function(callback, 2) do
-    source = IO.chardata_to_string(source)
-    destination = IO.chardata_to_string(destination)
+  def cp_r(source, destination, callback \\ fn _, _ -> true end) when is_function(callback, 2) do
+    source =
+      source
+      |> IO.chardata_to_string()
+      |> assert_no_null_byte!("File.cp_r/3")
+    destination =
+      destination
+      |> IO.chardata_to_string()
+      |> assert_no_null_byte!("File.cp_r/3")
 
     case do_cp_r(source, destination, callback, []) do
       {:error, _, _} = error -> error
@@ -947,7 +953,10 @@ defmodule File do
   """
   @spec rm_rf(Path.t) :: {:ok, [binary]} | {:error, posix, binary}
   def rm_rf(path) do
-    do_rm_rf(IO.chardata_to_string(path), {:ok, []})
+    path
+    |> IO.chardata_to_string()
+    |> assert_no_null_byte!("File.rm_rf/1")
+    |> do_rm_rf({:ok, []})
   end
 
   defp do_rm_rf(path, {:ok, _} = entry) do
@@ -1454,6 +1463,15 @@ defmodule File do
 
   @read_ahead_size 64 * 1024
 
+  defp assert_no_null_byte!(binary, operation) do
+    case :binary.match(binary, "\0") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute #{operation} for path with null byte, got: #{inspect binary}"
+      :nomatch ->
+        binary
+    end
+  end
+
   defp normalize_modes([:utf8 | rest], binary?) do
     [encoding: :utf8] ++ normalize_modes(rest, binary?)
   end

lib/elixir/lib/inspect.ex

@@ -356,20 +356,21 @@ defimpl Inspect, for: Map do
     open = color("%" <> name <> "{", :map, opts)
     sep = color(",", :map, opts)
     close = color("}", :map, opts)
-    surround_many(open, map, close, opts, traverse_fun(map), sep)
+    surround_many(open, map, close, opts, traverse_fun(map, opts), sep)
   end
 
-  defp traverse_fun(list) do
+  defp traverse_fun(list, opts) do
     if Inspect.List.keyword?(list) do
       &Inspect.List.keyword/2
     else
-      &to_map/2
+      sep = color(" => ", :map, opts)
+      &to_map(&1, &2, sep)
     end
   end
 
-  defp to_map({key, value}, opts) do
+  defp to_map({key, value}, opts, sep) do
     concat(
-      concat(to_doc(key, opts), " => "),
+      concat(to_doc(key, opts), sep),
       to_doc(value, opts)
     )
   end

lib/elixir/lib/inspect/algebra.ex

@@ -243,7 +243,7 @@ defmodule Inspect.Algebra do
             try do
               Process.put(:inspect_trap, true)
 
-              res = Inspect.Map.inspect(map, opts)
+              res = Inspect.Map.inspect(map, %{opts | syntax_colors: []})
               res = IO.iodata_to_binary(format(res, :infinity))
 
               exception = Inspect.Error.exception(

lib/elixir/lib/kernel.ex

@@ -3289,8 +3289,17 @@ defmodule Kernel do
       result
     end
 
-    {escaped, _} = :elixir_quote.escape(block, false)
-    module_vars  = module_vars(env.vars, 0)
+    escaped =
+      case env do
+        %{function: nil, lexical_tracker: pid} when is_pid(pid) ->
+          integer = Kernel.LexicalTracker.write_cache(pid, block)
+          quote do: Kernel.LexicalTracker.read_cache(unquote(pid), unquote(integer))
+        %{} ->
+          {escaped, _} = :elixir_quote.escape(block, false)
+          escaped
+      end
+
+    module_vars = module_vars(env.vars, 0)
 
     quote do
       unquote(with_alias)
@@ -3554,11 +3563,21 @@ defmodule Kernel do
   end
 
   defp define(kind, call, expr, env) do
-    assert_module_scope(env, kind, 2)
+    module = assert_module_scope(env, kind, 2)
     assert_no_function_scope(env, kind, 2)
 
-    {call, unquoted_call} = :elixir_quote.escape(call, true)
-    {expr, unquoted_expr} = :elixir_quote.escape(expr, true)
+    {escaped_call, unquoted_call} = :elixir_quote.escape(call, true)
+    {escaped_expr, unquoted_expr} = :elixir_quote.escape(expr, true)
+
+    escaped_expr =
+      case unquoted_expr do
+        true ->
+          escaped_expr
+        false ->
+          key = :erlang.unique_integer()
+          :elixir_module.write_cache(module, key, expr)
+          quote do: :elixir_module.read_cache(unquote(module), unquote(key))
+      end
 
     # Do not check clauses if any expression was unquoted
     check_clauses = not(unquoted_expr or unquoted_call)
@@ -3566,7 +3585,7 @@ defmodule Kernel do
 
     quote do
       :elixir_def.store_definition(unquote(kind), unquote(check_clauses),
-                                   unquote(call), unquote(expr), unquote(pos))
+                                   unquote(escaped_call), unquote(escaped_expr), unquote(pos))
     end
   end
 
@@ -4594,7 +4613,7 @@ defmodule Kernel do
   defp assert_module_scope(env, fun, arity) do
     case env.module do
       nil -> raise ArgumentError, "cannot invoke #{fun}/#{arity} outside module"
-      _   -> :ok
+      mod -> mod
     end
   end
 

lib/elixir/lib/kernel/lexical_tracker.ex

@@ -81,6 +81,18 @@ defmodule Kernel.LexicalTracker do
     :gen_server.cast(pid, {:alias_dispatch, module})
   end
 
+  @doc false
+  def write_cache(pid, value) do
+    key = :erlang.unique_integer()
+    :gen_server.cast(pid, {:write_cache, key, value})
+    key
+  end
+
+  @doc false
+  def read_cache(pid, key) do
+    :gen_server.call(pid, {:read_cache, key}, @timeout)
+  end
+
   @doc false
   def collect_unused_imports(pid) do
     unused(pid, :import)
@@ -99,7 +111,7 @@ defmodule Kernel.LexicalTracker do
 
   def init(dest) do
     {:ok, %{directives: %{}, references: %{}, compile: %{},
-            runtime: %{}, dest: dest}}
+            runtime: %{}, dest: dest, cache: %{}}}
   end
 
   @doc false
@@ -124,6 +136,14 @@ defmodule Kernel.LexicalTracker do
     {:reply, state.dest, state}
   end
 
+  def handle_call({:read_cache, key}, _from, %{cache: cache} = state) do
+    {:reply, Map.fetch!(cache, key), state}
+  end
+
+  def handle_cast({:write_cache, key, value}, %{cache: cache} = state) do
+    {:noreply, Map.put(state, :cache, Map.put(cache, key, value))}
+  end
+
   def handle_cast({:remote_reference, module, mode}, state) do
     {:noreply, %{state | references: add_reference(state.references, module, mode)}}
   end

lib/elixir/lib/kernel/typespec.ex

@@ -903,7 +903,7 @@ defmodule Kernel.Typespec do
   end
 
   # Handle tuples
-  defp typespec({:tuple, meta, args}, _vars, _caller) when args == [] or is_atom(args) do
+  defp typespec({:tuple, meta, []}, _vars, _caller) do
     {:type, line(meta), :tuple, :any}
   end
 

lib/elixir/lib/macro.ex

@@ -131,7 +131,7 @@ defmodule Macro do
       valid_alias?(charlist) ->
         :alias
       true ->
-        case :elixir_config.get(:identifier_tokenizer).tokenize(charlist) do
+        case :elixir_config.safe_get(:identifier_tokenizer, String.Tokenizer).tokenize(charlist) do
           {kind, _acc, [], _, _, special} ->
             if kind == :identifier and not :lists.member(?@, special) do
               :callable

lib/elixir/lib/macro/env.ex

@@ -44,6 +44,9 @@ defmodule Macro.Env do
 
     * `export_vars` - a list keeping all variables to be exported in a
       construct (may be `nil`)
+    * `match_vars` - controls how "new" variables are handled. Inside a
+      match it is a list with all variables in a match. Outside of a match
+      is either `:warn` or `:apply`
     * `prematch_vars` - a list of variables defined before a match (is
       `nil` when not inside a match)
 
@@ -64,6 +67,7 @@ defmodule Macro.Env do
   @type local :: atom | nil
 
   @opaque export_vars :: vars | nil
+  @opaque match_vars :: vars | :warn | :apply
   @opaque prematch_vars :: vars | nil
 
   @type t :: %{__struct__: __MODULE__,
@@ -80,6 +84,7 @@ defmodule Macro.Env do
                context_modules: context_modules,
                vars: vars,
                export_vars: export_vars,
+               match_vars: match_vars,
                prematch_vars: prematch_vars,
                lexical_tracker: lexical_tracker}
 
@@ -99,6 +104,7 @@ defmodule Macro.Env do
       vars: [],
       lexical_tracker: nil,
       export_vars: nil,
+      match_vars: :warn,
       prematch_vars: nil}
   end
 

lib/elixir/lib/map.ex

@@ -548,7 +548,8 @@ defmodule Map do
 
   If `key` is present in `map` with value `value`, `fun` is invoked with
   argument `value` and its result is used as the new value of `key`. If `key` is
-  not present in `map`, `initial` is inserted as the value of `key`.
+  not present in `map`, `initial` is inserted as the value of `key`. The initial
+  value will not be passed through the update function.
 
   ## Examples
 

lib/elixir/lib/map_set.ex

@@ -353,7 +353,8 @@ defmodule MapSet do
     %{map_set | map: Map.merge(map1, map2)}
   end
   def union(%MapSet{map: map1}, %MapSet{map: map2}) do
-    new_from_list(Map.keys(map1) ++ Map.keys(map2), [])
+    map = new_from_list(Map.keys(map1) ++ Map.keys(map2), [])
+    %MapSet{map: map}
   end
 
   defp order_by_size(map1, map2) when map_size(map1) > map_size(map2), do: {map2, map1}

lib/elixir/lib/module.ex

@@ -898,7 +898,7 @@ defmodule Module do
               "cannot make function #{function_name}/#{arity} overridable because it was not defined"
           clause ->
             neighbours =
-              if :elixir_compiler.get_opt(:internal) do
+              if :elixir_config.get(:bootstrap) do
                 []
               else
                 Module.LocalsTracker.yank(module, tuple)
@@ -1192,8 +1192,13 @@ defmodule Module do
     case :ets.take(table, :impl) do
       [{:impl, value, _, _}] ->
         impls = :ets.lookup_element(table, {:elixir, :impls}, 2)
-        impl = {{name, length(args)}, kind, line, file, value}
-        :ets.insert(table, {{:elixir, :impls}, [impl | impls]})
+        {total, defaults} = args_count(args, 0, 0)
+
+        impl = for arity <- total..(total - defaults), into: impls do
+          {{name, arity}, kind, line, file, value}
+        end
+
+        :ets.insert(table, {{:elixir, :impls}, impl})
       [] ->
         :ok
     end
@@ -1201,6 +1206,14 @@ defmodule Module do
     :ok
   end
 
+  defp args_count([{:\\, _, _} | tail], total, defaults) do
+    args_count(tail, total + 1, defaults + 1)
+  end
+  defp args_count([_head | tail], total, defaults) do
+    args_count(tail, total + 1, defaults)
+  end
+  defp args_count([], total, defaults), do: {total, defaults}
+
   @doc false
   def check_behaviours_and_impls(env, table, all_definitions, overridable_pairs) do
     behaviours = :ets.lookup_element(table, :behaviour, 2)
@@ -1299,9 +1312,9 @@ defmodule Module do
   end
   defp known_callbacks(callbacks) do
     formatted = for {{{name, arity}, kind}, module} <- callbacks do
-      "\n  * " <> Exception.format_mfa(module, name, arity) <> "(#{kind})"
+      "\n  * " <> Exception.format_mfa(module, name, arity) <> " (#{kind})"
     end
-    ". The known callbacks are:\n#{formatted}"
+    ". The known callbacks are:\n#{formatted}\n"
   end
 
   @doc false

lib/elixir/lib/module/locals_tracker.ex

@@ -138,8 +138,8 @@ defmodule Module.LocalsTracker do
 
   # Reattach a previously yanked node
   @doc false
-  def reattach(pid, kind, tuple, neighbours) do
-    :gen_server.cast(to_pid(pid), {:reattach, kind, tuple, neighbours})
+  def reattach(pid, tuple, kind, function, neighbours) do
+    :gen_server.cast(to_pid(pid), {:reattach, tuple, kind, function, neighbours})
   end
 
   # Collecting all conflicting imports with the given functions
@@ -162,15 +162,25 @@ defmodule Module.LocalsTracker do
   def collect_unused_locals(ref, private) do
     d = :gen_server.call(to_pid(ref), :digraph, @timeout)
     reachable = reachable_from(d, :local)
-    {unreachable(reachable, private), collect_warnings(reachable, private)}
+    reattached = :digraph.out_neighbours(d, :reattach)
+    {unreachable(reachable, reattached, private), collect_warnings(reachable, private)}
   end
 
-  defp unreachable(reachable, private) do
+  defp unreachable(reachable, reattached, private) do
     for {tuple, kind, _, _} <- private,
-        kind == :defmacrop or not :sets.is_element(tuple, reachable),
+        not reachable?(tuple, kind, reachable, reattached),
         do: tuple
   end
 
+  defp reachable?(tuple, :defmacrop, reachable, reattached) do
+    # All private micros are unreachable unless they have been
+    # reattached and they are reachable.
+    :lists.member(tuple, reattached) and :sets.is_element(tuple, reachable)
+  end
+  defp reachable?(tuple, :defp, reachable, _reattached) do
+    :sets.is_element(tuple, reachable)
+  end
+
   defp collect_warnings(reachable, private) do
     :lists.foldl(&collect_warnings(&1, &2, reachable), [], private)
   end
@@ -210,16 +220,6 @@ defmodule Module.LocalsTracker do
     last
   end
 
-  @doc false
-  def cache_env(pid, env) do
-    :gen_server.call(pid, {:cache_env, env}, @timeout)
-  end
-
-  @doc false
-  def get_cached_env(pid, ref) do
-    :gen_server.call(pid, {:get_cached_env, ref}, @timeout)
-  end
-
   # Stops the gen server
   @doc false
   def stop(pid) do
@@ -231,79 +231,76 @@ defmodule Module.LocalsTracker do
   def init([]) do
     d = :digraph.new([:protected])
     :digraph.add_vertex(d, :local)
-    {:ok, {d, []}}
+    :digraph.add_vertex(d, :reattach)
+    {:ok, d}
   end
 
-  @doc false
-  def handle_call({:cache_env, env}, _from, {d, cache}) do
-    case cache do
-      [{i, ^env} | _] ->
-        {:reply, i, {d, cache}}
-      t ->
-        i = length(t)
-        {:reply, i, {d, [{i, env} | t]}}
-    end
-  end
-
-  def handle_call({:get_cached_env, ref}, _from, {_, cache} = state) do
-    {^ref, env} = :lists.keyfind(ref, 1, cache)
-    {:reply, env, state}
-  end
-
-  def handle_call({:yank, local}, _from, {d, _} = state) do
+  def handle_call({:yank, local}, _from, d) do
     out_vertices = :digraph.out_neighbours(d, local)
     :digraph.del_edges(d, :digraph.out_edges(d, local))
-    {:reply, {[], out_vertices}, state}
+    {:reply, {[], out_vertices}, d}
   end
 
-  def handle_call(:digraph, _from, {d, _} = state) do
-    {:reply, d, state}
+  def handle_call(:digraph, _from, d) do
+    {:reply, d, d}
   end
 
   @doc false
-  def handle_info(_msg, state) do
-    {:noreply, state}
+  def handle_info(_msg, d) do
+    {:noreply, d}
   end
 
-  def handle_cast({:add_local, from, to}, {d, _} = state) do
+  def handle_cast({:add_local, from, to}, d) do
     handle_add_local(d, from, to)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_import, function, module, {name, arity}}, {d, _} = state) do
+  def handle_cast({:add_import, function, module, {name, arity}}, d) do
     handle_import(d, function, module, name, arity)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_definition, kind, tuple}, {d, _} = state) do
+  def handle_cast({:add_definition, kind, tuple}, d) do
     handle_add_definition(d, kind, tuple)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_defaults, kind, {name, arity}, defaults}, {d, _} = state) do
+  def handle_cast({:add_defaults, kind, {name, arity}, defaults}, d) do
     for i <- :lists.seq(arity - defaults, arity - 1) do
       handle_add_definition(d, kind, {name, i})
       handle_add_local(d, {name, i}, {name, arity})
     end
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:reattach, _kind, tuple, {in_neigh, out_neigh}}, {d, _} = state) do
+  def handle_cast({:reattach, tuple, kind, function, {in_neigh, out_neigh}}, d) do
+    # Reattach the old function
     for from <- in_neigh do
       :digraph.add_vertex(d, from)
-      replace_edge!(d, from, tuple)
+      replace_edge!(d, from, function)
     end
 
     for to <- out_neigh do
       :digraph.add_vertex(d, to)
-      replace_edge!(d, tuple, to)
+      replace_edge!(d, function, to)
     end
 
-    {:noreply, state}
+    # Add the new definition
+    handle_add_definition(d, kind, tuple)
+
+    # Make a call from the old function to the new one
+    if function != tuple do
+      handle_add_local(d, function, tuple)
+    end
+
+    # Finally marked the new one as reattached
+    replace_edge!(d, :reattach, tuple)
+
+    {:noreply, d}
   end
 
-  def handle_cast(:stop, state) do
-    {:stop, :normal, state}
+  def handle_cast(:stop, d) do
+    {:stop, :normal, d}
   end
 
   @doc false

lib/elixir/lib/path.ex

@@ -615,17 +615,19 @@ defmodule Path do
   def wildcard(glob, opts \\ []) do
     mod = if Keyword.get(opts, :match_dot), do: :file, else: Path.Wildcard
     glob
-    |> chardata_to_list()
+    |> chardata_to_list!()
     |> :filelib.wildcard(mod)
     |> Enum.map(&IO.chardata_to_string/1)
   end
 
-  # expand_dot the given path by expanding "..", "." and "~".
-
-  defp chardata_to_list(chardata) do
+  defp chardata_to_list!(chardata) do
     case :unicode.characters_to_list(chardata) do
       result when is_list(result) ->
-        result
+        if 0 in result do
+          raise ArgumentError, "cannot execute Path.wildcard/2 for path with null byte, got: #{inspect chardata}"
+        else
+          result
+        end
 
       {:error, encoded, rest} ->
         raise UnicodeConversionError, encoded: encoded, rest: rest, kind: :invalid
@@ -654,6 +656,8 @@ defmodule Path do
     end
   end
 
+  # expand_dot the given path by expanding "..", "." and "~".
+
   defp expand_dot(<<"/", rest::binary>>),
     do: "/" <> do_expand_dot(rest)
   defp expand_dot(<<letter, ":/", rest::binary>>) when letter in ?a..?z,

lib/elixir/lib/protocol.ex

@@ -254,21 +254,22 @@ defmodule Protocol do
     {:error, :not_a_protocol} |
     {:error, :no_beam_info}
   def consolidate(protocol, types) when is_atom(protocol) do
-    with {:ok, info} <- beam_protocol(protocol),
-         {:ok, code, docs} <- change_debug_info(info, types),
-         do: compile(code, docs)
+    with {:ok, ast_info, chunks_info} <- beam_protocol(protocol),
+         {:ok, code} <- change_debug_info(ast_info, types),
+         do: compile(protocol, code, chunks_info)
   end
 
   defp beam_protocol(protocol) do
-    chunk_ids = [:abstract_code, :attributes, 'ExDc']
+    chunk_ids = [:abstract_code, :attributes, :compile_info, 'ExDc']
     opts = [:allow_missing_chunks]
     case :beam_lib.chunks(beam_file(protocol), chunk_ids, opts) do
       {:ok, {^protocol, [{:abstract_code, {_raw, abstract_code}},
                          {:attributes, attributes},
+                         {:compile_info, compile_info},
                          {'ExDc', docs}]}} ->
         case attributes[:protocol] do
           [fallback_to_any: any] ->
-            {:ok, {protocol, any, abstract_code, docs}}
+            {:ok, {protocol, any, abstract_code}, {compile_info, docs}}
           _ ->
             {:error, :not_a_protocol}
         end
@@ -286,12 +287,12 @@ defmodule Protocol do
 
   # Change the debug information to the optimized
   # impl_for/1 dispatch version.
-  defp change_debug_info({protocol, any, code, docs}, types) do
+  defp change_debug_info({protocol, any, code}, types) do
     types   = if any, do: types, else: List.delete(types, Any)
     all     = [Any] ++ for {_guard, mod} <- __builtin__(), do: mod
     structs = types -- all
     case change_impl_for(code, protocol, types, structs, false, []) do
-      {:ok, ret} -> {:ok, ret, docs}
+      {:ok, ret} -> {:ok, ret}
       other      -> other
     end
   end
@@ -358,9 +359,9 @@ defmodule Protocol do
     change_impl_for(tail, protocol, info, types, protocol?, [head | acc])
   end
 
-  defp change_impl_for([], protocol, _info, _types, protocol?, acc) do
+  defp change_impl_for([], _protocol, _info, _types, protocol?, acc) do
     if protocol? do
-      {:ok, {protocol, Enum.reverse(acc)}}
+      {:ok, Enum.reverse(acc)}
     else
       {:error, :not_a_protocol}
     end
@@ -405,8 +406,9 @@ defmodule Protocol do
   end
 
   # Finally compile the module and emit its bytecode.
-  defp compile({protocol, code}, docs) do
-    opts = if Code.compiler_options[:debug_info], do: [:debug_info], else: []
+  defp compile(protocol, code, {compile_info, docs}) do
+    opts = Keyword.take(compile_info, [:source])
+    opts = if Code.compiler_options[:debug_info], do: [:debug_info | opts], else: opts
     {:ok, ^protocol, binary, _warnings} = :compile.forms(code, [:return | opts])
     {:ok,
       case docs do

lib/elixir/lib/registry.ex

@@ -304,7 +304,7 @@ defmodule Registry do
   @doc false
   def child_spec(opts) do
     %{
-      id: Registry,
+      id: Keyword.get(opts, :name, Registry),
       start: {Registry, :start_link, [opts]},
       type: :supervisor
     }
@@ -534,7 +534,8 @@ defmodule Registry do
   """
   @spec match(registry, key, match_pattern :: atom() | tuple(), guards :: list()) :: [{pid, term}]
   def match(registry, key, pattern, guards \\ []) when is_atom(registry) and is_list(guards) do
-    spec = [{{key, {:_, pattern}}, guards, [{:element, 2, :"$_"}]}]
+    guards = [{:"=:=", {:element, 1, :"$_"}, {:const, key}} | guards]
+    spec = [{{:_, {:_, pattern}}, guards, [{:element, 2, :"$_"}]}]
 
     case key_info!(registry) do
       {:unique, partitions, key_ets} ->
@@ -587,7 +588,16 @@ defmodule Registry do
   def keys(registry, pid) when is_atom(registry) and is_pid(pid) do
     {kind, partitions, _, pid_ets, _} = info!(registry)
     {_, pid_ets} = pid_ets || pid_ets!(registry, pid, partitions)
-    keys = safe_lookup_second(pid_ets, pid)
+
+    keys = try do
+      spec = [{{pid, :'$1', :'$2'}, [], [{{:'$1', :'$2'}}]}]
+      :ets.select(pid_ets, spec)
+    catch
+      :error, :badarg -> []
+    end
+
+    # Handle the possibility of fake keys
+    keys = gather_keys(keys, [], false)
 
     cond do
       kind == :unique -> Enum.uniq(keys)
@@ -595,6 +605,19 @@ defmodule Registry do
     end
   end
 
+  defp gather_keys([{key, {_, remaining}} | rest], acc, _fake) do
+    gather_keys(rest, [key | acc], {key, remaining})
+  end
+  defp gather_keys([{key, _} | rest], acc, fake) do
+    gather_keys(rest, [key | acc], fake)
+  end
+  defp gather_keys([], acc, {key, remaining}) do
+    List.duplicate(key, remaining) ++ Enum.reject(acc, & &1 === key)
+  end
+  defp gather_keys([], acc, false) do
+    acc
+  end
+
   @doc """
   Unregisters all entries for the given `key` associated to the current
   process in `registry`.
@@ -652,6 +675,95 @@ defmodule Registry do
     :ok
   end
 
+  @doc """
+  Unregister entries for a given key matching a pattern.
+
+  ## Examples
+
+  For unique registries it can be used to conditionally unregister a key on
+  the basis of whether or not it matches a particular value.
+
+      iex> Registry.start_link(:unique, Registry.UniqueUnregisterMatchTest)
+      iex> Registry.register(Registry.UniqueUnregisterMatchTest, "hello", :world)
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      ["hello"]
+      iex> Registry.unregister_match(Registry.UniqueUnregisterMatchTest, "hello", :foo)
+      :ok
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      ["hello"]
+      iex> Registry.unregister_match(Registry.UniqueUnregisterMatchTest, "hello", :world)
+      :ok
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      []
+
+  For duplicate registries:
+
+      iex> Registry.start_link(:duplicate, Registry.DuplicateUnregisterMatchTest)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_a)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_b)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_c)
+      iex> Registry.keys(Registry.DuplicateUnregisterMatchTest, self())
+      ["hello", "hello", "hello"]
+      iex> Registry.unregister_match(Registry.DuplicateUnregisterMatchTest, "hello", :world_a)
+      :ok
+      iex> Registry.keys(Registry.DuplicateUnregisterMatchTest, self())
+      ["hello", "hello"]
+      iex> Registry.lookup(Registry.DuplicateUnregisterMatchTest, "hello")
+      [{self(), :world_b}, {self(), :world_c}]
+  """
+  def unregister_match(registry, key, pattern, guards \\ []) when is_list(guards) do
+    self = self()
+
+    {kind, partitions, key_ets, pid_ets, listeners} = info!(registry)
+    {key_partition, pid_partition} = partitions(kind, key, self, partitions)
+    key_ets = key_ets || key_ets!(registry, key_partition)
+    {pid_server, pid_ets} = pid_ets || pid_ets!(registry, pid_partition)
+
+    # Remove first from the key_ets because in case of crashes
+    # the pid_ets will still be able to clean up. The last step is
+    # to clean if we have no more entries.
+
+    # Here we want to count all entries for this pid under this key, regardless
+    # of pattern.
+    underscore_guard = {:"=:=", {:element, 1, :"$_"}, {:const, key}}
+    total_spec = [{{:_, {self, :_}}, [underscore_guard], [true]}]
+    total = :ets.select_count(key_ets, total_spec)
+
+    # We only want to delete things that match the pattern
+    delete_spec = [{{:_, {self, pattern}}, [underscore_guard | guards] ,[true]}]
+    case :ets.select_delete(key_ets, delete_spec) do
+      # We deleted everything, we can just delete the object
+      ^total ->
+        true = :ets.delete_object(pid_ets, {self, key, key_ets})
+
+        unlink_if_unregistered(pid_server, pid_ets, self)
+
+        for listener <- listeners do
+          Kernel.send(listener, {:unregister, registry, key, self})
+        end
+
+      0 ->
+        :ok
+
+      deleted ->
+        # There are still entries remaining for this pid. delete_object/2 with
+        # duplicate_bag tables will remove every entry, but we only want to
+        # remove those we have deleted. The solution is to introduce a temp_entry
+        # that indicates how many keys WILL be remaining after the delete operation.
+        remaining = total - deleted
+        temp_entry = {self, key, {key_ets, remaining}}
+        true = :ets.insert(pid_ets, temp_entry)
+        true = :ets.delete_object(pid_ets, {self, key, key_ets})
+        real_keys = List.duplicate({self, key, key_ets}, remaining)
+        true = :ets.insert(pid_ets, real_keys)
+        # We've recreated the real remaining key entries, so we can now delete
+        # our temporary entry.
+        true = :ets.delete_object(pid_ets, temp_entry)
+    end
+
+    :ok
+  end
+
   @doc """
   Registers the current process under the given `key` in `registry`.
 
@@ -976,6 +1088,12 @@ defmodule Registry.Partition do
   def handle_info({:EXIT, pid, _reason}, ets) do
     entries = :ets.take(ets, pid)
     for {_pid, key, key_ets} <- entries do
+      key_ets = case key_ets do
+        # In case the fake key ets is being used. See unregister_match/2.
+        {key_ets, _} -> key_ets
+        _ -> key_ets
+      end
+
       try do
         :ets.match_delete(key_ets, {key, {pid, :_}})
       catch

lib/elixir/lib/stream.ex

@@ -121,63 +121,57 @@ defmodule Stream do
 
   ## Transformers
 
-  @doc """
-  Shortcut to `chunk(enum, n, n)`.
-  """
-  @spec chunk(Enumerable.t, pos_integer) :: Enumerable.t
+  # Deprecate on v1.7
+  @doc false
   def chunk(enum, n), do: chunk(enum, n, n, nil)
 
+  # Deprecate on v1.7
+  @doc false
+  def chunk(enum, n, step, leftover \\ nil)
+      when is_integer(n) and n > 0 and is_integer(step) and step > 0 do
+    chunk_every(enum, n, step, leftover || :discard)
+  end
+
   @doc """
-  Streams the enumerable in chunks, containing `n` items each, where
-  each new chunk starts `step` elements into the enumerable.
+  Shortcut to `chunk_every(enum, count, count)`.
+  """
+  @spec chunk_every(Enumerable.t, pos_integer) :: Enumerable.t
+  def chunk_every(enum, count), do: chunk_every(enum, count, count, [])
+
+  @doc """
+  Streams the enumerable in chunks, containing `count` items each,
+  where each new chunk starts `step` elements into the enumerable.
 
   `step` is optional and, if not passed, defaults to `count`, i.e.
   chunks do not overlap.
 
-  If the final chunk does not have `count` elements to fill the chunk,
-  the final chunk is dropped unless `leftover` is given.
+  If the last chunk does not have `count` elements to fill the chunk,
+  elements are taken from `leftover` to fill in the chunk. If `leftover`
+  does not have enough elements to fill the chunk, then a partial chunk
+  is returned with less than `count` elements.
 
-  If `leftover` is given, elements are taken from `leftover` to fill in
-  the chunk. If `leftover` is passed and does not have enough elements
-  to fill the chunk, then a partial chunk is returned with less than
-  `count` elements. Therefore, an empty list can be given to `leftover`
-  when one simply desires for the last chunk to not be discarded.
+  If `:discard` is given in `leftover`, the last chunk is discarded
+  unless it has exactly `count` elements.
 
   ## Examples
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 2) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 2) |> Enum.to_list
       [[1, 2], [3, 4], [5, 6]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 2) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard) |> Enum.to_list
       [[1, 2, 3], [3, 4, 5]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 2, [7]) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, [7]) |> Enum.to_list
       [[1, 2, 3], [3, 4, 5], [5, 6, 7]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 3, []) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 3, []) |> Enum.to_list
       [[1, 2, 3], [4, 5, 6]]
 
   """
-  @spec chunk(Enumerable.t, pos_integer, pos_integer, Enumerable.t | nil) :: Enumerable.t
-  def chunk(enum, n, step, leftover \\ nil)
-      when is_integer(n) and n > 0 and is_integer(step) and step > 0 do
-    limit = :erlang.max(n, step)
-    if is_nil(leftover) do
-      lazy enum, {[], 0}, fn(f1) -> R.chunk(n, step, limit, f1) end
-    else
-      lazy enum, {[], 0},
-           fn(f1) -> R.chunk(n, step, limit, f1) end,
-           &do_chunk(&1, n, leftover, &2)
-     end
-  end
-
-  defp do_chunk(acc(_, {_, 0}, _) = acc, _, _, _) do
-    {:cont, acc}
-  end
-
-  defp do_chunk(acc(h, {buffer, count} = old, t), n, leftover, f1) do
-    buffer = :lists.reverse(buffer, Enum.take(leftover, n - count))
-    next_with_acc(f1, buffer, h, old, t)
+  @spec chunk_every(Enumerable.t, pos_integer, pos_integer, Enumerable.t | :discard) :: Enumerable.t
+  def chunk_every(enum, count, step, leftover \\ [])
+      when is_integer(count) and count > 0 and is_integer(step) and step > 0 do
+    R.chunk_every(&chunk_while/4, enum, count, step, leftover)
   end
 
   @doc """
@@ -194,18 +188,7 @@ defmodule Stream do
   """
   @spec chunk_by(Enumerable.t, (element -> any)) :: Enumerable.t
   def chunk_by(enum, fun) do
-    chunk_by(enum, nil, fn
-      entry, nil ->
-        {:cont, {[entry], fun.(entry)}}
-      entry, {acc, value} ->
-        case fun.(entry) do
-          ^value -> {:cont, {[entry | acc], value}}
-          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
-        end
-    end, fn
-      nil -> {:cont, :done}
-      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
-    end)
+    R.chunk_by(&chunk_while/4, enum, fun)
   end
 
   @doc """
@@ -232,21 +215,21 @@ defmodule Stream do
       ...>   [] -> {:cont, []}
       ...>   acc -> {:cont, Enum.reverse(acc), []}
       ...> end
-      iex> stream = Stream.chunk_by(1..10, [], chunk_fun, after_fun)
+      iex> stream = Stream.chunk_while(1..10, [], chunk_fun, after_fun)
       iex> Enum.to_list(stream)
       [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
 
   """
-  @spec chunk_by(Enumerable.t, acc,
-                 (element, acc -> {:cont, chunk, acc} | {:cont, acc}),
-                 (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
-  def chunk_by(enum, acc, chunk_fun, after_fun) do
+  @spec chunk_while(Enumerable.t, acc,
+                    (element, acc -> {:cont, chunk, acc} | {:cont, acc} | {:halt, acc}),
+                    (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
+  def chunk_while(enum, acc, chunk_fun, after_fun) do
     lazy enum, acc,
-         fn(f1) -> R.chunk_by(chunk_fun, f1) end,
-         &after_chunk_by(&1, &2, after_fun)
+         fn(f1) -> R.chunk_while(chunk_fun, f1) end,
+         &after_chunk_while(&1, &2, after_fun)
   end
 
-  defp after_chunk_by(acc(h, acc, t), f1, after_fun) do
+  defp after_chunk_while(acc(h, acc, t), f1, after_fun) do
     case after_fun.(acc) do
       {:cont, emit, acc} -> next_with_acc(f1, emit, h, acc, t)
       {:cont, acc} -> {:cont, acc(h, acc, t)}
@@ -1155,6 +1138,10 @@ defmodule Stream do
   @spec cycle(Enumerable.t) :: Enumerable.t
   def cycle(enumerable)
 
+  def cycle([]) do
+    raise ArgumentError, "cannot cycle over empty enumerable"
+  end
+
   def cycle(enumerable) when is_list(enumerable) do
     unfold {enumerable, enumerable}, fn
       {source, [h | t]}      -> {h, {source, t}}
@@ -1185,6 +1172,8 @@ defmodule Stream do
       {:stream_cycle, acc} ->
         {:halted, acc}
     else
+      {state, []} when state in [:done, :halted] ->
+        raise ArgumentError, "cannot cycle over empty enumerable"
       {state, acc} when state in [:done, :halted] ->
         do_cycle(cycle, cycle, {:cont, acc})
       {:suspended, acc, continuation} ->

lib/elixir/lib/stream/reducers.ex

@@ -1,36 +1,57 @@
 defmodule Stream.Reducers do
-  # Collection of reducers shared by Enum and Stream.
+  # Collection of reducers and utilities shared by Enum and Stream.
   @moduledoc false
 
-  defmacro chunk(amount, step, limit, fun \\ nil) do
-    quote do
-      fn entry, acc(head, {buffer, count}, tail) ->
-        buffer = [entry | buffer]
-        count = count + 1
+  def chunk_every(chunk_by, enumerable, count, step, leftover) do
+    limit = :erlang.max(count, step)
+    chunk_by.(enumerable, {[], 0}, fn entry, {acc_buffer, acc_count} ->
+      acc_buffer = [entry | acc_buffer]
+      acc_count = acc_count + 1
 
-        new_state =
-          if count >= unquote(limit) do
-            left = count - unquote(step)
-            {Enum.take(buffer, left), left}
-          else
-            {buffer, count}
-          end
-
-        if count == unquote(amount) do
-          next_with_acc(unquote(fun), :lists.reverse(buffer), head, new_state, tail)
+      new_state =
+        if acc_count >= limit do
+          remaining = acc_count - step
+          {Enum.take(acc_buffer, remaining), remaining}
         else
-          skip(acc(head, new_state, tail))
+          {acc_buffer, acc_count}
         end
+
+      if acc_count == count do
+        {:cont, :lists.reverse(acc_buffer), new_state}
+      else
+        {:cont, new_state}
       end
-    end
+    end, fn {acc_buffer, acc_count} ->
+      if leftover == :discard or acc_count == 0 or (step > count and acc_count >= count) do
+        {:cont, []}
+      else
+        {:cont, :lists.reverse(acc_buffer, Enum.take(leftover, count - acc_count)), []}
+      end
+    end)
   end
 
-  defmacro chunk_by(callback, fun \\ nil) do
+  def chunk_by(chunk_by, enumerable, fun) do
+    chunk_by.(enumerable, nil, fn
+      entry, nil ->
+        {:cont, {[entry], fun.(entry)}}
+      entry, {acc, value} ->
+        case fun.(entry) do
+          ^value -> {:cont, {[entry | acc], value}}
+          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
+        end
+    end, fn
+      nil -> {:cont, :done}
+      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
+    end)
+  end
+
+  defmacro chunk_while(callback, fun \\ nil) do
     quote do
       fn entry, acc(head, acc, tail) ->
         case unquote(callback).(entry, acc) do
           {:cont, emit, acc} -> next_with_acc(unquote(fun), emit, head, acc, tail)
           {:cont, acc} -> skip(acc(head, acc, tail))
+          {:halt, acc} -> {:halt, acc(head, acc, tail)}
         end
       end
     end
@@ -38,7 +59,7 @@ defmodule Stream.Reducers do
 
   defmacro dedup(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, prev, tail) = acc) ->
+      fn entry, acc(head, prev, tail) = acc ->
         value = unquote(callback).(entry)
         case prev do
           {:value, ^value} -> skip(acc)
@@ -84,7 +105,7 @@ defmodule Stream.Reducers do
 
   defmacro filter(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -96,7 +117,7 @@ defmodule Stream.Reducers do
 
   defmacro filter_map(filter, mapper, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(filter).(entry) do
           next(unquote(fun), unquote(mapper).(entry), acc)
         else
@@ -108,7 +129,7 @@ defmodule Stream.Reducers do
 
   defmacro map(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         next(unquote(fun), unquote(callback).(entry), acc)
       end
     end
@@ -127,7 +148,7 @@ defmodule Stream.Reducers do
 
   defmacro reject(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         unless unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -151,7 +172,7 @@ defmodule Stream.Reducers do
 
   defmacro scan3(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, acc, tail)) ->
+      fn entry, acc(head, acc, tail) ->
         value = unquote(callback).(entry, acc)
         next_with_acc(unquote(fun), value, head, value, tail)
       end
@@ -160,7 +181,7 @@ defmodule Stream.Reducers do
 
   defmacro take(fun \\ nil) do
     quote do
-      fn(entry, acc(head, curr, tail) = original) ->
+      fn entry, acc(head, curr, tail) = original ->
         case curr do
           0 ->
             {:halt, original}
@@ -187,7 +208,7 @@ defmodule Stream.Reducers do
 
   defmacro take_while(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -199,7 +220,7 @@ defmodule Stream.Reducers do
 
   defmacro uniq_by(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, prev, tail) = original) ->
+      fn entry, acc(head, prev, tail) = original ->
         value = unquote(callback).(entry)
         if Map.has_key?(prev, value) do
           skip(original)
@@ -212,7 +233,7 @@ defmodule Stream.Reducers do
 
   defmacro with_index(fun \\ nil) do
     quote do
-      fn(entry, acc(head, counter, tail)) ->
+      fn entry, acc(head, counter, tail) ->
         next_with_acc(unquote(fun), {entry, counter}, head, counter + 1, tail)
       end
     end

lib/elixir/lib/string_io.ex

@@ -274,57 +274,64 @@ defmodule StringIO do
 
   ## get_until
 
-  defp get_until(encoding, prompt, mod, fun, args,
-                 %{input: input, output: output, capture_prompt: capture_prompt} = s) do
-    case :unicode.characters_to_list(input, encoding) do
-      {:error, _, _} ->
-        {:error, s}
-      {:incomplete, _, _} ->
-        {:error, s}
-      chars ->
-        {result, input, count} = do_get_until(chars, encoding, mod, fun, args)
-
+  defp get_until(encoding, prompt, mod, fun, args, %{input: input} = s) do
+    case do_get_until(input, encoding, mod, fun, args) do
+      {result, input, count} ->
         input =
-          case input do
-            :eof -> ""
-            _ -> :unicode.characters_to_binary(input, encoding)
-          end
+         case input do
+           :eof -> ""
+           _ -> list_to_binary(input, encoding)
+         end
 
-        s =
-          if capture_prompt do
-            %{s | output: <<output::binary, :binary.copy(IO.chardata_to_string(prompt), count)::binary>>}
-          else
-            s
-          end
+        {get_until_result(result, encoding), state_after_read(s, input, prompt, count)}
 
-        {result, %{s | input: input}}
+      :error ->
+        {:error, s}
     end
   end
 
   defp do_get_until(chars, encoding, mod, fun, args, continuation \\ [], count \\ 0)
 
-  defp do_get_until('', encoding, mod, fun, args, continuation, count) do
+  defp do_get_until("", encoding, mod, fun, args, continuation, count) do
     case apply(mod, fun, [continuation, :eof | args]) do
       {:done, result, rest} ->
         {result, rest, count + 1}
       {:more, next_continuation} ->
-        do_get_until('', encoding, mod, fun, args, next_continuation, count + 1)
+        do_get_until("", encoding, mod, fun, args, next_continuation, count + 1)
     end
   end
 
   defp do_get_until(chars, encoding, mod, fun, args, continuation, count) do
-    {line, rest} = collect_line(chars)
+    case bytes_until_eol(chars, encoding, 0) do
+      {r, c} when r in [:split, :replace_split] ->
+        {line, rest} = :erlang.split_binary(chars, c)
 
-    case apply(mod, fun, [continuation, line | args]) do
-      {:done, result, :eof} ->
-        {result, rest, count + 1}
-      {:done, result, extra} ->
-        {result, extra ++ rest, count + 1}
-      {:more, next_continuation} ->
-        do_get_until(rest, encoding, mod, fun, args, next_continuation, count + 1)
+        case apply(mod, fun, [continuation, binary_to_list(line, encoding) | args]) do
+          {:done, result, :eof} ->
+            {result, rest, count + 1}
+          {:done, result, extra} ->
+            {result, extra ++ binary_to_list(rest, encoding), count + 1}
+          {:more, next_continuation} ->
+            do_get_until(rest, encoding, mod, fun, args, next_continuation, count + 1)
+        end
+
+      :error ->
+        :error
     end
   end
 
+  defp binary_to_list(l, _)        when is_list(l),   do: l
+  defp binary_to_list(b, :unicode) when is_binary(b), do: to_charlist(b)
+  defp binary_to_list(b, :latin1)  when is_binary(b), do: :binary.bin_to_list(b)
+  defp list_to_binary(b, _)        when is_binary(b), do: b
+  defp list_to_binary(l, :unicode) when is_list(l),   do: to_string(l)
+  defp list_to_binary(l, :latin1)  when is_list(l),   do: :binary.list_to_bin(l)
+
+  # From http://erlang.org/doc/apps/stdlib/io_protocol.html: Result can be any
+  # Erlang term, but if it is a list(), the I/O server can convert it to a binary().
+  defp get_until_result(l, encoding) when is_list(l), do: list_to_binary(l, encoding)
+  defp get_until_result(other, _), do: other
+
   ## io_requests
 
   defp io_requests([r | rs], {:ok, s}) do
@@ -337,12 +344,14 @@ defmodule StringIO do
 
   ## helpers
 
-  defp state_after_read(%{capture_prompt: false} = s, remainder, _prompt) do
+  defp state_after_read(state, remainder, prompt, count \\ 1)
+
+  defp state_after_read(%{capture_prompt: false} = s, remainder, _prompt, _count) do
     %{s | input: remainder}
   end
 
-  defp state_after_read(%{capture_prompt: true, output: output} = s, remainder, prompt) do
-    %{s | input: remainder, output: <<output::binary, IO.chardata_to_string(prompt)::binary>>}
+  defp state_after_read(%{capture_prompt: true, output: output} = s, remainder, prompt, count) do
+    %{s | input: remainder, output: <<output::binary, :binary.copy(IO.chardata_to_string(prompt), count)::binary>>}
   end
 
   defp bytes_until_eol("", _, count), do: {:split, count}
@@ -359,26 +368,6 @@ defmodule StringIO do
 
   defp bytes_until_eol(<<_::binary>>, _, _), do: :error
 
-  defp collect_line(chars) do
-    collect_line(chars, [])
-  end
-
-  defp collect_line([], stack) do
-    {:lists.reverse(stack), []}
-  end
-
-  defp collect_line([?\r, ?\n | rest], stack) do
-    {:lists.reverse([?\n | stack]), rest}
-  end
-
-  defp collect_line([?\n | rest], stack) do
-    {:lists.reverse([?\n | stack]), rest}
-  end
-
-  defp collect_line([h | t], stack) do
-    collect_line(t, [h | stack])
-  end
-
   defp io_reply(from, reply_as, reply) do
     send from, {:io_reply, reply_as, reply}
   end

lib/elixir/lib/supervisor.ex

@@ -82,7 +82,7 @@ defmodule Supervisor do
   supervisor will automatically start a new one, with the initial stack
   of `[:hello]`:
 
-      GenServer.call(MyStack, :pop)
+      GenServer.call(Stack, :pop)
       #=> :hello
 
   Supervisors support different strategies; in the example above, we
@@ -187,7 +187,7 @@ defmodule Supervisor do
   `GenServer` with a shutdown limit of 10 seconds (10_000 miliseconds),
   one might do:
 
-      use GenServer, shutdown: 10000
+      use GenServer, shutdown: 10_000
 
   Let's understand what the `:shutdown` and `:restart` options control.
 
@@ -333,14 +333,14 @@ defmodule Supervisor do
   The first argument given to `start_link/2` is a list of children which may
   be either:
 
-      * a module - such as `Stack`. In this case, it is equivalent to passing
-        `{Stack, []}` (which means `Stack.child_spec/1` is invoked with an empty
-        keywords list)
-      * a tuple with a module as first element and the start argument as second -
-        such as `{Stack, [:hello]}`. When such format is used, the supervisor
-        will retrieve the child specification from the given module.
-      * a map representing the child specification itself - such as the child
-        specification map outlined in the previous section.
+    * a module - such as `Stack`. In this case, it is equivalent to passing
+      `{Stack, []}` (which means `Stack.child_spec/1` is invoked with an empty
+      keywords list)
+    * a tuple with a module as first element and the start argument as second -
+      such as `{Stack, [:hello]}`. When such format is used, the supervisor
+      will retrieve the child specification from the given module.
+    * a map representing the child specification itself - such as the child
+      specification map outlined in the previous section.
 
   The second argument is a keyword list of options:
 
@@ -528,7 +528,7 @@ defmodule Supervisor do
   process and exits not only on crashes but also if the parent process exits
   with `:normal` reason.
   """
-  @spec start_link([child_spec | {module, term} | module], options) :: on_start
+  @spec start_link([:supervisor.child_spec | {module, term} | module], options) :: on_start
   def start_link(children, options) when is_list(children) do
     {sup_opts, start_opts} = Keyword.split(options, [:strategy, :max_seconds, :max_restarts])
     start_link(Supervisor.Default, {children, sup_opts}, start_opts)
@@ -570,7 +570,7 @@ defmodule Supervisor do
   is allowed within 5 seconds. Check the `Supervisor` module for a detailed
   description of the available strategies.
   """
-  @spec init([child_spec | {module, term} | module], flag) :: {:ok, tuple}
+  @spec init([:supervisor.child_spec | {module, term} | module], flag) :: {:ok, tuple}
   def init(children, options) when is_list(children) and is_list(options) do
     unless strategy = options[:strategy] do
       raise ArgumentError, "expected :strategy option to be given"
@@ -591,12 +591,7 @@ defmodule Supervisor do
       e in UndefinedFunctionError ->
         case System.stacktrace do
           [{^module, :child_spec, [^arg], _} | _] ->
-            raise ArgumentError,
-                  "The module #{inspect module} was given as a child to a supervisor but it " <>
-                  "does not implement child_spec/1. If you own the given module, please define " <>
-                  "a child_spec/1 function that receives an argument and returns a child " <>
-                  "specification. If not, pass a map child specification instead of the " <>
-                  "given module to the supervisor. See the Supervisor module for documentation."
+            raise ArgumentError, child_spec_error(module)
           stack ->
             reraise e, stack
         end
@@ -605,10 +600,59 @@ defmodule Supervisor do
   defp init_child(map) when is_map(map) do
     map
   end
+  defp init_child({_, _, _, _, _, _} = tuple) do
+    tuple
+  end
   defp init_child(other) do
-    raise ArgumentError,
-          "supervisors expect the child to be a module, a {module, arg} tuple or " <>
-          "a map with the child specification, got: #{inspect other}"
+    raise ArgumentError, """
+    supervisors expect each child to be one of:
+
+      * a module
+      * a {module, arg} tuple
+      * a child specification as a map with at least the :id and :start fields
+      * or a tuple with 6 elements generated by Supervisor.Spec (deprecated)
+
+    Got: #{inspect other}
+    """
+  end
+
+  defp child_spec_error(module) do
+    if Code.ensure_loaded?(module) do
+      """
+      The module #{inspect module} was given as a child to a supervisor
+      but it does not implement child_spec/1.
+
+      If you own the given module, please define a child_spec/1 function
+      that receives an argument and returns a child specification as a map.
+      For example:
+
+          def child_spec(opts) do
+            %{
+              id: __MODULE__,
+              start: {__MODULE__, :start_link, [opts]},
+              type: :worker,
+              restart: :permanent,
+              shutdown: 500
+            }
+          end
+
+      Note that "use Agent", "use GenServer" and so on automatically define
+      this function for you.
+
+      However, if you don't own the given module and it doesn't implement
+      child_spec/1, instead of passing the module name directly as a supervisor
+      child, you will have to pass a child specification as a map:
+
+          %{
+            id: #{inspect module},
+            start: {#{inspect module}, :start_link, [arg1, arg2]}
+          }
+
+      See the Supervisor documentation for more information.
+      """
+    else
+      "The module #{inspect module} was given as a child to a supervisor but it does not exist."
+    end
   end
 
   @doc """
@@ -645,7 +689,14 @@ defmodule Supervisor do
             start: {Agent, :start_link, []}}
 
   """
-  @spec child_spec(child_spec | {module, arg :: term} | module, keyword) :: child_spec()
+  @spec child_spec(child_spec() | {module, arg :: term} | module, keyword) :: child_spec()
+  def child_spec(module_or_map, overrides)
+
+  def child_spec({_, _, _, _, _, _} = tuple, _overrides) do
+    raise ArgumentError, "old tuple-based child specification #{inspect tuple} " <>
+                         "is not supported in Supervisor.child_spec/2"
+  end
+
   def child_spec(module_or_map, overrides) do
     Enum.reduce overrides, init_child(module_or_map), fn
       {key, value}, acc when key in [:id, :start, :restart, :shutdown, :type, :modules] ->
@@ -731,7 +782,7 @@ defmodule Supervisor do
   """
   # TODO: Once we add DynamicSupervisor, we need to enforce receiving
   # a map here and deprecate the list and tuple formats.
-  @spec start_child(supervisor, child_spec | [term]) :: on_start_child
+  @spec start_child(supervisor, :supervisor.child_spec | [term]) :: on_start_child
   def start_child(supervisor, child_spec_or_args) do
     call(supervisor, {:start_child, child_spec_or_args})
   end

lib/elixir/lib/supervisor/default.ex

@@ -1,10 +1,6 @@
 defmodule Supervisor.Default do
   @moduledoc false
 
-  def init({[{_, _, _, _, _, _} | _] = children, opts}) do
-    Supervisor.Spec.supervise(children, opts)
-  end
-
   def init({children, opts}) do
     Supervisor.init(children, opts)
   end

lib/elixir/lib/supervisor/spec.ex

@@ -1,7 +1,8 @@
 defmodule Supervisor.Spec do
   @moduledoc """
-  NOTE: The functions in this module are deprecated. Please see
-  `Supervisor` instead.
+  NOTE: The functions in this module are deprecated and they do not
+  work with the module-based child specs introduced in Elixir v1.5.
+  Please see the `Supervisor` documentation instead.
 
   Convenience functions for defining supervisor specifications.
 
@@ -172,10 +173,21 @@ defmodule Supervisor.Spec do
     maxR = Keyword.get(options, :max_restarts, 3)
     maxS = Keyword.get(options, :max_seconds, 5)
 
-    assert_unique_ids(Enum.map(children, &elem(&1, 0)))
+    assert_unique_ids(Enum.map(children, &get_id/1))
     {:ok, {{strategy, maxR, maxS}, children}}
   end
 
+  defp get_id({id, _, _, _, _, _}) do
+    id
+  end
+  defp get_id(other) do
+    raise ArgumentError,
+      "invalid tuple specification given to supervise/2. If you are trying to use " <>
+      "the map child specification that is part of the Elixir v1.5, use Supervisor.init/2 " <>
+      "instead of Supervisor.Spec.supervise/2. See the Supervisor module for more information. " <>
+      "Got: #{inspect other}"
+  end
+
   defp assert_unique_ids([id | rest]) do
     if id in rest do
       raise ArgumentError,

lib/elixir/lib/system.ex

@@ -339,6 +339,8 @@ defmodule System do
   """
   @spec find_executable(binary) :: binary | nil
   def find_executable(program) when is_binary(program) do
+    assert_no_null_byte!(program, "System.find_executable/1")
+
     case :os.find_executable(String.to_charlist(program)) do
       false -> nil
       other -> List.to_string(other)
@@ -393,8 +395,13 @@ defmodule System do
   """
   @spec put_env(binary, binary) :: :ok
   def put_env(varname, value) when is_binary(varname) and is_binary(value) do
-   :os.putenv String.to_charlist(varname), String.to_charlist(value)
-   :ok
+    case :binary.match(varname, "=") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute System.put_env/2 for key with \"=\", got: #{inspect varname}"
+      :nomatch ->
+        :os.putenv String.to_charlist(varname), String.to_charlist(value)
+        :ok
+    end
   end
 
   @doc """
@@ -588,6 +595,7 @@ defmodule System do
   @spec cmd(binary, [binary], keyword) ::
         {Collectable.t, exit_status :: non_neg_integer}
   def cmd(command, args, opts \\ []) when is_binary(command) and is_list(args) do
+    assert_no_null_byte!(command, "System.cmd/3")
     cmd = String.to_charlist(command)
 
     cmd =
@@ -833,6 +841,15 @@ defmodule System do
     :erlang.unique_integer(modifiers)
   end
 
+  defp assert_no_null_byte!(binary, operation) do
+    case :binary.match(binary, "\0") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute #{operation} for program with null byte, got: #{inspect binary}"
+      :nomatch ->
+        binary
+    end
+  end
+
   defp normalize_time_unit(:native),
     do: :native
 

lib/elixir/lib/task.ex

@@ -345,9 +345,8 @@ defmodule Task do
   terminates all tasks.
 
   When streamed, each task will emit `{:ok, value}` upon successful
-  completion or `{:exit, reason, element}` if the caller is trapping
-  exits, where `element` is the stream element. Results are emitted
-  in the same order as the original `enumerable`.
+  completion or `{:exit, reason}` if the caller is trapping exits.
+  Results are emitted in the same order as the original `enumerable`.
 
   The level of concurrency can be controlled via the `:max_concurrency`
   option and defaults to `System.schedulers_online/0`. A timeout
@@ -373,8 +372,7 @@ defmodule Task do
       values are:
       * `:exit` (default) - the process that spawned the tasks exits.
       * `:kill_task` - the task that timed out is killed. The value
-        emitted for that task is `{:exit, :timeout, element}`, where
-        `element` is the element it timed out on.
+        emitted for that task is `{:exit, :timeout}`.
 
   ## Example
 

lib/elixir/lib/task/supervised.ex

@@ -405,7 +405,13 @@ defmodule Task.Supervised do
       {:spawn, position, value} ->
         {type, pid} = spawn.(parent_pid, normalize_mfa_with_arg(mfa, value))
         ref = Process.monitor(pid)
-        timer_ref = Process.send_after(self(), {:timeout, {monitor_ref, ref}}, timeout)
+
+        # Schedule a timeout message to ourselves, unless the timeout was set to :infinity
+        timer_ref = case timeout do
+          :infinity -> nil
+          timeout -> Process.send_after(self(), {:timeout, {monitor_ref, ref}}, timeout)
+        end
+
         send(parent_pid, {:spawned, {monitor_ref, position}, pid})
         task_info = %{
           position: position,
@@ -443,7 +449,9 @@ defmodule Task.Supervised do
       # this and keep going.
       {:DOWN, ref, _, _, reason} ->
         {%{position: position, timer_ref: timer_ref, timed_out?: timed_out?}, running_tasks} = Map.pop(running_tasks, ref)
-        :ok = Process.cancel_timer(timer_ref, async: true, info: false)
+        if timer_ref != nil do
+          :ok = Process.cancel_timer(timer_ref, async: true, info: false)
+        end
         message_kind = if(timed_out?, do: :timed_out, else: :down)
         send(parent_pid, {message_kind, {monitor_ref, position}, reason})
         stream_monitor_loop(running_tasks, config)

lib/elixir/lib/task/supervisor.ex

@@ -153,9 +153,8 @@ defmodule Task.Supervisor do
   linked to the current process, similarly to `async/4`.
 
   When streamed, each task will emit `{:ok, value}` upon successful
-  completion or `{:exit, reason, element}` if the caller is trapping
-  exits, where `element` is the stream element. Results are emitted
-  in the same order as the original `enumerable`.
+  completion or `{:exit, reason}` if the caller is trapping exits.
+  Results are emitted in the same order as the original `enumerable`.
 
   The level of concurrency can be controlled via the `:max_concurrency`
   option and defaults to `System.schedulers_online/0`. A timeout
@@ -185,8 +184,7 @@ defmodule Task.Supervisor do
       values are:
       * `:exit` (default) - the process that spawned the tasks exits.
       * `:kill_task` - the task that timed out is killed. The value
-        emitted for that task is `{:exit, :timeout, element}`, where
-        `element` is the element it timed out on.
+        emitted for that task is `{:exit, :timeout}`.
 
   ## Examples
 

lib/elixir/lib/uri.ex

@@ -42,7 +42,7 @@ defmodule URI do
   """
   @spec default_port(binary) :: nil | non_neg_integer
   def default_port(scheme) when is_binary(scheme) do
-    :elixir_config.get({:uri, scheme})
+    :elixir_config.safe_get({:uri, scheme}, nil)
   end
 
   @doc """

lib/elixir/pages/Deprecations.md

@@ -33,6 +33,7 @@ Deprecated feature                               | Deprecated in | Replaced by (
 `:char_lists` key in `t:Inspect.Opts.t/0` type   | [v1.5]        | `:charlists` (v1.3)
 `char_list/0` type                               | [v1.5]        | `charlist/0` type (v1.3)
 `@compile {:parse_transform, _}` in `Module`     | [v1.5]        | *None*
+EEx: `<%=` in middle and end expressions         | [v1.5]        | Use `<%` (= is allowed only on start expressions) (v1.0)
 `Access.key/1`                                   | [v1.4]        | `Access.key/2` (v1.3)
 `Behaviour` module                               | [v1.4]        | `@callback` (v1.0)
 `Enum.uniq/2`                                    | [v1.4]        | `Enum.uniq_by/2` (v1.2)
@@ -68,4 +69,4 @@ Empty string in `String.starts_with?/2`, `String.ends_with?/2`, `String.contains
 [v1.2]: https://github.com/elixir-lang/elixir/blob/v1.2/CHANGELOG.md#changelog-for-elixir-v12
 [v1.3]: https://github.com/elixir-lang/elixir/blob/v1.3/CHANGELOG.md#4-deprecations
 [v1.4]: https://github.com/elixir-lang/elixir/blob/v1.4/CHANGELOG.md#4-deprecations
-[v1.5]: https://github.com/elixir-lang/elixir/blob/master/CHANGELOG.md#4-deprecations
+[v1.5]: https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md#4-deprecations

lib/elixir/pages/Guards.md

@@ -104,14 +104,14 @@ Other constructs are `for`, `with`, `try`/`rescue`/`catch`/`else`/, and the `mat
 
 ## Failing guards
 
-Errors in guards do not result in a runtime error, but in the erroring guard fail. For example, the `length/1` function only works with lists, and if we use it on anything else it fails:
+Errors in guards do not result in runtime errors, but in guards failing. For example, the `length/1` function only works with lists. If we use it with anything else, a runtime error is raised:
 
 ```elixir
 iex> length("hello")
 ** (ArgumentError) argument error
 ```
 
-However, when used in guards, it simply makes the corresponding clause fail (i.e., not match):
+However, when used in guards, the corresponding clause simply fails to match:
 
 ```elixir
 iex> case "hello" do
@@ -123,7 +123,7 @@ iex> case "hello" do
 :length_failed
 ```
 
-In many cases, we can take advantage of this: in the code above, for example, we can use `length/1` to both check that the given thing is a list *and* check some properties of its length (instead of using `is_list(something) and length(something) > 0`).
+In many cases, we can take advantage of this. In the code above, we used `length/1` to both check that the given thing is a list *and* check some properties of its length (instead of using `is_list(something) and length(something) > 0`).
 
 ## Expressions in guard clauses
 

lib/elixir/pages/Syntax Reference.md

@@ -82,13 +82,13 @@ end
 
 ### Strings
 
-Strings in Elixir are written between double-quotes, such as `"foo"`. Any double-quote inside the string must be escaped with `\`. Strings support Unicode characters and are stored in UTF-8 encoding.
+Strings in Elixir are written between double-quotes, such as `"foo"`. Any double-quote inside the string must be escaped with `\ `. Strings support Unicode characters and are stored in UTF-8 encoding.
 
 Strings are always represented as themselves in the AST.
 
 ### Charlists
 
-Charlists in Elixir are written in single-quotes, such as `'foo'`. Any single-quote inside the string must be escaped with `\`. Charlists are a list of integers, each integer representing a Unicode character.
+Charlists in Elixir are written in single-quotes, such as `'foo'`. Any single-quote inside the string must be escaped with `\ `. Charlists are a list of integers, each integer representing a Unicode character.
 
 Charlists are always represented as themselves in the AST.
 

lib/elixir/src/elixir.erl

@@ -102,8 +102,10 @@ start(_Type, _Args) ->
                    relative_paths => true},
   {ok, [[Home] | _]} = init:get_argument(home),
   Config = [{at_exit, []},
-            {home, unicode:characters_to_binary(Home, Encoding, Encoding)},
+            {argv, []},
+            {bootstrap, false},
             {compiler_options, CompilerOpts},
+            {home, unicode:characters_to_binary(Home, Encoding, Encoding)},
             {identifier_tokenizer, Tokenizer}
             | URIConfig],
   Tab = elixir_config:new(Config),
@@ -183,8 +185,14 @@ env_for_eval(Env, Opts) ->
     false -> nil
   end,
 
+  FA = case lists:keyfind(function, 1, Opts) of
+    {function, {Function, Arity}} when is_atom(Function), is_integer(Arity) -> {Function, Arity};
+    {function, nil} -> nil;
+    false -> nil
+  end,
+
   Env#{
-    file := File, module := Module,
+    file := File, module := Module, function := FA,
     macros := Macros, functions := Functions,
     requires := Requires, aliases := Aliases, line := Line
   }.

lib/elixir/src/elixir.hrl

@@ -5,15 +5,13 @@
 -define(var_context, ?MODULE).
 
 -record(elixir_erl, {
-  def=nil,                 %% a tuple with the current definition {def | ..., name, arity}
   context=nil,             %% can be match, guards or nil
   extra=nil,               %% extra information about the context, like pin_guard and map_key
   caller=false,            %% when true, it means caller was invoked
   vars=#{},                %% a map of defined variables and their alias
   backup_vars=nil,         %% a copy of vars to be used on ^var
-  match_vars=nil,          %% a set of all variables defined in a particular match
   export_vars=nil,         %% a dict of all variables defined in a particular clause
-  extra_guards=nil,        %% extra guards from args expansion
+  extra_guards=[],         %% extra guards from args expansion
   counter=#{},             %% a map counting the variables defined
   file=(<<"nofile">>)      %% the current scope filename
 }).

lib/elixir/src/elixir_aliases.erl

@@ -4,7 +4,7 @@
 -include("elixir.hrl").
 
 inspect(Atom) when is_atom(Atom) ->
-  case elixir_compiler:get_opt(internal) of
+  case elixir_config:get(bootstrap) of
     true  -> atom_to_binary(Atom, utf8);
     false -> 'Elixir.Inspect.Atom':inspect(Atom)
   end.

lib/elixir/src/elixir_bitstring.erl

@@ -22,8 +22,8 @@ expand(BitstrMeta, Fun, [{'::', Meta, [Left, Right]} | T], Acc, E, RequireSize)
   %% on subparts, however we can't assign new variables.
   {ER, MatchSize} =
     case E of
-      {EExtracted, _} -> {EExtracted, false};          %% expand_arg,  no assigns
-      _               -> {E#{context := nil}, T /= []} %% expand, revert assigns
+      {EExtracted, _} -> {EExtracted, false}; %% expand_arg,  no assigns
+      _ -> {E#{context := nil, match_vars := warn}, T /= []} %% expand, revert assigns
     end,
 
   ERight = expand_specs(expr_type(ELeft), Meta, Right, ER, RequireSize or MatchSize),

lib/elixir/src/elixir_clauses.erl

@@ -2,20 +2,20 @@
 %% fn, receive and friends. try is handled in elixir_try.
 -module(elixir_clauses).
 -export([match/3, clause/5, def/2, head/2,
-         'case'/3, 'receive'/3, 'try'/3, 'cond'/3,
+         'case'/3, 'receive'/3, 'try'/3, 'cond'/3, with/3,
          format_error/1]).
 -import(elixir_errors, [form_error/4]).
 -include("elixir.hrl").
 
 match(Fun, Expr, #{context := match} = E) ->
   Fun(Expr, E);
-match(Fun, Expr, #{context := Context, prematch_vars := nil, vars := Vars} = E) ->
-  {EExpr, EE} = Fun(Expr, E#{context := match, prematch_vars := Vars}),
-  {EExpr, EE#{context := Context, prematch_vars := nil}}.
+match(Fun, Expr, #{context := Context, match_vars := Match, prematch_vars := nil, vars := Vars} = E) ->
+  {EExpr, EE} = Fun(Expr, E#{context := match, match_vars := [], prematch_vars := Vars}),
+  {EExpr, EE#{context := Context, match_vars := Match, prematch_vars := nil}}.
 
-def({Meta, Args, Guards, Body}, E) ->
-  {EArgs, EA}   = elixir_expand:expand(Args, E#{context := match}),
-  {EGuards, EG} = guard(Guards, EA#{context := guard}),
+def({Meta, Args, Guards, Body}, #{match_vars := Match} = E) ->
+  {EArgs, EA}   = elixir_expand:expand(Args, E#{context := match, match_vars := []}),
+  {EGuards, EG} = guard(Guards, EA#{context := guard, match_vars := Match}),
   {EBody, _}    = elixir_expand:expand(Body, EG#{context := ?key(E, context)}),
   {Meta, EArgs, EGuards, EBody}.
 
@@ -112,6 +112,66 @@ expand_receive(Meta, {'after', _}, _Acc, E) ->
 expand_receive(Meta, {Key, _}, _Acc, E) ->
   form_error(Meta, ?key(E, file), ?MODULE, {unexpected_option, 'receive', Key}).
 
+%% With
+
+with(Meta, Args, E) ->
+  {Exprs, Opts0} =
+    case elixir_utils:split_last(Args) of
+      {_, LastArg} = SplitResult when is_list(LastArg) ->
+        SplitResult;
+      _ ->
+        {Args, []}
+    end,
+
+  {EExprs, {EE, HasMatch}} = lists:mapfoldl(fun expand_with/2, {E, false}, Exprs),
+  {EDo, Opts1} = expand_with_do(Meta, Opts0, EE),
+  {EOpts, Opts2} = expand_with_else(Meta, Opts1, E, HasMatch),
+
+  case Opts2 of
+    [{Key, _} | _] ->
+      form_error(Meta, ?key(E, file), elixir_clauses, {unexpected_option, with, Key});
+    [] ->
+      ok
+  end,
+
+  {{with, Meta, EExprs ++ [[{do, EDo} | EOpts]]}, E}.
+
+expand_with({'<-', Meta, [{Name, _, Ctx}, _] = Args}, Acc) when is_atom(Name), is_atom(Ctx) ->
+  expand_with({'=', Meta, Args}, Acc);
+expand_with({'<-', Meta, [Left, Right]}, {E, _HasMatch}) ->
+  {ERight, ER} = elixir_expand:expand(Right, E),
+  {[ELeft], EL}  = head([Left], E),
+  {{'<-', Meta, [ELeft, ERight]}, {elixir_env:mergev(EL, ER), true}};
+expand_with(Expr, {E, HasMatch}) ->
+  {EExpr, EE} = elixir_expand:expand(Expr, E),
+  {EExpr, {EE, HasMatch}}.
+
+expand_with_do(Meta, Opts, E) ->
+  case lists:keytake(do, 1, Opts) of
+    {value, {do, Expr}, RestOpts} ->
+      {EExpr, _} = elixir_expand:expand(Expr, E),
+      {EExpr, RestOpts};
+    false ->
+      form_error(Meta, ?key(E, file), elixir_expand, {missing_option, 'with', [do]})
+  end.
+
+expand_with_else(Meta, Opts, E, HasMatch) ->
+  case lists:keytake(else, 1, Opts) of
+    {value, Pair, RestOpts} ->
+      if
+        HasMatch ->
+          ok;
+        true ->
+          Message = "\"else\" clauses will never match because all patterns in \"with\" will always match",
+          elixir_errors:warn(?line(Meta), ?key(E, file), Message)
+      end,
+      Fun = expand_one(Meta, 'with', 'else', fun head/2),
+      EPair = expand_without_export(Meta, 'with', Fun, Pair, E),
+      {[EPair], RestOpts};
+    false ->
+      {[], Opts}
+  end.
+
 %% Try
 
 'try'(Meta, [], E) ->
@@ -168,6 +228,11 @@ expand_rescue(Meta, _, E) ->
 expand_rescue({Name, _, Atom} = Var, E) when is_atom(Name), is_atom(Atom) ->
   match(fun elixir_expand:expand/2, Var, E);
 
+%% rescue var in _ => rescue var
+expand_rescue({in, _, [{Name, _, VarContext} = Var, {'_', _, UnderscoreContext}]}, E)
+    when is_atom(Name), is_atom(VarContext), is_atom(UnderscoreContext) ->
+  expand_rescue(Var, E);
+
 %% rescue var in [Exprs]
 expand_rescue({in, Meta, [Left, Right]}, E) ->
   {ELeft, EL}  = match(fun elixir_expand:expand/2, Left, E),

lib/elixir/src/elixir_compiler.erl

@@ -125,9 +125,9 @@ allows_fast_compilation(_) -> false.
 
 bootstrap() ->
   {ok, _} = application:ensure_all_started(elixir),
-  Update = fun(Old) -> maps:merge(Old, #{docs => false, internal => true,
-                                         relative_paths => false}) end,
+  Update = fun(Old) -> maps:merge(Old, #{docs => false, relative_paths => false}) end,
   _ = elixir_config:update(compiler_options, Update),
+  _ = elixir_config:put(bootstrap, true),
   [bootstrap_file(File) || File <- bootstrap_main()].
 
 bootstrap_file(File) ->

lib/elixir/src/elixir_config.erl

@@ -1,6 +1,6 @@
 -module(elixir_config).
 -compile({no_auto_import, [get/1]}).
--export([new/1, delete/1, put/2, get/1, update/2, get_and_put/2]).
+-export([new/1, delete/1, put/2, get/1, safe_get/2, update/2, get_and_put/2]).
 -export([start_link/0, init/1, handle_call/3, handle_cast/2,
   handle_info/2, code_change/3, terminate/2]).
 -behaviour(gen_server).
@@ -19,9 +19,15 @@ put(Key, Value) ->
   gen_server:call(?MODULE, {put, Key, Value}).
 
 get(Key) ->
-  case ets:lookup(?MODULE, Key) of
+  [{_, Value}] = ets:lookup(?MODULE, Key),
+  Value.
+
+safe_get(Key, Default) ->
+  try ets:lookup(?MODULE, Key) of
     [{_, Value}] -> Value;
-    []          -> nil
+    [] -> Default
+  catch
+    _:_ -> Default
   end.
 
 update(Key, Fun) ->

lib/elixir/src/elixir_def.erl

@@ -22,7 +22,7 @@ local_for(Module, Name, Arity, Kinds) ->
   of
     {[{_, Kind, Meta, File, _, _}], Clauses} ->
       case (Kinds == all) orelse (lists:member(Kind, Kinds)) of
-        true -> elixir_erl:definition_to_anonymous(File, Module, Tuple, Kind, Meta,
+        true -> elixir_erl:definition_to_anonymous(File, Module, Kind, Meta,
                                                    [Clause || {_, Clause} <- Clauses]);
         false -> false
       end;
@@ -34,10 +34,11 @@ local_for(Module, Name, Arity, Kinds) ->
 
 %% Take a definition out of the table
 
-take_definition(Module, Tuple) ->
+take_definition(Module, {Name, Arity} = Tuple) ->
   Table = elixir_module:defs_table(Module),
   case ets:take(Table, {def, Tuple}) of
-    [Result] ->
+    [{{def, Tuple}, _, _, _, _, {Defaults, _, _}} = Result] ->
+      ets:delete_object(Table, {{default, Name}, Arity, Defaults}),
       {Result, [Clause || {_, Clause} <- ets:take(Table, {clauses, Tuple})]};
     [] ->
       false
@@ -58,8 +59,8 @@ fetch_definition([[Tuple] | T], File, Module, Table, All, Private) ->
 
   try ets:lookup_element(Table, {clauses, Tuple}, 2) of
     Clauses ->
-      Unwrapped =
-        {Tuple, Kind, Meta, Clauses},
+      NewAll =
+        [{Tuple, Kind, Meta, Clauses} | All],
       NewPrivate =
         case (Kind == defp) orelse (Kind == defmacrop) of
           true ->
@@ -68,7 +69,7 @@ fetch_definition([[Tuple] | T], File, Module, Table, All, Private) ->
           false ->
             Private
         end,
-      fetch_definition(T, File, Module, Table, [Unwrapped | All], NewPrivate)
+      fetch_definition(T, File, Module, Table, NewAll, NewPrivate)
   catch
     error:badarg ->
       warn_bodyless_function(Check, Meta, File, Module, Kind, Tuple),
@@ -95,6 +96,10 @@ store_definition(Kind, CheckClauses, Call, Body, Pos) ->
   %% extract meta information like file and context.
   {_, Meta, _} = Call,
   DoCheckClauses = (not lists:keymember(context, 1, Meta)) andalso (CheckClauses),
+  Generated = case lists:keyfind(generated, 1, Meta) of
+    {generated, true} -> ?generated([]);
+    _ -> []
+  end,
 
   %% Check if there is a file information in the definition.
   %% If so, we assume this come from another source and
@@ -113,7 +118,6 @@ store_definition(Kind, CheckClauses, Call, Body, Pos) ->
   LinifyArgs   = elixir_quote:linify(Line, Key, Args),
   LinifyGuards = elixir_quote:linify(Line, Key, Guards),
   LinifyBody   = elixir_quote:linify(Line, Key, Body),
-  Generated    = case DoCheckClauses of true -> []; false -> ?generated([]) end,
 
   {EL, MetaLocation} =
     case retrieve_location(Location, ?key(E, module)) of
@@ -186,6 +190,13 @@ store_definition(Check, Kind, Meta, Name, Arity, File, Module, Defaults, Clauses
   Tuple   = {Name, Arity},
   HasBody = Clauses =/= [],
 
+  if
+    Defaults > 0 ->
+      ets:insert(Defs, {{default, Name}, Arity, Defaults});
+    true ->
+      ok
+  end,
+
   MaxDefaults =
     case ets:take(Defs, {def, Tuple}) of
       [{_, StoredKind, StoredMeta, StoredFile, StoredCheck,
@@ -194,14 +205,14 @@ store_definition(Check, Kind, Meta, Name, Arity, File, Module, Defaults, Clauses
         (Check and StoredCheck) andalso
           check_valid_clause(Meta, File, Name, Arity, Kind, Data, StoredMeta, StoredFile),
         check_valid_defaults(Meta, File, Name, Arity, Kind, Defaults, StoredDefaults, LastDefaults, LastHasBody),
-        {max(Defaults, StoredDefaults), HasBody, Defaults};
+        max(Defaults, StoredDefaults);
       [] ->
-        {Defaults, HasBody, Defaults}
+        Defaults
     end,
 
   Check andalso ets:insert(Data, {?last_def, Tuple}),
   ets:insert(Defs, [{{clauses, Tuple}, Clause} || Clause <- Clauses]),
-  ets:insert(Defs, {{def, Tuple}, Kind, Meta, File, Check, MaxDefaults}).
+  ets:insert(Defs, {{def, Tuple}, Kind, Meta, File, Check, {MaxDefaults, HasBody, Defaults}}).
 
 %% Handling of defaults
 
@@ -212,7 +223,7 @@ unpack_defaults(Kind, Meta, Name, Args, E) ->
 unpack_defaults(Kind, Meta, Name, [{'\\\\', DefaultMeta, [Expr, _]} | T] = List, Acc, Clauses) ->
   Base = match_defaults(Acc, length(Acc), []),
   {Args, Invoke} = extract_defaults(List, length(Base), [], []),
-  Clause = {Meta, Base ++ Args, [], {super, DefaultMeta, Base ++ Invoke}},
+  Clause = {Meta, Base ++ Args, [], {super, DefaultMeta, [{Kind, Name} | Base] ++ Invoke}},
   unpack_defaults(Kind, Meta, Name, T, [Expr | Acc], [Clause | Clauses]);
 unpack_defaults(Kind, Meta, Name, [H | T], Acc, Clauses) ->
   unpack_defaults(Kind, Meta, Name, T, [H | Acc], Clauses);
@@ -271,7 +282,7 @@ check_valid_defaults(Meta, File, Name, Arity, Kind, Defaults, 0, 0, _) when Defa
 check_valid_defaults(Meta, File, Name, Arity, Kind, 0, _, LastDefaults, true) when LastDefaults > 0 ->
   elixir_errors:form_warn(Meta, File, ?MODULE, {clauses_with_defaults, {Kind, Name, Arity}});
 % Clause without defaults
-check_valid_defaults(_Meta, _File, _Name, _Arity, _Kind, 0, _, _, _) -> [].
+check_valid_defaults(_Meta, _File, _Name, _Arity, _Kind, 0, _, _, _) -> ok.
 
 warn_bodyless_function(Check, _Meta, _File, Module, _Kind, _Tuple)
     when Check == false; Module == 'Elixir.Module' ->
@@ -289,12 +300,11 @@ invalid_arg({Name, _, Kind}) when is_atom(Name), is_atom(Kind) -> false;
 invalid_arg(_) -> true.
 
 check_previous_defaults(Meta, Module, Name, Arity, Kind, Defaults, E) ->
-  Matches = ets:match(elixir_module:defs_table(Module),
-                      {{def, {Name, '$2'}}, '$1', '_', '_', '_', {'$3', '_', '_'}}),
+  Matches = ets:lookup(elixir_module:defs_table(Module), {default, Name}),
   [begin
      elixir_errors:form_error(Meta, ?key(E, file), ?MODULE,
-       {defs_with_defaults, Name, {Kind, Arity}, {K, A}})
-   end || [K, A, D] <- Matches, A /= Arity, D /= 0, defaults_conflict(A, D, Arity, Defaults)].
+       {defs_with_defaults, Kind, Name, Arity, A})
+   end || {_, A, D} <- Matches, A /= Arity, D /= 0, defaults_conflict(A, D, Arity, Defaults)].
 
 defaults_conflict(A, D, Arity, Defaults) ->
   ((Arity >= (A - D)) andalso (Arity < A)) orelse
@@ -324,13 +334,13 @@ format_error({bodyless_clause, Kind, {Name, Arity}}) ->
 format_error({no_module, {Kind, Name, Arity}}) ->
   io_lib:format("cannot define function outside module, invalid scope for ~ts ~ts/~B", [Kind, Name, Arity]);
 
-format_error({defs_with_defaults, Name, {Kind, Arity}, {K, A}}) when Arity > A ->
-  io_lib:format("~ts ~ts/~B defaults conflicts with ~ts ~ts/~B",
-    [Kind, Name, Arity, K, Name, A]);
+format_error({defs_with_defaults, Kind, Name, Arity, A}) when Arity > A ->
+  io_lib:format("~ts ~ts/~B defaults conflicts with ~ts/~B",
+    [Kind, Name, Arity, Name, A]);
 
-format_error({defs_with_defaults, Name, {Kind, Arity}, {K, A}}) when Arity < A ->
-  io_lib:format("~ts ~ts/~B conflicts with defaults from ~ts ~ts/~B",
-    [Kind, Name, Arity, K, Name, A]);
+format_error({defs_with_defaults, Kind, Name, Arity, A}) when Arity < A ->
+  io_lib:format("~ts ~ts/~B conflicts with defaults from ~ts/~B",
+    [Kind, Name, Arity, Name, A]);
 
 format_error({clauses_with_defaults, {Kind, Name, Arity}}) ->
   io_lib:format(""

lib/elixir/src/elixir_env.erl

@@ -19,10 +19,13 @@ new() ->
     lexical_tracker => nil,                %% holds the lexical tracker PID
     vars => [],                            %% a set of defined variables
     export_vars => nil,                    %% a set of variables to be exported in some constructs
-    prematch_vars => nil}.                 %% a set of variables defined before the current match
+    prematch_vars => nil,                  %% a set of variables defined before the current match
+    match_vars => warn}.                   %% handling of new variables
 
 linify({Line, Env}) ->
-  Env#{line := Line}.
+  Env#{line := Line};
+linify(#{} = Env) ->
+  Env.
 
 env_to_scope(#{file := File, context := Context}) ->
   #elixir_erl{file=File, context=Context}.

lib/elixir/src/elixir_erl.erl

@@ -1,8 +1,8 @@
 %% Compiler backend to Erlang.
 -module(elixir_erl).
--export([elixir_to_erl/1, definition_to_anonymous/6, compile/1,
+-export([elixir_to_erl/1, definition_to_anonymous/5, compile/1,
          get_ann/1, remote/4, add_beam_chunks/2, debug_info/4,
-         definition_scope/5]).
+         definition_scope/2]).
 -include("elixir.hrl").
 
 %% TODO: Remove extra chunk functionality when OTP 20+.
@@ -61,8 +61,8 @@ remote(Ann, Module, Function, Args) when is_atom(Module), is_atom(Function), is_
 
 %% Converts an Elixir definition to an anonymous function.
 
-definition_to_anonymous(File, Module, {Name, Arity}, Kind, Meta, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Name, Arity, Clause, File) || Clause <- Clauses],
+definition_to_anonymous(File, Module, Kind, Meta, Clauses) ->
+  ErlClauses = [translate_clause(Kind, Clause, File) || Clause <- Clauses],
   Fun = {'fun', ?ann(Meta), {clauses, ErlClauses}},
   LocalHandler = fun(LocalName, LocalArgs) -> invoke_local(Module, LocalName, LocalArgs) end,
   {value, Result, _Binding} = erl_eval:expr(Fun, [], {value, LocalHandler}),
@@ -121,9 +121,9 @@ elixir_to_erl(Function) when is_function(Function) ->
     false ->
       error(badarg)
   end;
-elixir_to_erl(PidOrRef) when is_pid(PidOrRef); is_reference(PidOrRef) ->
+elixir_to_erl(Pid) when is_pid(Pid) ->
   elixir_erl:remote(0, erlang, binary_to_term,
-    [elixir_erl:elixir_to_erl(term_to_binary(PidOrRef))]);
+    [elixir_erl:elixir_to_erl(term_to_binary(Pid))]);
 elixir_to_erl(_Other) ->
   error(badarg).
 
@@ -137,14 +137,14 @@ elixir_to_erl_cons2([], Acc) ->
 
 %% Returns a definition scope for translation.
 
-definition_scope(Meta, Kind, Name, Arity, File) ->
+definition_scope(Meta, File) ->
   %% TODO: We only need to do this dance because some
   %% warnings are raised in elixir_erl_pass. Once we remove
   %% all warnings from the Erlang pass, we can remove the
   %% file field from #elixir_erl and clean up the code.
   case lists:keyfind(location, 1, Meta) of
-    {location, {F, _}} -> #elixir_erl{def = {Kind, Name, Arity}, file = F};
-    false -> #elixir_erl{def = {Kind, Name, Arity}, file = File}
+    {location, {F, _}} -> #elixir_erl{file=F};
+    false -> #elixir_erl{file=File}
   end.
 
 %% Compilation hook.
@@ -153,43 +153,63 @@ compile(#{module := Module} = Map) ->
   Data = elixir_module:data_table(Module),
   {Prefix, Forms, Defmacro, Unreachable} = dynamic_form(Map),
   Specs =
-    case elixir_compiler:get_opt(internal) of
+    case elixir_config:get(bootstrap) of
       true -> [];
       false -> specs_form(Data, Defmacro, Unreachable, types_form(Data, []))
     end,
   load_form(Map, Data, Prefix, Forms, Specs).
 
+dynamic_form(#{module := Module, line := Line, file := File, attributes := Attributes,
+               definitions := Definitions, unreachable := Unreachable}) ->
+  {Def, Defmacro, Macros, Exports, Functions} =
+    split_definition(Definitions, File, Unreachable, [], [], [], [], {[], []}),
+
+  Location = {elixir_utils:characters_to_list(elixir_utils:relative_to_cwd(File)), Line},
+  Prefix = [{attribute, Line, file, Location},
+            {attribute, Line, module, Module},
+            {attribute, Line, compile, no_auto_import}],
+
+  Forms0 = functions_form(Line, Module, Def, Defmacro, Exports, Functions),
+  Forms1 = attributes_form(Line, Attributes, Forms0),
+  {Prefix, Forms1, Macros, Unreachable}.
+
 % Definitions
 
-split_definition([{Tuple, def, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  {_, _, N, A, _} = Function = translate_definition(def, Meta, File, Tuple, Clauses),
-  split_definition(T, File, Unreachable, [Tuple | Def], Defmacro, [{N, A} | Exports],
-                   add_definition(Meta, Function, Functions));
-
-split_definition([{Tuple, defp, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  Function = translate_definition(defp, Meta, File, Tuple, Clauses),
+split_definition([{Tuple, Kind, Meta, Clauses} | T], File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
   case lists:member(Tuple, Unreachable) of
     false ->
-      split_definition(T, File, Unreachable, Def, Defmacro, Exports,
-                       add_definition(Meta, Function, Functions));
+      split_definition(Tuple, Kind, Meta, Clauses, T, File, Unreachable,
+                       Def, Defmacro, Macros, Exports, Functions);
     true ->
-      split_definition(T, File, Unreachable, Def, Defmacro, Exports, Functions)
+      split_definition(T, File, Unreachable, Def, Defmacro, Macros, Exports, Functions)
   end;
+split_definition([], _File, _Unreachable, Def, Defmacro, Macros, Exports, {Head, Tail}) ->
+  {Def, Defmacro, Macros, Exports, Head ++ Tail}.
 
-split_definition([{Tuple, defmacro, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  {_, _, N, A, _} = Function = translate_definition(defmacro, Meta, File, Tuple, Clauses),
-  split_definition(T, File, Unreachable, Def, [Tuple | Defmacro], [{N, A} | Exports],
-                   add_definition(Meta, Function, Functions));
+split_definition(Tuple, def, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  {_, _, N, A, _} = Entry = translate_definition(def, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, [Tuple | Def], Defmacro, Macros, [{N, A} | Exports],
+                   add_definition(Meta, Entry, Functions));
 
-split_definition([{_, defmacrop, _Meta, _Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  split_definition(T, File, Unreachable, Def, Defmacro, Exports, Functions);
+split_definition(Tuple, defp, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  Entry = translate_definition(defp, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, Defmacro, Macros, Exports,
+                   add_definition(Meta, Entry, Functions));
 
-split_definition([], _File, _Unreachable, Def, Defmacro, Exports, {Head, Tail}) ->
-  {Def, Defmacro, Exports, Head ++ Tail}.
+split_definition(Tuple, defmacro, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  {_, _, N, A, _} = Entry = translate_definition(defmacro, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, [Tuple | Defmacro], [Tuple | Macros], [{N, A} | Exports],
+                   add_definition(Meta, Entry, Functions));
+
+split_definition(Tuple, defmacrop, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  Entry = translate_definition(defmacro, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, Defmacro, [Tuple | Macros], Exports,
+                   add_definition(Meta, Entry, Functions)).
 
 add_definition(Meta, Body, {Head, Tail}) ->
   case lists:keyfind(location, 1, Meta) of
@@ -206,15 +226,15 @@ add_definition(Meta, Body, {Head, Tail}) ->
   end.
 
 translate_definition(Kind, Meta, File, {Name, Arity}, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Name, Arity, Clause, File) || Clause <- Clauses],
+  ErlClauses = [translate_clause(Kind, Clause, File) || Clause <- Clauses],
 
   case is_macro(Kind) of
     true -> {function, ?ann(Meta), elixir_utils:macro_name(Name), Arity + 1, ErlClauses};
     false -> {function, ?ann(Meta), Name, Arity, ErlClauses}
   end.
 
-translate_clause(Kind, Name, Arity, {Meta, Args, Guards, Body}, File) ->
-  S = definition_scope(Meta, Kind, Name, Arity, File),
+translate_clause(Kind, {Meta, Args, Guards, Body}, File) ->
+  S = definition_scope(Meta, File),
 
   {TClause, TS} = elixir_erl_clauses:clause(Meta,
                     fun elixir_erl_pass:translate_args/2, Args, Body, Guards, S),
@@ -245,20 +265,6 @@ is_macro(_)         -> false.
 
 % Functions
 
-dynamic_form(#{module := Module, line := Line, file := File, attributes := Attributes,
-               definitions := Definitions, unreachable := Unreachable}) ->
-  {Def, Defmacro, Exports, Functions} =
-    split_definition(Definitions, File, Unreachable, [], [], [], {[], []}),
-
-  Location = {elixir_utils:characters_to_list(elixir_utils:relative_to_cwd(File)), Line},
-  Prefix = [{attribute, Line, file, Location},
-            {attribute, Line, module, Module},
-            {attribute, Line, compile, no_auto_import}],
-
-  Forms0 = functions_form(Line, Module, Def, Defmacro, Exports, Functions),
-  Forms1 = attributes_form(Line, Attributes, Forms0),
-  {Prefix, Forms1, Defmacro, Unreachable}.
-
 functions_form(Line, Module, Def, Defmacro, Exports, Body) ->
   {Spec, Info} = add_info_function(Line, Module, Def, Defmacro),
   [{attribute, Line, export, lists:sort([{'__info__', 1} | Exports])}, Spec, Info | Body].

lib/elixir/src/elixir_erl_clauses.erl

@@ -1,49 +1,39 @@
 %% Handle code related to args, guard and -> matching for case,
 %% fn, receive and friends. try is handled in elixir_erl_try.
 -module(elixir_erl_clauses).
--export([match/3, clause/6, clauses/3, guards/3, get_clauses/3, get_clauses/4]).
+-export([match/3, clause/6, clauses/3, guards/3, get_clauses/3]).
 -include("elixir.hrl").
 
 %% Get clauses under the given key.
 
 get_clauses(Key, Keyword, As) ->
-  get_clauses(Key, Keyword, As, false).
-get_clauses(Key, Keyword, As, AllowNil) ->
   case lists:keyfind(Key, 1, Keyword) of
     {Key, Clauses} when is_list(Clauses) ->
       [{As, Meta, Left, Right} || {'->', Meta, [Left, Right]} <- Clauses];
-    {Key, nil} when AllowNil ->
-      [];
-    false ->
+    _ ->
       []
   end.
 
 %% Translate matches
 
-match(Fun, Args, #elixir_erl{context=Context, match_vars=MatchVars,
-    backup_vars=BackupVars, vars=Vars} = S) when Context /= match ->
-  {Result, NewS} = match(Fun, Args, S#elixir_erl{context=match,
-                         match_vars=#{}, backup_vars=Vars}),
-  {Result, NewS#elixir_erl{context=Context,
-      match_vars=MatchVars, backup_vars=BackupVars}};
+match(Fun, Args, #elixir_erl{context=Context, backup_vars=BackupVars, vars=Vars} = S) when Context /= match ->
+  {Result, NewS} = match(Fun, Args, S#elixir_erl{context=match, backup_vars=Vars}),
+  {Result, NewS#elixir_erl{context=Context, backup_vars=BackupVars}};
 match(Fun, Args, S) -> Fun(Args, S).
 
 %% Translate clauses with args, guards and expressions
 
 clause(Meta, Fun, Args, Expr, Guards, S) when is_list(Meta) ->
-  {TArgs, SA} = match(Fun, Args, S#elixir_erl{extra_guards=[]}),
-  {TExpr, SE} = elixir_erl_pass:translate(Expr,
-                  SA#elixir_erl{extra_guards=nil, export_vars=S#elixir_erl.export_vars}),
-
-  Extra = SA#elixir_erl.extra_guards,
-  TGuards = guards(Guards, Extra, SA),
+  {TArgs, SA} = match(Fun, Args, S),
+  SG = SA#elixir_erl{extra_guards=[]},
+  TGuards = guards(Guards, SA#elixir_erl.extra_guards, SG),
+  {TExpr, SE} = elixir_erl_pass:translate(Expr, SG#elixir_erl{export_vars=S#elixir_erl.export_vars}),
   {{clause, ?ann(Meta), TArgs, TGuards, unblock(TExpr)}, SE}.
 
 % Translate/Extract guards from the given expression.
 
 guards(Guards, Extra, S) ->
-  SG = S#elixir_erl{context=guard, extra_guards=nil},
-
+  SG = S#elixir_erl{context=guard},
   case Guards of
     [] -> case Extra of [] -> []; _ -> [Extra] end;
     _  -> [translate_guard(Guard, Extra, SG) || Guard <- Guards]

lib/elixir/src/elixir_erl_compiler.erl

@@ -58,7 +58,7 @@ handle_file_warning(_, _File, {_Line, erl_lint, {shadowed_var, _Var, _Where}}) -
 handle_file_warning(_, _File, {_Line, erl_lint, {exported_var, _Var, _Where}}) -> ok;
 
 handle_file_warning(_, File, {Line, erl_lint, {undefined_behaviour, Module}}) ->
-  case elixir_compiler:get_opt(internal) of
+  case elixir_config:get(bootstrap) of
     true ->
       ok;
     false ->

lib/elixir/src/elixir_erl_for.erl

@@ -50,11 +50,11 @@ translate_gen(_Meta, Left, Right, T, S) ->
   {TRight, SR} = elixir_erl_pass:translate(Right, S),
   {LeftArgs, LeftGuards} = elixir_utils:extract_guards(Left),
   {TLeft, SL} = elixir_erl_clauses:match(fun elixir_erl_pass:translate/2, LeftArgs,
-                                     SR#elixir_erl{extra=pin_guard, extra_guards=[]}),
+                                         SR#elixir_erl{extra=pin_guard}),
 
   TLeftGuards = elixir_erl_clauses:guards(LeftGuards, [], SL),
   ExtraGuards = [{nil, X} || X <- SL#elixir_erl.extra_guards],
-  SF = SL#elixir_erl{extra=S#elixir_erl.extra, extra_guards=nil},
+  SF = SL#elixir_erl{extra=S#elixir_erl.extra, extra_guards=[]},
 
   {TT, {TFilters, TS}} = translate_filters(T, SF),
 

lib/elixir/src/elixir_erl_pass.erl

@@ -41,15 +41,16 @@ translate({'__block__', Meta, Args}, S) when is_list(Args) ->
 translate({'__CALLER__', Meta, Atom}, S) when is_atom(Atom) ->
   {{var, ?ann(Meta), '__CALLER__'}, S#elixir_erl{caller=true}};
 
-translate({'super', Meta, Args}, #elixir_erl{def={Kind, Name, _}} = S) ->
+translate({'super', Meta, [{Kind, Name} | Args]}, S) ->
   %% In the expanded AST, super is used to invoke a function
-  %% with the same name but possibly different arity.
+  %% in the current module originated from a default clause
+  %% or a super call.
   {TArgs, SA} = translate_args(Args, S),
   Ann = ?ann(Meta),
   if
     Kind == defmacro; Kind == defmacrop ->
       MacroName = elixir_utils:macro_name(Name),
-      {{call, Ann, {atom, Ann, MacroName}, [{var, Ann, '_@CALLER'} | TArgs]}, SA};
+      {{call, Ann, {atom, Ann, MacroName}, [{var, Ann, '__CALLER__'} | TArgs]}, SA#elixir_erl{caller=true}};
     Kind == def; Kind == defp ->
       {{call, Ann, {atom, Ann, Name}, TArgs}, SA}
   end;
@@ -70,8 +71,8 @@ translate({'&', Meta, [{'/', _, [{Fun, _, Atom}, Arity]}]}, S)
 translate({fn, Meta, Clauses}, S) ->
   Transformer = fun({'->', CMeta, [ArgsWithGuards, Expr]}, Acc) ->
     {Args, Guards} = elixir_utils:extract_splat_guards(ArgsWithGuards),
-    {TClause, TS } = elixir_erl_clauses:clause(CMeta, fun translate_fn_match/2,
-                                            Args, Expr, Guards, Acc),
+    {TClause, TS} = elixir_erl_clauses:clause(CMeta, fun translate_fn_match/2,
+                                              Args, Expr, Guards, Acc),
     {TClause, elixir_erl_var:mergec(S, TS)}
   end,
   {TClauses, NS} = lists:mapfoldl(Transformer, S, Clauses),
@@ -101,29 +102,28 @@ translate({'case', Meta, [Expr, Opts]}, S) ->
 %% Try
 
 translate({'try', Meta, [Opts]}, S) ->
-  SN = S#elixir_erl{extra=nil},
   Do = proplists:get_value('do', Opts, nil),
-  {TDo, SB} = translate(Do, SN),
+  {TDo, SB} = translate(Do, S),
 
   Catch = [Tuple || {X, _} = Tuple <- Opts, X == 'rescue' orelse X == 'catch'],
-  {TCatch, SC} = elixir_erl_try:clauses(Meta, Catch, mergec(SN, SB)),
+  {TCatch, SC} = elixir_erl_try:clauses(Meta, Catch, mergec(S, SB)),
 
   {TAfter, SA} = case lists:keyfind('after', 1, Opts) of
     {'after', After} ->
-      {TBlock, SAExtracted} = translate(After, mergec(SN, SC)),
+      {TBlock, SAExtracted} = translate(After, mergec(S, SC)),
       {unblock(TBlock), SAExtracted};
     false ->
-      {[], mergec(SN, SC)}
+      {[], mergec(S, SC)}
   end,
 
   Else = elixir_erl_clauses:get_clauses(else, Opts, match),
-  {TElse, SE} = elixir_erl_clauses:clauses(Meta, Else, mergec(SN, SA)),
+  {TElse, SE} = elixir_erl_clauses:clauses(Meta, Else, mergec(S, SA)),
   {{'try', ?ann(Meta), unblock(TDo), TElse, TCatch, TAfter}, mergec(S, SE)};
 
 %% Receive
 
 translate({'receive', Meta, [Opts]}, S) ->
-  Do = elixir_erl_clauses:get_clauses(do, Opts, match, true),
+  Do = elixir_erl_clauses:get_clauses(do, Opts, match),
 
   case lists:keyfind('after', 1, Opts) of
     false ->
@@ -137,17 +137,22 @@ translate({'receive', Meta, [Opts]}, S) ->
       {{'receive', ?ann(Meta), FClauses, FExpr, FAfter}, SC}
   end;
 
-%% Comprehensions
+%% Comprehensions and with
 
 translate({for, Meta, [_ | _] = Args}, S) ->
   elixir_erl_for:translate(Meta, Args, true, S);
 
+translate({with, Meta, [_ | _] = Args}, S) ->
+  {Exprs, [{do, Do} | Opts]} = elixir_utils:split_last(Args),
+  {ElseClause, SE} = translate_with_else(Meta, Opts, S),
+  {With, SD} = translate_with_do(Exprs, Do, ElseClause, elixir_erl_var:mergec(S, SE)),
+  {With, elixir_erl_var:mergec(S, SD)};
+
 %% Variables
 
 translate({'^', Meta, [{Name, VarMeta, Kind}]}, #elixir_erl{context=match, file=File} = S) when is_atom(Name), is_atom(Kind) ->
   Tuple = {Name, var_kind(VarMeta, Kind)},
   {ok, {Value, _Counter, Safe}} = maps:find(Tuple, S#elixir_erl.backup_vars),
-  elixir_erl_var:warn_underscored_var_access(VarMeta, File, Name),
   elixir_erl_var:warn_unsafe_var(VarMeta, File, Name, Safe),
 
   PAnn = ?ann(Meta),
@@ -235,10 +240,10 @@ translate(Other, S) ->
 translate_case(true, Meta, Expr, Opts, S) ->
   Clauses = elixir_erl_clauses:get_clauses(do, Opts, match),
   {TExpr, SE} = translate(Expr, S),
-  {TClauses, SC} = elixir_erl_clauses:clauses(Meta, Clauses, SE#elixir_erl{extra=nil}),
-  {{'case', ?ann(Meta), TExpr, TClauses}, SC#elixir_erl{extra=SE#elixir_erl.extra}};
+  {TClauses, SC} = elixir_erl_clauses:clauses(Meta, Clauses, SE),
+  {{'case', ?ann(Meta), TExpr, TClauses}, SC};
 translate_case(false, Meta, Expr, Opts, S) ->
-  {Case, SC} = translate_case(true, Meta, Expr, Opts, S#elixir_erl{extra=nil}),
+  {Case, SC} = translate_case(true, Meta, Expr, Opts, S),
   {Case, elixir_erl_var:mergec(S, SC)}.
 
 translate_list([{'|', _, [_, _]=Args}], Fun, Acc, List) ->
@@ -350,6 +355,63 @@ returns_boolean(Condition, Body) ->
     false -> false
   end.
 
+%% with
+
+translate_with_else(Meta, [], S) ->
+  Ann = ?ann(?generated(Meta)),
+  {VarName, _, SC} = elixir_erl_var:build('_', S),
+  Var = {var, Ann, VarName},
+  {{clause, Ann, [Var], [], [Var]}, SC};
+translate_with_else(Meta, [{else, Else}], S) ->
+  Generated = ?generated(Meta),
+  ElseVarEx = {else, Generated, ?var_context},
+  {ElseVarErl, SV} = elixir_erl_var:assign(Generated, else, ?var_context, S),
+
+  RaiseVar = {catch_all, Generated, ?var_context},
+  RaiseExpr = {{'.', Generated, [erlang, error]}, Generated, [{with_clause, RaiseVar}]},
+  RaiseClause = {'->', Generated, [[RaiseVar], RaiseExpr]},
+
+  GeneratedElse = [build_generated_clause(Generated, ElseClause) || ElseClause <- Else],
+
+  Case = {'case', [{export_vars, false} | Generated], [ElseVarEx, [{do, GeneratedElse ++ [RaiseClause]}]]},
+  {TranslatedCase, SC} = elixir_erl_pass:translate(Case, SV),
+  {{clause, ?ann(Generated), [ElseVarErl], [], [TranslatedCase]}, SC}.
+
+build_generated_clause(Generated, {'->', _, [Args, Clause]}) ->
+  NewArgs = [build_generated_clause_arg(Generated, Arg) || Arg <- Args],
+  {'->', Generated, [NewArgs, Clause]}.
+
+build_generated_clause_arg(Generated, Arg) ->
+  {Expr, Guards} = elixir_utils:extract_guards(Arg),
+  NewGuards = [build_generated_guard(Generated, Guard) || Guard <- Guards],
+  concat_guards(Generated, Expr, NewGuards).
+
+build_generated_guard(Generated, {{'.', _, _} = Call, _, Args}) ->
+  {Call, Generated, [build_generated_guard(Generated, Arg) || Arg <- Args]};
+build_generated_guard(_, Expr) ->
+  Expr.
+
+concat_guards(_Meta, Expr, []) ->
+  Expr;
+concat_guards(Meta, Expr, [Guard | Tail]) ->
+  {'when', Meta, [Expr, concat_guards(Meta, Guard, Tail)]}.
+
+translate_with_do([{'<-', Meta, [Left, Expr]} | Rest], Do, Else, S) ->
+  {Args, Guards} = elixir_utils:extract_guards(Left),
+  {TExpr, SR} = elixir_erl_pass:translate(Expr, S),
+  {TArgs, SA} = elixir_erl_clauses:match(fun elixir_erl_pass:translate/2, Args, SR),
+  TGuards = elixir_erl_clauses:guards(Guards, [], SA),
+  {TBody, SB} = translate_with_do(Rest, Do, Else, SA),
+
+  Clause = {clause, ?ann(Meta), [TArgs], TGuards, unblock(TBody)},
+  {{'case', ?ann(?generated(Meta)), TExpr, [Clause, Else]}, SB};
+translate_with_do([Expr | Rest], Do, Else, S) ->
+  {TExpr, TS} = elixir_erl_pass:translate(Expr, S),
+  {TRest, RS} = translate_with_do(Rest, Do, Else, TS),
+  {{block, 0, [TExpr | unblock(TRest)]}, RS};
+translate_with_do([], Do, _Else, S) ->
+  elixir_erl_pass:translate(Do, S).
+
 %% Maps and structs
 
 translate_map(Meta, [{'|', _Meta, [Update, Assocs]}], S) ->
@@ -513,7 +575,7 @@ translate_remote(Left, Right, Meta, Args, S) ->
       {{call, Ann, {remote, Ann, TLeft, TRight}, TArgs}, SC}
   end.
 
-is_always_string({{'.', _, [Module, Function]}, Args}) ->
+is_always_string({{'.', _, [Module, Function]}, _, Args}) ->
   is_always_string(Module, Function, length(Args));
 %% Binary literals were already excluded in earlier passes.
 is_always_string(_Ast) ->

lib/elixir/src/elixir_erl_try.erl

@@ -23,19 +23,21 @@ each_clause({'catch', Meta, Raw, Expr}, S) ->
 
   Condition = [{'{}', Meta, Final}],
   {TC, TS} = elixir_erl_clauses:clause(Meta, fun elixir_erl_pass:translate_args/2,
-                                   Condition, Expr, Guards, S),
+                                       Condition, Expr, Guards, S),
   {[TC], TS};
 
 each_clause({rescue, Meta, [{in, _, [Left, Right]}], Expr}, S) ->
-  {VarName, _, CS} = elixir_erl_var:build('_', S),
-  Var = {VarName, Meta, nil},
-  {Parts, Safe, FS} = rescue_guards(Meta, Var, Right, CS),
-  Body = rescue_clause_body(Left, Expr, Safe, Var, Meta),
+  {TempName, _, CS} = elixir_erl_var:build('_', S),
+  TempVar = {TempName, Meta, ?var_context},
+  {Parts, Safe, FS} = rescue_guards(Meta, TempVar, Right, CS),
+  Body = rescue_clause_body(Left, Expr, Safe, TempVar, Meta),
   build_rescue(Meta, Parts, Body, FS);
 
-each_clause({rescue, Meta, [{VarName, _, Atom} = Var], Expr}, S) when is_atom(VarName), is_atom(Atom) ->
-  Body = rescue_clause_body(Var, Expr, false, Var, Meta),
-  build_rescue(Meta, _Parts = [{Var, []}], Body, S).
+each_clause({rescue, Meta, [{VarName, _, Context} = Left], Expr}, S) when is_atom(VarName), is_atom(Context) ->
+  {TempName, _, CS} = elixir_erl_var:build('_', S),
+  TempVar = {TempName, Meta, ?var_context},
+  Body = rescue_clause_body(Left, Expr, false, TempVar, Meta),
+  build_rescue(Meta, [{TempVar, []}], Body, CS).
 
 rescue_clause_body({'_', _, Atom}, Expr, _Safe, _Var, _Meta) when is_atom(Atom) ->
   Expr;
@@ -54,7 +56,7 @@ build_rescue(Meta, Parts, Body, S) ->
 
   {{clause, Line, TMatches, _, TBody}, TS} =
     elixir_erl_clauses:clause(Meta, fun elixir_erl_pass:translate_args/2,
-                          Matches, Body, [], S),
+                              Matches, Body, [], S),
 
   TClauses =
     [begin
@@ -74,7 +76,7 @@ rescue_guards(Meta, Var, Aliases, S) ->
       [] -> {[], S};
       _  ->
         {VarName, _, CS} = elixir_erl_var:build('_', S),
-        StructVar = {VarName, Meta, nil},
+        StructVar = {VarName, Meta, 'Elixir'},
         Map = {'%{}', Meta, [{'__struct__', StructVar}, {'__exception__', true}]},
         Match = {'=', Meta, [Map, Var]},
         Guards = [{erl(Meta, '=='), Meta, [StructVar, Mod]} || Mod <- Elixir],

lib/elixir/src/elixir_erl_var.erl

@@ -1,64 +1,68 @@
 %% Convenience functions used to manipulate scope and its variables.
 -module(elixir_erl_var).
--export([translate/4, build/2, context_info/1,
+-export([translate/4, build/2, assign/4,
   load_binding/2, dump_binding/2,
   mergev/2, mergec/2, merge_vars/2, merge_opt_vars/2,
-  warn_unsafe_var/4, warn_underscored_var_access/3, format_error/1
+  warn_unsafe_var/4, format_error/1
 ]).
 -include("elixir.hrl").
 
 %% VAR HANDLING
 
 translate(Meta, Name, Kind, S) when is_atom(Kind); is_integer(Kind) ->
-  Ann = ?ann(Meta),
   Tuple = {Name, Kind},
-  Vars = S#elixir_erl.vars,
 
-  {Current, Exists, Safe} =
-    case maps:find({Name, Kind}, Vars) of
-      {ok, {VarC, _, VarS}} -> {VarC, true, VarS};
-      error -> {nil, false, true}
+  {Current, Safe} =
+    case maps:find(Tuple, S#elixir_erl.vars) of
+      {ok, {VarC, _, VarS}} -> {VarC, VarS};
+      error -> {nil, true}
     end,
 
   case S#elixir_erl.context of
     match ->
-      MatchVars = S#elixir_erl.match_vars,
+      Previous =
+        case maps:find(Tuple, S#elixir_erl.backup_vars) of
+          {ok, {BackupVarC, _, _}} -> BackupVarC;
+          error -> nil
+        end,
 
-      case Exists andalso maps:get(Tuple, MatchVars, false) of
+      if
+        Current /= nil, Current /= Previous ->
+          {{var, ?ann(Meta), Current}, S};
         true ->
-          warn_underscored_var_repeat(Meta, S#elixir_erl.file, Name, Kind),
-          {{var, Ann, Current}, S};
-        false ->
-          %% We attempt to give vars a nice name because we
-          %% still use the unused vars warnings from erl_lint.
-          %%
-          %% Once we move the warning to Elixir compiler, we
-          %% can name vars as _@COUNTER.
-          {NewVar, Counter, NS} =
-            if
-              Kind /= nil ->
-                build('_', S);
-              true ->
-                build(Name, S)
-            end,
-
-          FS = NS#elixir_erl{
-            vars=maps:put(Tuple, {NewVar, Counter, true}, Vars),
-            match_vars=maps:put(Tuple, true, MatchVars),
-            export_vars=case S#elixir_erl.export_vars of
-              nil -> nil;
-              EV  -> maps:put(Tuple, {NewVar, Counter, true}, EV)
-            end
-          },
-
-          {{var, Ann, NewVar}, FS}
+          assign(Meta, Name, Kind, S)
       end;
-    _  when Exists ->
-      warn_underscored_var_access(Meta, S#elixir_erl.file, Name),
+    _  when Current /= nil ->
       warn_unsafe_var(Meta, S#elixir_erl.file, Name, Safe),
-      {{var, Ann, Current}, S}
+      {{var, ?ann(Meta), Current}, S}
   end.
 
+assign(Meta, Name, Kind, S) ->
+  Tuple = {Name, Kind},
+
+  %% We attempt to give vars a nice name because we
+  %% still use the unused vars warnings from erl_lint.
+  %%
+  %% Once we move the warning to Elixir compiler, we
+  %% can name vars as _@COUNTER.
+  {NewVar, Counter, NS} =
+    if
+      Kind /= nil ->
+        build('_', S);
+      true ->
+        build(Name, S)
+    end,
+
+  FS = NS#elixir_erl{
+    vars=maps:put(Tuple, {NewVar, Counter, true}, S#elixir_erl.vars),
+    export_vars=case S#elixir_erl.export_vars of
+      nil -> nil;
+      EV  -> maps:put(Tuple, {NewVar, Counter, true}, EV)
+    end
+  },
+
+  {{var, ?ann(Meta), NewVar}, FS}.
+
 build(Key, #elixir_erl{counter=Counter} = S) ->
   Cnt =
     case maps:find(Key, Counter) of
@@ -69,43 +73,14 @@ build(Key, #elixir_erl{counter=Counter} = S) ->
    Cnt,
    S#elixir_erl{counter=maps:put(Key, Cnt, Counter)}}.
 
-context_info(Kind) when Kind == nil; is_integer(Kind) -> "";
-context_info(Kind) -> io_lib:format(" (context ~ts)", [elixir_aliases:inspect(Kind)]).
-
-warn_underscored_var_repeat(Meta, File, Name, Kind) ->
-  Warn = should_warn(Meta),
-  case atom_to_list(Name) of
-    "_@" ++ _ ->
-      ok; %% Automatically generated variables
-    "_" ++ _ when Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {unused_match, Name, Kind});
-    _ ->
-      ok
-  end.
-
 warn_unsafe_var(Meta, File, Name, Safe) ->
-  Warn = should_warn(Meta),
-  if
-    (not Safe) and Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {unsafe_var, Name});
+  case (not Safe) andalso (lists:keyfind(generated, 1, Meta) /= {generated, true}) of
     true ->
+      elixir_errors:form_warn(Meta, File, ?MODULE, {unsafe_var, Name});
+    false ->
       ok
   end.
 
-warn_underscored_var_access(Meta, File, Name) ->
-  Warn = should_warn(Meta),
-  case atom_to_list(Name) of
-    "_@" ++ _ ->
-      ok; %% Automatically generated variables
-    "_" ++ _ when Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {underscore_var_access, Name});
-    _ ->
-      ok
-  end.
-
-should_warn(Meta) ->
-  lists:keyfind(generated, 1, Meta) /= {generated, true}.
-
 %% SCOPE MERGING
 
 %% Receives two scopes and return a new scope based on
@@ -193,13 +168,6 @@ dump_binding(Binding, #elixir_erl{vars=Vars}) ->
 
 %% Errors
 
-format_error({unused_match, Name, Kind}) ->
-  io_lib:format("the underscored variable \"~ts\"~ts appears more than once in a "
-                "match. This means the pattern will only match if all \"~ts\" bind "
-                "to the same value. If this is the intended behaviour, please "
-                "remove the leading underscore from the variable name, otherwise "
-                "give the variables different names", [Name, context_info(Kind), Name]);
-
 format_error({unsafe_var, Name}) ->
   io_lib:format("the variable \"~ts\" is unsafe as it has been set inside "
                 "one of: case, cond, receive, if, and, or, &&, ||. "
@@ -214,10 +182,4 @@ format_error({unsafe_var, Name}) ->
                 "        1 -> :one\n"
                 "        2 -> :two\n"
                 "      end\n\n"
-                "Unsafe variable found at:", [Name]);
-
-format_error({underscore_var_access, Name}) ->
-  io_lib:format("the underscored variable \"~ts\" is used after being set. "
-                "A leading underscore indicates that the value of the variable "
-                "should be ignored. If this is intended please rename the "
-                "variable to remove the underscore", [Name]).
+                "Unsafe variable found at:", [Name]).

lib/elixir/src/elixir_expand.erl

@@ -6,7 +6,7 @@
 %% =
 
 expand({'=', Meta, [Left, Right]}, E) ->
-  assert_no_guard_scope(Meta, '=', E),
+  assert_no_guard_scope(Meta, "=", E),
   {ERight, ER} = expand(Right, E),
   {ELeft, EL}  = elixir_clauses:match(fun expand/2, Left, E),
   {{'=', Meta, [ELeft, ERight]}, elixir_env:mergev(EL, ER)};
@@ -62,7 +62,7 @@ expand({Kind, Meta, [{{'.', _, [Base, '{}']}, _, Refs} | Rest]}, E)
 expand({alias, Meta, [Ref]}, E) ->
   expand({alias, Meta, [Ref, []]}, E);
 expand({alias, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, alias, E),
+  assert_no_match_or_guard_scope(Meta, "alias", E),
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, alias, [as, warn], no_alias_opts(Opts), ER),
 
@@ -76,7 +76,7 @@ expand({alias, Meta, [Ref, Opts]}, E) ->
 expand({require, Meta, [Ref]}, E) ->
   expand({require, Meta, [Ref, []]}, E);
 expand({require, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, require, E),
+  assert_no_match_or_guard_scope(Meta, "require", E),
 
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, require, [as, warn], no_alias_opts(Opts), ER),
@@ -93,7 +93,7 @@ expand({import, Meta, [Left]}, E) ->
   expand({import, Meta, [Left, []]}, E);
 
 expand({import, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, import, E),
+  assert_no_match_or_guard_scope(Meta, "import", E),
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, import, [only, except, warn], Opts, ER),
 
@@ -209,7 +209,7 @@ expand({quote, Meta, [_, _]}, E) ->
 %% Functions
 
 expand({'&', Meta, [Arg]}, E) ->
-  assert_no_match_or_guard_scope(Meta, '&', E),
+  assert_no_match_or_guard_scope(Meta, "&", E),
   case elixir_fn:capture(Meta, Arg, E) of
     {remote, Remote, Fun, Arity} ->
       is_atom(Remote) andalso
@@ -222,34 +222,36 @@ expand({'&', Meta, [Arg]}, E) ->
   end;
 
 expand({fn, Meta, Pairs}, E) ->
-  assert_no_match_or_guard_scope(Meta, fn, E),
+  assert_no_match_or_guard_scope(Meta, "fn", E),
   elixir_fn:expand(Meta, Pairs, E);
 
 %% Case/Receive/Try
 
 expand({'cond', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'cond', E),
+  assert_no_match_or_guard_scope(Meta, "cond", E),
   assert_no_underscore_clause_in_cond(Opts, E),
   {EClauses, EC} = elixir_clauses:'cond'(Meta, Opts, E),
   {{'cond', Meta, [EClauses]}, EC};
 
-expand({'case', Meta, [Expr, Options]}, Env) ->
+expand({'case', Meta, [Expr, Options]}, E) ->
+  assert_no_match_or_guard_scope(Meta, "case", E),
   ShouldExportVars = proplists:get_value(export_vars, Meta, true),
-  expand_case(ShouldExportVars, Meta, Expr, Options, Env);
+  expand_case(ShouldExportVars, Meta, Expr, Options, E);
 
 expand({'receive', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'receive', E),
+  assert_no_match_or_guard_scope(Meta, "receive", E),
   {EClauses, EC} = elixir_clauses:'receive'(Meta, Opts, E),
   {{'receive', Meta, [EClauses]}, EC};
 
 expand({'try', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'try', E),
+  assert_no_match_or_guard_scope(Meta, "try", E),
   {EClauses, EC} = elixir_clauses:'try'(Meta, Opts, E),
   {{'try', Meta, [EClauses]}, EC};
 
 %% Comprehensions
 
 expand({for, Meta, [_ | _] = Args}, E) ->
+  assert_no_match_or_guard_scope(Meta, "for", E),
   {Cases, Block} =
     case elixir_utils:split_last(Args) of
       {OuterCases, OuterOpts} when is_list(OuterOpts) ->
@@ -281,25 +283,22 @@ expand({for, Meta, [_ | _] = Args}, E) ->
 %% With
 
 expand({with, Meta, [_ | _] = Args}, E) ->
-  elixir_with:expand(Meta, Args, E);
+  assert_no_match_or_guard_scope(Meta, "with", E),
+  elixir_clauses:with(Meta, Args, E);
 
 %% Super
 
 expand({super, Meta, Args}, #{file := File} = E) when is_list(Args) ->
+  assert_no_match_or_guard_scope(Meta, "super", E),
   Module = assert_module_scope(Meta, super, E),
   Function = assert_function_scope(Meta, super, E),
   {_, Arity} = Function,
 
   case length(Args) of
     Arity ->
-      {OName, OArity} = elixir_overridable:super(Meta, File, Module, Function),
+      {Kind, Name} = elixir_overridable:super(Meta, File, Module, Function),
       {EArgs, EA} = expand_args(Args, E),
-      OArgs =
-        if
-          OArity > Arity -> [{'__CALLER__', [], nil} | EArgs];
-          true -> EArgs
-        end,
-      {{OName, Meta, OArgs}, EA};
+      {{super, Meta, [{Kind, Name} | EArgs]}, EA};
     _ ->
       form_error(Meta, File, ?MODULE, wrong_number_of_args_for_super)
   end;
@@ -315,8 +314,9 @@ expand({'^', Meta, [Arg]}, #{context := match, prematch_vars := PrematchVars} =
       %% If the variable was defined, then we return the expanded ^, otherwise
       %% we raise. We cannot use the expanded env because it would contain the
       %% variable.
-      case lists:member({VarName, var_kind(VarMeta, Kind)}, PrematchVars) of
+      case lists:member({VarName, var_context(VarMeta, Kind)}, PrematchVars) of
         true ->
+          warn_underscored_var_access(VarMeta, VarName, Kind, E),
           {{'^', Meta, [Var]}, EA};
         false ->
           form_error(Meta, ?key(EA, file), ?MODULE, {unbound_variable_pin, VarName})
@@ -332,27 +332,43 @@ expand({'_', _Meta, Kind} = Var, #{context := match} = E) when is_atom(Kind) ->
 expand({'_', Meta, Kind}, E) when is_atom(Kind) ->
   form_error(Meta, ?key(E, file), ?MODULE, unbound_underscore);
 
-expand({Name, Meta, Kind} = Var, #{context := match, export_vars := Export} = E) when is_atom(Name), is_atom(Kind) ->
-  Pair      = {Name, var_kind(Meta, Kind)},
-  NewVars   = ordsets:add_element(Pair, ?key(E, vars)),
-  NewExport = case (Export /= nil) of
-    true  -> ordsets:add_element(Pair, Export);
-    false -> Export
-  end,
-  {Var, E#{vars := NewVars, export_vars := NewExport}};
+expand({Name, Meta, Kind} = Var, #{context := match} = E) when is_atom(Name), is_atom(Kind) ->
+  #{vars := Vars, match_vars := Match, export_vars := Export} = E,
+  Pair = {Name, var_context(Meta, Kind)},
+  NewVars = ordsets:add_element(Pair, Vars),
+
+  NewExport =
+    case (Export /= nil) of
+      true  -> ordsets:add_element(Pair, Export);
+      false -> Export
+    end,
+
+  NewMatch =
+    case lists:member(Pair, Match) of
+      true -> warn_underscored_var_repeat(Meta, Name, Kind, E), Match;
+      false -> [Pair | Match]
+    end,
+
+  {Var, E#{vars := NewVars, match_vars := NewMatch, export_vars := NewExport}};
 expand({Name, Meta, Kind} = Var, #{vars := Vars} = E) when is_atom(Name), is_atom(Kind) ->
-  case lists:member({Name, var_kind(Meta, Kind)}, Vars) of
+  case lists:member({Name, var_context(Meta, Kind)}, Vars) of
     true ->
+      warn_underscored_var_access(Meta, Name, Kind, E),
       {Var, E};
     false ->
       case lists:keyfind(var, 1, Meta) of
         {var, true} ->
           form_error(Meta, ?key(E, file), ?MODULE, {undefined_var, Name, Kind});
         _ ->
-          Message =
-            io_lib:format("variable \"~ts\" does not exist and is being expanded to \"~ts()\","
-              " please use parentheses to remove the ambiguity or change the variable name", [Name, Name]),
-          elixir_errors:warn(?line(Meta), ?key(E, file), Message),
+          case ?key(E, match_vars) of
+            warn ->
+              Message =
+                io_lib:format("variable \"~ts\" does not exist and is being expanded to \"~ts()\","
+                  " please use parentheses to remove the ambiguity or change the variable name", [Name, Name]),
+              elixir_errors:warn(?line(Meta), ?key(E, file), Message);
+            apply ->
+              ok
+          end,
           expand({Name, Meta, []}, E)
       end
   end;
@@ -391,6 +407,7 @@ expand({{'.', DotMeta, [Left, Right]}, Meta, Args}, E)
 %% Anonymous calls
 
 expand({{'.', DotMeta, [Expr]}, Meta, Args}, E) when is_list(Args) ->
+  assert_no_match_or_guard_scope(Meta, "anonymous call", E),
   {EExpr, EE} = expand(Expr, E),
   if
     is_atom(EExpr) ->
@@ -431,14 +448,15 @@ expand(Function, E) when is_function(Function) ->
   end;
 
 
-expand(PidOrRef, E) when is_pid(PidOrRef); is_reference(PidOrRef) ->
+expand(Pid, E) when is_pid(Pid) ->
   case ?key(E, function) of
     nil ->
-      PidOrRef;
+      {Pid, E};
     Function ->
       %% TODO: Make me an error on 2.0
       elixir_errors:form_warn([], ?key(E, file), ?MODULE,
-                              {invalid_pid_or_ref_in_function, PidOrRef, Function})
+                              {invalid_pid_in_function, Pid, Function}),
+      {Pid, E}
   end;
 
 expand(Other, E) when is_number(Other); is_atom(Other); is_binary(Other) ->
@@ -554,7 +572,7 @@ expand_args(Args, E) ->
 
 %% Match/var helpers
 
-var_kind(Meta, Kind) ->
+var_context(Meta, Kind) ->
   case lists:keyfind(counter, 1, Meta) of
     {counter, Counter} -> Counter;
     false -> Kind
@@ -563,7 +581,6 @@ var_kind(Meta, Kind) ->
 %% Case
 
 expand_case(true, Meta, Expr, Opts, E) ->
-  assert_no_match_or_guard_scope(Meta, 'case', E),
   {EExpr, EE} = expand(Expr, E),
   {EOpts, EO} = elixir_clauses:'case'(Meta, Opts, EE),
   ROpts =
@@ -789,11 +806,11 @@ assert_function_scope(_Meta, _Kind, #{function := Function}) -> Function.
 assert_no_match_or_guard_scope(Meta, Kind, E) ->
   assert_no_match_scope(Meta, Kind, E),
   assert_no_guard_scope(Meta, Kind, E).
-assert_no_match_scope(Meta, _Kind, #{context := match, file := File}) ->
-  form_error(Meta, File, ?MODULE, invalid_pattern_in_match);
+assert_no_match_scope(Meta, Kind, #{context := match, file := File}) ->
+  form_error(Meta, File, ?MODULE, {invalid_pattern_in_match, Kind});
 assert_no_match_scope(_Meta, _Kind, _E) -> [].
-assert_no_guard_scope(Meta, _Kind, #{context := guard, file := File}) ->
-  form_error(Meta, File, ?MODULE, invalid_expr_in_guard);
+assert_no_guard_scope(Meta, Kind, #{context := guard, file := File}) ->
+  form_error(Meta, File, ?MODULE, {invalid_expr_in_guard, Kind});
 assert_no_guard_scope(_Meta, _Kind, _E) -> [].
 
 %% Here we look into the Clauses "optimistically", that is, we don't check for
@@ -814,6 +831,30 @@ assert_no_underscore_clause_in_cond(_Other, _E) ->
 
 %% Warnings
 
+warn_underscored_var_repeat(Meta, Name, Kind, E) ->
+  Warn = should_warn(Meta),
+  case atom_to_list(Name) of
+    "_" ++ _ when Warn ->
+      elixir_errors:form_warn(Meta, ?key(E, file), ?MODULE, {underscored_var_repeat, Name, Kind});
+    _ ->
+      ok
+  end.
+
+warn_underscored_var_access(Meta, Name, Kind, E) ->
+  Warn = should_warn(Meta),
+  case atom_to_list(Name) of
+    "_" ++ _ when Warn ->
+      elixir_errors:form_warn(Meta, ?key(E, file), ?MODULE, {underscored_var_access, Name, Kind});
+    _ ->
+      ok
+  end.
+
+context_info(Kind) when Kind == nil; is_integer(Kind) -> "";
+context_info(Kind) -> io_lib:format(" (context ~ts)", [elixir_aliases:inspect(Kind)]).
+
+should_warn(Meta) ->
+  lists:keyfind(generated, 1, Meta) /= {generated, true}.
+
 format_error({useless_literal, Term}) ->
   io_lib:format("code block contains unused literal ~ts "
                 "(remove the literal or assign it to _ to avoid warnings)",
@@ -866,14 +907,14 @@ format_error({undefined_var, Name, Kind}) ->
   Message =
     "expected variable \"~ts\"~ts to expand to an existing variable "
     "or be part of a match",
-  io_lib:format(Message, [Name, elixir_erl_var:context_info(Kind)]);
+  io_lib:format(Message, [Name, context_info(Kind)]);
 format_error(underscore_in_cond) ->
   "unbound variable _ inside \"cond\". If you want the last clause to always match, "
     "you probably meant to use: true ->";
-format_error(invalid_expr_in_guard) ->
-  "invalid expression in guard";
-format_error(invalid_pattern_in_match) ->
-  "invalid pattern in match";
+format_error({invalid_expr_in_guard, Kind}) ->
+  io_lib:format("invalid expression in guard, ~ts is not allowed in guards", [Kind]);
+format_error({invalid_pattern_in_match, Kind}) ->
+  io_lib:format("invalid pattern in match, ~ts is not allowed in matches", [Kind]);
 format_error({invalid_expr_in_scope, Scope, Kind}) ->
   io_lib:format("cannot invoke ~ts outside ~ts", [Kind, Scope]);
 format_error({invalid_alias, Expr}) ->
@@ -908,9 +949,9 @@ format_error({invalid_local_invocation, Context, {Name, _, Args} = Call}) ->
 format_error({invalid_remote_invocation, Context, Receiver, Right, Arity}) ->
   io_lib:format("cannot invoke remote function ~ts.~ts/~B inside ~ts",
                 ['Elixir.Macro':to_string(Receiver), Right, Arity, Context]);
-format_error({invalid_pid_or_ref_in_function, PidOrRef, {Name, Arity}}) ->
-  io_lib:format("cannot compile PID/Reference ~ts inside quoted expression for function ~ts/~B",
-                ['Elixir.Kernel':inspect(PidOrRef, []), Name, Arity]);
+format_error({invalid_pid_in_function, Pid, {Name, Arity}}) ->
+  io_lib:format("cannot compile PID ~ts inside quoted expression for function ~ts/~B",
+                ['Elixir.Kernel':inspect(Pid, []), Name, Arity]);
 format_error({unsupported_option, Kind, Key}) ->
   io_lib:format("unsupported option ~ts given to ~s",
                 ['Elixir.Macro':to_string(Key), Kind]);
@@ -918,4 +959,15 @@ format_error({options_are_not_keyword, Kind, Opts}) ->
   io_lib:format("invalid options for ~s, expected a keyword list, got: ~ts",
                 [Kind, 'Elixir.Macro':to_string(Opts)]);
 format_error({undefined_function, Name, Args}) ->
-  io_lib:format("undefined function ~ts/~B", [Name, length(Args)]).
+  io_lib:format("undefined function ~ts/~B", [Name, length(Args)]);
+format_error({underscored_var_repeat, Name, Kind}) ->
+  io_lib:format("the underscored variable \"~ts\"~ts appears more than once in a "
+                "match. This means the pattern will only match if all \"~ts\" bind "
+                "to the same value. If this is the intended behaviour, please "
+                "remove the leading underscore from the variable name, otherwise "
+                "give the variables different names", [Name, context_info(Kind), Name]);
+format_error({underscored_var_access, Name, Kind}) ->
+  io_lib:format("the underscored variable \"~ts\"~ts is used after being set. "
+                "A leading underscore indicates that the value of the variable "
+                "should be ignored. If this is intended please rename the "
+                "variable to remove the underscore", [Name, context_info(Kind)]).

lib/elixir/src/elixir_lexical.erl

@@ -11,7 +11,7 @@
 -define(tracker, 'Elixir.Kernel.LexicalTracker').
 
 run(File, Dest, Callback) ->
-  case elixir_compiler:get_opt(internal) of
+  case elixir_config:get(bootstrap) of
     false ->
       {ok, Pid} = ?tracker:start_link(Dest),
       try Callback(Pid) of

lib/elixir/src/elixir_locals.erl

@@ -3,16 +3,17 @@
 -export([
   setup/1, cleanup/1, cache_env/1, get_cached_env/1,
   record_local/2, record_local/3, record_import/4,
-  record_definition/3, record_defaults/4,
+  record_definition/3, record_defaults/4, reattach/5,
   ensure_no_import_conflict/3, warn_unused_local/3, format_error/1
 ]).
 
 -include("elixir.hrl").
 -define(attr, {elixir, locals_tracker}).
+-define(cache, {elixir, cache_env}).
 -define(tracker, 'Elixir.Module.LocalsTracker').
 
 setup(Module) ->
-  case elixir_compiler:get_opt(internal) of
+  case elixir_config:get(bootstrap) of
     false ->
       {ok, Pid} = ?tracker:start_link(),
       ets:insert(elixir_module:data_table(Module), {?attr, Pid}),
@@ -24,6 +25,9 @@ setup(Module) ->
 cleanup(Module) ->
   if_tracker(Module, fun(Pid) -> unlink(Pid), ?tracker:stop(Pid), ok end).
 
+reattach(Tuple, Kind, Module, Function, Neighbours) ->
+  if_tracker(Module, fun(Pid) -> ?tracker:reattach(Pid, Tuple, Kind, Function, Neighbours) end).
+
 record_local(Tuple, Module) when is_atom(Module) ->
   if_tracker(Module, fun(Pid) -> ?tracker:add_local(Pid, Tuple), ok end).
 record_local(Tuple, _Module, Function)
@@ -56,18 +60,27 @@ if_tracker(Module, Default, Callback) ->
 
 %% CACHING
 
-cache_env(#{module := Module, line := Line} = E) ->
-  try ets:lookup_element(elixir_module:data_table(Module), ?attr, 2) of
-    Pid ->
-      {Pid, {Line, ?tracker:cache_env(Pid, E#{line := nil, vars := []})}}
-  catch
-    error:badarg ->
-      {Escaped, _} = elixir_quote:escape(E#{vars := []}, false),
-      Escaped
-  end.
+cache_env(#{line := Line, module := Module} = E) ->
+  Table = elixir_module:data_table(Module),
+  Cache = E#{line := nil, vars := []},
 
-get_cached_env({Pid, {Line, Ref}}) -> (?tracker:get_cached_env(Pid, Ref))#{line := Line};
-get_cached_env(Env) -> Env.
+  Pos =
+    case ets:lookup(Table, ?cache) of
+      [{_, Key, Cache}] ->
+        Key;
+      [{_, PrevKey, _}] ->
+        Key = PrevKey + 1,
+        ets:insert(Table, {{cache_env, Key}, Cache}),
+        ets:insert(Table, {?cache, Key, Cache}),
+        Key
+    end,
+
+  {Module, {Line, Pos}}.
+
+get_cached_env({Module, {Line, Pos}}) ->
+  (ets:lookup_element(elixir_module:data_table(Module), {cache_env, Pos}, 2))#{line := Line};
+get_cached_env(Env) ->
+  Env.
 
 %% ERROR HANDLING
 

lib/elixir/src/elixir_map.erl

@@ -121,10 +121,7 @@ validate_struct(_, _) -> false.
 
 load_struct(Meta, Name, Args, InContext, E) ->
   Arity = length(Args),
-
-  Local =
-    not(ensure_loaded(Name)) andalso
-      (InContext orelse wait_for_struct(Name)),
+  Local = InContext orelse (not(ensure_loaded(Name)) andalso wait_for_struct(Name)),
 
   try
     case Local andalso elixir_def:local_for(Name, '__struct__', Arity, [def]) of

lib/elixir/src/elixir_module.erl

@@ -1,7 +1,8 @@
 -module(elixir_module).
 -export([data_table/1, defs_table/1, is_open/1, delete_doc/6,
          compile/4, expand_callback/6, format_error/1,
-         compiler_modules/0, delete_impl/6]).
+         compiler_modules/0, delete_impl/6,
+         write_cache/3, read_cache/2]).
 -include("elixir.hrl").
 
 -define(lexical_attr, {elixir, lexical_tracker}).
@@ -39,6 +40,12 @@ delete_impl(#{module := Module}, _, _, _, _, _) ->
   ets:delete(data_table(Module), impl),
   ok.
 
+write_cache(Module, Key, Value) ->
+  ets:insert(data_table(Module), {{cache, Key}, Value}).
+
+read_cache(Module, Key) ->
+  ets:lookup_element(data_table(Module), {cache, Key}, 2).
+
 %% Compilation hook
 
 compile(Module, Block, Vars, #{line := Line} = Env) when is_atom(Module) ->
@@ -79,7 +86,7 @@ compile(Line, Module, Block, Vars, E) ->
 
     {AllDefinitions, Unreachable} = elixir_def:fetch_definitions(File, Module),
 
-    (not elixir_compiler:get_opt(internal)) andalso
+    (not elixir_config:get(bootstrap)) andalso
      'Elixir.Module':check_behaviours_and_impls(E, Data, AllDefinitions, OverridablePairs),
 
     CompileOpts = lists:flatten(ets:lookup_element(Data, compile, 2)),
@@ -179,7 +186,7 @@ build(Line, File, Module, Lexical) ->
       end,
 
   ImplOnDefinition =
-      case elixir_compiler:get_opt(internal) of
+      case elixir_config:get(bootstrap) of
         true -> [{elixir_module, delete_impl}];
         _    -> [{'Elixir.Module', compile_impl}]
       end,
@@ -210,7 +217,8 @@ build(Line, File, Module, Lexical) ->
     {typep, [], true, nil},
 
     % Internal
-    {{elixir, impls}, []}
+    {{elixir, impls}, []},
+    {{elixir, cache_env}, 0, nil}
   ]),
 
   Persisted = [behaviour, on_load, compile, external_resource, dialyzer, vsn],
@@ -310,7 +318,7 @@ beam_location(#{lexical_tracker := Pid, module := Module}) ->
 %% Handle unused attributes warnings and special cases.
 
 warn_unused_attributes(File, Data, PersistedAttrs) ->
-  ReservedAttrs = [after_compile, before_compile, moduledoc, on_definition | PersistedAttrs],
+  ReservedAttrs = [after_compile, before_compile, deprecated, moduledoc, on_definition, since | PersistedAttrs],
   Keys = ets:select(Data, [{{'$1', '_', '_', '$2'}, [{is_atom, '$1'}, {is_integer, '$2'}], [['$1', '$2']]}]),
   [elixir_errors:form_warn([{line, Line}], File, ?MODULE, {unused_attribute, Key}) ||
    [Key, Line] <- Keys, not lists:member(Key, ReservedAttrs)].

lib/elixir/src/elixir_overridable.erl

@@ -15,15 +15,15 @@ overridable(Module, Value) ->
 
 super(Meta, File, Module, Function) ->
   case store(Module, Function, true) of
-    {ok, Name} ->
-      Name;
+    {_, _} = KindName ->
+      KindName;
     error ->
       elixir_errors:form_error(Meta, File, ?MODULE, {no_super, Module, Function})
   end.
 
 store_pending(Module) ->
   [begin
-    {ok, _} = store(Module, Pair, false),
+    {_, _} = store(Module, Pair, false),
     Pair
    end || {Pair, {_, _, _, false}} <- maps:to_list(overridable(Module)),
           not 'Elixir.Module':'defines?'(Module, Pair)].
@@ -42,7 +42,7 @@ store(Module, Function, Hidden) ->
           false ->
             {Kind, Name, Arity, Clauses};
           true when Kind == defmacro; Kind == defmacrop ->
-            {defp, name(Name, Count), Arity + 1, rewrite_clauses(Clauses)};
+            {defmacrop, name(Name, Count), Arity, Clauses};
           true ->
             {defp, name(Name, Count), Arity, Clauses}
         end,
@@ -52,25 +52,18 @@ store(Module, Function, Hidden) ->
       case Overridden of
         false ->
           overridable(Module, maps:put(Function, {Count, Def, Neighbours, true}, Overridable)),
-          (not elixir_compiler:get_opt(internal)) andalso
-            'Elixir.Module.LocalsTracker':reattach(Module, Kind, Function, Neighbours),
           elixir_def:store_definition(false, FinalKind, Meta, FinalName, FinalArity,
                                       File, Module, Defaults, FinalClauses),
-          elixir_locals:record_definition(Tuple, FinalKind, Module),
-          elixir_locals:record_local(Tuple, Module, Function);
+          elixir_locals:reattach(Tuple, FinalKind, Module, Function, Neighbours);
         true ->
           ok
       end,
 
-      {ok, Tuple};
+      {FinalKind, FinalName};
     error ->
       error
   end.
 
-rewrite_clauses(Clauses) ->
-  [{Meta, [{'__CALLER__', [], nil} | Args], Guards, Body} ||
-   {Meta, Args, Guards, Body} <- Clauses].
-
 name(Name, Count) when is_integer(Count) ->
   list_to_atom(atom_to_list(Name) ++ " (overridable " ++ integer_to_list(Count) ++ ")").
 
@@ -83,6 +76,6 @@ format_error({no_super, Module, {Name, Arity}}) ->
     [Name, Arity, elixir_aliases:inspect(Module), Joined]).
 
 format_fa({Name, Arity}) ->
-  A = atom_to_binary(Name, utf8),
+  A = 'Elixir.Inspect.Function':escape_name(Name),
   B = integer_to_binary(Arity),
   <<A/binary, $/, B/binary>>.

lib/elixir/src/elixir_parser.yrl

@@ -378,9 +378,9 @@ capture_op_eol -> capture_op : '$1'.
 capture_op_eol -> capture_op eol : '$1'.
 
 unary_op_eol -> unary_op : '$1'.
-unary_op_eol -> unary_op eol : '$1'.
+unary_op_eol -> unary_op eol : warn_unary_operator_eol('$1'), '$1'.
 unary_op_eol -> dual_op : '$1'.
-unary_op_eol -> dual_op eol : '$1'.
+unary_op_eol -> dual_op eol : warn_unary_operator_eol('$1'), '$1'.
 
 match_op_eol -> match_op : '$1'.
 match_op_eol -> match_op eol : '$1'.
@@ -857,6 +857,25 @@ warn_empty_stab_clause({stab_op, {Line, _Begin, _End}, '->'}) ->
     "an expression is always required on the right side of ->. "
     "Please provide a value after ->").
 
+warn_unary_operator_eol({dual_op, {Line, _Begin, _End}, Op}) ->
+  elixir_errors:warn(Line, ?file(),
+    io_lib:format(
+      "unary operator ~ts followed by new line. "
+      "This may happen when you try to use a binary operator over multiple lines. "
+      "Please make sure that the operator and all arguments are on the same line",
+      [Op]
+    )
+  );
+warn_unary_operator_eol({unary_op, {Line, _Begin, _End}, Op}) ->
+  elixir_errors:warn(Line, ?file(),
+    io_lib:format(
+      "unary operator ~ts followed by new line. "
+      "Please make sure the unary operator and the expression that follows "
+      "the operator are on the same line",
+      [Op]
+     )
+  ).
+
 warn_pipe({arrow_op, {Line, _Begin, _End}, Op}, {_, [_ | _], [_ | _]}) ->
   elixir_errors:warn(Line, ?file(),
     io_lib:format(

lib/elixir/src/elixir_tokenizer.erl

@@ -142,7 +142,7 @@ tokenize(String, Line, Column, Opts) ->
     existing_atoms_only=ExistingAtomsOnly,
     check_terminators=CheckTerminators,
     preserve_comments=PreserveComments,
-    identifier_tokenizer=elixir_config:get(identifier_tokenizer)
+    identifier_tokenizer=elixir_config:safe_get(identifier_tokenizer, 'Elixir.String.Tokenizer')
   }).
 
 tokenize(String, Line, Opts) ->
@@ -670,7 +670,7 @@ eol(Line, Column, Tokens) -> [{eol, {Line, Column, Column + 1}} | Tokens].
 unsafe_to_atom(Part, Line, #elixir_tokenizer{}) when
     is_binary(Part) andalso size(Part) > 255;
     is_list(Part) andalso length(Part) > 255 ->
-  {error, {Line, "atom length must be less than system limit", ":"}};
+  {error, {Line, "atom length must be less than system limit: ", [$: | Part]}};
 unsafe_to_atom(Binary, _Line, #elixir_tokenizer{existing_atoms_only=true}) when is_binary(Binary) ->
   {ok, binary_to_existing_atom(Binary, utf8)};
 unsafe_to_atom(Binary, _Line, #elixir_tokenizer{}) when is_binary(Binary) ->

lib/elixir/src/elixir_utils.erl

@@ -86,15 +86,17 @@ change_universal_time(Name, {{Y, M, D}, {H, Min, Sec}}=Time)
     file:write_file_info(Name, #file_info{mtime=Time}, [{time, universal}]).
 
 relative_to_cwd(Path) ->
-  case elixir_compiler:get_opt(relative_paths) of
+  try elixir_compiler:get_opt(relative_paths) of
     true  -> 'Elixir.Path':relative_to_cwd(Path);
     false -> Path
+  catch
+    _:_ -> Path
   end.
 
 characters_to_list(Data) when is_list(Data) ->
   Data;
 characters_to_list(Data) ->
-  case elixir_compiler:get_opt(internal) of
+  case elixir_config:safe_get(bootstrap, true) of
     true  -> unicode:characters_to_list(Data);
     false -> 'Elixir.String':to_charlist(Data)
   end.
@@ -102,8 +104,8 @@ characters_to_list(Data) ->
 characters_to_binary(Data) when is_binary(Data) ->
   Data;
 characters_to_binary(Data) ->
-  case elixir_compiler:get_opt(internal) of
-    true  -> unicode:characters_to_binary(Data);
+  case elixir_config:safe_get(bootstrap, true) of
+    true -> unicode:characters_to_binary(Data);
     false -> 'Elixir.List':to_string(Data)
   end.
 

lib/elixir/src/elixir_with.erl

@@ -1,105 +0,0 @@
--module(elixir_with).
--export([expand/3]).
--include("elixir.hrl").
-
-expand(Meta, Args, Env) ->
-  {Exprs, Opts} =
-    case elixir_utils:split_last(Args) of
-      {_, LastArg} = SplitResult when is_list(LastArg) ->
-        SplitResult;
-      _ ->
-        {Args, []}
-    end,
-
-  {DoExpr, OtherOpts1} =
-    case lists:keytake(do, 1, Opts) of
-      {value, {do, DoValue}, RestOpts1} ->
-        {DoValue, RestOpts1};
-      false ->
-        elixir_errors:form_error(Meta, ?key(Env, file), elixir_expand, {missing_option, 'with', [do]})
-    end,
-
-  {ElseExpr, OtherOpts2} =
-    case lists:keytake(else, 1, OtherOpts1) of
-      {value, {else, ElseValue}, RestOpts2} ->
-        assert_clauses(Meta, ElseValue, Env),
-        {ElseValue, RestOpts2};
-      false ->
-        {nil, OtherOpts1}
-    end,
-
-  case OtherOpts2 of
-    [{Key, _} | _] ->
-      elixir_errors:form_error(Meta, ?key(Env, file), elixir_clauses, {unexpected_option, with, Key});
-    [] ->
-      ok
-  end,
-
-  ResultCase =
-    case ElseExpr of
-      nil ->
-        {MainCase, _} = build_main_case(Exprs, DoExpr, fun(Ret) -> Ret end, false),
-        MainCase;
-      _ ->
-        Wrapper = fun(Ret) -> {error, Ret} end,
-        case build_main_case(Exprs, {ok, DoExpr}, Wrapper, false) of
-          {MainCase, false} ->
-            Message =
-              "\"else\" clauses will never match"
-              " because all patterns in \"with\" will always match",
-            elixir_errors:warn(?line(Meta), ?key(Env, file), Message),
-            {{'.', Meta, [erlang, element]}, Meta, [MainCase, 2]};
-          {MainCase, true} ->
-            build_else_case(Meta, MainCase, ElseExpr, Wrapper)
-        end
-    end,
-  elixir_expand:expand(ResultCase, Env).
-
-%% Helpers
-
-assert_clauses(_Meta, [], _Env) ->
-  ok;
-assert_clauses(Meta, [{'->', _, [_, _]} | Rest], Env) ->
-  assert_clauses(Meta, Rest, Env);
-assert_clauses(Meta, _Other, Env) ->
-  elixir_errors:form_error(Meta, ?key(Env, file), elixir_clauses, {bad_or_missing_clauses, {with, else}}).
-
-build_main_case([{'<-', Meta, [{Name, _, Ctx}, _] = Args} | Rest], DoExpr, Wrapper, HasMatch)
-    when is_atom(Name) andalso is_atom(Ctx) ->
-  build_main_case([{'=', Meta, Args} | Rest], DoExpr, Wrapper, HasMatch);
-build_main_case([{'<-', Meta, [Left, Right]} | Rest], DoExpr, Wrapper, _HasMatch) ->
-  {InnerCase, true} = build_main_case(Rest, DoExpr, Wrapper, true),
-  Generated = ?generated(Meta),
-  Other = {other, Generated, ?var_context},
-  Clauses = [
-    {'->', Generated, [[Left], InnerCase]},
-    {'->', Generated, [[Other], Wrapper(Other)]}
-  ],
-  {{'case', [{export_vars, false} | Meta], [Right, [{do, Clauses}]]}, true};
-build_main_case([Expr | Rest], DoExpr, Wrapper, HasMatch) ->
-  {InnerCase, InnerHasMatch} = build_main_case(Rest, DoExpr, Wrapper, HasMatch),
-  {{'__block__', [], [Expr, InnerCase]}, InnerHasMatch};
-build_main_case([], DoExpr, _Wrapper, HasMatch) ->
-  {DoExpr, HasMatch}.
-
-build_else_case(Meta, MainCase, Clauses, Wrapper) ->
-  Generated = ?generated(Meta),
-
-  Return = {return, Generated, ?var_context},
-  ReturnClause = {'->', Generated, [[{ok, Return}], Return]},
-
-  Other = {other, Generated, ?var_context},
-  RaiseError = {{'.', Generated, [erlang, error]}, Meta, [{with_clause, Other}]},
-  RaiseErrorClause = {'->', Generated, [[Wrapper(Other)], RaiseError]},
-
-  ClauseWrapper = fun(Clause) -> wrap_clause_pattern(Clause, Wrapper) end,
-  ResultClauses = [ReturnClause] ++ lists:map(ClauseWrapper, Clauses) ++ [RaiseErrorClause],
-  {'case', [{export_vars, false} | Meta], [MainCase, [{do, ResultClauses}]]}.
-
-wrap_clause_pattern({'->', Meta, [[Left], Right]}, Wrapper) ->
-  {'->', Meta, [[wrap_pattern(Left, Wrapper)], Right]}.
-
-wrap_pattern({'when', Meta, [Left, Right]}, Wrapper) ->
-  {'when', Meta, [Wrapper(Left), Right]};
-wrap_pattern(Expr, Wrapper) ->
-  Wrapper(Expr).

lib/elixir/test/elixir/base_test.exs

@@ -750,4 +750,30 @@ defmodule BaseTest do
   test "hex_decode32!/2 with :mixed case and ignoring padding" do
     assert "fo" == hex_decode32!("cPNg", case: :mixed, padding: false)
   end
+
+  test "encode then decode is identity" do
+    for {encode, decode} <- [{&encode16/2, &decode16!/2},
+                             {&encode32/2, &decode32!/2},
+                             {&hex_encode32/2, &hex_decode32!/2},
+                             {&encode64/2, &decode64!/2},
+                             {&url_encode64/2, &url_decode64!/2}],
+        encode_case <- [:upper, :lower],
+        decode_case <- [:upper, :lower, :mixed],
+        (encode_case == decode_case) or (decode_case == :mixed),
+        pad? <- [true, false],
+        len <- 0..256 do
+      data =
+        0
+        |> :lists.seq(len - 1)
+        |> Enum.shuffle()
+        |> IO.iodata_to_binary()
+
+      expected =
+        data
+        |> encode.([case: encode_case, pad: pad?])
+        |> decode.([case: decode_case, pad: pad?])
+
+      assert data == expected, "identity did not match for #{inspect data} when #{inspect encode} (#{encode_case})"
+    end
+  end
 end

lib/elixir/test/elixir/calendar/date_range_test.exs

@@ -1,5 +1,5 @@
 Code.require_file "../test_helper.exs", __DIR__
-Code.require_file "../fixtures/calendar/julian.exs", __DIR__
+Code.require_file "../fixtures/calendar/holocene.exs", __DIR__
 
 defmodule Date.RangeTest do
   use ExUnit.Case, async: true
@@ -13,7 +13,7 @@ defmodule Date.RangeTest do
       assert Enum.member?(@asc_range, ~D[2000-01-01])
       assert Enum.member?(@asc_range, ~D[2001-01-01])
       refute Enum.member?(@asc_range, ~D[2002-01-01])
-      refute Enum.member?(@asc_range, Calendar.Julian.date(1999, 12, 19))
+      refute Enum.member?(@asc_range, Calendar.Holocene.date(12000, 1, 1))
     end
 
     test "for descending range" do
@@ -21,7 +21,7 @@ defmodule Date.RangeTest do
       assert Enum.member?(@desc_range, ~D[2000-01-01])
       assert Enum.member?(@desc_range, ~D[2001-01-01])
       refute Enum.member?(@desc_range, ~D[1999-01-01])
-      refute Enum.member?(@asc_range, Calendar.Julian.date(1999, 12, 19))
+      refute Enum.member?(@asc_range, Calendar.Holocene.date(12000, 1, 1))
     end
   end
 
@@ -49,16 +49,16 @@ defmodule Date.RangeTest do
 
   test "both dates must have matching calendars" do
     first = ~D[2000-01-01]
-    last = Calendar.Julian.date(2001, 01, 01)
+    last = Calendar.Holocene.date(12001, 1, 1)
 
     assert_raise ArgumentError, "both dates must have matching calendars", fn ->
       Date.range(first, last)
     end
   end
 
-  test "accepts equal but not Calendar.ISO calendars" do
-    first = Calendar.Julian.date(2000, 01, 01)
-    last = Calendar.Julian.date(2001, 01, 01)
+  test "accepts equal but non Calendar.ISO calendars" do
+    first = Calendar.Holocene.date(12000, 1, 1)
+    last = Calendar.Holocene.date(12001, 1, 1)
     range = Date.range(first, last)
     assert range
     assert first in range

lib/elixir/test/elixir/calendar_test.exs

@@ -1,5 +1,5 @@
 Code.require_file "test_helper.exs", __DIR__
-Code.require_file "fixtures/calendar/julian.exs", __DIR__
+Code.require_file "fixtures/calendar/holocene.exs", __DIR__
 
 defmodule FakeCalendar do
   def date_to_string(_, _, _), do: "boom"
@@ -37,7 +37,10 @@ defmodule DateTest do
 
   test "compare/2 across calendars" do
     date1 = ~D[2000-01-01]
-    date2 = Calendar.Julian.date(2000, 01, 01)
+    date2 = Calendar.Holocene.date(12000, 01, 01)
+    assert Date.compare(date1, date2) == :eq
+
+    date2 = Calendar.Holocene.date(12001, 01, 01)
     assert Date.compare(date1, date2) == :lt
     assert Date.compare(date2, date1) == :gt
   end
@@ -53,14 +56,23 @@ defmodule DateTest do
   end
 
   test "convert/2" do
-    assert Date.convert(~D[2000-01-01], Calendar.Julian) ==
-           {:ok, Calendar.Julian.date(1999, 12, 19)}
-    assert ~D[2000-01-01] |> Date.convert!(Calendar.Julian) |> Date.convert!(Calendar.ISO) ==
+    assert Date.convert(~D[2000-01-01], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.date(12000, 01, 01)}
+    assert ~D[2000-01-01] |> Date.convert!(Calendar.Holocene) |> Date.convert!(Calendar.ISO) ==
            ~D[2000-01-01]
     assert Date.convert(~D[2016-02-03], FakeCalendar) ==
            {:error, :incompatible_calendars}
-    assert Date.convert(~N[2000-01-01 00:00:00], Calendar.Julian) ==
-           {:ok, Calendar.Julian.date(1999, 12, 19)}
+    assert Date.convert(~N[2000-01-01 00:00:00], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.date(12000, 01, 01)}
+  end
+
+  test "add/2" do
+    assert_raise FunctionClauseError, fn ->
+      Date.add(~D[0000-01-01], 3652425)
+    end
+    assert_raise FunctionClauseError, fn ->
+      Date.add(~D[0000-01-01], -1)
+    end
   end
 
   test "diff/2" do
@@ -68,7 +80,7 @@ defmodule DateTest do
     assert Date.diff(~D[2000-01-01], ~D[2000-01-31]) == -30
 
     date1 = ~D[2000-01-01]
-    date2 = Calendar.Julian.date(2000, 01, 01)
+    date2 = Calendar.Holocene.date(12000, 01, 14)
     assert Date.diff(date1, date2) == -13
     assert Date.diff(date2, date1) == 13
   end
@@ -145,33 +157,40 @@ defmodule NaiveDateTimeTest do
 
   test "add/2 with other calendars" do
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.add(10, :second) ==
-           %NaiveDateTime{calendar: Calendar.Julian, year: 1999, month: 12, day: 19,
+           %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
                           hour: 12, minute: 34, second: 25, microsecond: {123456, 6}}
   end
 
   test "diff/2 with other calendars" do
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.add(10, :second)
            |> NaiveDateTime.diff(~N[2000-01-01 12:34:15.123456]) == 10
   end
 
   test "convert/2" do
-    assert NaiveDateTime.convert(~N[2000-01-01 12:34:15.123400], Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1999, 12, 19, 12, 34, 15, 123400)}
+    assert NaiveDateTime.convert(~N[2000-01-01 12:34:15.123400], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(12000, 1, 1, 12, 34, 15, {123400, 6})}
+
+    assert ~N[2000-01-01 12:34:15]
+           |> NaiveDateTime.convert!(Calendar.Holocene)
+           |> NaiveDateTime.convert!(Calendar.ISO) ==
+           ~N[2000-01-01 12:34:15]
+
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.convert!(Calendar.ISO) ==
            ~N[2000-01-01 12:34:15.123456]
+
     assert NaiveDateTime.convert(~N[2016-02-03 00:00:01], FakeCalendar) ==
            {:error, :incompatible_calendars}
 
-    assert NaiveDateTime.convert(~N[1970-01-01 00:00:00], Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1969, 12, 19, 0, 0, 0, {0, 0})}
-    assert NaiveDateTime.convert(DateTime.from_unix!(0, :seconds), Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1969, 12, 19, 0, 0, 0, {0, 0})}
+    assert NaiveDateTime.convert(~N[1970-01-01 00:00:00], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(11970, 1, 1, 0, 0, 0, {0, 0})}
+    assert NaiveDateTime.convert(DateTime.from_unix!(0, :seconds), Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(11970, 1, 1, 0, 0, 0, {0, 0})}
   end
 end
 
@@ -211,13 +230,20 @@ defmodule DateTimeTest do
     assert DateTime.from_unix(-62167219201) == {:error, :invalid_unix_time}
 
     max_datetime = %DateTime{
-      calendar: Calendar.ISO, day: 31, hour: 23, microsecond: {0, 0},
+      calendar: Calendar.ISO, day: 31, hour: 23, microsecond: {999999, 6},
       minute: 59, month: 12, second: 59, std_offset: 0, time_zone: "Etc/UTC",
       utc_offset: 0, year: 9999, zone_abbr: "UTC"
     }
 
-    assert DateTime.from_unix(253402300799) == {:ok, max_datetime}
+    assert DateTime.from_unix(253402300799999999, :microsecond) == {:ok, max_datetime}
     assert DateTime.from_unix(253402300800) == {:error, :invalid_unix_time}
+
+    minus_datetime = %DateTime{
+      calendar: Calendar.ISO, day: 31, hour: 23, microsecond: {999999, 6},
+      minute: 59, month: 12, second: 59, std_offset: 0, time_zone: "Etc/UTC",
+      utc_offset: 0, year: 1969, zone_abbr: "UTC"
+    }
+    assert DateTime.from_unix(-1, :microsecond) == {:ok, minus_datetime}
   end
 
   test "from_unix!/2" do
@@ -260,6 +286,31 @@ defmodule DateTimeTest do
     assert DateTime.compare(datetime2, datetime1) == :gt
   end
 
+  test "convert/2" do
+    datetime_iso = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "CET",
+                             hour: 23, minute: 0, second: 7, microsecond: {0, 0},
+                             utc_offset: 3600, std_offset: 0, time_zone: "Europe/Warsaw"}
+    datetime_hol = %DateTime{year: 12000, month: 2, day: 29, zone_abbr: "CET",
+                             hour: 23, minute: 0, second: 7, microsecond: {0, 0},
+                             utc_offset: 3600, std_offset: 0, time_zone: "Europe/Warsaw",
+                             calendar: Calendar.Holocene}
+
+    assert DateTime.convert(datetime_iso, Calendar.Holocene) == {:ok, datetime_hol}
+
+    assert datetime_iso
+           |> DateTime.convert!(Calendar.Holocene)
+           |> DateTime.convert!(Calendar.ISO) ==
+           datetime_iso
+
+    assert %{datetime_iso | microsecond: {123, 6}}
+           |> DateTime.convert!(Calendar.Holocene)
+           |> DateTime.convert!(Calendar.ISO) ==
+           %{datetime_iso | microsecond: {123, 6}}
+
+    assert DateTime.convert(datetime_iso, FakeCalendar) ==
+           {:error, :incompatible_calendars}
+  end
+
   test "from_iso8601/1 with tz offsets" do
     assert DateTime.from_iso8601("2017-06-02T14:00:00+01:00") |> elem(1) ==
            %DateTime{year: 2017, month: 6, day: 2, zone_abbr: "UTC",

lib/elixir/test/elixir/enum_test.exs

@@ -59,6 +59,24 @@ defmodule EnumTest do
     assert Enum.chunk([1, 2, 3, 4, 5], 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
+  test "chunk_every/2" do
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4], [5]]
+  end
+
+  test "chunk_every/4" do
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 2, 2, [6]) == [[1, 2], [3, 4], [5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard) == [[1, 2, 3], [3, 4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 2, 3, :discard) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, []) == [[1, 2, 3], [3, 4, 5], [5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 3, []) == [[1, 2, 3], [4, 5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 2, 3, []) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 2, 3, []) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6, 7], 2, 3, []) == [[1, 2], [4, 5], [7]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6, 7], 2, 3, [8]) == [[1, 2], [4, 5], [7, 8]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6, 7], 2, 4, []) == [[1, 2], [5, 6]]
+  end
+
   test "chunk_by/2" do
     assert Enum.chunk_by([1, 2, 2, 3, 4, 4, 6, 7, 7], &(rem(&1, 2) == 1)) == [[1], [2, 2], [3], [4, 4, 6], [7, 7]]
     assert Enum.chunk_by([1, 2, 3, 4], fn _ -> true end) == [[1, 2, 3, 4]]
@@ -66,12 +84,15 @@ defmodule EnumTest do
     assert Enum.chunk_by([1], fn _ -> true end) == [[1]]
   end
 
-  test "chunk_by/4" do
+  test "chunk_while/4" do
     chunk_fun = fn i, acc ->
-      if rem(i, 2) == 0 do
-        {:cont, Enum.reverse([i | acc]), []}
-      else
-        {:cont, [i | acc]}
+      cond do
+        i > 10 ->
+          {:halt, acc}
+        rem(i, 2) == 0 ->
+          {:cont, Enum.reverse([i | acc]), []}
+        true ->
+          {:cont, [i | acc]}
       end
     end
 
@@ -80,12 +101,16 @@ defmodule EnumTest do
       acc -> {:cont, Enum.reverse(acc), []}
     end
 
-    assert Enum.chunk_by([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], chunk_fun, after_fun) ==
+    assert Enum.chunk_while([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], chunk_fun, after_fun) ==
            [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert Enum.chunk_by(0..10, [], chunk_fun, after_fun) ==
+    assert Enum.chunk_while(0..9, [], chunk_fun, after_fun) ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9]]
+    assert Enum.chunk_while(0..10, [], chunk_fun, after_fun) ==
            [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert Enum.chunk_by(0..11, [], chunk_fun, after_fun) ==
-           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11]]
+    assert Enum.chunk_while(0..11, [], chunk_fun, after_fun) ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
+    assert Enum.chunk_while([5, 7, 9, 11], [], chunk_fun, after_fun) ==
+           [[5, 7, 9]]
   end
 
   test "concat/1" do
@@ -833,6 +858,18 @@ defmodule EnumTest.Range do
     assert Enum.chunk(1..5, 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
+  test "chunk_every/2" do
+    assert Enum.chunk_every(1..5, 2) == [[1, 2], [3, 4], [5]]
+  end
+
+  test "chunk_every/4" do
+    assert Enum.chunk_every(1..5, 2, 2) == [[1, 2], [3, 4], [5]]
+    assert Enum.chunk_every(1..6, 3, 2, :discard) == [[1, 2, 3], [3, 4, 5]]
+    assert Enum.chunk_every(1..6, 2, 3, :discard) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every(1..6, 3, 2, []) == [[1, 2, 3], [3, 4, 5], [5, 6]]
+    assert Enum.chunk_every(1..5, 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
+  end
+
   test "chunk_by/2" do
     assert Enum.chunk_by(1..4, fn _ -> true end) == [[1, 2, 3, 4]]
     assert Enum.chunk_by(1..4, &(rem(&1, 2) == 1)) == [[1], [2], [3], [4]]
@@ -1566,3 +1603,13 @@ defmodule EnumTest.SideEffects do
     assert Enum.take(iterator, 0) == []
   end
 end
+
+defmodule EnumTest.Function do
+  use ExUnit.Case, async: true
+
+  test "raises Protocol.UndefinedError for funs of wrong arity" do
+    assert_raise Protocol.UndefinedError, fn ->
+      Enum.to_list(fn -> nil end)
+    end
+  end
+end

lib/elixir/test/elixir/file_test.exs

@@ -460,6 +460,11 @@ defmodule FileTest do
       end
     end
 
+    test "cp_r raises on path with null byte" do
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.cp_r("source", "foo\0bar") end
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.cp_r("foo\0bar", "dest") end
+    end
+
     test "cp_r with src file and dest file" do
       src  = fixture_path("file.txt")
       dest = tmp_path("sample.txt")
@@ -1113,6 +1118,10 @@ defmodule FileTest do
       refute File.exists?(fixture)
     end
 
+    test "rm_rf raises on path with null byte" do
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.rm_rf("foo\0bar") end
+    end
+
     test "rm_rf with symlink" do
       from = tmp_path("tmp/from")
       to   = tmp_path("tmp/to")

lib/elixir/test/elixir/fixtures/calendar/holocene.exs

@@ -0,0 +1,44 @@
+defmodule Calendar.Holocene do
+  # This calendar is used to test conversions between calendars.
+  # It implements the Holocene calendar, which is based on the
+  # Propleptic Gregorian calendar with every year + 10000.
+
+  def date(year, month, day) do
+    %Date{year: year, month: month, day: day, calendar: __MODULE__}
+  end
+
+  def naive_datetime(year, month, day, hour, minute, second, microsecond \\ {0, 0}) do
+    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second,
+                   microsecond: microsecond, calendar: __MODULE__}
+  end
+
+  def date_to_string(year, month, day) do
+    "#{year}-#{month}-#{day} (HE)"
+  end
+
+  def naive_datetime_to_string(year, month, day, hour, minute, second, microsecond) do
+    "#{year}-#{month}-#{day} #{Calendar.ISO.time_to_string(hour, minute, second, microsecond)} (HE)"
+  end
+
+  defdelegate time_to_string(hour, minute, second, microsecond), to: Calendar.ISO
+
+  def day_rollover_relative_to_midnight_utc(), do: {0, 1}
+
+  def naive_datetime_from_iso_days(entry) do
+    {year, month, day, hour, minute, second, microsecond} =
+      Calendar.ISO.naive_datetime_from_iso_days(entry)
+    {year + 10000, month, day, hour, minute, second, microsecond}
+  end
+
+  def naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) do
+    Calendar.ISO.naive_datetime_to_iso_days(year - 10000, month, day, hour, minute, second, microsecond)
+  end
+
+  defdelegate time_from_day_fraction(day_fraction), to: Calendar.ISO
+
+  defdelegate time_to_day_fraction(hour, minute, second, microsecond), to: Calendar.ISO
+
+  defdelegate leap_year?(year), to: Calendar.ISO
+
+  defdelegate days_in_month(year, month), to: Calendar.ISO
+end

lib/elixir/test/elixir/fixtures/calendar/julian.exs

@@ -1,99 +0,0 @@
-defmodule Calendar.Julian do
-  # This calendar is used to test conversions between calendars.
-  # It implements the Julian calendar.
-
-  import Integer, only: [floor_div: 2]
-
-  def date_to_string(year, month, day) do
-    "#{year}-#{month}-#{day} (O.S.)"
-  end
-
-  def naive_datetime_to_string(year, month, day, hour, minute, second, microsecond) do
-    "#{year}-#{month}-#{day} #{Calendar.ISO.time_to_string(hour, minute, second, microsecond)} (O.S.)"
-  end
-
-  def time_to_string(hour, minute, second, microsecond) do
-    Calendar.ISO.time_to_string(hour, minute, second, microsecond)
-  end
-
-  def date(year, month, day) do
-    %Date{year: year, month: month, day: day, calendar: Calendar.Julian}
-  end
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond \\ {0, 0})
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond) when is_integer(microsecond) do
-    naive_datetime(year, month, day, hour, minute, second, {microsecond, 6})
-  end
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond) do
-    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: Calendar.Julian}
-  end
-
-  def day_rollover_relative_to_midnight_utc(), do: {0, 1}
-
-  def naive_datetime_from_rata_die({days, day_fraction}) do
-    {year, month, day} = date_from_rata_die(days)
-    {hour, minute, second, microsecond} = time_from_day_fraction(day_fraction)
-    {year, month, day, hour, minute, second, microsecond}
-  end
-
-  defp date_from_rata_die(days) do
-    approx = floor_div((4 * (days - epoch())) + 1464, 1461)
-    year = if approx <= 0, do: approx - 1, else: approx
-    prior_days = days - date_to_rata_die(year, 1, 1)
-    correction = cond do
-      days < date_to_rata_die(year, 3, 1) -> 0
-      leap_year?(year) ->  1
-      true -> 2
-    end
-    month = floor_div(12*(prior_days + correction) + 373, 367)
-    day = 1 + days - date_to_rata_die(year, month, 1)
-    {year, month, day}
-  end
-
-  def naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) do
-    days = date_to_rata_die(year, month, day)
-    day_fraction = time_to_day_fraction(hour, minute, second, microsecond)
-    {days, day_fraction}
-  end
-
-  def time_from_day_fraction(day_fraction), do: Calendar.ISO.time_from_day_fraction(day_fraction)
-
-  def time_to_day_fraction(hour, minute, second, microsecond) when is_integer(microsecond) do
-    time_to_day_fraction(hour, minute, second, {microsecond, 6})
-  end
-
-  def time_to_day_fraction(hour, minute, second, microsecond) do
-    Calendar.ISO.time_to_day_fraction(hour, minute, second, microsecond)
-  end
-
-  def leap_year?(year) when is_integer(year) and year >= 0 do
-    rem(year, 4) == if year > 0, do: 0, else: 3
-  end
-
-  def days_in_month(year, month)
-  def days_in_month(year, 2) do
-    if leap_year?(year), do: 29, else: 28
-  end
-  def days_in_month(_, month) when month in [4, 6, 9, 11], do: 30
-  def days_in_month(_, month) when month in 1..12, do: 31
-
-  defp date_to_rata_die(year, month, day) do
-    year = if year < 0, do: year + 1, else: year
-
-    epoch() - 1 + (365 * (year - 1)) + floor_div(year - 1, 4) +
-    floor_div(367 * month - 362, 12) +
-    adjustment_for_leap_year(year, month) + day
-  end
-
-  defp epoch(), do: -1
-
-  defp adjustment_for_leap_year(year, month) do
-    cond do
-      month <= 2 -> 0
-      leap_year?(year) -> -1
-      true -> -2
-    end
-  end
-end

lib/elixir/test/elixir/fixtures/dialyzer/defmacrop.ex

@@ -0,0 +1,11 @@
+defmodule Dialyzer.Defmacrop do
+  defmacrop good_macro(id) do
+    quote do
+      {:good, {:good_macro, unquote(id)}}
+    end
+  end
+
+  def run() do
+    good_macro("Not So Bad")
+  end
+end

lib/elixir/test/elixir/fixtures/dialyzer/with.ex

@@ -0,0 +1,45 @@
+defmodule Dialyzer.With do
+  def with_else do
+    with :ok <- ok_or_error(),
+         :ok <- ok_or_other_error(),
+         :ok <- ok_or_tuple_error(),
+         :ok <- ok_or_tuple_list_error() do
+      :ok
+    else
+      :error ->
+        :error
+
+      :other_error ->
+        :other_error
+
+      {:error, msg} when is_list(msg) or is_tuple(msg) ->
+        :error
+
+      {:error, msg} when is_list(msg) when is_tuple(msg) ->
+        :error
+
+      {:error, _msg} ->
+        :error
+    end
+  end
+
+  @spec ok_or_error() :: :ok | :error
+  defp ok_or_error do
+    Enum.random([:ok, :error])
+  end
+
+  @spec ok_or_other_error() :: :ok | :other_error
+  defp ok_or_other_error do
+    Enum.random([:ok, :other_error])
+  end
+
+  @spec ok_or_tuple_error() :: :ok | {:error, :err}
+  defp ok_or_tuple_error do
+    Enum.random([:ok, {:error, :err}])
+  end
+
+  @spec ok_or_tuple_list_error() :: :ok | {:error, [:err]}
+  defp ok_or_tuple_list_error do
+    Enum.random([:ok, {:error, [:err]}])
+  end
+end

lib/elixir/test/elixir/inspect_test.exs

@@ -452,6 +452,14 @@ defmodule Inspect.MapTest do
            inspect(%Inspect.Error{message: "#{msg}"})
   end
 
+  test "bad implementation safe disables colors" do
+    msg = "got KeyError with message \\\"key :unknown not found in: " <>
+          "%{__struct__: Inspect.MapTest.Failing, key: 0}\\\" while " <>
+          "inspecting %{__struct__: Inspect.MapTest.Failing, key: 0}"
+
+    assert inspect(%Failing{}, syntax_colors: [atom: [:green]]) =~ msg
+  end
+
   test "exception" do
     assert inspect(%RuntimeError{message: "runtime error"}) ==
            "%RuntimeError{message: \"runtime error\"}"

lib/elixir/test/elixir/kernel/dialyzer_test.exs

@@ -97,6 +97,18 @@ defmodule Kernel.DialyzerTest do
     assert_dialyze_no_warnings! context
   end
 
+  if :erlang.system_info(:otp_release) >= '20' do
+    test "no warnings on with/else", context do
+      copy_beam! context, Dialyzer.With
+      assert_dialyze_no_warnings! context
+    end
+  end
+
+  test "no warnings on defmacrop", context do
+    copy_beam! context, Dialyzer.Defmacrop
+    assert_dialyze_no_warnings! context
+  end
+
   defp copy_beam!(context, module) do
     name = "#{module}.beam"
     File.cp! Path.join(context[:base_dir], name),

lib/elixir/test/elixir/kernel/errors_test.exs

@@ -15,6 +15,12 @@ defmodule Kernel.ErrorsTest do
       '[foo: \u200B]\noops'
   end
 
+  test "invalid __CALLER__" do
+    assert_compile_fail CompileError,
+      "nofile:1: variable '__CALLER__' is unbound",
+      'defmodule Sample do def hello do __CALLER__ end end'
+  end
+
   test "invalid quoted token" do
     assert_compile_fail SyntaxError,
       "nofile:1: syntax error before: \"world\"",
@@ -232,7 +238,7 @@ defmodule Kernel.ErrorsTest do
 
   test "different defs with defaults" do
     assert_compile_fail CompileError,
-      "nofile:3: def hello/3 defaults conflicts with def hello/2",
+      "nofile:3: def hello/3 defaults conflicts with hello/2",
       ~C'''
       defmodule Kernel.ErrorsTest.DifferentDefsWithDefaults1 do
         def hello(a, b \\ nil), do: a + b
@@ -241,7 +247,7 @@ defmodule Kernel.ErrorsTest do
       '''
 
     assert_compile_fail CompileError,
-      "nofile:3: def hello/2 conflicts with defaults from def hello/3",
+      "nofile:3: def hello/2 conflicts with defaults from hello/3",
       ~C'''
       defmodule Kernel.ErrorsTest.DifferentDefsWithDefaults2 do
         def hello(a, b \\ nil, c \\ nil), do: a + b + c

lib/elixir/test/elixir/kernel/expansion_test.exs

@@ -169,10 +169,6 @@ defmodule Kernel.ExpansionTest do
       assert expand(quote do: __DIR__) == __DIR__
     end
 
-    test "__CALLER__" do
-      assert expand(quote do: __CALLER__) == quote do: __CALLER__
-    end
-
     test "__ENV__" do
       env = %{__ENV__ | line: 0}
       assert expand_env(quote(do: __ENV__), env) ==
@@ -498,50 +494,18 @@ defmodule Kernel.ExpansionTest do
 
   describe "with" do
     test "variables do not leak" do
-      input = quote(do: (with({foo} <- {bar}, do: baz = :ok); baz))
-      other = Macro.var(:other, :elixir_with)
-      result = quote do
-        case {bar()} do
-          {foo} -> baz = :ok
-          unquote(other) -> unquote(other)
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: baz = :ok); baz)))
+             quote(do: (with({foo} <- {bar()}, do: baz = :ok); baz()))
     end
 
     test "variables are available in do option" do
-      input = quote(do: (with({foo} <- {bar}, do: baz = foo); baz))
-      other = Macro.var(:other, :elixir_with)
-      result = quote do
-        case {bar()} do
-          {foo} -> baz = foo
-          unquote(other) -> unquote(other)
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: baz = foo); baz))) ==
+             quote(do: (with({foo} <- {bar()}, do: baz = foo); baz()))
     end
 
     test "variables inside else do not leak" do
-      input = quote(do: (with({foo} <- {bar}, do: :ok, else: (baz -> baz)); baz))
-      other = Macro.var(:other, :elixir_with)
-      return = Macro.var(:return, :elixir_with)
-      result = quote do
-        case(case {bar()} do
-          {foo} -> {:ok, :ok}
-          unquote(other) -> {:error, unquote(other)}
-        end) do
-          {:ok, unquote(return)} -> unquote(return)
-          {:error, baz} -> baz
-          {:error, unquote(other)} -> :erlang.error({:with_clause, unquote(other)})
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: :ok, else: (baz -> baz)); baz))) ==
+             quote(do: (with({foo} <- {bar()}, do: :ok, else: (baz -> baz)); baz()))
     end
 
     test "fails if \"do\" is missing" do
@@ -1187,6 +1151,24 @@ defmodule Kernel.ExpansionTest do
       end)
     end
 
+    assert_raise CompileError, ~r"invalid pattern in match", fn ->
+      expand(quote do
+        x = &(&1)
+        case true do
+          x.(false) -> true
+        end
+      end)
+    end
+
+    assert_raise CompileError, ~r"invalid expression in guard", fn ->
+      expand(quote do
+        x = &(&1)
+        case true do
+          true when x.(true) -> true
+        end
+      end)
+    end
+
     assert_raise CompileError, ~r"invalid call foo\(1\)\(2\)", fn ->
       expand(quote do: foo(1)(2))
     end
@@ -1198,6 +1180,14 @@ defmodule Kernel.ExpansionTest do
     assert_raise CompileError, ~r"unhandled operator ->", fn ->
       expand(quote do: (foo -> bar))
     end
+
+    assert_raise CompileError,
+      ~r"size in bitstring expects an integer or a variable as argument, got: foo\(\)",
+      fn ->
+        expand(quote do
+          fn <<_::size(foo)>> -> :ok end
+        end)
+      end
   end
 
   ## Helpers

lib/elixir/test/elixir/kernel/impl_test.exs

@@ -17,7 +17,11 @@ defmodule Kernel.ImplTest do
   end
 
   defmodule Behaviour do
-    @callback foo :: any
+    @callback foo() :: any
+  end
+
+  defmodule BehaviourWithArgument do
+    @callback foo(any) :: any
   end
 
   defmodule MacroBehaviour do
@@ -285,6 +289,20 @@ defmodule Kernel.ImplTest do
     end) =~ "got @impl Kernel.ImplTest.MacroBehaviour for def bar/0 but the given behaviour was not declared with @behaviour"
   end
 
+  test "does not warn for @impl when using default arguments" do
+    assert capture_err(fn ->
+      Code.eval_string ~S"""
+      defmodule Kernel.ImplTest.ImplAttributes do
+        @behaviour Kernel.ImplTest.Behaviour
+        @behaviour Kernel.ImplTest.BehaviourWithArgument
+
+        @impl true
+        def foo(args \\ []), do: args
+      end
+      """
+    end) == ""
+  end
+
   test "does not warn for no @impl when overriding callback" do
     assert capture_err(fn ->
       Code.eval_string """

lib/elixir/test/elixir/kernel/macros_test.exs

@@ -15,10 +15,12 @@ defmodule Kernel.MacrosTest do
 
   Kernel.MacrosTest.Nested = require Kernel.MacrosTest.Nested, as: Nested
 
+  @spec my_macro :: Macro.t
   defmacro my_macro do
     quote do: 1 + 1
   end
 
+  @spec my_private_macro :: Macro.t
   defmacrop my_private_macro do
     quote do: 1 + 3
   end

lib/elixir/test/elixir/kernel/overridable_test.exs

@@ -101,27 +101,6 @@ defmodule Kernel.OverridableExampleBehaviour do
 end
 
 defmodule Kernel.OverridableTest do
-  defmodule OverridableOrder do
-    def not_private(str) do
-      process_url(str)
-    end
-
-    def process_url(_str) do
-      :first
-    end
-
-    # There was a bug where the order in which we removed
-    # overridable expressions lead to errors. This module
-    # aims to guarantee removing process_url/1 before we
-    # remove the function that depends on it does not cause
-    # errors. If it compiles, it works!
-    defoverridable [process_url: 1, not_private: 1]
-
-    def process_url(_str) do
-      :second
-    end
-  end
-
   require Kernel.Overridable, as: Overridable
   use ExUnit.Case
 
@@ -130,6 +109,60 @@ defmodule Kernel.OverridableTest do
     :code.delete(module)
   end
 
+  test "overridable keeps function ordering" do
+    defmodule OverridableOrder do
+      def not_private(str) do
+        process_url(str)
+      end
+
+      def process_url(_str) do
+        :first
+      end
+
+      # There was a bug where the order in which we removed
+      # overridable expressions lead to errors. This module
+      # aims to guarantee removing process_url/1 before we
+      # remove the function that depends on it does not cause
+      # errors. If it compiles, it works!
+      defoverridable [process_url: 1, not_private: 1]
+
+      def process_url(_str) do
+        :second
+      end
+    end
+  end
+
+  test "overridable works with defaults" do
+    defmodule OverridableDefault do
+      def fun(value, opt \\ :from_parent) do
+        {value, opt}
+      end
+
+      defmacro macro(value, opt \\ :from_parent) do
+        {{value, opt}, Macro.escape(__CALLER__)}
+      end
+
+      # There was a bug where the default function would
+      # attempt to call its overridable name instead of
+      # func/1. If it compiles, it works!
+      defoverridable [fun: 1, fun: 2, macro: 1, macro: 2]
+
+      def fun(value) do
+        {value, super(value)}
+      end
+
+      defmacro macro(value) do
+        {{value, super(value)}, Macro.escape(__CALLER__)}
+      end
+    end
+
+    defmodule OverridableCall do
+      require OverridableDefault
+      OverridableDefault.fun(:foo)
+      OverridableDefault.macro(:bar)
+    end
+  end
+
   test "overridable is made concrete if no other is defined" do
     assert Overridable.sample == 1
   end

lib/elixir/test/elixir/kernel/quote_test.exs

@@ -195,14 +195,14 @@ end
 
 # DO NOT MOVE THIS LINE
 defmodule Kernel.QuoteTest.Errors do
-  defmacro defadd do
+  defmacro defraise do
     quote location: :keep do
-      def add(a, b), do: a + b
+      def will_raise(_a, _b), do: raise "oops"
     end
   end
 
   defmacro will_raise do
-    quote location: :keep, do: raise "omg"
+    quote location: :keep, do: raise "oops"
   end
 end
 
@@ -211,16 +211,16 @@ defmodule Kernel.QuoteTest.ErrorsTest do
   import Kernel.QuoteTest.Errors
 
   # Defines the add function
-  defadd()
+  defraise()
 
   test "inside function error" do
-    assert_raise ArithmeticError, fn ->
-      add(:a, :b)
+    assert_raise RuntimeError, fn ->
+      will_raise(:a, :b)
     end
 
     mod  = Kernel.QuoteTest.ErrorsTest
     file = __ENV__.file |> Path.relative_to_cwd |> String.to_charlist
-    assert [{^mod, :add, 2, [file: ^file, line: 200]} | _] = System.stacktrace
+    assert [{^mod, :will_raise, 2, [file: ^file, line: 200]} | _] = System.stacktrace
   end
 
   test "outside function error" do

lib/elixir/test/elixir/kernel/raise_test.exs

@@ -133,265 +133,300 @@ defmodule Kernel.RaiseTest do
     end
   end
 
-  test "rescue with underscore no exception" do
-    result = try do
-      RescueUndefinedModule.go
-    rescue
-      _ -> true
-    end
-
-    assert result
-  end
-
-  test "rescue with higher precedence than catch" do
-    result = try do
-      RescueUndefinedModule.go
-    catch
-      _, _ -> false
-    rescue
-      _ -> true
-    end
-
-    assert result
-  end
-
-  test "rescue runtime error" do
-    result = try do
-      raise "an exception"
-    rescue
-      RuntimeError -> true
-    catch
-      :error, _ -> false
-    end
-
-    assert result
-
-    result = try do
-      raise "an exception"
-    rescue
-      AnotherError -> true
-    catch
-      :error, _ -> false
-    end
-
-    refute result
-  end
-
-  test "rescue named runtime error" do
-    result = try do
-      raise "an exception"
-    rescue
-      x in [RuntimeError] -> Exception.message(x)
-    catch
-      :error, _ -> false
-    end
-
-    assert result == "an exception"
-  end
-
-  test "rescue argument error from elixir" do
-    result = try do
-     raise ArgumentError, ""
-    rescue
-      ArgumentError -> true
-    end
-
-    assert result
-  end
-
-  test "rescue named without aliases" do
-    result = try do
-      raise "an exception"
-    rescue
-      x -> Exception.message(x)
-    end
-
-    assert result == "an exception"
-  end
-
-  test "wrap custom Erlang error" do
-    result = try do
-      :erlang.error(:sample)
-    rescue
-      x in [RuntimeError, ErlangError] -> Exception.message(x)
-    end
-
-    assert result == "Erlang error: :sample"
-  end
-
-  test "undefined function error" do
-    result = try do
-      DoNotExist.for_sure()
-    rescue
-      x in [UndefinedFunctionError] -> Exception.message(x)
-    end
-
-    assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
-  end
-
-  test "function clause error" do
-    result = try do
-      zero(1)
-    rescue
-      x in [FunctionClauseError] -> Exception.message(x)
-    end
-
-    assert result == "no function clause matching in Kernel.RaiseTest.zero/1"
-  end
-
-  test "badarg error" do
-    result = try do
-      :erlang.error(:badarg)
-    rescue
-      x in [ArgumentError] -> Exception.message(x)
-    end
-
-    assert result == "argument error"
-  end
-
-  test "tuple badarg error" do
-    result = try do
-      :erlang.error({:badarg, [1, 2, 3]})
-    rescue
-      x in [ArgumentError] -> Exception.message(x)
-    end
-
-    assert result == "argument error: [1, 2, 3]"
-  end
-
-  test "badarith error" do
-    result = try do
-      :erlang.error(:badarith)
-    rescue
-      x in [ArithmeticError] -> Exception.message(x)
-    end
-
-    assert result == "bad argument in arithmetic expression"
-  end
-
-  test "badarity error" do
-    fun    = fn(x) -> x end
-    string = "#{inspect(fun)} with arity 1 called with 2 arguments (1, 2)"
-
-    result = try do
-      fun.(1, 2)
-    rescue
-      x in [BadArityError] -> Exception.message(x)
-    end
-
-    assert result == string
-  end
-
-  test "badfun error" do
-    # Avoid "invalid function call" warning in >= OTP 19
-    x = fn -> :example end
-    result = try do
-      x.().(2)
-    rescue
-      x in [BadFunctionError] -> Exception.message(x)
-    end
-
-    assert result == "expected a function, got: :example"
-  end
-
-  test "badmatch error" do
-    x = :example
-    result = try do
-      ^x = zero(0)
-    rescue
-      x in [MatchError] -> Exception.message(x)
-    end
-
-    assert result == "no match of right hand side value: 0"
-  end
-
-  test "bad key error" do
-    result = try do
-      %{%{} | foo: :bar}
-    rescue
-      x in [KeyError] -> Exception.message(x)
-    end
-
-    assert result == "key :foo not found"
-
-    result = try do
-      %{}.foo
-    rescue
-      x in [KeyError] -> Exception.message(x)
-    end
-
-    assert result == "key :foo not found in: %{}"
-  end
-
-  test "bad map error" do
-    result = try do
-      %{zero(0) | foo: :bar}
-    rescue
-      x in [BadMapError] -> Exception.message(x)
-    end
-
-    assert result == "expected a map, got: 0"
-  end
-
-  test "bad boolean error" do
-    result = try do
-      1 and true
-    rescue
-      x in [BadBooleanError] -> Exception.message(x)
-    end
-
-    assert result == "expected a boolean on left-side of \"and\", got: 1"
-  end
-
-  test "case clause error" do
-    x = :example
-    result = try do
-      case zero(0) do
-        ^x -> nil
+  describe "rescue" do
+    test "runtime error" do
+      result = try do
+        raise "an exception"
+      rescue
+        RuntimeError -> true
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [CaseClauseError] -> Exception.message(x)
-    end
 
-    assert result == "no case clause matching: 0"
-  end
+      assert result
 
-  test "cond clause error" do
-    result = try do
-      cond do
-        !zero(0) -> :ok
+      result = try do
+        raise "an exception"
+      rescue
+        AnotherError -> true
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [CondClauseError] -> Exception.message(x)
+
+      refute result
     end
 
-    assert result == "no cond clause evaluated to a true value"
-  end
-
-  test "try clause error" do
-    f = fn() -> :example end
-    result = try do
-      try do
-        f.()
-      else
-        :other ->
-          :ok
+    test "named runtime error" do
+      result = try do
+        raise "an exception"
+      rescue
+        x in [RuntimeError] -> Exception.message(x)
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [TryClauseError] -> Exception.message(x)
+
+      assert result == "an exception"
     end
 
-    assert result == "no try clause matching: :example"
+    test "with higher precedence than catch" do
+      result = try do
+        raise "an exception"
+      catch
+        _, _ -> false
+      rescue
+        _ -> true
+      end
+
+      assert result
+    end
+
+    test "argument error from erlang" do
+      result = try do
+        :erlang.error(:badarg)
+      rescue
+        ArgumentError -> true
+      end
+
+      assert result
+    end
+
+    test "argument error from elixir" do
+      result = try do
+        raise ArgumentError, ""
+      rescue
+        ArgumentError -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all variable" do
+      result = try do
+        raise "an exception"
+      rescue
+        x -> Exception.message(x)
+      end
+
+      assert result == "an exception"
+    end
+
+    test "catch-all underscore" do
+      result = try do
+        raise "an exception"
+      rescue
+        _ -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all unused variable" do
+      result = try do
+        raise "an exception"
+      rescue
+        _any -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all with \"x in _\" syntax" do
+      result = try do
+        raise "an exception"
+      rescue
+        exception in _ ->
+          Exception.message(exception)
+      end
+
+      assert result == "an exception"
+    end
   end
 
-  test "undefined function error as Erlang error" do
-    result = try do
-      DoNotExist.for_sure()
-    rescue
-      x in [ErlangError] -> Exception.message(x)
+  describe "normalize" do
+    test "wrap custom Erlang error" do
+      result = try do
+        :erlang.error(:sample)
+      rescue
+        x in [ErlangError] -> Exception.message(x)
+      end
+
+      assert result == "Erlang error: :sample"
     end
 
-    assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    test "undefined function error" do
+      result = try do
+        DoNotExist.for_sure()
+      rescue
+        x in [UndefinedFunctionError] -> Exception.message(x)
+      end
+
+      assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    end
+
+    test "function clause error" do
+      result = try do
+        zero(1)
+      rescue
+        x in [FunctionClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no function clause matching in Kernel.RaiseTest.zero/1"
+    end
+
+    test "badarg error" do
+      result = try do
+        :erlang.error(:badarg)
+      rescue
+        x in [ArgumentError] -> Exception.message(x)
+      end
+
+      assert result == "argument error"
+    end
+
+    test "tuple badarg error" do
+      result = try do
+        :erlang.error({:badarg, [1, 2, 3]})
+      rescue
+        x in [ArgumentError] -> Exception.message(x)
+      end
+
+      assert result == "argument error: [1, 2, 3]"
+    end
+
+    test "badarith error" do
+      result = try do
+        :erlang.error(:badarith)
+      rescue
+        x in [ArithmeticError] -> Exception.message(x)
+      end
+
+      assert result == "bad argument in arithmetic expression"
+    end
+
+    test "badarity error" do
+      fun    = fn(x) -> x end
+      string = "#{inspect(fun)} with arity 1 called with 2 arguments (1, 2)"
+
+      result = try do
+        fun.(1, 2)
+      rescue
+        x in [BadArityError] -> Exception.message(x)
+      end
+
+      assert result == string
+    end
+
+    test "badfun error" do
+      # Avoid "invalid function call" warning in >= OTP 19
+      x = fn -> :example end
+      result = try do
+        x.().(2)
+      rescue
+        x in [BadFunctionError] -> Exception.message(x)
+      end
+
+      assert result == "expected a function, got: :example"
+    end
+
+    test "badmatch error" do
+      x = :example
+      result = try do
+        ^x = zero(0)
+      rescue
+        x in [MatchError] -> Exception.message(x)
+      end
+
+      assert result == "no match of right hand side value: 0"
+    end
+
+    test "bad key error" do
+      result = try do
+        %{%{} | foo: :bar}
+      rescue
+        x in [KeyError] -> Exception.message(x)
+      end
+
+      assert result == "key :foo not found"
+
+      result = try do
+        %{}.foo
+      rescue
+        x in [KeyError] -> Exception.message(x)
+      end
+
+      assert result == "key :foo not found in: %{}"
+    end
+
+    test "bad map error" do
+      result = try do
+        %{zero(0) | foo: :bar}
+      rescue
+        x in [BadMapError] -> Exception.message(x)
+      end
+
+      assert result == "expected a map, got: 0"
+    end
+
+    test "bad boolean error" do
+      result = try do
+        1 and true
+      rescue
+        x in [BadBooleanError] -> Exception.message(x)
+      end
+
+      assert result == "expected a boolean on left-side of \"and\", got: 1"
+    end
+
+    test "case clause error" do
+      x = :example
+      result = try do
+        case zero(0) do
+          ^x -> nil
+        end
+      rescue
+        x in [CaseClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no case clause matching: 0"
+    end
+
+    test "cond clause error" do
+      result = try do
+        cond do
+          !zero(0) -> :ok
+        end
+      rescue
+        x in [CondClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no cond clause evaluated to a true value"
+    end
+
+    test "try clause error" do
+      f = fn() -> :example end
+      result = try do
+        try do
+          f.()
+        else
+          :other ->
+            :ok
+        end
+      rescue
+        x in [TryClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no try clause matching: :example"
+    end
+
+    test "undefined function error as Erlang error" do
+      result = try do
+        DoNotExist.for_sure()
+      rescue
+        x in [ErlangError] -> Exception.message(x)
+      end
+
+      assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    end
   end
 
   defmacrop exceptions do

lib/elixir/test/elixir/kernel/typespec_test.exs

@@ -507,6 +507,19 @@ defmodule Kernel.TypespecTest do
            specs(bytecode)
   end
 
+  test "@spec(spec) with tuples and tuple vars" do
+    bytecode = test_module do
+      def my_fun1(x), do: x
+      def my_fun2(x), do: x
+      @spec my_fun1(tuple) :: tuple
+      @spec my_fun2(tuple) :: tuple when tuple: {integer, integer}
+    end
+
+    assert [{{:my_fun1, 1}, [{:type, _, :fun, [{:type, _, :product, [{:type, _, :tuple, :any}]}, {:type, _, :tuple, :any}]}]},
+            {{:my_fun2, 1}, [{:type, _, :bounded_fun, [{:type, _, :fun, [{:type, _, :product, [{:var, _, :tuple}]}, {:var, _, :tuple}]}, _]}]}] =
+           specs(bytecode)
+  end
+
   test "@spec(spec) for unreachable private function" do
     # Run it inside capture_io/2 so that the "my_fun/1 is unused"
     # warning doesn't get printed among the ExUnit test results.

lib/elixir/test/elixir/kernel/warning_test.exs

@@ -84,6 +84,25 @@ defmodule Kernel.WarningTest do
     end) =~ message
   end
 
+  test "unary operator followed by new line" do
+    assert capture_err(fn ->
+      Code.eval_string """
+        1
+        +
+        1
+      """
+    end) =~ "unary operator + followed by new line"
+
+    assert capture_err(fn ->
+      Code.eval_string """
+        !
+        1
+      """
+    end) =~ "unary operator ! followed by new line"
+  after
+    purge Sample
+  end
+
   test "useless attr" do
     message = capture_err(fn ->
       Code.eval_string """
@@ -468,6 +487,27 @@ defmodule Kernel.WarningTest do
     purge Sample
   end
 
+  test "generated clause not match" do
+    assert capture_err(fn ->
+             Code.eval_string("""
+             defmodule Sample do
+               defmacro __using__(_) do
+                 quote do
+                   def hello, do: nil
+                   def hello, do: nil
+                 end
+               end
+             end
+             defmodule UseSample do
+               use Sample
+             end
+             """)
+           end) =~ "this clause cannot match because a previous clause at line 10 always matches"
+  after
+    purge(Sample)
+    purge(UseSample)
+  end
+
   test "clause with defaults should be first" do
     assert capture_err(fn ->
       Code.eval_string ~S"""
@@ -676,6 +716,31 @@ defmodule Kernel.WarningTest do
     purge Sample
   end
 
+  test "with and do clauses emit errors, else clauses do not" do
+    assert capture_err(fn ->
+      Code.compile_string """
+      with {:first, int} when is_integer(int) <- {:second, Integer.gcd(2, 4)} do
+        int
+      end
+      """
+    end) =~ "this clause cannot match"
+
+    assert capture_err(fn ->
+      Code.compile_string """
+      with {:first, int1} when is_integer(int1) <- {:first, Integer.gcd(2, 4)},
+           {:second, int2} when is_integer(int2) <- {:second, Integer.gcd(2, 4)} do
+        {:ok, int1 + int2}
+      else
+        {:first, nil} -> {:error, "first number is not integer"}
+        {:second, nil} -> {:error, "second number is not integer"}
+      end
+      """
+    end) == ""
+  after
+    purge Sample1
+    purge Sample2
+  end
+
   test "warning on codepoint escape" do
     assert capture_err(fn ->
       Code.eval_string "? "

lib/elixir/test/elixir/kernel/with_test.exs

@@ -10,7 +10,7 @@ defmodule Kernel.WithTest do
 
   test "matching with" do
     assert with(_..42 <- 1..42, do: :ok) == :ok
-    assert with({:ok, res} <- :error, do: res) == :error
+    assert with({:ok, res} <- error(), do: res) == :error
     assert with({:ok, _} = res <- ok(42), do: elem(res, 1)) == 42
   end
 
@@ -29,7 +29,7 @@ defmodule Kernel.WithTest do
     result = with({:ok, n1} <- ok(11), n2 <- 22, do: n1 + n2)
     assert result == 33
 
-    result = with(n1 <- 11, {:ok, n2} <- :error, do: n1 + n2)
+    result = with(n1 <- 11, {:ok, n2} <- error(), do: n1 + n2)
     assert result == :error
   end
 
@@ -65,9 +65,9 @@ defmodule Kernel.WithTest do
   end
 
   test "else conditions" do
-    assert with({:ok, res} <- 41, do: res, else: ({:error, error} -> error; res -> res + 1)) == 42
-    assert with({:ok, res} <- 41, do: res, else: (res when res == 41 -> res + 1; res -> res)) == 42
-    assert with({:ok, res} <- 41, do: res, else: (_ -> :error)) == :error
+    assert with({:ok, res} <- four(), do: res, else: ({:error, error} -> error; res -> res + 1)) == 5
+    assert with({:ok, res} <- four(), do: res, else: (res when res == 4 -> res + 1; res -> res)) == 5
+    assert with({:ok, res} <- four(), do: res, else: (_ -> :error)) == :error
   end
 
   test "else conditions with match error" do

lib/elixir/test/elixir/map_set_test.exs

@@ -108,4 +108,38 @@ defmodule MapSetTest do
     list = MapSet.to_list(MapSet.new(5..120))
     assert Enum.sort(list) == Enum.to_list(5..120)
   end
+
+  test "MapSet v1 compatibility" do
+    result = 1..5 |> map_set_v1() |> MapSet.new()
+    assert MapSet.equal?(result, MapSet.new(1..5))
+
+    result = MapSet.put(map_set_v1(1..5), 6)
+    assert MapSet.equal?(result, MapSet.new(1..6))
+
+    result = MapSet.union(map_set_v1(1..5), MapSet.new(6..10))
+    assert MapSet.equal?(result, MapSet.new(1..10))
+
+    result = MapSet.intersection(map_set_v1(1..10), MapSet.new(6..15))
+    assert MapSet.equal?(result, MapSet.new(6..10))
+
+    result = MapSet.difference(map_set_v1(1..10), MapSet.new(6..50))
+    assert MapSet.equal?(result, MapSet.new(1..5))
+
+    result = MapSet.delete(map_set_v1(1..10), 1)
+    assert MapSet.equal?(result, MapSet.new(2..10))
+
+    assert MapSet.size(map_set_v1(1..5)) == 5
+    assert MapSet.to_list(map_set_v1(1..5)) == Enum.to_list(1..5)
+
+    assert MapSet.disjoint?(map_set_v1(1..5), MapSet.new(10..15))
+    refute MapSet.disjoint?(map_set_v1(1..5), MapSet.new(5..10))
+
+    assert MapSet.subset?(map_set_v1(3..7), MapSet.new(1..10))
+    refute MapSet.subset?(map_set_v1(7..12), MapSet.new(1..10))
+  end
+
+  defp map_set_v1(enumerable) do
+    map = Map.new(enumerable, &{&1, true})
+    %{__struct__: MapSet, map: map}
+  end
 end

lib/elixir/test/elixir/map_test.exs

@@ -216,7 +216,7 @@ defmodule MapTest do
     end
   end
 
-  test "defstruct allow keys to be enforced" do
+  test "defstruct allows keys to be enforced" do
     message = "the following keys must also be given when building struct TestMod: [:foo]"
     assert_raise ArgumentError, message, fn ->
       Code.eval_string("""
@@ -243,6 +243,26 @@ defmodule MapTest do
     end
   end
 
+  test "struct always expands context module" do
+    Code.compiler_options(ignore_module_conflict: true)
+
+    defmodule LocalPoint do
+      defstruct x: 0
+      def new, do: %LocalPoint{}
+    end
+
+    assert LocalPoint.new == %{__struct__: LocalPoint, x: 0}
+
+    defmodule LocalPoint do
+      defstruct x: 0, y: 0
+      def new, do: %LocalPoint{}
+    end
+
+    assert LocalPoint.new == %{__struct__: LocalPoint, x: 0, y: 0}
+  after
+    Code.compiler_options(ignore_module_conflict: false)
+  end
+
   defmodule LocalUser do
     defmodule NestedUser do
       defstruct []

lib/elixir/test/elixir/module/locals_tracker_test.exs

@@ -60,11 +60,11 @@ defmodule Module.LocalsTrackerTest do
     D.add_local(config[:pid], {:foo, 1}, {:bar, 1})
     D.add_definition(config[:pid], :defp, {:bar, 1})
 
-    {infoo, outfoo}   = D.yank(config[:pid], {:foo, 1})
+    {infoo, outfoo} = D.yank(config[:pid], {:foo, 1})
     {inbar, outbar} = D.yank(config[:pid], {:bar, 1})
 
-    D.reattach(config[:pid], :defp, {:bar, 1}, {inbar, outbar})
-    D.reattach(config[:pid], :def, {:foo, 1}, {infoo, outfoo})
+    D.reattach(config[:pid], {:bar, 1}, :defp, {:bar, 1}, {inbar, outbar})
+    D.reattach(config[:pid], {:foo, 1}, :def, {:foo, 1}, {infoo, outfoo})
     assert {:bar, 1} in D.reachable(config[:pid])
   end
 

lib/elixir/test/elixir/module_test.exs

@@ -307,6 +307,12 @@ defmodule ModuleTest do
     end
   end
 
+  test "does not use ETS tables named after the module" do
+    in_module do
+      assert :ets.info(__MODULE__) == :undefined
+    end
+  end
+
   ## Definitions
 
   test "defines?" do

lib/elixir/test/elixir/path_test.exs

@@ -32,6 +32,10 @@ defmodule PathTest do
     File.rm_rf tmp_path("wildcard")
   end
 
+  test "wildcard/2 raises on null byte" do
+    assert_raise ArgumentError, ~r/null byte/, fn -> Path.wildcard("foo\0bar") end
+  end
+
   if windows?() do
     describe "Windows" do
       test "relative/1" do