Release v1.5.0

Tables named after modules are very likely to be used
the application itself, which can lead to conflicts in
scenarios such as code reloading.

Closes #6361

CHANGELOG.md

@@ -18,6 +18,28 @@ Elixir follows the recommendations in [Unicode Annex #31](http://unicode.org/rep
 
 For a complete reference on Elixir syntax, see the [Syntax Reference](https://hexdocs.pm/elixir/syntax-reference.html). For technical details on Unicode support, see [Unicode Syntax](https://hexdocs.pm/elixir/unicode-syntax.html).
 
+## IEx improvements
+
+IEx got many improvements. The autocompletion system is now capable of autocompleting variables and user imports. New helpers have also been added:
+
+  * `exports/1` lists all exports (functions and macros) in a given module
+  * `open/1` opens up the source of a module or function directly in your editor. For example, `open MyApp.Module`
+  * `runtime_info/0` prints general information about the running system, such as number of cores, runtime version, allocation of memory in the VM and more
+
+IEx also features a breakpoint system for code debugging. The following functions have been added to aid debugging:
+
+  * `break!/2` - sets up a breakpoint for a given `Mod.fun/arity`
+  * `break!/4` - sets up a breakpoint for the given module, function, arity
+  * `breaks/0` - prints all breakpoints and their ids
+  * `continue/0` - continues until the next breakpoint in the same process
+  * `open/0` - opens editor on the current breakpoint
+  * `remove_breaks/0` - removes all breakpoints in all modules
+  * `remove_breaks/1` - removes all breakpoints in a given module
+  * `reset_break/1` - sets the number of stops on the given id to zero
+  * `reset_break/3` - sets the number of stops on the given module, function, arity to zer
+  * `respawn/0` - starts a new shell (breakpoints will ask for permission once more)
+  * `whereami/1` - shows the current location
+
 ## Exception.blame
 
 `Exception.blame/3` is a new function in Elixir that is capable of attaching debug information to certain exceptions. Currently this is used to augment `FunctionClauseError`s with a summary of all clauses and which parts of clause match and which ones didn't. For example:
@@ -134,38 +156,48 @@ Overall, using `@impl` has the following advantages:
 
 This release brings further improvements to Calendar types. It adds arithmetic and others functions to `Time`, `Date`, `NaiveDateTime` and `Datetime` as well as conversion between different calendars.
 
-## v1.5.0-dev
+## v1.5.0 (2017-07-25)
 
 ### 1. Enhancements
 
 #### Elixir
 
-  * [Access] Optimize Access.get/2
-  * [Calendar] Limit `Calendar.ISO` up to year 10000
-  * [Calendar] Add Rata Die format for conversions between Calendars and `Date.convert/2`, `Time.convert/2`, `NaiveDateTime.convert/2` and `DateTime.convert/2` (as well as bang variants)
+  * [Access] Optimize `Access.get/2`
+  * [Base] Optimise Base encode/decode
+  * [Calendar] Implement Inspect for DateTime with Calendar.ISO
+  * [Calendar] Add "ISO days" format for conversions between Calendars and `Date.convert/2`, `Time.convert/2`, `NaiveDateTime.convert/2` and `DateTime.convert/2` (as well as bang variants)
   * [Calendar] Add `:calendar` field to `Time` struct
   * [Calendar] Add `Time.diff/3`, `Date.add/2`, `Date.diff/2`, `DateTime.diff/3`
   * [Calendar] Add `Date.range/2`
+  * [Calendar] Add `Date.new/4`, `DateTime.utc_now/1`, `NaiveDateTime.new/8` and `Time.new/5` that allow specifing calendar
   * [Enum] Add `Enum.chunk_by/4` and `Stream.chunk_by/4`
+  * [Enum] Add `Enum.chunk_every/2` and `Enum.chunk_every/4` with a more explicit API than `Enum.chunk/2` and `Enum.chunk/4`
   * [Exception] Add `Exception.blame/3` that adds metadata to exceptions
   * [File] Add `File.read_link/1` and `File.read_link!/1`
   * [File] Introduce `:trim_bom` option for `File.stream!/2`
   * [Inspect] Add `:printable_limit` to control the limit of printable structures
   * [Integer] Add `Integer.gcd/2`
+  * [Kernel] Add `left not in right` to check that the left side is not in the enumerable on the right
   * [Kernel] Use the new `debug_info` chunk in OTP 20. This provides a mechanism for tools to retrieve the Elixir AST from beam files
   * [Kernel] `defoverridable/1` accepts a module name as argument and marks all callbacks as overridable
   * [Kernel] Allow non-quoted Unicode atoms and variables according to Unicode Annex #31 (see Unicode Syntax document)
   * [Kernel] Warn when a `:__struct__` key is used when building/updating structs
+  * [Kernel] Cache the AST on definitions. This speeds up the compilation time from 10% to 15% measured across different projects
+  * [Kernel] Improve compiler error message on invalid patterns and guards
   * [Keyword] Add `replace/3` and `replace!/3` for replacing an existing key
   * [List] `List.starts_with?/2`
   * [Macro] Introduce `Macro.generate_arguments/2`
   * [Map] Optimize `Map.merge/3` by choosing merging direction
   * [Map] Add `replace/3` and `replace!/3` for replacing an existing key
+  * [Map] Raise `BadMapError` in `Map.equal?/2` when either of the two arguments is not a map
   * [MapSet] Reduce `MapSet` size when serialized to approximately half
   * [Process] Add `Process.cancel_timer/2`
   * [Protocol] Show available implementations on `Protocol.UndefinedError` if the protocol has been consolidated
   * [Registry] Support ETS guard conditions in `Registry.match/3`
   * [Registry] Support `parallel: true` in `Registry.dispatch/3`
+  * [Registry] Introduce `Registry.unregister_match/4`
+  * [Stream] Add `Stream.chunk_every/2` and `Stream.chunk_every/4` with a more explicit API than `Stream.chunk/2` and `Stream.chunk/4`
+  * [String] Optimise binary pattern matching in `String.split/1` and `String.trim_*/1`
   * [Supervisor] Add `Supervisor.init/2` and `Supervisor.child_spec/2`
   * [Supervisor] Allow `module` and `{module, arg}` to be given to `Supervisor.start_link/2` and invoke `module.child_spec(arg)` on each argument
   * [Task] Support `:on_timeout` in `Task.async_stream` to control how tasks are terminated
@@ -174,15 +206,20 @@ This release brings further improvements to Calendar types. It adds arithmetic a
 #### ExUnit
 
   * [ExUnit] Show code snippet from test source file in case of test errors
-  * [ExUnit] Show the value of variables used in an assertion
   * [ExUnit] Use `Exception.blame/3` when formatting test errors
   * [ExUnit] Make `assert_raise/2` fail if the underlying exception has a broken `message/1` implementation
   * [ExUnit] Add `start_supervised/2` and `stop_supervised/1` to ExUnit. Processes started by this function are automatically shut down when the test exits
 
 #### IEx
 
+  * [IEx.Autocomplete] Support autocompletion of variable names
+  * [IEx.Autocomplete] Support autocompletion of functions imported using `import Mod, only: [...]`
   * [IEx.Evaluator] Use `Exception.blame/3` when showing errors in the terminal
-  * [IEx.Helpers] Add `e/1` IEx helper to list all exports in a module
+  * [IEx.Helpers] Add `exports/1` IEx helper to list all exports in a module
+  * [IEx.Helpers] Add `break!/2`, `break!/4`, `breaks/0`, `continue/0`, `open/0`, `remove_breaks/0`, `remove_breaks/1`, `reset_break/1`, `reset_break/3` and `whereami/1` for code debugging
+  * [IEx.Helpers] No longer emit warnings for IEx commands without parentheses
+  * [IEx.Helpers] Add `runtime_info/0` for printing runtime system information
+  * [IEx.Helpers] Add `open/1` to open the source of a given module/function in your editor
   * [IEx.Info] Implement `IEx.Info` protocol for calendar types
 
 #### Logger
@@ -196,16 +233,32 @@ This release brings further improvements to Calendar types. It adds arithmetic a
   * [mix loadpaths] Ensure `--no-deps-check` do not trigger SCM callbacks (such as `git`)
   * [mix local.hex] Add `--if-missing` flag to `local.hex` mix task
   * [mix profile.cprof] Add `Mix.Tasks.Profile.Cprof` for count-based profiling
+  * [mix new] New styling for generated applications
 
 ### 2. Bug fixes
 
 #### Elixir
 
+  * [Calendar] Ensure `Calendar.ISO` raises a readable error when reaching up the year 10000 restriction
+  * [Calendar] Return `{:error, :invalid_time}` for wrong precision instead of crashing when parsing ISO dates
+  * [Enumerable] Raise `Protocol.UndefinedError` on bad functions in Enumerable implementation
+  * [File] Ensure recursive file operations raise on paths with null bytes (*security issue reported by Griffin Byatt*)
   * [File] Support `:ram`/`:raw` files in `File.copy/2`
+  * [Inspect] Do not use colors when inspecting error messages
   * [Kernel] Support guards on anonymous functions of zero arity
   * [Kernel] Fix compilation of maps used as maps keys inside matches
+  * [Kernel] Ensure `do` clause in `with` is tail call optimizable
+  * [Module] `on_definition/6` callback receives body wrapped in a keyword list, such as `[do: body]`. This solves a bug where it was impossible to distinguish between a bodyless clause and a function that returns `nil`.
+  * [Path] Ensure recursive path operations raise on paths with null bytes (*security issue reported by Griffin Byatt*)
+  * [Protocol] Do not lose source compile info on protocol consolidation
   * [Record] Properly escape quoted expressions passed to `defrecord`
+  * [Regex] Fix `inspect/2` for regexes with `/` terminator in them
+  * [Registry] Ensure `Registry.match/4` works with `:_` as key
+  * [Stream] Fix stream cycle over empty enumerable
   * [String] Consider Unicode non-characters valid according to the specification in `String.valid?/1`
+  * [StringIO] Fix encoding and performance issues in `StringIO.get_until`
+  * [System] Raise on paths with null bytes in `System.cmd/2` and in `System.find_executable/1` (*security issue reported by Griffin Byatt*)
+  * [System] Raise on ill-formed environment variables (*security issue reported by Griffin Byatt*)
 
 #### ExUnit
 
@@ -221,7 +274,9 @@ This release brings further improvements to Calendar types. It adds arithmetic a
 
   * [mix compile.elixir] Store multiple sources in case of module conflicts. This solves an issue where `_build` would get corrupted when compiling Elixir projects with module conflicts
   * [mix compile.erlang] Do not silently discard Erlang compile errors
+  * [mix compile.erlang] Properly track `-compile` module attribute when specified as a list
   * [mix compile.protocols] Ensure protocol implementations do not "disappear" when switching between applications in umbrella projects by having separate consolidation paths per project
+  * [mix compile.protocols] Do not raise when consolidating a protocol that was converted into a module
 
 ### 3. Soft deprecations (no warnings emitted)
 
@@ -247,6 +302,10 @@ This release brings further improvements to Calendar types. It adds arithmetic a
   * [String] `String.lstrip/2` and `String.rstrip/2` are deprecated in favor of `String.trim_leading/2` and `String.trim_trailing/2` with a binary as second argument
   * [Typespec] `char_list/0` type is deprecated in favor of `charlist/0`
 
+#### EEx
+
+  * [EEx] Deprecate `<%= ` in "middle" and "end" expressions, e.g.: `<%= else %>` and `<%= end %>`
+
 ## v1.4
 
 The CHANGELOG for v1.4 releases can be found [in the v1.4 branch](https://github.com/elixir-lang/elixir/blob/v1.4/CHANGELOG.md).

Makefile

@@ -1,7 +1,7 @@
 REBAR ?= "$(CURDIR)/rebar"
 PREFIX ?= /usr/local
 SHARE_PREFIX ?= $(PREFIX)/share
-CANONICAL := master/
+CANONICAL := v1.5/
 ELIXIRC := bin/elixirc --verbose --ignore-module-conflict
 ERLC := erlc -I lib/elixir/include
 ERL := erl -I lib/elixir/include -noshell -pa lib/elixir/ebin

RELEASE.md

@@ -10,18 +10,17 @@ This document simply outlines the release process:
 
 3. Ensure CHANGELOG is updated and add current date
 
-4. If a new `vMAJOR.MINOR`, update in [Deprecations](lib/elixir/pages/Deprecations.md) page the link
-   to [vVERSION]'s changelog (located at the end of the file) by replacing "master" with "vVERSION"
+4. If a new `vMAJOR.MINOR`, replace "master" with "vVERSION" in the "Deprecations" page and commit
 
-5. Commit changes above with title "Release vVERSION" and generate new tag
+5. If a new `vMAJOR.MINOR`, create a new branch "vMAJOR.MINOR" and set `CANONICAL=` in Makefile
 
-6. Run `make clean test` to ensure all tests pass from scratch and the CI is green
+6. Commit changes above with title "Release vVERSION" and generate new tag
 
-7. Recompile an existing project (for example, Ecto) to ensure manifests can be upgraded
+7. Run `make clean test` to ensure all tests pass from scratch and the CI is green
 
-8. Push branch and the new tag
+8. Recompile an existing project (for example, Ecto) to ensure manifests can be upgraded
 
-9. If a new `vMAJOR.MINOR`, create a new branch "vMAJOR.MINOR" and set `CANONICAL=` in Makefile before building docs
+9. Push branch and the new tag
 
 10. Publish new zips with `make zips`, upload `Precompiled.zip` and `Docs.zip` to GitHub Releases
 

VERSION

@@ -1 +1 @@
-1.5.0-dev
+1.5.0

lib/elixir/lib/base.ex

@@ -92,58 +92,145 @@ defmodule Base do
 
   """
 
-  b16_alphabet    = Enum.with_index '0123456789ABCDEF'
-  b64_alphabet    = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
-  b64url_alphabet = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
-  b32_alphabet    = Enum.with_index 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
-  b32hex_alphabet = Enum.with_index '0123456789ABCDEFGHIJKLMNOPQRSTUV'
+  b16_alphabet    = '0123456789ABCDEF'
+  b64_alphabet    = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
+  b64url_alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
+  b32_alphabet    = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'
+  b32hex_alphabet = '0123456789ABCDEFGHIJKLMNOPQRSTUV'
 
-  Enum.each [{:enc16,    :dec16,    b16_alphabet},
-             {:enc32,    :dec32,    b32_alphabet},
-             {:enc64,    :dec64,    b64_alphabet},
-             {:enc64url, :dec64url, b64url_alphabet},
-             {:enc32hex, :dec32hex, b32hex_alphabet}], fn({enc, dec, alphabet}) ->
-    for {encoding, value} <- alphabet do
-      defp unquote(enc)(unquote(value)), do: unquote(encoding)
-      defp unquote(dec)(unquote(encoding)), do: unquote(value)
-    end
-    defp unquote(dec)(c) do
-      raise ArgumentError, "non-alphabet digit found: #{inspect <<c>>, binaries: :as_strings} (byte #{c})"
+  defmacrop encode_pair(alphabet, case, value) do
+    quote do
+      case unquote(value) do
+        unquote(encode_pair_clauses(alphabet, case))
+      end
     end
   end
 
-  @compile {:inline, from_upper: 1, from_lower: 1, from_mixed: 1,
-                     to_lower: 1, to_upper: 1, enc16: 1, dec16: 1,
-                     enc32: 1, dec32: 1, enc32hex: 1, dec32hex: 1,
-                     enc64: 1, dec64: 1, enc64url: 1, dec64url: 1}
+  defp encode_pair_clauses(alphabet, case) when case in [:sensitive, :upper] do
+    shift = shift(alphabet)
+    alphabet
+    |> Enum.with_index()
+    |> encode_clauses(shift)
+  end
 
-  defp to_lower(char) when char in ?A..?Z,
-    do: char + (?a - ?A)
-  defp to_lower(char),
-    do: char
+  defp encode_pair_clauses(alphabet, :lower) do
+    shift = shift(alphabet)
+    alphabet
+    |> Stream.map(fn c -> (if c in ?A..?Z, do: c - ?A + ?a, else: c) end)
+    |> Enum.with_index()
+    |> encode_clauses(shift)
+  end
 
-  defp to_upper(char), do: char
+  defp shift(alphabet) do
+    alphabet
+    |> length()
+    |> :math.log2()
+    |> round()
+  end
 
-  defp from_upper(char), do: char
+  defp encode_clauses(alphabet, shift) do
+    for {encoding1, value1} <- alphabet, {encoding2, value2} <- alphabet do
+      encoding = bsl(encoding1, 8) + encoding2
+      value = bsl(value1, shift) + value2
+      [clause] = quote do: (unquote(value) -> unquote(encoding))
+      clause
+    end
+  end
 
-  defp from_lower(char) when char in ?a..?z,
-    do: char - (?a - ?A)
-  defp from_lower(char) when char not in ?A..?Z,
-    do: char
-  defp from_lower(char),
-    do: raise(ArgumentError, "non-alphabet digit found: \"#{<<char>>}\" (byte #{char})")
+  defmacrop decode_char(alphabet, case, encoding) do
+    quote do
+      case unquote(encoding) do
+        unquote(decode_char_clauses(alphabet, case))
+      end
+    end
+  end
 
-  defp from_mixed(char) when char in ?a..?z,
-    do: char - (?a - ?A)
-  defp from_mixed(char),
-    do: char
+  defp decode_char_clauses(alphabet, case) when case in [:sensitive, :upper] do
+    clauses =
+      alphabet
+      |> Enum.with_index()
+      |> decode_clauses()
+    clauses ++ bad_digit_clause()
+  end
 
-  defp maybe_pad(subject, false, _, _),
-    do: subject
-  defp maybe_pad(subject, _, group_size, pad) do
-    case rem(byte_size(subject), group_size) do
-      0 -> subject
-      x -> subject <> String.duplicate(pad, group_size - x)
+  defp decode_char_clauses(alphabet, :lower) do
+    {uppers, rest} =
+      alphabet
+      |> Stream.with_index()
+      |> Enum.split_with(fn {encoding, _} -> encoding in ?A..?Z end)
+    lowers =
+      Enum.map(uppers, fn {encoding, value} -> {encoding - ?A + ?a, value} end)
+
+    if length(uppers) > length(rest) do
+      decode_mixed_clauses(lowers, rest)
+    else
+      decode_mixed_clauses(rest, lowers)
+    end
+  end
+
+  defp decode_char_clauses(alphabet, :mixed) when length(alphabet) == 16 do
+    alphabet = Enum.with_index(alphabet)
+    lowers =
+      alphabet
+      |> Stream.filter(fn {encoding, _} -> encoding in ?A..?Z end)
+      |> Enum.map(fn {encoding, value} -> {encoding - ?A + ?a, value} end)
+    decode_mixed_clauses(alphabet, lowers)
+  end
+
+  defp decode_char_clauses(alphabet, :mixed) when length(alphabet) == 32 do
+    alphabet
+    |> Stream.with_index()
+    |> Enum.flat_map(fn {encoding, value} = pair ->
+         if encoding in ?A..?Z do
+           [pair, {encoding - ?A + ?a, value}]
+         else
+           [pair]
+         end
+       end)
+    |> decode_clauses()
+  end
+
+  defp decode_mixed_clauses(first, second) do
+    first_clauses = decode_clauses(first)
+    second_clauses = decode_clauses(second) ++ bad_digit_clause()
+    join_clause =
+      quote do
+        encoding ->
+          case encoding do
+            unquote(second_clauses)
+          end
+      end
+    first_clauses ++ join_clause
+  end
+
+  defp decode_clauses(alphabet) do
+    for {encoding, value} <- alphabet do
+      [clause] = quote do: (unquote(encoding) -> unquote(value))
+      clause
+    end
+  end
+
+  defp bad_digit_clause() do
+    quote do
+      c ->
+        raise ArgumentError,
+              "non-alphabet digit found: #{inspect <<c>>, binaries: :as_strings} (byte #{c})"
+    end
+  end
+
+  defp maybe_pad(body, "", _, _),
+    do: body
+  defp maybe_pad(body, tail, false, _),
+    do: body <> tail
+  defp maybe_pad(body, tail, _, group_size) do
+    case group_size - rem(byte_size(tail), group_size) do
+      ^group_size -> body <> tail
+      6 -> body <> tail <> "======"
+      5 -> body <> tail <> "====="
+      4 -> body <> tail <> "===="
+      3 -> body <> tail <> "==="
+      2 -> body <> tail <> "=="
+      1 -> body <> tail <> "="
     end
   end
 
@@ -587,7 +674,7 @@ defmodule Base do
   def hex_encode32(data, opts \\ []) when is_binary(data) do
     case = Keyword.get(opts, :case, :upper)
     pad? = Keyword.get(opts, :padding, true)
-    do_hex_encode32(case, data, pad?)
+    do_encode32hex(case, data, pad?)
   end
 
   @doc """
@@ -678,7 +765,7 @@ defmodule Base do
   def hex_decode32!(string, opts \\ []) when is_binary(string) do
     case = Keyword.get(opts, :case, :upper)
     pad? = Keyword.get(opts, :padding, true)
-    do_hex_decode32(case, string, pad?)
+    do_decode32hex(case, string, pad?)
   end
 
   defp remove_ignored(string, nil), do: string
@@ -686,244 +773,298 @@ defmodule Base do
     for <<char::8 <- string>>, char not in '\s\t\r\n', into: <<>>, do: <<char::8>>
   end
 
-  defp do_encode16(_, <<>>), do: <<>>
-  defp do_encode16(:upper, data) do
-    for <<c::4 <- data>>, into: <<>>, do: <<enc16(c)::8>>
+  enc16 = [upper: :enc16_upper, lower: :enc16_lower]
+
+  for {case, fun} <- enc16 do
+    defp unquote(fun)(char) do
+      encode_pair(unquote(b16_alphabet), unquote(case), char)
+    end
   end
-  defp do_encode16(:lower, data) do
-    for <<c::4 <- data>>, into: <<>>, do: <<to_lower(enc16(c))::8>>
+
+  defp do_encode16(_, <<>>), do: <<>>
+
+  for {case, fun} <- enc16 do
+    defp do_encode16(unquote(case), data) do
+      split =  8 * div(byte_size(data), 8)
+      <<main::size(split)-binary, rest::binary>> = data
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16,
+            unquote(fun)(c7)::16, unquote(fun)(c8)::16>>
+        end
+      case rest do
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16,
+            unquote(fun)(c7)::16>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16, unquote(fun)(c6)::16>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16,
+            unquote(fun)(c5)::16>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16, unquote(fun)(c4)::16>>
+        <<c1::8, c2::8, c3::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16,
+            unquote(fun)(c3)::16>>
+        <<c1::8, c2::8>> ->
+          <<main::binary, unquote(fun)(c1)::16, unquote(fun)(c2)::16>>
+        <<c1::8>> ->
+          <<main::binary, unquote(fun)(c1)::16>>
+        <<>> ->
+          main
+      end
+    end
+  end
+
+  dec16 = [upper: :dec16_upper, lower: :dec16_lower, mixed: :dec16_mixed]
+
+  for {case, fun} <- dec16 do
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(b16_alphabet), unquote(case), encoding)
+    end
   end
 
   defp do_decode16(_, <<>>), do: <<>>
-  defp do_decode16(:upper, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(c1)::4, dec16(c2)::4>>
-    end
-  end
-  defp do_decode16(:lower, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(from_lower(c1))::4, dec16(from_lower(c2))::4>>
-    end
-  end
-  defp do_decode16(:mixed, string) when rem(byte_size(string), 2) == 0 do
-    for <<c1::8, c2::8 <- string>>, into: <<>> do
-      <<dec16(from_mixed(c1))::4, dec16(from_mixed(c2))::4>>
+
+  for {case, fun} <- dec16 do
+    defp do_decode16(unquote(case), string) do
+      split =  8 * div(byte_size(string), 8)
+      <<main::size(split)-binary, rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4,
+            unquote(fun)(c5)::4, unquote(fun)(c6)::4,
+            unquote(fun)(c7)::4, unquote(fun)(c8)::4>>
+        end
+      case rest do
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4,
+            unquote(fun)(c5)::4, unquote(fun)(c6)::4>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4,
+            unquote(fun)(c3)::4, unquote(fun)(c4)::4>>
+        <<c1::8, c2::8>> ->
+          <<main::bits, unquote(fun)(c1)::4, unquote(fun)(c2)::4>>
+        <<_::8>> ->
+          raise ArgumentError, "odd-length string"
+        <<>> ->
+          main
+      end
     end
   end
 
-  defp do_encode64(<<>>, _), do: <<>>
-  defp do_encode64(data, pad?) do
-    split =  3 * div(byte_size(data), 3)
-    <<main::size(split)-binary, rest::binary>> = data
-    main = for <<c::6 <- main>>, into: <<>>, do: <<enc64(c)::8>>
-    tail = case rest do
-      <<c1::6, c2::6, c3::4>> ->
-        <<enc64(c1)::8, enc64(c2)::8, enc64(bsl(c3, 2))::8>>
-      <<c1::6, c2::2>> ->
-        <<enc64(c1)::8, enc64(bsl(c2, 4))::8>>
-      <<>> ->
-        <<>>
+  for {base, alphabet} <- ["64": b64_alphabet, "64url": b64url_alphabet] do
+    pair = :"enc#{base}_pair"
+    char = :"enc#{base}_char"
+    do_encode = :"do_encode#{base}"
+
+    defp unquote(pair)(value) do
+      encode_pair(unquote(alphabet), :sensitive, value)
     end
-    main <> maybe_pad(tail, pad?, 4, "=")
-  end
 
-  defp do_decode64(<<>>, _), do: <<>>
-  defp do_decode64(string, false) do
-    maybe_pad(string, true, 4, "=") |> do_decode64(true)
-  end
-  defp do_decode64(string, _pad?) when rem(byte_size(string), 4) == 0 do
-    split = byte_size(string) - 4
-    <<main::size(split)-binary, rest::binary>> = string
-    main = for <<c::8 <- main>>, into: <<>>, do: <<dec64(c)::6>>
-    tail = case rest do
-      <<c1::8, c2::8, ?=, ?=>> ->
-        <<dec64(c1)::6, bsr(dec64(c2), 4)::2>>
-      <<c1::8, c2::8, c3::8, ?=>> ->
-        <<dec64(c1)::6, dec64(c2)::6, bsr(dec64(c3), 2)::4>>
-      <<c1::8, c2::8, c3::8, c4::8>> ->
-        <<dec64(c1)::6, dec64(c2)::6, dec64(c3)::6, dec64(c4)::6>>
-      <<>> ->
-        <<>>
+    defp unquote(char)(value) do
+      value
+      |> unquote(pair)()
+      |> band(0x00FF)
     end
-    main <> tail
-  end
-  defp do_decode64(_, _) do
-    raise ArgumentError, "incorrect padding"
-  end
 
-  defp do_encode64url(<<>>, _), do: <<>>
-  defp do_encode64url(data, pad?) do
-    split =  3 * div(byte_size(data), 3)
-    <<main::size(split)-binary, rest::binary>> = data
-    main = for <<c::6 <- main>>, into: <<>>, do: <<enc64url(c)::8>>
-    tail = case rest do
-      <<c1::6, c2::6, c3::4>> ->
-        <<enc64url(c1)::8, enc64url(c2)::8, enc64url(bsl(c3, 2))::8>>
-      <<c1::6, c2::2>> ->
-        <<enc64url(c1)::8, enc64url(bsl(c2, 4))::8>>
-      <<>> ->
-        <<>>
+    defp unquote(do_encode)(<<>>, _), do: <<>>
+    defp unquote(do_encode)(data, pad?) do
+      split =  6 * div(byte_size(data), 6)
+      <<main::size(split)-binary, rest::binary>> = data
+      main = for <<c1::12, c2::12, c3::12, c4::12 <- main>>, into: <<>> do
+        <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+          unquote(pair)(c3)::16, unquote(pair)(c4)::16>>
+      end
+      tail = case rest do
+        <<c1::12, c2::12, c3::12, c::4>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16, unquote(pair)(c3)::16,
+            unquote(char)(bsl(c, 2))::8>>
+        <<c1::12, c2::12, c3::8>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(bsl(c3, 4))::16>>
+        <<c1::12, c2::12>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16>>
+        <<c1::12, c2::4>> ->
+          <<unquote(pair)(c1)::16, unquote(char)(bsl(c2, 2))::8>>
+        <<c1::8>> ->
+          <<unquote(pair)(bsl(c1, 4))::16>>
+        <<>> ->
+          <<>>
+      end
+      maybe_pad(main, tail, pad?, 4)
     end
-    main <> maybe_pad(tail, pad?, 4, "=")
   end
 
-  defp do_decode64url(<<>>, _), do: <<>>
-  defp do_decode64url(string, false) do
-    maybe_pad(string, true, 4, "=") |> do_decode64url(true)
-  end
-  defp do_decode64url(string, _pad?) when rem(byte_size(string), 4) == 0 do
-    split = byte_size(string) - 4
-    <<main::size(split)-binary, rest::binary>> = string
-    main = for <<c::8 <- main>>, into: <<>>, do: <<dec64url(c)::6>>
-    tail = case rest do
-      <<c1::8, c2::8, ?=, ?=>> ->
-        <<dec64url(c1)::6, bsr(dec64url(c2), 4)::2>>
-      <<c1::8, c2::8, c3::8, ?=>> ->
-        <<dec64url(c1)::6, dec64url(c2)::6, bsr(dec64url(c3), 2)::4>>
-      <<c1::8, c2::8, c3::8, c4::8>> ->
-        <<dec64url(c1)::6, dec64url(c2)::6, dec64url(c3)::6, dec64url(c4)::6>>
-      <<>> ->
-        <<>>
+  for {base, alphabet} <- ["64": b64_alphabet, "64url": b64url_alphabet] do
+    fun = :"dec#{base}"
+    do_decode = :"do_decode#{base}"
+
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(alphabet), :sensitive, encoding)
+    end
+
+    defp unquote(do_decode)(<<>>, _), do: <<>>
+    defp unquote(do_decode)(string, pad?) do
+      segs = div(byte_size(string) + 7, 8) - 1
+      <<main::size(segs)-binary-unit(64), rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::6, unquote(fun)(c2)::6, unquote(fun)(c3)::6,
+            unquote(fun)(c4)::6, unquote(fun)(c5)::6, unquote(fun)(c6)::6,
+            unquote(fun)(c7)::6, unquote(fun)(c8)::6>>
+        end
+      case rest do
+        <<c1::8, c2::8, ?=, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, bsr(unquote(fun)(c2), 4)::2>>
+        <<c1::8, c2::8, c3::8, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            bsr(unquote(fun)(c3), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, ?=, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            bsr(unquote(fun)(c6), 4)::2>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, bsr(unquote(fun)(c7), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, unquote(fun)(c7)::6, unquote(fun)(c8)::6>>
+        <<c1::8, c2::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, bsr(unquote(fun)(c2), 4)::2>>
+        <<c1::8, c2::8, c3::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            bsr(unquote(fun)(c3), 2)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            bsr(unquote(fun)(c6), 4)::2>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::6, unquote(fun)(c2)::6,
+            unquote(fun)(c3)::6, unquote(fun)(c4)::6, unquote(fun)(c5)::6,
+            unquote(fun)(c6)::6, bsr(unquote(fun)(c7), 2)::4>>
+        _ ->
+          raise ArgumentError, "incorrect padding"
+      end
     end
-    main <> tail
-  end
-  defp do_decode64url(_, _) do
-    raise ArgumentError, "incorrect padding"
   end
 
-  defp do_encode32(_, <<>>, _), do: <<>>
+  for {base, alphabet} <- ["32": b32_alphabet, "32hex": b32hex_alphabet],
+      case <- [:upper, :lower] do
+    pair = :"enc#{base}_#{case}_pair"
+    char = :"enc#{base}_#{case}_char"
+    do_encode = :"do_encode#{base}"
 
-  for {case, fun} <- [upper: :to_upper, lower: :to_lower] do
-    defp do_encode32(unquote(case), data, pad?) do
+    defp unquote(pair)(value) do
+      encode_pair(unquote(alphabet), unquote(case), value)
+    end
+
+    defp unquote(char)(value) do
+      value
+      |> unquote(pair)()
+      |> band(0x00FF)
+    end
+
+    defp unquote(do_encode)(_, <<>>, _), do: <<>>
+    defp unquote(do_encode)(unquote(case), data, pad?) do
       split =  5 * div(byte_size(data), 5)
       <<main::size(split)-binary, rest::binary>> = data
-      main = for <<c::5 <- main>>, into: <<>>, do: <<unquote(fun)(enc32(c))::8>>
+      main =
+        for <<c1::10, c2::10, c3::10, c4::10 <- main>>, into: <<>> do
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(c3)::16, unquote(pair)(c4)::16>>
+        end
       tail = case rest do
-        <<c1::5, c2::5, c3::5, c4::5, c5::5, c6::5, c7::2>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(c4))::8,
-            unquote(fun)(enc32(c5))::8, unquote(fun)(enc32(c6))::8,
-            unquote(fun)(enc32(bsl(c7, 3)))::8>>
-        <<c1::5, c2::5, c3::5, c4::5, c5::4>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(c4))::8,
-            unquote(fun)(enc32(bsl(c5, 1)))::8>>
-        <<c1::5, c2::5, c3::5, c4::1>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(c2))::8,
-            unquote(fun)(enc32(c3))::8, unquote(fun)(enc32(bsl(c4, 4)))::8>>
-        <<c1::5, c2::3>> ->
-          <<unquote(fun)(enc32(c1))::8, unquote(fun)(enc32(bsl(c2, 2)))::8>>
+        <<c1::10, c2::10, c3::10, c4::2>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(pair)(c3)::16, unquote(char)(bsl(c4, 3))::8>>
+        <<c1::10, c2::10, c3::4>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(c2)::16,
+            unquote(char)(bsl(c3, 1))::8>>
+        <<c1::10, c2::6>> ->
+          <<unquote(pair)(c1)::16, unquote(pair)(bsl(c2, 4))::16>>
+        <<c1::8>> ->
+          <<unquote(pair)(bsl(c1, 2))::16>>
         <<>> ->
           <<>>
       end
-      main <> maybe_pad(tail, pad?, 8, "=")
+      maybe_pad(main, tail, pad?, 8)
     end
   end
 
-  defp do_decode32(_, <<>>, _), do: <<>>
-  defp do_decode32(case, string, false),
-    do: do_decode32(case, maybe_pad(string, true, 8, "="), true)
+  for {base, alphabet} <- ["32": b32_alphabet, "32hex": b32hex_alphabet],
+      case <- [:upper, :lower, :mixed] do
+    fun = :"dec#{base}_#{case}"
+    do_decode = :"do_decode#{base}"
 
-  for {case, fun} <- [upper: :from_upper, lower: :from_lower, mixed: :from_mixed] do
-    defp do_decode32(unquote(case), string, _pad?) when rem(byte_size(string), 8) == 0 do
-      split = byte_size(string) - 8
-      <<main::size(split)-binary, rest::binary>> = string
-      main = for <<c::8 <- main>>, into: <<>>, do: <<dec32(unquote(fun)(c))::5>>
-      tail = case rest do
+    defp unquote(fun)(encoding) do
+      decode_char(unquote(alphabet), unquote(case), encoding)
+    end
+
+    defp unquote(do_decode)(_, <<>>, _), do: <<>>
+
+    defp unquote(do_decode)(unquote(case), string, pad?) do
+      segs = div(byte_size(string) + 7, 8) - 1
+      <<main::size(segs)-binary-unit(64), rest::binary>> = string
+      main =
+        for <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8 <- main>>, into: <<>> do
+          <<unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            unquote(fun)(c7)::5, unquote(fun)(c8)::5>>
+        end
+      case rest do
         <<c1::8, c2::8, ?=, ?=, ?=, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, bsr(dec32(unquote(fun)(c2)), 2)::3>>
+          <<main::bits, unquote(fun)(c1)::5, bsr(unquote(fun)(c2), 2)::3>>
         <<c1::8, c2::8, c3::8, c4::8, ?=, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, bsr(dec32(unquote(fun)(c4)), 4)::1>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, bsr(unquote(fun)(c4), 4)::1>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, ?=, ?=, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            bsr(dec32(unquote(fun)(c5)), 1)::4>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            bsr(unquote(fun)(c5), 1)::4>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            dec32(unquote(fun)(c5))::5, dec32(unquote(fun)(c6))::5,
-            bsr(dec32(unquote(fun)(c7)), 3)::2>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            bsr(unquote(fun)(c7), 3)::2>>
         <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
-          <<dec32(unquote(fun)(c1))::5, dec32(unquote(fun)(c2))::5,
-            dec32(unquote(fun)(c3))::5, dec32(unquote(fun)(c4))::5,
-            dec32(unquote(fun)(c5))::5, dec32(unquote(fun)(c6))::5,
-            dec32(unquote(fun)(c7))::5, dec32(unquote(fun)(c8))::5>>
-        <<>> ->
-          <<>>
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            unquote(fun)(c7)::5, unquote(fun)(c8)::5>>
+        <<c1::8, c2::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, bsr(unquote(fun)(c2), 2)::3>>
+        <<c1::8, c2::8, c3::8, c4::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, bsr(unquote(fun)(c4), 4)::1>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            bsr(unquote(fun)(c5), 1)::4>>
+        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8>> when not pad? ->
+          <<main::bits, unquote(fun)(c1)::5, unquote(fun)(c2)::5,
+            unquote(fun)(c3)::5, unquote(fun)(c4)::5,
+            unquote(fun)(c5)::5, unquote(fun)(c6)::5,
+            bsr(unquote(fun)(c7), 3)::2>>
+        _ ->
+          raise ArgumentError, "incorrect padding"
       end
-      main <> tail
     end
   end
-
-  defp do_decode32(_, _, _),
-    do: raise ArgumentError, "incorrect padding"
-
-  defp do_hex_encode32(_, <<>>, _), do: <<>>
-
-  for {case, fun} <- [upper: :to_upper, lower: :to_lower] do
-    defp do_hex_encode32(unquote(case), data, pad?) do
-      split =  5 * div(byte_size(data), 5)
-      <<main::size(split)-binary, rest::binary>> = data
-      main = for <<c::5 <- main>>, into: <<>>, do: <<unquote(fun)(enc32hex(c))::8>>
-      tail = case rest do
-        <<c1::5, c2::5, c3::5, c4::5, c5::5, c6::5, c7::2>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(c4))::8,
-            unquote(fun)(enc32hex(c5))::8, unquote(fun)(enc32hex(c6))::8,
-            unquote(fun)(enc32hex(bsl(c7, 3)))::8>>
-        <<c1::5, c2::5, c3::5, c4::5, c5::4>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(c4))::8,
-            unquote(fun)(enc32hex(bsl(c5, 1)))::8>>
-        <<c1::5, c2::5, c3::5, c4::1>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(c2))::8,
-            unquote(fun)(enc32hex(c3))::8, unquote(fun)(enc32hex(bsl(c4, 4)))::8>>
-        <<c1::5, c2::3>> ->
-          <<unquote(fun)(enc32hex(c1))::8, unquote(fun)(enc32hex(bsl(c2, 2)))::8>>
-        <<>> ->
-          <<>>
-      end
-      main <> maybe_pad(tail, pad?, 8, "=")
-    end
-  end
-
-  defp do_hex_decode32(_, <<>>, _), do: <<>>
-  defp do_hex_decode32(case, string, false),
-    do: do_hex_decode32(case, maybe_pad(string, true, 8, "="), true)
-
-  for {case, fun} <- [upper: :from_upper, lower: :from_lower, mixed: :from_mixed] do
-    defp do_hex_decode32(unquote(case), string, _pad?) when rem(byte_size(string), 8) == 0 do
-      split = byte_size(string) - 8
-      <<main::size(split)-binary, rest::binary>> = string
-      main = for <<c::8 <- main>>, into: <<>>, do: <<dec32hex(unquote(fun)(c))::5>>
-      tail = case rest do
-        <<c1::8, c2::8, ?=, ?=, ?=, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, bsr(dec32hex(unquote(fun)(c2)), 2)::3>>
-        <<c1::8, c2::8, c3::8, c4::8, ?=, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, bsr(dec32hex(unquote(fun)(c4)), 4)::1>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, ?=, ?=, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            bsr(dec32hex(unquote(fun)(c5)), 1)::4>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, ?=>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            dec32hex(unquote(fun)(c5))::5, dec32hex(unquote(fun)(c6))::5,
-            bsr(dec32hex(unquote(fun)(c7)), 3)::2>>
-        <<c1::8, c2::8, c3::8, c4::8, c5::8, c6::8, c7::8, c8::8>> ->
-          <<dec32hex(unquote(fun)(c1))::5, dec32hex(unquote(fun)(c2))::5,
-            dec32hex(unquote(fun)(c3))::5, dec32hex(unquote(fun)(c4))::5,
-            dec32hex(unquote(fun)(c5))::5, dec32hex(unquote(fun)(c6))::5,
-            dec32hex(unquote(fun)(c7))::5, dec32hex(unquote(fun)(c8))::5>>
-        <<>> ->
-          <<>>
-      end
-      main <> tail
-    end
-  end
-
-  defp do_hex_decode32(_, _, _),
-    do: raise ArgumentError, "incorrect padding"
 end

lib/elixir/lib/calendar.ex

@@ -35,17 +35,16 @@ defmodule Calendar do
   @typedoc """
   The internal date format that is used when converting between calendars.
 
-  This is the amount of days including the fractional part that has passed of the last day,
-  since midnight 1 January AD 1 of the Proleptic Gregorian Calendar
-  (0000-01-01+00:00T00:00.00000 in ISO 8601 notation).
+  This is the amount of days including the fractional part that has passed of
+  the last day since 0000-01-01+00:00T00:00.00000 in ISO 8601 notation (also
+  known as midnight 1 January BC 1 of the Proleptic Gregorian Calendar).
 
   The `parts_per_day` represent how many subparts the current day is subdivided in
   (for different calendars, picking a different `parts_per_day` might make sense).
-  The `parts_in_day` represents how many of these `parts_per_day` have passed in the last day.
-
-  Thus, a Rata Die like `{1234, {1, 2}}` should be read as `1234½`.
+  The `parts_in_day` represents how many of these `parts_per_day` have passed in the
+  last day.
   """
-  @type rata_die :: {days :: integer, day_fraction}
+  @type iso_days :: {days :: integer, day_fraction}
 
   @typedoc """
   Microseconds with stored precision.
@@ -127,14 +126,14 @@ defmodule Calendar do
   @callback time_to_string(hour, minute, second, microsecond) :: String.t
 
   @doc """
-  Converts the given datetime (with time zone) into the `t:rata_die` format.
+  Converts the given datetime (with time zone) into the `t:iso_days` format.
   """
-  @callback naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) :: rata_die
+  @callback naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) :: iso_days
 
   @doc """
-  Converts `t:rata_die` to the Calendar's datetime format.
+  Converts `t:iso_days` to the Calendar's datetime format.
   """
-  @callback naive_datetime_from_rata_die(rata_die) :: {year, month, day, hour, minute, second, microsecond}
+  @callback naive_datetime_from_iso_days(iso_days) :: {year, month, day, hour, minute, second, microsecond}
 
   @doc """
   Converts the given time to the `t:day_fraction` format.

lib/elixir/lib/calendar/date.ex

@@ -67,8 +67,11 @@ defmodule Date do
 
   ## Examples
 
-      iex> Date.range(~D[2000-01-01], ~D[2001-01-01])
-      #DateRange<~D[2000-01-01], ~D[2001-01-01]>
+      iex> Date.range(~D[1999-01-01], ~D[2000-01-01])
+      #DateRange<~D[1999-01-01], ~D[2000-01-01]>
+
+      iex> Date.range(~N[2000-01-01 09:00:00], ~D[1999-01-01])
+      #DateRange<~N[2000-01-01 09:00:00], ~D[1999-01-01]>
 
   A range of dates implements the `Enumerable` protocol, which means
   functions in the `Enum` module can be used to work with
@@ -79,24 +82,25 @@ defmodule Date do
       366
       iex> Enum.member?(range, ~D[2001-02-01])
       true
-      iex> Enum.reduce(range, 0, fn(_date, acc) -> acc - 1 end)
+      iex> Enum.reduce(range, 0, fn _date, acc -> acc - 1 end)
       -366
   """
 
-  @spec range(Date.t, Date.t) :: Date.Range.t
+  @spec range(Calendar.date, Calendar.date) :: Date.Range.t
   def range(%{calendar: calendar} = first, %{calendar: calendar} = last) do
-    {first_days, _} = to_rata_die(first)
-    {last_days, _} = to_rata_die(last)
+    {first_days, _} = to_iso_days(first)
+    {last_days, _} = to_iso_days(last)
 
     %Date.Range{
       first: first,
       last: last,
-      first_rata_die: first_days,
-      last_rata_die: last_days,
+      first_in_iso_days: first_days,
+      last_in_iso_days: last_days,
     }
   end
 
-  def range(%Date{}, %Date{}) do
+  def range(%{calendar: _, year: _, month: _, day: _},
+            %{calendar: _, year: _, month: _, day: _}) do
     raise ArgumentError, "both dates must have matching calendars"
   end
 
@@ -125,7 +129,7 @@ defmodule Date do
   end
 
   @doc """
-  Returns true if the year in `date` is a leap year.
+  Returns true if the year in the given `date` is a leap year.
 
   ## Examples
 
@@ -149,7 +153,7 @@ defmodule Date do
   end
 
   @doc """
-  Returns the number of days in the given date month.
+  Returns the number of days in the given `date` month.
 
   ## Examples
 
@@ -309,7 +313,7 @@ defmodule Date do
   end
 
   @doc """
-  Converts a `Date` struct to an Erlang date tuple.
+  Converts the given `date` to an Erlang date tuple.
 
   Only supports converting dates which are in the ISO calendar,
   or other calendars in which the days also start at midnight.
@@ -375,7 +379,7 @@ defmodule Date do
   end
 
   @doc """
-  Compares two `Date` structs.
+  Compares two date structs.
 
   Returns `:gt` if first date is later than the second
   and `:lt` for vice versa. If the two dates are equal
@@ -408,7 +412,7 @@ defmodule Date do
   end
   def compare(date1, date2) do
     if Calendar.compatible_calendars?(date1.calendar, date2.calendar) do
-      case {to_rata_die(date1), to_rata_die(date2)} do
+      case {to_iso_days(date1), to_iso_days(date2)} do
         {first, second} when first > second -> :gt
         {first, second} when first < second -> :lt
         _ -> :eq
@@ -424,21 +428,33 @@ defmodule Date do
   end
 
   @doc """
-  Converts a date from one calendar to another.
+  Converts the given `date` from it's calendar to the given `calendar`.
 
   Returns `{:ok, date}` if the calendars are compatible,
   or `{:error, :incompatible_calendars}` if they are not.
 
   See also `Calendar.compatible_calendars?/2`.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Date.convert(~D[2000-01-01], Calendar.Holocene)
+      {:ok, %Date{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1}}
+
   """
-  @spec convert(Calendar.date(), Calendar.calendar) :: {:ok, Date.t} | {:error, :incompatible_calendars}
-  def convert(%Date{calendar: calendar} = date, calendar), do: {:ok, date}
+  @spec convert(Calendar.date, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day}, calendar) do
+    {:ok, %Date{calendar: calendar, year: year, month: month, day: day}}
+  end
   def convert(%{calendar: calendar} = date, target_calendar) do
     if Calendar.compatible_calendars?(calendar, target_calendar) do
       result_date =
         date
-        |> to_rata_die()
-        |> from_rata_die(target_calendar)
+        |> to_iso_days()
+        |> from_iso_days(target_calendar)
       {:ok, result_date}
     else
       {:error, :incompatible_calendars}
@@ -448,8 +464,18 @@ defmodule Date do
   @doc """
   Similar to `Date.convert/2`, but raises an `ArgumentError`
   if the conversion between the two calendars is not possible.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Date.convert!(~D[2000-01-01], Calendar.Holocene)
+      %Date{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1}
+
   """
-  @spec convert!(Date.t, Calendar.calendar) :: Date.t
+  @spec convert!(Calendar.date, Calendar.calendar) :: t
   def convert!(date, calendar) do
     case convert(date, calendar) do
       {:ok, value} ->
@@ -460,7 +486,7 @@ defmodule Date do
   end
 
   @doc """
-  Adds the number of days to the given date.
+  Adds the number of days to the given `date`.
 
   The days are counted as gregorian days. The date is returned in the same
   calendar as it was given in.
@@ -472,11 +498,14 @@ defmodule Date do
       iex> Date.add(~D[2000-01-01], 2)
       ~D[2000-01-03]
 
+      iex> Date.add(~N[2000-01-01 09:00:00], 2)
+      ~D[2000-01-03]
+
   """
-  @spec add(Date.t, integer()) :: Date.t
-  def add(%Date{calendar: calendar} = date, days) do
-    {rata_days, fraction} = to_rata_die(date)
-    from_rata_die({rata_days + days, fraction}, calendar)
+  @spec add(Calendar.date, integer()) :: t
+  def add(%{calendar: calendar} = date, days) do
+    {iso_days_days, fraction} = to_iso_days(date)
+    from_iso_days({iso_days_days + days, fraction}, calendar)
   end
 
   @doc """
@@ -493,42 +522,45 @@ defmodule Date do
       iex> Date.diff(~D[2000-01-01], ~D[2000-01-03])
       -2
 
+      iex> Date.diff(~D[2000-01-01], ~N[2000-01-03 09:00:00])
+      -2
+
   """
-  @spec diff(Date.t, Date.t) :: integer
-  def diff(%Date{calendar: Calendar.ISO, year: year1, month: month1, day: day1},
-           %Date{calendar: Calendar.ISO, year: year2, month: month2, day: day2}) do
-    Calendar.ISO.date_to_rata_die_days(year1, month1, day1) -
-      Calendar.ISO.date_to_rata_die_days(year2, month2, day2)
+  @spec diff(Calendar.date, Calendar.date) :: integer
+  def diff(%{calendar: Calendar.ISO, year: year1, month: month1, day: day1},
+           %{calendar: Calendar.ISO, year: year2, month: month2, day: day2}) do
+    Calendar.ISO.date_to_iso_days_days(year1, month1, day1) -
+      Calendar.ISO.date_to_iso_days_days(year2, month2, day2)
   end
 
-  def diff(%Date{} = date1, %Date{} = date2) do
-    if Calendar.compatible_calendars?(date1.calendar, date2.calendar) do
-      {days1, _} = to_rata_die(date1)
-      {days2, _} = to_rata_die(date2)
+  def diff(%{calendar: calendar1} = date1, %{calendar: calendar2} = date2) do
+    if Calendar.compatible_calendars?(calendar1, calendar2) do
+      {days1, _} = to_iso_days(date1)
+      {days2, _} = to_iso_days(date2)
       days1 - days2
     else
       raise ArgumentError, "cannot calculate the difference between #{inspect date1} and #{inspect date2} because their calendars are not compatible and thus the result would be ambiguous"
     end
   end
 
-  defp to_rata_die(%{calendar: Calendar.ISO, year: year, month: month, day: day}) do
-    {Calendar.ISO.date_to_rata_die_days(year, month, day), {0, 86400000000}}
+  defp to_iso_days(%{calendar: Calendar.ISO, year: year, month: month, day: day}) do
+    {Calendar.ISO.date_to_iso_days_days(year, month, day), {0, 86400000000}}
   end
-  defp to_rata_die(%{calendar: calendar, year: year, month: month, day: day}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, 0, 0, 0, {0, 0})
+  defp to_iso_days(%{calendar: calendar, year: year, month: month, day: day}) do
+    calendar.naive_datetime_to_iso_days(year, month, day, 0, 0, 0, {0, 0})
   end
 
-  defp from_rata_die({days, _}, Calendar.ISO) do
-    {year, month, day} = Calendar.ISO.date_from_rata_die_days(days)
+  defp from_iso_days({days, _}, Calendar.ISO) do
+    {year, month, day} = Calendar.ISO.date_from_iso_days_days(days)
     %Date{year: year, month: month, day: day, calendar: Calendar.ISO}
   end
-  defp from_rata_die(rata_die, target_calendar) do
-    {year, month, day, _, _, _, _} = target_calendar.naive_datetime_from_rata_die(rata_die)
+  defp from_iso_days(iso_days, target_calendar) do
+    {year, month, day, _, _, _, _} = target_calendar.naive_datetime_from_iso_days(iso_days)
     %Date{year: year, month: month, day: day, calendar: target_calendar}
   end
 
   @doc """
-  Calculates the day of the week of a given `Date` struct.
+  Calculates the day of the week of a given `date`.
 
   Returns the day of the week as an integer. For the ISO 8601
   calendar (the default), it is an integer from 1 to 7, where

lib/elixir/lib/calendar/date_range.ex

@@ -13,22 +13,21 @@ defmodule Date.Range do
   """
 
   @type t :: %__MODULE__{first: Date.t, last: Date.t,
-                         first_rata_die: rata_die_days, last_rata_die: rata_die_days}
+                         first_in_iso_days: Calendar.days,
+                         last_in_iso_days: Calendar.days}
 
-  @opaque rata_die_days :: Calendar.days
-
-  defstruct [:first, :last, :first_rata_die, :last_rata_die]
+  defstruct [:first, :last, :first_in_iso_days, :last_in_iso_days]
 
   defimpl Enumerable do
     def member?(%{first: %{calendar: calendar, year: first_year, month: first_month, day: first_day},
                   last: %{calendar: calendar, year: last_year, month: last_month, day: last_day},
-                  first_rata_die: first_rata_die, last_rata_die: last_rata_die},
+                  first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days},
                 %Date{calendar: calendar, year: year, month: month, day: day}) do
       first = {first_year, first_month, first_day}
       last = {last_year, last_month, last_day}
       date = {year, month, day}
 
-      if first_rata_die <= last_rata_die do
+      if first_in_iso_days <= last_in_iso_days do
         {:ok, date >= first and date <= last}
       else
         {:ok, date >= last and date <= first}
@@ -39,13 +38,13 @@ defmodule Date.Range do
       {:ok, false}
     end
 
-    def count(%Date.Range{first_rata_die: first_rata_die, last_rata_die: last_rata_die}) do
-      {:ok, abs(first_rata_die - last_rata_die) + 1}
+    def count(%Date.Range{first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days}) do
+      {:ok, abs(first_in_iso_days - last_in_iso_days) + 1}
     end
 
-    def reduce(%Date.Range{first_rata_die: first_rata_die, last_rata_die: last_rata_die,
+    def reduce(%Date.Range{first_in_iso_days: first_in_iso_days, last_in_iso_days: last_in_iso_days,
                            first: %{calendar: calendar}}, acc, fun) do
-      reduce(first_rata_die, last_rata_die, acc, fun, calendar, first_rata_die <= last_rata_die)
+      reduce(first_in_iso_days, last_in_iso_days, acc, fun, calendar, first_in_iso_days <= last_in_iso_days)
     end
 
     defp reduce(_x, _y, {:halt, acc}, _fun, _calendar, _up?) do
@@ -57,24 +56,24 @@ defmodule Date.Range do
     end
 
     defp reduce(x, y, {:cont, acc}, fun, calendar, up? = true) when x <= y do
-      reduce(x + 1, y, fun.(date_from_rata_days(x, calendar), acc), fun, calendar, up?)
+      reduce(x + 1, y, fun.(date_from_iso_days_days(x, calendar), acc), fun, calendar, up?)
     end
 
     defp reduce(x, y, {:cont, acc}, fun, calendar, up? = false) when x >= y do
-      reduce(x - 1, y, fun.(date_from_rata_days(x, calendar), acc), fun, calendar, up?)
+      reduce(x - 1, y, fun.(date_from_iso_days_days(x, calendar), acc), fun, calendar, up?)
     end
 
     defp reduce(_, _, {:cont, acc}, _fun, _calendar, _up) do
       {:done, acc}
     end
 
-    defp date_from_rata_days(days, Calendar.ISO) do
-      {year, month, day} = Calendar.ISO.date_from_rata_die_days(days)
+    defp date_from_iso_days_days(days, Calendar.ISO) do
+      {year, month, day} = Calendar.ISO.date_from_iso_days_days(days)
       %Date{year: year, month: month, day: day, calendar: Calendar.ISO}
     end
 
-    defp date_from_rata_days(days, calendar) do
-      {year, month, day, _, _, _, _} = calendar.naive_datetime_from_rata_die({days, {0, 86400000000}})
+    defp date_from_iso_days_days(days, calendar) do
+      {year, month, day, _, _, _, _} = calendar.naive_datetime_from_iso_days({days, {0, 86400000000}})
       %Date{year: year, month: month, day: day, calendar: calendar}
     end
   end

lib/elixir/lib/calendar/datetime.ex

@@ -12,6 +12,11 @@ defmodule DateTime do
   are structural and based on the DateTime struct fields. For proper
   comparison between datetimes, use the `compare/2` function.
 
+  The functions on this module work with the `DateTime` struct as well
+  as any struct that contains the same fields as the `DateTime` struct.
+  Such functions expect `t:Calendar.datetime/0` in their typespecs
+  (instead of `t:t/0`).
+
   Developers should avoid creating the DateTime struct directly
   and instead rely on the functions provided by this module as
   well as the ones in 3rd party calendar libraries.
@@ -40,7 +45,7 @@ defmodule DateTime do
                          time_zone: Calendar.time_zone, zone_abbr: Calendar.zone_abbr,
                          utc_offset: Calendar.utc_offset, std_offset: Calendar.std_offset}
 
-  @unix_days :calendar.date_to_gregorian_days({1970, 1, 1}) - 365
+  @unix_days :calendar.date_to_gregorian_days({1970, 1, 1})
 
   @doc """
   Returns the current datetime in UTC.
@@ -52,13 +57,13 @@ defmodule DateTime do
       "Etc/UTC"
 
   """
-  @spec utc_now(Calendar.calendar) :: DateTime.t
+  @spec utc_now(Calendar.calendar) :: t
   def utc_now(calendar \\ Calendar.ISO) do
     System.os_time |> from_unix!(:native, calendar)
   end
 
   @doc """
-  Converts the given Unix time to DateTime.
+  Converts the given Unix time to `DateTime`.
 
   The integer can be given in different unit
   according to `System.convert_time_unit/3` and it will
@@ -69,27 +74,24 @@ defmodule DateTime do
 
   ## Examples
 
-      iex> DateTime.from_unix(1464096368)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {0, 0}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2016, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(1464096368)
+      iex> datetime
+      #DateTime<2016-05-24 13:26:08Z>
 
-      iex> DateTime.from_unix(1432560368868569, :microsecond)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 25, hour: 13, microsecond: {868569, 6}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2015, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(1432560368868569, :microsecond)
+      iex> datetime
+      #DateTime<2015-05-25 13:26:08.868569Z>
 
   The unit can also be an integer as in `t:System.time_unit/0`:
 
-      iex> DateTime.from_unix(143256036886856, 1024)
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 17, hour: 7, microsecond: {320312, 3},
-        minute: 5, month: 3, second: 22, std_offset: 0, time_zone: "Etc/UTC",
-        utc_offset: 0, year: 6403, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_unix(143256036886856, 1024)
+      iex> datetime
+      #DateTime<6403-03-17 07:05:22.320Z>
 
   Negative Unix times are supported, up to -62167219200 seconds,
   which is equivalent to "0000-01-01T00:00:00Z" or 0 Gregorian seconds.
   """
-  @spec from_unix(integer, :native | System.time_unit, Calendar.calendar) :: {:ok, DateTime.t} | {:error, atom}
+  @spec from_unix(integer, :native | System.time_unit, Calendar.calendar) :: {:ok, t} | {:error, atom}
   def from_unix(integer, unit \\ :second, calendar \\ Calendar.ISO) when is_integer(integer) do
     case Calendar.ISO.from_unix(integer, unit) do
       {:ok, {year, month, day}, {hour, minute, second}, microsecond} ->
@@ -103,7 +105,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given Unix time to DateTime.
+  Converts the given Unix time to `DateTime`.
 
   The integer can be given in different unit
   according to `System.convert_time_unit/3` and it will
@@ -116,22 +118,16 @@ defmodule DateTime do
 
       # An easy way to get the Unix epoch is passing 0 to this function
       iex> DateTime.from_unix!(0)
-      %DateTime{calendar: Calendar.ISO, day: 1, hour: 0, microsecond: {0, 0},
-                minute: 0, month: 1, second: 0, std_offset: 0, time_zone: "Etc/UTC",
-                utc_offset: 0, year: 1970, zone_abbr: "UTC"}
+      #DateTime<1970-01-01 00:00:00Z>
 
       iex> DateTime.from_unix!(1464096368)
-      %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {0, 0}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2016, zone_abbr: "UTC"}
+      #DateTime<2016-05-24 13:26:08Z>
 
       iex> DateTime.from_unix!(1432560368868569, :microsecond)
-      %DateTime{calendar: Calendar.ISO, day: 25, hour: 13, microsecond: {868569, 6}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2015, zone_abbr: "UTC"}
+      #DateTime<2015-05-25 13:26:08.868569Z>
 
   """
-  @spec from_unix!(integer, :native | System.time_unit, Calendar.calendar) :: DateTime.t
+  @spec from_unix!(integer, :native | System.time_unit, Calendar.calendar) :: t
   def from_unix!(integer, unit \\ :second, calendar \\ Calendar.ISO) when is_atom(unit) do
     case from_unix(integer, unit, calendar) do
       {:ok, datetime} ->
@@ -142,19 +138,19 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given NaiveDateTime to DateTime.
+  Converts the given `NaiveDateTime` to `DateTime`.
 
   It expects a time zone to put the NaiveDateTime in.
   Currently it only supports "Etc/UTC" as time zone.
 
   ## Examples
 
-      iex> DateTime.from_naive(~N[2016-05-24 13:26:08.003], "Etc/UTC")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {3000, 3}, minute: 26,
-                      month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                      year: 2016, zone_abbr: "UTC"}}
+      iex> {:ok, datetime} = DateTime.from_naive(~N[2016-05-24 13:26:08.003], "Etc/UTC")
+      iex> datetime
+      #DateTime<2016-05-24 13:26:08.003Z>
+
   """
-  @spec from_naive(NaiveDateTime.t, Calendar.time_zone) :: {:ok, DateTime.t}
+  @spec from_naive(NaiveDateTime.t, Calendar.time_zone) :: {:ok, t}
   def from_naive(naive_datetime, time_zone)
 
   def from_naive(%NaiveDateTime{calendar: calendar,
@@ -166,7 +162,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given NaiveDateTime to DateTime.
+  Converts the given `NaiveDateTime` to `DateTime`.
 
   It expects a time zone to put the NaiveDateTime in.
   Currently it only supports "Etc/UTC" as time zone.
@@ -174,12 +170,10 @@ defmodule DateTime do
   ## Examples
 
       iex> DateTime.from_naive!(~N[2016-05-24 13:26:08.003], "Etc/UTC")
-      %DateTime{calendar: Calendar.ISO, day: 24, hour: 13, microsecond: {3000, 3}, minute: 26,
-                month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
-                year: 2016, zone_abbr: "UTC"}
+      #DateTime<2016-05-24 13:26:08.003Z>
 
   """
-  @spec from_naive!(non_neg_integer, :native | System.time_unit) :: DateTime.t
+  @spec from_naive!(NaiveDateTime.t, Calendar.time_zone) :: t
   def from_naive!(naive_datetime, time_zone) do
     case from_naive(naive_datetime, time_zone) do
       {:ok, datetime} ->
@@ -190,9 +184,9 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given DateTime to Unix time.
+  Converts the given `datetime` to Unix time.
 
-  The DateTime is expected to be using the ISO calendar
+  The `datetime` is expected to be using the ISO calendar
   with a year greater than or equal to 0.
 
   It will return the integer with the given unit,
@@ -216,18 +210,18 @@ defmodule DateTime do
       -17412508655
 
   """
-  @spec to_unix(DateTime.t, System.time_unit) :: non_neg_integer
+  @spec to_unix(Calendar.datetime, System.time_unit) :: integer
   def to_unix(datetime, unit \\ :second)
 
-  def to_unix(%DateTime{utc_offset: utc_offset, std_offset: std_offset} = datetime, unit) do
-    {days, fraction} = to_rata_die(datetime)
-    unix_units = Calendar.ISO.rata_die_to_unit({days - @unix_days, fraction}, unit)
+  def to_unix(%{utc_offset: utc_offset, std_offset: std_offset} = datetime, unit) do
+    {days, fraction} = to_iso_days(datetime)
+    unix_units = Calendar.ISO.iso_days_to_unit({days - @unix_days, fraction}, unit)
     offset_units = System.convert_time_unit(utc_offset + std_offset, :second, unit)
     unix_units - offset_units
   end
 
   @doc """
-  Converts a `DateTime` into a `NaiveDateTime`.
+  Converts the given `datetime` into a `NaiveDateTime`.
 
   Because `NaiveDateTime` does not hold time zone information,
   any time zone related data will be lost during the conversion.
@@ -241,7 +235,8 @@ defmodule DateTime do
       ~N[2000-02-29 23:00:07.0]
 
   """
-  def to_naive(%DateTime{year: year, month: month, day: day, calendar: calendar,
+  @spec to_naive(t) :: NaiveDateTime.t
+  def to_naive(%DateTime{calendar: calendar, year: year, month: month, day: day,
                          hour: hour, minute: minute, second: second, microsecond: microsecond}) do
     %NaiveDateTime{year: year, month: month, day: day, calendar: calendar,
                    hour: hour, minute: minute, second: second, microsecond: microsecond}
@@ -262,6 +257,7 @@ defmodule DateTime do
       ~D[2000-02-29]
 
   """
+  @spec to_date(t) :: Date.t
   def to_date(%DateTime{year: year, month: month, day: day, calendar: calendar}) do
     %Date{year: year, month: month, day: day, calendar: calendar}
   end
@@ -281,6 +277,7 @@ defmodule DateTime do
       ~T[23:00:07.0]
 
   """
+  @spec to_time(t) :: Time.t
   def to_time(%DateTime{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}) do
     %Time{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}
   end
@@ -367,15 +364,17 @@ defmodule DateTime do
 
   ## Examples
 
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07Z")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 23, microsecond: {0, 0}, minute: 50, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 0}
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07.123+02:30")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 21, microsecond: {123000, 3}, minute: 20, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 9000}
-      iex> DateTime.from_iso8601("2015-01-23T23:50:07,123+02:30")
-      {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 21, microsecond: {123000, 3}, minute: 20, month: 1, second: 7, std_offset: 0,
-                      time_zone: "Etc/UTC", utc_offset: 0, year: 2015, zone_abbr: "UTC"}, 9000}
+      iex> {:ok, datetime, 0} = DateTime.from_iso8601("2015-01-23T23:50:07Z")
+      iex> datetime
+      #DateTime<2015-01-23 23:50:07Z>
+
+      iex> {:ok, datetime, 9000} = DateTime.from_iso8601("2015-01-23T23:50:07.123+02:30")
+      iex> datetime
+      #DateTime<2015-01-23 21:20:07.123Z>
+
+      iex> {:ok, datetime, 9000} = DateTime.from_iso8601("2015-01-23T23:50:07,123+02:30")
+      iex> datetime
+      #DateTime<2015-01-23 21:20:07.123Z>
 
       iex> DateTime.from_iso8601("2015-01-23P23:50:07")
       {:error, :invalid_format}
@@ -411,11 +410,12 @@ defmodule DateTime do
          {:ok, offset} <- parse_offset(rest) do
       %{year: year, month: month, day: day} = date
       %{hour: hour, minute: minute, second: second, microsecond: microsecond} = time
+      {_, precision} = microsecond
 
       datetime =
-        Calendar.ISO.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+        Calendar.ISO.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
         |> apply_tz_offset(offset)
-        |> from_rata_die("Etc/UTC", "UTC", 0, 0, calendar)
+        |> from_iso_days("Etc/UTC", "UTC", 0, 0, calendar, precision)
 
       {:ok, %{datetime | microsecond: microsecond}, offset}
     else
@@ -437,7 +437,7 @@ defmodule DateTime do
   end
 
   @doc """
-  Converts the given datetime to a string according to its calendar.
+  Converts the given `datetime` to a string according to its calendar.
 
   ### Examples
 
@@ -479,8 +479,21 @@ defmodule DateTime do
     end
   end
 
+  defimpl Inspect do
+    def inspect(%{calendar: Calendar.ISO, year: year, month: month, day: day,
+                  hour: hour, minute: minute, second: second, microsecond: microsecond,
+                  time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, _) do
+      "#DateTime<" <> Calendar.ISO.datetime_to_string(year, month, day, hour, minute, second, microsecond,
+                                                      time_zone, zone_abbr, utc_offset, std_offset) <> ">"
+    end
+
+    def inspect(datetime, opts) do
+      Inspect.Any.inspect(datetime, opts)
+    end
+  end
+
   @doc """
-  Compares two `DateTime` structs.
+  Compares two datetime structs.
 
   Returns `:gt` if first datetime is later than the second
   and `:lt` for vice versa. If the two datetimes are equal
@@ -501,24 +514,24 @@ defmodule DateTime do
       :gt
 
   """
-  @spec compare(DateTime.t, DateTime.t) :: :lt | :eq | :gt
+  @spec compare(Calendar.datetime, Calendar.datetime) :: :lt | :eq | :gt
   def compare(%DateTime{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
               %DateTime{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2) do
     {days1, {parts1, ppd1}} =
       datetime1
-      |> to_rata_die()
+      |> to_iso_days()
       |> apply_tz_offset(utc_offset1 + std_offset1)
 
     {days2, {parts2, ppd2}} =
       datetime2
-      |> to_rata_die()
+      |> to_iso_days()
       |> apply_tz_offset(utc_offset2 + std_offset2)
 
     # Ensure fraction tuples have same denominator.
-    rata_die1 = {days1, parts1 * ppd2}
-    rata_die2 = {days2, parts2 * ppd1}
+    iso_days1 = {days1, parts1 * ppd2}
+    iso_days2 = {days2, parts2 * ppd1}
 
-    case {rata_die1, rata_die2}  do
+    case {iso_days1, iso_days2}  do
       {first, second} when first > second -> :gt
       {first, second} when first < second -> :lt
       _ -> :eq
@@ -547,90 +560,111 @@ defmodule DateTime do
       -18000
 
   """
-  @spec diff(DateTime.t, DateTime.t) :: integer()
-  def diff(%DateTime{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
-           %DateTime{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2, unit \\ :second) do
+  @spec diff(Calendar.datetime, Calendar.datetime) :: integer()
+  def diff(%{utc_offset: utc_offset1, std_offset: std_offset1} = datetime1,
+           %{utc_offset: utc_offset2, std_offset: std_offset2} = datetime2, unit \\ :second) do
     naive_diff =
-      (datetime1 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)) -
-      (datetime2 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit))
+      (datetime1 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)) -
+      (datetime2 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit))
     offset_diff =
       (utc_offset2 + std_offset2) - (utc_offset1 + std_offset1)
     naive_diff + System.convert_time_unit(offset_diff, :second, unit)
   end
 
   @doc """
-  Converts a DateTime from one calendar to another.
+  Converts a given `datetime` from one calendar to another.
 
-  If this conversion fails for some reason, an `{:error, reason}` tuple is returned.
+  If it is not possible to convert unambiguously between the calendars
+  (see `Calendar.compatible_calendars?/2`), an `{:error, :incompatible_calendars}` tuple
+  is returned.
 
   ## Examples
 
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
       iex> dt1 = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
       ...>                 hour: 23, minute: 0, second: 7, microsecond: {0, 0},
       ...>                 utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
-      iex> DateTime.convert(dt1, Calendar.ISO)
-      {:ok, %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
-                      hour: 23, minute: 0, second: 7, microsecond: {0, 0},
-                      utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}}
+      iex> DateTime.convert(dt1, Calendar.Holocene)
+      {:ok, %DateTime{calendar: Calendar.Holocene, day: 29, hour: 23,
+                      microsecond: {0, 0}, minute: 0, month: 2, second: 7, std_offset: 0,
+                      time_zone: "America/Manaus", utc_offset: -14400, year: 12000,
+                      zone_abbr: "AMT"}}
 
   """
-  @spec convert(DateTime.t, Calendar.calendar) :: {:ok, DateTime.t} | {:error, atom}
-  def convert(%DateTime{calendar: calendar} = datetime, calendar) do
-    {:ok, datetime}
+  @spec convert(Calendar.datetime, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day,
+                hour: hour, minute: minute, second: second, microsecond: microsecond,
+                time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}, calendar) do
+    {:ok, %DateTime{calendar: calendar, year: year, month: month, day: day,
+                    hour: hour, minute: minute, second: second, microsecond: microsecond,
+                    time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}}
   end
 
-  def convert(%DateTime{} = datetime, calendar) do
-    result_datetime =
-      datetime
-      |> to_rata_die
-      |> from_rata_die(datetime, calendar)
-    {:ok, result_datetime}
+  def convert(%{calendar: dt_calendar, microsecond: {_, precision}} = datetime, calendar) do
+    if Calendar.compatible_calendars?(dt_calendar, calendar) do
+      result_datetime =
+        datetime
+        |> to_iso_days
+        |> from_iso_days(datetime, calendar, precision)
+      {:ok, result_datetime}
+    else
+      {:error, :incompatible_calendars}
+    end
   end
 
   @doc """
-  Converts a `DateTime` struct from one calendar to another.
+  Converts a given `datetime` from one calendar to another.
 
-  If this conversion fails for some reason, an `ArgumentError` is raised.
+  If it is not possible to convert unambiguously between the calendars
+  (see `Calendar.compatible_calendars?/2`), an ArgumentError is raised.
 
   ## Examples
 
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
       iex> dt1 = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
       ...>                 hour: 23, minute: 0, second: 7, microsecond: {0, 0},
       ...>                 utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
-      iex> DateTime.convert!(dt1, Calendar.ISO)
-      %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "AMT",
-                hour: 23, minute: 0, second: 7, microsecond: {0, 0},
-                utc_offset: -14400, std_offset: 0, time_zone: "America/Manaus"}
+      iex> DateTime.convert!(dt1, Calendar.Holocene)
+      %DateTime{calendar: Calendar.Holocene, day: 29, hour: 23,
+                microsecond: {0, 0}, minute: 0, month: 2, second: 7, std_offset: 0,
+                time_zone: "America/Manaus", utc_offset: -14400, year: 12000,
+                zone_abbr: "AMT"}
 
   """
-  @spec convert!(DateTime.t, Calendar.calendar) :: DateTime.t
+  @spec convert!(Calendar.datetime, Calendar.calendar) :: t | no_return
   def convert!(datetime, calendar) do
     case convert(datetime, calendar) do
       {:ok, value} ->
         value
-      {:error, reason} ->
-        raise ArgumentError, "cannot convert #{inspect datetime} to target calendar #{inspect calendar}, reason: #{inspect reason}"
+      {:error, :incompatible_calendars} ->
+        raise ArgumentError, "cannot convert #{inspect datetime} to target calendar #{inspect calendar}, reason: #{inspect datetime.calendar} and #{inspect calendar} have different day rollover moments, making this conversion ambiguous"
     end
   end
 
-  defp to_rata_die(%DateTime{calendar: calendar,year: year, month: month, day: day,
-                             hour: hour, minute: minute, second: second, microsecond: microsecond}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+  defp to_iso_days(%{calendar: calendar,year: year, month: month, day: day,
+                     hour: hour, minute: minute, second: second, microsecond: microsecond}) do
+    calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
   end
 
-  defp from_rata_die(rata_die, datetime, calendar) do
+  defp from_iso_days(iso_days, datetime, calendar, precision) do
     %{time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset} = datetime
-    from_rata_die(rata_die, time_zone, zone_abbr, utc_offset, std_offset, calendar)
+    from_iso_days(iso_days, time_zone, zone_abbr, utc_offset, std_offset, calendar, precision)
   end
 
-  defp from_rata_die(rata_die, time_zone, zone_abbr, utc_offset, std_offset, calendar) do
-    {year, month, day, hour, minute, second, microsecond} = calendar.naive_datetime_from_rata_die(rata_die)
-    %DateTime{year: year, month: month, day: day,
-              hour: hour, minute: minute, second: second, microsecond: microsecond,
+  defp from_iso_days(iso_days, time_zone, zone_abbr, utc_offset, std_offset, calendar, precision) do
+    {year, month, day, hour, minute, second, {microsecond, _}} = calendar.naive_datetime_from_iso_days(iso_days)
+    %DateTime{calendar: calendar, year: year, month: month, day: day,
+              hour: hour, minute: minute, second: second, microsecond: {microsecond, precision},
               time_zone: time_zone, zone_abbr: zone_abbr, utc_offset: utc_offset, std_offset: std_offset}
   end
 
-  defp apply_tz_offset(rata_die, offset) do
-    Calendar.ISO.add_day_fraction_to_rata_die(rata_die, -offset, 86400)
+  defp apply_tz_offset(iso_days, offset) do
+    Calendar.ISO.add_day_fraction_to_iso_days(iso_days, -offset, 86400)
   end
 end

lib/elixir/lib/calendar/iso.ex

@@ -30,46 +30,44 @@ defmodule Calendar.ISO do
   @microseconds_per_second 1_000_000
 
   @doc """
-  Returns the normalized Rata Die representation of the specified date.
+  Returns the `t:Calendar.iso_days` format of the specified date.
 
   ## Examples
 
-      iex> Calendar.ISO.naive_datetime_to_rata_die(1, 1, 1, 0, 0, 0, {0, 6})
-      {1, {0, 86400000000}}
-      iex> Calendar.ISO.naive_datetime_to_rata_die(2000, 1, 1, 12, 0, 0, {0, 6})
-      {730120, {43200000000, 86400000000}}
-      iex> Calendar.ISO.naive_datetime_to_rata_die(2000, 1, 1, 13, 0, 0, {0, 6})
-      {730120, {46800000000, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(0, 1, 1, 0, 0, 0, {0, 6})
+      {0, {0, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(2000, 1, 1, 12, 0, 0, {0, 6})
+      {730485, {43200000000, 86400000000}}
+      iex> Calendar.ISO.naive_datetime_to_iso_days(2000, 1, 1, 13, 0, 0, {0, 6})
+      {730485, {46800000000, 86400000000}}
 
   """
-  @spec naive_datetime_to_rata_die(Calendar.year, Calendar.month, Calendar.day,
+  @spec naive_datetime_to_iso_days(Calendar.year, Calendar.month, Calendar.day,
                                    Calendar.hour, Calendar.minute, Calendar.second,
-                                   Calendar.microsecond) :: Calendar.rata_die
-  def naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) do
-    {date_to_rata_die_days(year, month, day),
+                                   Calendar.microsecond) :: Calendar.iso_days
+  def naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) do
+    {date_to_iso_days_days(year, month, day),
      time_to_day_fraction(hour, minute, second, microsecond)}
   end
 
   @doc """
-  Converts a Rata Die to the datetime format specified by this calendar.
+  Converts the `t:Calendar.iso_days` format to the datetime format specified by this calendar.
 
   ## Examples
 
-      iex> Calendar.ISO.naive_datetime_from_rata_die({1, {0, 86400}})
-      {1, 1, 1, 0, 0, 0, {0, 6}}
-
-      iex> Calendar.ISO.naive_datetime_from_rata_die({730120, {0, 86400}})
+      iex> Calendar.ISO.naive_datetime_from_iso_days({0, {0, 86400}})
+      {0, 1, 1, 0, 0, 0, {0, 6}}
+      iex> Calendar.ISO.naive_datetime_from_iso_days({730485, {0, 86400}})
       {2000, 1, 1, 0, 0, 0, {0, 6}}
-
-      iex> Calendar.ISO.naive_datetime_from_rata_die({730120, {43200, 86400}})
+      iex> Calendar.ISO.naive_datetime_from_iso_days({730485, {43200, 86400}})
       {2000, 1, 1, 12, 0, 0, {0, 6}}
 
   """
-  @spec naive_datetime_from_rata_die(Calendar.rata_die) ::
+  @spec naive_datetime_from_iso_days(Calendar.iso_days) ::
         {Calendar.year, Calendar.month, Calendar.day,
          Calendar.hour, Calendar.minute, Calendar.second, Calendar.microsecond}
-  def naive_datetime_from_rata_die({days, day_fraction}) do
-    {year, month, day} = date_from_rata_die_days(days)
+  def naive_datetime_from_iso_days({days, day_fraction}) do
+    {year, month, day} = date_from_iso_days_days(days)
     {hour, minute, second, microsecond} = time_from_day_fraction(day_fraction)
     {year, month, day, hour, minute, second, microsecond}
   end
@@ -116,23 +114,22 @@ defmodule Calendar.ISO do
     {hours, minutes, seconds, {microseconds, 6}}
   end
 
-  # Converts a year, month, day in only a count of days since the Rata Die epoch.
+  # Converts year, month, day to count of days since 0000-01-01.
   @doc false
-  def date_to_rata_die_days(0, 1, 1) do
-    -365
+  def date_to_iso_days_days(0, 1, 1) do
+    0
   end
-  def date_to_rata_die_days(1970, 1, 1) do
-    719163
+  def date_to_iso_days_days(1970, 1, 1) do
+    719528
   end
-  def date_to_rata_die_days(year, month, day) do
-    # Rata Die starts at year 1, rather than at year 0.
-    :calendar.date_to_gregorian_days(year, month, day) - 365
+  def date_to_iso_days_days(year, month, day) do
+    :calendar.date_to_gregorian_days(year, month, day)
   end
 
-  # Calculates {year, month, day} from the count of days since the Rata Die epoch.
+  # Converts count of days since 0000-01-01 to {year, month, day} tuple.
   @doc false
-  def date_from_rata_die_days(days) do
-    :calendar.gregorian_days_to_date(days + 365)
+  def date_from_iso_days_days(days) do
+    :calendar.gregorian_days_to_date(days)
   end
 
   defp div_mod(int1, int2) do
@@ -273,8 +270,9 @@ defmodule Calendar.ISO do
     year <= 9999 and :calendar.valid_date(year, month, day)
   end
 
-  def valid_time?(hour, minute, second, {microsecond, _}) do
-    hour in 0..23 and minute in 0..59 and second in 0..60 and microsecond in 0..999_999
+  def valid_time?(hour, minute, second, {microsecond, precision}) do
+    hour in 0..23 and minute in 0..59 and second in 0..60 and
+      microsecond in 0..999_999 and precision in 0..6
   end
 
   def day_rollover_relative_to_midnight_utc() do
@@ -418,26 +416,26 @@ defmodule Calendar.ISO do
   end
 
   @doc false
-  def rata_die_to_unit({days, {parts, ppd}}, unit) do
+  def iso_days_to_unit({days, {parts, ppd}}, unit) do
     day_microseconds = days * @seconds_per_day * @microseconds_per_second
     microseconds = div(parts * @seconds_per_day * @microseconds_per_second, ppd)
     System.convert_time_unit(day_microseconds + microseconds, :microsecond, unit)
   end
 
   @doc false
-  def add_day_fraction_to_rata_die({days, {parts, ppd}}, add, ppd) do
-    normalize_rata_die(days, parts + add, ppd)
+  def add_day_fraction_to_iso_days({days, {parts, ppd}}, add, ppd) do
+    normalize_iso_days(days, parts + add, ppd)
   end
-  def add_day_fraction_to_rata_die({days, {parts, ppd}}, add, add_ppd) do
+  def add_day_fraction_to_iso_days({days, {parts, ppd}}, add, add_ppd) do
     parts = parts * add_ppd
     add = add * ppd
     gcd = Integer.gcd(ppd, add_ppd)
     result_parts = div(parts + add, gcd)
     result_ppd = div(ppd * add_ppd, gcd)
-    normalize_rata_die(days, result_parts, result_ppd)
+    normalize_iso_days(days, result_parts, result_ppd)
   end
 
-  defp normalize_rata_die(days, parts, ppd) do
+  defp normalize_iso_days(days, parts, ppd) do
     days_offset = div(parts, ppd)
     parts = rem(parts, ppd)
     if parts < 0 do

lib/elixir/lib/calendar/naive_datetime.ex

@@ -27,6 +27,11 @@ defmodule NaiveDateTime do
   `NaiveDateTime` is not validated against a time zone, such errors
   would go unnoticed.
 
+  The functions on this module work with the `NaiveDateTime` struct as well
+  as any struct that contains the same fields as the `NaiveDateTime` struct,
+  such as `DateTime`. Such functions expect
+  `t:Calendar.naive_datetime/0` in their typespecs (instead of `t:t/0`).
+
   Developers should avoid creating the NaiveDateTime structs directly
   and instead rely on the functions provided by this module as well
   as the ones in 3rd party calendar libraries.
@@ -149,7 +154,7 @@ defmodule NaiveDateTime do
   def new(date, time)
 
   def new(%Date{calendar: calendar, year: year, month: month, day: day},
-          %Time{hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: calendar}) do
+          %Time{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}) do
     {:ok, %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
                          hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
@@ -195,9 +200,9 @@ defmodule NaiveDateTime do
           integer, unit \\ :second) when is_integer(integer) do
     ppd = System.convert_time_unit(86400, :second, unit)
     naive_datetime
-    |> to_rata_die()
-    |> Calendar.ISO.add_day_fraction_to_rata_die(integer, ppd)
-    |> from_rata_die(calendar, precision)
+    |> to_iso_days()
+    |> Calendar.ISO.add_day_fraction_to_iso_days(integer, ppd)
+    |> from_iso_days(calendar, precision)
   end
 
   @doc """
@@ -232,8 +237,8 @@ defmodule NaiveDateTime do
       raise ArgumentError, "cannot calculate the difference between #{inspect naive_datetime1} and #{inspect naive_datetime2} because their calendars are not compatible and thus the result would be ambiguous"
     end
 
-    units1 = naive_datetime1 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)
-    units2 = naive_datetime2 |> to_rata_die() |> Calendar.ISO.rata_die_to_unit(unit)
+    units1 = naive_datetime1 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)
+    units2 = naive_datetime2 |> to_iso_days() |> Calendar.ISO.iso_days_to_unit(unit)
     units1 - units2
   end
 
@@ -509,6 +514,7 @@ defmodule NaiveDateTime do
       {:error, :invalid_date}
       iex> NaiveDateTime.from_erl({{2000, 13, 1},{13, 30, 15}})
       {:error, :invalid_date}
+
   """
   @spec from_erl(:calendar.datetime, Calendar.microsecond) :: {:ok, t} | {:error, atom}
   def from_erl(tuple, microsecond \\ {0, 0}, calendar \\ Calendar.ISO)
@@ -532,6 +538,7 @@ defmodule NaiveDateTime do
       ~N[2000-01-01 13:30:15.005]
       iex> NaiveDateTime.from_erl!({{2000, 13, 1}, {13, 30, 15}})
       ** (ArgumentError) cannot convert {{2000, 13, 1}, {13, 30, 15}} to naive datetime, reason: :invalid_date
+
   """
   @spec from_erl!(:calendar.datetime, Calendar.microsecond) :: t | no_return
   def from_erl!(tuple, microsecond \\ {0, 0}) do
@@ -574,7 +581,7 @@ defmodule NaiveDateTime do
   @spec compare(Calendar.naive_datetime, Calendar.naive_datetime) :: :lt | :eq | :gt
   def compare(%{calendar: calendar1} = naive_datetime1, %{calendar: calendar2} = naive_datetime2) do
     if Calendar.compatible_calendars?(calendar1, calendar2) do
-      case {to_rata_die(naive_datetime1), to_rata_die(naive_datetime2)} do
+      case {to_iso_days(naive_datetime1), to_iso_days(naive_datetime2)} do
         {first, second} when first > second -> :gt
         {first, second} when first < second -> :lt
         _ -> :eq
@@ -590,23 +597,36 @@ defmodule NaiveDateTime do
   end
 
   @doc """
-  Converts a `NaiveDateTime` struct from one calendar to another.
+  Converts the given `naive_datetime` from one calendar to another.
 
   If it is not possible to convert unambiguously between the calendars
   (see `Calendar.compatible_calendars?/2`), an `{:error, :incompatible_calendars}` tuple
   is returned.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> NaiveDateTime.convert(~N[2000-01-01 13:30:15], Calendar.Holocene)
+      {:ok, %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
+                           hour: 13, minute: 30, second: 15, microsecond: {0, 0}}}
+
   """
-  @spec convert(Calendar.naive_datetime, Calendar.calendar) :: {:ok, NaiveDateTime.t} | {:error, :incompatible_calendars}
-  def convert(%NaiveDateTime{calendar: calendar} = naive_datetime, calendar) do
-    {:ok, naive_datetime}
+  @spec convert(Calendar.naive_datetime, Calendar.calendar) :: {:ok, t} | {:error, :incompatible_calendars}
+  def convert(%{calendar: calendar, year: year, month: month, day: day,
+                hour: hour, minute: minute, second: second, microsecond: microsecond}, calendar) do
+    {:ok, %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
+                         hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
 
   def convert(%{calendar: ndt_calendar, microsecond: {_, precision}} = naive_datetime, calendar) do
     if Calendar.compatible_calendars?(ndt_calendar, calendar) do
       result_naive_datetime =
         naive_datetime
-        |> to_rata_die
-        |> from_rata_die(calendar, precision)
+        |> to_iso_days
+        |> from_iso_days(calendar, precision)
       {:ok, result_naive_datetime}
     else
       {:error, :incompatible_calendars}
@@ -614,40 +634,49 @@ defmodule NaiveDateTime do
   end
 
   @doc """
-  Converts a NaiveDateTime from one calendar to another.
+  Converts the given `naive_datetime` from one calendar to another.
 
   If it is not possible to convert unambiguously between the calendars
   (see `Calendar.compatible_calendars?/2`), an ArgumentError is raised.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> NaiveDateTime.convert!(~N[2000-01-01 13:30:15], Calendar.Holocene)
+      %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
+                     hour: 13, minute: 30, second: 15, microsecond: {0, 0}}
+
   """
-  @spec convert!(NaiveDateTime.t, Calendar.calendar) :: NaiveDateTime.t
+  @spec convert!(Calendar.naive_datetime, Calendar.calendar) :: t
   def convert!(naive_datetime, calendar) do
     case convert(naive_datetime, calendar) do
       {:ok, value} ->
         value
       {:error, :incompatible_calendars} ->
         raise ArgumentError, "cannot convert #{inspect naive_datetime} to target calendar #{inspect calendar}, reason: #{inspect naive_datetime.calendar} and #{inspect calendar} have different day rollover moments, making this conversion ambiguous"
-      {:error, reason} ->
-        raise ArgumentError, "cannot convert #{inspect naive_datetime} to target calendar #{inspect calendar}, reason: #{inspect reason}"
     end
   end
 
   ## Helpers
 
-  defp to_rata_die(%{calendar: calendar, year: year, month: month, day: day,
+  defp to_iso_days(%{calendar: calendar, year: year, month: month, day: day,
                      hour: hour, minute: minute, second: second, microsecond: microsecond}) do
-    calendar.naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond)
+    calendar.naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond)
   end
 
-  defp from_rata_die(rata_die, calendar, precision) do
+  defp from_iso_days(iso_days, calendar, precision) do
     {year, month, day, hour, minute, second, {microsecond, _}} =
-      calendar.naive_datetime_from_rata_die(rata_die)
-    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second,
-                   microsecond: {microsecond, precision}, calendar: calendar}
+      calendar.naive_datetime_from_iso_days(iso_days)
+    %NaiveDateTime{calendar: calendar, year: year, month: month, day: day,
+                   hour: hour, minute: minute, second: second, microsecond: {microsecond, precision}}
   end
 
   defimpl String.Chars do
     def to_string(%{calendar: calendar, year: year, month: month, day: day,
-                     hour: hour, minute: minute, second: second, microsecond: microsecond}) do
+                    hour: hour, minute: minute, second: second, microsecond: microsecond}) do
       calendar.naive_datetime_to_string(year, month, day, hour, minute, second, microsecond)
     end
   end

lib/elixir/lib/calendar/time.ex

@@ -63,7 +63,9 @@ defmodule Time do
   Expects all values to be integers. Returns `{:ok, time}` if each
   entry fits its appropriate range, returns `{:error, reason}` otherwise.
 
-  Note a time may have 60 seconds in case of leap seconds.
+  Note a time may have 60 seconds in case of leap seconds. Microseconds
+  can also be given with a precision, which must be an integer between
+  0 and 6.
 
   ## Examples
 
@@ -87,9 +89,13 @@ defmodule Time do
       iex> Time.new(23, 59, 59, 1_000_000)
       {:error, :invalid_time}
 
+      # Invalid precision
+      Time.new(23, 59, 59, {999_999, 10})
+      {:error, :invalid_time}
+
   """
   @spec new(Calendar.hour, Calendar.minute, Calendar.second, Calendar.microsecond, Calendar.calendar) ::
-        {:ok, Time.t} | {:error, atom}
+        {:ok, t} | {:error, atom}
   def new(hour, minute, second, microsecond \\ {0, 0}, calendar \\ Calendar.ISO)
 
   def new(hour, minute, second, microsecond, calendar) when is_integer(microsecond) do
@@ -108,7 +114,7 @@ defmodule Time do
   end
 
   @doc """
-  Converts the given time to a string.
+  Converts the given `time` to a string.
 
   ### Examples
 
@@ -252,7 +258,7 @@ defmodule Time do
   end
 
   @doc """
-  Converts a `Time` struct to an Erlang time tuple.
+  Converts given `time` to an Erlang time tuple.
 
   WARNING: Loss of precision may occur, as Erlang time tuples
   only contain hours/minutes/seconds.
@@ -315,7 +321,7 @@ defmodule Time do
   end
 
   @doc """
-  Compares two `Time` structs.
+  Compares two time structs.
 
   Returns `:gt` if first time is later than the second
   and `:lt` for vice versa. If the two times are equal
@@ -325,12 +331,16 @@ defmodule Time do
 
       iex> Time.compare(~T[16:04:16], ~T[16:04:28])
       :lt
+      iex> Time.compare(~T[16:04:16], ~T[16:04:16])
+      :eq
       iex> Time.compare(~T[16:04:16.01], ~T[16:04:16.001])
       :gt
 
   This function can also be used to compare across more
   complex calendar types by considering only the time fields:
 
+      iex> Time.compare(~N[1900-01-01 16:04:16], ~N[2015-01-01 16:04:16])
+      :eq
       iex> Time.compare(~N[2015-01-01 16:04:16], ~N[2015-01-01 16:04:28])
       :lt
       iex> Time.compare(~N[2015-01-01 16:04:16.01], ~N[2000-01-01 16:04:16.001])
@@ -359,14 +369,24 @@ defmodule Time do
   end
 
   @doc """
-  Converts the `Time` struct to a different calendar.
+  Converts given `time` to a different calendar.
 
   Returns `{:ok, time}` if the conversion was successful,
   or `{:error, reason}` if it was not, for some reason.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Time.convert(~T[13:30:15], Calendar.Holocene)
+      {:ok, %Time{calendar: Calendar.Holocene, hour: 13, minute: 30, second: 15, microsecond: {0, 0}}}
+
   """
-  @spec convert(Calendar.time, Calendar.calendar) :: {:ok, Time.t} | {:error, atom}
-  def convert(%Time{calendar: calendar} = time, calendar) do
-    {:ok, time}
+  @spec convert(Calendar.time, Calendar.calendar) :: {:ok, t} | {:error, atom}
+  def convert(%{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}, calendar) do
+    {:ok, %Time{calendar: calendar, hour: hour, minute: minute, second: second, microsecond: microsecond}}
   end
 
   def convert(%{microsecond: {_, precision}} = time, calendar) do
@@ -381,8 +401,18 @@ defmodule Time do
   @doc """
   Similar to `Time.convert/2`, but raises an `ArgumentError`
   if the conversion between the two calendars is not possible.
+
+  ## Examples
+
+  Imagine someone implements `Calendar.Holocene`, a calendar based on the
+  Gregorian calendar that adds exactly 10,000 years to the current Gregorian
+  year:
+
+      iex> Time.convert!(~T[13:30:15], Calendar.Holocene)
+      %Time{calendar: Calendar.Holocene, hour: 13, minute: 30, second: 15, microsecond: {0, 0}}
+
   """
-  @spec convert!(Calendar.time, Calendar.calendar) :: Time.t
+  @spec convert!(Calendar.time, Calendar.calendar) :: t
   def convert!(time, calendar) do
     case convert(time, calendar) do
       {:ok, value} ->
@@ -393,7 +423,14 @@ defmodule Time do
   end
 
   @doc """
-  Returns the difference between two `Time` structs.
+  Returns the difference between two times, considering only the hour, minute
+  second and microsecond.
+
+  As with the `compare/2` function both `Time` structs and other structures
+  containing time can be used. If for instance a `NaiveDateTime` or `DateTime`
+  is passed, only the hour, month, second, and microsecond is considered. Any
+  additional information about a date or time zone is ignored when calculating
+  the difference.
 
   The answer can be returned in any `unit` available from
   `t:System.time_unit/0`. If the first unit is smaller than
@@ -406,18 +443,28 @@ defmodule Time do
 
       iex> Time.diff(~T[00:29:12], ~T[00:29:10])
       2
+
+      # When passing a `NaiveDateTime` the date part is ignored.
+      iex> Time.diff(~N[2017-01-01 00:29:12], ~T[00:29:10])
+      2
+
+      # Two `NaiveDateTime` structs could have big differences in the date
+      # but only the time part is considered.
+      iex> Time.diff(~N[2017-01-01 00:29:12], (~N[1900-02-03 00:29:10]))
+      2
+
       iex> Time.diff(~T[00:29:12], ~T[00:29:10], :microsecond)
       2_000_000
       iex> Time.diff(~T[00:29:10], ~T[00:29:12], :microsecond)
       -2_000_000
 
   """
-  @spec diff(Time.t, Time.t, System.time_unit) :: integer
-  def diff(%Time{} = time1, %Time{} = time2, unit \\ :second) do
+  @spec diff(Calendar.time, Calendar.time, System.time_unit) :: integer
+  def diff(time1, time2, unit \\ :second) do
     fraction1 = to_day_fraction(time1)
     fraction2 = to_day_fraction(time2)
-    Calendar.ISO.rata_die_to_unit({0, fraction1}, unit) -
-      Calendar.ISO.rata_die_to_unit({0, fraction2}, unit)
+    Calendar.ISO.iso_days_to_unit({0, fraction1}, unit) -
+      Calendar.ISO.iso_days_to_unit({0, fraction2}, unit)
   end
 
   ## Helpers

lib/elixir/lib/enum.ex

@@ -314,12 +314,22 @@ defmodule Enum do
     end
   end
 
-  @doc """
-  Shortcut to `chunk(enumerable, count, count)`.
-  """
-  @spec chunk(t, pos_integer) :: [list]
+  # Deprecate on v1.7
+  @doc false
   def chunk(enumerable, count), do: chunk(enumerable, count, count, nil)
 
+  # Deprecate on v1.7
+  @doc false
+  def chunk(enumerable, count, step, leftover \\ nil) do
+    chunk_every(enumerable, count, step, leftover || :discard)
+  end
+
+  @doc """
+  Shortcut to `chunk_every(enumerable, count, count)`.
+  """
+  @spec chunk_every(t, pos_integer) :: [list]
+  def chunk_every(enumerable, count), do: chunk_every(enumerable, count, count, [])
+
   @doc """
   Returns list of lists containing `count` items each, where
   each new chunk starts `step` elements into the enumerable.
@@ -327,55 +337,48 @@ defmodule Enum do
   `step` is optional and, if not passed, defaults to `count`, i.e.
   chunks do not overlap.
 
-  If the final chunk does not have `count` elements to fill the chunk,
-  the final chunk is dropped unless `leftover` is given.
+  If the last chunk does not have `count` elements to fill the chunk,
+  elements are taken from `leftover` to fill in the chunk. If `leftover`
+  does not have enough elements to fill the chunk, then a partial chunk
+  is returned with less than `count` elements.
 
-  If `leftover` is given, elements are taken from `leftover` to fill in
-  the chunk. If `leftover` is passed and does not have enough elements
-  to fill the chunk, then a partial chunk is returned with less than
-  `count` elements. Therefore, an empty list can be given to `leftover`
-  when one simply desires for the last chunk to not be discarded.
+  If `:discard` is given in `leftover`, the last chunk is discarded
+  unless it has exactly `count` elements.
 
   ## Examples
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 2)
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 2)
       [[1, 2], [3, 4], [5, 6]]
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 3, 2)
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard)
       [[1, 2, 3], [3, 4, 5]]
 
-      iex> Enum.chunk([1, 2, 3, 4, 5, 6], 3, 2, [7])
+      iex> Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, [7])
       [[1, 2, 3], [3, 4, 5], [5, 6, 7]]
 
-      iex> Enum.chunk([1, 2, 3, 4], 3, 3, [])
+      iex> Enum.chunk_every([1, 2, 3, 4], 3, 3, [])
       [[1, 2, 3], [4]]
 
-      iex> Enum.chunk([1, 2, 3, 4], 10)
-      []
-
-      iex> Enum.chunk([1, 2, 3, 4], 10, 10, [])
+      iex> Enum.chunk_every([1, 2, 3, 4], 10)
       [[1, 2, 3, 4]]
 
   """
-  @spec chunk(t, pos_integer, pos_integer, t | nil) :: [list]
-  def chunk(enumerable, count, step, leftover \\ nil)
+  @spec chunk_every(t, pos_integer, pos_integer, t | :discard) :: [list]
+  def chunk_every(enumerable, count, step, leftover \\ [])
       when is_integer(count) and count > 0 and is_integer(step) and step > 0 do
-    limit = :erlang.max(count, step)
-
-    {acc, {buffer, i}} =
-      reduce(enumerable, {[], {[], 0}}, R.chunk(count, step, limit))
-
-    if is_nil(leftover) || i == 0 do
-      :lists.reverse(acc)
-    else
-      buffer = :lists.reverse(buffer, take(leftover, count - i))
-      :lists.reverse([buffer | acc])
-    end
+    R.chunk_every(&chunk_while/4, enumerable, count, step, leftover)
   end
 
   @doc """
-  Splits enumerable on every element for which `fun` returns a new
-  value.
+  Chunks the `enum` with fine grained control when every chunk is emitted.
+
+  `chunk_fun` receives the current element and the accumulator and
+  must return `{:cont, element, acc}` to emit the given chunk and
+  continue with accumulator or `{:cont, acc}` to not emit any chunk
+  and continue with the return accumulator.
+
+  `after_fun` is invoked when iteration is done and must also return
+  `{:cont, element, acc}` or `{:cont, acc}`.
 
   Returns a list of lists.
 
@@ -392,15 +395,22 @@ defmodule Enum do
       ...>   [] -> {:cont, []}
       ...>   acc -> {:cont, Enum.reverse(acc), []}
       ...> end
-      iex> Enum.chunk_by(1..10, [], chunk_fun, after_fun)
+      iex> Enum.chunk_while(1..10, [], chunk_fun, after_fun)
       [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
 
   """
-  @spec chunk_by(t, acc,
-                 (element, acc -> {:cont, chunk, acc} | {:cont, acc}),
-                 (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
-  def chunk_by(enum, acc, chunk_fun, after_fun) do
-    {res, acc} = reduce(enum, {[], acc}, R.chunk_by(chunk_fun))
+  @spec chunk_while(t, acc,
+                    (element, acc -> {:cont, chunk, acc} | {:cont, acc} | {:halt, acc}),
+                    (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
+  def chunk_while(enum, acc, chunk_fun, after_fun) do
+    {_, {res, acc}} =
+      Enumerable.reduce(enum, {:cont, {[], acc}}, fn entry, {buffer, acc} ->
+        case chunk_fun.(entry, acc) do
+          {:cont, emit, acc} -> {:cont, {[emit | buffer], acc}}
+          {:cont, acc} -> {:cont, {buffer, acc}}
+          {:halt, acc} -> {:halt, {buffer, acc}}
+        end
+      end)
 
     case after_fun.(acc) do
       {:cont, _acc} -> :lists.reverse(res)
@@ -422,18 +432,7 @@ defmodule Enum do
   """
   @spec chunk_by(t, (element -> any)) :: [list]
   def chunk_by(enumerable, fun) do
-    chunk_by(enumerable, nil, fn
-      entry, nil ->
-        {:cont, {[entry], fun.(entry)}}
-      entry, {acc, value} ->
-        case fun.(entry) do
-          ^value -> {:cont, {[entry | acc], value}}
-          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
-        end
-    end, fn
-      nil -> {:cont, :done}
-      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
-    end)
+    R.chunk_by(&chunk_while/4, enumerable, fun)
   end
 
   @doc """
@@ -3192,6 +3191,13 @@ defimpl Enumerable, for: Function do
   def member?(_function, _value),
     do: {:error, __MODULE__}
 
-  def reduce(function, acc, fun),
+  def reduce(function, acc, fun) when is_function(function, 2),
     do: function.(acc, fun)
+
+  def reduce(function, _acc, _fun) do
+    raise Protocol.UndefinedError,
+      protocol: @protocol,
+      value: function,
+      description: "only anonymous functions of arity 2 are enumerable"
+  end
 end

lib/elixir/lib/exception.ex

@@ -230,7 +230,7 @@ defmodule Exception do
          {_, kind, _, clauses} <- List.keyfind(defs, {function, arity}, 0) do
       clauses =
         for {meta, ex_args, guards, _block} <- clauses do
-          scope = :elixir_erl.definition_scope(meta, kind, function, arity, "nofile")
+          scope = :elixir_erl.definition_scope(meta, "nofile")
           {erl_args, scope} =
             :elixir_erl_clauses.match(&:elixir_erl_pass.translate_args/2, ex_args, scope)
           {args, binding} =
@@ -862,8 +862,7 @@ defmodule UndefinedFunctionError do
   end
 
   defp format_fa({_dist, fun, arity}) do
-    fun = with ":" <> fun <- inspect(fun), do: fun
-    ["      * ", fun, ?/, Integer.to_string(arity), ?\n]
+    ["      * ", Inspect.Function.escape_name(fun), ?/, Integer.to_string(arity), ?\n]
   end
 
   defp exports_for(module) do

lib/elixir/lib/file.ex

@@ -670,15 +670,21 @@ defmodule File do
       File.cp_r "samples", "tmp"
 
       # Same as before, but asks the user how to proceed in case of conflicts
-      File.cp_r "samples", "tmp", fn(source, destination) ->
+      File.cp_r "samples", "tmp", fn source, destination ->
         IO.gets("Overwriting #{destination} by #{source}. Type y to confirm. ") == "y\n"
       end
 
   """
   @spec cp_r(Path.t, Path.t, (Path.t, Path.t -> boolean)) :: {:ok, [binary]} | {:error, posix, binary}
-  def cp_r(source, destination, callback \\ fn(_, _) -> true end) when is_function(callback, 2) do
-    source = IO.chardata_to_string(source)
-    destination = IO.chardata_to_string(destination)
+  def cp_r(source, destination, callback \\ fn _, _ -> true end) when is_function(callback, 2) do
+    source =
+      source
+      |> IO.chardata_to_string()
+      |> assert_no_null_byte!("File.cp_r/3")
+    destination =
+      destination
+      |> IO.chardata_to_string()
+      |> assert_no_null_byte!("File.cp_r/3")
 
     case do_cp_r(source, destination, callback, []) do
       {:error, _, _} = error -> error
@@ -947,7 +953,10 @@ defmodule File do
   """
   @spec rm_rf(Path.t) :: {:ok, [binary]} | {:error, posix, binary}
   def rm_rf(path) do
-    do_rm_rf(IO.chardata_to_string(path), {:ok, []})
+    path
+    |> IO.chardata_to_string()
+    |> assert_no_null_byte!("File.cp_r/3")
+    |> do_rm_rf({:ok, []})
   end
 
   defp do_rm_rf(path, {:ok, _} = entry) do
@@ -1454,6 +1463,15 @@ defmodule File do
 
   @read_ahead_size 64 * 1024
 
+  defp assert_no_null_byte!(binary, operation) do
+    case :binary.match(binary, "\0") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute #{operation} for path with null byte, got: #{inspect binary}"
+      :nomatch ->
+        binary
+    end
+  end
+
   defp normalize_modes([:utf8 | rest], binary?) do
     [encoding: :utf8] ++ normalize_modes(rest, binary?)
   end

lib/elixir/lib/inspect.ex

@@ -356,20 +356,21 @@ defimpl Inspect, for: Map do
     open = color("%" <> name <> "{", :map, opts)
     sep = color(",", :map, opts)
     close = color("}", :map, opts)
-    surround_many(open, map, close, opts, traverse_fun(map), sep)
+    surround_many(open, map, close, opts, traverse_fun(map, opts), sep)
   end
 
-  defp traverse_fun(list) do
+  defp traverse_fun(list, opts) do
     if Inspect.List.keyword?(list) do
       &Inspect.List.keyword/2
     else
-      &to_map/2
+      sep = color(" => ", :map, opts)
+      &to_map(&1, &2, sep)
     end
   end
 
-  defp to_map({key, value}, opts) do
+  defp to_map({key, value}, opts, sep) do
     concat(
-      concat(to_doc(key, opts), " => "),
+      concat(to_doc(key, opts), sep),
       to_doc(value, opts)
     )
   end

lib/elixir/lib/inspect/algebra.ex

@@ -243,7 +243,7 @@ defmodule Inspect.Algebra do
             try do
               Process.put(:inspect_trap, true)
 
-              res = Inspect.Map.inspect(map, opts)
+              res = Inspect.Map.inspect(map, %{opts | syntax_colors: []})
               res = IO.iodata_to_binary(format(res, :infinity))
 
               exception = Inspect.Error.exception(

lib/elixir/lib/kernel.ex

@@ -3289,8 +3289,17 @@ defmodule Kernel do
       result
     end
 
-    {escaped, _} = :elixir_quote.escape(block, false)
-    module_vars  = module_vars(env.vars, 0)
+    escaped =
+      case env do
+        %{function: nil, lexical_tracker: pid} when is_pid(pid) ->
+          integer = Kernel.LexicalTracker.write_cache(pid, block)
+          quote do: Kernel.LexicalTracker.read_cache(unquote(pid), unquote(integer))
+        %{} ->
+          {escaped, _} = :elixir_quote.escape(block, false)
+          escaped
+      end
+
+    module_vars = module_vars(env.vars, 0)
 
     quote do
       unquote(with_alias)
@@ -3554,19 +3563,30 @@ defmodule Kernel do
   end
 
   defp define(kind, call, expr, env) do
-    assert_module_scope(env, kind, 2)
+    module = assert_module_scope(env, kind, 2)
     assert_no_function_scope(env, kind, 2)
+    table = :elixir_module.data_table(module)
 
-    {call, unquoted_call} = :elixir_quote.escape(call, true)
-    {expr, unquoted_expr} = :elixir_quote.escape(expr, true)
+    {escaped_call, unquoted_call} = :elixir_quote.escape(call, true)
+    {escaped_expr, unquoted_expr} = :elixir_quote.escape(expr, true)
+
+    escaped_expr =
+      case unquoted_expr do
+        true ->
+          escaped_expr
+        false ->
+          key = {:cache_def, :erlang.unique_integer()}
+          :ets.insert(table, {key, expr})
+          quote do: :ets.lookup_element(unquote(table), unquote(key), 2)
+      end
 
     # Do not check clauses if any expression was unquoted
     check_clauses = not(unquoted_expr or unquoted_call)
-    pos = :elixir_locals.cache_env(env)
+    pos = :elixir_locals.cache_env(table, env)
 
     quote do
       :elixir_def.store_definition(unquote(kind), unquote(check_clauses),
-                                   unquote(call), unquote(expr), unquote(pos))
+                                   unquote(escaped_call), unquote(escaped_expr), unquote(pos))
     end
   end
 
@@ -4594,7 +4614,7 @@ defmodule Kernel do
   defp assert_module_scope(env, fun, arity) do
     case env.module do
       nil -> raise ArgumentError, "cannot invoke #{fun}/#{arity} outside module"
-      _   -> :ok
+      mod -> mod
     end
   end
 

lib/elixir/lib/kernel/lexical_tracker.ex

@@ -81,6 +81,18 @@ defmodule Kernel.LexicalTracker do
     :gen_server.cast(pid, {:alias_dispatch, module})
   end
 
+  @doc false
+  def write_cache(pid, value) do
+    key = :erlang.unique_integer()
+    :gen_server.cast(pid, {:write_cache, key, value})
+    key
+  end
+
+  @doc false
+  def read_cache(pid, key) do
+    :gen_server.call(pid, {:read_cache, key}, @timeout)
+  end
+
   @doc false
   def collect_unused_imports(pid) do
     unused(pid, :import)
@@ -99,7 +111,7 @@ defmodule Kernel.LexicalTracker do
 
   def init(dest) do
     {:ok, %{directives: %{}, references: %{}, compile: %{},
-            runtime: %{}, dest: dest}}
+            runtime: %{}, dest: dest, cache: %{}}}
   end
 
   @doc false
@@ -124,6 +136,14 @@ defmodule Kernel.LexicalTracker do
     {:reply, state.dest, state}
   end
 
+  def handle_call({:read_cache, key}, _from, %{cache: cache} = state) do
+    {:reply, Map.fetch!(cache, key), state}
+  end
+
+  def handle_cast({:write_cache, key, value}, %{cache: cache} = state) do
+    {:noreply, Map.put(state, :cache, Map.put(cache, key, value))}
+  end
+
   def handle_cast({:remote_reference, module, mode}, state) do
     {:noreply, %{state | references: add_reference(state.references, module, mode)}}
   end

lib/elixir/lib/macro/env.ex

@@ -44,6 +44,9 @@ defmodule Macro.Env do
 
     * `export_vars` - a list keeping all variables to be exported in a
       construct (may be `nil`)
+    * `match_vars` - controls how "new" variables are handled. Inside a
+      match it is a list with all variables in a match. Outside of a match
+      is either `:warn` or `:apply`
     * `prematch_vars` - a list of variables defined before a match (is
       `nil` when not inside a match)
 
@@ -64,6 +67,7 @@ defmodule Macro.Env do
   @type local :: atom | nil
 
   @opaque export_vars :: vars | nil
+  @opaque match_vars :: vars | :warn | :apply
   @opaque prematch_vars :: vars | nil
 
   @type t :: %{__struct__: __MODULE__,
@@ -80,6 +84,7 @@ defmodule Macro.Env do
                context_modules: context_modules,
                vars: vars,
                export_vars: export_vars,
+               match_vars: match_vars,
                prematch_vars: prematch_vars,
                lexical_tracker: lexical_tracker}
 
@@ -99,6 +104,7 @@ defmodule Macro.Env do
       vars: [],
       lexical_tracker: nil,
       export_vars: nil,
+      match_vars: :warn,
       prematch_vars: nil}
   end
 

lib/elixir/lib/map.ex

@@ -548,7 +548,8 @@ defmodule Map do
 
   If `key` is present in `map` with value `value`, `fun` is invoked with
   argument `value` and its result is used as the new value of `key`. If `key` is
-  not present in `map`, `initial` is inserted as the value of `key`.
+  not present in `map`, `initial` is inserted as the value of `key`. The initial
+  value will not be passed through the update function.
 
   ## Examples
 

lib/elixir/lib/module.ex

@@ -1299,9 +1299,9 @@ defmodule Module do
   end
   defp known_callbacks(callbacks) do
     formatted = for {{{name, arity}, kind}, module} <- callbacks do
-      "\n  * " <> Exception.format_mfa(module, name, arity) <> "(#{kind})"
+      "\n  * " <> Exception.format_mfa(module, name, arity) <> " (#{kind})"
     end
-    ". The known callbacks are:\n#{formatted}"
+    ". The known callbacks are:\n#{formatted}\n"
   end
 
   @doc false

lib/elixir/lib/module/locals_tracker.ex

@@ -166,8 +166,8 @@ defmodule Module.LocalsTracker do
   end
 
   defp unreachable(reachable, private) do
-    for {tuple, kind, _, _} <- private,
-        kind == :defmacrop or not :sets.is_element(tuple, reachable),
+    for {tuple, _, _, _} <- private,
+        not :sets.is_element(tuple, reachable),
         do: tuple
   end
 
@@ -210,16 +210,6 @@ defmodule Module.LocalsTracker do
     last
   end
 
-  @doc false
-  def cache_env(pid, env) do
-    :gen_server.call(pid, {:cache_env, env}, @timeout)
-  end
-
-  @doc false
-  def get_cached_env(pid, ref) do
-    :gen_server.call(pid, {:get_cached_env, ref}, @timeout)
-  end
-
   # Stops the gen server
   @doc false
   def stop(pid) do
@@ -231,64 +221,48 @@ defmodule Module.LocalsTracker do
   def init([]) do
     d = :digraph.new([:protected])
     :digraph.add_vertex(d, :local)
-    {:ok, {d, []}}
+    {:ok, d}
   end
 
-  @doc false
-  def handle_call({:cache_env, env}, _from, {d, cache}) do
-    case cache do
-      [{i, ^env} | _] ->
-        {:reply, i, {d, cache}}
-      t ->
-        i = length(t)
-        {:reply, i, {d, [{i, env} | t]}}
-    end
-  end
-
-  def handle_call({:get_cached_env, ref}, _from, {_, cache} = state) do
-    {^ref, env} = :lists.keyfind(ref, 1, cache)
-    {:reply, env, state}
-  end
-
-  def handle_call({:yank, local}, _from, {d, _} = state) do
+  def handle_call({:yank, local}, _from, d) do
     out_vertices = :digraph.out_neighbours(d, local)
     :digraph.del_edges(d, :digraph.out_edges(d, local))
-    {:reply, {[], out_vertices}, state}
+    {:reply, {[], out_vertices}, d}
   end
 
-  def handle_call(:digraph, _from, {d, _} = state) do
-    {:reply, d, state}
+  def handle_call(:digraph, _from, d) do
+    {:reply, d, d}
   end
 
   @doc false
-  def handle_info(_msg, state) do
-    {:noreply, state}
+  def handle_info(_msg, d) do
+    {:noreply, d}
   end
 
-  def handle_cast({:add_local, from, to}, {d, _} = state) do
+  def handle_cast({:add_local, from, to}, d) do
     handle_add_local(d, from, to)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_import, function, module, {name, arity}}, {d, _} = state) do
+  def handle_cast({:add_import, function, module, {name, arity}}, d) do
     handle_import(d, function, module, name, arity)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_definition, kind, tuple}, {d, _} = state) do
+  def handle_cast({:add_definition, kind, tuple}, d) do
     handle_add_definition(d, kind, tuple)
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:add_defaults, kind, {name, arity}, defaults}, {d, _} = state) do
+  def handle_cast({:add_defaults, kind, {name, arity}, defaults}, d) do
     for i <- :lists.seq(arity - defaults, arity - 1) do
       handle_add_definition(d, kind, {name, i})
       handle_add_local(d, {name, i}, {name, arity})
     end
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast({:reattach, _kind, tuple, {in_neigh, out_neigh}}, {d, _} = state) do
+  def handle_cast({:reattach, _kind, tuple, {in_neigh, out_neigh}}, d) do
     for from <- in_neigh do
       :digraph.add_vertex(d, from)
       replace_edge!(d, from, tuple)
@@ -299,11 +273,11 @@ defmodule Module.LocalsTracker do
       replace_edge!(d, tuple, to)
     end
 
-    {:noreply, state}
+    {:noreply, d}
   end
 
-  def handle_cast(:stop, state) do
-    {:stop, :normal, state}
+  def handle_cast(:stop, d) do
+    {:stop, :normal, d}
   end
 
   @doc false

lib/elixir/lib/path.ex

@@ -615,17 +615,19 @@ defmodule Path do
   def wildcard(glob, opts \\ []) do
     mod = if Keyword.get(opts, :match_dot), do: :file, else: Path.Wildcard
     glob
-    |> chardata_to_list()
+    |> chardata_to_list!()
     |> :filelib.wildcard(mod)
     |> Enum.map(&IO.chardata_to_string/1)
   end
 
-  # expand_dot the given path by expanding "..", "." and "~".
-
-  defp chardata_to_list(chardata) do
+  defp chardata_to_list!(chardata) do
     case :unicode.characters_to_list(chardata) do
       result when is_list(result) ->
-        result
+        if 0 in result do
+          raise ArgumentError, "cannot execute Path.wildcard/2 for path with null byte, got: #{inspect chardata}"
+        else
+          result
+        end
 
       {:error, encoded, rest} ->
         raise UnicodeConversionError, encoded: encoded, rest: rest, kind: :invalid
@@ -654,6 +656,8 @@ defmodule Path do
     end
   end
 
+  # expand_dot the given path by expanding "..", "." and "~".
+
   defp expand_dot(<<"/", rest::binary>>),
     do: "/" <> do_expand_dot(rest)
   defp expand_dot(<<letter, ":/", rest::binary>>) when letter in ?a..?z,

lib/elixir/lib/protocol.ex

@@ -254,21 +254,22 @@ defmodule Protocol do
     {:error, :not_a_protocol} |
     {:error, :no_beam_info}
   def consolidate(protocol, types) when is_atom(protocol) do
-    with {:ok, info} <- beam_protocol(protocol),
-         {:ok, code, docs} <- change_debug_info(info, types),
-         do: compile(code, docs)
+    with {:ok, ast_info, chunks_info} <- beam_protocol(protocol),
+         {:ok, code} <- change_debug_info(ast_info, types),
+         do: compile(protocol, code, chunks_info)
   end
 
   defp beam_protocol(protocol) do
-    chunk_ids = [:abstract_code, :attributes, 'ExDc']
+    chunk_ids = [:abstract_code, :attributes, :compile_info, 'ExDc']
     opts = [:allow_missing_chunks]
     case :beam_lib.chunks(beam_file(protocol), chunk_ids, opts) do
       {:ok, {^protocol, [{:abstract_code, {_raw, abstract_code}},
                          {:attributes, attributes},
+                         {:compile_info, compile_info},
                          {'ExDc', docs}]}} ->
         case attributes[:protocol] do
           [fallback_to_any: any] ->
-            {:ok, {protocol, any, abstract_code, docs}}
+            {:ok, {protocol, any, abstract_code}, {compile_info, docs}}
           _ ->
             {:error, :not_a_protocol}
         end
@@ -286,12 +287,12 @@ defmodule Protocol do
 
   # Change the debug information to the optimized
   # impl_for/1 dispatch version.
-  defp change_debug_info({protocol, any, code, docs}, types) do
+  defp change_debug_info({protocol, any, code}, types) do
     types   = if any, do: types, else: List.delete(types, Any)
     all     = [Any] ++ for {_guard, mod} <- __builtin__(), do: mod
     structs = types -- all
     case change_impl_for(code, protocol, types, structs, false, []) do
-      {:ok, ret} -> {:ok, ret, docs}
+      {:ok, ret} -> {:ok, ret}
       other      -> other
     end
   end
@@ -358,9 +359,9 @@ defmodule Protocol do
     change_impl_for(tail, protocol, info, types, protocol?, [head | acc])
   end
 
-  defp change_impl_for([], protocol, _info, _types, protocol?, acc) do
+  defp change_impl_for([], _protocol, _info, _types, protocol?, acc) do
     if protocol? do
-      {:ok, {protocol, Enum.reverse(acc)}}
+      {:ok, Enum.reverse(acc)}
     else
       {:error, :not_a_protocol}
     end
@@ -405,8 +406,9 @@ defmodule Protocol do
   end
 
   # Finally compile the module and emit its bytecode.
-  defp compile({protocol, code}, docs) do
-    opts = if Code.compiler_options[:debug_info], do: [:debug_info], else: []
+  defp compile(protocol, code, {compile_info, docs}) do
+    opts = Keyword.take(compile_info, [:source])
+    opts = if Code.compiler_options[:debug_info], do: [:debug_info | opts], else: opts
     {:ok, ^protocol, binary, _warnings} = :compile.forms(code, [:return | opts])
     {:ok,
       case docs do

lib/elixir/lib/registry.ex

@@ -534,7 +534,8 @@ defmodule Registry do
   """
   @spec match(registry, key, match_pattern :: atom() | tuple(), guards :: list()) :: [{pid, term}]
   def match(registry, key, pattern, guards \\ []) when is_atom(registry) and is_list(guards) do
-    spec = [{{key, {:_, pattern}}, guards, [{:element, 2, :"$_"}]}]
+    guards = [{:"=:=", {:element, 1, :"$_"}, {:const, key}} | guards]
+    spec = [{{:_, {:_, pattern}}, guards, [{:element, 2, :"$_"}]}]
 
     case key_info!(registry) do
       {:unique, partitions, key_ets} ->
@@ -587,7 +588,16 @@ defmodule Registry do
   def keys(registry, pid) when is_atom(registry) and is_pid(pid) do
     {kind, partitions, _, pid_ets, _} = info!(registry)
     {_, pid_ets} = pid_ets || pid_ets!(registry, pid, partitions)
-    keys = safe_lookup_second(pid_ets, pid)
+
+    keys = try do
+      spec = [{{pid, :'$1', :'$2'}, [], [{{:'$1', :'$2'}}]}]
+      :ets.select(pid_ets, spec)
+    catch
+      :error, :badarg -> []
+    end
+
+    # Handle the possibility of fake keys
+    keys = gather_keys(keys, [], false)
 
     cond do
       kind == :unique -> Enum.uniq(keys)
@@ -595,6 +605,19 @@ defmodule Registry do
     end
   end
 
+  defp gather_keys([{key, {_, remaining}} | rest], acc, _fake) do
+    gather_keys(rest, [key | acc], {key, remaining})
+  end
+  defp gather_keys([{key, _} | rest], acc, fake) do
+    gather_keys(rest, [key | acc], fake)
+  end
+  defp gather_keys([], acc, {key, remaining}) do
+    List.duplicate(key, remaining) ++ Enum.reject(acc, & &1 === key)
+  end
+  defp gather_keys([], acc, false) do
+    acc
+  end
+
   @doc """
   Unregisters all entries for the given `key` associated to the current
   process in `registry`.
@@ -652,6 +675,95 @@ defmodule Registry do
     :ok
   end
 
+  @doc """
+  Unregister entries for a given key matching a pattern.
+
+  ## Examples
+
+  For unique registries it can be used to conditionally unregister a key on
+  the basis of whether or not it matches a particular value.
+
+      iex> Registry.start_link(:unique, Registry.UniqueUnregisterMatchTest)
+      iex> Registry.register(Registry.UniqueUnregisterMatchTest, "hello", :world)
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      ["hello"]
+      iex> Registry.unregister_match(Registry.UniqueUnregisterMatchTest, "hello", :foo)
+      :ok
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      ["hello"]
+      iex> Registry.unregister_match(Registry.UniqueUnregisterMatchTest, "hello", :world)
+      :ok
+      iex> Registry.keys(Registry.UniqueUnregisterMatchTest, self())
+      []
+
+  For duplicate registries:
+
+      iex> Registry.start_link(:duplicate, Registry.DuplicateUnregisterMatchTest)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_a)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_b)
+      iex> Registry.register(Registry.DuplicateUnregisterMatchTest, "hello", :world_c)
+      iex> Registry.keys(Registry.DuplicateUnregisterMatchTest, self())
+      ["hello", "hello", "hello"]
+      iex> Registry.unregister_match(Registry.DuplicateUnregisterMatchTest, "hello", :world_a)
+      :ok
+      iex> Registry.keys(Registry.DuplicateUnregisterMatchTest, self())
+      ["hello", "hello"]
+      iex> Registry.lookup(Registry.DuplicateUnregisterMatchTest, "hello")
+      [{self(), :world_b}, {self(), :world_c}]
+  """
+  def unregister_match(registry, key, pattern, guards \\ []) when is_list(guards) do
+    self = self()
+
+    {kind, partitions, key_ets, pid_ets, listeners} = info!(registry)
+    {key_partition, pid_partition} = partitions(kind, key, self, partitions)
+    key_ets = key_ets || key_ets!(registry, key_partition)
+    {pid_server, pid_ets} = pid_ets || pid_ets!(registry, pid_partition)
+
+    # Remove first from the key_ets because in case of crashes
+    # the pid_ets will still be able to clean up. The last step is
+    # to clean if we have no more entries.
+
+    # Here we want to count all entries for this pid under this key, regardless
+    # of pattern.
+    underscore_guard = {:"=:=", {:element, 1, :"$_"}, {:const, key}}
+    total_spec = [{{:_, {self, :_}}, [underscore_guard], [true]}]
+    total = :ets.select_count(key_ets, total_spec)
+
+    # We only want to delete things that match the pattern
+    delete_spec = [{{:_, {self, pattern}}, [underscore_guard | guards] ,[true]}]
+    case :ets.select_delete(key_ets, delete_spec) do
+      # We deleted everything, we can just delete the object
+      ^total ->
+        true = :ets.delete_object(pid_ets, {self, key, key_ets})
+
+        unlink_if_unregistered(pid_server, pid_ets, self)
+
+        for listener <- listeners do
+          Kernel.send(listener, {:unregister, registry, key, self})
+        end
+
+      0 ->
+        :ok
+
+      deleted ->
+        # There are still entries remaining for this pid. delete_object/2 with
+        # duplicate_bag tables will remove every entry, but we only want to
+        # remove those we have deleted. The solution is to introduce a temp_entry
+        # that indicates how many keys WILL be remaining after the delete operation.
+        remaining = total - deleted
+        temp_entry = {self, key, {key_ets, remaining}}
+        true = :ets.insert(pid_ets, temp_entry)
+        true = :ets.delete_object(pid_ets, {self, key, key_ets})
+        real_keys = List.duplicate({self, key, key_ets}, remaining)
+        true = :ets.insert(pid_ets, real_keys)
+        # We've recreated the real remaining key entries, so we can now delete
+        # our temporary entry.
+        true = :ets.delete_object(pid_ets, temp_entry)
+    end
+
+    :ok
+  end
+
   @doc """
   Registers the current process under the given `key` in `registry`.
 
@@ -976,6 +1088,12 @@ defmodule Registry.Partition do
   def handle_info({:EXIT, pid, _reason}, ets) do
     entries = :ets.take(ets, pid)
     for {_pid, key, key_ets} <- entries do
+      key_ets = case key_ets do
+        # In case the fake key ets is being used. See unregister_match/2.
+        {key_ets, _} -> key_ets
+        _ -> key_ets
+      end
+
       try do
         :ets.match_delete(key_ets, {key, {pid, :_}})
       catch

lib/elixir/lib/stream.ex

@@ -121,63 +121,57 @@ defmodule Stream do
 
   ## Transformers
 
-  @doc """
-  Shortcut to `chunk(enum, n, n)`.
-  """
-  @spec chunk(Enumerable.t, pos_integer) :: Enumerable.t
+  # Deprecate on v1.7
+  @doc false
   def chunk(enum, n), do: chunk(enum, n, n, nil)
 
+  # Deprecate on v1.7
+  @doc false
+  def chunk(enum, n, step, leftover \\ nil)
+      when is_integer(n) and n > 0 and is_integer(step) and step > 0 do
+    chunk_every(enum, n, step, leftover || :discard)
+  end
+
   @doc """
-  Streams the enumerable in chunks, containing `n` items each, where
-  each new chunk starts `step` elements into the enumerable.
+  Shortcut to `chunk_every(enum, count, count)`.
+  """
+  @spec chunk_every(Enumerable.t, pos_integer) :: Enumerable.t
+  def chunk_every(enum, count), do: chunk_every(enum, count, count, [])
+
+  @doc """
+  Streams the enumerable in chunks, containing `count` items each,
+  where each new chunk starts `step` elements into the enumerable.
 
   `step` is optional and, if not passed, defaults to `count`, i.e.
   chunks do not overlap.
 
-  If the final chunk does not have `count` elements to fill the chunk,
-  the final chunk is dropped unless `leftover` is given.
+  If the last chunk does not have `count` elements to fill the chunk,
+  elements are taken from `leftover` to fill in the chunk. If `leftover`
+  does not have enough elements to fill the chunk, then a partial chunk
+  is returned with less than `count` elements.
 
-  If `leftover` is given, elements are taken from `leftover` to fill in
-  the chunk. If `leftover` is passed and does not have enough elements
-  to fill the chunk, then a partial chunk is returned with less than
-  `count` elements. Therefore, an empty list can be given to `leftover`
-  when one simply desires for the last chunk to not be discarded.
+  If `:discard` is given in `leftover`, the last chunk is discarded
+  unless it has exactly `count` elements.
 
   ## Examples
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 2) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 2) |> Enum.to_list
       [[1, 2], [3, 4], [5, 6]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 2) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard) |> Enum.to_list
       [[1, 2, 3], [3, 4, 5]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 2, [7]) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, [7]) |> Enum.to_list
       [[1, 2, 3], [3, 4, 5], [5, 6, 7]]
 
-      iex> Stream.chunk([1, 2, 3, 4, 5, 6], 3, 3, []) |> Enum.to_list
+      iex> Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 3, []) |> Enum.to_list
       [[1, 2, 3], [4, 5, 6]]
 
   """
-  @spec chunk(Enumerable.t, pos_integer, pos_integer, Enumerable.t | nil) :: Enumerable.t
-  def chunk(enum, n, step, leftover \\ nil)
-      when is_integer(n) and n > 0 and is_integer(step) and step > 0 do
-    limit = :erlang.max(n, step)
-    if is_nil(leftover) do
-      lazy enum, {[], 0}, fn(f1) -> R.chunk(n, step, limit, f1) end
-    else
-      lazy enum, {[], 0},
-           fn(f1) -> R.chunk(n, step, limit, f1) end,
-           &do_chunk(&1, n, leftover, &2)
-     end
-  end
-
-  defp do_chunk(acc(_, {_, 0}, _) = acc, _, _, _) do
-    {:cont, acc}
-  end
-
-  defp do_chunk(acc(h, {buffer, count} = old, t), n, leftover, f1) do
-    buffer = :lists.reverse(buffer, Enum.take(leftover, n - count))
-    next_with_acc(f1, buffer, h, old, t)
+  @spec chunk_every(Enumerable.t, pos_integer, pos_integer, Enumerable.t | :discard) :: Enumerable.t
+  def chunk_every(enum, count, step, leftover \\ [])
+      when is_integer(count) and count > 0 and is_integer(step) and step > 0 do
+    R.chunk_every(&chunk_while/4, enum, count, step, leftover)
   end
 
   @doc """
@@ -194,18 +188,7 @@ defmodule Stream do
   """
   @spec chunk_by(Enumerable.t, (element -> any)) :: Enumerable.t
   def chunk_by(enum, fun) do
-    chunk_by(enum, nil, fn
-      entry, nil ->
-        {:cont, {[entry], fun.(entry)}}
-      entry, {acc, value} ->
-        case fun.(entry) do
-          ^value -> {:cont, {[entry | acc], value}}
-          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
-        end
-    end, fn
-      nil -> {:cont, :done}
-      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
-    end)
+    R.chunk_by(&chunk_while/4, enum, fun)
   end
 
   @doc """
@@ -232,21 +215,21 @@ defmodule Stream do
       ...>   [] -> {:cont, []}
       ...>   acc -> {:cont, Enum.reverse(acc), []}
       ...> end
-      iex> stream = Stream.chunk_by(1..10, [], chunk_fun, after_fun)
+      iex> stream = Stream.chunk_while(1..10, [], chunk_fun, after_fun)
       iex> Enum.to_list(stream)
       [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
 
   """
-  @spec chunk_by(Enumerable.t, acc,
-                 (element, acc -> {:cont, chunk, acc} | {:cont, acc}),
-                 (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
-  def chunk_by(enum, acc, chunk_fun, after_fun) do
+  @spec chunk_while(Enumerable.t, acc,
+                    (element, acc -> {:cont, chunk, acc} | {:cont, acc} | {:halt, acc}),
+                    (acc -> {:cont, chunk, acc} | {:cont, acc})) :: Enumerable.t when chunk: any
+  def chunk_while(enum, acc, chunk_fun, after_fun) do
     lazy enum, acc,
-         fn(f1) -> R.chunk_by(chunk_fun, f1) end,
-         &after_chunk_by(&1, &2, after_fun)
+         fn(f1) -> R.chunk_while(chunk_fun, f1) end,
+         &after_chunk_while(&1, &2, after_fun)
   end
 
-  defp after_chunk_by(acc(h, acc, t), f1, after_fun) do
+  defp after_chunk_while(acc(h, acc, t), f1, after_fun) do
     case after_fun.(acc) do
       {:cont, emit, acc} -> next_with_acc(f1, emit, h, acc, t)
       {:cont, acc} -> {:cont, acc(h, acc, t)}
@@ -1155,6 +1138,10 @@ defmodule Stream do
   @spec cycle(Enumerable.t) :: Enumerable.t
   def cycle(enumerable)
 
+  def cycle([]) do
+    raise ArgumentError, "cannot cycle over empty enumerable"
+  end
+
   def cycle(enumerable) when is_list(enumerable) do
     unfold {enumerable, enumerable}, fn
       {source, [h | t]}      -> {h, {source, t}}
@@ -1185,6 +1172,8 @@ defmodule Stream do
       {:stream_cycle, acc} ->
         {:halted, acc}
     else
+      {state, []} when state in [:done, :halted] ->
+        raise ArgumentError, "cannot cycle over empty enumerable"
       {state, acc} when state in [:done, :halted] ->
         do_cycle(cycle, cycle, {:cont, acc})
       {:suspended, acc, continuation} ->

lib/elixir/lib/stream/reducers.ex

@@ -1,36 +1,57 @@
 defmodule Stream.Reducers do
-  # Collection of reducers shared by Enum and Stream.
+  # Collection of reducers and utilities shared by Enum and Stream.
   @moduledoc false
 
-  defmacro chunk(amount, step, limit, fun \\ nil) do
-    quote do
-      fn entry, acc(head, {buffer, count}, tail) ->
-        buffer = [entry | buffer]
-        count = count + 1
+  def chunk_every(chunk_by, enumerable, count, step, leftover) do
+    limit = :erlang.max(count, step)
+    chunk_by.(enumerable, {[], 0}, fn entry, {acc_buffer, acc_count} ->
+      acc_buffer = [entry | acc_buffer]
+      acc_count = acc_count + 1
 
-        new_state =
-          if count >= unquote(limit) do
-            left = count - unquote(step)
-            {Enum.take(buffer, left), left}
-          else
-            {buffer, count}
-          end
-
-        if count == unquote(amount) do
-          next_with_acc(unquote(fun), :lists.reverse(buffer), head, new_state, tail)
+      new_state =
+        if acc_count >= limit do
+          remaining = acc_count - step
+          {Enum.take(acc_buffer, remaining), remaining}
         else
-          skip(acc(head, new_state, tail))
+          {acc_buffer, acc_count}
         end
+
+      if acc_count == count do
+        {:cont, :lists.reverse(acc_buffer), new_state}
+      else
+        {:cont, new_state}
       end
-    end
+    end, fn {acc_buffer, acc_count} ->
+      if leftover == :discard or acc_count == 0 do
+        {:cont, []}
+      else
+        {:cont, :lists.reverse(acc_buffer, Enum.take(leftover, count - acc_count)), []}
+      end
+    end)
   end
 
-  defmacro chunk_by(callback, fun \\ nil) do
+  def chunk_by(chunk_by, enumerable, fun) do
+    chunk_by.(enumerable, nil, fn
+      entry, nil ->
+        {:cont, {[entry], fun.(entry)}}
+      entry, {acc, value} ->
+        case fun.(entry) do
+          ^value -> {:cont, {[entry | acc], value}}
+          new_value -> {:cont, :lists.reverse(acc), {[entry], new_value}}
+        end
+    end, fn
+      nil -> {:cont, :done}
+      {acc, _value} -> {:cont, :lists.reverse(acc), :done}
+    end)
+  end
+
+  defmacro chunk_while(callback, fun \\ nil) do
     quote do
       fn entry, acc(head, acc, tail) ->
         case unquote(callback).(entry, acc) do
           {:cont, emit, acc} -> next_with_acc(unquote(fun), emit, head, acc, tail)
           {:cont, acc} -> skip(acc(head, acc, tail))
+          {:halt, acc} -> {:halt, acc(head, acc, tail)}
         end
       end
     end
@@ -38,7 +59,7 @@ defmodule Stream.Reducers do
 
   defmacro dedup(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, prev, tail) = acc) ->
+      fn entry, acc(head, prev, tail) = acc ->
         value = unquote(callback).(entry)
         case prev do
           {:value, ^value} -> skip(acc)
@@ -84,7 +105,7 @@ defmodule Stream.Reducers do
 
   defmacro filter(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -96,7 +117,7 @@ defmodule Stream.Reducers do
 
   defmacro filter_map(filter, mapper, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(filter).(entry) do
           next(unquote(fun), unquote(mapper).(entry), acc)
         else
@@ -108,7 +129,7 @@ defmodule Stream.Reducers do
 
   defmacro map(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         next(unquote(fun), unquote(callback).(entry), acc)
       end
     end
@@ -127,7 +148,7 @@ defmodule Stream.Reducers do
 
   defmacro reject(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         unless unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -151,7 +172,7 @@ defmodule Stream.Reducers do
 
   defmacro scan3(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, acc, tail)) ->
+      fn entry, acc(head, acc, tail) ->
         value = unquote(callback).(entry, acc)
         next_with_acc(unquote(fun), value, head, value, tail)
       end
@@ -160,7 +181,7 @@ defmodule Stream.Reducers do
 
   defmacro take(fun \\ nil) do
     quote do
-      fn(entry, acc(head, curr, tail) = original) ->
+      fn entry, acc(head, curr, tail) = original ->
         case curr do
           0 ->
             {:halt, original}
@@ -187,7 +208,7 @@ defmodule Stream.Reducers do
 
   defmacro take_while(callback, fun \\ nil) do
     quote do
-      fn(entry, acc) ->
+      fn entry, acc ->
         if unquote(callback).(entry) do
           next(unquote(fun), entry, acc)
         else
@@ -199,7 +220,7 @@ defmodule Stream.Reducers do
 
   defmacro uniq_by(callback, fun \\ nil) do
     quote do
-      fn(entry, acc(head, prev, tail) = original) ->
+      fn entry, acc(head, prev, tail) = original ->
         value = unquote(callback).(entry)
         if Map.has_key?(prev, value) do
           skip(original)
@@ -212,7 +233,7 @@ defmodule Stream.Reducers do
 
   defmacro with_index(fun \\ nil) do
     quote do
-      fn(entry, acc(head, counter, tail)) ->
+      fn entry, acc(head, counter, tail) ->
         next_with_acc(unquote(fun), {entry, counter}, head, counter + 1, tail)
       end
     end

lib/elixir/lib/string_io.ex

@@ -274,57 +274,64 @@ defmodule StringIO do
 
   ## get_until
 
-  defp get_until(encoding, prompt, mod, fun, args,
-                 %{input: input, output: output, capture_prompt: capture_prompt} = s) do
-    case :unicode.characters_to_list(input, encoding) do
-      {:error, _, _} ->
-        {:error, s}
-      {:incomplete, _, _} ->
-        {:error, s}
-      chars ->
-        {result, input, count} = do_get_until(chars, encoding, mod, fun, args)
-
+  defp get_until(encoding, prompt, mod, fun, args, %{input: input} = s) do
+    case do_get_until(input, encoding, mod, fun, args) do
+      {result, input, count} ->
         input =
-          case input do
-            :eof -> ""
-            _ -> :unicode.characters_to_binary(input, encoding)
-          end
+         case input do
+           :eof -> ""
+           _ -> list_to_binary(input, encoding)
+         end
 
-        s =
-          if capture_prompt do
-            %{s | output: <<output::binary, :binary.copy(IO.chardata_to_string(prompt), count)::binary>>}
-          else
-            s
-          end
+        {get_until_result(result, encoding), state_after_read(s, input, prompt, count)}
 
-        {result, %{s | input: input}}
+      :error ->
+        {:error, s}
     end
   end
 
   defp do_get_until(chars, encoding, mod, fun, args, continuation \\ [], count \\ 0)
 
-  defp do_get_until('', encoding, mod, fun, args, continuation, count) do
+  defp do_get_until("", encoding, mod, fun, args, continuation, count) do
     case apply(mod, fun, [continuation, :eof | args]) do
       {:done, result, rest} ->
         {result, rest, count + 1}
       {:more, next_continuation} ->
-        do_get_until('', encoding, mod, fun, args, next_continuation, count + 1)
+        do_get_until("", encoding, mod, fun, args, next_continuation, count + 1)
     end
   end
 
   defp do_get_until(chars, encoding, mod, fun, args, continuation, count) do
-    {line, rest} = collect_line(chars)
+    case bytes_until_eol(chars, encoding, 0) do
+      {r, c} when r in [:split, :replace_split] ->
+        {line, rest} = :erlang.split_binary(chars, c)
 
-    case apply(mod, fun, [continuation, line | args]) do
-      {:done, result, :eof} ->
-        {result, rest, count + 1}
-      {:done, result, extra} ->
-        {result, extra ++ rest, count + 1}
-      {:more, next_continuation} ->
-        do_get_until(rest, encoding, mod, fun, args, next_continuation, count + 1)
+        case apply(mod, fun, [continuation, binary_to_list(line, encoding) | args]) do
+          {:done, result, :eof} ->
+            {result, rest, count + 1}
+          {:done, result, extra} ->
+            {result, extra ++ binary_to_list(rest, encoding), count + 1}
+          {:more, next_continuation} ->
+            do_get_until(rest, encoding, mod, fun, args, next_continuation, count + 1)
+        end
+
+      :error ->
+        :error
     end
   end
 
+  defp binary_to_list(l, _)        when is_list(l),   do: l
+  defp binary_to_list(b, :unicode) when is_binary(b), do: to_charlist(b)
+  defp binary_to_list(b, :latin1)  when is_binary(b), do: :binary.bin_to_list(b)
+  defp list_to_binary(b, _)        when is_binary(b), do: b
+  defp list_to_binary(l, :unicode) when is_list(l),   do: to_string(l)
+  defp list_to_binary(l, :latin1)  when is_list(l),   do: :binary.list_to_bin(l)
+
+  # From http://erlang.org/doc/apps/stdlib/io_protocol.html: Result can be any
+  # Erlang term, but if it is a list(), the I/O server can convert it to a binary().
+  defp get_until_result(l, encoding) when is_list(l), do: list_to_binary(l, encoding)
+  defp get_until_result(other, _), do: other
+
   ## io_requests
 
   defp io_requests([r | rs], {:ok, s}) do
@@ -337,12 +344,14 @@ defmodule StringIO do
 
   ## helpers
 
-  defp state_after_read(%{capture_prompt: false} = s, remainder, _prompt) do
+  defp state_after_read(state, remainder, prompt, count \\ 1)
+
+  defp state_after_read(%{capture_prompt: false} = s, remainder, _prompt, _count) do
     %{s | input: remainder}
   end
 
-  defp state_after_read(%{capture_prompt: true, output: output} = s, remainder, prompt) do
-    %{s | input: remainder, output: <<output::binary, IO.chardata_to_string(prompt)::binary>>}
+  defp state_after_read(%{capture_prompt: true, output: output} = s, remainder, prompt, count) do
+    %{s | input: remainder, output: <<output::binary, :binary.copy(IO.chardata_to_string(prompt), count)::binary>>}
   end
 
   defp bytes_until_eol("", _, count), do: {:split, count}
@@ -359,26 +368,6 @@ defmodule StringIO do
 
   defp bytes_until_eol(<<_::binary>>, _, _), do: :error
 
-  defp collect_line(chars) do
-    collect_line(chars, [])
-  end
-
-  defp collect_line([], stack) do
-    {:lists.reverse(stack), []}
-  end
-
-  defp collect_line([?\r, ?\n | rest], stack) do
-    {:lists.reverse([?\n | stack]), rest}
-  end
-
-  defp collect_line([?\n | rest], stack) do
-    {:lists.reverse([?\n | stack]), rest}
-  end
-
-  defp collect_line([h | t], stack) do
-    collect_line(t, [h | stack])
-  end
-
   defp io_reply(from, reply_as, reply) do
     send from, {:io_reply, reply_as, reply}
   end

lib/elixir/lib/system.ex

@@ -339,6 +339,8 @@ defmodule System do
   """
   @spec find_executable(binary) :: binary | nil
   def find_executable(program) when is_binary(program) do
+    assert_no_null_byte!(program, "System.find_executable/1")
+
     case :os.find_executable(String.to_charlist(program)) do
       false -> nil
       other -> List.to_string(other)
@@ -393,8 +395,13 @@ defmodule System do
   """
   @spec put_env(binary, binary) :: :ok
   def put_env(varname, value) when is_binary(varname) and is_binary(value) do
-   :os.putenv String.to_charlist(varname), String.to_charlist(value)
-   :ok
+    case :binary.match(varname, "=") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute System.put_env/2 for key with \"=\", got: #{inspect varname}"
+      :nomatch ->
+        :os.putenv String.to_charlist(varname), String.to_charlist(value)
+        :ok
+    end
   end
 
   @doc """
@@ -588,6 +595,7 @@ defmodule System do
   @spec cmd(binary, [binary], keyword) ::
         {Collectable.t, exit_status :: non_neg_integer}
   def cmd(command, args, opts \\ []) when is_binary(command) and is_list(args) do
+    assert_no_null_byte!(command, "System.cmd/3")
     cmd = String.to_charlist(command)
 
     cmd =
@@ -833,6 +841,15 @@ defmodule System do
     :erlang.unique_integer(modifiers)
   end
 
+  defp assert_no_null_byte!(binary, operation) do
+    case :binary.match(binary, "\0") do
+      {_, _} ->
+        raise ArgumentError, "cannot execute #{operation} for program with null byte, got: #{inspect binary}"
+      :nomatch ->
+        binary
+    end
+  end
+
   defp normalize_time_unit(:native),
     do: :native
 

lib/elixir/pages/Deprecations.md

@@ -33,6 +33,7 @@ Deprecated feature                               | Deprecated in | Replaced by (
 `:char_lists` key in `t:Inspect.Opts.t/0` type   | [v1.5]        | `:charlists` (v1.3)
 `char_list/0` type                               | [v1.5]        | `charlist/0` type (v1.3)
 `@compile {:parse_transform, _}` in `Module`     | [v1.5]        | *None*
+EEx: `<%=` in middle and end expressions         | [v1.5]        | Use `<%` (= is allowed only on start expressions) (v1.0)
 `Access.key/1`                                   | [v1.4]        | `Access.key/2` (v1.3)
 `Behaviour` module                               | [v1.4]        | `@callback` (v1.0)
 `Enum.uniq/2`                                    | [v1.4]        | `Enum.uniq_by/2` (v1.2)
@@ -68,4 +69,4 @@ Empty string in `String.starts_with?/2`, `String.ends_with?/2`, `String.contains
 [v1.2]: https://github.com/elixir-lang/elixir/blob/v1.2/CHANGELOG.md#changelog-for-elixir-v12
 [v1.3]: https://github.com/elixir-lang/elixir/blob/v1.3/CHANGELOG.md#4-deprecations
 [v1.4]: https://github.com/elixir-lang/elixir/blob/v1.4/CHANGELOG.md#4-deprecations
-[v1.5]: https://github.com/elixir-lang/elixir/blob/master/CHANGELOG.md#4-deprecations
+[v1.5]: https://github.com/elixir-lang/elixir/blob/v1.5/CHANGELOG.md#4-deprecations

lib/elixir/pages/Guards.md

@@ -104,14 +104,14 @@ Other constructs are `for`, `with`, `try`/`rescue`/`catch`/`else`/, and the `mat
 
 ## Failing guards
 
-Errors in guards do not result in a runtime error, but in the erroring guard fail. For example, the `length/1` function only works with lists, and if we use it on anything else it fails:
+Errors in guards do not result in runtime errors, but in guards failing. For example, the `length/1` function only works with lists. If we use it with anything else, a runtime error is raised:
 
 ```elixir
 iex> length("hello")
 ** (ArgumentError) argument error
 ```
 
-However, when used in guards, it simply makes the corresponding clause fail (i.e., not match):
+However, when used in guards, the corresponding clause simply fails to match:
 
 ```elixir
 iex> case "hello" do
@@ -123,7 +123,7 @@ iex> case "hello" do
 :length_failed
 ```
 
-In many cases, we can take advantage of this: in the code above, for example, we can use `length/1` to both check that the given thing is a list *and* check some properties of its length (instead of using `is_list(something) and length(something) > 0`).
+In many cases, we can take advantage of this. In the code above, we used `length/1` to both check that the given thing is a list *and* check some properties of its length (instead of using `is_list(something) and length(something) > 0`).
 
 ## Expressions in guard clauses
 

lib/elixir/src/elixir.erl

@@ -183,8 +183,14 @@ env_for_eval(Env, Opts) ->
     false -> nil
   end,
 
+  FA = case lists:keyfind(function, 1, Opts) of
+    {function, {Function, Arity}} when is_atom(Function), is_integer(Arity) -> {Function, Arity};
+    {function, nil} -> nil;
+    false -> nil
+  end,
+
   Env#{
-    file := File, module := Module,
+    file := File, module := Module, function := FA,
     macros := Macros, functions := Functions,
     requires := Requires, aliases := Aliases, line := Line
   }.

lib/elixir/src/elixir.hrl

@@ -5,15 +5,13 @@
 -define(var_context, ?MODULE).
 
 -record(elixir_erl, {
-  def=nil,                 %% a tuple with the current definition {def | ..., name, arity}
   context=nil,             %% can be match, guards or nil
   extra=nil,               %% extra information about the context, like pin_guard and map_key
   caller=false,            %% when true, it means caller was invoked
   vars=#{},                %% a map of defined variables and their alias
   backup_vars=nil,         %% a copy of vars to be used on ^var
-  match_vars=nil,          %% a set of all variables defined in a particular match
   export_vars=nil,         %% a dict of all variables defined in a particular clause
-  extra_guards=nil,        %% extra guards from args expansion
+  extra_guards=[],         %% extra guards from args expansion
   counter=#{},             %% a map counting the variables defined
   file=(<<"nofile">>)      %% the current scope filename
 }).

lib/elixir/src/elixir_clauses.erl

@@ -2,20 +2,20 @@
 %% fn, receive and friends. try is handled in elixir_try.
 -module(elixir_clauses).
 -export([match/3, clause/5, def/2, head/2,
-         'case'/3, 'receive'/3, 'try'/3, 'cond'/3,
+         'case'/3, 'receive'/3, 'try'/3, 'cond'/3, with/3,
          format_error/1]).
 -import(elixir_errors, [form_error/4]).
 -include("elixir.hrl").
 
 match(Fun, Expr, #{context := match} = E) ->
   Fun(Expr, E);
-match(Fun, Expr, #{context := Context, prematch_vars := nil, vars := Vars} = E) ->
-  {EExpr, EE} = Fun(Expr, E#{context := match, prematch_vars := Vars}),
-  {EExpr, EE#{context := Context, prematch_vars := nil}}.
+match(Fun, Expr, #{context := Context, match_vars := Match, prematch_vars := nil, vars := Vars} = E) ->
+  {EExpr, EE} = Fun(Expr, E#{context := match, match_vars := [], prematch_vars := Vars}),
+  {EExpr, EE#{context := Context, match_vars := Match, prematch_vars := nil}}.
 
-def({Meta, Args, Guards, Body}, E) ->
-  {EArgs, EA}   = elixir_expand:expand(Args, E#{context := match}),
-  {EGuards, EG} = guard(Guards, EA#{context := guard}),
+def({Meta, Args, Guards, Body}, #{match_vars := Match} = E) ->
+  {EArgs, EA}   = elixir_expand:expand(Args, E#{context := match, match_vars := []}),
+  {EGuards, EG} = guard(Guards, EA#{context := guard, match_vars := Match}),
   {EBody, _}    = elixir_expand:expand(Body, EG#{context := ?key(E, context)}),
   {Meta, EArgs, EGuards, EBody}.
 
@@ -112,6 +112,66 @@ expand_receive(Meta, {'after', _}, _Acc, E) ->
 expand_receive(Meta, {Key, _}, _Acc, E) ->
   form_error(Meta, ?key(E, file), ?MODULE, {unexpected_option, 'receive', Key}).
 
+%% With
+
+with(Meta, Args, E) ->
+  {Exprs, Opts0} =
+    case elixir_utils:split_last(Args) of
+      {_, LastArg} = SplitResult when is_list(LastArg) ->
+        SplitResult;
+      _ ->
+        {Args, []}
+    end,
+
+  {EExprs, {EE, HasMatch}} = lists:mapfoldl(fun expand_with/2, {E, false}, Exprs),
+  {EDo, Opts1} = expand_with_do(Meta, Opts0, EE),
+  {EOpts, Opts2} = expand_with_else(Meta, Opts1, E, HasMatch),
+
+  case Opts2 of
+    [{Key, _} | _] ->
+      form_error(Meta, ?key(E, file), elixir_clauses, {unexpected_option, with, Key});
+    [] ->
+      ok
+  end,
+
+  {{with, Meta, EExprs ++ [[{do, EDo} | EOpts]]}, E}.
+
+expand_with({'<-', Meta, [{Name, _, Ctx}, _] = Args}, Acc) when is_atom(Name), is_atom(Ctx) ->
+  expand_with({'=', Meta, Args}, Acc);
+expand_with({'<-', Meta, [Left, Right]}, {E, _HasMatch}) ->
+  {ERight, ER} = elixir_expand:expand(Right, E),
+  {[ELeft], EL}  = head([Left], E),
+  {{'<-', Meta, [ELeft, ERight]}, {elixir_env:mergev(EL, ER), true}};
+expand_with(Expr, {E, HasMatch}) ->
+  {EExpr, EE} = elixir_expand:expand(Expr, E),
+  {EExpr, {EE, HasMatch}}.
+
+expand_with_do(Meta, Opts, E) ->
+  case lists:keytake(do, 1, Opts) of
+    {value, {do, Expr}, RestOpts} ->
+      {EExpr, _} = elixir_expand:expand(Expr, E),
+      {EExpr, RestOpts};
+    false ->
+      form_error(Meta, ?key(E, file), elixir_expand, {missing_option, 'with', [do]})
+  end.
+
+expand_with_else(Meta, Opts, E, HasMatch) ->
+  case lists:keytake(else, 1, Opts) of
+    {value, Pair, RestOpts} ->
+      if
+        HasMatch ->
+          ok;
+        true ->
+          Message = "\"else\" clauses will never match because all patterns in \"with\" will always match",
+          elixir_errors:warn(?line(Meta), ?key(E, file), Message)
+      end,
+      Fun = expand_one(Meta, 'with', 'else', fun head/2),
+      EPair = expand_without_export(Meta, 'with', Fun, Pair, E),
+      {[EPair], RestOpts};
+    false ->
+      {[], Opts}
+  end.
+
 %% Try
 
 'try'(Meta, [], E) ->
@@ -168,6 +228,11 @@ expand_rescue(Meta, _, E) ->
 expand_rescue({Name, _, Atom} = Var, E) when is_atom(Name), is_atom(Atom) ->
   match(fun elixir_expand:expand/2, Var, E);
 
+%% rescue var in _ => rescue var
+expand_rescue({in, _, [{Name, _, VarContext} = Var, {'_', _, UnderscoreContext}]}, E)
+    when is_atom(Name), is_atom(VarContext), is_atom(UnderscoreContext) ->
+  expand_rescue(Var, E);
+
 %% rescue var in [Exprs]
 expand_rescue({in, Meta, [Left, Right]}, E) ->
   {ELeft, EL}  = match(fun elixir_expand:expand/2, Left, E),

lib/elixir/src/elixir_def.erl

@@ -22,7 +22,7 @@ local_for(Module, Name, Arity, Kinds) ->
   of
     {[{_, Kind, Meta, File, _, _}], Clauses} ->
       case (Kinds == all) orelse (lists:member(Kind, Kinds)) of
-        true -> elixir_erl:definition_to_anonymous(File, Module, Tuple, Kind, Meta,
+        true -> elixir_erl:definition_to_anonymous(File, Module, Kind, Meta,
                                                    [Clause || {_, Clause} <- Clauses]);
         false -> false
       end;
@@ -212,7 +212,7 @@ unpack_defaults(Kind, Meta, Name, Args, E) ->
 unpack_defaults(Kind, Meta, Name, [{'\\\\', DefaultMeta, [Expr, _]} | T] = List, Acc, Clauses) ->
   Base = match_defaults(Acc, length(Acc), []),
   {Args, Invoke} = extract_defaults(List, length(Base), [], []),
-  Clause = {Meta, Base ++ Args, [], {super, DefaultMeta, Base ++ Invoke}},
+  Clause = {Meta, Base ++ Args, [], {super, DefaultMeta, [{Kind, Name} | Base] ++ Invoke}},
   unpack_defaults(Kind, Meta, Name, T, [Expr | Acc], [Clause | Clauses]);
 unpack_defaults(Kind, Meta, Name, [H | T], Acc, Clauses) ->
   unpack_defaults(Kind, Meta, Name, T, [H | Acc], Clauses);

lib/elixir/src/elixir_env.erl

@@ -19,10 +19,13 @@ new() ->
     lexical_tracker => nil,                %% holds the lexical tracker PID
     vars => [],                            %% a set of defined variables
     export_vars => nil,                    %% a set of variables to be exported in some constructs
-    prematch_vars => nil}.                 %% a set of variables defined before the current match
+    prematch_vars => nil,                  %% a set of variables defined before the current match
+    match_vars => warn}.                   %% handling of new variables
 
 linify({Line, Env}) ->
-  Env#{line := Line}.
+  Env#{line := Line};
+linify(#{} = Env) ->
+  Env.
 
 env_to_scope(#{file := File, context := Context}) ->
   #elixir_erl{file=File, context=Context}.

lib/elixir/src/elixir_erl.erl

@@ -1,8 +1,8 @@
 %% Compiler backend to Erlang.
 -module(elixir_erl).
--export([elixir_to_erl/1, definition_to_anonymous/6, compile/1,
+-export([elixir_to_erl/1, definition_to_anonymous/5, compile/1,
          get_ann/1, remote/4, add_beam_chunks/2, debug_info/4,
-         definition_scope/5]).
+         definition_scope/2]).
 -include("elixir.hrl").
 
 %% TODO: Remove extra chunk functionality when OTP 20+.
@@ -61,8 +61,8 @@ remote(Ann, Module, Function, Args) when is_atom(Module), is_atom(Function), is_
 
 %% Converts an Elixir definition to an anonymous function.
 
-definition_to_anonymous(File, Module, {Name, Arity}, Kind, Meta, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Name, Arity, Clause, File) || Clause <- Clauses],
+definition_to_anonymous(File, Module, Kind, Meta, Clauses) ->
+  ErlClauses = [translate_clause(Kind, Clause, File) || Clause <- Clauses],
   Fun = {'fun', ?ann(Meta), {clauses, ErlClauses}},
   LocalHandler = fun(LocalName, LocalArgs) -> invoke_local(Module, LocalName, LocalArgs) end,
   {value, Result, _Binding} = erl_eval:expr(Fun, [], {value, LocalHandler}),
@@ -137,14 +137,14 @@ elixir_to_erl_cons2([], Acc) ->
 
 %% Returns a definition scope for translation.
 
-definition_scope(Meta, Kind, Name, Arity, File) ->
+definition_scope(Meta, File) ->
   %% TODO: We only need to do this dance because some
   %% warnings are raised in elixir_erl_pass. Once we remove
   %% all warnings from the Erlang pass, we can remove the
   %% file field from #elixir_erl and clean up the code.
   case lists:keyfind(location, 1, Meta) of
-    {location, {F, _}} -> #elixir_erl{def = {Kind, Name, Arity}, file = F};
-    false -> #elixir_erl{def = {Kind, Name, Arity}, file = File}
+    {location, {F, _}} -> #elixir_erl{file=F};
+    false -> #elixir_erl{file=File}
   end.
 
 %% Compilation hook.
@@ -159,37 +159,57 @@ compile(#{module := Module} = Map) ->
     end,
   load_form(Map, Data, Prefix, Forms, Specs).
 
+dynamic_form(#{module := Module, line := Line, file := File, attributes := Attributes,
+               definitions := Definitions, unreachable := Unreachable}) ->
+  {Def, Defmacro, Macros, Exports, Functions} =
+    split_definition(Definitions, File, Unreachable, [], [], [], [], {[], []}),
+
+  Location = {elixir_utils:characters_to_list(elixir_utils:relative_to_cwd(File)), Line},
+  Prefix = [{attribute, Line, file, Location},
+            {attribute, Line, module, Module},
+            {attribute, Line, compile, no_auto_import}],
+
+  Forms0 = functions_form(Line, Module, Def, Defmacro, Exports, Functions),
+  Forms1 = attributes_form(Line, Attributes, Forms0),
+  {Prefix, Forms1, Macros, Unreachable}.
+
 % Definitions
 
-split_definition([{Tuple, def, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  {_, _, N, A, _} = Function = translate_definition(def, Meta, File, Tuple, Clauses),
-  split_definition(T, File, Unreachable, [Tuple | Def], Defmacro, [{N, A} | Exports],
-                   add_definition(Meta, Function, Functions));
-
-split_definition([{Tuple, defp, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  Function = translate_definition(defp, Meta, File, Tuple, Clauses),
+split_definition([{Tuple, Kind, Meta, Clauses} | T], File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
   case lists:member(Tuple, Unreachable) of
     false ->
-      split_definition(T, File, Unreachable, Def, Defmacro, Exports,
-                       add_definition(Meta, Function, Functions));
+      split_definition(Tuple, Kind, Meta, Clauses, T, File, Unreachable,
+                       Def, Defmacro, Macros, Exports, Functions);
     true ->
-      split_definition(T, File, Unreachable, Def, Defmacro, Exports, Functions)
+      split_definition(T, File, Unreachable, Def, Defmacro, Macros, Exports, Functions)
   end;
+split_definition([], _File, _Unreachable, Def, Defmacro, Macros, Exports, {Head, Tail}) ->
+  {Def, Defmacro, Macros, Exports, Head ++ Tail}.
 
-split_definition([{Tuple, defmacro, Meta, Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  {_, _, N, A, _} = Function = translate_definition(defmacro, Meta, File, Tuple, Clauses),
-  split_definition(T, File, Unreachable, Def, [Tuple | Defmacro], [{N, A} | Exports],
-                   add_definition(Meta, Function, Functions));
+split_definition(Tuple, def, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  {_, _, N, A, _} = Entry = translate_definition(def, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, [Tuple | Def], Defmacro, Macros, [{N, A} | Exports],
+                   add_definition(Meta, Entry, Functions));
 
-split_definition([{_, defmacrop, _Meta, _Clauses} | T], File, Unreachable,
-                 Def, Defmacro, Exports, Functions) ->
-  split_definition(T, File, Unreachable, Def, Defmacro, Exports, Functions);
+split_definition(Tuple, defp, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  Entry = translate_definition(defp, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, Defmacro, Macros, Exports,
+                   add_definition(Meta, Entry, Functions));
 
-split_definition([], _File, _Unreachable, Def, Defmacro, Exports, {Head, Tail}) ->
-  {Def, Defmacro, Exports, Head ++ Tail}.
+split_definition(Tuple, defmacro, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  {_, _, N, A, _} = Entry = translate_definition(defmacro, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, [Tuple | Defmacro], [Tuple | Macros], [{N, A} | Exports],
+                   add_definition(Meta, Entry, Functions));
+
+split_definition(Tuple, defmacrop, Meta, Clauses, T, File, Unreachable,
+                 Def, Defmacro, Macros, Exports, Functions) ->
+  Entry = translate_definition(defmacro, Meta, File, Tuple, Clauses),
+  split_definition(T, File, Unreachable, Def, Defmacro, [Tuple | Macros], Exports,
+                   add_definition(Meta, Entry, Functions)).
 
 add_definition(Meta, Body, {Head, Tail}) ->
   case lists:keyfind(location, 1, Meta) of
@@ -206,15 +226,15 @@ add_definition(Meta, Body, {Head, Tail}) ->
   end.
 
 translate_definition(Kind, Meta, File, {Name, Arity}, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Name, Arity, Clause, File) || Clause <- Clauses],
+  ErlClauses = [translate_clause(Kind, Clause, File) || Clause <- Clauses],
 
   case is_macro(Kind) of
     true -> {function, ?ann(Meta), elixir_utils:macro_name(Name), Arity + 1, ErlClauses};
     false -> {function, ?ann(Meta), Name, Arity, ErlClauses}
   end.
 
-translate_clause(Kind, Name, Arity, {Meta, Args, Guards, Body}, File) ->
-  S = definition_scope(Meta, Kind, Name, Arity, File),
+translate_clause(Kind, {Meta, Args, Guards, Body}, File) ->
+  S = definition_scope(Meta, File),
 
   {TClause, TS} = elixir_erl_clauses:clause(Meta,
                     fun elixir_erl_pass:translate_args/2, Args, Body, Guards, S),
@@ -245,20 +265,6 @@ is_macro(_)         -> false.
 
 % Functions
 
-dynamic_form(#{module := Module, line := Line, file := File, attributes := Attributes,
-               definitions := Definitions, unreachable := Unreachable}) ->
-  {Def, Defmacro, Exports, Functions} =
-    split_definition(Definitions, File, Unreachable, [], [], [], {[], []}),
-
-  Location = {elixir_utils:characters_to_list(elixir_utils:relative_to_cwd(File)), Line},
-  Prefix = [{attribute, Line, file, Location},
-            {attribute, Line, module, Module},
-            {attribute, Line, compile, no_auto_import}],
-
-  Forms0 = functions_form(Line, Module, Def, Defmacro, Exports, Functions),
-  Forms1 = attributes_form(Line, Attributes, Forms0),
-  {Prefix, Forms1, Defmacro, Unreachable}.
-
 functions_form(Line, Module, Def, Defmacro, Exports, Body) ->
   {Spec, Info} = add_info_function(Line, Module, Def, Defmacro),
   [{attribute, Line, export, lists:sort([{'__info__', 1} | Exports])}, Spec, Info | Body].

lib/elixir/src/elixir_erl_clauses.erl

@@ -1,49 +1,39 @@
 %% Handle code related to args, guard and -> matching for case,
 %% fn, receive and friends. try is handled in elixir_erl_try.
 -module(elixir_erl_clauses).
--export([match/3, clause/6, clauses/3, guards/3, get_clauses/3, get_clauses/4]).
+-export([match/3, clause/6, clauses/3, guards/3, get_clauses/3]).
 -include("elixir.hrl").
 
 %% Get clauses under the given key.
 
 get_clauses(Key, Keyword, As) ->
-  get_clauses(Key, Keyword, As, false).
-get_clauses(Key, Keyword, As, AllowNil) ->
   case lists:keyfind(Key, 1, Keyword) of
     {Key, Clauses} when is_list(Clauses) ->
       [{As, Meta, Left, Right} || {'->', Meta, [Left, Right]} <- Clauses];
-    {Key, nil} when AllowNil ->
-      [];
-    false ->
+    _ ->
       []
   end.
 
 %% Translate matches
 
-match(Fun, Args, #elixir_erl{context=Context, match_vars=MatchVars,
-    backup_vars=BackupVars, vars=Vars} = S) when Context /= match ->
-  {Result, NewS} = match(Fun, Args, S#elixir_erl{context=match,
-                         match_vars=#{}, backup_vars=Vars}),
-  {Result, NewS#elixir_erl{context=Context,
-      match_vars=MatchVars, backup_vars=BackupVars}};
+match(Fun, Args, #elixir_erl{context=Context, backup_vars=BackupVars, vars=Vars} = S) when Context /= match ->
+  {Result, NewS} = match(Fun, Args, S#elixir_erl{context=match, backup_vars=Vars}),
+  {Result, NewS#elixir_erl{context=Context, backup_vars=BackupVars}};
 match(Fun, Args, S) -> Fun(Args, S).
 
 %% Translate clauses with args, guards and expressions
 
 clause(Meta, Fun, Args, Expr, Guards, S) when is_list(Meta) ->
-  {TArgs, SA} = match(Fun, Args, S#elixir_erl{extra_guards=[]}),
-  {TExpr, SE} = elixir_erl_pass:translate(Expr,
-                  SA#elixir_erl{extra_guards=nil, export_vars=S#elixir_erl.export_vars}),
-
-  Extra = SA#elixir_erl.extra_guards,
-  TGuards = guards(Guards, Extra, SA),
+  {TArgs, SA} = match(Fun, Args, S),
+  SG = SA#elixir_erl{extra_guards=[]},
+  TGuards = guards(Guards, SA#elixir_erl.extra_guards, SG),
+  {TExpr, SE} = elixir_erl_pass:translate(Expr, SG#elixir_erl{export_vars=S#elixir_erl.export_vars}),
   {{clause, ?ann(Meta), TArgs, TGuards, unblock(TExpr)}, SE}.
 
 % Translate/Extract guards from the given expression.
 
 guards(Guards, Extra, S) ->
-  SG = S#elixir_erl{context=guard, extra_guards=nil},
-
+  SG = S#elixir_erl{context=guard},
   case Guards of
     [] -> case Extra of [] -> []; _ -> [Extra] end;
     _  -> [translate_guard(Guard, Extra, SG) || Guard <- Guards]

lib/elixir/src/elixir_erl_for.erl

@@ -50,11 +50,11 @@ translate_gen(_Meta, Left, Right, T, S) ->
   {TRight, SR} = elixir_erl_pass:translate(Right, S),
   {LeftArgs, LeftGuards} = elixir_utils:extract_guards(Left),
   {TLeft, SL} = elixir_erl_clauses:match(fun elixir_erl_pass:translate/2, LeftArgs,
-                                     SR#elixir_erl{extra=pin_guard, extra_guards=[]}),
+                                         SR#elixir_erl{extra=pin_guard}),
 
   TLeftGuards = elixir_erl_clauses:guards(LeftGuards, [], SL),
   ExtraGuards = [{nil, X} || X <- SL#elixir_erl.extra_guards],
-  SF = SL#elixir_erl{extra=S#elixir_erl.extra, extra_guards=nil},
+  SF = SL#elixir_erl{extra=S#elixir_erl.extra, extra_guards=[]},
 
   {TT, {TFilters, TS}} = translate_filters(T, SF),
 

lib/elixir/src/elixir_erl_pass.erl

@@ -41,15 +41,16 @@ translate({'__block__', Meta, Args}, S) when is_list(Args) ->
 translate({'__CALLER__', Meta, Atom}, S) when is_atom(Atom) ->
   {{var, ?ann(Meta), '__CALLER__'}, S#elixir_erl{caller=true}};
 
-translate({'super', Meta, Args}, #elixir_erl{def={Kind, Name, _}} = S) ->
+translate({'super', Meta, [{Kind, Name} | Args]}, S) ->
   %% In the expanded AST, super is used to invoke a function
-  %% with the same name but possibly different arity.
+  %% in the current module originated from a default clause
+  %% or a super call.
   {TArgs, SA} = translate_args(Args, S),
   Ann = ?ann(Meta),
   if
     Kind == defmacro; Kind == defmacrop ->
       MacroName = elixir_utils:macro_name(Name),
-      {{call, Ann, {atom, Ann, MacroName}, [{var, Ann, '_@CALLER'} | TArgs]}, SA};
+      {{call, Ann, {atom, Ann, MacroName}, [{var, Ann, '__CALLER__'} | TArgs]}, SA#elixir_erl{caller=true}};
     Kind == def; Kind == defp ->
       {{call, Ann, {atom, Ann, Name}, TArgs}, SA}
   end;
@@ -70,8 +71,8 @@ translate({'&', Meta, [{'/', _, [{Fun, _, Atom}, Arity]}]}, S)
 translate({fn, Meta, Clauses}, S) ->
   Transformer = fun({'->', CMeta, [ArgsWithGuards, Expr]}, Acc) ->
     {Args, Guards} = elixir_utils:extract_splat_guards(ArgsWithGuards),
-    {TClause, TS } = elixir_erl_clauses:clause(CMeta, fun translate_fn_match/2,
-                                            Args, Expr, Guards, Acc),
+    {TClause, TS} = elixir_erl_clauses:clause(CMeta, fun translate_fn_match/2,
+                                              Args, Expr, Guards, Acc),
     {TClause, elixir_erl_var:mergec(S, TS)}
   end,
   {TClauses, NS} = lists:mapfoldl(Transformer, S, Clauses),
@@ -101,29 +102,28 @@ translate({'case', Meta, [Expr, Opts]}, S) ->
 %% Try
 
 translate({'try', Meta, [Opts]}, S) ->
-  SN = S#elixir_erl{extra=nil},
   Do = proplists:get_value('do', Opts, nil),
-  {TDo, SB} = translate(Do, SN),
+  {TDo, SB} = translate(Do, S),
 
   Catch = [Tuple || {X, _} = Tuple <- Opts, X == 'rescue' orelse X == 'catch'],
-  {TCatch, SC} = elixir_erl_try:clauses(Meta, Catch, mergec(SN, SB)),
+  {TCatch, SC} = elixir_erl_try:clauses(Meta, Catch, mergec(S, SB)),
 
   {TAfter, SA} = case lists:keyfind('after', 1, Opts) of
     {'after', After} ->
-      {TBlock, SAExtracted} = translate(After, mergec(SN, SC)),
+      {TBlock, SAExtracted} = translate(After, mergec(S, SC)),
       {unblock(TBlock), SAExtracted};
     false ->
-      {[], mergec(SN, SC)}
+      {[], mergec(S, SC)}
   end,
 
   Else = elixir_erl_clauses:get_clauses(else, Opts, match),
-  {TElse, SE} = elixir_erl_clauses:clauses(Meta, Else, mergec(SN, SA)),
+  {TElse, SE} = elixir_erl_clauses:clauses(Meta, Else, mergec(S, SA)),
   {{'try', ?ann(Meta), unblock(TDo), TElse, TCatch, TAfter}, mergec(S, SE)};
 
 %% Receive
 
 translate({'receive', Meta, [Opts]}, S) ->
-  Do = elixir_erl_clauses:get_clauses(do, Opts, match, true),
+  Do = elixir_erl_clauses:get_clauses(do, Opts, match),
 
   case lists:keyfind('after', 1, Opts) of
     false ->
@@ -137,17 +137,22 @@ translate({'receive', Meta, [Opts]}, S) ->
       {{'receive', ?ann(Meta), FClauses, FExpr, FAfter}, SC}
   end;
 
-%% Comprehensions
+%% Comprehensions and with
 
 translate({for, Meta, [_ | _] = Args}, S) ->
   elixir_erl_for:translate(Meta, Args, true, S);
 
+translate({with, Meta, [_ | _] = Args}, S) ->
+  {Exprs, [{do, Do} | Opts]} = elixir_utils:split_last(Args),
+  {ElseClause, SE} = translate_with_else(Meta, Opts, S),
+  {With, SD} = translate_with_do(Exprs, Do, ElseClause, elixir_erl_var:mergec(S, SE)),
+  {With, elixir_erl_var:mergec(S, SD)};
+
 %% Variables
 
 translate({'^', Meta, [{Name, VarMeta, Kind}]}, #elixir_erl{context=match, file=File} = S) when is_atom(Name), is_atom(Kind) ->
   Tuple = {Name, var_kind(VarMeta, Kind)},
   {ok, {Value, _Counter, Safe}} = maps:find(Tuple, S#elixir_erl.backup_vars),
-  elixir_erl_var:warn_underscored_var_access(VarMeta, File, Name),
   elixir_erl_var:warn_unsafe_var(VarMeta, File, Name, Safe),
 
   PAnn = ?ann(Meta),
@@ -235,10 +240,10 @@ translate(Other, S) ->
 translate_case(true, Meta, Expr, Opts, S) ->
   Clauses = elixir_erl_clauses:get_clauses(do, Opts, match),
   {TExpr, SE} = translate(Expr, S),
-  {TClauses, SC} = elixir_erl_clauses:clauses(Meta, Clauses, SE#elixir_erl{extra=nil}),
-  {{'case', ?ann(Meta), TExpr, TClauses}, SC#elixir_erl{extra=SE#elixir_erl.extra}};
+  {TClauses, SC} = elixir_erl_clauses:clauses(Meta, Clauses, SE),
+  {{'case', ?ann(Meta), TExpr, TClauses}, SC};
 translate_case(false, Meta, Expr, Opts, S) ->
-  {Case, SC} = translate_case(true, Meta, Expr, Opts, S#elixir_erl{extra=nil}),
+  {Case, SC} = translate_case(true, Meta, Expr, Opts, S),
   {Case, elixir_erl_var:mergec(S, SC)}.
 
 translate_list([{'|', _, [_, _]=Args}], Fun, Acc, List) ->
@@ -350,6 +355,43 @@ returns_boolean(Condition, Body) ->
     false -> false
   end.
 
+%% with
+
+translate_with_else(Meta, [], S) ->
+  Ann = ?ann(?generated(Meta)),
+  {VarName, _, SC} = elixir_erl_var:build('_', S),
+  Var = {var, Ann, VarName},
+  {{clause, Ann, [Var], [], [Var]}, SC};
+translate_with_else(Meta, [{else, Else}], S) ->
+  Generated = ?generated(Meta),
+  ElseVarEx = {else, Generated, ?var_context},
+  {ElseVarErl, SV} = elixir_erl_var:assign(Generated, else, ?var_context, S),
+
+  RaiseVar = {catch_all, Generated, ?var_context},
+  RaiseExpr = {{'.', Generated, [erlang, error]}, Generated, [{with_clause, RaiseVar}]},
+  RaiseClause = {'->', Generated, [[RaiseVar], RaiseExpr]},
+
+  Case = {'case', [{export_vars, false} | Meta], [ElseVarEx, [{do, Else ++ [RaiseClause]}]]},
+  {TranslatedCase, SC} = elixir_erl_pass:translate(Case, SV),
+  {{clause, ?ann(Generated), [ElseVarErl], [], [TranslatedCase]}, SC}.
+
+translate_with_do([{'<-', Meta, [Left, Expr]} | Rest], Do, Else, S) ->
+  {Args, Guards} = elixir_utils:extract_guards(Left),
+  {TExpr, SR} = elixir_erl_pass:translate(Expr, S),
+  {TArgs, SA} = elixir_erl_clauses:match(fun elixir_erl_pass:translate/2, Args, SR),
+  TGuards = elixir_erl_clauses:guards(Guards, [], SA),
+  {TBody, SB} = translate_with_do(Rest, Do, Else, SA),
+
+  Ann = ?ann(?generated(Meta)),
+  Clause = {clause, Ann, [TArgs], TGuards, unblock(TBody)},
+  {{'case', Ann, TExpr, [Clause, Else]}, SB};
+translate_with_do([Expr | Rest], Do, Else, S) ->
+  {TExpr, TS} = elixir_erl_pass:translate(Expr, S),
+  {TRest, RS} = translate_with_do(Rest, Do, Else, TS),
+  {{block, 0, [TExpr | unblock(TRest)]}, RS};
+translate_with_do([], Do, _Else, S) ->
+  elixir_erl_pass:translate(Do, S).
+
 %% Maps and structs
 
 translate_map(Meta, [{'|', _Meta, [Update, Assocs]}], S) ->
@@ -513,7 +555,7 @@ translate_remote(Left, Right, Meta, Args, S) ->
       {{call, Ann, {remote, Ann, TLeft, TRight}, TArgs}, SC}
   end.
 
-is_always_string({{'.', _, [Module, Function]}, Args}) ->
+is_always_string({{'.', _, [Module, Function]}, _, Args}) ->
   is_always_string(Module, Function, length(Args));
 %% Binary literals were already excluded in earlier passes.
 is_always_string(_Ast) ->

lib/elixir/src/elixir_erl_try.erl

@@ -23,19 +23,21 @@ each_clause({'catch', Meta, Raw, Expr}, S) ->
 
   Condition = [{'{}', Meta, Final}],
   {TC, TS} = elixir_erl_clauses:clause(Meta, fun elixir_erl_pass:translate_args/2,
-                                   Condition, Expr, Guards, S),
+                                       Condition, Expr, Guards, S),
   {[TC], TS};
 
 each_clause({rescue, Meta, [{in, _, [Left, Right]}], Expr}, S) ->
-  {VarName, _, CS} = elixir_erl_var:build('_', S),
-  Var = {VarName, Meta, nil},
-  {Parts, Safe, FS} = rescue_guards(Meta, Var, Right, CS),
-  Body = rescue_clause_body(Left, Expr, Safe, Var, Meta),
+  {TempName, _, CS} = elixir_erl_var:build('_', S),
+  TempVar = {TempName, Meta, ?var_context},
+  {Parts, Safe, FS} = rescue_guards(Meta, TempVar, Right, CS),
+  Body = rescue_clause_body(Left, Expr, Safe, TempVar, Meta),
   build_rescue(Meta, Parts, Body, FS);
 
-each_clause({rescue, Meta, [{VarName, _, Atom} = Var], Expr}, S) when is_atom(VarName), is_atom(Atom) ->
-  Body = rescue_clause_body(Var, Expr, false, Var, Meta),
-  build_rescue(Meta, _Parts = [{Var, []}], Body, S).
+each_clause({rescue, Meta, [{VarName, _, Context} = Left], Expr}, S) when is_atom(VarName), is_atom(Context) ->
+  {TempName, _, CS} = elixir_erl_var:build('_', S),
+  TempVar = {TempName, Meta, ?var_context},
+  Body = rescue_clause_body(Left, Expr, false, TempVar, Meta),
+  build_rescue(Meta, [{TempVar, []}], Body, CS).
 
 rescue_clause_body({'_', _, Atom}, Expr, _Safe, _Var, _Meta) when is_atom(Atom) ->
   Expr;
@@ -54,7 +56,7 @@ build_rescue(Meta, Parts, Body, S) ->
 
   {{clause, Line, TMatches, _, TBody}, TS} =
     elixir_erl_clauses:clause(Meta, fun elixir_erl_pass:translate_args/2,
-                          Matches, Body, [], S),
+                              Matches, Body, [], S),
 
   TClauses =
     [begin
@@ -74,7 +76,7 @@ rescue_guards(Meta, Var, Aliases, S) ->
       [] -> {[], S};
       _  ->
         {VarName, _, CS} = elixir_erl_var:build('_', S),
-        StructVar = {VarName, Meta, nil},
+        StructVar = {VarName, Meta, 'Elixir'},
         Map = {'%{}', Meta, [{'__struct__', StructVar}, {'__exception__', true}]},
         Match = {'=', Meta, [Map, Var]},
         Guards = [{erl(Meta, '=='), Meta, [StructVar, Mod]} || Mod <- Elixir],

lib/elixir/src/elixir_erl_var.erl

@@ -1,64 +1,68 @@
 %% Convenience functions used to manipulate scope and its variables.
 -module(elixir_erl_var).
--export([translate/4, build/2, context_info/1,
+-export([translate/4, build/2, assign/4,
   load_binding/2, dump_binding/2,
   mergev/2, mergec/2, merge_vars/2, merge_opt_vars/2,
-  warn_unsafe_var/4, warn_underscored_var_access/3, format_error/1
+  warn_unsafe_var/4, format_error/1
 ]).
 -include("elixir.hrl").
 
 %% VAR HANDLING
 
 translate(Meta, Name, Kind, S) when is_atom(Kind); is_integer(Kind) ->
-  Ann = ?ann(Meta),
   Tuple = {Name, Kind},
-  Vars = S#elixir_erl.vars,
 
-  {Current, Exists, Safe} =
-    case maps:find({Name, Kind}, Vars) of
-      {ok, {VarC, _, VarS}} -> {VarC, true, VarS};
-      error -> {nil, false, true}
+  {Current, Safe} =
+    case maps:find(Tuple, S#elixir_erl.vars) of
+      {ok, {VarC, _, VarS}} -> {VarC, VarS};
+      error -> {nil, true}
     end,
 
   case S#elixir_erl.context of
     match ->
-      MatchVars = S#elixir_erl.match_vars,
+      Previous =
+        case maps:find(Tuple, S#elixir_erl.backup_vars) of
+          {ok, {BackupVarC, _, _}} -> BackupVarC;
+          error -> nil
+        end,
 
-      case Exists andalso maps:get(Tuple, MatchVars, false) of
+      if
+        Current /= nil, Current /= Previous ->
+          {{var, ?ann(Meta), Current}, S};
         true ->
-          warn_underscored_var_repeat(Meta, S#elixir_erl.file, Name, Kind),
-          {{var, Ann, Current}, S};
-        false ->
-          %% We attempt to give vars a nice name because we
-          %% still use the unused vars warnings from erl_lint.
-          %%
-          %% Once we move the warning to Elixir compiler, we
-          %% can name vars as _@COUNTER.
-          {NewVar, Counter, NS} =
-            if
-              Kind /= nil ->
-                build('_', S);
-              true ->
-                build(Name, S)
-            end,
-
-          FS = NS#elixir_erl{
-            vars=maps:put(Tuple, {NewVar, Counter, true}, Vars),
-            match_vars=maps:put(Tuple, true, MatchVars),
-            export_vars=case S#elixir_erl.export_vars of
-              nil -> nil;
-              EV  -> maps:put(Tuple, {NewVar, Counter, true}, EV)
-            end
-          },
-
-          {{var, Ann, NewVar}, FS}
+          assign(Meta, Name, Kind, S)
       end;
-    _  when Exists ->
-      warn_underscored_var_access(Meta, S#elixir_erl.file, Name),
+    _  when Current /= nil ->
       warn_unsafe_var(Meta, S#elixir_erl.file, Name, Safe),
-      {{var, Ann, Current}, S}
+      {{var, ?ann(Meta), Current}, S}
   end.
 
+assign(Meta, Name, Kind, S) ->
+  Tuple = {Name, Kind},
+
+  %% We attempt to give vars a nice name because we
+  %% still use the unused vars warnings from erl_lint.
+  %%
+  %% Once we move the warning to Elixir compiler, we
+  %% can name vars as _@COUNTER.
+  {NewVar, Counter, NS} =
+    if
+      Kind /= nil ->
+        build('_', S);
+      true ->
+        build(Name, S)
+    end,
+
+  FS = NS#elixir_erl{
+    vars=maps:put(Tuple, {NewVar, Counter, true}, S#elixir_erl.vars),
+    export_vars=case S#elixir_erl.export_vars of
+      nil -> nil;
+      EV  -> maps:put(Tuple, {NewVar, Counter, true}, EV)
+    end
+  },
+
+  {{var, ?ann(Meta), NewVar}, FS}.
+
 build(Key, #elixir_erl{counter=Counter} = S) ->
   Cnt =
     case maps:find(Key, Counter) of
@@ -69,43 +73,14 @@ build(Key, #elixir_erl{counter=Counter} = S) ->
    Cnt,
    S#elixir_erl{counter=maps:put(Key, Cnt, Counter)}}.
 
-context_info(Kind) when Kind == nil; is_integer(Kind) -> "";
-context_info(Kind) -> io_lib:format(" (context ~ts)", [elixir_aliases:inspect(Kind)]).
-
-warn_underscored_var_repeat(Meta, File, Name, Kind) ->
-  Warn = should_warn(Meta),
-  case atom_to_list(Name) of
-    "_@" ++ _ ->
-      ok; %% Automatically generated variables
-    "_" ++ _ when Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {unused_match, Name, Kind});
-    _ ->
-      ok
-  end.
-
 warn_unsafe_var(Meta, File, Name, Safe) ->
-  Warn = should_warn(Meta),
-  if
-    (not Safe) and Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {unsafe_var, Name});
+  case (not Safe) andalso (lists:keyfind(generated, 1, Meta) /= {generated, true}) of
     true ->
+      elixir_errors:form_warn(Meta, File, ?MODULE, {unsafe_var, Name});
+    false ->
       ok
   end.
 
-warn_underscored_var_access(Meta, File, Name) ->
-  Warn = should_warn(Meta),
-  case atom_to_list(Name) of
-    "_@" ++ _ ->
-      ok; %% Automatically generated variables
-    "_" ++ _ when Warn ->
-      elixir_errors:form_warn(Meta, File, ?MODULE, {underscore_var_access, Name});
-    _ ->
-      ok
-  end.
-
-should_warn(Meta) ->
-  lists:keyfind(generated, 1, Meta) /= {generated, true}.
-
 %% SCOPE MERGING
 
 %% Receives two scopes and return a new scope based on
@@ -193,13 +168,6 @@ dump_binding(Binding, #elixir_erl{vars=Vars}) ->
 
 %% Errors
 
-format_error({unused_match, Name, Kind}) ->
-  io_lib:format("the underscored variable \"~ts\"~ts appears more than once in a "
-                "match. This means the pattern will only match if all \"~ts\" bind "
-                "to the same value. If this is the intended behaviour, please "
-                "remove the leading underscore from the variable name, otherwise "
-                "give the variables different names", [Name, context_info(Kind), Name]);
-
 format_error({unsafe_var, Name}) ->
   io_lib:format("the variable \"~ts\" is unsafe as it has been set inside "
                 "one of: case, cond, receive, if, and, or, &&, ||. "
@@ -214,10 +182,4 @@ format_error({unsafe_var, Name}) ->
                 "        1 -> :one\n"
                 "        2 -> :two\n"
                 "      end\n\n"
-                "Unsafe variable found at:", [Name]);
-
-format_error({underscore_var_access, Name}) ->
-  io_lib:format("the underscored variable \"~ts\" is used after being set. "
-                "A leading underscore indicates that the value of the variable "
-                "should be ignored. If this is intended please rename the "
-                "variable to remove the underscore", [Name]).
+                "Unsafe variable found at:", [Name]).

lib/elixir/src/elixir_expand.erl

@@ -6,7 +6,7 @@
 %% =
 
 expand({'=', Meta, [Left, Right]}, E) ->
-  assert_no_guard_scope(Meta, '=', E),
+  assert_no_guard_scope(Meta, "=", E),
   {ERight, ER} = expand(Right, E),
   {ELeft, EL}  = elixir_clauses:match(fun expand/2, Left, E),
   {{'=', Meta, [ELeft, ERight]}, elixir_env:mergev(EL, ER)};
@@ -62,7 +62,7 @@ expand({Kind, Meta, [{{'.', _, [Base, '{}']}, _, Refs} | Rest]}, E)
 expand({alias, Meta, [Ref]}, E) ->
   expand({alias, Meta, [Ref, []]}, E);
 expand({alias, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, alias, E),
+  assert_no_match_or_guard_scope(Meta, "alias", E),
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, alias, [as, warn], no_alias_opts(Opts), ER),
 
@@ -76,7 +76,7 @@ expand({alias, Meta, [Ref, Opts]}, E) ->
 expand({require, Meta, [Ref]}, E) ->
   expand({require, Meta, [Ref, []]}, E);
 expand({require, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, require, E),
+  assert_no_match_or_guard_scope(Meta, "require", E),
 
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, require, [as, warn], no_alias_opts(Opts), ER),
@@ -93,7 +93,7 @@ expand({import, Meta, [Left]}, E) ->
   expand({import, Meta, [Left, []]}, E);
 
 expand({import, Meta, [Ref, Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, import, E),
+  assert_no_match_or_guard_scope(Meta, "import", E),
   {ERef, ER} = expand_without_aliases_report(Ref, E),
   {EOpts, ET}  = expand_opts(Meta, import, [only, except, warn], Opts, ER),
 
@@ -209,7 +209,7 @@ expand({quote, Meta, [_, _]}, E) ->
 %% Functions
 
 expand({'&', Meta, [Arg]}, E) ->
-  assert_no_match_or_guard_scope(Meta, '&', E),
+  assert_no_match_or_guard_scope(Meta, "&", E),
   case elixir_fn:capture(Meta, Arg, E) of
     {remote, Remote, Fun, Arity} ->
       is_atom(Remote) andalso
@@ -222,34 +222,36 @@ expand({'&', Meta, [Arg]}, E) ->
   end;
 
 expand({fn, Meta, Pairs}, E) ->
-  assert_no_match_or_guard_scope(Meta, fn, E),
+  assert_no_match_or_guard_scope(Meta, "fn", E),
   elixir_fn:expand(Meta, Pairs, E);
 
 %% Case/Receive/Try
 
 expand({'cond', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'cond', E),
+  assert_no_match_or_guard_scope(Meta, "cond", E),
   assert_no_underscore_clause_in_cond(Opts, E),
   {EClauses, EC} = elixir_clauses:'cond'(Meta, Opts, E),
   {{'cond', Meta, [EClauses]}, EC};
 
-expand({'case', Meta, [Expr, Options]}, Env) ->
+expand({'case', Meta, [Expr, Options]}, E) ->
+  assert_no_match_or_guard_scope(Meta, "case", E),
   ShouldExportVars = proplists:get_value(export_vars, Meta, true),
-  expand_case(ShouldExportVars, Meta, Expr, Options, Env);
+  expand_case(ShouldExportVars, Meta, Expr, Options, E);
 
 expand({'receive', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'receive', E),
+  assert_no_match_or_guard_scope(Meta, "receive", E),
   {EClauses, EC} = elixir_clauses:'receive'(Meta, Opts, E),
   {{'receive', Meta, [EClauses]}, EC};
 
 expand({'try', Meta, [Opts]}, E) ->
-  assert_no_match_or_guard_scope(Meta, 'try', E),
+  assert_no_match_or_guard_scope(Meta, "try", E),
   {EClauses, EC} = elixir_clauses:'try'(Meta, Opts, E),
   {{'try', Meta, [EClauses]}, EC};
 
 %% Comprehensions
 
 expand({for, Meta, [_ | _] = Args}, E) ->
+  assert_no_match_or_guard_scope(Meta, "for", E),
   {Cases, Block} =
     case elixir_utils:split_last(Args) of
       {OuterCases, OuterOpts} when is_list(OuterOpts) ->
@@ -281,25 +283,22 @@ expand({for, Meta, [_ | _] = Args}, E) ->
 %% With
 
 expand({with, Meta, [_ | _] = Args}, E) ->
-  elixir_with:expand(Meta, Args, E);
+  assert_no_match_or_guard_scope(Meta, "with", E),
+  elixir_clauses:with(Meta, Args, E);
 
 %% Super
 
 expand({super, Meta, Args}, #{file := File} = E) when is_list(Args) ->
+  assert_no_match_or_guard_scope(Meta, "super", E),
   Module = assert_module_scope(Meta, super, E),
   Function = assert_function_scope(Meta, super, E),
   {_, Arity} = Function,
 
   case length(Args) of
     Arity ->
-      {OName, OArity} = elixir_overridable:super(Meta, File, Module, Function),
+      {Kind, Name} = elixir_overridable:super(Meta, File, Module, Function),
       {EArgs, EA} = expand_args(Args, E),
-      OArgs =
-        if
-          OArity > Arity -> [{'__CALLER__', [], nil} | EArgs];
-          true -> EArgs
-        end,
-      {{OName, Meta, OArgs}, EA};
+      {{super, Meta, [{Kind, Name} | EArgs]}, EA};
     _ ->
       form_error(Meta, File, ?MODULE, wrong_number_of_args_for_super)
   end;
@@ -315,8 +314,9 @@ expand({'^', Meta, [Arg]}, #{context := match, prematch_vars := PrematchVars} =
       %% If the variable was defined, then we return the expanded ^, otherwise
       %% we raise. We cannot use the expanded env because it would contain the
       %% variable.
-      case lists:member({VarName, var_kind(VarMeta, Kind)}, PrematchVars) of
+      case lists:member({VarName, var_context(VarMeta, Kind)}, PrematchVars) of
         true ->
+          warn_underscored_var_access(VarMeta, VarName, Kind, E),
           {{'^', Meta, [Var]}, EA};
         false ->
           form_error(Meta, ?key(EA, file), ?MODULE, {unbound_variable_pin, VarName})
@@ -332,27 +332,43 @@ expand({'_', _Meta, Kind} = Var, #{context := match} = E) when is_atom(Kind) ->
 expand({'_', Meta, Kind}, E) when is_atom(Kind) ->
   form_error(Meta, ?key(E, file), ?MODULE, unbound_underscore);
 
-expand({Name, Meta, Kind} = Var, #{context := match, export_vars := Export} = E) when is_atom(Name), is_atom(Kind) ->
-  Pair      = {Name, var_kind(Meta, Kind)},
-  NewVars   = ordsets:add_element(Pair, ?key(E, vars)),
-  NewExport = case (Export /= nil) of
-    true  -> ordsets:add_element(Pair, Export);
-    false -> Export
-  end,
-  {Var, E#{vars := NewVars, export_vars := NewExport}};
+expand({Name, Meta, Kind} = Var, #{context := match} = E) when is_atom(Name), is_atom(Kind) ->
+  #{vars := Vars, match_vars := Match, export_vars := Export} = E,
+  Pair = {Name, var_context(Meta, Kind)},
+  NewVars = ordsets:add_element(Pair, Vars),
+
+  NewExport =
+    case (Export /= nil) of
+      true  -> ordsets:add_element(Pair, Export);
+      false -> Export
+    end,
+
+  NewMatch =
+    case lists:member(Pair, Match) of
+      true -> warn_underscored_var_repeat(Meta, Name, Kind, E), Match;
+      false -> [Pair | Match]
+    end,
+
+  {Var, E#{vars := NewVars, match_vars := NewMatch, export_vars := NewExport}};
 expand({Name, Meta, Kind} = Var, #{vars := Vars} = E) when is_atom(Name), is_atom(Kind) ->
-  case lists:member({Name, var_kind(Meta, Kind)}, Vars) of
+  case lists:member({Name, var_context(Meta, Kind)}, Vars) of
     true ->
+      warn_underscored_var_access(Meta, Name, Kind, E),
       {Var, E};
     false ->
       case lists:keyfind(var, 1, Meta) of
         {var, true} ->
           form_error(Meta, ?key(E, file), ?MODULE, {undefined_var, Name, Kind});
         _ ->
-          Message =
-            io_lib:format("variable \"~ts\" does not exist and is being expanded to \"~ts()\","
-              " please use parentheses to remove the ambiguity or change the variable name", [Name, Name]),
-          elixir_errors:warn(?line(Meta), ?key(E, file), Message),
+          case ?key(E, match_vars) of
+            warn ->
+              Message =
+                io_lib:format("variable \"~ts\" does not exist and is being expanded to \"~ts()\","
+                  " please use parentheses to remove the ambiguity or change the variable name", [Name, Name]),
+              elixir_errors:warn(?line(Meta), ?key(E, file), Message);
+            apply ->
+              ok
+          end,
           expand({Name, Meta, []}, E)
       end
   end;
@@ -391,6 +407,7 @@ expand({{'.', DotMeta, [Left, Right]}, Meta, Args}, E)
 %% Anonymous calls
 
 expand({{'.', DotMeta, [Expr]}, Meta, Args}, E) when is_list(Args) ->
+  assert_no_match_or_guard_scope(Meta, "anonymous call", E),
   {EExpr, EE} = expand(Expr, E),
   if
     is_atom(EExpr) ->
@@ -434,11 +451,12 @@ expand(Function, E) when is_function(Function) ->
 expand(PidOrRef, E) when is_pid(PidOrRef); is_reference(PidOrRef) ->
   case ?key(E, function) of
     nil ->
-      PidOrRef;
+      {PidOrRef, E};
     Function ->
       %% TODO: Make me an error on 2.0
       elixir_errors:form_warn([], ?key(E, file), ?MODULE,
-                              {invalid_pid_or_ref_in_function, PidOrRef, Function})
+                              {invalid_pid_or_ref_in_function, PidOrRef, Function}),
+      {PidOrRef, E}
   end;
 
 expand(Other, E) when is_number(Other); is_atom(Other); is_binary(Other) ->
@@ -554,7 +572,7 @@ expand_args(Args, E) ->
 
 %% Match/var helpers
 
-var_kind(Meta, Kind) ->
+var_context(Meta, Kind) ->
   case lists:keyfind(counter, 1, Meta) of
     {counter, Counter} -> Counter;
     false -> Kind
@@ -563,7 +581,6 @@ var_kind(Meta, Kind) ->
 %% Case
 
 expand_case(true, Meta, Expr, Opts, E) ->
-  assert_no_match_or_guard_scope(Meta, 'case', E),
   {EExpr, EE} = expand(Expr, E),
   {EOpts, EO} = elixir_clauses:'case'(Meta, Opts, EE),
   ROpts =
@@ -789,11 +806,11 @@ assert_function_scope(_Meta, _Kind, #{function := Function}) -> Function.
 assert_no_match_or_guard_scope(Meta, Kind, E) ->
   assert_no_match_scope(Meta, Kind, E),
   assert_no_guard_scope(Meta, Kind, E).
-assert_no_match_scope(Meta, _Kind, #{context := match, file := File}) ->
-  form_error(Meta, File, ?MODULE, invalid_pattern_in_match);
+assert_no_match_scope(Meta, Kind, #{context := match, file := File}) ->
+  form_error(Meta, File, ?MODULE, {invalid_pattern_in_match, Kind});
 assert_no_match_scope(_Meta, _Kind, _E) -> [].
-assert_no_guard_scope(Meta, _Kind, #{context := guard, file := File}) ->
-  form_error(Meta, File, ?MODULE, invalid_expr_in_guard);
+assert_no_guard_scope(Meta, Kind, #{context := guard, file := File}) ->
+  form_error(Meta, File, ?MODULE, {invalid_expr_in_guard, Kind});
 assert_no_guard_scope(_Meta, _Kind, _E) -> [].
 
 %% Here we look into the Clauses "optimistically", that is, we don't check for
@@ -814,6 +831,30 @@ assert_no_underscore_clause_in_cond(_Other, _E) ->
 
 %% Warnings
 
+warn_underscored_var_repeat(Meta, Name, Kind, E) ->
+  Warn = should_warn(Meta),
+  case atom_to_list(Name) of
+    "_" ++ _ when Warn ->
+      elixir_errors:form_warn(Meta, ?key(E, file), ?MODULE, {underscored_var_repeat, Name, Kind});
+    _ ->
+      ok
+  end.
+
+warn_underscored_var_access(Meta, Name, Kind, E) ->
+  Warn = should_warn(Meta),
+  case atom_to_list(Name) of
+    "_" ++ _ when Warn ->
+      elixir_errors:form_warn(Meta, ?key(E, file), ?MODULE, {underscored_var_access, Name, Kind});
+    _ ->
+      ok
+  end.
+
+context_info(Kind) when Kind == nil; is_integer(Kind) -> "";
+context_info(Kind) -> io_lib:format(" (context ~ts)", [elixir_aliases:inspect(Kind)]).
+
+should_warn(Meta) ->
+  lists:keyfind(generated, 1, Meta) /= {generated, true}.
+
 format_error({useless_literal, Term}) ->
   io_lib:format("code block contains unused literal ~ts "
                 "(remove the literal or assign it to _ to avoid warnings)",
@@ -866,14 +907,14 @@ format_error({undefined_var, Name, Kind}) ->
   Message =
     "expected variable \"~ts\"~ts to expand to an existing variable "
     "or be part of a match",
-  io_lib:format(Message, [Name, elixir_erl_var:context_info(Kind)]);
+  io_lib:format(Message, [Name, context_info(Kind)]);
 format_error(underscore_in_cond) ->
   "unbound variable _ inside \"cond\". If you want the last clause to always match, "
     "you probably meant to use: true ->";
-format_error(invalid_expr_in_guard) ->
-  "invalid expression in guard";
-format_error(invalid_pattern_in_match) ->
-  "invalid pattern in match";
+format_error({invalid_expr_in_guard, Kind}) ->
+  io_lib:format("invalid expression in guard, ~ts is not allowed in guards", [Kind]);
+format_error({invalid_pattern_in_match, Kind}) ->
+  io_lib:format("invalid pattern in match, ~ts is not allowed in matches", [Kind]);
 format_error({invalid_expr_in_scope, Scope, Kind}) ->
   io_lib:format("cannot invoke ~ts outside ~ts", [Kind, Scope]);
 format_error({invalid_alias, Expr}) ->
@@ -918,4 +959,15 @@ format_error({options_are_not_keyword, Kind, Opts}) ->
   io_lib:format("invalid options for ~s, expected a keyword list, got: ~ts",
                 [Kind, 'Elixir.Macro':to_string(Opts)]);
 format_error({undefined_function, Name, Args}) ->
-  io_lib:format("undefined function ~ts/~B", [Name, length(Args)]).
+  io_lib:format("undefined function ~ts/~B", [Name, length(Args)]);
+format_error({underscored_var_repeat, Name, Kind}) ->
+  io_lib:format("the underscored variable \"~ts\"~ts appears more than once in a "
+                "match. This means the pattern will only match if all \"~ts\" bind "
+                "to the same value. If this is the intended behaviour, please "
+                "remove the leading underscore from the variable name, otherwise "
+                "give the variables different names", [Name, context_info(Kind), Name]);
+format_error({underscored_var_access, Name, Kind}) ->
+  io_lib:format("the underscored variable \"~ts\"~ts is used after being set. "
+                "A leading underscore indicates that the value of the variable "
+                "should be ignored. If this is intended please rename the "
+                "variable to remove the underscore", [Name, context_info(Kind)]).

lib/elixir/src/elixir_locals.erl

@@ -1,7 +1,7 @@
 %% Module responsible for tracking invocations of module calls.
 -module(elixir_locals).
 -export([
-  setup/1, cleanup/1, cache_env/1, get_cached_env/1,
+  setup/1, cleanup/1, cache_env/1, cache_env/2, get_cached_env/1,
   record_local/2, record_local/3, record_import/4,
   record_definition/3, record_defaults/4,
   ensure_no_import_conflict/3, warn_unused_local/3, format_error/1
@@ -9,6 +9,7 @@
 
 -include("elixir.hrl").
 -define(attr, {elixir, locals_tracker}).
+-define(cache, {elixir, cache_env}).
 -define(tracker, 'Elixir.Module.LocalsTracker').
 
 setup(Module) ->
@@ -56,18 +57,29 @@ if_tracker(Module, Default, Callback) ->
 
 %% CACHING
 
-cache_env(#{module := Module, line := Line} = E) ->
-  try ets:lookup_element(elixir_module:data_table(Module), ?attr, 2) of
-    Pid ->
-      {Pid, {Line, ?tracker:cache_env(Pid, E#{line := nil, vars := []})}}
-  catch
-    error:badarg ->
-      {Escaped, _} = elixir_quote:escape(E#{vars := []}, false),
-      Escaped
-  end.
+cache_env(#{module := Module} = E) ->
+  cache_env(elixir_module:data_table(Module), E).
 
-get_cached_env({Pid, {Line, Ref}}) -> (?tracker:get_cached_env(Pid, Ref))#{line := Line};
-get_cached_env(Env) -> Env.
+cache_env(Table, #{line := Line} = E) ->
+  Cache = E#{line := nil, vars := []},
+
+  Pos =
+    case ets:lookup(Table, ?cache) of
+      [{_, Key, Cache}] ->
+        Key;
+      [{_, PrevKey, _}] ->
+        Key = PrevKey + 1,
+        ets:insert(Table, {{cache_env, Key}, Cache}),
+        ets:insert(Table, {?cache, Key, Cache}),
+        Key
+    end,
+
+  {Table, {Line, Pos}}.
+
+get_cached_env({Table, {Line, Pos}}) ->
+  (ets:lookup_element(Table, {cache_env, Pos}, 2))#{line := Line};
+get_cached_env(Env) ->
+  Env.
 
 %% ERROR HANDLING
 

lib/elixir/src/elixir_module.erl

@@ -210,7 +210,8 @@ build(Line, File, Module, Lexical) ->
     {typep, [], true, nil},
 
     % Internal
-    {{elixir, impls}, []}
+    {{elixir, impls}, []},
+    {{elixir, cache_env}, 0, nil}
   ]),
 
   Persisted = [behaviour, on_load, compile, external_resource, dialyzer, vsn],

lib/elixir/src/elixir_overridable.erl

@@ -15,15 +15,15 @@ overridable(Module, Value) ->
 
 super(Meta, File, Module, Function) ->
   case store(Module, Function, true) of
-    {ok, Name} ->
-      Name;
+    {_, _} = KindName ->
+      KindName;
     error ->
       elixir_errors:form_error(Meta, File, ?MODULE, {no_super, Module, Function})
   end.
 
 store_pending(Module) ->
   [begin
-    {ok, _} = store(Module, Pair, false),
+    {_, _} = store(Module, Pair, false),
     Pair
    end || {Pair, {_, _, _, false}} <- maps:to_list(overridable(Module)),
           not 'Elixir.Module':'defines?'(Module, Pair)].
@@ -42,7 +42,7 @@ store(Module, Function, Hidden) ->
           false ->
             {Kind, Name, Arity, Clauses};
           true when Kind == defmacro; Kind == defmacrop ->
-            {defp, name(Name, Count), Arity + 1, rewrite_clauses(Clauses)};
+            {defmacrop, name(Name, Count), Arity, Clauses};
           true ->
             {defp, name(Name, Count), Arity, Clauses}
         end,
@@ -62,15 +62,11 @@ store(Module, Function, Hidden) ->
           ok
       end,
 
-      {ok, Tuple};
+      {FinalKind, FinalName};
     error ->
       error
   end.
 
-rewrite_clauses(Clauses) ->
-  [{Meta, [{'__CALLER__', [], nil} | Args], Guards, Body} ||
-   {Meta, Args, Guards, Body} <- Clauses].
-
 name(Name, Count) when is_integer(Count) ->
   list_to_atom(atom_to_list(Name) ++ " (overridable " ++ integer_to_list(Count) ++ ")").
 
@@ -83,6 +79,6 @@ format_error({no_super, Module, {Name, Arity}}) ->
     [Name, Arity, elixir_aliases:inspect(Module), Joined]).
 
 format_fa({Name, Arity}) ->
-  A = atom_to_binary(Name, utf8),
+  A = 'Elixir.Inspect.Function':escape_name(Name),
   B = integer_to_binary(Arity),
   <<A/binary, $/, B/binary>>.

lib/elixir/src/elixir_with.erl

@@ -1,105 +0,0 @@
--module(elixir_with).
--export([expand/3]).
--include("elixir.hrl").
-
-expand(Meta, Args, Env) ->
-  {Exprs, Opts} =
-    case elixir_utils:split_last(Args) of
-      {_, LastArg} = SplitResult when is_list(LastArg) ->
-        SplitResult;
-      _ ->
-        {Args, []}
-    end,
-
-  {DoExpr, OtherOpts1} =
-    case lists:keytake(do, 1, Opts) of
-      {value, {do, DoValue}, RestOpts1} ->
-        {DoValue, RestOpts1};
-      false ->
-        elixir_errors:form_error(Meta, ?key(Env, file), elixir_expand, {missing_option, 'with', [do]})
-    end,
-
-  {ElseExpr, OtherOpts2} =
-    case lists:keytake(else, 1, OtherOpts1) of
-      {value, {else, ElseValue}, RestOpts2} ->
-        assert_clauses(Meta, ElseValue, Env),
-        {ElseValue, RestOpts2};
-      false ->
-        {nil, OtherOpts1}
-    end,
-
-  case OtherOpts2 of
-    [{Key, _} | _] ->
-      elixir_errors:form_error(Meta, ?key(Env, file), elixir_clauses, {unexpected_option, with, Key});
-    [] ->
-      ok
-  end,
-
-  ResultCase =
-    case ElseExpr of
-      nil ->
-        {MainCase, _} = build_main_case(Exprs, DoExpr, fun(Ret) -> Ret end, false),
-        MainCase;
-      _ ->
-        Wrapper = fun(Ret) -> {error, Ret} end,
-        case build_main_case(Exprs, {ok, DoExpr}, Wrapper, false) of
-          {MainCase, false} ->
-            Message =
-              "\"else\" clauses will never match"
-              " because all patterns in \"with\" will always match",
-            elixir_errors:warn(?line(Meta), ?key(Env, file), Message),
-            {{'.', Meta, [erlang, element]}, Meta, [MainCase, 2]};
-          {MainCase, true} ->
-            build_else_case(Meta, MainCase, ElseExpr, Wrapper)
-        end
-    end,
-  elixir_expand:expand(ResultCase, Env).
-
-%% Helpers
-
-assert_clauses(_Meta, [], _Env) ->
-  ok;
-assert_clauses(Meta, [{'->', _, [_, _]} | Rest], Env) ->
-  assert_clauses(Meta, Rest, Env);
-assert_clauses(Meta, _Other, Env) ->
-  elixir_errors:form_error(Meta, ?key(Env, file), elixir_clauses, {bad_or_missing_clauses, {with, else}}).
-
-build_main_case([{'<-', Meta, [{Name, _, Ctx}, _] = Args} | Rest], DoExpr, Wrapper, HasMatch)
-    when is_atom(Name) andalso is_atom(Ctx) ->
-  build_main_case([{'=', Meta, Args} | Rest], DoExpr, Wrapper, HasMatch);
-build_main_case([{'<-', Meta, [Left, Right]} | Rest], DoExpr, Wrapper, _HasMatch) ->
-  {InnerCase, true} = build_main_case(Rest, DoExpr, Wrapper, true),
-  Generated = ?generated(Meta),
-  Other = {other, Generated, ?var_context},
-  Clauses = [
-    {'->', Generated, [[Left], InnerCase]},
-    {'->', Generated, [[Other], Wrapper(Other)]}
-  ],
-  {{'case', [{export_vars, false} | Meta], [Right, [{do, Clauses}]]}, true};
-build_main_case([Expr | Rest], DoExpr, Wrapper, HasMatch) ->
-  {InnerCase, InnerHasMatch} = build_main_case(Rest, DoExpr, Wrapper, HasMatch),
-  {{'__block__', [], [Expr, InnerCase]}, InnerHasMatch};
-build_main_case([], DoExpr, _Wrapper, HasMatch) ->
-  {DoExpr, HasMatch}.
-
-build_else_case(Meta, MainCase, Clauses, Wrapper) ->
-  Generated = ?generated(Meta),
-
-  Return = {return, Generated, ?var_context},
-  ReturnClause = {'->', Generated, [[{ok, Return}], Return]},
-
-  Other = {other, Generated, ?var_context},
-  RaiseError = {{'.', Generated, [erlang, error]}, Meta, [{with_clause, Other}]},
-  RaiseErrorClause = {'->', Generated, [[Wrapper(Other)], RaiseError]},
-
-  ClauseWrapper = fun(Clause) -> wrap_clause_pattern(Clause, Wrapper) end,
-  ResultClauses = [ReturnClause] ++ lists:map(ClauseWrapper, Clauses) ++ [RaiseErrorClause],
-  {'case', [{export_vars, false} | Meta], [MainCase, [{do, ResultClauses}]]}.
-
-wrap_clause_pattern({'->', Meta, [[Left], Right]}, Wrapper) ->
-  {'->', Meta, [[wrap_pattern(Left, Wrapper)], Right]}.
-
-wrap_pattern({'when', Meta, [Left, Right]}, Wrapper) ->
-  {'when', Meta, [Wrapper(Left), Right]};
-wrap_pattern(Expr, Wrapper) ->
-  Wrapper(Expr).

lib/elixir/test/elixir/base_test.exs

@@ -750,4 +750,30 @@ defmodule BaseTest do
   test "hex_decode32!/2 with :mixed case and ignoring padding" do
     assert "fo" == hex_decode32!("cPNg", case: :mixed, padding: false)
   end
+
+  test "encode then decode is identity" do
+    for {encode, decode} <- [{&encode16/2, &decode16!/2},
+                             {&encode32/2, &decode32!/2},
+                             {&hex_encode32/2, &hex_decode32!/2},
+                             {&encode64/2, &decode64!/2},
+                             {&url_encode64/2, &url_decode64!/2}],
+        encode_case <- [:upper, :lower],
+        decode_case <- [:upper, :lower, :mixed],
+        (encode_case == decode_case) or (decode_case == :mixed),
+        pad? <- [true, false],
+        len <- 0..256 do
+      data =
+        0
+        |> :lists.seq(len - 1)
+        |> Enum.shuffle()
+        |> IO.iodata_to_binary()
+
+      expected =
+        data
+        |> encode.([case: encode_case, pad: pad?])
+        |> decode.([case: decode_case, pad: pad?])
+
+      assert data == expected, "identity did not match for #{inspect data} when #{inspect encode} (#{encode_case})"
+    end
+  end
 end

lib/elixir/test/elixir/calendar/date_range_test.exs

@@ -1,5 +1,5 @@
 Code.require_file "../test_helper.exs", __DIR__
-Code.require_file "../fixtures/calendar/julian.exs", __DIR__
+Code.require_file "../fixtures/calendar/holocene.exs", __DIR__
 
 defmodule Date.RangeTest do
   use ExUnit.Case, async: true
@@ -13,7 +13,7 @@ defmodule Date.RangeTest do
       assert Enum.member?(@asc_range, ~D[2000-01-01])
       assert Enum.member?(@asc_range, ~D[2001-01-01])
       refute Enum.member?(@asc_range, ~D[2002-01-01])
-      refute Enum.member?(@asc_range, Calendar.Julian.date(1999, 12, 19))
+      refute Enum.member?(@asc_range, Calendar.Holocene.date(12000, 1, 1))
     end
 
     test "for descending range" do
@@ -21,7 +21,7 @@ defmodule Date.RangeTest do
       assert Enum.member?(@desc_range, ~D[2000-01-01])
       assert Enum.member?(@desc_range, ~D[2001-01-01])
       refute Enum.member?(@desc_range, ~D[1999-01-01])
-      refute Enum.member?(@asc_range, Calendar.Julian.date(1999, 12, 19))
+      refute Enum.member?(@asc_range, Calendar.Holocene.date(12000, 1, 1))
     end
   end
 
@@ -49,16 +49,16 @@ defmodule Date.RangeTest do
 
   test "both dates must have matching calendars" do
     first = ~D[2000-01-01]
-    last = Calendar.Julian.date(2001, 01, 01)
+    last = Calendar.Holocene.date(12001, 1, 1)
 
     assert_raise ArgumentError, "both dates must have matching calendars", fn ->
       Date.range(first, last)
     end
   end
 
-  test "accepts equal but not Calendar.ISO calendars" do
-    first = Calendar.Julian.date(2000, 01, 01)
-    last = Calendar.Julian.date(2001, 01, 01)
+  test "accepts equal but non Calendar.ISO calendars" do
+    first = Calendar.Holocene.date(12000, 1, 1)
+    last = Calendar.Holocene.date(12001, 1, 1)
     range = Date.range(first, last)
     assert range
     assert first in range

lib/elixir/test/elixir/calendar_test.exs

@@ -1,5 +1,5 @@
 Code.require_file "test_helper.exs", __DIR__
-Code.require_file "fixtures/calendar/julian.exs", __DIR__
+Code.require_file "fixtures/calendar/holocene.exs", __DIR__
 
 defmodule FakeCalendar do
   def date_to_string(_, _, _), do: "boom"
@@ -37,7 +37,10 @@ defmodule DateTest do
 
   test "compare/2 across calendars" do
     date1 = ~D[2000-01-01]
-    date2 = Calendar.Julian.date(2000, 01, 01)
+    date2 = Calendar.Holocene.date(12000, 01, 01)
+    assert Date.compare(date1, date2) == :eq
+
+    date2 = Calendar.Holocene.date(12001, 01, 01)
     assert Date.compare(date1, date2) == :lt
     assert Date.compare(date2, date1) == :gt
   end
@@ -53,14 +56,14 @@ defmodule DateTest do
   end
 
   test "convert/2" do
-    assert Date.convert(~D[2000-01-01], Calendar.Julian) ==
-           {:ok, Calendar.Julian.date(1999, 12, 19)}
-    assert ~D[2000-01-01] |> Date.convert!(Calendar.Julian) |> Date.convert!(Calendar.ISO) ==
+    assert Date.convert(~D[2000-01-01], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.date(12000, 01, 01)}
+    assert ~D[2000-01-01] |> Date.convert!(Calendar.Holocene) |> Date.convert!(Calendar.ISO) ==
            ~D[2000-01-01]
     assert Date.convert(~D[2016-02-03], FakeCalendar) ==
            {:error, :incompatible_calendars}
-    assert Date.convert(~N[2000-01-01 00:00:00], Calendar.Julian) ==
-           {:ok, Calendar.Julian.date(1999, 12, 19)}
+    assert Date.convert(~N[2000-01-01 00:00:00], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.date(12000, 01, 01)}
   end
 
   test "diff/2" do
@@ -68,7 +71,7 @@ defmodule DateTest do
     assert Date.diff(~D[2000-01-01], ~D[2000-01-31]) == -30
 
     date1 = ~D[2000-01-01]
-    date2 = Calendar.Julian.date(2000, 01, 01)
+    date2 = Calendar.Holocene.date(12000, 01, 14)
     assert Date.diff(date1, date2) == -13
     assert Date.diff(date2, date1) == 13
   end
@@ -145,33 +148,40 @@ defmodule NaiveDateTimeTest do
 
   test "add/2 with other calendars" do
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.add(10, :second) ==
-           %NaiveDateTime{calendar: Calendar.Julian, year: 1999, month: 12, day: 19,
+           %NaiveDateTime{calendar: Calendar.Holocene, year: 12000, month: 1, day: 1,
                           hour: 12, minute: 34, second: 25, microsecond: {123456, 6}}
   end
 
   test "diff/2 with other calendars" do
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.add(10, :second)
            |> NaiveDateTime.diff(~N[2000-01-01 12:34:15.123456]) == 10
   end
 
   test "convert/2" do
-    assert NaiveDateTime.convert(~N[2000-01-01 12:34:15.123400], Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1999, 12, 19, 12, 34, 15, 123400)}
+    assert NaiveDateTime.convert(~N[2000-01-01 12:34:15.123400], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(12000, 1, 1, 12, 34, 15, {123400, 6})}
+
+    assert ~N[2000-01-01 12:34:15]
+           |> NaiveDateTime.convert!(Calendar.Holocene)
+           |> NaiveDateTime.convert!(Calendar.ISO) ==
+           ~N[2000-01-01 12:34:15]
+
     assert ~N[2000-01-01 12:34:15.123456]
-           |> NaiveDateTime.convert!(Calendar.Julian)
+           |> NaiveDateTime.convert!(Calendar.Holocene)
            |> NaiveDateTime.convert!(Calendar.ISO) ==
            ~N[2000-01-01 12:34:15.123456]
+
     assert NaiveDateTime.convert(~N[2016-02-03 00:00:01], FakeCalendar) ==
            {:error, :incompatible_calendars}
 
-    assert NaiveDateTime.convert(~N[1970-01-01 00:00:00], Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1969, 12, 19, 0, 0, 0, {0, 0})}
-    assert NaiveDateTime.convert(DateTime.from_unix!(0, :seconds), Calendar.Julian) ==
-           {:ok, Calendar.Julian.naive_datetime(1969, 12, 19, 0, 0, 0, {0, 0})}
+    assert NaiveDateTime.convert(~N[1970-01-01 00:00:00], Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(11970, 1, 1, 0, 0, 0, {0, 0})}
+    assert NaiveDateTime.convert(DateTime.from_unix!(0, :seconds), Calendar.Holocene) ==
+           {:ok, Calendar.Holocene.naive_datetime(11970, 1, 1, 0, 0, 0, {0, 0})}
   end
 end
 
@@ -260,6 +270,31 @@ defmodule DateTimeTest do
     assert DateTime.compare(datetime2, datetime1) == :gt
   end
 
+  test "convert/2" do
+    datetime_iso = %DateTime{year: 2000, month: 2, day: 29, zone_abbr: "CET",
+                             hour: 23, minute: 0, second: 7, microsecond: {0, 0},
+                             utc_offset: 3600, std_offset: 0, time_zone: "Europe/Warsaw"}
+    datetime_hol = %DateTime{year: 12000, month: 2, day: 29, zone_abbr: "CET",
+                             hour: 23, minute: 0, second: 7, microsecond: {0, 0},
+                             utc_offset: 3600, std_offset: 0, time_zone: "Europe/Warsaw",
+                             calendar: Calendar.Holocene}
+
+    assert DateTime.convert(datetime_iso, Calendar.Holocene) == {:ok, datetime_hol}
+
+    assert datetime_iso
+           |> DateTime.convert!(Calendar.Holocene)
+           |> DateTime.convert!(Calendar.ISO) ==
+           datetime_iso
+
+    assert %{datetime_iso | microsecond: {123, 6}}
+           |> DateTime.convert!(Calendar.Holocene)
+           |> DateTime.convert!(Calendar.ISO) ==
+           %{datetime_iso | microsecond: {123, 6}}
+
+    assert DateTime.convert(datetime_iso, FakeCalendar) ==
+           {:error, :incompatible_calendars}
+  end
+
   test "from_iso8601/1 with tz offsets" do
     assert DateTime.from_iso8601("2017-06-02T14:00:00+01:00") |> elem(1) ==
            %DateTime{year: 2017, month: 6, day: 2, zone_abbr: "UTC",

lib/elixir/test/elixir/enum_test.exs

@@ -59,6 +59,19 @@ defmodule EnumTest do
     assert Enum.chunk([1, 2, 3, 4, 5], 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
+  test "chunk_every/2" do
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 2) == [[1, 2], [3, 4], [5]]
+  end
+
+  test "chunk_every/4" do
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 2, 2, [6]) == [[1, 2], [3, 4], [5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard) == [[1, 2, 3], [3, 4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 2, 3, :discard) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, []) == [[1, 2, 3], [3, 4, 5], [5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5, 6], 3, 3, []) == [[1, 2, 3], [4, 5, 6]]
+    assert Enum.chunk_every([1, 2, 3, 4, 5], 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
+  end
+
   test "chunk_by/2" do
     assert Enum.chunk_by([1, 2, 2, 3, 4, 4, 6, 7, 7], &(rem(&1, 2) == 1)) == [[1], [2, 2], [3], [4, 4, 6], [7, 7]]
     assert Enum.chunk_by([1, 2, 3, 4], fn _ -> true end) == [[1, 2, 3, 4]]
@@ -66,12 +79,15 @@ defmodule EnumTest do
     assert Enum.chunk_by([1], fn _ -> true end) == [[1]]
   end
 
-  test "chunk_by/4" do
+  test "chunk_while/4" do
     chunk_fun = fn i, acc ->
-      if rem(i, 2) == 0 do
-        {:cont, Enum.reverse([i | acc]), []}
-      else
-        {:cont, [i | acc]}
+      cond do
+        i > 10 ->
+          {:halt, acc}
+        rem(i, 2) == 0 ->
+          {:cont, Enum.reverse([i | acc]), []}
+        true ->
+          {:cont, [i | acc]}
       end
     end
 
@@ -80,12 +96,16 @@ defmodule EnumTest do
       acc -> {:cont, Enum.reverse(acc), []}
     end
 
-    assert Enum.chunk_by([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], chunk_fun, after_fun) ==
+    assert Enum.chunk_while([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], chunk_fun, after_fun) ==
            [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert Enum.chunk_by(0..10, [], chunk_fun, after_fun) ==
+    assert Enum.chunk_while(0..9, [], chunk_fun, after_fun) ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9]]
+    assert Enum.chunk_while(0..10, [], chunk_fun, after_fun) ==
            [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert Enum.chunk_by(0..11, [], chunk_fun, after_fun) ==
-           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11]]
+    assert Enum.chunk_while(0..11, [], chunk_fun, after_fun) ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
+    assert Enum.chunk_while([5, 7, 9, 11], [], chunk_fun, after_fun) ==
+           [[5, 7, 9]]
   end
 
   test "concat/1" do
@@ -833,6 +853,18 @@ defmodule EnumTest.Range do
     assert Enum.chunk(1..5, 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
+  test "chunk_every/2" do
+    assert Enum.chunk_every(1..5, 2) == [[1, 2], [3, 4], [5]]
+  end
+
+  test "chunk_every/4" do
+    assert Enum.chunk_every(1..5, 2, 2) == [[1, 2], [3, 4], [5]]
+    assert Enum.chunk_every(1..6, 3, 2, :discard) == [[1, 2, 3], [3, 4, 5]]
+    assert Enum.chunk_every(1..6, 2, 3, :discard) == [[1, 2], [4, 5]]
+    assert Enum.chunk_every(1..6, 3, 2, []) == [[1, 2, 3], [3, 4, 5], [5, 6]]
+    assert Enum.chunk_every(1..5, 4, 4, 6..10) == [[1, 2, 3, 4], [5, 6, 7, 8]]
+  end
+
   test "chunk_by/2" do
     assert Enum.chunk_by(1..4, fn _ -> true end) == [[1, 2, 3, 4]]
     assert Enum.chunk_by(1..4, &(rem(&1, 2) == 1)) == [[1], [2], [3], [4]]
@@ -1566,3 +1598,13 @@ defmodule EnumTest.SideEffects do
     assert Enum.take(iterator, 0) == []
   end
 end
+
+defmodule EnumTest.Function do
+  use ExUnit.Case, async: true
+
+  test "raises Protocol.UndefinedError for funs of wrong arity" do
+    assert_raise Protocol.UndefinedError, fn ->
+      Enum.to_list(fn -> nil end)
+    end
+  end
+end

lib/elixir/test/elixir/file_test.exs

@@ -460,6 +460,11 @@ defmodule FileTest do
       end
     end
 
+    test "cp_r raises on path with null byte" do
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.cp_r("source", "foo\0bar") end
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.cp_r("foo\0bar", "dest") end
+    end
+
     test "cp_r with src file and dest file" do
       src  = fixture_path("file.txt")
       dest = tmp_path("sample.txt")
@@ -1113,6 +1118,10 @@ defmodule FileTest do
       refute File.exists?(fixture)
     end
 
+    test "rm_rf raises on path with null byte" do
+      assert_raise ArgumentError, ~r/null byte/, fn -> File.rm_rf("foo\0bar") end
+    end
+
     test "rm_rf with symlink" do
       from = tmp_path("tmp/from")
       to   = tmp_path("tmp/to")

lib/elixir/test/elixir/fixtures/calendar/holocene.exs

@@ -0,0 +1,44 @@
+defmodule Calendar.Holocene do
+  # This calendar is used to test conversions between calendars.
+  # It implements the Holocene calendar, which is based on the
+  # Propleptic Gregorian calendar with every year + 10000.
+
+  def date(year, month, day) do
+    %Date{year: year, month: month, day: day, calendar: __MODULE__}
+  end
+
+  def naive_datetime(year, month, day, hour, minute, second, microsecond \\ {0, 0}) do
+    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second,
+                   microsecond: microsecond, calendar: __MODULE__}
+  end
+
+  def date_to_string(year, month, day) do
+    "#{year}-#{month}-#{day} (HE)"
+  end
+
+  def naive_datetime_to_string(year, month, day, hour, minute, second, microsecond) do
+    "#{year}-#{month}-#{day} #{Calendar.ISO.time_to_string(hour, minute, second, microsecond)} (HE)"
+  end
+
+  defdelegate time_to_string(hour, minute, second, microsecond), to: Calendar.ISO
+
+  def day_rollover_relative_to_midnight_utc(), do: {0, 1}
+
+  def naive_datetime_from_iso_days(entry) do
+    {year, month, day, hour, minute, second, microsecond} =
+      Calendar.ISO.naive_datetime_from_iso_days(entry)
+    {year + 10000, month, day, hour, minute, second, microsecond}
+  end
+
+  def naive_datetime_to_iso_days(year, month, day, hour, minute, second, microsecond) do
+    Calendar.ISO.naive_datetime_to_iso_days(year - 10000, month, day, hour, minute, second, microsecond)
+  end
+
+  defdelegate time_from_day_fraction(day_fraction), to: Calendar.ISO
+
+  defdelegate time_to_day_fraction(hour, minute, second, microsecond), to: Calendar.ISO
+
+  defdelegate leap_year?(year), to: Calendar.ISO
+
+  defdelegate days_in_month(year, month), to: Calendar.ISO
+end

lib/elixir/test/elixir/fixtures/calendar/julian.exs

@@ -1,99 +0,0 @@
-defmodule Calendar.Julian do
-  # This calendar is used to test conversions between calendars.
-  # It implements the Julian calendar.
-
-  import Integer, only: [floor_div: 2]
-
-  def date_to_string(year, month, day) do
-    "#{year}-#{month}-#{day} (O.S.)"
-  end
-
-  def naive_datetime_to_string(year, month, day, hour, minute, second, microsecond) do
-    "#{year}-#{month}-#{day} #{Calendar.ISO.time_to_string(hour, minute, second, microsecond)} (O.S.)"
-  end
-
-  def time_to_string(hour, minute, second, microsecond) do
-    Calendar.ISO.time_to_string(hour, minute, second, microsecond)
-  end
-
-  def date(year, month, day) do
-    %Date{year: year, month: month, day: day, calendar: Calendar.Julian}
-  end
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond \\ {0, 0})
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond) when is_integer(microsecond) do
-    naive_datetime(year, month, day, hour, minute, second, {microsecond, 6})
-  end
-
-  def naive_datetime(year, month, day, hour, minute, second, microsecond) do
-    %NaiveDateTime{year: year, month: month, day: day, hour: hour, minute: minute, second: second, microsecond: microsecond, calendar: Calendar.Julian}
-  end
-
-  def day_rollover_relative_to_midnight_utc(), do: {0, 1}
-
-  def naive_datetime_from_rata_die({days, day_fraction}) do
-    {year, month, day} = date_from_rata_die(days)
-    {hour, minute, second, microsecond} = time_from_day_fraction(day_fraction)
-    {year, month, day, hour, minute, second, microsecond}
-  end
-
-  defp date_from_rata_die(days) do
-    approx = floor_div((4 * (days - epoch())) + 1464, 1461)
-    year = if approx <= 0, do: approx - 1, else: approx
-    prior_days = days - date_to_rata_die(year, 1, 1)
-    correction = cond do
-      days < date_to_rata_die(year, 3, 1) -> 0
-      leap_year?(year) ->  1
-      true -> 2
-    end
-    month = floor_div(12*(prior_days + correction) + 373, 367)
-    day = 1 + days - date_to_rata_die(year, month, 1)
-    {year, month, day}
-  end
-
-  def naive_datetime_to_rata_die(year, month, day, hour, minute, second, microsecond) do
-    days = date_to_rata_die(year, month, day)
-    day_fraction = time_to_day_fraction(hour, minute, second, microsecond)
-    {days, day_fraction}
-  end
-
-  def time_from_day_fraction(day_fraction), do: Calendar.ISO.time_from_day_fraction(day_fraction)
-
-  def time_to_day_fraction(hour, minute, second, microsecond) when is_integer(microsecond) do
-    time_to_day_fraction(hour, minute, second, {microsecond, 6})
-  end
-
-  def time_to_day_fraction(hour, minute, second, microsecond) do
-    Calendar.ISO.time_to_day_fraction(hour, minute, second, microsecond)
-  end
-
-  def leap_year?(year) when is_integer(year) and year >= 0 do
-    rem(year, 4) == if year > 0, do: 0, else: 3
-  end
-
-  def days_in_month(year, month)
-  def days_in_month(year, 2) do
-    if leap_year?(year), do: 29, else: 28
-  end
-  def days_in_month(_, month) when month in [4, 6, 9, 11], do: 30
-  def days_in_month(_, month) when month in 1..12, do: 31
-
-  defp date_to_rata_die(year, month, day) do
-    year = if year < 0, do: year + 1, else: year
-
-    epoch() - 1 + (365 * (year - 1)) + floor_div(year - 1, 4) +
-    floor_div(367 * month - 362, 12) +
-    adjustment_for_leap_year(year, month) + day
-  end
-
-  defp epoch(), do: -1
-
-  defp adjustment_for_leap_year(year, month) do
-    cond do
-      month <= 2 -> 0
-      leap_year?(year) -> -1
-      true -> -2
-    end
-  end
-end

lib/elixir/test/elixir/inspect_test.exs

@@ -452,6 +452,14 @@ defmodule Inspect.MapTest do
            inspect(%Inspect.Error{message: "#{msg}"})
   end
 
+  test "bad implementation safe disables colors" do
+    msg = "got KeyError with message \\\"key :unknown not found in: " <>
+          "%{__struct__: Inspect.MapTest.Failing, key: 0}\\\" while " <>
+          "inspecting %{__struct__: Inspect.MapTest.Failing, key: 0}"
+
+    assert inspect(%Failing{}, syntax_colors: [atom: [:green]]) =~ msg
+  end
+
   test "exception" do
     assert inspect(%RuntimeError{message: "runtime error"}) ==
            "%RuntimeError{message: \"runtime error\"}"

lib/elixir/test/elixir/kernel/errors_test.exs

@@ -15,6 +15,12 @@ defmodule Kernel.ErrorsTest do
       '[foo: \u200B]\noops'
   end
 
+  test "invalid __CALLER__" do
+    assert_compile_fail CompileError,
+      "nofile:1: variable '__CALLER__' is unbound",
+      'defmodule Sample do def hello do __CALLER__ end end'
+  end
+
   test "invalid quoted token" do
     assert_compile_fail SyntaxError,
       "nofile:1: syntax error before: \"world\"",

lib/elixir/test/elixir/kernel/expansion_test.exs

@@ -169,10 +169,6 @@ defmodule Kernel.ExpansionTest do
       assert expand(quote do: __DIR__) == __DIR__
     end
 
-    test "__CALLER__" do
-      assert expand(quote do: __CALLER__) == quote do: __CALLER__
-    end
-
     test "__ENV__" do
       env = %{__ENV__ | line: 0}
       assert expand_env(quote(do: __ENV__), env) ==
@@ -498,50 +494,18 @@ defmodule Kernel.ExpansionTest do
 
   describe "with" do
     test "variables do not leak" do
-      input = quote(do: (with({foo} <- {bar}, do: baz = :ok); baz))
-      other = Macro.var(:other, :elixir_with)
-      result = quote do
-        case {bar()} do
-          {foo} -> baz = :ok
-          unquote(other) -> unquote(other)
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: baz = :ok); baz)))
+             quote(do: (with({foo} <- {bar()}, do: baz = :ok); baz()))
     end
 
     test "variables are available in do option" do
-      input = quote(do: (with({foo} <- {bar}, do: baz = foo); baz))
-      other = Macro.var(:other, :elixir_with)
-      result = quote do
-        case {bar()} do
-          {foo} -> baz = foo
-          unquote(other) -> unquote(other)
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: baz = foo); baz))) ==
+             quote(do: (with({foo} <- {bar()}, do: baz = foo); baz()))
     end
 
     test "variables inside else do not leak" do
-      input = quote(do: (with({foo} <- {bar}, do: :ok, else: (baz -> baz)); baz))
-      other = Macro.var(:other, :elixir_with)
-      return = Macro.var(:return, :elixir_with)
-      result = quote do
-        case(case {bar()} do
-          {foo} -> {:ok, :ok}
-          unquote(other) -> {:error, unquote(other)}
-        end) do
-          {:ok, unquote(return)} -> unquote(return)
-          {:error, baz} -> baz
-          {:error, unquote(other)} -> :erlang.error({:with_clause, unquote(other)})
-        end
-        baz()
-      end
-
-      assert input |> expand() |> clean_meta([:export_vars, :generated]) == result
+      assert expand(quote(do: (with({foo} <- {bar}, do: :ok, else: (baz -> baz)); baz))) ==
+             quote(do: (with({foo} <- {bar()}, do: :ok, else: (baz -> baz)); baz()))
     end
 
     test "fails if \"do\" is missing" do
@@ -1187,6 +1151,24 @@ defmodule Kernel.ExpansionTest do
       end)
     end
 
+    assert_raise CompileError, ~r"invalid pattern in match", fn ->
+      expand(quote do
+        x = &(&1)
+        case true do
+          x.(false) -> true
+        end
+      end)
+    end
+
+    assert_raise CompileError, ~r"invalid expression in guard", fn ->
+      expand(quote do
+        x = &(&1)
+        case true do
+          true when x.(true) -> true
+        end
+      end)
+    end
+
     assert_raise CompileError, ~r"invalid call foo\(1\)\(2\)", fn ->
       expand(quote do: foo(1)(2))
     end

lib/elixir/test/elixir/kernel/macros_test.exs

@@ -15,10 +15,12 @@ defmodule Kernel.MacrosTest do
 
   Kernel.MacrosTest.Nested = require Kernel.MacrosTest.Nested, as: Nested
 
+  @spec my_macro :: Macro.t
   defmacro my_macro do
     quote do: 1 + 1
   end
 
+  @spec my_private_macro :: Macro.t
   defmacrop my_private_macro do
     quote do: 1 + 3
   end

lib/elixir/test/elixir/kernel/overridable_test.exs

@@ -101,27 +101,6 @@ defmodule Kernel.OverridableExampleBehaviour do
 end
 
 defmodule Kernel.OverridableTest do
-  defmodule OverridableOrder do
-    def not_private(str) do
-      process_url(str)
-    end
-
-    def process_url(_str) do
-      :first
-    end
-
-    # There was a bug where the order in which we removed
-    # overridable expressions lead to errors. This module
-    # aims to guarantee removing process_url/1 before we
-    # remove the function that depends on it does not cause
-    # errors. If it compiles, it works!
-    defoverridable [process_url: 1, not_private: 1]
-
-    def process_url(_str) do
-      :second
-    end
-  end
-
   require Kernel.Overridable, as: Overridable
   use ExUnit.Case
 
@@ -130,6 +109,60 @@ defmodule Kernel.OverridableTest do
     :code.delete(module)
   end
 
+  test "overridable keeps function ordering" do
+    defmodule OverridableOrder do
+      def not_private(str) do
+        process_url(str)
+      end
+
+      def process_url(_str) do
+        :first
+      end
+
+      # There was a bug where the order in which we removed
+      # overridable expressions lead to errors. This module
+      # aims to guarantee removing process_url/1 before we
+      # remove the function that depends on it does not cause
+      # errors. If it compiles, it works!
+      defoverridable [process_url: 1, not_private: 1]
+
+      def process_url(_str) do
+        :second
+      end
+    end
+  end
+
+  test "overridable works with defaults" do
+    defmodule OverridableDefault do
+      def fun(value, opt \\ :from_parent) do
+        {value, opt}
+      end
+
+      defmacro macro(value, opt \\ :from_parent) do
+        {{value, opt}, Macro.escape(__CALLER__)}
+      end
+
+      # There was a bug where the default function would
+      # attempt to call its overridable name instead of
+      # func/1. If it compiles, it works!
+      defoverridable [fun: 1, fun: 2, macro: 1, macro: 2]
+
+      def fun(value) do
+        {value, super(value)}
+      end
+
+      defmacro macro(value) do
+        {{value, super(value)}, Macro.escape(__CALLER__)}
+      end
+    end
+
+    defmodule OverridableCall do
+      require OverridableDefault
+      OverridableDefault.fun(:foo)
+      OverridableDefault.macro(:bar)
+    end
+  end
+
   test "overridable is made concrete if no other is defined" do
     assert Overridable.sample == 1
   end

lib/elixir/test/elixir/kernel/quote_test.exs

@@ -195,14 +195,14 @@ end
 
 # DO NOT MOVE THIS LINE
 defmodule Kernel.QuoteTest.Errors do
-  defmacro defadd do
+  defmacro defraise do
     quote location: :keep do
-      def add(a, b), do: a + b
+      def will_raise(_a, _b), do: raise "oops"
     end
   end
 
   defmacro will_raise do
-    quote location: :keep, do: raise "omg"
+    quote location: :keep, do: raise "oops"
   end
 end
 
@@ -211,16 +211,16 @@ defmodule Kernel.QuoteTest.ErrorsTest do
   import Kernel.QuoteTest.Errors
 
   # Defines the add function
-  defadd()
+  defraise()
 
   test "inside function error" do
-    assert_raise ArithmeticError, fn ->
-      add(:a, :b)
+    assert_raise RuntimeError, fn ->
+      will_raise(:a, :b)
     end
 
     mod  = Kernel.QuoteTest.ErrorsTest
     file = __ENV__.file |> Path.relative_to_cwd |> String.to_charlist
-    assert [{^mod, :add, 2, [file: ^file, line: 200]} | _] = System.stacktrace
+    assert [{^mod, :will_raise, 2, [file: ^file, line: 200]} | _] = System.stacktrace
   end
 
   test "outside function error" do

lib/elixir/test/elixir/kernel/raise_test.exs

@@ -133,265 +133,300 @@ defmodule Kernel.RaiseTest do
     end
   end
 
-  test "rescue with underscore no exception" do
-    result = try do
-      RescueUndefinedModule.go
-    rescue
-      _ -> true
-    end
-
-    assert result
-  end
-
-  test "rescue with higher precedence than catch" do
-    result = try do
-      RescueUndefinedModule.go
-    catch
-      _, _ -> false
-    rescue
-      _ -> true
-    end
-
-    assert result
-  end
-
-  test "rescue runtime error" do
-    result = try do
-      raise "an exception"
-    rescue
-      RuntimeError -> true
-    catch
-      :error, _ -> false
-    end
-
-    assert result
-
-    result = try do
-      raise "an exception"
-    rescue
-      AnotherError -> true
-    catch
-      :error, _ -> false
-    end
-
-    refute result
-  end
-
-  test "rescue named runtime error" do
-    result = try do
-      raise "an exception"
-    rescue
-      x in [RuntimeError] -> Exception.message(x)
-    catch
-      :error, _ -> false
-    end
-
-    assert result == "an exception"
-  end
-
-  test "rescue argument error from elixir" do
-    result = try do
-     raise ArgumentError, ""
-    rescue
-      ArgumentError -> true
-    end
-
-    assert result
-  end
-
-  test "rescue named without aliases" do
-    result = try do
-      raise "an exception"
-    rescue
-      x -> Exception.message(x)
-    end
-
-    assert result == "an exception"
-  end
-
-  test "wrap custom Erlang error" do
-    result = try do
-      :erlang.error(:sample)
-    rescue
-      x in [RuntimeError, ErlangError] -> Exception.message(x)
-    end
-
-    assert result == "Erlang error: :sample"
-  end
-
-  test "undefined function error" do
-    result = try do
-      DoNotExist.for_sure()
-    rescue
-      x in [UndefinedFunctionError] -> Exception.message(x)
-    end
-
-    assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
-  end
-
-  test "function clause error" do
-    result = try do
-      zero(1)
-    rescue
-      x in [FunctionClauseError] -> Exception.message(x)
-    end
-
-    assert result == "no function clause matching in Kernel.RaiseTest.zero/1"
-  end
-
-  test "badarg error" do
-    result = try do
-      :erlang.error(:badarg)
-    rescue
-      x in [ArgumentError] -> Exception.message(x)
-    end
-
-    assert result == "argument error"
-  end
-
-  test "tuple badarg error" do
-    result = try do
-      :erlang.error({:badarg, [1, 2, 3]})
-    rescue
-      x in [ArgumentError] -> Exception.message(x)
-    end
-
-    assert result == "argument error: [1, 2, 3]"
-  end
-
-  test "badarith error" do
-    result = try do
-      :erlang.error(:badarith)
-    rescue
-      x in [ArithmeticError] -> Exception.message(x)
-    end
-
-    assert result == "bad argument in arithmetic expression"
-  end
-
-  test "badarity error" do
-    fun    = fn(x) -> x end
-    string = "#{inspect(fun)} with arity 1 called with 2 arguments (1, 2)"
-
-    result = try do
-      fun.(1, 2)
-    rescue
-      x in [BadArityError] -> Exception.message(x)
-    end
-
-    assert result == string
-  end
-
-  test "badfun error" do
-    # Avoid "invalid function call" warning in >= OTP 19
-    x = fn -> :example end
-    result = try do
-      x.().(2)
-    rescue
-      x in [BadFunctionError] -> Exception.message(x)
-    end
-
-    assert result == "expected a function, got: :example"
-  end
-
-  test "badmatch error" do
-    x = :example
-    result = try do
-      ^x = zero(0)
-    rescue
-      x in [MatchError] -> Exception.message(x)
-    end
-
-    assert result == "no match of right hand side value: 0"
-  end
-
-  test "bad key error" do
-    result = try do
-      %{%{} | foo: :bar}
-    rescue
-      x in [KeyError] -> Exception.message(x)
-    end
-
-    assert result == "key :foo not found"
-
-    result = try do
-      %{}.foo
-    rescue
-      x in [KeyError] -> Exception.message(x)
-    end
-
-    assert result == "key :foo not found in: %{}"
-  end
-
-  test "bad map error" do
-    result = try do
-      %{zero(0) | foo: :bar}
-    rescue
-      x in [BadMapError] -> Exception.message(x)
-    end
-
-    assert result == "expected a map, got: 0"
-  end
-
-  test "bad boolean error" do
-    result = try do
-      1 and true
-    rescue
-      x in [BadBooleanError] -> Exception.message(x)
-    end
-
-    assert result == "expected a boolean on left-side of \"and\", got: 1"
-  end
-
-  test "case clause error" do
-    x = :example
-    result = try do
-      case zero(0) do
-        ^x -> nil
+  describe "rescue" do
+    test "runtime error" do
+      result = try do
+        raise "an exception"
+      rescue
+        RuntimeError -> true
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [CaseClauseError] -> Exception.message(x)
-    end
 
-    assert result == "no case clause matching: 0"
-  end
+      assert result
 
-  test "cond clause error" do
-    result = try do
-      cond do
-        !zero(0) -> :ok
+      result = try do
+        raise "an exception"
+      rescue
+        AnotherError -> true
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [CondClauseError] -> Exception.message(x)
+
+      refute result
     end
 
-    assert result == "no cond clause evaluated to a true value"
-  end
-
-  test "try clause error" do
-    f = fn() -> :example end
-    result = try do
-      try do
-        f.()
-      else
-        :other ->
-          :ok
+    test "named runtime error" do
+      result = try do
+        raise "an exception"
+      rescue
+        x in [RuntimeError] -> Exception.message(x)
+      catch
+        :error, _ -> false
       end
-    rescue
-      x in [TryClauseError] -> Exception.message(x)
+
+      assert result == "an exception"
     end
 
-    assert result == "no try clause matching: :example"
+    test "with higher precedence than catch" do
+      result = try do
+        raise "an exception"
+      catch
+        _, _ -> false
+      rescue
+        _ -> true
+      end
+
+      assert result
+    end
+
+    test "argument error from erlang" do
+      result = try do
+        :erlang.error(:badarg)
+      rescue
+        ArgumentError -> true
+      end
+
+      assert result
+    end
+
+    test "argument error from elixir" do
+      result = try do
+        raise ArgumentError, ""
+      rescue
+        ArgumentError -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all variable" do
+      result = try do
+        raise "an exception"
+      rescue
+        x -> Exception.message(x)
+      end
+
+      assert result == "an exception"
+    end
+
+    test "catch-all underscore" do
+      result = try do
+        raise "an exception"
+      rescue
+        _ -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all unused variable" do
+      result = try do
+        raise "an exception"
+      rescue
+        _any -> true
+      end
+
+      assert result
+    end
+
+    test "catch-all with \"x in _\" syntax" do
+      result = try do
+        raise "an exception"
+      rescue
+        exception in _ ->
+          Exception.message(exception)
+      end
+
+      assert result == "an exception"
+    end
   end
 
-  test "undefined function error as Erlang error" do
-    result = try do
-      DoNotExist.for_sure()
-    rescue
-      x in [ErlangError] -> Exception.message(x)
+  describe "normalize" do
+    test "wrap custom Erlang error" do
+      result = try do
+        :erlang.error(:sample)
+      rescue
+        x in [ErlangError] -> Exception.message(x)
+      end
+
+      assert result == "Erlang error: :sample"
     end
 
-    assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    test "undefined function error" do
+      result = try do
+        DoNotExist.for_sure()
+      rescue
+        x in [UndefinedFunctionError] -> Exception.message(x)
+      end
+
+      assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    end
+
+    test "function clause error" do
+      result = try do
+        zero(1)
+      rescue
+        x in [FunctionClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no function clause matching in Kernel.RaiseTest.zero/1"
+    end
+
+    test "badarg error" do
+      result = try do
+        :erlang.error(:badarg)
+      rescue
+        x in [ArgumentError] -> Exception.message(x)
+      end
+
+      assert result == "argument error"
+    end
+
+    test "tuple badarg error" do
+      result = try do
+        :erlang.error({:badarg, [1, 2, 3]})
+      rescue
+        x in [ArgumentError] -> Exception.message(x)
+      end
+
+      assert result == "argument error: [1, 2, 3]"
+    end
+
+    test "badarith error" do
+      result = try do
+        :erlang.error(:badarith)
+      rescue
+        x in [ArithmeticError] -> Exception.message(x)
+      end
+
+      assert result == "bad argument in arithmetic expression"
+    end
+
+    test "badarity error" do
+      fun    = fn(x) -> x end
+      string = "#{inspect(fun)} with arity 1 called with 2 arguments (1, 2)"
+
+      result = try do
+        fun.(1, 2)
+      rescue
+        x in [BadArityError] -> Exception.message(x)
+      end
+
+      assert result == string
+    end
+
+    test "badfun error" do
+      # Avoid "invalid function call" warning in >= OTP 19
+      x = fn -> :example end
+      result = try do
+        x.().(2)
+      rescue
+        x in [BadFunctionError] -> Exception.message(x)
+      end
+
+      assert result == "expected a function, got: :example"
+    end
+
+    test "badmatch error" do
+      x = :example
+      result = try do
+        ^x = zero(0)
+      rescue
+        x in [MatchError] -> Exception.message(x)
+      end
+
+      assert result == "no match of right hand side value: 0"
+    end
+
+    test "bad key error" do
+      result = try do
+        %{%{} | foo: :bar}
+      rescue
+        x in [KeyError] -> Exception.message(x)
+      end
+
+      assert result == "key :foo not found"
+
+      result = try do
+        %{}.foo
+      rescue
+        x in [KeyError] -> Exception.message(x)
+      end
+
+      assert result == "key :foo not found in: %{}"
+    end
+
+    test "bad map error" do
+      result = try do
+        %{zero(0) | foo: :bar}
+      rescue
+        x in [BadMapError] -> Exception.message(x)
+      end
+
+      assert result == "expected a map, got: 0"
+    end
+
+    test "bad boolean error" do
+      result = try do
+        1 and true
+      rescue
+        x in [BadBooleanError] -> Exception.message(x)
+      end
+
+      assert result == "expected a boolean on left-side of \"and\", got: 1"
+    end
+
+    test "case clause error" do
+      x = :example
+      result = try do
+        case zero(0) do
+          ^x -> nil
+        end
+      rescue
+        x in [CaseClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no case clause matching: 0"
+    end
+
+    test "cond clause error" do
+      result = try do
+        cond do
+          !zero(0) -> :ok
+        end
+      rescue
+        x in [CondClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no cond clause evaluated to a true value"
+    end
+
+    test "try clause error" do
+      f = fn() -> :example end
+      result = try do
+        try do
+          f.()
+        else
+          :other ->
+            :ok
+        end
+      rescue
+        x in [TryClauseError] -> Exception.message(x)
+      end
+
+      assert result == "no try clause matching: :example"
+    end
+
+    test "undefined function error as Erlang error" do
+      result = try do
+        DoNotExist.for_sure()
+      rescue
+        x in [ErlangError] -> Exception.message(x)
+      end
+
+      assert result == "function DoNotExist.for_sure/0 is undefined (module DoNotExist is not available)"
+    end
   end
 
   defmacrop exceptions do

lib/elixir/test/elixir/kernel/with_test.exs

@@ -10,7 +10,7 @@ defmodule Kernel.WithTest do
 
   test "matching with" do
     assert with(_..42 <- 1..42, do: :ok) == :ok
-    assert with({:ok, res} <- :error, do: res) == :error
+    assert with({:ok, res} <- error(), do: res) == :error
     assert with({:ok, _} = res <- ok(42), do: elem(res, 1)) == 42
   end
 
@@ -29,7 +29,7 @@ defmodule Kernel.WithTest do
     result = with({:ok, n1} <- ok(11), n2 <- 22, do: n1 + n2)
     assert result == 33
 
-    result = with(n1 <- 11, {:ok, n2} <- :error, do: n1 + n2)
+    result = with(n1 <- 11, {:ok, n2} <- error(), do: n1 + n2)
     assert result == :error
   end
 
@@ -65,9 +65,9 @@ defmodule Kernel.WithTest do
   end
 
   test "else conditions" do
-    assert with({:ok, res} <- 41, do: res, else: ({:error, error} -> error; res -> res + 1)) == 42
-    assert with({:ok, res} <- 41, do: res, else: (res when res == 41 -> res + 1; res -> res)) == 42
-    assert with({:ok, res} <- 41, do: res, else: (_ -> :error)) == :error
+    assert with({:ok, res} <- four(), do: res, else: ({:error, error} -> error; res -> res + 1)) == 5
+    assert with({:ok, res} <- four(), do: res, else: (res when res == 4 -> res + 1; res -> res)) == 5
+    assert with({:ok, res} <- four(), do: res, else: (_ -> :error)) == :error
   end
 
   test "else conditions with match error" do

lib/elixir/test/elixir/module_test.exs

@@ -307,6 +307,12 @@ defmodule ModuleTest do
     end
   end
 
+  test "does not use ETS tables named after the module" do
+    in_module do
+      assert :ets.info(__MODULE__) == :undefined
+    end
+  end
+
   ## Definitions
 
   test "defines?" do

lib/elixir/test/elixir/path_test.exs

@@ -32,6 +32,10 @@ defmodule PathTest do
     File.rm_rf tmp_path("wildcard")
   end
 
+  test "wildcard/2 raises on null byte" do
+    assert_raise ArgumentError, ~r/null byte/, fn -> Path.wildcard("foo\0bar") end
+  end
+
   if windows?() do
     describe "Windows" do
       test "relative/1" do

lib/elixir/test/elixir/protocol_test.exs

@@ -377,6 +377,10 @@ defmodule Protocol.ConsolidationTest do
            List.keyfind(docs, {:ok, 1}, 0)
   end
 
+  test "consolidation keeps source" do
+    assert Sample.__info__(:compile)[:source]
+  end
+
   test "consolidated keeps callbacks" do
     callbacks = Kernel.Typespec.beam_callbacks(@sample_binary)
     assert callbacks != []

lib/elixir/test/elixir/registry_test.exs

@@ -69,6 +69,10 @@ defmodule RegistryTest do
                [{self(), value}]
         assert Registry.match(registry, "hello", {:"$1", :_, :"$1"}) ==
                [{self(), value}]
+        assert Registry.match(registry, "hello", :_) ==
+               [{self(), value}]
+        assert Registry.match(registry, :_, :_) ==
+               []
       end
 
       test "supports guard conditions", %{registry: registry} do
@@ -82,6 +86,42 @@ defmodule RegistryTest do
                [{self(), value}]
       end
 
+      test "unregister_match supports patterns", %{registry: registry} do
+        value = {1, :atom, 1}
+        {:ok, _} = Registry.register(registry, "hello", value)
+
+        Registry.unregister_match(registry, "hello", {2, :_, :_})
+        assert Registry.lookup(registry, "hello") ==
+               [{self(), value}]
+        Registry.unregister_match(registry, "hello", {1.0, :_, :_})
+        assert Registry.lookup(registry, "hello") ==
+               [{self(), value}]
+        Registry.unregister_match(registry, "hello", {:_, :atom, :_})
+        assert Registry.lookup(registry, "hello") ==
+              []
+      end
+
+      test "unregister_match supports guards", %{registry: registry} do
+        value = {1, :atom, 1}
+        {:ok, _} = Registry.register(registry, "hello", value)
+
+        Registry.unregister_match(registry, "hello", {:"$1", :_, :_}, [{:<, :"$1", 2}])
+        assert Registry.lookup(registry, "hello") ==
+               []
+      end
+
+      test "unregister_match supports tricky keys", %{registry: registry} do
+        {:ok, _} = Registry.register(registry, :_, :foo)
+        {:ok, _} = Registry.register(registry, "hello", "b")
+
+        Registry.unregister_match(registry, :_, :foo)
+        assert Registry.lookup(registry, :_) ==
+               []
+
+        assert Registry.keys(registry, self()) |> Enum.sort ==
+               ["hello"]
+      end
+
       test "compares using ===", %{registry: registry} do
         {:ok, _} = Registry.register(registry, 1.0, :value)
         {:ok, _} = Registry.register(registry, 1, :value)
@@ -347,6 +387,50 @@ defmodule RegistryTest do
                [{self(), value1}, {self(), value2}]
       end
 
+      test "unregister_match supports patterns", %{registry: registry} do
+        value1 = {1, :atom, 1}
+        {:ok, _} = Registry.register(registry, "hello", value1)
+        value2 = {2, :atom, 2}
+        {:ok, _} = Registry.register(registry, "hello", value2)
+
+        Registry.unregister_match(registry, "hello", {2, :_, :_})
+        assert Registry.lookup(registry, "hello") ==
+               [{self(), value1}]
+
+        {:ok, _} = Registry.register(registry, "hello", value2)
+        Registry.unregister_match(registry, "hello", {2.0, :_, :_})
+        assert Registry.lookup(registry, "hello") ==
+               [{self(), value1}, {self(), value2}]
+        Registry.unregister_match(registry, "hello", {:_, :atom, :_})
+        assert Registry.lookup(registry, "hello") ==
+               []
+      end
+
+      test "unregister_match supports guards", %{registry: registry} do
+        value1 = {1, :atom, 1}
+        {:ok, _} = Registry.register(registry, "hello", value1)
+        value2 = {2, :atom, 2}
+        {:ok, _} = Registry.register(registry, "hello", value2)
+
+        Registry.unregister_match(registry, "hello", {:"$1", :_, :_}, [{:<, :"$1", 2}])
+        assert Registry.lookup(registry, "hello") ==
+               [{self(), value2}]
+      end
+
+      test "unregister_match supports tricky keys", %{registry: registry} do
+        {:ok, _} = Registry.register(registry, :_, :foo)
+        {:ok, _} = Registry.register(registry, :_, :bar)
+        {:ok, _} = Registry.register(registry, "hello", "a")
+        {:ok, _} = Registry.register(registry, "hello", "b")
+
+        Registry.unregister_match(registry, :_, :foo)
+        assert Registry.lookup(registry, :_) ==
+               [{self(), :bar}]
+
+        assert Registry.keys(registry, self()) |> Enum.sort ==
+               [:_, "hello", "hello"]
+      end
+
       @tag listener: :"duplicate_listener_#{partitions}"
       test "allows listeners", %{registry: registry, listener: listener} do
         Process.register(self(), listener)

lib/elixir/test/elixir/stream_test.exs

@@ -59,22 +59,39 @@ defmodule StreamTest do
            [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
-  test "chunk/4 is zippable" do
-    stream = Stream.chunk([1, 2, 3, 4, 5, 6], 3, 2, [])
+  test "chunk_every/2, chunk_every/3 and chunk_every/4" do
+    assert Stream.chunk_every([1, 2, 3, 4, 5], 2) |> Enum.to_list ==
+           [[1, 2], [3, 4], [5]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5], 2, 2, [6]) |> Enum.to_list ==
+           [[1, 2], [3, 4], [5, 6]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, :discard) |> Enum.to_list ==
+           [[1, 2, 3], [3, 4, 5]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5, 6], 2, 3, :discard) |> Enum.to_list ==
+           [[1, 2], [4, 5]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, []) |> Enum.to_list ==
+           [[1, 2, 3], [3, 4, 5], [5, 6]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 3, []) |> Enum.to_list ==
+           [[1, 2, 3], [4, 5, 6]]
+    assert Stream.chunk_every([1, 2, 3, 4, 5], 4, 4, 6..10) |> Enum.to_list ==
+           [[1, 2, 3, 4], [5, 6, 7, 8]]
+  end
+
+  test "chunk_every/4 is zippable" do
+    stream = Stream.chunk_every([1, 2, 3, 4, 5, 6], 3, 2, [])
     list   = Enum.to_list(stream)
     assert Enum.zip(list, list) == Enum.zip(stream, stream)
   end
 
-  test "chunk/4 is haltable" do
-    assert 1..10 |> Stream.take(6) |> Stream.chunk(4, 4, [7, 8]) |> Enum.to_list ==
+  test "chunk_every/4 is haltable" do
+    assert 1..10 |> Stream.take(6) |> Stream.chunk_every(4, 4, [7, 8]) |> Enum.to_list ==
            [[1, 2, 3, 4], [5, 6, 7, 8]]
-    assert 1..10 |> Stream.take(6) |> Stream.chunk(4, 4, [7, 8]) |> Stream.take(3) |> Enum.to_list ==
+    assert 1..10 |> Stream.take(6) |> Stream.chunk_every(4, 4, [7, 8]) |> Stream.take(3) |> Enum.to_list ==
            [[1, 2, 3, 4], [5, 6, 7, 8]]
-    assert 1..10 |> Stream.take(6) |> Stream.chunk(4, 4, [7, 8]) |> Stream.take(2) |> Enum.to_list ==
+    assert 1..10 |> Stream.take(6) |> Stream.chunk_every(4, 4, [7, 8]) |> Stream.take(2) |> Enum.to_list ==
            [[1, 2, 3, 4], [5, 6, 7, 8]]
-    assert 1..10 |> Stream.take(6) |> Stream.chunk(4, 4, [7, 8]) |> Stream.take(1) |> Enum.to_list ==
+    assert 1..10 |> Stream.take(6) |> Stream.chunk_every(4, 4, [7, 8]) |> Stream.take(1) |> Enum.to_list ==
            [[1, 2, 3, 4]]
-    assert 1..6 |> Stream.take(6) |> Stream.chunk(4, 4, [7, 8]) |> Enum.to_list ==
+    assert 1..6 |> Stream.take(6) |> Stream.chunk_every(4, 4, [7, 8]) |> Enum.to_list ==
            [[1, 2, 3, 4], [5, 6, 7, 8]]
   end
 
@@ -86,7 +103,7 @@ defmodule StreamTest do
            [[1], [2, 2], [3], [4, 4, 6], [7, 7]]
     assert stream |> Stream.take(3) |> Enum.to_list ==
            [[1], [2, 2], [3]]
-    assert 1..10 |> Stream.chunk(2) |> Enum.take(2) ==
+    assert 1..10 |> Stream.chunk_every(2) |> Enum.take(2) ==
            [[1, 2], [3, 4]]
   end
 
@@ -96,12 +113,15 @@ defmodule StreamTest do
     assert Enum.zip(list, list) == Enum.zip(stream, stream)
   end
 
-  test "chunk_by/4" do
+  test "chunk_while/4" do
     chunk_fun = fn i, acc ->
-      if rem(i, 2) == 0 do
-        {:cont, Enum.reverse([i | acc]), []}
-      else
-        {:cont, [i | acc]}
+      cond do
+        i > 10 ->
+          {:halt, acc}
+        rem(i, 2) == 0 ->
+          {:cont, Enum.reverse([i | acc]), []}
+        true ->
+          {:cont, [i | acc]}
       end
     end
 
@@ -110,15 +130,16 @@ defmodule StreamTest do
       acc -> {:cont, Enum.reverse(acc), []}
     end
 
-    stream = Stream.chunk_by(1..10, [], chunk_fun, after_fun)
-    assert lazy?(stream)
-
-    assert 1..10 |> Stream.chunk_by([], chunk_fun, after_fun) |> Enum.to_list() ==
+    assert Stream.chunk_while([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], chunk_fun, after_fun) |> Enum.to_list ==
            [[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert 0..10 |> Stream.chunk_by([], chunk_fun, after_fun) |> Enum.to_list() ==
+    assert Stream.chunk_while(0..9, [], chunk_fun, after_fun) |> Enum.to_list ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9]]
+    assert Stream.chunk_while(0..10, [], chunk_fun, after_fun) |> Enum.to_list ==
            [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
-    assert 0..11 |> Stream.chunk_by([], chunk_fun, after_fun) |> Enum.to_list() ==
-           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11]]
+    assert Stream.chunk_while(0..11, [], chunk_fun, after_fun) |> Enum.to_list ==
+           [[0], [1, 2], [3, 4], [5, 6], [7, 8], [9, 10]]
+    assert Stream.chunk_while([5, 7, 9, 11], [], chunk_fun, after_fun) |> Enum.to_list ==
+           [[5, 7, 9]]
   end
 
   test "concat/1" do
@@ -168,6 +189,14 @@ defmodule StreamTest do
     stream = Stream.cycle([1, 2, 3])
     assert is_function(stream)
 
+    assert_raise ArgumentError, "cannot cycle over empty enumerable", fn ->
+      Stream.cycle([])
+    end
+
+    assert_raise ArgumentError, "cannot cycle over empty enumerable", fn ->
+      Stream.cycle(%{}) |> Enum.to_list()
+    end
+
     assert Stream.cycle([1, 2, 3]) |> Stream.take(5) |> Enum.to_list == [1, 2, 3, 1, 2]
     assert Enum.take(stream, 5) == [1, 2, 3, 1, 2]
   end

lib/elixir/test/elixir/string_io_test.exs

@@ -248,4 +248,117 @@ defmodule StringIOTest do
     assert IO.stream(pid, 2) |> Enum.to_list == ["ab", "c"]
     assert contents(pid) == {"", ""}
   end
+
+  defp get_until(pid, encoding, prompt, module, function, extra_args \\ []) do
+    :io.request(pid, {:get_until, encoding, prompt, module, function, extra_args})
+  end
+
+  defmodule GetUntilCallbacks do
+    def until_eof(continuation, :eof) do
+      {:done, continuation, :eof}
+    end
+
+    def until_eof(continuation, content) do
+      {:more, continuation ++ content}
+    end
+
+    def until_eof_then_try_more('magic-stop-prefix' ++ continuation, :eof) do
+      {:done, continuation, :eof}
+    end
+
+    def until_eof_then_try_more(continuation, :eof) do
+      {:more, 'magic-stop-prefix' ++ continuation}
+    end
+
+    def until_eof_then_try_more(continuation, content) do
+      {:more, continuation ++ content}
+    end
+
+    def up_to_3_bytes(continuation, :eof) do
+      {:done, continuation, :eof}
+    end
+
+    def up_to_3_bytes(continuation, content) do
+      case continuation ++ content do
+        [a, b, c | tail] -> {:done, [a, b, c], tail}
+        str -> {:more, str}
+      end
+    end
+
+    def up_to_3_bytes_discard_rest(continuation, :eof) do
+      {:done, continuation, :eof}
+    end
+
+    def up_to_3_bytes_discard_rest(continuation, content) do
+      case continuation ++ content do
+        [a, b, c | _tail] -> {:done, [a, b, c], :eof}
+        str -> {:more, str}
+      end
+    end
+  end
+
+  test "get_until with up_to_3_bytes" do
+    pid = start("abcdefg")
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :up_to_3_bytes)
+    assert result == "abc"
+    assert IO.read(pid, :all) == "defg"
+  end
+
+  test "get_until with up_to_3_bytes_discard_rest" do
+    pid = start("abcdefg")
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :up_to_3_bytes_discard_rest)
+    assert result == "abc"
+    assert IO.read(pid, :all) == ""
+  end
+
+  test "get_until with until_eof" do
+    pid = start("abc\nd")
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :until_eof)
+    assert result == "abc\nd"
+  end
+
+  test "get_until with until_eof and \\r\\n" do
+    pid = start("abc\r\nd")
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :until_eof)
+    assert result == "abc\r\nd"
+  end
+
+  test "get_until with until_eof capturing prompt" do
+    pid = start("abc\nd", capture_prompt: true)
+    result = get_until(pid, :unicode, ">", GetUntilCallbacks, :until_eof)
+    assert result == "abc\nd"
+    assert StringIO.contents(pid) == {"", ">>>"}
+  end
+
+  test "get_until with until_eof_then_try_more" do
+    pid = start("abc\nd")
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :until_eof_then_try_more)
+    assert result == "abc\nd"
+  end
+
+  test "get_until with invalid UTF-8" do
+    pid = start(<<130, 227, 129, 132, 227, 129, 134>>)
+    result = get_until(pid, :unicode, "", GetUntilCallbacks, :until_eof)
+    assert result == :error
+  end
+
+  test "get_until with raw bytes (latin1)" do
+    pid = start(<<181, 255, 194, ?\n>>)
+    result = get_until(pid, :latin1, "", GetUntilCallbacks, :until_eof)
+    assert result == <<181, 255, 194, ?\n>>
+  end
+
+  test ":io.erl_scan_form/2" do
+    pid = start("1.")
+    result = :io.scan_erl_form(pid, 'p>')
+    assert result == {:ok, [{:integer, 1, 1}, {:dot, 1}], 1}
+    assert StringIO.contents(pid) == {"", ""}
+  end
+
+  test ":io.erl_scan_form/2 with capture_prompt" do
+    pid = start("1.", capture_prompt: true)
+    result = :io.scan_erl_form(pid, 'p>')
+    assert result == {:ok, [{:integer, 1, 1}, {:dot, 1}], 1}
+    assert StringIO.contents(pid) == {"", "p>p>"}
+  end
 end

lib/elixir/test/elixir/system_test.exs

@@ -73,6 +73,16 @@ defmodule SystemTest do
 
     System.put_env(%{@test_var => "OTHER_SAMPLE"})
     assert System.get_env(@test_var) == "OTHER_SAMPLE"
+
+    assert_raise ArgumentError, ~r[cannot execute System.put_env/2 for key with \"=\"], fn ->
+      System.put_env("FOO=BAR", "BAZ")
+    end
+  end
+
+  test "cmd/2 raises for null bytes" do
+    assert_raise ArgumentError, ~r"cannot execute System.cmd/3 for program with null byte", fn ->
+      System.cmd("null\0byte", [])
+    end
   end
 
   if windows?() do
@@ -107,11 +117,11 @@ defmodule SystemTest do
       File.rm_rf! Path.dirname(tmp_path(@echo))
     end
   else
-    test "cmd/2 Unix" do
+    test "cmd/2 unix" do
       assert {"hello\n", 0} = System.cmd "echo", ["hello"]
     end
 
-    test "cmd/3 (with options) Unix" do
+    test "cmd/3 (with options) unix" do
       assert {["hello\n"], 0} = System.cmd "echo", ["hello"],
                                   into: [], cd: System.cwd!, env: %{"foo" => "bar", "baz" => nil},
                                   arg0: "echo", stderr_to_stdout: true, parallelism: true
@@ -119,7 +129,7 @@ defmodule SystemTest do
 
     @echo "echo-elixir-test"
 
-    test "cmd/2 with absolute and relative paths Unix" do
+    test "cmd/2 with absolute and relative paths ynix" do
       echo = tmp_path(@echo)
       File.mkdir_p! Path.dirname(echo)
       File.cp! System.find_executable("echo"), echo
@@ -143,6 +153,9 @@ defmodule SystemTest do
     assert System.find_executable("erl")
     assert is_binary System.find_executable("erl")
     assert !System.find_executable("does-not-really-exist-from-elixir")
+    assert_raise ArgumentError, ~r"cannot execute System.find_executable/1 for program with null byte", fn ->
+      System.find_executable("null\0byte")
+    end
   end
 
   test "monotonic_time/0" do

lib/elixir/test/erlang/control_test.erl

@@ -222,3 +222,9 @@ optimized_oror_test() ->
 
 no_after_in_try_test() ->
   {'try', _, [_], [_], _, []} = to_erl("try do :foo.bar() else _ -> :ok end").
+
+optimized_inspect_interpolation_test() ->
+    {bin, _,
+     [{bin_element, _,
+       {call, _, {remote, _,{atom, _, 'Elixir.Kernel'}, {atom, _, inspect}}, [_]},
+       default, [binary]}]} = to_erl("\"#{inspect(1)}\"").

lib/elixir/unicode/properties.ex

@@ -70,15 +70,15 @@ defmodule String.Casing do
 
   # Downcase
 
-  def downcase(string), do: downcase(string, "")
+  def downcase(string) when is_binary(string), do: downcase(string, "")
 
   for {codepoint, _upper, lower, _title} <- codes, lower && lower != codepoint do
-    defp downcase(unquote(codepoint) <> rest, acc) do
+    defp downcase(<<unquote(codepoint), rest::bits>>, acc) do
       downcase(rest, acc <> unquote(lower))
     end
   end
 
-  defp downcase(<<char, rest::binary>>, acc) do
+  defp downcase(<<char, rest::bits>>, acc) do
     downcase(rest, <<acc::binary, char>>)
   end
 
@@ -86,15 +86,15 @@ defmodule String.Casing do
 
   # Upcase
 
-  def upcase(string), do: upcase(string, "")
+  def upcase(string) when is_binary(string), do: upcase(string, "")
 
   for {codepoint, upper, _lower, _title} <- codes, upper && upper != codepoint do
-    defp upcase(unquote(codepoint) <> rest, acc) do
+    defp upcase(<<unquote(codepoint), rest::bits>>, acc) do
       upcase(rest, acc <> unquote(upper))
     end
   end
 
-  defp upcase(<<char, rest::binary>>, acc) do
+  defp upcase(<<char, rest::bits>>, acc) do
     upcase(rest, <<acc::binary, char>>)
   end
 
@@ -140,11 +140,14 @@ defmodule String.Break do
 
   # trim_leading
 
-  for codepoint <- whitespace do
-    def trim_leading(unquote(codepoint) <> rest), do: trim_leading(rest)
+  def trim_leading(string) when is_binary(string) do
+    do_trim_leading(string)
   end
-  def trim_leading(""), do: ""
-  def trim_leading(string) when is_binary(string), do: string
+
+  for codepoint <- whitespace do
+    def do_trim_leading(<<unquote(codepoint), rest::bits>>), do: do_trim_leading(rest)
+  end
+  def do_trim_leading(<<rest::bits>>), do: rest
 
   # trim_trailing
 

lib/ex_unit/lib/ex_unit.ex

@@ -147,7 +147,6 @@ defmodule ExUnit do
     {:ok, _} = Application.ensure_all_started(:ex_unit)
 
     configure(options)
-    config = put_defaults(configuration())
 
     if Application.fetch_env!(:ex_unit, :autorun) do
       Application.put_env(:ex_unit, :autorun, false)
@@ -155,6 +154,7 @@ defmodule ExUnit do
       System.at_exit fn
         0 ->
           time = ExUnit.Server.cases_loaded()
+          config = persist_defaults(configuration())
           %{failures: failures} = ExUnit.Runner.run(config, time)
           System.at_exit fn _ ->
             if failures > 0, do: exit({:shutdown, 1})
@@ -212,7 +212,8 @@ defmodule ExUnit do
     * `:timeout` - sets the timeout for the tests, defaults to `60_000` milliseconds;
 
     * `:trace` - sets ExUnit into trace mode, this sets `:max_cases` to `1` and
-      prints each test case and test while running.
+      prints each test case and test while running. Note that in trace mode test timeouts
+      will be ignored.
 
   """
   def configure(options) do
@@ -226,6 +227,8 @@ defmodule ExUnit do
   """
   def configuration do
     Application.get_all_env(:ex_unit)
+    |> put_seed()
+    |> put_max_cases()
   end
 
   @doc """
@@ -260,28 +263,25 @@ defmodule ExUnit do
   of failures and the number of skipped tests.
   """
   def run do
-    config = put_defaults(configuration())
+    config = persist_defaults(configuration())
     ExUnit.Runner.run(config, nil)
   end
 
-  defp put_defaults(opts) do
-    opts
-    |> put_seed()
-    |> put_max_cases()
+  # Persists default values in application
+  # environment before the test suite starts.
+  defp persist_defaults(config) do
+    config |> Keyword.take([:seed, :max_cases]) |> configure()
+    config
   end
 
   defp put_seed(opts) do
     Keyword.put_new_lazy(opts, :seed, fn ->
-      seed = :os.timestamp |> elem(2)
-      Application.put_env(:ex_unit, :seed, seed)
-      seed
+      :os.timestamp |> elem(2)
     end)
   end
 
   defp put_max_cases(opts) do
-    max_cases = max_cases(opts)
-    Application.put_env(:ex_unit, :max_cases, max_cases)
-    Keyword.put(opts, :max_cases, max_cases)
+    Keyword.put(opts, :max_cases, max_cases(opts))
   end
 
   defp max_cases(opts) do

lib/ex_unit/lib/ex_unit/doc_test.ex

@@ -23,7 +23,7 @@ defmodule ExUnit.DocTest do
 
   To run doctests include them in an ExUnit case with a `doctest` macro:
 
-      defmodule MyModule.Test do
+      defmodule MyModuleTest do
         use ExUnit.Case, async: true
         doctest MyModule
       end
@@ -193,8 +193,8 @@ defmodule ExUnit.DocTest do
         end
       end
 
-    tests = quote bind_quoted: binding() do
-      file = "(for doctest at) " <> Path.relative_to_cwd(mod.__info__(:compile)[:source])
+    tests = quote bind_quoted: [mod: mod, opts: opts] do
+      file = ExUnit.DocTest.__file__(mod)
       for {name, test} <- ExUnit.DocTest.__doctests__(mod, opts) do
         @tag :doctest
         @file file
@@ -205,6 +205,13 @@ defmodule ExUnit.DocTest do
     [require, tests]
   end
 
+  @doc false
+  def __file__(module) do
+    source = module.__info__(:compile)[:source] ||
+             raise "#{inspect module} does not have compile-time source information"
+    "(for doctest at) " <> Path.relative_to_cwd(source)
+  end
+
   @doc false
   def __doctests__(module, opts) do
     do_import = Keyword.get(opts, :import, false)

lib/iex/lib/iex.ex

@@ -2,10 +2,7 @@ defmodule IEx do
   @moduledoc ~S"""
   Elixir's interactive shell.
 
-  This module is the main entry point for Interactive Elixir and
-  in this documentation we will talk a bit about how IEx works.
-
-  Notice that some of the functionalities described here will not be available
+  Some of the functionalities described here will not be available
   depending on your terminal. In particular, if you get a message
   saying that the smart terminal could not be run, some of the
   features described here won't work.
@@ -47,12 +44,78 @@ defmodule IEx do
 
       set ERL_AFLAGS "-kernel shell_history enabled"
 
+  ## Expressions in IEx
+
+  As an interactive shell, IEx evaluates expressions. This has some
+  interesting consequences that are worth discussing.
+
+  The first one is that the code is truly evaluated and not compiled.
+  This means that any benchmarking done in the shell is going to have
+  skewed results. So never run any profiling nor benchmarks in the shell.
+
+  Second, IEx allows you to break an expression into many lines,
+  since this is common in Elixir. For example:
+
+      iex(1)> "ab
+      ...(1)> c"
+      "ab\nc"
+
+  In the example above, the shell will be expecting more input until it
+  finds the closing quote. Sometimes it is not obvious which character
+  the shell is expecting, and the user may find themselves trapped in
+  the state of incomplete expression with no ability to terminate it other
+  than by exiting the shell.
+
+  For such cases, there is a special break-trigger (`#iex:break`) that when
+  encountered on a line by itself will force the shell to break out of any
+  pending expression and return to its normal state:
+
+      iex(1)> ["ab
+      ...(1)> c"
+      ...(1)> "
+      ...(1)> ]
+      ...(1)> #iex:break
+      ** (TokenMissingError) iex:1: incomplete expression
+
   ## The Break command
 
   Inside IEx, hitting `Ctrl+C` will open up the `BREAK` menu. In this
   menu you can quit the shell, see process and ets tables information
   and much more.
 
+  ## Exiting the shell
+
+  There are a few ways to quit the IEx shell:
+
+    * via the `BREAK` menu (available via `Ctrl+C`) by typing `q`, pressing enter
+    * by hitting `Ctrl+C`, `Ctrl+C`
+    * by hitting `Ctrl+\`
+
+  If you are connected to remote shell, it remains alive after disconnection.
+
+  ## Prying and breakpoints
+
+  IEx also has the ability to set breakpoints on Elixir code and
+  "pry" into running processes. This allows the developer to have
+  an IEx session run inside a given function.
+
+  `IEx.pry/0` can be used when you are able to modify the source
+  code directly and recompile it:
+
+      def my_fun(arg1, arg2) do
+        require IEx; IEx.pry
+        ... implementation ...
+      end
+
+  When the code is executed, it will ask you for permission to be
+  introspected.
+
+  Alternatively, you can use `IEx.break!/4` to setup a breakpoint
+  on a given module, function and arity you have no control of.
+  While `IEx.break!/4` is more flexible, it requires OTP 20+ and
+  it does not contain information about imports and aliases from
+  the source code.
+
   ## The User Switch command
 
   Besides the break command, one can type `Ctrl+G` to get to the
@@ -85,8 +148,16 @@ defmodule IEx do
   Since shells are isolated from each other, you can't access the
   variables defined in one shell from the other one.
 
-  The user switch command menu also allows developers to connect to remote
-  shells using the `r` command. A topic which we will discuss next.
+  The User Switch command can also be used to terminate an existing
+  session, for example when the evaluator gets stuck in an infinite
+  loop or when you are stuck typing an expression:
+
+      User switch command
+       --> i
+       --> c
+
+  The user switch command menu also allows developers to connect to
+  remote shells using the `r` command. A topic which we will discuss next.
 
   ## Remote shells
 
@@ -209,48 +280,6 @@ defmodule IEx do
       iex(1)> [1, 2, 3, 4, 5]
       [1, 2, 3, ...]
 
-  ## Expressions in IEx
-
-  As an interactive shell, IEx evaluates expressions. This has some
-  interesting consequences that are worth discussing.
-
-  The first one is that the code is truly evaluated and not compiled.
-  This means that any benchmarking done in the shell is going to have
-  skewed results. So never run any profiling nor benchmarks in the shell.
-
-  Second, IEx allows you to break an expression into many lines,
-  since this is common in Elixir. For example:
-
-      iex(1)> "ab
-      ...(1)> c"
-      "ab\nc"
-
-  In the example above, the shell will be expecting more input until it
-  finds the closing quote. Sometimes it is not obvious which character
-  the shell is expecting, and the user may find themselves trapped in
-  the state of incomplete expression with no ability to terminate it other
-  than by exiting the shell.
-
-  For such cases, there is a special break-trigger (`#iex:break`) that when
-  encountered on a line by itself will force the shell to break out of any
-  pending expression and return to its normal state:
-
-      iex(1)> ["ab
-      ...(1)> c"
-      ...(1)> "
-      ...(1)> ]
-      ...(1)> #iex:break
-      ** (TokenMissingError) iex:1: incomplete expression
-
-  ## Exiting the shell
-
-  There are a few ways to quit the IEx shell:
-
-    * via the `BREAK` menu (available via `Ctrl+C`) by typing `q`, `Enter`
-    * by hitting `Ctrl+C`, `Ctrl+C`
-    * by hitting `Ctrl+\`
-
-  If you are connected to remote shell, it remains alive after disconnection.
   """
 
   @doc """
@@ -419,17 +448,23 @@ defmodule IEx do
   Pries into the process environment.
 
   This is useful for debugging a particular chunk of code
-  and inspect the state of a particular process. The process
-  is temporarily changed to trap exits (i.e. the process flag
-  `:trap_exit` is set to `true`) and has the `group_leader` changed
-  to support ANSI escape codes. Those values are reverted by
-  calling `respawn/0`, which starts a new IEx shell, freeing up
-  the pried one.
+  when executed by a particular process. The process becomes
+  the evaluator of IEx commands and is temporarily changed to
+  have a custom group leader. Those values are reverted by
+  calling `IEx.Helpers.respawn/0`, which starts a new IEx shell,
+  freeing up the pried one.
 
-  When a process is pried, all code runs inside IEx and, as
-  such, it is evaluated and cannot access private functions
-  of the module being pried. Module functions still need to be
-  accessed via `Mod.fun(args)`.
+  When a process is pried, all code runs inside IEx and has
+  access to all imports and aliases from the original code.
+  However, the code is evaluated and therefore cannot access
+  private functions of the module being pried. Module functions
+  still need to be accessed via `Mod.fun(args)`.
+
+  Alternatively, you can use `IEx.break!/4` to setup a breakpoint
+  on a given module, function and arity you have no control of.
+  While `IEx.break!/4` is more flexible, it requires OTP 20+ and
+  it does not contain information about imports and aliases from
+  the source code.
 
   ## Examples
 
@@ -440,12 +475,12 @@ defmodule IEx do
   the process information. Let's see an example:
 
       import Enum, only: [map: 2]
-      require IEx
+
 
       defmodule Adder do
         def add(a, b) do
           c = a + b
-          IEx.pry
+          require IEx; IEx.pry
         end
       end
 
@@ -470,56 +505,147 @@ defmodule IEx do
       Interactive Elixir - press Ctrl+C to exit (type h() ENTER for help)
 
   Setting variables or importing modules in IEx does not
-  affect the caller's environment (hence it is called `pry`).
+  affect the caller's environment. However, sending and
+  receiving messages will change the process state.
+
+  ## Pry and mix test
+
+  To use `IEx.pry/0` during tests, you need to run Mix inside
+  `iex` and pass the `--trace` to `mix test` to avoid running
+  into timeouts:
+
+      iex -S mix test --trace
+      iex -S mix test path/to/file:line --trace
   """
-  defmacro pry(timeout \\ 5000) do
+  defmacro pry() do
     quote do
-      IEx.pry(binding(), __ENV__, unquote(timeout))
+      IEx.Pry.pry(binding(), __ENV__)
     end
   end
 
   @doc """
-  Callback for `IEx.pry/1`.
-
-  You can invoke this function directly when you are not able to invoke
-  `IEx.pry/1` as a macro. This function expects the binding (from
-  `Kernel.binding/0`), the environment (from `__ENV__/0`) and the timeout
-  (a sensible default is 5000).
+  Macro-based shortcut for `IEx.break!/4`.
   """
-  def pry(binding, env, timeout) do
-    opts = [binding: binding, dot_iex_path: "", env: env, prefix: "pry"]
-    meta = "#{inspect self()} at #{Path.relative_to_cwd(env.file)}:#{env.line}"
-    desc =
-      if File.regular?(env.file) do
-        parse_file(env)
-      else
-        ""
+  defmacro break!(ast, stops \\ 1) do
+    with {:/, _, [call, arity]} when is_integer(arity) <- ast,
+         {mod, fun, []} <- Macro.decompose_call(call) do
+      quote do
+        IEx.break!(unquote(mod), unquote(fun), unquote(arity), unquote(stops))
       end
-
-    res = IEx.Server.take_over("Request to pry #{meta}#{desc}", opts, timeout)
-
-    # We cannot use colors because IEx may be off.
-    case res do
-      {:error, :no_iex} ->
-        extra =
-          case :os.type do
-            {:win32, _} -> " If you are Windows, you may need to start IEx with the --werl flag."
-            _           -> ""
-          end
-        IO.puts :stdio, "Cannot pry #{meta}. Is an IEx shell running?" <> extra
+    else
       _ ->
-        :ok
+        raise ArgumentError, "expected Mod.fun/arity, such as URI.parse/1, got: #{Macro.to_string(ast)}"
     end
-
-    res
   end
 
-  defp parse_file(env) do
-    lines =
-      env.file
-      |> File.stream!
-      |> Enum.slice(max(env.line - 3, 0), 5)
-    Enum.intersperse(["\n\n" | lines], "    ")
+  @doc """
+  Sets up a breakpoint in `module`, `function` and `arity` with
+  the given number of `stops`.
+
+  This function will instrument the given module and load a new
+  version in memory with breakpoints at the given function and
+  arity. If the module is recompiled, all breakpoints are lost.
+
+  When a breakpoint is reached, IEx will ask if you want to `pry`
+  the given function and arity. In other words, this works similar
+  to `IEx.pry/0` as the running process becomes the evaluator of
+  IEx commands and is temporarily changed to have a custom group
+  leader. However, differently from `IEx.pry/0`, aliases and imports
+  from the source code won't be available in the shell.
+
+  IEx helpers includes many conveniences related to breakpoints.
+  Below they are listed with the full module, such as `IEx.Helpers.breaks/0`,
+  but remember it can be called directly as `breaks()` inside IEx.
+  They are:
+
+    * `IEx.Helpers.break!/2` - sets up a breakpoint for a given `Mod.fun/arity`
+    * `IEx.Helpers.break!/4` - sets up a breakpoint for the given module, function, arity
+    * `IEx.Helpers.breaks/0` - prints all breakpoints and their ids
+    * `IEx.Helpers.continue/0` - continues until the next breakpoint in the same shell
+    * `IEx.Helpers.open/0` - opens editor on the current breakpoint
+    * `IEx.Helpers.remove_breaks/0` - removes all breakpoints in all modules
+    * `IEx.Helpers.remove_breaks/1` - removes all breakpoints in a given module
+    * `IEx.Helpers.reset_break/1` - sets the number of stops on the given id to zero
+    * `IEx.Helpers.reset_break/3` - sets the number of stops on the given module, function, arity to zero
+    * `IEx.Helpers.respawn/0` - starts a new shell (breakpoints will ask for permission once more)
+    * `IEx.Helpers.whereami/1` - shows the current location
+
+  By default, the number of stops in a breakpoint is 1. Any follow-up
+  call won't stop the code execution unless another breakpoint is set.
+
+  Alternatively, the number of be increased by passing the `stops`
+  argument. `IEx.Helpers.reset_break/1` and `IEx.Helpers.reset_break/3`
+  can be used to reset the number back to zero. Note the module remains
+  "instrumented" even after all stops on all breakpoints are consumed.
+  You can remove the instrumentation in a given module by calling
+  `IEx.Helpers.remove_breaks/1` and on all modules by calling
+  `IEx.Helpers.remove_breaks/0`.
+
+  To exit a breakpoint, the developer can either invoke `continue()`,
+  which will block the shell until the next breakpoint is found or
+  the process terminates, or invoke `respawn()`, which starts a new IEx
+  shell, freeing up the pried one.
+
+  This functionality only works on Elixir code and requires OTP 20+.
+
+  ## Examples
+
+  The following sets up a breakpoint on `URI.decode_query/2`:
+
+      IEx.break!(URI, :decode_query, 2)
+
+  This call will setup a breakpoint that stops once.
+  To set a breakpoint that will stop 10 times:
+
+      IEx.break!(URI, :decode_query, 2, 10)
+
+  `IEx.break!/2` is a convenience macro that allows breakpoints
+  to be given in the `Mod.fun/arity` format:
+
+      require IEx
+      IEx.break!(URI.decode_query/2)
+
+  Or to set a breakpoint that will stop 10 times:
+
+      IEx.break!(URI.decode_query/2, 10)
+
+  This function returns the breakpoint ID and will raise if there
+  is an error setting up the breakpoint.
+
+  ## Breaks and mix test
+
+  To use `IEx.break!/4` during tests, you need to run Mix inside
+  `iex` and pass the `--trace` to `mix test` to avoid running
+  into timeouts:
+
+      iex -S mix test --trace
+      iex -S mix test path/to/file:line --trace
+
+  """
+  def break!(module, function, arity, stops \\ 1) do
+    case IEx.Pry.break(module, function, arity, stops) do
+      {:ok, id} ->
+        id
+      {:error, kind} ->
+        message =
+          case kind do
+            :missing_debug_info ->
+              "module #{inspect module} was not compiled with debug_info"
+            :no_beam_file ->
+              "could not find .beam file for #{inspect module}"
+            :non_elixir_module ->
+              "module #{inspect module} was not written in Elixir"
+            :otp_20_is_required ->
+              "you are running on an earlier OTP version than OTP 20"
+            :outdated_debug_info ->
+              "module #{inspect module} was not compiled with the latest debug_info"
+            :recompilation_failed ->
+              "the module could not be compiled with breakpoints (likely an internal error)"
+            :unknown_function_arity ->
+              "unknown function/macro #{Exception.format_mfa(module, function, arity)}"
+          end
+        raise "could not set breakpoint, " <> message
+    end
   end
 
   ## Callbacks

lib/iex/lib/iex/app.ex

@@ -4,18 +4,7 @@ defmodule IEx.App do
   use Application
 
   def start(_type, _args) do
-    tab = IEx.Config.new()
-
-    case Supervisor.start_link([IEx.Config], strategy: :one_for_one, name: IEx.Supervisor) do
-      {:ok, pid} ->
-        {:ok, pid, tab}
-      {:error, _} = error ->
-        IEx.Config.delete(tab)
-        error
-    end
-  end
-
-  def stop(tab) do
-    IEx.Config.delete(tab)
+    children = [IEx.Config, IEx.Pry]
+    Supervisor.start_link(children, strategy: :one_for_one, name: IEx.Supervisor)
   end
 end

lib/iex/lib/iex/autocomplete.ex

@@ -395,8 +395,8 @@ defmodule IEx.Autocomplete do
   ## Evaluator interface
 
   defp imports_from_env(server) do
-    with evaluator when is_pid(evaluator) <- server.evaluator(),
-         env_fields = IEx.Evaluator.fields_from_env(evaluator, [:functions, :macros]),
+    with {evaluator, server} <- server.evaluator(),
+         env_fields = IEx.Evaluator.fields_from_env(evaluator, server, [:functions, :macros]),
          %{functions: funs, macros: macros} <- env_fields do
       Enum.flat_map(funs ++ macros, &elem(&1, 1))
     else
@@ -405,8 +405,8 @@ defmodule IEx.Autocomplete do
   end
 
   defp aliases_from_env(server) do
-    with evaluator when is_pid(evaluator) <- server.evaluator,
-         %{aliases: aliases} <- IEx.Evaluator.fields_from_env(evaluator, [:aliases]) do
+    with {evaluator, server} <- server.evaluator(),
+         %{aliases: aliases} <- IEx.Evaluator.fields_from_env(evaluator, server, [:aliases]) do
       aliases
     else
       _ -> []
@@ -414,17 +414,17 @@ defmodule IEx.Autocomplete do
   end
 
   defp variables_from_binding(hint, server) do
-    with evaluator when is_pid(evaluator) <- server.evaluator() do
-      IEx.Evaluator.variables_from_binding(evaluator, hint)
+    with {evaluator, server} <- server.evaluator() do
+      IEx.Evaluator.variables_from_binding(evaluator, server, hint)
     else
       _ -> []
     end
   end
 
   defp value_from_binding(ast_node, server) do
-    with evaluator when is_pid(evaluator) <- server.evaluator(),
+    with {evaluator, server} <- server.evaluator(),
          {var, map_key_path} <- extract_from_ast(ast_node, []) do
-      IEx.Evaluator.value_from_binding(evaluator, var, map_key_path)
+      IEx.Evaluator.value_from_binding(evaluator, server, var, map_key_path)
     else
       _ -> :error
     end

lib/iex/lib/iex/config.ex

@@ -119,17 +119,7 @@ defmodule IEx.Config do
   # Agent API
 
   def start_link(_) do
-    Agent.start_link(__MODULE__, :handle_init, [@table], [name: @agent])
-  end
-
-  def new() do
-    tab = :ets.new(@table, [:named_table, :public])
-    true = :ets.insert_new(tab, [after_spawn: []])
-    tab
-  end
-
-  def delete(__MODULE__) do
-    :ets.delete(__MODULE__)
+    Agent.start_link(__MODULE__, :handle_init, [], name: @agent)
   end
 
   def after_spawn(fun) do
@@ -146,9 +136,10 @@ defmodule IEx.Config do
 
   # Agent callbacks
 
-  def handle_init(tab) do
-    :public = :ets.info(tab, :protection)
-    tab
+  def handle_init do
+    :ets.new(@table, [:named_table, :public])
+    true = :ets.insert_new(@table, [after_spawn: []])
+    @table
   end
 
   def handle_after_spawn(tab, fun) do
@@ -164,10 +155,9 @@ defmodule IEx.Config do
   end
 
   defp update_configuration(config) do
-    put = fn({key, value}) when key in @keys ->
+    Enum.each(config, fn {key, value} when key in @keys ->
       Application.put_env(:iex, key, value)
-    end
-    Enum.each(config, put)
+    end)
   end
 
   defp merge_option(:colors, old, new) when is_list(new), do: Keyword.merge(old, new)

lib/iex/lib/iex/evaluator.ex

@@ -14,11 +14,11 @@ defmodule IEx.Evaluator do
     old_leader = Process.group_leader
     Process.group_leader(self(), leader)
 
-    state = loop_state(opts)
+    state = loop_state(server, IEx.History.init, opts)
     command == :ack && :proc_lib.init_ack(self())
 
     try do
-      loop(server, IEx.History.init, state)
+      loop(state)
     after
       Process.group_leader(self(), old_leader)
     end
@@ -28,10 +28,10 @@ defmodule IEx.Evaluator do
   Gets a value out of the binding, using the provided
   variable name and map key path.
   """
-  @spec value_from_binding(pid, atom, [atom]) :: {:ok, any} | :error
-  def value_from_binding(evaluator, var_name, map_key_path) do
+  @spec value_from_binding(pid, pid, atom, [atom]) :: {:ok, any} | :error
+  def value_from_binding(evaluator, server, var_name, map_key_path) do
     ref = make_ref()
-    send evaluator, {:value_from_binding, ref, self(), var_name, map_key_path}
+    send evaluator, {:value_from_binding, server, ref, self(), var_name, map_key_path}
 
     receive do
       {^ref, result} -> result
@@ -44,10 +44,10 @@ defmodule IEx.Evaluator do
   Gets a list of variables out of the binding that match the passed
   variable prefix.
   """
-  @spec variables_from_binding(pid, String.t) :: [String.t]
-  def variables_from_binding(evaluator, variable_prefix) do
+  @spec variables_from_binding(pid, pid, String.t) :: [String.t]
+  def variables_from_binding(evaluator, server, variable_prefix) do
     ref = make_ref()
-    send evaluator, {:variables_from_binding, ref, self(), variable_prefix}
+    send evaluator, {:variables_from_binding, server, ref, self(), variable_prefix}
 
     receive do
       {^ref, result} -> result
@@ -59,10 +59,10 @@ defmodule IEx.Evaluator do
   @doc """
   Returns the named fields from the current session environment.
   """
-  @spec fields_from_env(pid, [atom]) :: %{atom => term}
-  def fields_from_env(evaluator, fields) do
+  @spec fields_from_env(pid, pid, [atom]) :: %{atom => term}
+  def fields_from_env(evaluator, server, fields) do
     ref = make_ref()
-    send evaluator, {:fields_from_env, ref, self(), fields}
+    send evaluator, {:fields_from_env, server, ref, self(), fields}
 
     receive do
       {^ref, result} -> result
@@ -71,23 +71,23 @@ defmodule IEx.Evaluator do
     end
   end
 
-  defp loop(server, history, state) do
+  defp loop(%{server: server} = state) do
     receive do
       {:eval, ^server, code, iex_state} ->
-        {result, history, state} = eval(code, iex_state, history, state)
+        {result, state} = eval(code, iex_state, state)
         send server, {:evaled, self(), result}
-        loop(server, history, state)
-      {:fields_from_env, ref, receiver, fields} ->
+        loop(state)
+      {:fields_from_env, ^server, ref, receiver, fields} ->
         send receiver, {ref, Map.take(state.env, fields)}
-        loop(server, history, state)
-      {:value_from_binding, ref, receiver, var_name, map_key_path} ->
+        loop(state)
+      {:value_from_binding, ^server, ref, receiver, var_name, map_key_path} ->
         value = traverse_binding(state.binding, var_name, map_key_path)
         send receiver, {ref, value}
-        loop(server, history, state)
-      {:variables_from_binding, ref, receiver, var_prefix} ->
+        loop(state)
+      {:variables_from_binding, ^server, ref, receiver, var_prefix} ->
         value = find_matched_variables(state.binding, var_prefix)
         send receiver, {ref, value}
-        loop(server, history, state)
+        loop(state)
       {:done, ^server} ->
         :ok
     end
@@ -110,18 +110,13 @@ defmodule IEx.Evaluator do
         do: var_name
   end
 
-  defp loop_state(opts) do
-    env =
-      if env = opts[:env] do
-        :elixir.env_for_eval(env, [])
-      else
-        :elixir.env_for_eval(file: "iex")
-      end
-
+  defp loop_state(server, history, opts) do
+    env = opts[:env] || :elixir.env_for_eval(file: "iex")
+    env = %{env | match_vars: :apply}
     {_, _, env, scope} = :elixir.eval('import IEx.Helpers', [], env)
 
     binding = Keyword.get(opts, :binding, [])
-    state  = %{binding: binding, scope: scope, env: env}
+    state = %{binding: binding, scope: scope, env: env, server: server, history: history}
 
     case opts[:dot_iex_path] do
       ""   -> state
@@ -178,34 +173,48 @@ defmodule IEx.Evaluator do
   # https://github.com/elixir-lang/elixir/issues/1089 for discussion.
   @break_trigger '#iex:break\n'
 
-  defp eval(code, iex_state, history, state) do
+  defp eval(code, iex_state, state) do
     try do
-      do_eval(String.to_charlist(code), iex_state, history, state)
+      do_eval(String.to_charlist(code), iex_state, state)
     catch
       kind, error ->
         print_error(kind, error, System.stacktrace)
-        {%{iex_state | cache: ''}, history, state}
+        {%{iex_state | cache: ''}, state}
     end
   end
 
-  defp do_eval(@break_trigger, %IEx.State{cache: ''} = iex_state, history, state) do
-    {iex_state, history, state}
+  defp do_eval(@break_trigger, %IEx.State{cache: ''} = iex_state, state) do
+    {iex_state, state}
   end
 
-  defp do_eval(@break_trigger, iex_state, _history, _state) do
+  defp do_eval(@break_trigger, iex_state, _state) do
     :elixir_errors.parse_error(iex_state.counter, "iex", "incomplete expression", "")
   end
 
-  defp do_eval(latest_input, iex_state, history, state) do
+  defp do_eval(latest_input, iex_state, state) do
     code = iex_state.cache ++ latest_input
     line = iex_state.counter
-    Process.put(:iex_history, history)
-    handle_eval(Code.string_to_quoted(code, [line: line, file: "iex"]), code, line, iex_state, history, state)
+    put_history(state)
+    put_whereami(state)
+    quoted = Code.string_to_quoted(code, line: line, file: "iex")
+    handle_eval(quoted, code, line, iex_state, state)
   after
     Process.delete(:iex_history)
+    Process.delete(:iex_whereami)
   end
 
-  defp handle_eval({:ok, forms}, code, line, iex_state, history, state) do
+  defp put_history(%{history: history}) do
+    Process.put(:iex_history, history)
+  end
+
+  defp put_whereami(%{env: %{file: "iex"}}) do
+    :ok
+  end
+  defp put_whereami(%{env: %{file: file, line: line}}) do
+    Process.put(:iex_whereami, {file, line})
+  end
+
+  defp handle_eval({:ok, forms}, code, line, iex_state, state) do
     {result, binding, env, scope} =
       :elixir.eval_forms(forms, state.binding, state.env, state.scope)
     unless result == IEx.dont_display_result, do: io_inspect(result)
@@ -218,22 +227,23 @@ defmodule IEx.Evaluator do
                 scope: scope,
                 binding: binding}
 
-    {iex_state, update_history(history, line, code, result), state}
+    {iex_state, update_history(state, line, code, result)}
   end
 
-  defp handle_eval({:error, {_, _, ""}}, code, _line, iex_state, history, state) do
+  defp handle_eval({:error, {_, _, ""}}, code, _line, iex_state, state) do
     # Update iex_state.cache so that IEx continues to add new input to
     # the unfinished expression in "code"
-    {%{iex_state | cache: code}, history, state}
+    {%{iex_state | cache: code}, state}
   end
 
-  defp handle_eval({:error, {line, error, token}}, _code, _line, _iex_state, _, _state) do
+  defp handle_eval({:error, {line, error, token}}, _code, _line, _iex_state, _state) do
     # Encountered malformed expression
     :elixir_errors.parse_error(line, "iex", error, token)
   end
 
-  defp update_history(history, counter, cache, result) do
-    IEx.History.append(history, {counter, cache, result}, IEx.Config.history_size)
+  defp update_history(state, counter, cache, result) do
+    history_size = IEx.Config.history_size
+    update_in(state.history, &IEx.History.append(&1, {counter, cache, result}, history_size))
   end
 
   defp io_inspect(result) do

lib/iex/lib/iex/helpers.ex

@@ -22,27 +22,28 @@ defmodule IEx.Helpers do
 
   There are many other helpers available, here are some examples:
 
-    * `b/1`         - prints callbacks info and docs for a given module
-    * `c/1`         - compiles a file into the current directory
-    * `c/2`         - compiles a file to the given path
-    * `cd/1`        - changes the current directory
-    * `clear/0`     - clears the screen
-    * `e/1`         - shows all exports (functions + macros) in a module
-    * `flush/0`     - flushes all messages sent to the shell
-    * `h/0`         - prints this help message
-    * `h/1`         - prints help for the given module, function or macro
-    * `i/0`         - prints information about the last value
-    * `i/1`         - prints information about the given term
-    * `ls/0`        - lists the contents of the current directory
-    * `ls/1`        - lists the contents of the specified directory
-    * `pid/1`       - creates a PID from a string
-    * `pid/3`       - creates a PID with the 3 integer arguments passed
-    * `pwd/0`       - prints the current working directory
-    * `r/1`         - recompiles the given module's source file
-    * `recompile/0` - recompiles the current project
-    * `respawn/0`   - respawns the current shell
-    * `v/0`         - retrieves the last value from the history
-    * `v/1`         - retrieves the nth value from the history
+    * `b/1`            - prints callbacks info and docs for a given module
+    * `c/1`            - compiles a file into the current directory
+    * `c/2`            - compiles a file to the given path
+    * `cd/1`           - changes the current directory
+    * `clear/0`        - clears the screen
+    * `exports/1`      - shows all exports (functions + macros) in a module
+    * `flush/0`        - flushes all messages sent to the shell
+    * `h/0`            - prints this help message
+    * `h/1`            - prints help for the given module, function or macro
+    * `i/0`            - prints information about the last value
+    * `i/1`            - prints information about the given term
+    * `ls/0`           - lists the contents of the current directory
+    * `ls/1`           - lists the contents of the specified directory
+    * `open/1`         - opens the source for the given module or function in your editor
+    * `pid/1`          - creates a PID from a string
+    * `pid/3`          - creates a PID with the 3 integer arguments passed
+    * `pwd/0`          - prints the current working directory
+    * `r/1`            - recompiles the given module's source file
+    * `recompile/0`    - recompiles the current project
+    * `runtime_info/0` - prints runtime info (versions, memory usage, stats)
+    * `v/0`            - retrieves the last value from the history
+    * `v/1`            - retrieves the nth value from the history
 
   Help for all of those functions can be consulted directly from
   the command line using the `h/1` helper itself. Try:
@@ -52,7 +53,10 @@ defmodule IEx.Helpers do
   To list all IEx helpers available, which is effectively all
   exports (functions and macros) in the `IEx.Helpers` module:
 
-      iex> e(IEx.Helpers)
+      iex> exports(IEx.Helpers)
+
+  This module also include helpers for debugging purposes, see
+  `IEx.break!/4` for more information.
 
   To learn more about IEx as a whole, type `h(IEx)`.
   """
@@ -193,12 +197,74 @@ defmodule IEx.Helpers do
     dont_display_result()
   end
 
+  @doc """
+  Opens the current prying location.
+
+  This command only works inside a pry session started manually
+  via `IEx.pry/0` or a breakpoint set via `IEx.break!/4`. Calling
+  this function during a regular `IEx` session will print an error.
+
+  Keep in mind the `open/0` location may not exist when prying
+  precompiled source code, such as Elixir itself.
+
+  For more information and to open any module or function, see
+  `open/1`.
+  """
+  def open() do
+    case Process.get(:iex_whereami) do
+      {file, line} ->
+        IEx.Introspection.open({file, line})
+      _ ->
+        IO.puts IEx.color(:eval_error, "Pry session is not currently enabled")
+    end
+
+    dont_display_result()
+  end
+
+  @doc """
+  Opens the given module, module/function/arity or `{file, line}`.
+
+  This function uses the `ELIXIR_EDITOR` environment variable
+  and falls back to `EDITOR` if the former is not available.
+
+  By default, it attempts to open the file and line using the
+  `file:line` notation. For example, if your editor is called
+  `subl`, it will open the file as:
+
+      subl path/to/file:line
+
+  Custom editors are supported by using the __FILE__ and __LINE__
+  notations. For example, vi/vim users can set `ELIXIR_EDITOR` to:
+
+      ELIXIR_EDITOR="vi +__LINE__ __FILE__"
+
+  and Elixir will properly interpolate values.
+
+  Since this function prints the result returned by the editor,
+  `ELIXIR_EDITOR` can be set "echo" if you prefer to display the
+  location rather than opening it.
+
+  Keep in mind the location may not exist when opening
+  precompiled source code, such as Elixir itself.
+
+  ## Examples
+
+      iex> open MyApp
+      iex> open MyApp.fun/2
+      iex> open {"path/to/file", 1}
+
+  """
+  defmacro open(term) do
+    quote do
+      IEx.Introspection.open(unquote(IEx.Introspection.decompose(term)))
+    end
+  end
+
   @doc """
   Prints the documentation for `IEx.Helpers`.
   """
   def h() do
     IEx.Introspection.h(IEx.Helpers)
-    dont_display_result()
   end
 
   @doc """
@@ -217,42 +283,9 @@ defmodule IEx.Helpers do
       iex> h Enum.all?
 
   """
-  @h_modules [__MODULE__, Kernel, Kernel.SpecialForms]
-
-  defmacro h(term)
-  defmacro h({:/, _, [call, arity]} = term) do
-    args =
-      case Macro.decompose_call(call) do
-        {_mod, :__info__, []} when arity == 1 ->
-          [Module, :__info__, 1]
-        {mod, fun, []} ->
-          [mod, fun, arity]
-        {fun, []} ->
-          [@h_modules, fun, arity]
-        _ ->
-          [term]
-      end
-
+  defmacro h(term) do
     quote do
-      IEx.Introspection.h(unquote_splicing(args))
-    end
-  end
-
-  defmacro h(call) do
-    args =
-      case Macro.decompose_call(call) do
-        {_mod, :__info__, []} ->
-          [Module, :__info__, 1]
-        {mod, fun, []} ->
-          [mod, fun]
-        {fun, []} ->
-          [@h_modules, fun]
-        _ ->
-          [call]
-      end
-
-    quote do
-      IEx.Introspection.h(unquote_splicing(args))
+      IEx.Introspection.h(unquote(IEx.Introspection.decompose(term)))
     end
   end
 
@@ -268,22 +301,9 @@ defmodule IEx.Helpers do
       iex> b(Mix.Task.run)
       iex> b(GenServer)
   """
-  defmacro b(term)
-  defmacro b({:/, _, [{{:., _, [mod, fun]}, _, []}, arity]}) do
+  defmacro b(term) do
     quote do
-      IEx.Introspection.b(unquote(mod), unquote(fun), unquote(arity))
-    end
-  end
-
-  defmacro b({{:., _, [mod, fun]}, _, []}) do
-    quote do
-      IEx.Introspection.b(unquote(mod), unquote(fun))
-    end
-  end
-
-  defmacro b(module) do
-    quote do
-      IEx.Introspection.b(unquote(module))
+      IEx.Introspection.b(unquote(IEx.Introspection.decompose(term)))
     end
   end
 
@@ -305,22 +325,9 @@ defmodule IEx.Helpers do
       @type t() :: Enumerable.t()
 
   """
-  defmacro t(term)
-  defmacro t({:/, _, [{{:., _, [mod, fun]}, _, []}, arity]}) do
+  defmacro t(term) do
     quote do
-      IEx.Introspection.t(unquote(mod), unquote(fun), unquote(arity))
-    end
-  end
-
-  defmacro t({{:., _, [mod, fun]}, _, []}) do
-    quote do
-      IEx.Introspection.t(unquote(mod), unquote(fun))
-    end
-  end
-
-  defmacro t(module) do
-    quote do
-      IEx.Introspection.t(unquote(module))
+      IEx.Introspection.t(unquote(IEx.Introspection.decompose(term)))
     end
   end
 
@@ -336,30 +343,9 @@ defmodule IEx.Helpers do
       iex> s(is_atom/1)
 
   """
-  defmacro s(term)
-  defmacro s({:/, _, [call, arity]} = term) do
-    args =
-      case Macro.decompose_call(call) do
-        {mod, fun, []} -> [mod, fun, arity]
-        {fun, []} -> [Kernel, fun, arity]
-        _ -> [term]
-      end
-
+  defmacro s(term) do
     quote do
-      IEx.Introspection.s(unquote_splicing(args))
-    end
-  end
-
-  defmacro s(call) do
-    args =
-      case Macro.decompose_call(call) do
-        {mod, fun, []} -> [mod, fun]
-        {fun, []} -> [Kernel, fun]
-        _ -> [call]
-      end
-
-    quote do
-      IEx.Introspection.s(unquote_splicing(args))
+      IEx.Introspection.s(unquote(IEx.Introspection.decompose(term)))
     end
   end
 
@@ -497,6 +483,77 @@ defmodule IEx.Helpers do
     |> Enum.map_join(", ", &inspect/1)
   end
 
+  @doc """
+  Prints vm/runtime information such as versions, memory usage and statistics.
+  """
+  def runtime_info do
+    print_pane("System and architecture")
+
+    print_entry("Elixir version", System.version)
+    print_entry("OTP version", :erlang.system_info(:otp_release))
+    print_entry("ERTS version", :erlang.system_info(:version))
+    print_entry("Compiled for", :erlang.system_info(:system_architecture))
+    print_entry("Schedulers", :erlang.system_info(:schedulers))
+    print_entry("Schedulers online", :erlang.system_info(:schedulers_online))
+
+    print_pane("Memory")
+    print_memory("Total", :total, :MB)
+    print_memory("Atoms", :atom)
+    print_memory("Binaries", :binary)
+    print_memory("Code", :code)
+    print_memory("ETS", :ets)
+    print_memory("Processes", :processes)
+
+    print_pane("Statistics / limits")
+    print_uptime()
+    print_entry("Run queue", :erlang.statistics(:run_queue))
+    if :erlang.system_info(:otp_release) >= '20' do
+      print_percentage("Atoms", :atom_count, :atom_limit)
+    end
+    print_percentage("ETS", :ets_count, :ets_limit)
+    print_percentage("Ports", :port_count, :port_limit)
+    print_percentage("Processes", :process_count, :process_limit)
+
+    IO.puts ""
+    dont_display_result()
+  end
+
+  defp print_pane(msg) do
+    IO.puts IEx.color(:eval_result, ["\n## ", msg, " \n"])
+  end
+
+  defp print_entry(_key, nil), do: :ok
+  defp print_entry(key, value), do: IO.puts "#{pad_key(key)}#{value}"
+
+  defp print_uptime() do
+    IO.write pad_key("Uptime")
+    :c.uptime()
+  end
+
+  defp print_percentage(key, min, max) do
+    min = get_stat(min)
+    max = get_stat(max)
+    percentage = trunc((min / max) * 100)
+    IO.puts "#{pad_key(key)}#{min} / #{max} (#{percentage}% used)"
+  end
+
+  defp get_stat(:ets_count), do: length(:ets.all())
+  defp get_stat(other), do: :erlang.system_info(other)
+
+  defp print_memory(key, memory, unit \\ :kB) do
+    value =
+      memory
+      |> :erlang.memory()
+      |> div(memory_unit(unit))
+      |> round()
+    IO.puts "#{pad_key(key)}#{value} #{unit}"
+  end
+
+  defp memory_unit(:MB), do: 1024 * 1024
+  defp memory_unit(:kB), do: 1024
+
+  defp pad_key(key), do: String.pad_trailing("#{key}:", 20, " ")
+
   @doc """
   Flushes all messages sent to the shell and prints them out.
   """
@@ -546,14 +603,14 @@ defmodule IEx.Helpers do
   @doc """
   Prints a list of all the functions and macros exported by the given module.
   """
-  def e(module \\ Kernel) do
-    IEx.Autocomplete.exports(module) |> print_exports()
-    dont_display_result()
-  end
-
-  defp print_exports(functions) do
-    list = Enum.map(functions, fn({name, arity}) -> Atom.to_string(name) <> "/" <> Integer.to_string(arity) end)
+  def exports(module \\ Kernel) do
+    exports = IEx.Autocomplete.exports(module)
+    list =
+      Enum.map(exports, fn {name, arity} ->
+        Atom.to_string(name) <> "/" <> Integer.to_string(arity)
+      end)
     print_table(list)
+    dont_display_result()
   end
 
   @doc """
@@ -632,14 +689,177 @@ defmodule IEx.Helpers do
 
   @doc """
   Respawns the current shell by starting a new shell process.
-
-  Returns `true` if it worked.
   """
   def respawn do
     if whereis = IEx.Server.whereis do
       send whereis, {:respawn, self()}
-      dont_display_result()
     end
+    dont_display_result()
+  end
+
+  @doc """
+  Continues execution of the current process.
+
+  This is usually called by sessions started with `IEx.pry/0`
+  or `IEx.break!/4`. This allows the current to execute until
+  the next breakpoint, which will automatically yield control
+  back to IEx without requesting permission to pry.
+
+  If the running process terminates, a new IEx session is
+  started.
+
+  While the process executes, the user will no longer have
+  control of the shell. If you would rather start a new shell,
+  use `respawn/0` instead.
+  """
+  def continue do
+    if whereis = IEx.Server.whereis do
+      send whereis, {:continue, self()}
+    end
+    dont_display_result()
+  end
+
+  @doc """
+  Macro-based shortcut for `IEx.break!/4`.
+  """
+  defmacro break!(ast, stops \\ 1) do
+    quote do
+      require IEx
+      IEx.break!(unquote(ast), unquote(stops))
+    end
+  end
+
+  @doc """
+  Sets up a breakpoint in `module`, `function` and `arity`
+  with the given number of `stops`.
+
+  See `IEx.break!/4` for a complete description of breakpoints
+  in IEx.
+  """
+  defdelegate break!(module, function, arity, stops \\ 1), to: IEx
+
+  @doc """
+  Prints all breakpoints to the terminal.
+  """
+  def breaks do
+    breaks(IEx.Pry.breaks())
+  end
+
+  defp breaks([]) do
+    IO.puts IEx.color(:eval_info, "No breakpoints set")
+    dont_display_result()
+  end
+
+  defp breaks(breaks) do
+    entries =
+      for {id, module, {function, arity}, stops} <- breaks do
+        {Integer.to_string(id),
+         Exception.format_mfa(module, function, arity),
+         Integer.to_string(stops)}
+      end
+
+    entries = [{"ID", "Module.function/arity", "Pending stops"} | entries]
+
+    {id_max, mfa_max, stops_max} =
+      Enum.reduce(entries, {0, 0, 0}, fn {id, mfa, stops}, {id_max, mfa_max, stops_max} ->
+        {max(byte_size(id), id_max),
+         max(byte_size(mfa), mfa_max),
+         max(byte_size(stops), stops_max)}
+      end)
+
+    [header | entries] = entries
+
+    IO.puts ""
+    print_break(header, id_max, mfa_max)
+    IO.puts [String.duplicate("-", id_max + 2), ?\s,
+             String.duplicate("-", mfa_max + 2), ?\s,
+             String.duplicate("-", stops_max + 2)]
+    Enum.each(entries, &print_break(&1, id_max, mfa_max))
+    IO.puts ""
+
+    dont_display_result()
+  end
+
+  defp print_break({id, mfa, stops}, id_max, mfa_max) do
+    IO.puts [?\s, String.pad_trailing(id, id_max + 2),
+             ?\s, String.pad_trailing(mfa, mfa_max + 2),
+             ?\s, stops]
+  end
+
+  @doc """
+  Sets the number of pending stops in the breakpoint
+  with the given id to zero.
+
+  Returns `:ok` if there is such breakpoint id. `:not_found`
+  otherwise.
+
+  Note the module remains "instrumented" on reset. If you would
+  like to effectively remove all breakpoints and instrumentation
+  code from a module, use `remove_breaks/1` instead.
+  """
+  defdelegate reset_break(id), to: IEx.Pry
+
+  @doc """
+  Sets the number of pending stops in the given module,
+  function and arity to zero.
+
+  If the module is not instrumented or if the given function
+  does not have a breakpoint, it is a no-op and it returns
+  `:not_found`. Otherwise it returns `:ok`.
+
+  Note the module remains "instrumented" on reset. If you would
+  like to effectively remove all breakpoints and instrumentation
+  code from a module, use `remove_breaks/1` instead.
+  """
+  defdelegate reset_break(module, function, arity), to: IEx.Pry
+
+  @doc """
+  Removes all breakpoints and instrumentation from `module`.
+  """
+  defdelegate remove_breaks(module), to: IEx.Pry
+
+  @doc """
+  Removes all breakpoints and instrumentation from all modules.
+  """
+  defdelegate remove_breaks(), to: IEx.Pry
+
+  @doc """
+  Prints the current location in a pry session.
+
+  It expects a `radius` which chooses how many lines before and after
+  the current line we should print. By default the `radius` is of two
+  lines:
+
+      Location: lib/iex/lib/iex/helpers.ex:79
+
+      77:
+      78:   def recompile do
+      79:     require IEx; IEx.pry
+      80:     if mix_started?() do
+      81:       config = Mix.Project.config
+
+  This command only works inside a pry session started manually
+  via `IEx.pry/0` or a breakpoint set via `IEx.break!/4`. Calling
+  this function during a regular `IEx` session will print an error.
+
+  Keep in mind the `whereami/1` location may not exist when prying
+  precompiled source code, such as Elixir itself.
+  """
+  def whereami(radius \\ 2) do
+    case Process.get(:iex_whereami) do
+      {file, line} ->
+        IO.puts IEx.color(:eval_info, ["Location: ", Path.relative_to_cwd(file), ":", Integer.to_string(line)])
+        case IEx.Pry.whereami(file, line, radius) do
+          {:ok, lines} ->
+            IO.write [?\n, lines, ?\n]
+          :error ->
+            IO.puts IEx.color(:eval_error, "Could not extract source snippet. Location is not available.")
+        end
+      _ ->
+        IO.puts IEx.color(:eval_error, "Pry session is not currently enabled")
+    end
+
+    dont_display_result()
   end
 
   @doc """

lib/iex/lib/iex/introspection.ex

@@ -1,5 +1,5 @@
-# Convenience helpers for showing docs, specs and types
-# from modules. Invoked directly from IEx.Helpers.
+# Convenience helpers for showing docs, specs, types
+# and opening modules. Invoked directly from IEx.Helpers.
 defmodule IEx.Introspection do
   @moduledoc false
 
@@ -8,7 +8,158 @@ defmodule IEx.Introspection do
   alias Kernel.Typespec
 
   @doc """
-  Prints the documentation for the given module.
+  Decomposes an introspection call into `{mod, fun, arity}`,
+  `{mod, fun}` or `mod`.
+  """
+  @default_modules [IEx.Helpers, Kernel, Kernel.SpecialForms]
+
+  def decompose({:/, _, [call, arity]} = term) do
+    case Macro.decompose_call(call) do
+      {_mod, :__info__, []} when arity == 1 ->
+        {:{}, [], [Module, :__info__, 1]}
+      {mod, fun, []} ->
+        {:{}, [], [mod, fun, arity]}
+      {fun, []} ->
+        {:{}, [], [find_decompose_fun_arity(fun, arity), fun, arity]}
+      _ ->
+        term
+    end
+  end
+
+  def decompose(call) do
+    case Macro.decompose_call(call) do
+      {_mod, :__info__, []} ->
+        Macro.escape {Module, :__info__, 1}
+      {mod, fun, []} ->
+        {mod, fun}
+      {fun, []} ->
+        {find_decompose_fun(fun), fun}
+      _ ->
+        call
+    end
+  end
+
+  defp find_decompose_fun(fun) do
+    Enum.find(@default_modules, Kernel, fn mod ->
+      Keyword.has_key?(mod.__info__(:functions), fun) or
+        Keyword.has_key?(mod.__info__(:macros), fun)
+    end)
+  end
+
+  defp find_decompose_fun_arity(fun, arity) do
+    pair = {fun, arity}
+    Enum.find(@default_modules, Kernel, fn mod ->
+      pair in mod.__info__(:functions) or pair in mod.__info__(:macros)
+    end)
+  end
+
+  @doc """
+  Opens the given module, mfa, file/line, binary.
+  """
+  def open(module) when is_atom(module) do
+    case open_mfa(module, :__info__, 1) do
+      {source, nil, _} -> open(source)
+      {_, tuple, _} -> open(tuple)
+      :error -> puts_error("Could not open: #{inspect module}. Module is not available.")
+    end
+    dont_display_result()
+  end
+
+  def open({module, function}) when is_atom(module) and is_atom(function) do
+    case open_mfa(module, function, :*) do
+      {_, _, nil} -> puts_error("Could not open: #{inspect module}.#{function}. Function/macro is not available.")
+      {_, _, tuple} -> open(tuple)
+      :error -> puts_error("Could not open: #{inspect module}.#{function}. Module is not available.")
+    end
+    dont_display_result()
+  end
+
+  def open({module, function, arity}) when is_atom(module) and is_atom(function) and is_integer(arity) do
+    case open_mfa(module, function, arity) do
+      {_, _, nil} -> puts_error("Could not open: #{inspect module}.#{function}/#{arity}. Function/macro is not available.")
+      {_, _, tuple} -> open(tuple)
+      :error -> puts_error("Could not open: #{inspect module}.#{function}/#{arity}. Module is not available.")
+    end
+    dont_display_result()
+  end
+
+  def open({file, line}) when is_binary(file) and is_integer(line) do
+    cond do
+      not File.regular?(file) ->
+        puts_error("Could not open: #{inspect file}. File is not available.")
+      editor = System.get_env("ELIXIR_EDITOR") || System.get_env("EDITOR") ->
+        command =
+          if editor =~ "__FILE__" or editor =~ "__LINE__" do
+            editor
+            |> String.replace("__FILE__", inspect(file))
+            |> String.replace("__LINE__", Integer.to_string(line))
+          else
+            "#{editor} #{inspect file}:#{line}"
+          end
+        IO.write IEx.color(:eval_info, :os.cmd(String.to_charlist(command)))
+      true ->
+        puts_error("Could not open: #{inspect file}. " <>
+                   "Please set the ELIXIR_EDITOR or EDITOR environment variables with the " <>
+                   "command line invocation of your favorite EDITOR.")
+    end
+
+    dont_display_result()
+  end
+
+  def open(invalid) do
+    puts_error("Invalid arguments for open helper: #{inspect invalid}")
+    dont_display_result()
+  end
+
+  defp open_mfa(module, fun, arity) do
+    with {:module, _} <- Code.ensure_loaded(module),
+         source when is_list(source) <- module.module_info(:compile)[:source] do
+      open_abstract_code(module, fun, arity, List.to_string(source))
+    else
+      _ -> :error
+    end
+  end
+
+  defp open_abstract_code(module, fun, arity, source) do
+    fun = Atom.to_string(fun)
+
+    with beam when is_list(beam) <- :code.which(module),
+         {:ok, {_, [abstract_code: {:raw_abstract_v1, code}]}} <- :beam_lib.chunks(beam, [:abstract_code]) do
+      {_, module_pair, fa_pair} =
+        Enum.reduce(code, {source, nil, nil}, &open_abstract_code_reduce(&1, &2, fun, arity))
+      {source, module_pair, fa_pair}
+    else
+      _ ->
+        {source, nil, nil}
+    end
+  end
+
+  defp open_abstract_code_reduce(entry, {file, module_pair, fa_pair}, fun, arity) do
+    case entry do
+      {:attribute, _, :file, {ann_file, _}} ->
+        if Path.type(ann_file) == :absolute do
+          {List.to_string(ann_file), module_pair, fa_pair}
+        else
+          {file, module_pair, fa_pair}
+        end
+      {:attribute, ann, :module, _} ->
+        {file, {file, :erl_anno.line(ann)}, fa_pair}
+      {:function, ann, ann_fun, ann_arity, _} ->
+        case Atom.to_string(ann_fun) do
+          "MACRO-" <> ^fun when arity == :* or ann_arity == arity + 1 ->
+            {file, module_pair, fa_pair || {file, :erl_anno.line(ann)}}
+          ^fun when arity == :* or ann_arity == arity->
+            {file, module_pair, fa_pair || {file, :erl_anno.line(ann)}}
+          _ ->
+            {file, module_pair, fa_pair}
+        end
+      _ ->
+        {file, module_pair, fa_pair}
+    end
+  end
+
+  @doc """
+  Prints documentation.
   """
   def h(module) when is_atom(module) do
     case Code.ensure_loaded(module) do
@@ -31,27 +182,7 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  def h(_) do
-    puts_error("Invalid arguments for h helper")
-    dont_display_result()
-  end
-
-  @doc """
-  Prints the documentation for the given function
-  with any arity in the list of modules.
-  """
-  def h(modules, function) when is_list(modules) and is_atom(function) do
-    printed? =
-      Enum.any?(modules, fn module ->
-        h_mod_fun(module, function) == :ok
-      end)
-
-    unless printed?, do: no_docs(function)
-
-    dont_display_result()
-  end
-
-  def h(module, function) when is_atom(module) and is_atom(function) do
+  def h({module, function}) when is_atom(module) and is_atom(function) do
     case h_mod_fun(module, function) do
       :ok ->
         :ok
@@ -66,41 +197,7 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  defp h_mod_fun(mod, fun) when is_atom(mod) do
-    if docs = Code.get_docs(mod, :docs) do
-      result = for {{^fun, arity}, _, _, _, _} = doc <- docs, has_content?(doc) do
-        h(mod, fun, arity)
-      end
-
-      cond do
-        result != [] ->
-          :ok
-        has_callback?(mod, fun) ->
-          :behaviour_found
-        true ->
-          :not_found
-      end
-    else
-      :no_docs
-    end
-  end
-
-  @doc """
-  Prints the documentation for the given function
-  and arity in the list of modules.
-  """
-  def h(modules, function, arity) when is_list(modules) and is_atom(function) and is_integer(arity) do
-    printed? =
-      Enum.any?(modules, fn module ->
-        h_mod_fun_arity(module, function, arity) == :ok
-      end)
-
-    unless printed?, do: no_docs("#{function}/#{arity}")
-
-    dont_display_result()
-  end
-
-  def h(module, function, arity) when is_atom(module) and is_atom(function) and is_integer(arity) do
+  def h({module, function, arity}) when is_atom(module) and is_atom(function) and is_integer(arity) do
     case h_mod_fun_arity(module, function, arity) do
       :ok ->
         :ok
@@ -115,6 +212,31 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
+  def h(invalid) do
+    puts_error("Invalid arguments for h helper: #{inspect invalid}")
+    dont_display_result()
+  end
+
+  defp h_mod_fun(mod, fun) when is_atom(mod) do
+    if docs = Code.get_docs(mod, :docs) do
+      result =
+        for {{^fun, arity}, _, _, _, _} = doc <- docs, has_content?(doc) do
+          h({mod, fun, arity})
+        end
+
+      cond do
+        result != [] ->
+          :ok
+        has_callback?(mod, fun) ->
+          :behaviour_found
+        true ->
+          :not_found
+      end
+    else
+      :no_docs
+    end
+  end
+
   defp h_mod_fun_arity(mod, fun, arity) when is_atom(mod) do
     docs = Code.get_docs(mod, :docs)
 
@@ -214,7 +336,7 @@ defmodule IEx.Introspection do
   end
 
   @doc """
-  Prints the list of behaviour callbacks for the given module.
+  Prints the list of behaviour callbacks or a given callback.
   """
   def b(mod) when is_atom(mod) do
     printer = fn heading, _doc -> puts_info(heading) end
@@ -228,10 +350,7 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  @doc """
-  Prints documentation for the given callback function with any arity.
-  """
-  def b(mod, fun) when is_atom(mod) and is_atom(fun) do
+  def b({mod, fun}) when is_atom(mod) and is_atom(fun) do
     filter = &match?({^fun, _}, elem(&1, 0))
     case print_callback_docs(mod, filter, &print_doc/2) do
       :ok        -> :ok
@@ -243,10 +362,7 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  @doc """
-  Prints documentation for the given callback function and arity.
-  """
-  def b(mod, fun, arity) when is_atom(mod) and is_atom(fun) and is_integer(arity) do
+  def b({mod, fun, arity}) when is_atom(mod) and is_atom(fun) and is_integer(arity) do
     filter = &match?({^fun, ^arity}, elem(&1, 0))
     case print_callback_docs(mod, filter, &print_doc/2) do
       :ok        -> :ok
@@ -258,6 +374,11 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
+  def b(invalid) do
+    puts_error("Invalid arguments for b helper: #{inspect invalid}")
+    dont_display_result()
+  end
+
   defp print_callback_docs(mod, filter, printer) do
     case get_callback_docs(mod) do
       {callbacks, docs} ->
@@ -273,8 +394,8 @@ defmodule IEx.Introspection do
           [] -> :not_found
           _ -> :ok
         end
-
-      other -> other
+      other ->
+        other
     end
   end
 
@@ -307,24 +428,25 @@ defmodule IEx.Introspection do
     do: other
 
   @doc """
-  Prints the types for the given module.
+  Prints the types for the given module and type documentation.
   """
   def t(module) when is_atom(module) do
-    _ = case Typespec.beam_types(module) do
-      nil   -> no_beam(module)
-      []    -> no_types(inspect module)
-      types -> Enum.each(types, &print_type/1)
+    case Typespec.beam_types(module) do
+      nil ->
+        no_beam(module)
+      [] ->
+        no_types(inspect module)
+      types ->
+        Enum.each(types, &print_type/1)
     end
 
     dont_display_result()
   end
 
-  @doc """
-  Prints the given type in module with any arity.
-  """
-  def t(module, type) when is_atom(module) and is_atom(type) do
+  def t({module, type}) when is_atom(module) and is_atom(type) do
     case Typespec.beam_types(module) do
-      nil   -> no_beam(module)
+      nil ->
+        no_beam(module)
       types ->
         printed =
           for {_, {^type, _, _args}} = typespec <- types do
@@ -341,12 +463,10 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  @doc """
-  Prints the type in module with given arity.
-  """
-  def t(module, type, arity) when is_atom(module) and is_atom(type) and is_integer(arity) do
+  def t({module, type, arity}) when is_atom(module) and is_atom(type) and is_integer(arity) do
     case Typespec.beam_types(module) do
-      nil   -> no_beam(module)
+      nil ->
+        no_beam(module)
       types ->
         printed =
           for {_, {^type, _, args}} = typespec <- types, length(args) == arity do
@@ -363,6 +483,11 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
+  def t(invalid) do
+    puts_error("Invalid arguments for t helper: #{inspect invalid}")
+    dont_display_result()
+  end
+
   defp print_type_doc(module, type) do
     docs  = Code.get_docs(module, :type_docs)
     {_, _, _, content} = Enum.find(docs, fn({{name, _}, _, _, _}) ->
@@ -372,16 +497,17 @@ defmodule IEx.Introspection do
   end
 
   @doc """
-  Prints the specs for given module.
+  Prints the specs for given module or function/arity.
   """
   def s(module) when is_atom(module) do
-    case beam_specs(module) do
-      nil   -> no_beam(module)
-      []    -> no_specs(inspect module)
+    case Typespec.beam_specs(module) do
+      nil ->
+        no_beam(module)
       specs ->
-        printed = for {_kind, {{fun, _arity}, _spec}} = spec <- specs, fun != :__info__ do
-          print_spec(spec)
-        end
+        printed =
+          for {{fun, _}, _} = spec <- specs, fun != :__info__ do
+            print_spec(spec)
+          end
         if printed == [] do
           no_specs(inspect module)
         end
@@ -390,17 +516,14 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  @doc """
-  Prints the specs for given module and function.
-  """
-  def s(module, function) when is_atom(module) and is_atom(function) do
-    case beam_specs(module) do
-      nil   -> no_beam(module)
+  def s({module, function}) when is_atom(module) and is_atom(function) do
+    case Typespec.beam_specs(module) do
+      nil ->
+        no_beam(module)
       specs ->
         printed =
-          for {_kind, {{^function, _arity}, _spec}} = spec <- specs do
+          for {{^function, _}, _} = spec <- specs do
             print_spec(spec)
-            spec
           end
 
         if printed == [] do
@@ -411,17 +534,14 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  @doc """
-  Prints the spec in given module, with arity.
-  """
-  def s(module, function, arity) when is_atom(module) and is_atom(function) and is_integer(arity) do
-    case beam_specs(module) do
-      nil   -> no_beam(module)
+  def s({module, function, arity}) when is_atom(module) and is_atom(function) and is_integer(arity) do
+    case Typespec.beam_specs(module) do
+      nil ->
+        no_beam(module)
       specs ->
         printed =
-          for {_kind, {{^function, ^arity}, _spec}} = spec <- specs do
+          for {{^function, ^arity}, _} = spec <- specs do
             print_spec(spec)
-            spec
           end
 
         if printed == [] do
@@ -432,15 +552,9 @@ defmodule IEx.Introspection do
     dont_display_result()
   end
 
-  defp beam_specs(module) do
-    specs = beam_specs_tag(Typespec.beam_specs(module), :spec)
-    callbacks = beam_specs_tag(Typespec.beam_callbacks(module), :callback)
-    specs && callbacks && Enum.concat(specs, callbacks)
-  end
-
-  defp beam_specs_tag(nil, _), do: nil
-  defp beam_specs_tag(specs, tag) do
-    Enum.map(specs, &{tag, &1})
+  def s(invalid) do
+    puts_error("Invalid arguments for s helper: #{inspect invalid}")
+    dont_display_result()
   end
 
   defp print_type({:opaque, type}) do
@@ -455,10 +569,10 @@ defmodule IEx.Introspection do
     true
   end
 
-  defp print_spec({kind, {{name, _arity}, specs}}) do
+  defp print_spec({{name, _arity}, specs}) do
     Enum.each specs, fn(spec) ->
       binary = Macro.to_string Typespec.spec_to_ast(name, spec)
-      puts_info("@#{kind} #{binary}")
+      puts_info("@spec #{binary}")
     end
     true
   end

lib/iex/lib/iex/pry.ex

@@ -0,0 +1,393 @@
+defmodule IEx.Pry do
+  @moduledoc """
+  The low-level API for prying sessions and setting up breakpoints.
+  """
+
+  use GenServer
+
+  @table __MODULE__
+  @server __MODULE__
+  @timeout :infinity
+
+  @type id :: integer()
+  @type break :: {id, module, {function, arity}, pending :: non_neg_integer}
+
+  @doc """
+  Callback for `IEx.pry/1`.
+
+  You can invoke this function directly when you are not able to invoke
+  `IEx.pry/1` as a macro. This function expects the binding (from
+  `Kernel.binding/0`) and the environment (from `__ENV__/0`).
+  """
+  def pry(binding, %Macro.Env{} = env) do
+    %{file: file, line: line, module: module, function: function_arity} = env
+    self = self()
+    opts = [binding: binding, dot_iex_path: "", env: env, prefix: "pry"]
+
+    location =
+      case function_arity do
+        {function, arity} ->
+          "#{Exception.format_mfa(module, function, arity)} (#{Path.relative_to_cwd(file)}:#{line})"
+        _ ->
+          "#{Path.relative_to_cwd(file)}:#{line}"
+      end
+
+    whereami =
+      case whereami(file, line, 2) do
+        {:ok, lines} -> [?\n, ?\n, lines]
+        :error -> []
+      end
+
+    # If we are the current evaluator, it is because we just
+    # reached a pry/breakpoint and the user hit continue().
+    # In both cases, we are safe to print and the request will
+    # suceed.
+    request =
+      case IEx.Server.evaluator do
+        {^self, _} ->
+          IO.puts IEx.color :eval_interrupt, "Break reached: #{location}#{whereami}"
+          "Prying #{inspect self} at #{location}"
+        _ ->
+          "Request to pry #{inspect self} at #{location}#{whereami}"
+      end
+
+    # We cannot use colors because IEx may be off
+    case IEx.Server.take_over(request, opts) do
+      :ok ->
+        :ok
+      {:error, :no_iex} ->
+        extra =
+          if match?({:win32, _}, :os.type) do
+            " If you are using Windows, you may need to start IEx with the --werl flag."
+          else
+            ""
+          end
+        IO.puts :stdio, "Cannot pry #{inspect self} at #{location}. Is an IEx shell running?" <> extra
+        {:error, :no_iex}
+      {:error, _} = error ->
+        error
+    end
+  end
+
+  def pry(binding, opts) when is_list(opts) do
+    vars = for {k, _} when is_atom(k) <- binding, do: {k, nil}
+    pry(binding, %{:elixir.env_for_eval(opts) | vars: vars})
+  end
+
+  @doc """
+  Formats the location for `whereami/3` prying.
+
+  It receives the `file`, `line` and the snippet `radius` and
+  returns `{:ok, lines}`, where lines is a list of chardata
+  containing each formatted line, or `:error`.
+
+  The actual line is especially formatted in bold.
+  """
+  def whereami(file, line, radius)
+      when is_binary(file) and is_integer(line) and is_integer(radius) and radius > 0 do
+    with true <- File.regular?(file),
+         [_ | _] = lines <- whereami_lines(file, line, radius) do
+      {:ok, lines}
+    else
+      _ -> :error
+    end
+  end
+
+  defp whereami_lines(file, line, radius) do
+    min = max(line - radius - 1, 0)
+    max = line + radius - 1
+
+    file
+    |> File.stream!
+    |> Enum.slice(min..max)
+    |> Enum.with_index(min + 1)
+    |> Enum.map(&whereami_format_line(&1, line))
+  end
+
+  defp whereami_format_line({line_text, line_number}, line) do
+    gutter = String.pad_leading(Integer.to_string(line_number), 5, " ")
+    if line_number == line do
+      IO.ANSI.format_fragment([:bright, gutter, ": ", line_text, :normal])
+    else
+      [gutter, ": ", line_text]
+    end
+  end
+
+  @doc """
+  Sets up a breakpoint on the given module/function or
+  given module/function/arity.
+  """
+  @spec break(module, function, arity, pos_integer) ::
+        {:ok, id} |
+        {:error, :recompilation_failed | :no_beam_file | :unknown_function_arity |
+                 :otp_20_is_required | :missing_debug_info | :outdated_debug_info |
+                 :non_elixir_module}
+  def break(module, function, arity, breaks \\ 1)
+      when is_atom(module) and is_atom(function) and is_integer(arity) and arity >= 0 and
+           is_integer(breaks) and breaks > 0 do
+    GenServer.call(@server, {:break, module, {function, arity}, breaks}, @timeout)
+  end
+
+  @doc """
+  Resets the breaks on a given breakpoint id.
+  """
+  @spec reset_break(id) :: :ok | :not_found
+  def reset_break(id) when is_integer(id) do
+    GenServer.call(@server, {:reset_break, {id, :_, :_, :_}}, @timeout)
+  end
+
+  @doc """
+  Resets the breaks for the given module, function and arity.
+
+  If the module is not instrumented or if the given function
+  does not have a breakpoint, it is a no-op and it returns
+  `:not_found`. Otherwise it returns `:ok`.
+  """
+  @spec reset_break(module, function, arity) :: :ok | :not_found
+  def reset_break(module, function, arity) do
+    GenServer.call(@server, {:reset_break, {:_, module, {function, arity}, :_}}, @timeout)
+  end
+
+  @doc """
+  Removes all breakpoints on all modules.
+
+  This effectively loads the non-instrumented version of
+  currently instrumented modules into memory.
+  """
+  @spec remove_breaks :: :ok
+  def remove_breaks do
+    GenServer.call(@server, :remove_breaks, @timeout)
+  end
+
+  @doc """
+  Removes breakpoints in the given module.
+
+  This effectively loads the non-instrumented version of
+  the module into memory.
+  """
+  @spec remove_breaks(module) :: :ok | {:error, :no_beam_file}
+  def remove_breaks(module) do
+    GenServer.call(@server, {:remove_breaks, module}, @timeout)
+  end
+
+  @doc """
+  Returns all breakpoints.
+  """
+  @spec breaks :: [break]
+  def breaks do
+    @server
+    |> GenServer.call(:breaks, @timeout)
+    |> Enum.sort()
+  end
+
+  ## Callbacks
+
+  @doc false
+  def start_link(_) do
+    GenServer.start_link(__MODULE__, :ok, name: @server)
+  end
+
+  def init(:ok) do
+    Process.flag(:trap_exit, true)
+    :ets.new(@table, [:named_table, :public, write_concurrency: true])
+    {:ok, 0}
+  end
+
+  def handle_call({:break, module, fa, breaks}, _from, counter) do
+    # If there is a match for the given module and fa, and
+    # the module is still instrumented, we just increment
+    # the breaks counter.
+    #
+    # Otherwise we need to invoke the whole instrumentation
+    # tool chain.
+    case :ets.match_object(@table, {:_, module, fa, :_}) do
+      [{ref, module, fa, _}] ->
+        if instrumented?(module) do
+          :ets.insert(@table, {ref, module, fa, breaks})
+          {:reply, {:ok, ref}, counter}
+        else
+          :ets.delete(@table, ref)
+          instrument_and_reply(module, fa, breaks, counter)
+        end
+      [] ->
+        instrument_and_reply(module, fa, breaks, counter)
+    end
+  end
+
+  def handle_call({:reset_break, pattern}, _from, counter) do
+    reset =
+      for {ref, module, fa, _} <- :ets.match_object(@table, pattern) do
+        if instrumented?(module) do
+          :ets.insert(@table, {ref, module, fa, 0})
+          true
+        else
+          :ets.delete(@table, ref)
+          false
+        end
+      end
+
+    if Enum.any?(reset) do
+      {:reply, :ok, counter}
+    else
+      {:reply, :not_found, counter}
+    end
+  end
+
+  def handle_call(:breaks, _from, counter) do
+    entries =
+      for {id, module, function_arity, breaks} <- :ets.tab2list(@table),
+          keep_instrumented(id, module) == :ok do
+        {id, module, function_arity, max(breaks, 0)}
+      end
+    {:reply, entries, counter}
+  end
+
+  def handle_call(:remove_breaks, _from, _counter) do
+    # Make sure to deinstrument before clearing
+    # up the table to avoid race conditions.
+    @table
+    |> :ets.match({:_, :"$1", :_, :_})
+    |> List.flatten
+    |> Enum.uniq
+    |> Enum.each(&deinstrument_if_instrumented/1)
+    true = :ets.delete_all_objects(@table)
+    {:reply, :ok, 0}
+  end
+
+  def handle_call({:remove_breaks, module}, _from, counter) do
+    # Make sure to deinstrumented before clearing
+    # up the table to avoid race conditions.
+    reply = deinstrument_if_instrumented(module)
+    true = :ets.match_delete(@table, {:_, module, :_, :_})
+    {:reply, reply, counter}
+  end
+
+  defp keep_instrumented(id, module) do
+    if instrumented?(module) do
+      :ok
+    else
+      :ets.delete(@table, id)
+      :error
+    end
+  end
+
+  defp deinstrument_if_instrumented(module) do
+    if instrumented?(module) do
+      deinstrument(module)
+    else
+      :ok
+    end
+  end
+
+  defp deinstrument(module) do
+    with beam when is_list(beam) <- :code.which(module),
+         {:ok, binary} = File.read(beam) do
+      :code.purge(module)
+      {:module, _} = :code.load_binary(module, beam, binary)
+      :ok
+    else
+      _ -> {:error, :no_beam_file}
+    end
+  end
+
+  defp instrument_and_reply(module, fa, breaks, counter) do
+    case fetch_elixir_debug_info_with_fa_check(module, fa) do
+      {:ok, beam, backend, elixir} ->
+        counter = counter + 1
+        true = :ets.insert_new(@table, {counter, module, fa, breaks})
+        entries = :ets.match_object(@table, {:_, module, :_, :_})
+        {:reply, instrument(beam, backend, elixir, counter, entries), counter}
+      {:error, _} = error ->
+        {:reply, error, counter}
+    end
+  end
+
+  defp fetch_elixir_debug_info_with_fa_check(module, fa) do
+    case :code.which(module) do
+      beam when is_list(beam) ->
+        case :beam_lib.chunks(beam, [:debug_info]) do
+          {:ok, {_, [debug_info: {:debug_info_v1, backend, {:elixir_v1, map, _} = elixir}]}} ->
+            case List.keyfind(map.definitions, fa, 0) do
+              {_, _, _, _} -> {:ok, beam, backend, elixir}
+              nil -> {:error, :unknown_function_arity}
+            end
+          {:ok, {_, [debug_info: {:debug_info_v1, _, _}]}} ->
+            {:error, :non_elixir_module}
+          {:error, :beam_lib, {:unknown_chunk, _, _}} ->
+            {:error, :otp_20_is_required} # TODO: Remove this when we require OTP 20+
+          {:error, :beam_lib, {:missing_chunk, _, _}} ->
+            {:error, :missing_debug_info}
+          _ ->
+            {:error, :outdated_debug_info}
+        end
+      _ ->
+        {:error, :no_beam_file}
+    end
+  end
+
+  defp instrument(beam, backend, {:elixir_v1, map, specs}, counter, entries) do
+    %{attributes: attributes, definitions: definitions, module: module} = map
+    map = %{map | attributes: [{:iex_pry, true} | attributes],
+                  definitions: Enum.map(definitions, &instrument_definition(&1, map, entries))}
+
+    with {:ok, forms} <- backend.debug_info(:erlang_v1, module, {:elixir_v1, map, specs}, []),
+         {:ok, _, binary, _} <- :compile.noenv_forms(forms, [:return | map.compile_opts]) do
+      :code.purge(module)
+      {:module, _} = :code.load_binary(module, beam, binary)
+      {:ok, counter}
+    else
+      _error ->
+        {:error, :recompilation_failed}
+    end
+  end
+
+  defp instrument_definition({fa, kind, meta, clauses} = definition, map, entries) do
+    case List.keyfind(entries, fa, 2) do
+      {ref, _, ^fa, _} ->
+        %{module: module, file: file} = map
+        file =
+          case meta[:location] do
+            {file, _} -> file
+            _ -> file
+          end
+        opts = [module: module, file: file, function: fa]
+        clauses = Enum.map(clauses, &instrument_clause(&1, ref, opts))
+        {fa, kind, meta, clauses}
+      nil ->
+        definition
+    end
+  end
+
+  defp instrument_clause({meta, args, guards, clause}, ref, opts) do
+    opts = [line: Keyword.get(meta, :line, 1)] ++ opts
+
+    # We store variables on a map ignoring the context.
+    # In the rare case where variables in different contexts
+    # have the same name, the last one wins.
+    {_, binding} =
+      Macro.prewalk(args, %{}, fn
+        {name, _, ctx} = var, acc when name != :_ and is_atom(name) and is_atom(ctx) ->
+          {var, Map.put(acc, name, var)}
+        expr, acc ->
+          {expr, acc}
+      end)
+
+    # Generate the take_over condition with the ets lookup.
+    # Remember this is expanded AST, so no aliases allowed,
+    # no locals (such as the unary -) and so on.
+    condition =
+      quote do
+        # :ets.update_counter(table, key, {pos, inc, threshold, reset})
+        case :ets.update_counter(unquote(@table), unquote(ref), {4, unquote(-1), unquote(-1), unquote(-1)}) do
+          unquote(-1) -> :ok
+          _ -> :"Elixir.IEx.Pry".pry(unquote(Map.to_list(binding)), unquote(opts))
+        end
+      end
+
+    {meta, args, guards, {:__block__, [], [condition, clause]}}
+  end
+
+  defp instrumented?(module) do
+    module.module_info(:attributes)[:iex_pry] == [true]
+  end
+end

lib/iex/lib/iex/server.ex

@@ -1,5 +1,7 @@
 defmodule IEx.State do
   @moduledoc false
+  # This state is exchanged between IEx.Server and
+  # IEx.Evaluator which is why it is a struct.
   defstruct cache: '', counter: 1, prefix: "iex"
   @type t :: %__MODULE__{}
 end
@@ -45,13 +47,11 @@ defmodule IEx.Server do
   @doc """
   Returns the PID of the IEx evaluator process if it exists.
   """
-  @spec evaluator :: pid | nil
+  @spec evaluator :: {evaluator :: pid, server :: pid} | nil
   def evaluator() do
-    case IEx.Server.local do
-      nil -> nil
-      pid ->
-        {:dictionary, dictionary} = Process.info(pid, :dictionary)
-        dictionary[:evaluator]
+    if pid = IEx.Server.local do
+      {:dictionary, dictionary} = Process.info(pid, :dictionary)
+      {dictionary[:evaluator], pid}
     end
   end
 
@@ -59,31 +59,22 @@ defmodule IEx.Server do
   Requests to take over the given shell from the
   current process.
   """
-  @spec take_over(binary, keyword, pos_integer) ::
+  @spec take_over(binary, keyword) ::
         :ok | {:error, :no_iex} | {:error, :refused}
-  def take_over(identifier, opts, timeout \\ 1000, server \\ whereis()) do
-    cond do
-      is_nil(server) ->
-        {:error, :no_iex}
-      true ->
-        ref = make_ref()
-        send server, {:take?, self(), ref}
+  def take_over(identifier, opts, server \\ whereis()) do
+    if is_nil(server) do
+      {:error, :no_iex}
+    else
+      ref = make_ref()
+      opts = [evaluator: self()] ++ opts
+      send server, {:take, self(), identifier, ref, opts}
 
-        receive do
-          ^ref ->
-            opts = [evaluator: self()] ++ opts
-            send server, {:take, self(), identifier, ref, opts}
-
-            receive do
-              {^ref, nil} ->
-                {:error, :refused}
-              {^ref, leader} ->
-                IEx.Evaluator.init(:no_ack, server, leader, opts)
-            end
-        after
-          timeout ->
-            {:error, :no_iex}
-        end
+      receive do
+        {^ref, nil} ->
+          {:error, :refused}
+        {^ref, leader} ->
+          IEx.Evaluator.init(:no_ack, server, leader, opts)
+      end
     end
   end
 
@@ -110,16 +101,12 @@ defmodule IEx.Server do
 
   defp start_loop(opts, pid, ref) do
     receive do
-      {:take?, other, ref} ->
-        send(other, ref)
-        start_loop(opts, pid, ref)
-
-      {:take, other, identifier, ref, opts} ->
-        if allow_take?(identifier) do
-          send(other, {ref, Process.group_leader})
-          run(opts)
+      {:take, take_pid, take_identifier, take_ref, take_opts} ->
+        if allow_take?(take_identifier) do
+          send(take_pid, {take_ref, Process.group_leader})
+          run(take_opts)
         else
-          send(other, {ref, nil})
+          send(take_pid, {take_ref, nil})
           start_loop(opts, pid, ref)
         end
 
@@ -137,7 +124,14 @@ defmodule IEx.Server do
   defp run(opts) when is_list(opts) do
     IO.puts "Interactive Elixir (#{System.version}) - press Ctrl+C to exit (type h() ENTER for help)"
     evaluator = start_evaluator(opts)
-    loop(run_state(opts), evaluator, Process.monitor(evaluator))
+    loop(iex_state(opts), evaluator, Process.monitor(evaluator))
+  end
+
+  defp rerun(opts, evaluator, evaluator_ref, input) do
+    kill_input(input)
+    IO.puts("")
+    stop_evaluator(evaluator, evaluator_ref)
+    run(opts)
   end
 
   @doc """
@@ -145,29 +139,16 @@ defmodule IEx.Server do
   """
   @spec start_evaluator(keyword) :: pid
   def start_evaluator(opts) do
-    self_pid = self()
-    self_leader = Process.group_leader
     evaluator = opts[:evaluator] ||
-                :proc_lib.start(IEx.Evaluator, :init, [:ack, self_pid, self_leader, opts])
+                :proc_lib.start(IEx.Evaluator, :init, [:ack, self(), Process.group_leader, opts])
     Process.put(:evaluator, evaluator)
     evaluator
   end
 
-  defp reset_loop(opts, evaluator, evaluator_ref) do
-    IO.puts("")
-    # We only kill the evaluator if the new evaluator
-    # is not the same as the current one. Otherwise
-    # we end up killing the process that is taking over.
-    exit_loop(evaluator, evaluator_ref, opts[:evaluator] != evaluator)
-    run(opts)
-  end
-
-  defp exit_loop(evaluator, evaluator_ref, done? \\ true) do
+  defp stop_evaluator(evaluator, evaluator_ref) do
     Process.delete(:evaluator)
     Process.demonitor(evaluator_ref, [:flush])
-    if done? do
-      send(evaluator, {:done, self()})
-    end
+    send(evaluator, {:done, self()})
     :ok
   end
 
@@ -189,11 +170,11 @@ defmodule IEx.Server do
         io_error "** (EXIT) interrupted"
         loop(%{state | cache: ''}, evaluator, evaluator_ref)
       {:input, ^input, :eof} ->
-        exit_loop(evaluator, evaluator_ref)
+        stop_evaluator(evaluator, evaluator_ref)
       {:input, ^input, {:error, :terminated}} ->
-        exit_loop(evaluator, evaluator_ref)
+        stop_evaluator(evaluator, evaluator_ref)
       msg ->
-        handle_take_over(msg, evaluator, evaluator_ref, input, fn ->
+        handle_take_over(msg, state, evaluator, evaluator_ref, input, fn state ->
           wait_input(state, evaluator, evaluator_ref, input)
         end)
     end
@@ -203,65 +184,75 @@ defmodule IEx.Server do
     receive do
       {:evaled, ^evaluator, new_state} ->
         loop(new_state, evaluator, evaluator_ref)
-      {:EXIT, _pid, :interrupt} ->
-        # User did ^G while the evaluator was busy or stuck
-        io_error "** (EXIT) interrupted"
-        Process.delete(:evaluator)
-        Process.exit(evaluator, :kill)
-        Process.demonitor(evaluator_ref, [:flush])
-        evaluator = start_evaluator([])
-        loop(%{state | cache: ''}, evaluator, Process.monitor(evaluator))
       msg ->
-        handle_take_over(msg, evaluator, evaluator_ref, nil,
-                         fn -> wait_eval(state, evaluator, evaluator_ref) end)
+        handle_take_over(msg, state, evaluator, evaluator_ref, nil,
+                         fn state -> wait_eval(state, evaluator, evaluator_ref) end)
+    end
+  end
+
+  defp wait_take_over(state, evaluator, evaluator_ref) do
+    receive do
+      msg ->
+        handle_take_over(msg, state, evaluator, evaluator_ref, nil,
+                         fn state -> wait_take_over(state, evaluator, evaluator_ref) end)
     end
   end
 
   # Take process.
   #
-  # The take? message is received out of band, so we can
-  # go back to wait for the same input. The take message
-  # needs to take hold of the IO, so it kills the input,
-  # re-runs the server OR goes back to the main loop.
-  #
   # A take process may also happen if the evaluator dies,
   # then a new evaluator is created to replace the dead one.
-  defp handle_take_over({:take?, other, ref}, _evaluator, _evaluator_ref, _input, callback) do
-    send(other, ref)
-    callback.()
-  end
-
-  defp handle_take_over({:take, other, identifier, ref, opts}, evaluator, evaluator_ref, input, callback) do
-    kill_input(input)
-
-    if allow_take?(identifier) do
-      send other, {ref, Process.group_leader}
-      reset_loop(opts, evaluator, evaluator_ref)
-    else
-      send other, {ref, nil}
-      callback.()
+  defp handle_take_over({:take, other, identifier, ref, opts},
+                        state, evaluator, evaluator_ref, input, callback) do
+    cond do
+      evaluator == opts[:evaluator]  ->
+        send other, {ref, Process.group_leader}
+        kill_input(input)
+        loop(iex_state(opts), evaluator, evaluator_ref)
+      allow_take?(identifier) ->
+        send other, {ref, Process.group_leader}
+        rerun(opts, evaluator, evaluator_ref, input)
+      true ->
+        send other, {ref, nil}
+        callback.(state)
     end
   end
 
-  defp handle_take_over({:respawn, evaluator}, evaluator, evaluator_ref, input, _callback) do
+  # User did ^G while the evaluator was busy or stuck
+  defp handle_take_over({:EXIT, _pid, :interrupt}, state, evaluator, evaluator_ref, input, _callback) do
     kill_input(input)
-    reset_loop([], evaluator, evaluator_ref)
+    io_error "** (EXIT) interrupted"
+    Process.delete(:evaluator)
+    Process.exit(evaluator, :kill)
+    Process.demonitor(evaluator_ref, [:flush])
+    evaluator = start_evaluator([])
+    loop(%{state | cache: ''}, evaluator, Process.monitor(evaluator))
+  end
+
+  defp handle_take_over({:respawn, evaluator}, _state, evaluator, evaluator_ref, input, _callback) do
+    rerun([], evaluator, evaluator_ref, input)
+  end
+
+  defp handle_take_over({:continue, evaluator}, state, evaluator, evaluator_ref, input, _callback) do
+    kill_input(input)
+    send(evaluator, {:done, self()})
+    wait_take_over(state, evaluator, evaluator_ref)
   end
 
   defp handle_take_over({:DOWN, evaluator_ref, :process, evaluator, reason},
-                        evaluator, evaluator_ref, input, _callback) do
+                        _state, evaluator, evaluator_ref, input, _callback) do
     try do
-      io_error Exception.format_banner({:EXIT, evaluator}, reason)
+      io_error "** (EXIT from #{inspect evaluator}) evaluator process exited with reason: " <>
+               Exception.format_exit(reason)
     catch
       type, detail ->
         io_error "** (IEx.Error) #{type} when printing EXIT message: #{inspect detail}"
     end
-    kill_input(input)
-    reset_loop([], evaluator, evaluator_ref)
+    rerun([], evaluator, evaluator_ref, input)
   end
 
-  defp handle_take_over(_, _evaluator, _evaluator_ref, _input, callback) do
-    callback.()
+  defp handle_take_over(_, state, _evaluator, _evaluator_ref, _input, callback) do
+    callback.(state)
   end
 
   defp kill_input(nil),   do: :ok
@@ -278,9 +269,8 @@ defmodule IEx.Server do
 
   ## State
 
-  defp run_state(opts) do
+  defp iex_state(opts) do
     prefix = Keyword.get(opts, :prefix, "iex")
-
     %IEx.State{prefix: prefix}
   end
 

lib/iex/test/iex/autocomplete_test.exs

@@ -11,13 +11,13 @@ defmodule IEx.AutocompleteTest do
 
   defmodule MyServer do
     def evaluator do
-      Process.get(:evaluator)
+      {Process.get(:evaluator), self()}
     end
   end
 
   defp eval(line) do
     ExUnit.CaptureIO.capture_io(fn ->
-      evaluator = MyServer.evaluator
+      {evaluator, _} = MyServer.evaluator
       Process.group_leader(evaluator, Process.group_leader)
       send evaluator, {:eval, self(), line <> "\n", %IEx.State{}}
       assert_receive {:evaled, _, _}

lib/iex/test/iex/interaction_test.exs

@@ -3,8 +3,6 @@ Code.require_file "../test_helper.exs", __DIR__
 defmodule IEx.InteractionTest do
   use IEx.Case
 
-  ## Basic interaction
-
   test "whole output" do
     assert capture_io("IO.puts \"Hello world\"", fn ->
       IEx.Server.start([dot_iex_path: ""], {IEx, :dont_display_result, []})
@@ -20,6 +18,10 @@ defmodule IEx.InteractionTest do
     assert capture_iex("1 + 2") == "3"
   end
 
+  test "invalid input" do
+    assert capture_iex("if true do ) false end") =~ "** (SyntaxError) iex:1: \"do\" is missing terminator \"end\". unexpected token: \")\" at line 1"
+  end
+
   test "multiple vars" do
     code = """
     << a, b, c, d, e, f, g, h, i, j :: binary >> = <<1, 2, 3, 4, 5, 6, 7, 8, 9, 10>>
@@ -37,22 +39,6 @@ defmodule IEx.InteractionTest do
     assert capture_iex(code) =~ "3"
   end
 
-  test "exception" do
-    exception = Regex.escape("** (ArithmeticError) bad argument in arithmetic expression")
-    assert capture_iex("1 + :atom\n:this_is_still_working")
-           =~ ~r/^#{exception}.+\n:this_is_still_working$/s
-    refute capture_iex("1 + :atom\n:this_is_still_working")
-           =~ ~r/erl_eval/s
-  end
-
-  test "empty history at the start" do
-    assert capture_iex("v(-1)") =~ "** (RuntimeError) v(-1) is out of bounds"
-  end
-
-  test "empty history at the start redux" do
-    assert capture_iex("v(1)") =~ "** (RuntimeError) v(1) is out of bounds"
-  end
-
   test "no break" do
     input = """
       ["a
@@ -72,10 +58,6 @@ defmodule IEx.InteractionTest do
     assert capture_iex(input) =~ "** (TokenMissingError) iex:1: incomplete expression"
   end
 
-  test "invalid input" do
-    assert capture_iex("if true do ) false end") =~ "** (SyntaxError) iex:1: \"do\" is missing terminator \"end\". unexpected token: \")\" at line 1"
-  end
-
   test "module definition" do
     input = """
     defmodule Sample do
@@ -112,19 +94,25 @@ defmodule IEx.InteractionTest do
            "<<45, 46, 47>>\n[45, 46, 47]\n%{__struct__: IO.Stream, device: nil, line_or_bytes: :line, raw: true}"
   end
 
-  test "history size" do
-    opts = [history_size: 3]
-    assert capture_iex("1\n2\n3\nv(1)", opts) == "1\n2\n3\n1"
-    assert "1\n2\n3\n4\n** (RuntimeError) v(1) is out of bounds" <> _ = capture_iex("1\n2\n3\n4\nv(1)", opts)
-    assert "1\n2\n3\n4\n** (RuntimeError) v(-4) is out of bounds" <> _ = capture_iex("1\n2\n3\n4\nv(-4)", opts)
-    assert "1\n2\n3\n4\n2\n** (RuntimeError) v(2) is out of bounds" <> _ = capture_iex("1\n2\n3\n4\nv(2)\nv(2)", opts)
+  test "exception" do
+    exception = Regex.escape("** (ArithmeticError) bad argument in arithmetic expression")
+    assert capture_iex("1 + :atom\n:this_is_still_working") =~
+           ~r/^#{exception}.+\n:this_is_still_working$/s
+    refute capture_iex("1 + :atom\n:this_is_still_working") =~
+           ~r/erl_eval/s
+  end
+
+  test "exception while invoking conflicting helpers" do
+    import File, only: [open: 1], warn: false
+    assert capture_iex("open('README.md')", [], [env: __ENV__]) =~
+           ~r"function open/1 imported from both File and IEx.Helpers"
   end
 
   test "receive exit" do
     assert capture_iex("spawn_link(fn -> exit(:bye) end); Process.sleep(1000)") =~
-           ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) :bye"
+           ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) evaluator process exited with reason: :bye"
     assert capture_iex("spawn_link(fn -> exit({:bye, [:world]}) end); Process.sleep(1000)") =~
-           ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) {:bye, \[:world\]}"
+           ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) evaluator process exited with reason: {:bye, \[:world\]}"
   end
 
   test "receive exit from exception" do
@@ -133,7 +121,7 @@ defmodule IEx.InteractionTest do
     content = capture_iex("spawn_link(fn -> exit({%ArgumentError{},
                            [{:not_a_real_module, :function, 0, []}]}) end);
                            Process.sleep(1000)")
-    assert content =~ ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) an exception was raised:\n"
+    assert content =~ ~r"\*\* \(EXIT from #PID<\d+\.\d+\.\d+>\) evaluator process exited with reason: an exception was raised:\n"
     assert content =~ ~r"\s{4}\*\* \(ArgumentError\) argument error\n"
     assert content =~ ~r"\s{8}:not_a_real_module\.function/0"
   end

lib/iex/test/iex/pry_test.exs

@@ -0,0 +1,201 @@
+Code.require_file "../test_helper.exs", __DIR__
+
+defmodule IEx.PryTest do
+  use ExUnit.Case
+
+  setup do
+    on_exit fn -> IEx.Pry.remove_breaks() end
+  end
+
+  describe "whereami" do
+    test "shows lines with radius" do
+      Application.put_env(:elixir, :ansi_enabled, true)
+
+      {:ok, contents} = IEx.Pry.whereami(__ENV__.file, 3, 2)
+      assert IO.iodata_to_binary(contents) == """
+          1: Code.require_file "../test_helper.exs", __DIR__
+          2:\s
+      \e[1m    3: defmodule IEx.PryTest do
+      \e[22m    4:   use ExUnit.Case
+          5:\s
+      """
+
+      {:ok, contents} = IEx.Pry.whereami(__ENV__.file, 1, 4)
+      assert IO.iodata_to_binary(contents) == """
+      \e[1m    1: Code.require_file "../test_helper.exs", __DIR__
+      \e[22m    2:\s
+          3: defmodule IEx.PryTest do
+          4:   use ExUnit.Case
+          5:\s
+      """
+    after
+      Application.delete_env(:elixir, :ansi_enabled)
+    end
+
+    test "returns error for unknown files" do
+      assert IEx.Pry.whereami("unknown", 3, 2) == :error
+    end
+
+    test "returns error for out of range lines" do
+      assert IEx.Pry.whereami(__ENV__.file, 1000, 2) == :error
+    end
+  end
+
+  if :erlang.system_info(:otp_release) >= '20' do
+    describe "break" do
+      test "sets up a breakpoint on the given module" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert instrumented?(URI)
+        assert [_] = IEx.Pry.breaks()
+      end
+
+      test "sets up multiple breakpoints in the same module" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert instrumented?(URI)
+        assert IEx.Pry.break(URI, :parse, 1) == {:ok, 2}
+        assert instrumented?(URI)
+        assert [_, _] = IEx.Pry.breaks()
+      end
+
+      test "reinstruments if module has been reloaded" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert instrumented?(URI)
+        deinstrument!(URI)
+        refute instrumented?(URI)
+        assert IEx.Pry.break(URI, :parse, 1) == {:ok, 2}
+        assert instrumented?(URI)
+        assert [_, _] = IEx.Pry.breaks()
+      end
+
+      test "returns id when breakpoint is already set" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert [_] = IEx.Pry.breaks()
+      end
+
+      test "builds new id when breakpoint is already set on deinstrument" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        deinstrument!(URI)
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 2}
+        assert [_] = IEx.Pry.breaks()
+      end
+
+      test "errors when setting up a break with no beam" do
+        assert IEx.Pry.break(__MODULE__, :setup, 2) == {:error, :no_beam_file}
+      end
+
+      test "errors when setting up a break for unknown function" do
+        assert IEx.Pry.break(URI, :unknown, 2) == {:error, :unknown_function_arity}
+      end
+
+      test "errors for non elixir modules" do
+        assert IEx.Pry.break(:elixir, :unknown, 2) == {:error, :non_elixir_module}
+      end
+    end
+
+    describe "breaks" do
+      test "returns all breaks" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.breaks() ==
+               [{1, URI, {:decode_query, 2}, 1}]
+
+        assert IEx.Pry.break(URI, :decode_query, 2, 10) == {:ok, 1}
+        assert IEx.Pry.breaks() ==
+               [{1, URI, {:decode_query, 2}, 10}]
+
+        assert IEx.Pry.break(URI, :parse, 1, 1) == {:ok, 2}
+        assert IEx.Pry.breaks() ==
+               [{1, URI, {:decode_query, 2}, 10}, {2, URI, {:parse, 1}, 1}]
+      end
+
+      test "sets negative break to 0" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        :ets.insert(IEx.Pry, {1, URI, {:decode_query, 2}, -1})
+        assert IEx.Pry.breaks() == [{1, URI, {:decode_query, 2}, 0}]
+      end
+
+      test "do not return break points for deinstrumented modules" do
+        assert IEx.Pry.break(URI, :parse, 1) == {:ok, 1}
+        assert IEx.Pry.breaks() == [{1, URI, {:parse, 1}, 1}]
+        deinstrument!(URI)
+        assert IEx.Pry.breaks() == []
+      end
+    end
+
+    describe "reset_break" do
+      test "resets break for given id" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.reset_break(1) == :ok
+        assert IEx.Pry.breaks() == [{1, URI, {:decode_query, 2}, 0}]
+      end
+
+      test "resets break for given mfa" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.reset_break(URI, :decode_query, 2) == :ok
+        assert IEx.Pry.breaks() == [{1, URI, {:decode_query, 2}, 0}]
+      end
+
+      test "returns not_found if module is deinstrumented" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        deinstrument!(URI)
+        assert IEx.Pry.reset_break(URI, :decode_query, 2) == :not_found
+        assert IEx.Pry.breaks() == []
+      end
+
+      test "returns not_found if mfa has no break" do
+        assert IEx.Pry.reset_break(URI, :decode_query, 2) == :not_found
+      end
+
+      test "returns not_found if id is deinstrumented" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        deinstrument!(URI)
+        assert IEx.Pry.reset_break(1) == :not_found
+        assert IEx.Pry.breaks() == []
+      end
+
+      test "returns not_found if id has no break" do
+        assert IEx.Pry.reset_break(1) == :not_found
+      end
+    end
+
+    describe "remove_breaks" do
+      test "removes all breaks" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.remove_breaks() == :ok
+        assert IEx.Pry.breaks() == []
+      end
+
+      test "removes all breaks even if module is deinstrumented" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        deinstrument!(URI)
+        assert IEx.Pry.remove_breaks() == :ok
+        assert IEx.Pry.breaks() == []
+      end
+
+      test "remove breaks in a given module" do
+        assert IEx.Pry.remove_breaks(Date.Range) == :ok
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        assert IEx.Pry.break(Date.Range, :__struct__, 1) == {:ok, 2}
+        assert IEx.Pry.remove_breaks(Date.Range) == :ok
+        assert IEx.Pry.breaks() == [{1, URI, {:decode_query, 2}, 1}]
+      end
+
+      test "remove breaks in a given module even if deinstrumented" do
+        assert IEx.Pry.break(URI, :decode_query, 2) == {:ok, 1}
+        deinstrument!(URI)
+        assert IEx.Pry.breaks() == []
+      end
+    end
+  end
+
+  defp instrumented?(module) do
+    module.module_info(:attributes)[:iex_pry] == [true]
+  end
+
+  defp deinstrument!(module) do
+    beam = :code.which(module)
+    :code.purge(module)
+    {:module, _} = :code.load_binary(module, beam, File.read!(beam))
+    :ok
+  end
+end

lib/iex/test/iex/server_test.exs

@@ -3,40 +3,50 @@ Code.require_file "../test_helper.exs", __DIR__
 defmodule IEx.ServerTest do
   use IEx.Case
 
-  # Options
+  describe "options" do
+    test "prefix" do
+      assert capture_io(fn ->
+        boot([prefix: "pry"])
+      end) =~ "pry(1)> "
+    end
 
-  test "prefix option" do
-    assert capture_io(fn ->
-      boot([prefix: "pry"])
-    end) =~ "pry(1)> "
+    test "env" do
+      assert capture_io("__ENV__.file", fn ->
+        boot([env: __ENV__])
+      end) =~ "server_test.exs"
+    end
   end
 
-  test "env option" do
-    assert capture_io("__ENV__.file", fn ->
-      boot([env: __ENV__])
-    end) =~ "server_test.exs"
-  end
+  describe "take_over" do
+    test "allows take over of the shell during boot" do
+      assert capture_io("Y\na+b", fn ->
+        server = self()
+        boot([], fn ->
+          opts = [prefix: "dbg", binding: [a: 1, b: 2]]
+          IEx.Server.take_over("iex:13", opts, server)
+        end)
+      end) =~ "dbg(1)> "
+    end
 
-  # Take over
+    test "continues if take over is refused" do
+      assert capture_io("N\n", fn ->
+        server = self()
+        boot([], fn ->
+          opts = [prefix: "dbg", binding: [a: 1, b: 2]]
+          IEx.Server.take_over("iex:13", opts, server)
+        end)
+      end) =~ "iex(1)> "
+    end
 
-  test "allows take over of the shell during boot" do
-    assert capture_io("Y\na+b", fn ->
-      server = self()
-      boot([], fn ->
-        opts = [prefix: "dbg", binding: [a: 1, b: 2]]
-        IEx.Server.take_over("iex:13", opts, 1000, server)
-      end)
-    end) =~ "dbg(1)> "
-  end
+    test "fails if callback during boot fails" do
+      assert capture_io(fn ->
+        boot([], fn -> exit(0) end)
+      end) == ""
+    end
 
-  test "does not operate if callback during boot fails" do
-    assert capture_io(fn ->
-      boot([], fn -> exit(0) end)
-    end) == ""
-  end
-
-  test "take over fails when there is no shell" do
-    assert IEx.Server.take_over("iex:13", [], 10) == {:error, :no_iex}
+    test "fails when there is no shell" do
+      assert IEx.Server.take_over("iex:13", []) == {:error, :no_iex}
+    end
   end
 
   test "pry wraps around take over" do

lib/iex/test/test_helper.exs

@@ -1,3 +1,5 @@
+System.put_env("ELIXIR_EDITOR", "echo")
+
 :ok = Application.start(:iex)
 IEx.configure([colors: [enabled: false]])
 ExUnit.start [trace: "--trace" in System.argv, assert_receive_timeout: 500]

lib/mix/lib/mix.ex

@@ -201,6 +201,15 @@ defmodule Mix do
 
   @doc """
   Returns the Mix environment.
+
+  This function should not be used at runtime in application code (as opposed
+  to infrastructure and build code like Mix tasks). Mix is a build tool and may
+  not be available after the code is compiled (for example in a release).
+
+  To differentiate the program behavior depending on the environment, it is
+  recommended to use application environment through `Application.get_env/3`.
+  Proper configuration can be set in `Mix.Config` files, often per-environment
+  (see `Mix.Config.import_config/1` for more information).
   """
   def env do
     # env is not available on bootstrapping, so set a :dev default
@@ -212,6 +221,9 @@ defmodule Mix do
 
   Be careful when invoking this function as any project
   configuration won't be reloaded.
+
+  This function should not be used at runtime in application code
+  (see `env/0` for more information).
   """
   def env(env) when is_atom(env) do
     Mix.State.put(:env, env)

lib/mix/lib/mix/compilers/erlang.ex

@@ -51,9 +51,9 @@ defmodule Mix.Compilers.Erlang do
     compile(manifest, files, src_ext, opts, callback)
   end
 
-  def compile(manifest, mappings, src_ext, dest_ext, force, callback) when is_boolean(force) do
-    IO.warn "Mix.Compilers.Erlang.compile/6 with a boolean as 5th argument is deprecated, " <>
-            "please pass [force: true]"
+  def compile(manifest, mappings, src_ext, dest_ext, force, callback) when is_boolean(force) or is_nil(force) do
+    IO.warn "Mix.Compilers.Erlang.compile/6 with a boolean or nil as 5th argument is deprecated, " <>
+            "please pass [force: true] or [] instead"
     compile(manifest, mappings, src_ext, dest_ext, [force: force], callback)
   end
 

lib/mix/lib/mix/tasks/compile.erlang.ex

@@ -147,6 +147,8 @@ defmodule Mix.Tasks.Compile.Erlang do
         end
       {:attribute, _, :behaviour, behaviour} ->
         %{erl | behaviours: [behaviour | erl.behaviours]}
+      {:attribute, _, :compile, value} when is_list(value) ->
+        %{erl | compile: value ++ erl.compile}
       {:attribute, _, :compile, value} ->
         %{erl | compile: [value | erl.compile]}
       _ ->

lib/mix/lib/mix/tasks/compile.protocols.ex

@@ -193,14 +193,20 @@ defmodule Mix.Tasks.Compile.Protocols do
           Map.put(protocols, protocol, true)
       end)
 
+    removed_metadata =
+      old_metadata -- new_metadata
+
+    removed_protocols =
+      for {protocol, :protocol, _beam} <- removed_metadata,
+          remove_consolidated(protocol, output),
+          do: {protocol, true},
+          into: %{}
+
     protocols =
-      Enum.reduce(old_metadata -- new_metadata, protocols, fn
-        {_, {:impl, protocol}, _beam}, protocols ->
-          Map.put(protocols, protocol, true)
-        {protocol, :protocol, _beam}, protocols ->
-          remove_consolidated(protocol, output)
-          protocols
-      end)
+      for {_, {:impl, protocol}, _beam} <- removed_metadata,
+          not Map.has_key?(removed_protocols, protocol),
+          do: {protocol, true},
+          into: protocols
 
     Map.keys(protocols)
   end

lib/mix/lib/mix/tasks/escript.build.ex

@@ -13,7 +13,16 @@ defmodule Mix.Tasks.Escript.Build do
   be installed, as Elixir is embedded as part of the escript.
 
   This task guarantees the project and its dependencies are
-  compiled and packages them inside an escript.
+  compiled and packages them inside an escript. Before invoking
+  `mix escript.build`, it is only necessary to define a `:escript`
+  key with a `:main_module` option in your `mix.exs` file:
+
+      escript: [main_module: MyApp.CLI]
+
+  By default, this task removes documentation and debugging
+  chunks from the compiled `.beam` files to reduce the size of
+  the escript. If this is not desired, check the `:strip_beams`
+  option.
 
   > Note: escripts do not support projects and dependencies
   > that need to store or read artifacts from the priv directory.
@@ -261,7 +270,7 @@ defmodule Mix.Tasks.Escript.Build do
 
   defp strip_beam(beam) when is_binary(beam) do
     {:ok, _, all_chunks} = :beam_lib.all_chunks(beam)
-    filtered_chunks = ['CInf', 'Abst', 'Dbgi']
+    filtered_chunks = ['Abst', 'CInf', 'Dbgi', 'ExDc']
     significant_chunks =
       for {name, _} = pair <- all_chunks, name not in filtered_chunks, do: pair
     {:ok, built_module} = :beam_lib.build_module(significant_chunks)

lib/mix/lib/mix/tasks/run.ex

@@ -83,9 +83,7 @@ defmodule Mix.Tasks.Run do
 
     # Start app after rewriting System.argv,
     # but before requiring and evaling.
-    if opts[:start] != false do
-      Mix.Task.run "app.start", args
-    end
+    Mix.Task.run "app.start", args
 
     process_load(opts, expr_evaluator)
 

lib/mix/test/mix/tasks/compile.protocols_test.exs

@@ -42,6 +42,41 @@ defmodule Mix.Tasks.Compile.ProtocolsTest do
     end
   end
 
+  test "compiles after converting a protocol into a standard module", context do
+    Mix.Project.push MixTest.Case.Sample
+
+    in_tmp context.test, fn ->
+      File.mkdir_p!("lib")
+      assert Mix.Task.run("compile")
+
+      # Define a local protocol
+      File.write!("lib/protocol.ex", """
+      defprotocol Compile.Protocol do
+        def foo(a)
+      end
+
+      defimpl Compile.Protocol, for: Integer do
+        def foo(a), do: a
+      end
+      """)
+      assert compile_elixir_and_protocols() == :ok
+      mark_as_old!("_build/dev/lib/sample/consolidated/Elixir.Compile.Protocol.beam")
+      File.rm!("lib/protocol.ex")
+
+      # Define a standard module
+      File.write!("lib/protocol.ex", """
+      defmodule Compile.Protocol do
+      end
+      """)
+      assert compile_elixir_and_protocols() == :noop
+
+      # Delete a local protocol
+      File.rm!("lib/protocol.ex")
+      assert compile_elixir_and_protocols() == :noop
+      refute File.regular?("_build/dev/lib/sample/consolidated/Elixir.Compile.Protocol.beam")
+    end
+  end
+
   test "compiles and consolidates deps protocols", context do
     Mix.Project.push MixTest.Case.Sample
 

lib/mix/test/mix/tasks/run_test.exs

@@ -37,6 +37,19 @@ defmodule Mix.Tasks.RunTest do
     purge [GitRepo]
   end
 
+  test "does not start applications on --no-start", context do
+    in_tmp context.test, fn ->
+      Mix.Tasks.Run.run ["--no-start", "-e", "send self(), {:apps, Application.started_applications}"]
+      assert_received {:apps, apps}
+      refute List.keyfind(apps, :sample, 0)
+      Mix.Task.clear
+
+      Mix.Tasks.Run.run ["-e", "send self(), {:apps, Application.started_applications}"]
+      assert_received {:apps, apps}
+      assert List.keyfind(apps, :sample, 0)
+    end
+  end
+
   test "run errors on missing files", context do
     in_tmp context.test, fn ->
       assert_raise Mix.Error, "No files matched pattern \"non-existent\" given to --require", fn ->

src/elixir.app.src

@@ -1,6 +1,6 @@
 {application, elixir,
 [{description, "elixir"},
- {vsn, "1.5.0-dev"},
+ {vsn, "1.5.0"},
  {modules, [
 	elixir
   ]},