Release v1.3.4

Prior to this change, calling a protocol function with an opaque type
would yield a warning, as Dialyzer concludes that the impl_for/1
function can't handle opaque arguments, since all clauses would
destructure their arguments in some way.

By adding a catch-all clause that does not destructure its argument,
Dialyzer no longer draws this conclusion, and the warnings go away.

As noted in the protocol.ex comment, this is technically a hack as it
relies on Dialyzer not being smart enough. However, I would not expect
it to break soon, if ever.

Signed-off-by: José Valim <jose.valim@plataformatec.com.br>

CHANGELOG.md

@@ -102,11 +102,11 @@ In any case, the previous behaviour can be brought back with the `--verbose` fla
 
 ### mix xref
 
-Speaking about warnings, Mix v1.3 includes a new task called `xref` that performs cross reference checks in your code. One of such checks is the ability to find calls to modules and functions that do not exist. For example, if in your library code you call `ThisModuleDoesNotExist.foo(1, 2, 3)`, `mix xref --unreachable` will be able to find such code and let you know about it.
+Speaking about warnings, Mix v1.3 includes a new task called `xref` that performs cross reference checks in your code. One of such checks is the ability to find calls to modules and functions that do not exist. For example, if in your library code you call `ThisModuleDoesNotExist.foo(1, 2, 3)`, `mix xref unreachable` will be able to find such code and let you know about it.
 
 Since such checks can discover possible bugs in your codebase, a new compiler called `xref` has been added to `Mix.compilers/0`, so they run by default every time you compile your code.
 
-We have included other modes in `xref`, such as `mix xref --callers Foo`, to find all places in your code that a function from the module `Foo` is called. We hope other tools and text editors can leverage such features to provide useful functionality for their users.
+We have included other modes in `xref`, such as `mix xref callers Foo`, to find all places in your code that a function from the module `Foo` is called. We hope other tools and text editors can leverage such features to provide useful functionality for their users.
 
 ### Better dependency tracking
 
@@ -268,7 +268,121 @@ end
 
 By restricting hierarchies in favor of named setups, it is straight-forward for the developer to glance at each describe block and know exactly the setup steps involved.
 
-## v1.3.0-rc.1 (2016-08-06)
+## v1.3.4 (2016-10-09)
+
+### 1. Bug fixes
+
+#### Elixir
+
+  * [Kernel] Ensure the compiler does not generate unecessary variable bindings inside case statements. This improves the code emitted and make sure "unused variable warnings" are not mistakenly silenced
+  * [Kernel] Move `raise` checks to runtime to avoid crashing cover on Erlang 19.1
+  * [Protocol] Do not emit warnings when using protocols on opaque types
+
+#### ExUnit
+
+  * [ExUnit.CaptureLog] Flush Erlang's `:error_logger` before capturing to avoid mixed messages
+
+## v1.3.3 (2016-09-17)
+
+### 1. Enhancements
+
+#### Elixir
+
+  * [DateTime] Support negative integer in `DateTime.from_unix/2`
+  * [Kernel.LexicalTracker] Do not consider remote typespecs as a compile-time dependency
+  * [Kernel.ParallelCompiler] Do not emit deadlock messages when the process is waiting on itself
+  * [Kernel.Typespec] Mark struct update syntax as generated to avoid false positives from dialyzer
+
+#### ExUnit
+
+  * [ExUnit] Make ExUnit server timeout configurable
+
+#### Logger
+
+  * [Logger] Use `:ansi_color` if one is available in metadata
+
+#### Mix
+
+  * [Mix] Add support for the `:sparse` option in `Mix.SCM.Git`
+  * [Mix] Skip dependendency loading if `MIX_NO_DEPS` is set to `1`
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [System] Use `NUL` instead of `/dev/null` on Windows when building `System.build_info`
+
+#### IEx
+
+  * [IEx.Autocomplete] Resolves issue with autocompletion on structs not working
+
+#### Mix
+
+  * [Mix] Also store external resources that are not part of the current working directory in compilation manifest
+  * [Mix] Always include the compiled file source in manifests
+
+## v1.3.2 (2016-07-15)
+
+### 1. Enhancements
+
+#### Elixir
+
+  * [Kernel] Support guards in `else` clauses in `with`
+
+#### Mix
+
+  * [Mix] Add `MIX_NO_DEPS` env var for disabling dep loading. Used for third-party scripts and tools like Nix package manager
+  * [Mix] Add `mix test --listen-on-stdin` that automatically reruns tests on stdin
+  * [Mix] Disable `--warnings-as-errors` when compiling dependencies
+  * [Mix] Add `--filter` option to `mix deps.unlock` to unlock only matching dependencies
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [Enum] Return `nil` if enumerable halts in `Enum.find_index/3`
+  * [Kernel] Do not attempt to load modules that have not been required when invoking regular functions, otherwise this invalidates the `@compile {:autoload, false}` directive.
+
+#### Mix
+
+  * [Mix] Ensure missing protocol dependencies are discarded in umbrella projects with shared build
+
+#### ExUnit
+
+  * [ExUnit.Diff] Ensure no leading or trailing when diffing some maps
+
+## v1.3.1 (2016-06-28)
+
+### 1. Enhancements
+
+#### IEx
+
+  * [IEx.Helpers] `Add import_file_if_available` for importing files only if they are available
+  * [IEx.Helpers] `Add import_if_available` for importing modules only if they are available
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [Kernel] Ensure structs can be expanded in dynamic module names
+  * [Kernel] Ensure aliases warnings are not accidentally discarded when the same module is imported
+  * [Kernel.ParallelCompiler] Ensure two modules with cyclic struct dependencies cannot run into a deadlock when compiling
+  * [Kernel.Typespec] Support module attributes in remote types
+  * [Module] Do not expect stacktraces to be always present when dispatching to locals during the module compilation
+
+#### IEx
+
+  * [IEx.Helpers] Fix `h` helper for operators
+
+#### Mix
+
+  * [Mix] Do not load modules for xref purposes, instead use BEAM info
+  * [Mix] Ensure `deps.check` does not check archives (that's done in loadpaths)
+  * [Mix] Validate application properties before traversing them
+  * [Mix] Check for proper Makefile when compiling on Windows
+  * [Mix] Enforce space after comma in `mix do`
+
+## v1.3.0 (2016-06-21)
 
 ### 1. Enhancements
 
@@ -283,6 +397,7 @@ By restricting hierarchies in favor of named setups, it is straight-forward for
   * [CLI] Add `--logger-otp-reports BOOL` and `--logger-sasl-reports BOOL` switches
   * [Compiler] Emit a summary of compilation errors when modules are missing
   * [Enum] Add `Enum.group_by/3` that allows developers to map on the value being grouped
+  * [Enum] Make list values in maps returned by `Enum.group_by/2` and `Enum.group_by/3` preserve the order of the input enumerable instead of reversing it.
   * [Enum] Add `Enum.drop_every/2` that drops every `nth`, including the first one
   * [Exception] Suggest possible functions on `UndefinedFunctionError` for existing modules
   * [Exception] Warn if unknown fields are given to `raise/2`

Makefile

@@ -1,7 +1,7 @@
 REBAR ?= "$(CURDIR)/rebar"
 PREFIX ?= /usr/local
-DOCS := master
-CANONICAL := master
+DOCS := v1.3
+CANONICAL := stable
 ELIXIRC := bin/elixirc --verbose --ignore-module-conflict
 ERLC := erlc -I lib/elixir/include
 ERL := erl -I lib/elixir/include -noshell -pa lib/elixir/ebin

RELEASE.md

@@ -28,7 +28,7 @@ This document simply outlines the release process:
 
 11. Create a new branch "vMAJOR.MINOR"
 
-12. Move docs generation to `docs/vMAJOR.MINOR` in Makefile, set CANONICAL reference and copy them to `docs/stable` (change index.html accordingly)
+12. Move docs generation to `docs/vMAJOR.MINOR` in Makefile, set CANONICAL to stable and copy them to `docs/stable` (change index.html accordingly)
 
 13. In master, bump versions, start new CHANGELOG, add `-dev` back and commit "Start vVERSION+1"
 

VERSION

@@ -1 +1 @@
-1.3.0-rc.1
+1.3.4

lib/elixir/lib/access.ex

@@ -435,6 +435,13 @@ defmodule Access do
       ...>   prev -> {prev, String.upcase(prev)}
       ...> end)
       {"john", [%{name: "JOHN"}, %{name: "mary"}]}
+
+  `at/1` can also be used to pop elements out of a list or
+  a key inside of a list:
+
+      iex> list = [%{name: "john"}, %{name: "mary"}]
+      iex> pop_in(list, [Access.at(0)])
+      {%{name: "john"}, [%{name: "mary"}]}
       iex> pop_in(list, [Access.at(0), :name])
       {"john", [%{}, %{name: "mary"}]}
 
@@ -477,7 +484,7 @@ defmodule Access do
   defp get_and_update_at([head | rest], 0, next, updates) do
     case next.(head) do
       {get, update} -> {get, :lists.reverse([update | updates], rest)}
-      :pop -> {head, :lists.reverse([head | updates], rest)}
+      :pop -> {head, :lists.reverse(updates, rest)}
     end
   end
 

lib/elixir/lib/calendar.ex

@@ -1004,19 +1004,32 @@ defmodule DateTime do
       {:ok, %DateTime{calendar: Calendar.ISO, day: 23, hour: 22, microsecond: {211914, 3}, minute: 53,
                       month: 1, second: 43, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
                       year: 46302, zone_abbr: "UTC"}}
+  
+  Negative Unix times are supported, up to -#{@unix_epoch} seconds, 
+  which is equivalent to "0000-01-01T00:00:00Z" or 0 gregorian seconds.
 
+      iex> DateTime.from_unix(-12345678910)
+      {:ok, %DateTime{calendar: Calendar.ISO, day: 13, hour: 4, microsecond: {0, 0}, minute: 44, 
+                      month: 10, second: 50, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0, 
+                      year: 1578, zone_abbr: "UTC"}}
+  
+  When a Unix time before that moment is passed to `from_unix/2`, `:error` will be returned.
   """
-  @spec from_unix(non_neg_integer, :native | System.time_unit) :: {:ok, DateTime.t}
-  def from_unix(integer, unit \\ :seconds) when is_integer(integer) and integer >= 0 do
+  @spec from_unix(integer, :native | System.time_unit) :: {:ok, DateTime.t}
+  def from_unix(integer, unit \\ :seconds) when is_integer(integer) do
     total = System.convert_time_unit(integer, unit, :microseconds)
-    microsecond = rem(total, 1_000_000)
-    precision = precision_for_unit(unit)
-    {{year, month, day}, {hour, minute, second}} =
-      :calendar.gregorian_seconds_to_datetime(@unix_epoch + div(total, 1_000_000))
+    if total < -@unix_epoch * 1_000_000 do
+      :error  
+    else
+      microsecond = rem(total, 1_000_000)
+      precision = precision_for_unit(unit)
+      {{year, month, day}, {hour, minute, second}} =
+        :calendar.gregorian_seconds_to_datetime(@unix_epoch + div(total, 1_000_000))
 
-    {:ok, %DateTime{year: year, month: month, day: day,
-                    hour: hour, minute: minute, second: second, microsecond: {microsecond, precision},
-                    std_offset: 0, utc_offset: 0, zone_abbr: "UTC", time_zone: "Etc/UTC"}}
+      {:ok, %DateTime{year: year, month: month, day: day,
+                      hour: hour, minute: minute, second: second, microsecond: {microsecond, precision},
+                      std_offset: 0, utc_offset: 0, zone_abbr: "UTC", time_zone: "Etc/UTC"}}
+    end
   end
 
   def precision_for_unit(unit) do
@@ -1034,6 +1047,13 @@ defmodule DateTime do
   @doc """
   Converts the given Unix time to DateTime.
 
+  The integer can be given in different unit
+  according to `System.convert_time_unit/3` and it will
+  be converted to microseconds internally.
+
+  Unix times are always in UTC and therefore the DateTime
+  will be returned in UTC.
+
   ## Examples
 
       iex> DateTime.from_unix!(1464096368)
@@ -1046,6 +1066,15 @@ defmodule DateTime do
                 month: 5, second: 8, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0,
                 year: 2015, zone_abbr: "UTC"}
 
+  Negative Unix times are supported, up to -#{@unix_epoch} seconds, 
+  which is equivalent to "0000-01-01T00:00:00Z" or 0 gregorian seconds.
+
+      iex> DateTime.from_unix(-12345678910)
+      {:ok, %DateTime{calendar: Calendar.ISO, day: 13, hour: 4, microsecond: {0, 0}, minute: 44, 
+                      month: 10, second: 50, std_offset: 0, time_zone: "Etc/UTC", utc_offset: 0, 
+                      year: 1578, zone_abbr: "UTC"}}
+
+  When a Unix time before that moment is passed to `from_unix!/2`, an ArgumentError will be raised.
   """
   @spec from_unix!(non_neg_integer, :native | System.time_unit) :: DateTime.t
   def from_unix!(integer, unit \\ :seconds) when is_atom(unit) do
@@ -1057,7 +1086,7 @@ defmodule DateTime do
   Converts the given DateTime to Unix time.
 
   The DateTime is expected to be using the ISO calendar
-  with a year greater than or equal to 1970.
+  with a year greater than or equal to 0.
 
   It will return the integer with the given unit,
   according to `System.convert_time_unit/3`.
@@ -1073,11 +1102,19 @@ defmodule DateTime do
       iex> DateTime.to_unix(dt)
       1416517099
 
+      iex> flamel = %DateTime{calendar: Calendar.ISO, day: 22, hour: 8, microsecond: {527771, 6},
+      ...>                minute: 2, month: 3, second: 25, std_offset: 0, time_zone: "Etc/UTC",
+      ...>                utc_offset: 0, year: 1418, zone_abbr: "UTC"}
+      iex> DateTime.to_unix(flamel)
+      -17412508655
+
   """
   @spec to_unix(DateTime.t, System.time_unit) :: non_neg_integer
+  def to_unix(datetime, unit \\ :seconds)
+  
   def to_unix(%DateTime{calendar: Calendar.ISO, std_offset: std_offset, utc_offset: utc_offset,
                         hour: hour, minute: minute, second: second, microsecond: {microsecond, _},
-                        year: year, month: month, day: day}, unit \\ :seconds) when year >= 1970 do
+                        year: year, month: month, day: day}, unit) when year >= 0 do
     seconds =
       :calendar.datetime_to_gregorian_seconds({{year, month, day}, {hour, minute, second}})
       |> Kernel.-(utc_offset)

lib/elixir/lib/enum.ex

@@ -861,13 +861,13 @@ defmodule Enum do
 
   def find_index(enumerable, fun) do
     res =
-      Enumerable.reduce(enumerable, {:cont, 0}, fn(entry, acc) ->
-        if fun.(entry), do: {:halt, acc}, else: {:cont, acc + 1}
+      Enumerable.reduce(enumerable, {:cont, {:not_found, 0}}, fn(entry, {_status, index}) ->
+        if fun.(entry), do: {:halt, {:found, index}}, else: {:cont, {:not_found, index + 1}}
       end)
 
     case res do
-      {:halted, entry} -> entry
-      {:done, _} -> nil
+      {_, {:found, index}} -> index
+      {_, {:not_found, _}} -> nil
     end
   end
 
@@ -997,7 +997,8 @@ defmodule Enum do
 
   # TODO: Remove on 2.0
   def group_by(enumerable, dict, fun) when is_function(fun, 1) do
-    IO.warn "Enum.group_by/3 with a map/dictionary as second element is deprecated, please use a map instead"
+    IO.warn "Enum.group_by/3 with a map/dictionary as second element is deprecated. " <>
+      "A map is used by default and it is no longer required to pass one to this function"
     reduce(reverse(enumerable), dict, fn(entry, categories) ->
       Dict.update(categories, fun.(entry), [entry], &[entry | &1])
     end)

lib/elixir/lib/exception.ex

@@ -15,7 +15,11 @@ defmodule Exception do
   """
 
   @typedoc "The exception type"
-  @type t :: %{__struct__: module, __exception__: true}
+  @type t :: %{
+    required(:__struct__) => module,
+    required(:__exception__) => true,
+    atom => any
+  }
 
   @typedoc "The kind handled by formatting functions"
   @type kind :: :error | :exit | :throw | {:EXIT, pid}

lib/elixir/lib/gen_server.ex

@@ -746,7 +746,7 @@ defmodule GenServer do
   @spec reply(from, term) :: :ok
   def reply(client, reply)
 
-  def reply({to, tag}, reply) do
+  def reply({to, tag}, reply) when is_pid(to) do
     try do
       send(to, {tag, reply})
       :ok

lib/elixir/lib/kernel.ex

@@ -1350,31 +1350,9 @@ defmodule Kernel do
           :erlang.error unquote(alias).exception([])
         end
       _ ->
-        generated = fn fun, var ->
-          {fun, [generated: true, line: -1], [{var, [], __MODULE__}]}
+        quote do
+          :erlang.error Kernel.Utils.raise(unquote(msg))
         end
-
-        {fun, meta, [arg, [do: clauses]]} =
-          quote do
-            case unquote(msg) do
-              msg when unquote(generated.(:is_binary, :msg)) ->
-                :erlang.error RuntimeError.exception(msg)
-              atom when unquote(generated.(:is_atom, :atom)) ->
-                :erlang.error atom.exception([])
-              %{__struct__: struct, __exception__: true} = other when is_atom(struct) ->
-                :erlang.error other
-              other ->
-                message = "raise/1 expects an alias, string or exception as the first argument, got: #{inspect other}"
-                :erlang.error ArgumentError.exception(message)
-            end
-          end
-
-        clauses =
-          :lists.map(fn {:->, meta, args} ->
-            {:->, [generated: true] ++ Keyword.put(meta, :line, -1), args}
-          end, clauses)
-
-        {fun, meta, [arg, [do: clauses]]}
     end
   end
 
@@ -1862,7 +1840,7 @@ defmodule Kernel do
   In case any entry returns `nil`, its key will be removed
   and the deletion will be considered a success.
   """
-  @spec pop_in(Access.t, nonempty_list(term)) :: Access.t
+  @spec pop_in(Access.t, nonempty_list(term)) :: {term, Access.t}
   def pop_in(data, keys)
   def pop_in(nil, [h | _]), do: Access.pop(nil, h)
   def pop_in(data, keys), do: do_pop_in(data, keys)
@@ -3473,7 +3451,7 @@ defmodule Kernel do
       end
 
       unquote(builder)
-
+      Kernel.Utils.announce_struct(__MODULE__)
       fields
     end
   end
@@ -4314,7 +4292,7 @@ defmodule Kernel do
   end
 
   # TODO: Deprecate by v1.5
-  # @doc false
+  @doc false
   defmacro to_char_list(arg) do
     quote do: Kernel.to_charlist(unquote(arg))
   end

lib/elixir/lib/kernel/lexical_tracker.ex

@@ -148,17 +148,15 @@ defmodule Kernel.LexicalTracker do
 
   def handle_cast({:add_import, module, fas, line, warn}, state) when is_atom(module) do
     directives =
-      for {{:import, {import_module, _, _}}, _} = directive <- state.directives,
-          module != import_module,
-          do: directive,
-          into: %{}
-
-    directives = add_directive(directives, module, line, warn, :import)
+      state.directives
+      |> Enum.reject(&match?({{:import, {^module, _, _}}, _}, &1))
+      |> :maps.from_list
+      |> add_directive(module, line, warn, :import)
 
     directives =
-      Enum.reduce fas, directives, fn {function, arity}, directives ->
+      Enum.reduce(fas, directives, fn {function, arity}, directives ->
         add_directive(directives, {module, function, arity}, line, warn, :import)
-      end
+      end)
 
     {:noreply, %{state | directives: directives}}
   end

lib/elixir/lib/kernel/parallel_compiler.ex

@@ -107,6 +107,7 @@ defmodule Kernel.ParallelCompiler do
       :erlang.spawn_monitor fn ->
         # Set the elixir_compiler_pid used by our custom Kernel.ErrorHandler.
         :erlang.put(:elixir_compiler_pid, parent)
+        :erlang.put(:elixir_compiler_file, file)
         :erlang.process_flag(:error_handler, Kernel.ErrorHandler)
 
         exit(try do
@@ -206,8 +207,10 @@ defmodule Kernel.ParallelCompiler do
 
       {:waiting, kind, child, ref, on, defining} ->
         # Oops, we already got it, do not put it on waiting.
+        # OR
+        # We're waiting on ourselves, send :found so that we can crash with a better error
         waiting =
-          if :lists.any(&match?({^kind, ^on}, &1), result) do
+          if :lists.any(&match?({^kind, ^on}, &1), result) or on in defining do
             send child, {ref, :found}
             waiting
           else

lib/elixir/lib/kernel/parallel_require.ex

@@ -47,21 +47,22 @@ defmodule Kernel.ParallelRequire do
     wait_for_messages(files, waiting, callbacks, schedulers, result)
   end
 
-  defp spawn_requires([h | t], waiting, callbacks, schedulers, result) do
+  defp spawn_requires([file | files], waiting, callbacks, schedulers, result) do
     parent = self()
     {pid, ref} = :erlang.spawn_monitor fn ->
       :erlang.put(:elixir_compiler_pid, parent)
+      :erlang.put(:elixir_compiler_file, file)
 
       exit(try do
-        new = Code.require_file(h) || []
-        {:required, Enum.map(new, &elem(&1, 0)), h}
+        new = Code.require_file(file) || []
+        {:required, Enum.map(new, &elem(&1, 0)), file}
       catch
         kind, reason ->
           {:failure, kind, reason, System.stacktrace}
       end)
     end
 
-    spawn_requires(t, [{pid, ref} | waiting], callbacks, schedulers, result)
+    spawn_requires(files, [{pid, ref} | waiting], callbacks, schedulers, result)
   end
 
   defp wait_for_messages(files, waiting, callbacks, schedulers, result) do

lib/elixir/lib/kernel/typespec.ex

@@ -448,7 +448,7 @@ defmodule Kernel.Typespec do
   defp translate_spec(kind, meta, name, args, return, guard, caller) when is_atom(args),
     do: translate_spec(kind, meta, name, [], return, guard, caller)
   defp translate_spec(:macrocallback, meta, name, args, return, guard, caller),
-    do: translate_spec(:callback, meta, :"MACRO-#{name}", [quote(do: env :: Macro.Env.t) | args], return, guard, caller)
+    do: translate_spec(:callback, meta, :"MACRO-#{name}", macro_args(args), return, guard, caller)
   defp translate_spec(kind, meta, name, args, return, guard, caller) do
     ensure_no_defaults!(args)
 
@@ -470,6 +470,10 @@ defmodule Kernel.Typespec do
     {{kind, {name, arity}, spec}, caller.line}
   end
 
+  defp macro_args(args) do
+    [quote(do: {line :: Macro.Env.line, env :: Macro.Env.t}) | args]
+  end
+
   defp ensure_no_defaults!(args) do
     :lists.foreach fn
       {:::, _, [left, right]} ->
@@ -773,6 +777,8 @@ defmodule Kernel.Typespec do
   end
 
   defp typespec({:%, _, [name, {:%{}, meta, fields}]}, vars, caller) do
+    # We cannot set a function name to avoid tracking
+    # as a compile time dependency, because for structs it actually is one.
     module = Macro.expand(name, caller)
 
     struct =
@@ -809,6 +815,8 @@ defmodule Kernel.Typespec do
   end
 
   defp typespec({:record, meta, [atom, fields]}, vars, caller) do
+    # We cannot set a function name to avoid tracking
+    # as a compile time dependency because for records it actually is one.
     case Macro.expand({atom, [], [{atom, [], []}]}, caller) do
       keyword when is_list(keyword) ->
         types =
@@ -864,9 +872,24 @@ defmodule Kernel.Typespec do
     {:op, line(meta), op, {:integer, line(meta), integer}}
   end
 
+  # Handle remote calls in the form of @module_attribute.type.
+  # These are not handled by the general remote type clause as calling
+  # Macro.expand/2 on the remote does not expand module attributes (but expands
+  # things like __MODULE__).
+  defp typespec({{:., meta, [{:@, _, [{attr, _, _}]}, name]}, _, args} = orig, vars, caller) do
+    remote = Module.get_attribute(caller.module, attr)
+    unless is_atom(remote) and remote != nil do
+      message = "invalid remote in typespec: #{Macro.to_string(orig)} (@#{attr} is #{inspect remote})"
+      compile_error(caller, message)
+    end
+    remote_type({typespec(remote, vars, caller), meta, typespec(name, vars, caller), args}, vars, caller)
+  end
+
   # Handle remote calls
   defp typespec({{:., meta, [remote, name]}, _, args} = orig, vars, caller) do
-    remote = Macro.expand remote, caller
+    # We set a function name to avoid tracking
+    # aliases in typespecs as compile time dependencies.
+    remote = Macro.expand remote, %{caller | function: {:typespec, 0}}
     unless is_atom(remote) do
       compile_error(caller, "invalid remote in typespec: #{Macro.to_string(orig)}")
     end

lib/elixir/lib/kernel/utils.ex

@@ -3,6 +3,9 @@ import Kernel, except: [destructure: 2, defdelegate: 2, defstruct: 2]
 defmodule Kernel.Utils do
   @moduledoc false
 
+  @doc """
+  Callback for destructure.
+  """
   def destructure(list, count) when is_list(list), do: destructure_list(list, count)
   def destructure(nil, count), do: destructure_nil(count)
 
@@ -13,6 +16,9 @@ defmodule Kernel.Utils do
   defp destructure_nil(0), do: []
   defp destructure_nil(count), do: [nil | destructure_nil(count - 1)]
 
+  @doc """
+  Callback for defdelegate.
+  """
   def defdelegate(fun, opts) do
     append_first = Keyword.get(opts, :append_first, false)
 
@@ -56,6 +62,9 @@ defmodule Kernel.Utils do
       "defdelegate/2 only accepts function parameters, got: #{Macro.to_string(code)}"
   end
 
+  @doc """
+  Callback for defstruct.
+  """
   def defstruct(module, fields) do
     case fields do
       fs when is_list(fs) ->
@@ -84,4 +93,31 @@ defmodule Kernel.Utils do
      List.wrap(Module.get_attribute(module, :enforce_keys)),
      Module.get_attribute(module, :derive)}
   end
+
+  @doc """
+  Announcing callback for defstruct.
+  """
+  def announce_struct(module) do
+    case :erlang.get(:elixir_compiler_pid) do
+      :undefined -> :ok
+      pid -> send(pid, {:struct_available, module})
+    end
+  end
+
+  @doc """
+  Callback for raise.
+  """
+  def raise(msg) when is_binary(msg) do
+    RuntimeError.exception(msg)
+  end
+  def raise(atom) when is_atom(atom) do
+    atom.exception([])
+  end
+  def raise(%{__struct__: struct, __exception__: true} = exception) when is_atom(struct) do
+    exception
+  end
+  def raise(other) do
+    ArgumentError.exception("raise/1 expects an alias, string or exception as " <>
+                            "the first argument, got: #{inspect other}")
+  end
 end

lib/elixir/lib/macro.ex

@@ -1098,7 +1098,7 @@ defmodule Macro do
 
       case expand do
         {:ok, receiver, quoted} ->
-          next = :elixir_counter.next
+          next = :erlang.unique_integer()
           {:elixir_quote.linify_with_context_counter(0, {receiver, next}, quoted), true}
         {:ok, _receiver, _name, _args} ->
           {original, false}
@@ -1119,7 +1119,7 @@ defmodule Macro do
 
         case expand do
           {:ok, receiver, quoted} ->
-            next = :elixir_counter.next
+            next = :erlang.unique_integer()
             {:elixir_quote.linify_with_context_counter(0, {receiver, next}, quoted), true}
           :error ->
             {original, false}

lib/elixir/lib/module.ex

@@ -369,7 +369,7 @@ defmodule Module do
     * `@compile :debug_info` - includes `:debug_info` regardless of the
       setting in `Code.compiler_options`
 
-    * `@compile {:debug_info, false} - disables `:debug_info` regardless
+    * `@compile {:debug_info, false}` - disables `:debug_info` regardless
       of the setting in `Code.compiler_options`
 
     * `@compile {:inline, some_fun: 2, other_fun: 3}` - inlines the given
@@ -499,7 +499,7 @@ defmodule Module do
       raise ArgumentError, "expected :file to be given as option"
     end
 
-    next = :elixir_counter.next
+    next = :erlang.unique_integer()
     line = Keyword.get(opts, :line, 0)
     quoted = :elixir_quote.linify_with_context_counter(line, {module, next}, quoted)
     :elixir_module.compile(module, quoted, [], :elixir.env_for_eval(opts))

lib/elixir/lib/option_parser.ex

@@ -15,13 +15,24 @@ defmodule OptionParser do
   @doc """
   Parses `argv` into a keywords list.
 
-  It returns a three-element tuple as follows:
+  It returns a three-element tuple with the form `{parsed, args, invalid}`, where:
 
-     1. parsed switches,
-     2. remaining arguments,
-     3. invalid options.
+    * `parsed` is a keyword list of parsed switches with `{switch_name, value}`
+      tuples in it; `switch_name` is the atom representing the switch name while
+      `value` is the value for that switch parsed according to `opts` (see the
+      "Examples" section for more information)
+    * `args` is a list of the remaining arguments in `argv` as strings
+    * `invalid` is a list of invalid options as `{option_name, value}` where
+      `option_name` is the raw option and `value` is `nil` if the option wasn't
+      expected or the string value if the value didn't have the expected type for
+      the corresponding option
 
-  ## Examples
+  Elixir converts switches to underscored atoms, so `--source-path` becomes
+  `:source_path`. This is done to better suit Elixir conventions. However, this
+  means that switches can't contain underscores and switches that do contain
+  underscores are always returned in the list of invalid options.
+
+  Without any options, this function will try to parse all switches in the `argv`.
 
       iex> OptionParser.parse(["--debug"])
       {[debug: true], [], []}
@@ -32,65 +43,85 @@ defmodule OptionParser do
       iex> OptionParser.parse(["--source-path", "lib", "test/enum_test.exs", "--verbose"])
       {[source_path: "lib", verbose: true], ["test/enum_test.exs"], []}
 
-  By default, Elixir will try to automatically parse all switches.
   Switches followed by a value will be assigned the value, as a string.
-  Switches without an argument, like `--debug` will automatically
-  be set to `true`.
+  Switches without an argument, like `--debug` in the examples above, will
+  automatically be set to `true`.
 
-  Note: Elixir also converts the switches to underscore atoms, so
-  `--source-path` becomes `:source_path`, to better suit Elixir
-  conventions. This means that option names on the command line cannot
-  contain underscores; such options will be put in the invalid options
-  list.
+  ## Options
 
-  ## Switch Definitions
+  The following options are supported:
+
+    * `:switches` or `:strict` - see the "Switch definitions" section below
+    * `:aliases` - see the "Aliases" section below
+
+  ## Switch definitions
 
   Often it is better to explicitly list the known
-  switches and their formats. The switches can be specified via two
-  alternative options:
+  switches and their formats. The switches can be specified via one of two
+  options:
 
-    * `:switches` - defines some switches. An attempt is still made to parse
-      switches that do not appear in the list.
+    * `:switches` - defines some switches and their types. This function
+      still attempts to parse switches that are not in this list.
+    * `:strict` - defines strict switches. Any switch in `argv` that is not
+      specified in the list is returned in the invalid options list.
 
-    * `:strict` - the switches are strict. Any switch that is not specified
-      in the list is returned in the invalid options list.
+  Both these options accept a keyword list of `{name, type}` tuples where `name`
+  is an atom defining the name of the switch and `type` is an atom that
+  specifies the type for the value of this switch (see the "Types" section below
+  for the possible types and more information about type casting).
 
   Note that you should only supply the `:switches` or `:strict` option.
-  If you supply both, an error will be raised.
+  If you supply both, an `ArgumentError` exception will be raised.
 
   ### Types
 
-  Option parser switches may take 0 or 1 argument.
+  Switches parsed by `OptionParser` may take zero or one arguments.
 
-  The following switches take no argument:
+  The following switches types take no arguments:
 
-    * `:boolean` - sets the value to true when given
-    * `:count`   - counts the number of times the switch is given
+    * `:boolean` - sets the value to `true` when given (see also the
+      "Negation switches" section below)
+    * `:count` - counts the number of times the switch is given
 
-  The following switches take 1 argument:
+  The following switches take one argument:
 
-    * `:integer` - parses the upcoming value as an integer.
-    * `:float`   - parses the upcoming value as a float.
-    * `:string`  - parses the upcoming value as a string.
+    * `:integer` - parses the value as an integer
+    * `:float` - parses the value as a float
+    * `:string` - parses the value as a string
 
-  If a switch can't be parsed, it is returned in the invalid
-  options list.
+  If a switch can't be parsed according to the given type, it is returned
+  in the invalid options list.
 
   ### Modifiers
 
   Switches can be specified with modifiers, which change how
   they behave. The following modifiers are supported:
 
-    * `:keep` - keeps duplicated items instead of overriding them.
-      Works with all types except `:count`.
+    * `:keep` - keeps duplicated items instead of overriding them; works with
+      all types except `:count`. Specifying `switch_name: :keep` assumes the
+      type of `:switch_name` will be `:string`.
 
-  Note: if you want to use `:keep` with a non-string type, use a list, e.g.
-  `[foo: [:integer, :keep]]`.
+  Note that if you want to use `:keep` with a type other than `:string`, use a list
+  as the type for the switch. For example: `[foo: [:integer, :keep]]`.
 
-  ### Examples
+  ### Negation switches
 
-  Here are some examples of option parser working with different types
-  and modifiers:
+  In case a switch `SWITCH` is specified to have type `:boolean`, it may be
+  passed as `--no-SWITCH` as well which will set the option to `false`:
+
+      iex> OptionParser.parse(["--no-op", "path/to/file"], switches: [op: :boolean])
+      {[op: false], ["path/to/file"], []}
+
+  ## Aliases
+
+  A set of aliases can be specified in the `:aliases` option:
+
+      iex> OptionParser.parse(["-d"], aliases: [d: :debug])
+      {[debug: true], [], []}
+
+  ## Examples
+
+  Here are some examples of working with different types and modifiers:
 
       iex> OptionParser.parse(["--unlock", "path/to/file"], strict: [unlock: :boolean])
       {[unlock: true], ["path/to/file"], []}
@@ -121,21 +152,6 @@ defmodule OptionParser do
       iex> OptionParser.parse(["--unlock", "path/to/file", "--unlock", "path/to/another/file"], strict: [unlock: :keep])
       {[unlock: "path/to/file", unlock: "path/to/another/file"], [], []}
 
-  ### Negation switches
-
-  In case a switch is declared as boolean, it may be passed as `--no-SWITCH`
-  which will set the option to `false`:
-
-      iex> OptionParser.parse(["--no-op", "path/to/file"], switches: [op: :boolean])
-      {[op: false], ["path/to/file"], []}
-
-  ## Aliases
-
-  A set of aliases can be given as options too:
-
-      iex> OptionParser.parse(["-d"], aliases: [d: :debug])
-      {[debug: true], [], []}
-
   """
   @spec parse(argv, options) :: {parsed, argv, errors}
   def parse(argv, opts \\ []) when is_list(argv) and is_list(opts) do
@@ -146,14 +162,16 @@ defmodule OptionParser do
   The same as `parse/2` but raises an `OptionParser.ParseError`
   exception if any invalid options are given.
 
-  If there weren't any errors, returns a three-element tuple as follows:
+  If there are no errors, returns a `{parsed, rest}` tuple where:
 
-      1. parsed options,
-      2. remaining arguments,
-      3. empty list.
+    * `parsed` is the list of parsed switches (same as in `parse/2`)
+    * `rest` is the list of arguments (same as in `parse/2`)
 
   ## Examples
 
+      iex> OptionParser.parse!(["--debug", "path/to/file"], strict: [debug: :boolean])
+      {[debug: true], ["path/to/file"]}
+
       iex> OptionParser.parse!(["--limit", "xyz"], strict: [limit: :integer])
       ** (OptionParser.ParseError) 1 error found!
       --limit : Expected type integer, got "xyz"
@@ -172,7 +190,7 @@ defmodule OptionParser do
   @spec parse!(argv, options) :: {parsed, argv} | no_return
   def parse!(argv, opts \\ []) when is_list(argv) and is_list(opts) do
     case parse(argv, opts) do
-      {parsed, argv, []} -> {parsed, argv}
+      {parsed, args, []} -> {parsed, args}
       {_, _, errors} -> raise ParseError, format_errors(errors, opts)
     end
   end
@@ -201,14 +219,16 @@ defmodule OptionParser do
   The same as `parse_head/2` but raises an `OptionParser.ParseError`
   exception if any invalid options are given.
 
-  If there weren't any errors, returns a three-element tuple as follows:
+  If there are no errors, returns a `{parsed, rest}` tuple where:
 
-      1. parsed options,
-      2. remaining arguments,
-      3. empty list.
+    * `parsed` is the list of parsed switches (same as in `parse_head/2`)
+    * `rest` is the list of arguments (same as in `parse_head/2`)
 
   ## Examples
 
+      iex> OptionParser.parse_head!(["--source", "lib", "path/to/file", "--verbose"])
+      {[source: "lib"], ["path/to/file", "--verbose"]}
+
       iex> OptionParser.parse_head!(["--number", "lib", "test/enum_test.exs", "--verbose"], strict: [number: :integer])
       ** (OptionParser.ParseError) 1 error found!
       --number : Expected type integer, got "lib"
@@ -219,10 +239,10 @@ defmodule OptionParser do
       --verbose : Missing argument of type integer
       --source : Expected type integer, got "lib"
   """
-  @spec parse_head!(argv, options) :: {parsed, argv, errors} | no_return
+  @spec parse_head!(argv, options) :: {parsed, argv} | no_return
   def parse_head!(argv, opts \\ []) when is_list(argv) and is_list(opts) do
     case parse_head(argv, opts) do
-      {parsed, argv, []} -> {parsed, argv}
+      {parsed, args, []} -> {parsed, args}
       {_, _, errors} -> raise ParseError, format_errors(errors, opts)
     end
   end
@@ -250,7 +270,7 @@ defmodule OptionParser do
       {:error, ["--" | rest]} ->
         {Enum.reverse(opts), Enum.reverse(args, rest), Enum.reverse(invalid)}
 
-      {:error, [arg | rest]=remaining_args} ->
+      {:error, [arg | rest] = remaining_args} ->
         # there is no option
         if all? do
           do_parse(rest, config, opts, [arg | args], invalid, all?)
@@ -264,20 +284,20 @@ defmodule OptionParser do
   Low-level function that parses one option.
 
   It accepts the same options as `parse/2` and `parse_head/2`
-  as both functions are built on top of next. This function
+  as both functions are built on top of this function. This function
   may return:
 
     * `{:ok, key, value, rest}` - the option `key` with `value` was
       successfully parsed
 
     * `{:invalid, key, value, rest}` - the option `key` is invalid with `value`
-      (returned when the switch type does not match the one given via the
-      command line)
+      (returned when the value cannot be parsed according to the switch type)
 
     * `{:undefined, key, value, rest}` - the option `key` is undefined
       (returned in strict mode when the switch is unknown)
 
-    * `{:error, rest}` - there are no switches at the top of the given argv
+    * `{:error, rest}` - there are no switches at the head of the given `argv`
+
   """
 
   @spec next(argv, options) ::
@@ -295,15 +315,15 @@ defmodule OptionParser do
     {:error, []}
   end
 
-  defp next(["--" | _]=argv, _aliases, _switches, _strict) do
+  defp next(["--" | _] = argv, _aliases, _switches, _strict) do
     {:error, argv}
   end
 
-  defp next(["-" | _]=argv, _aliases, _switches, _strict) do
+  defp next(["-" | _] = argv, _aliases, _switches, _strict) do
     {:error, argv}
   end
 
-  defp next(["- " <> _ | _]=argv, _aliases, _switches, _strict) do
+  defp next(["- " <> _ | _] = argv, _aliases, _switches, _strict) do
     {:error, argv}
   end
 
@@ -334,8 +354,9 @@ defmodule OptionParser do
   @doc """
   Receives a key-value enumerable and converts it to argv.
 
-  Keys must be atoms. Keys with nil value are discarded,
+  Keys must be atoms. Keys with `nil` value are discarded,
   boolean values are converted to `--key` or `--no-key`
+  (if the value is `true` or `false`, respectively),
   and all other values are converted using `to_string/1`.
 
   ## Examples
@@ -364,6 +385,9 @@ defmodule OptionParser do
   @doc ~S"""
   Splits a string into argv chunks.
 
+  This function splits the given `string` into a list of strings in a similar
+  way to many shells.
+
   ## Examples
 
       iex> OptionParser.split("foo bar")
@@ -371,6 +395,7 @@ defmodule OptionParser do
 
       iex> OptionParser.split("foo \"bar baz\"")
       ["foo", "bar baz"]
+
   """
   @spec split(String.t) :: argv
   def split(string) do

lib/elixir/lib/protocol.ex

@@ -451,6 +451,16 @@ defmodule Protocol do
         end
       end, builtin)
 
+      # Define a catch-all impl_for/1 clause to pacify Dialyzer (since
+      # destructuring opaque types is illegal, Dialyzer will think none of the
+      # previous clauses matches opaque types, and without this clause, will
+      # conclude that impl_for can't handle an opaque argument). This is a hack
+      # since it relies on Dialyzer not being smart enough to conclude that all
+      # opaque types will get the any_impl_for/0 implementation.
+      Kernel.def impl_for(_) do
+        any_impl_for()
+      end
+
       @doc false
       @spec impl_for!(term) :: atom | no_return
       Kernel.def impl_for!(data) do

lib/elixir/lib/string.ex

@@ -350,6 +350,7 @@ defmodule String do
   defp parts_to_index(:infinity),                      do: 0
   defp parts_to_index(n) when is_integer(n) and n > 0, do: n
 
+  defp split_each("", _pattern, true, 1), do: []
   defp split_each(string, _pattern, _trim, 1) when is_binary(string), do: [string]
   defp split_each(string, pattern, trim, count) do
     case do_splitter(string, pattern, trim) do
@@ -402,6 +403,7 @@ defmodule String do
   end
 
   defp maybe_compile_pattern(""), do: ""
+  defp maybe_compile_pattern(pattern) when is_tuple(pattern), do: pattern
   defp maybe_compile_pattern(pattern), do: :binary.compile_pattern(pattern)
 
   @doc """

lib/elixir/lib/system.ex

@@ -34,7 +34,6 @@ defmodule System do
       next = System.monotonic_time()
       diff = next - prev
 
-
   Generally speaking, the VM provides three time measurements:
 
     * `os_time/0` - the time reported by the OS. This time may be
@@ -113,7 +112,15 @@ defmodule System do
   # Tries to run "git rev-parse --short HEAD". In the case of success returns
   # the short revision hash. If that fails, returns an empty string.
   defmacrop get_revision do
-    :os.cmd('git rev-parse --short HEAD 2> /dev/null')
+    null =
+      case :os.type do
+        {:win32, _} -> 'NUL'
+        _ -> '/dev/null'
+      end
+
+    'git rev-parse --short HEAD 2> '
+    |> Kernel.++(null)
+    |> :os.cmd()
     |> strip
   end
 
@@ -157,8 +164,7 @@ defmodule System do
     %{build:    build,
       date:     get_date,
       revision: revision,
-      version:  version,
-      }
+      version:  version}
   end
 
   # Returns a string of the build info
@@ -730,6 +736,30 @@ defmodule System do
     :os.system_time(normalize_time_unit(unit))
   end
 
+  @doc """
+  Returns the OTP release number.
+  """
+  @spec otp_release :: String.t
+  def otp_release do
+    :erlang.list_to_binary :erlang.system_info(:otp_release)
+  end
+
+  @doc """
+  Returns the number of schedulers in the VM.
+  """
+  @spec schedulers :: pos_integer
+  def schedulers do
+    :erlang.system_info(:schedulers)
+  end
+
+  @doc """
+  Returns the number of schedulers online in the VM.
+  """
+  @spec schedulers_online :: pos_integer
+  def schedulers_online do
+    :erlang.system_info(:schedulers_online)
+  end
+
   @doc """
   Generates and returns an integer that is unique in the current runtime
   instance.

lib/elixir/rebar.config

@@ -1,23 +1,24 @@
 {erl_opts, [
-            warn_unused_vars,
-            warn_export_all,
-            warn_shadow_vars,
-            warn_unused_import,
-            warn_unused_function,
-            warn_bif_clash,
-            warn_unused_record,
-            warn_deprecated_function,
-            warn_obsolete_guard,
-            strict_validation,
-            warn_exported_vars,
-            %% warn_export_vars,
-            %% warn_missing_spec,
-            %% warn_untyped_record,
-            %% warnings_as_errors,
-            debug_info
-           ]}.
+  warn_unused_vars,
+  warn_export_all,
+  warn_shadow_vars,
+  warn_unused_import,
+  warn_unused_function,
+  warn_bif_clash,
+  warn_unused_record,
+  warn_deprecated_function,
+  warn_obsolete_guard,
+  strict_validation,
+  warn_exported_vars,
+  %% warn_export_vars,
+  %% warn_missing_spec,
+  %% warn_untyped_record,
+  %% warnings_as_errors,
+  debug_info,
+  {platform_define, "^18.*", old_map_specs}
+]}.
 
 {yrl_opts, [
-            {report, true},
-            {verbose, false}
-           ]}.
+  {report, true},
+  {verbose, false}
+]}.

lib/elixir/src/elixir.erl

@@ -14,11 +14,16 @@
 -export_type([charlist/0, char_list/0, struct/0, as_boolean/1, keyword/0, keyword/1]).
 -type charlist() :: string().
 -type char_list() :: string().
--type struct() :: #{'__struct__' => atom()}.
 -type as_boolean(T) :: T.
 -type keyword() :: [{atom(), any()}].
 -type keyword(T) :: [{atom(), T}].
 
+-ifdef(old_map_specs).
+-type struct() :: #{'__struct__' => atom(), atom() => any()}.
+-else.
+-type struct() :: #{'__struct__' := atom(), atom() => any()}.
+-endif.
+
 %% OTP Application API
 
 -export([start/2, stop/1, config_change/3]).

lib/elixir/src/elixir_clauses.erl

@@ -198,17 +198,17 @@ normalize_vars(Key, {Ref, Counter, _Safe},
 % Generate match vars by checking if they were updated
 % or not and assigning the previous value.
 
-generate_match_vars([{Key, Value, Expr} | T], ClauseVars, Left, Right) ->
+generate_match_vars([{Key, {Value, _, _}, Expr} | T], ClauseVars, Left, Right) ->
   case maps:find(Key, ClauseVars) of
-    {ok, Value} ->
+    {ok, {Value, _, _}} ->
       generate_match_vars(T, ClauseVars, Left, Right);
-    {ok, Clause} ->
+    {ok, {Clause, _, _}} ->
       generate_match_vars(T, ClauseVars,
-        [{var, 0, element(1, Value)} | Left],
-        [{var, 0, element(1, Clause)} | Right]);
+        [{var, 0, Value} | Left],
+        [{var, 0, Clause} | Right]);
     error ->
       generate_match_vars(T, ClauseVars,
-        [{var, 0, element(1, Value)} | Left], [Expr | Right])
+        [{var, 0, Value} | Left], [Expr | Right])
   end;
 
 generate_match_vars([], _ClauseVars, Left, Right) ->

lib/elixir/src/elixir_counter.erl

@@ -1,38 +0,0 @@
--module(elixir_counter).
--export([start_link/0, init/1, handle_call/3, handle_cast/2,
-  handle_info/2, terminate/2, code_change/3, next/0]).
--behaviour(gen_server).
-
--define(timeout, 30000).    %% 30 seconds
--define(limit, 4294967295). %% 2^32 - 1
-
-next() ->
-  gen_server:call(?MODULE, next, ?timeout).
-
-start_link() ->
-  gen_server:start_link({local, ?MODULE}, ?MODULE, 0, []).
-
-init(Counter) ->
-  {ok, Counter}.
-
-handle_call(next, _From, Counter) ->
-  {reply, Counter, bump(Counter)};
-handle_call(Request, _From, Counter) ->
-  {stop, {badcall, Request}, Counter}.
-
-handle_cast(Request, Counter) ->
-  {stop, {badcast, Request}, Counter}.
-
-handle_info(_Request, Counter) ->
-  {noreply, Counter}.
-
-terminate(_Reason, _Counter) ->
-  ok.
-
-code_change(_Old, Counter, _Extra) ->
-  {ok, Counter}.
-
-bump(Counter) when Counter < ?limit ->
-  Counter + 1;
-bump(_Counter) ->
-  0.

lib/elixir/src/elixir_def.erl

@@ -105,8 +105,6 @@ store_definition(Meta, Line, Kind, CheckClauses, Name, Args, Guards, Body, KeepL
   store_each(CheckClauses, Kind, File, Location, Module, DefaultsLength, Function),
   [store_each(false, Kind, File, Location, Module, 0,
     default_function_for(Kind, Name, Default)) || Default <- Defaults],
-
-  make_struct_available(Kind, Module, Name, Args),
   {Name, Arity}.
 
 %% @on_definition
@@ -117,16 +115,6 @@ run_on_definition_callbacks(Kind, Line, Module, Name, Args, Guards, Expr, E) ->
   _ = [Mod:Fun(Env, Kind, Name, Args, Guards, Expr) || {Mod, Fun} <- Callbacks],
   ok.
 
-make_struct_available(def, Module, '__struct__', []) ->
-  case erlang:get(elixir_compiler_pid) of
-    undefined -> ok;
-    Pid ->
-      Pid ! {struct_available, Module},
-      ok
-  end;
-make_struct_available(_, _, _, _) ->
-  ok.
-
 %% Retrieve location from meta file (if Key == keep)
 %% or @file, otherwise nil
 
@@ -218,7 +206,7 @@ is_macro(_)         -> false.
 unwrap_definitions(File, Module) ->
   Table = elixir_module:defs_table(Module),
   Entries = ets:match(Table, {{def, '$1'}, '_', '_', '_', '_', '_', '_'}),
-  {All, Private} = unwrap_definition(Entries, File, Module, Table, [], []),
+  {All, Private} = unwrap_definition(lists:sort(Entries), File, Module, Table, [], []),
   Unreachable = elixir_locals:warn_unused_local(File, Module, Private),
   split_definition(All, Unreachable, [], [], [], [], [], {[], []}).
 

lib/elixir/src/elixir_dispatch.erl

@@ -60,7 +60,8 @@ import_function(Meta, Name, Arity, E) ->
   end.
 
 require_function(Meta, Receiver, Name, Arity, E) ->
-  case is_element({Name, Arity}, get_optional_macros(Receiver)) of
+  Required = is_element(Receiver, ?m(E, requires)),
+  case is_element({Name, Arity}, get_macros(Receiver, Required)) of
     true  -> false;
     false ->
       elixir_lexical:record_remote(Receiver, ?m(E, function), ?m(E, lexical_tracker)),
@@ -146,7 +147,7 @@ do_expand_import(Meta, {Name, Arity} = Tuple, Args, Module, E, Result) ->
       elixir_locals:record_import(Tuple, Receiver, Module, ?m(E, function)),
       {ok, Receiver, expand_macro_named(Meta, Receiver, Name, Arity, Args, E)};
     {import, Receiver} ->
-      case expand_require([{require, false} | Meta], Receiver, Tuple, Args, E) of
+      case expand_require([{required, true} | Meta], Receiver, Tuple, Args, E) of
         {ok, _, _} = Response -> Response;
         error -> {ok, Receiver, Name, Args}
       end;
@@ -161,19 +162,15 @@ do_expand_import(Meta, {Name, Arity} = Tuple, Args, Module, E, Result) ->
 
 expand_require(Meta, Receiver, {Name, Arity} = Tuple, Args, E) ->
   check_deprecation(Meta, Receiver, Name, Arity, E),
-  Module = ?m(E, module),
+  Required = (Receiver == ?m(E, module)) orelse is_element(Receiver, ?m(E, requires)) orelse required(Meta),
 
-  case is_element(Tuple, get_optional_macros(Receiver)) of
+  case is_element(Tuple, get_macros(Receiver, Required)) of
+    true when Required ->
+      elixir_lexical:record_remote(Receiver, Name, Arity, nil, ?line(Meta), ?m(E, lexical_tracker)),
+      {ok, Receiver, expand_macro_named(Meta, Receiver, Name, Arity, Args, E)};
     true ->
-      Requires = ?m(E, requires),
-      case (Receiver == Module) orelse is_element(Receiver, Requires) orelse skip_require(Meta) of
-        true  ->
-          elixir_lexical:record_remote(Receiver, Name, Arity, nil, ?line(Meta), ?m(E, lexical_tracker)),
-          {ok, Receiver, expand_macro_named(Meta, Receiver, Name, Arity, Args, E)};
-        false ->
-          Info = {unrequired_module, {Receiver, Name, length(Args), Requires}},
-          elixir_errors:form_error(Meta, ?m(E, file), ?MODULE, Info)
-      end;
+      Info = {unrequired_module, {Receiver, Name, length(Args), ?m(E, requires)}},
+      elixir_errors:form_error(Meta, ?m(E, file), ?MODULE, Info);
     false ->
       error
   end.
@@ -202,7 +199,7 @@ expand_macro_named(Meta, Receiver, Name, Arity, Args, E) ->
 
 expand_quoted(Meta, Receiver, Name, Arity, Quoted, E) ->
   Line = ?line(Meta),
-  Next = elixir_counter:next(),
+  Next = erlang:unique_integer(),
 
   try
     elixir_exp:expand(
@@ -220,8 +217,8 @@ caller(Line, E) ->
 
 %% Helpers
 
-skip_require(Meta) ->
-  lists:keyfind(require, 1, Meta) == {require, false}.
+required(Meta) ->
+  lists:keyfind(required, 1, Meta) == {required, true}.
 
 find_dispatch(Meta, Tuple, Extra, E) ->
   case is_import(Meta) of
@@ -289,9 +286,20 @@ format_error({ambiguous_call, {Mod1, Mod2, Name, Arity}}) ->
 %% INTROSPECTION
 
 %% Do not try to get macros from Erlang. Speeds up compilation a bit.
-get_optional_macros(erlang) -> [];
+get_macros(erlang, _) -> [];
 
-get_optional_macros(Receiver) ->
+get_macros(Receiver, false) ->
+  case code:is_loaded(Receiver) of
+    {file, _} ->
+      try
+        Receiver:'__info__'(macros)
+      catch
+        error:undef -> []
+      end;
+    false -> []
+  end;
+
+get_macros(Receiver, true) ->
   case code:ensure_loaded(Receiver) of
     {module, Receiver} ->
       try

lib/elixir/src/elixir_fn.erl

@@ -8,7 +8,7 @@ translate(Meta, Clauses, S) ->
     {Args, Guards} = elixir_clauses:extract_splat_guards(ArgsWithGuards),
     {TClause, TS } = elixir_clauses:clause(CMeta, fun translate_fn_match/2,
                                             Args, Expr, Guards, Acc),
-    {TClause, elixir_scope:mergef(S, TS)}
+    {TClause, elixir_scope:mergec(S, TS)}
   end,
 
   {TClauses, NS} = lists:mapfoldl(Transformer, S, Clauses),
@@ -111,7 +111,7 @@ handle_capture(false, Meta, Expr, E, Sequential) ->
   do_capture(Meta, Expr, E, Sequential).
 
 do_capture(Meta, Expr, E, Sequential) ->
-  case do_escape(Expr, elixir_counter:next(), E, []) of
+  case do_escape(Expr, erlang:unique_integer(), E, []) of
     {_, []} when not Sequential ->
       invalid_capture(Meta, Expr, E);
     {EExpr, EDict} ->

lib/elixir/src/elixir_locals.erl

@@ -31,7 +31,7 @@ local_for(Module, Name, Arity, Given) ->
     {Kind, Ann, [_ | _] = Clauses} when Given == nil; Kind == Given ->
       get_function(Ann, Module, Clauses);
     _ ->
-      [_ | T] = erlang:get_stacktrace(),
+      {current_stacktrace, [_ | T]} = erlang:process_info(self(), current_stacktrace),
       erlang:raise(error, undef, [{Module, Name, Arity, []} | T])
   end.
 

lib/elixir/src/elixir_map.erl

@@ -19,7 +19,7 @@ expand_struct(Meta, Left, Right, #{context := Context} = E) ->
     true when is_atom(ELeft) ->
       %% We always record structs when they are expanded
       %% as they expect the reference at compile time.
-      elixir_lexical:record_remote(ELeft, nil, ?m(E, lexical_tracker));
+      elixir_lexical:record_remote(ELeft, '__struct__', 1, nil, ?line(Meta), ?m(E, lexical_tracker));
     true ->
       ok;
     false when Context == match ->
@@ -31,7 +31,9 @@ expand_struct(Meta, Left, Right, #{context := Context} = E) ->
   end,
 
   EMeta =
-    case lists:member(ELeft, ?m(E, context_modules)) of
+    %% We also include the current module because it won't be present
+    %% in context module in case the module name is defined dynamically.
+    case lists:member(ELeft, [?m(E, module) | ?m(E, context_modules)]) of
       true  -> [{struct, context} | Meta];
       false -> Meta
     end,
@@ -90,9 +92,9 @@ translate_struct(Meta, Name, {'%{}', MapMeta, Args}, S) ->
 
       {TMap, TS} = translate_map(MapMeta, Assocs, Var, VS),
 
-      {{'case', Ann, TUpdate, [
+      {{'case', ?generated, TUpdate, [
         {clause, Ann, [Match], [], [TMap]},
-        {clause, Ann, [Var], [], [elixir_utils:erl_call(Ann, erlang, error, [Error])]}
+        {clause, ?generated, [Var], [], [elixir_utils:erl_call(Ann, erlang, error, [Error])]}
       ]}, TS};
     match ->
       translate_map(MapMeta, Assocs ++ [{'__struct__', Name}], nil, US);

lib/elixir/src/elixir_module.erl

@@ -378,7 +378,7 @@ load_form(Line, Data, Forms, Opts, E) ->
             ok;
           PID ->
             Ref = make_ref(),
-            PID ! {module_available, self(), Ref, ?m(E, file), Module, Binary},
+            PID ! {module_available, self(), Ref, get(elixir_compiler_file), Module, Binary},
             receive {Ref, ack} -> ok end
         end;
       _ ->
@@ -525,7 +525,7 @@ add_beam_chunk(Bin, Id, ChunkData)
 %% the callback can't be expanded, invokes the given
 %% fun passing a possibly expanded AM:AF(Args).
 expand_callback(Line, M, F, Args, E, Fun) ->
-  Meta = [{line, Line}, {require, false}],
+  Meta = [{line, Line}, {required, true}],
 
   {EE, ET} = elixir_dispatch:dispatch_require(Meta, M, F, Args, E, fun(AM, AF, AA) ->
     Fun(AM, AF, AA),

lib/elixir/src/elixir_scope.erl

@@ -2,8 +2,7 @@
 -module(elixir_scope).
 -export([translate_var/4, build_var/2, context_info/1,
   load_binding/2, dump_binding/2,
-  mergev/2, mergec/2, mergef/2,
-  merge_vars/2, merge_opt_vars/2,
+  mergev/2, mergec/2, merge_vars/2, merge_opt_vars/2,
   warn_unsafe_var/4, warn_underscored_var_access/3, format_error/1
 ]).
 -include("elixir.hrl").
@@ -128,14 +127,6 @@ mergec(S1, S2) ->
     caller=S2#elixir_scope.caller
  }.
 
-%% Similar to mergec but does not merge the user vars counter.
-
-mergef(S1, S2) ->
-  S1#elixir_scope{
-    super=S2#elixir_scope.super,
-    caller=S2#elixir_scope.caller
- }.
-
 %% Mergers.
 
 merge_vars(V, V) -> V;
@@ -212,8 +203,8 @@ format_error({unused_match, Name, Kind}) ->
 
 format_error({unsafe_var, Name}) ->
   io_lib:format("the variable \"~ts\" is unsafe as it has been set inside "
-                "a case/cond/receive/if/&&/||. Please explicitly return the "
-                "variable value instead. For example:\n\n"
+                "a case/cond/receive/if/&&/||. Please explicitly return "
+                "the variable value instead. For example:\n\n"
                 "    case int do\n"
                 "      1 -> atom = :one\n"
                 "      2 -> atom = :two\n"

lib/elixir/src/elixir_sup.erl

@@ -15,7 +15,7 @@ init(ok) ->
       2000,                         % Shutdown = brutal_kill | int() >= 0 | infinity
       worker,                       % Type     = worker | supervisor
       [elixir_config]               % Modules  = [Module] | dynamic
-   },
+    },
 
     {
       elixir_code_server,
@@ -25,17 +25,7 @@ init(ok) ->
       2000,                         % Shutdown = brutal_kill | int() >= 0 | infinity
       worker,                       % Type     = worker | supervisor
       [elixir_code_server]          % Modules  = [Module] | dynamic
-   },
-
-    {
-      elixir_counter,
-      {elixir_counter, start_link, []},
-
-      permanent,                    % Restart  = permanent | transient | temporary
-      2000,                         % Shutdown = brutal_kill | int() >= 0 | infinity
-      worker,                       % Type     = worker | supervisor
-      [elixir_counter]              % Modules  = [Module] | dynamic
-   }
+    }
   ],
 
   {ok, {{one_for_one, 3, 10}, Workers}}.

lib/elixir/src/elixir_tokenizer.erl

@@ -135,15 +135,15 @@ tokenize([], EndLine, _Column, #elixir_tokenizer{terminators=[{Start, {StartLine
 % Base integers
 
 tokenize([$0, $x, H | T], Line, Column, Scope, Tokens) when ?is_hex(H) ->
-  {Rest, Number, Length} = tokenize_hex([H | T], []),
+  {Rest, Number, Length} = tokenize_hex(T, [H], 1),
   tokenize(Rest, Line, Column + 2 + Length, Scope, [{number, {Line, Column, Column + 2 + Length}, Number} | Tokens]);
 
 tokenize([$0, $b, H | T], Line, Column, Scope, Tokens) when ?is_bin(H) ->
-  {Rest, Number, Length} = tokenize_bin([H | T], []),
+  {Rest, Number, Length} = tokenize_bin(T, [H], 1),
   tokenize(Rest, Line, Column + 2 + Length, Scope, [{number, {Line, Column, Column + 2 + Length}, Number} | Tokens]);
 
 tokenize([$0, $o, H | T], Line, Column, Scope, Tokens) when ?is_octal(H) ->
-  {Rest, Number, Length} = tokenize_octal([H | T], []),
+  {Rest, Number, Length} = tokenize_octal(T, [H], 1),
   tokenize(Rest, Line, Column + 2 + Length, Scope, [{number, {Line, Column, Column + 2 + Length}, Number} | Tokens]);
 
 % Comments
@@ -413,8 +413,8 @@ tokenize([$. | T], Line, Column, Scope, Tokens) ->
 
 % Integers and floats
 
-tokenize([H | _] = String, Line, Column, Scope, Tokens) when ?is_digit(H) ->
-  {Rest, Number, Length} = tokenize_number(String, [], false),
+tokenize([H | T], Line, Column, Scope, Tokens) when ?is_digit(H) ->
+  {Rest, Number, Length} = tokenize_number(T, [H], 1, false),
   tokenize(Rest, Line, Column + Length, Scope, [{number, {Line, Column, Column + Length}, Number} | Tokens]);
 
 % Identifiers (including aliases)
@@ -733,46 +733,55 @@ extract_heredoc_line(Marker, Rest, Buffer, _Counter) ->
 %% At this point, we are at least sure the first digit is a number.
 
 %% Check if we have a point followed by a number;
-tokenize_number([$., H | T], Acc, false) when ?is_digit(H) ->
-  tokenize_number(T, [H, $. | Acc], true);
+tokenize_number([$., H | T], Acc, Length, false) when ?is_digit(H) ->
+  tokenize_number(T, [H, $. | Acc], Length + 2, true);
 
 %% Check if we have an underscore followed by a number;
-tokenize_number([$_, H | T], Acc, Bool) when ?is_digit(H) ->
-  tokenize_number(T, [H | Acc], Bool);
+tokenize_number([$_, H | T], Acc, Length, Bool) when ?is_digit(H) ->
+  tokenize_number(T, [H | Acc], Length + 2, Bool);
 
 %% Check if we have e- followed by numbers (valid only for floats);
-tokenize_number([E, S, H | T], Acc, true)
+tokenize_number([E, S, H | T], Acc, Length, true)
     when (E == $E) or (E == $e), ?is_digit(H), S == $+ orelse S == $- ->
-  tokenize_number(T, [H, S, $e | Acc], true);
+  tokenize_number(T, [H, S, $e | Acc], Length + 3, true);
 
 %% Check if we have e followed by numbers (valid only for floats);
-tokenize_number([E, H | T], Acc, true)
+tokenize_number([E, H | T], Acc, Length, true)
     when (E == $E) or (E == $e), ?is_digit(H) ->
-  tokenize_number(T, [H, $e | Acc], true);
+  tokenize_number(T, [H, $e | Acc], Length + 2, true);
 
 %% Finally just numbers.
-tokenize_number([H | T], Acc, Bool) when ?is_digit(H) ->
-  tokenize_number(T, [H | Acc], Bool);
+tokenize_number([H | T], Acc, Length, Bool) when ?is_digit(H) ->
+  tokenize_number(T, [H | Acc], Length + 1, Bool);
 
 %% Cast to float...
-tokenize_number(Rest, Acc, true) ->
-  {Rest, list_to_float(lists:reverse(Acc)), length(Acc)};
+tokenize_number(Rest, Acc, Length, true) ->
+  {Rest, list_to_float(lists:reverse(Acc)), Length};
 
 %% Or integer.
-tokenize_number(Rest, Acc, false) ->
-  {Rest, list_to_integer(lists:reverse(Acc)), length(Acc)}.
+tokenize_number(Rest, Acc, Length, false) ->
+  {Rest, list_to_integer(lists:reverse(Acc)), Length}.
 
-tokenize_hex([H | T], Acc) when ?is_hex(H) -> tokenize_hex(T, [H | Acc]);
-tokenize_hex([$_, H | T], Acc) when ?is_hex(H) -> tokenize_hex(T, [H | Acc]);
-tokenize_hex(Rest, Acc) -> {Rest, list_to_integer(lists:reverse(Acc), 16), length(Acc)}.
+tokenize_hex([H | T], Acc, Length) when ?is_hex(H) ->
+  tokenize_hex(T, [H | Acc], Length + 1);
+tokenize_hex([$_, H | T], Acc, Length) when ?is_hex(H) ->
+  tokenize_hex(T, [H | Acc], Length + 2);
+tokenize_hex(Rest, Acc, Length) ->
+  {Rest, list_to_integer(lists:reverse(Acc), 16), Length}.
 
-tokenize_octal([H | T], Acc) when ?is_octal(H) -> tokenize_octal(T, [H | Acc]);
-tokenize_octal([$_, H | T], Acc) when ?is_octal(H) -> tokenize_octal(T, [H | Acc]);
-tokenize_octal(Rest, Acc) -> {Rest, list_to_integer(lists:reverse(Acc), 8), length(Acc)}.
+tokenize_octal([H | T], Acc, Length) when ?is_octal(H) ->
+  tokenize_octal(T, [H | Acc], Length + 1);
+tokenize_octal([$_, H | T], Acc, Length) when ?is_octal(H) ->
+  tokenize_octal(T, [H | Acc], Length + 2);
+tokenize_octal(Rest, Acc, Length) ->
+  {Rest, list_to_integer(lists:reverse(Acc), 8), Length}.
 
-tokenize_bin([H | T], Acc) when ?is_bin(H) -> tokenize_bin(T, [H | Acc]);
-tokenize_bin([$_, H | T], Acc) when ?is_bin(H) -> tokenize_bin(T, [H | Acc]);
-tokenize_bin(Rest, Acc) -> {Rest, list_to_integer(lists:reverse(Acc), 2), length(Acc)}.
+tokenize_bin([H | T], Acc, Length) when ?is_bin(H) ->
+  tokenize_bin(T, [H | Acc], Length + 1);
+tokenize_bin([$_, H | T], Acc, Length) when ?is_bin(H) ->
+  tokenize_bin(T, [H | Acc], Length + 2);
+tokenize_bin(Rest, Acc, Length) ->
+  {Rest, list_to_integer(lists:reverse(Acc), 2), Length}.
 
 %% Comments
 

lib/elixir/src/elixir_with.erl

@@ -45,7 +45,7 @@ expand(Meta, Args, E) ->
 
   {ECases, EC} = lists:mapfoldl(fun expand/2, E, Cases),
   {EDoExpr, _} = elixir_exp:expand(DoExpr, EC),
-  {EElseExpr, _} = expand_else(ElseExpr, E),
+  {EElseExpr, _} = expand_else(Meta, ElseExpr, E),
   {{with, Meta, ECases ++ [[{do, EDoExpr} | EElseExpr]]}, E}.
 
 expand({'<-', Meta, [Left, Right]}, E) ->
@@ -55,25 +55,36 @@ expand({'<-', Meta, [Left, Right]}, E) ->
 expand(X, E) ->
   elixir_exp:expand(X, E).
 
-expand_else(KV, E) when is_list(KV) ->
+expand_else(_Meta, KV, E) when is_list(KV) ->
   {[{do, EClauses}], EC} = elixir_exp_clauses:'case'([], [{do, KV}], E),
   {[{else, EClauses}], EC};
-expand_else(nil, E) ->
-  {[], E}.
+expand_else(_Meta, nil, E) ->
+  {[], E};
+expand_else(Meta, _KV, E) ->
+  Message = "expected -> clauses for else in with",
+  elixir_errors:compile_error(Meta, ?m(E, file), Message, []).
 
 %% Translation
 
 translate(Meta, Args, S) ->
   {Parts, [{do, Expr} | ExprList]} = elixir_utils:split_last(Args),
-  CaseExpr =
-    case ExprList of
-      [{else, ElseExpr}] ->
-        build_else(Meta, build_case(Parts, {ok, Expr}, fun(X) -> {error, X} end), ElseExpr);
-      [] ->
-        build_case(Parts, Expr, fun(X) -> X end)
-    end,
-  {TC, TS} = elixir_translator:translate(CaseExpr, S#elixir_scope{extra=nil}),
-  {TC, elixir_scope:mergec(S, TS)}.
+  case ExprList of
+    [{else, ElseExpr}] ->
+      {TCases, TS} = translate_case(Parts, {ok, Expr}, fun(X) -> {error, X} end, S),
+      translate_else(Meta, TCases, ElseExpr, TS);
+    [] ->
+      translate_case(Parts, Expr, fun(X) -> X end, S)
+  end.
+
+translate_case(Parts, DoExpr, Wrapper, S) ->
+  Cases = build_case(Parts, DoExpr, Wrapper),
+  {TCases, TS} = elixir_translator:translate(Cases, S#elixir_scope{extra=nil}),
+  {TCases, elixir_scope:mergec(S, TS)}.
+
+translate_else(Meta, WithCases, ElseExpr, S) ->
+  ElseClauses = build_else(Meta, ElseExpr),
+  {TClauses, TS} = elixir_clauses:clauses(Meta, ElseClauses, S#elixir_scope{extra=nil}),
+  {{'case', ?ann(Meta), WithCases, TClauses}, elixir_scope:mergec(S, TS)}.
 
 build_case([{'<-', Meta, [{Name, _, Ctx}, _] = Args} | Rest], DoExpr, Wrapper)
     when is_atom(Name) andalso is_atom(Ctx) ->
@@ -90,18 +101,20 @@ build_case([Expr | Rest], DoExpr, Wrapper) ->
 build_case([], DoExpr, _Wrapper) ->
   DoExpr.
 
-build_else(Meta, WithCases, ElseClauses) ->
+build_else(Meta, ElseClauses) ->
   Result = {result, Meta, ?MODULE},
-  Clauses = [
-    {'->', Meta, [[{ok, Result}], Result]}
-    | else_to_error_clause(ElseClauses)
-  ] ++ [build_raise(Meta)],
-  {'case', Meta, [WithCases, [{do, Clauses}]]}.
+  [{match, Meta, [{ok, Result}], Result} |
+    each_clause_to_error_match(ElseClauses)] ++ [build_raise(Meta)].
 
-else_to_error_clause(Clauses) ->
-  [{'->', Meta, [[{error, Match}], Expr]} ||
+each_clause_to_error_match(Clauses) ->
+  [{match, Meta, [error_match_for_match(Match)], Expr} ||
     {'->', Meta, [[Match], Expr]} <- Clauses].
 
+error_match_for_match({'when', Meta, [Left, Right]}) ->
+  {'when', Meta, [{error, Left}, Right]};
+error_match_for_match(Match) ->
+  {error, Match}.
+
 build_raise(Meta) ->
-  Other = {raise, Meta, ?MODULE},
-  {'->', ?generated, [[{error, Other}], {{'.', Meta, [erlang, error]}, Meta, [{with_clause, Other}]}]}.
+  Other = {other, Meta, ?MODULE},
+  {match, ?generated, [{error, Other}], {{'.', Meta, [erlang, error]}, Meta, [{with_clause, Other}]}}.

lib/elixir/test/elixir/behaviour_test.exs

@@ -3,30 +3,33 @@ Code.require_file "test_helper.exs", __DIR__
 defmodule BehaviourTest do
   use ExUnit.Case, async: true
 
-  defmodule Sample do
-    use Behaviour
+  {_, _, sample_binary, _} =
+    defmodule Sample do
+      use Behaviour
 
-    defcallback first(integer) :: integer
+      defcallback first(integer) :: integer
 
-    defcallback foo(atom(), binary) :: binary
+      defcallback foo(atom(), binary) :: binary
 
-    defcallback bar(External.hello, my_var :: binary) :: binary
+      defcallback bar(External.hello, my_var :: binary) :: binary
 
-    defcallback guarded(my_var) :: my_var when my_var: binary
+      defcallback guarded(my_var) :: my_var when my_var: binary
 
-    defcallback orr(atom | integer) :: atom
+      defcallback orr(atom | integer) :: atom
 
-    defcallback literal(123, {atom}, :atom, [integer], true) :: atom
+      defcallback literal(123, {atom}, :atom, [integer], true) :: atom
 
-    defmacrocallback last(integer) :: Macro.t
-  end
+      defmacrocallback last(integer) :: Macro.t
+    end
+
+  @sample_binary sample_binary
 
   test "callbacks" do
     assert Sample.__behaviour__(:callbacks) == [first: 1, guarded: 1, "MACRO-last": 2, literal: 5, orr: 1, foo: 2, bar: 2]
   end
 
   test "specs" do
-    assert length(Keyword.get_values(Sample.module_info[:attributes], :callback)) == 7
+    assert length(Kernel.Typespec.beam_callbacks(@sample_binary)) == 7
   end
 
   test "default is not supported" do

lib/elixir/test/elixir/calendar_test.exs

@@ -66,4 +66,25 @@ defmodule DateTimeTest do
                    utc_offset: -12600, std_offset: 3600, time_zone: "Brazil/Manaus"}
     assert to_string(dt) == "2000-02-29 23:00:07-02:30 BRM Brazil/Manaus"
   end
+
+  test "from_unix/2 works with Unix times back to 0 Gregorian Seconds" do
+    assert DateTime.from_unix(-62167219200) == {:ok,
+                    %DateTime{calendar: Calendar.ISO, day: 1, hour: 0, microsecond: {0, 0},
+                              minute: 0, month: 1, second: 0, std_offset: 0, time_zone: "Etc/UTC",
+                              utc_offset: 0, year: 0, zone_abbr: "UTC"}}
+
+    assert DateTime.from_unix(-62167219201) == :error
+  end
+
+  test "to_unix/2 works with Unix times back to 0 Gregorian Seconds" do
+    gregorian_0 = %DateTime{calendar: Calendar.ISO, day: 1, hour: 0, microsecond: {0, 0},
+                            minute: 0, month: 1, second: 0, std_offset: 0, time_zone: "Etc/UTC",
+                            utc_offset: 0, year: 0, zone_abbr: "UTC"}
+    before_gregorian_0 = %DateTime{gregorian_0 | year: -1}
+
+    assert DateTime.to_unix(gregorian_0) == -62167219200
+    assert_raise FunctionClauseError, fn ->
+      DateTime.to_unix(before_gregorian_0)
+    end
+  end
 end

lib/elixir/test/elixir/enum_test.exs

@@ -195,6 +195,8 @@ defmodule EnumTest do
   test "find_index/2" do
     assert Enum.find_index([2, 4, 6], fn(x) -> rem(x, 2) == 1 end) == nil
     assert Enum.find_index([2, 3, 4], fn(x) -> rem(x, 2) == 1 end) == 1
+    assert Stream.take(1..3, 3) |> Enum.find_index(fn _ -> false end) == nil
+    assert Stream.take(1..6, 6) |> Enum.find_index(fn x -> x == 5 end) == 4
   end
 
   test "find_value/2" do

lib/elixir/test/elixir/fixtures/dialyzer/macrocallback.ex

@@ -0,0 +1,11 @@
+defmodule Dialyzer.Macrocallback do
+  @macrocallback required(atom) :: Macro.t
+  @macrocallback optional(atom) :: Macro.t
+  @optional_callbacks [optional: 1]
+end
+
+defmodule Dialyzer.Macrocallback.Impl do
+  @behaviour Dialyzer.Macrocallback
+  defmacro required(var), do: Macro.expand(var, __CALLER__)
+  defmacro optional(var), do: Macro.expand(var, __CALLER__)
+end

lib/elixir/test/elixir/fixtures/dialyzer/protocol_opaque.ex

@@ -0,0 +1,21 @@
+defmodule Dialyzer.ProtocolOpaque do
+  def circus() do
+    Dialyzer.ProtocolOpaque.Entity.speak(Dialyzer.ProtocolOpaque.Duck.new)
+  end
+end
+
+defprotocol Dialyzer.ProtocolOpaque.Entity do
+  def speak(entity)
+end
+
+defmodule Dialyzer.ProtocolOpaque.Duck do
+  @opaque t :: %__MODULE__{}
+  defstruct feathers: :white_and_grey
+
+  @spec new :: t
+  def new(), do: %__MODULE__{}
+
+  defimpl Dialyzer.ProtocolOpaque.Entity do
+    def speak(%Dialyzer.ProtocolOpaque.Duck{}), do: "Quack!"
+  end
+end

lib/elixir/test/elixir/fixtures/dialyzer/struct_update.ex

@@ -0,0 +1,7 @@
+defmodule Dialyzer.StructUpdate do
+  defstruct [:foo]
+
+  def update(%__MODULE__{} = struct) do
+    %__MODULE__{struct | foo: :bar}
+  end
+end

lib/elixir/test/elixir/fixtures/parallel_struct/bar.ex

@@ -1,4 +1,3 @@
 defmodule Bar do
-  defstruct name: ""
-  def foo?(%Foo{}), do: true
+  defstruct name: "", foo: %Foo{}
 end

lib/elixir/test/elixir/fixtures/parallel_struct/undef.ex

@@ -0,0 +1,5 @@
+defmodule Undef do
+  def undef do
+    %__MODULE__{}
+  end
+end

lib/elixir/test/elixir/kernel/dialyzer_test.exs

@@ -16,8 +16,14 @@ defmodule Kernel.DialyzerTest do
       |> Path.join("base_plt")
       |> String.to_charlist()
 
-    # Add a few key elixir modules for types
-    files = Enum.map([Kernel, String, Keyword, Exception], &:code.which/1)
+    # Some OSs (like Windows) do not provide the HOME environment variable.
+    unless System.get_env("HOME") do
+      System.put_env("HOME", System.user_home())
+    end
+
+    # Add a few key elixir modules for types and macro functions
+    mods = [Kernel, String, Keyword, Exception, Macro, Macro.Env, :elixir_env]
+    files = Enum.map(mods, &:code.which/1)
     :dialyzer.run([analysis_type: :plt_build, output_plt: plt,
                    apps: [:erts], files: files])
 
@@ -48,6 +54,7 @@ defmodule Kernel.DialyzerTest do
     {:ok, [outdir: dir, dialyzer: dialyzer]}
   end
 
+  @tag otp19: false
   test "no warnings on valid remote calls", context do
     copy_beam! context, Dialyzer.RemoteCall
     assert_dialyze_no_warnings! context
@@ -63,6 +70,24 @@ defmodule Kernel.DialyzerTest do
     assert_dialyze_no_warnings! context
   end
 
+  test "no warnings on macrocallback", context do
+    copy_beam! context, Dialyzer.Macrocallback
+    copy_beam! context, Dialyzer.Macrocallback.Impl
+    assert_dialyze_no_warnings! context
+  end
+
+  test "no warnings on struct update", context do
+    copy_beam! context, Dialyzer.StructUpdate
+    assert_dialyze_no_warnings! context
+  end
+
+  test "no warnings on protocol calls with opaque types", context do
+    copy_beam! context, Dialyzer.ProtocolOpaque
+    copy_beam! context, Dialyzer.ProtocolOpaque.Entity
+    copy_beam! context, Dialyzer.ProtocolOpaque.Duck
+    assert_dialyze_no_warnings! context
+  end
+
   defp copy_beam!(context, module) do
     name = "#{module}.beam"
     File.cp! Path.join(context[:base_dir], name),

lib/elixir/test/elixir/kernel/fn_test.exs

@@ -18,6 +18,18 @@ defmodule Kernel.FnTest do
     refute (fn ^x -> true; _ -> false end).(1.0)
   end
 
+  test "case function hoisting does not affect anonymous fns" do
+    assert :undefined =
+             (if is_a?(:atom, 0) do
+                user = :defined
+              else
+                (fn() ->
+                  user = :undefined
+                  user
+                end).()
+              end)
+  end
+
   test "capture with access" do
     assert (&(&1[:hello])).([hello: :world]) == :world
   end

lib/elixir/test/elixir/kernel/lexical_tracker_test.exs

@@ -161,6 +161,7 @@ defmodule Kernel.LexicalTrackerTest do
           &R.func/0
           &Remote.func/0
           &Integer.is_even/1
+          %Macro.Env{}
         end
 
         &extract/2
@@ -171,6 +172,8 @@ defmodule Kernel.LexicalTrackerTest do
 
         &is_record/1; def b(a), do: is_record(a)
 
+        %Macro.Env{}
+
         Kernel.LexicalTracker.remote_dispatches(__ENV__.module)
       end |> elem(3)
       """)
@@ -181,14 +184,16 @@ defmodule Kernel.LexicalTrackerTest do
     assert {9, Integer, :is_even, 1} in compile_remote_calls
     assert {15, Record, :is_record, 1} in compile_remote_calls
     assert {18, Integer, :is_even, 1} in compile_remote_calls
-    assert {21, Record, :extract, 2} in compile_remote_calls
-    assert {22, Record, :is_record, 1} in compile_remote_calls
-    assert {23, Remote, :func, 0} in compile_remote_calls
+    assert {19, Macro.Env, :__struct__, 1} in compile_remote_calls
+    assert {22, Record, :extract, 2} in compile_remote_calls
+    assert {23, Record, :is_record, 1} in compile_remote_calls
     assert {24, Remote, :func, 0} in compile_remote_calls
-    assert {25, Integer, :is_even, 1} in compile_remote_calls
-    assert {27, Kernel, :def, 2} in compile_remote_calls
-    assert {27, Record, :is_record, 1} in compile_remote_calls
-    assert {29, Kernel.LexicalTracker, :remote_dispatches, 1} in compile_remote_calls
+    assert {25, Remote, :func, 0} in compile_remote_calls
+    assert {26, Integer, :is_even, 1} in compile_remote_calls
+    assert {28, Kernel, :def, 2} in compile_remote_calls
+    assert {28, Record, :is_record, 1} in compile_remote_calls
+    assert {30, Macro.Env, :__struct__, 1} in compile_remote_calls
+    assert {32, Kernel.LexicalTracker, :remote_dispatches, 1} in compile_remote_calls
 
     runtime_remote_calls = unroll_dispatches(runtime_remote_calls)
     assert {7, Record, :extract, 2} in runtime_remote_calls
@@ -200,7 +205,7 @@ defmodule Kernel.LexicalTrackerTest do
     assert {16, Remote, :func, 0} in runtime_remote_calls
     assert {17, Remote, :func, 0} in runtime_remote_calls
     assert {18, :erlang, :==, 2} in runtime_remote_calls
-    assert {27, :erlang, :is_tuple, 1} in runtime_remote_calls
+    assert {28, :erlang, :is_tuple, 1} in runtime_remote_calls
   end
 
   defp unroll_dispatches(dispatches) do

lib/elixir/test/elixir/kernel/macros_test.exs

@@ -48,7 +48,7 @@ defmodule Kernel.MacrosTest do
     assert_raise UndefinedFunctionError, fn -> x.value end
   end
 
-  test "bang do block" do
+  test "macros with bang and do block have proper precedence" do
     import Kernel.MacrosTest.Nested
     assert (do_identity! do 1 end) == 1
     assert (Kernel.MacrosTest.Nested.do_identity! do 1 end) == 1

lib/elixir/test/elixir/kernel/parallel_compiler_test.exs

@@ -28,6 +28,13 @@ defmodule Kernel.ParallelCompilerTest do
     end
   end
 
+  test "emits struct undefined error when local struct is undefined" do
+    fixtures = [fixture_path("parallel_struct/undef.ex")]
+    assert capture_io(fn ->
+      assert catch_exit(Kernel.ParallelCompiler.files(fixtures)) == {:shutdown, 1}
+    end) =~ "Undef.__struct__/1 is undefined, cannot expand struct Undef"
+  end
+
   test "does not hang on missing dependencies" do
     fixtures = [fixture_path("parallel_compiler/bat.ex")]
     assert capture_io(fn ->

lib/elixir/test/elixir/kernel/raise_test.exs

@@ -283,9 +283,10 @@ defmodule Kernel.RaiseTest do
   end
 
   test "badfun error" do
-    x = :example
+    # Avoid "invalid function call" warning in >= OTP 19
+    x = fn -> :example end
     result = try do
-      x.(2)
+      x.().(2)
     rescue
       x in [BadFunctionError] -> Exception.message(x)
     end

lib/elixir/test/elixir/kernel/typespec_test.exs

@@ -455,6 +455,27 @@ defmodule Kernel.TypespecTest do
            types(module)
   end
 
+  test "@type with module attributes" do
+    module = test_module do
+      @keyword Keyword
+      @type kw :: @keyword.t
+      @type kw(value) :: @keyword.t(value)
+    end
+
+    assert [type: {:kw, {:remote_type, _, [{:atom, _, Keyword}, {:atom, _, :t}, []]}, _},
+            type: {:kw, {:remote_type, _, [{:atom, _, Keyword}, {:atom, _, :t}, [{:var, _, :value}]]}, [{:var, _, :value}]}] =
+      types(module)
+  end
+
+  test "invalid remote @type with module attribute that does not evaluate to a module" do
+    assert_raise CompileError, ~r/\(@foo is "bar"\)/, fn ->
+      test_module do
+        @foo "bar"
+        @type t :: @foo.t
+      end
+    end
+  end
+
   test "defines_type?" do
     test_module do
       @type mytype :: tuple

lib/elixir/test/elixir/kernel/warning_test.exs

@@ -310,6 +310,20 @@ defmodule Kernel.WarningTest do
     purge Sample
   end
 
+  test "unused alias when also import" do
+    assert capture_err(fn ->
+      Code.compile_string """
+      defmodule Sample do
+        alias :lists, as: List
+        import MapSet
+        new()
+      end
+      """
+    end) =~ "unused alias List"
+  after
+    purge Sample
+  end
+
   test "unused inside dynamic module" do
     import List, only: [flatten: 1], warn: false
 
@@ -421,8 +435,8 @@ defmodule Kernel.WarningTest do
     assert capture_err(fn ->
       Code.eval_string ~S"""
       defmodule Sample do
-      def hello(arg \\ 0), do: nil
-      def hello(arg), do: nil
+        def hello(arg \\ 0), do: nil
+        def hello(arg), do: nil
       end
       """
     end) =~ "definitions with multiple clauses and default values require a function head"
@@ -551,7 +565,7 @@ defmodule Kernel.WarningTest do
       """
     end) =~ "undefined behaviour function foo/0 (for behaviour Sample1)"
   after
-    purge [Sample1, Sample2, Sample3]
+    purge [Sample1, Sample2]
   end
 
   test "undefined macro for behaviour" do
@@ -567,7 +581,7 @@ defmodule Kernel.WarningTest do
       """
     end) =~ "undefined behaviour macro foo/0 (for behaviour Sample1)"
   after
-    purge [Sample1, Sample2, Sample3]
+    purge [Sample1, Sample2]
   end
 
   test "undefined behavior" do

lib/elixir/test/elixir/kernel/with_test.exs

@@ -47,6 +47,13 @@ defmodule Kernel.WithTest do
     assert result == :error
   end
 
+  test "does not leak variables to else" do
+    state = 1
+    result = with 1 <- state, state = 2, :ok <- error(), do: state, else: (_ -> state)
+    assert result == 1
+    assert state == 1
+  end
+
   test "errors in with" do
     assert_raise RuntimeError, fn ->
       with({:ok, res} <- oops(), do: res)
@@ -59,6 +66,7 @@ defmodule Kernel.WithTest do
 
   test "else conditions" do
     assert with({:ok, res} <- 41, do: res, else: ({:error, error} -> error; res -> res + 1)) == 42
+    assert with({:ok, res} <- 41, do: res, else: (res when res == 41 -> res + 1; res -> res)) == 42
     assert with({:ok, res} <- 41, do: res, else: (_ -> :error)) == :error
   end
 
@@ -68,6 +76,12 @@ defmodule Kernel.WithTest do
     end
   end
 
+  test "invalid else form" do
+    assert_raise CompileError, "nofile:1: expected -> clauses for else in with",  fn ->
+      Code.eval_quoted(quote do: with(_ <- true, do: :ok, else: :error))
+    end
+  end
+
   defp four() do
     4
   end

lib/elixir/test/elixir/map_test.exs

@@ -147,6 +147,16 @@ defmodule MapTest do
     %^struct{name: "john"} = user
   end
 
+  test "structs when using dynamic modules" do
+    defmodule Module.concat(MapTest, DynamicUser) do
+      defstruct [:name, :age]
+
+      def sample do
+        %__MODULE__{}
+      end
+    end
+  end
+
   test "structs when quoted" do
     assert (quote do
       %User{foo: 1}

lib/elixir/test/elixir/module_test.exs

@@ -245,6 +245,17 @@ defmodule ModuleTest do
     assert ModuleHygiene.test == [1, 2, 3]
   end
 
+  test "ensure function clauses are ordered" do
+    {_, _, binary, _} =
+      defmodule Ordered do
+        def foo(:foo), do: :bar
+        def baz(:baz), do: :bat
+      end
+    atoms = :beam_lib.chunks(binary, [:atoms])
+    assert :erlang.phash2(atoms) == 53987778
+  end
+
+  # TODO: Remove this check once we depend only on 19
   if :erlang.system_info(:otp_release) >= '19' do
     test "create with generated true does not emit warnings" do
       contents =

lib/elixir/test/elixir/protocol_test.exs

@@ -5,18 +5,24 @@ defmodule ProtocolTest do
 
   doctest Protocol
 
-  defprotocol Sample do
-    @type t :: any
-    @doc "Ok"
-    @spec ok(t) :: boolean
-    def ok(term)
-  end
+  {_, _, sample_binary, _} =
+    defprotocol Sample do
+      @type t :: any
+      @doc "Ok"
+      @spec ok(t) :: boolean
+      def ok(term)
+    end
 
-  defprotocol WithAny do
-    @fallback_to_any true
-    @doc "Ok"
-    def ok(term)
-  end
+  @sample_binary sample_binary
+
+  {_, _, with_any_binary, _} =
+    defprotocol WithAny do
+      @fallback_to_any true
+      @doc "Ok"
+      def ok(term)
+    end
+
+  @with_any_binary with_any_binary
 
   defprotocol Derivable do
     def ok(a)
@@ -123,11 +129,11 @@ defmodule ProtocolTest do
   end
 
   test "protocol defines callbacks" do
-    assert get_callbacks(Sample, :ok, 1) ==
-      [{:type, [11], :fun, [{:type, [11], :product, [{:user_type, [11], :t, []}]}, {:type, [11], :boolean, []}]}]
+    assert get_callbacks(@sample_binary, :ok, 1) ==
+      [{:type, 12, :fun, [{:type, 12, :product, [{:user_type, 12, :t, []}]}, {:type, 12, :boolean, []}]}]
 
-    assert get_callbacks(WithAny, :ok, 1) ==
-      [{:type, [18], :fun, [{:type, [18], :product, [{:user_type, [18], :t, []}]}, {:type, [18], :term, []}]}]
+    assert get_callbacks(@with_any_binary, :ok, 1) ==
+      [{:type, 22, :fun, [{:type, 22, :product, [{:user_type, 22, :t, []}]}, {:type, 22, :term, []}]}]
   end
 
   test "protocol defines functions and attributes" do
@@ -182,8 +188,8 @@ defmodule ProtocolTest do
     assert Multi.test(:a) == :a
   end
 
-  defp get_callbacks(module, name, arity) do
-    callbacks = for {:callback, info} <- module.__info__(:attributes), do: hd(info)
+  defp get_callbacks(beam, name, arity) do
+    callbacks = Kernel.Typespec.beam_callbacks(beam)
     List.keyfind(callbacks, {name, arity}, 0) |> elem(1)
   end
 
@@ -308,6 +314,8 @@ defmodule Protocol.ConsolidationTest do
   {:ok, binary} = Protocol.consolidate(Sample, [Any, ImplStruct])
   :code.load_binary(Sample, 'protocol_test.exs', binary)
 
+  @sample_binary binary
+
   # Any should be moved to the end
   :code.purge(WithAny)
   :code.delete(WithAny)
@@ -367,7 +375,7 @@ defmodule Protocol.ConsolidationTest do
   end
 
   test "consolidated keeps callbacks" do
-    callbacks = for {:callback, info} <- Sample.__info__(:attributes), do: hd(info)
+    callbacks = Kernel.Typespec.beam_callbacks(@sample_binary)
     assert callbacks != []
   end
 

lib/elixir/test/elixir/string_test.exs

@@ -66,6 +66,9 @@ defmodule StringTest do
     assert String.split("abé", "", trim: true, parts: 2) == ["a", "bé"]
 
     assert String.split("noël", "") == ["n", "o", "ë", "l", ""]
+
+    assert String.split("x-", "-", parts: 2, trim: true) == ["x"]
+    assert String.split("x-x-", "-", parts: 3, trim: true) == ["x", "x"]
   end
 
   test "split with regex" do
@@ -79,6 +82,14 @@ defmodule StringTest do
     assert String.split("a,b", ~r{\.}) == ["a,b"]
   end
 
+  test "split with compiled pattern" do
+    pattern = :binary.compile_pattern("-")
+
+    assert String.split("x-", pattern) == ["x", ""]
+    assert String.split("x-", pattern, parts: 2, trim: true) == ["x"]
+    assert String.split("x-x-", pattern, parts: 3, trim: true) == ["x", "x"]
+  end
+
   test "splitter" do
     assert String.splitter("a,b,c", ",") |> Enum.to_list == ["a", "b", "c"]
     assert String.splitter("a,b", ".") |> Enum.to_list == ["a,b"]

lib/elixir/test/elixir/system_test.exs

@@ -187,4 +187,16 @@ defmodule SystemTest do
     time = System.monotonic_time(:nanoseconds)
     assert abs(System.convert_time_unit(time, :nanoseconds, :microseconds)) < abs(time)
   end
+
+  test "schedulers/0" do
+    assert System.schedulers >= 1
+  end
+
+  test "schedulers_online/0" do
+    assert System.schedulers_online >= 1
+  end
+
+  test "otp_release/0" do
+    assert is_binary System.otp_release
+  end
 end

lib/elixir/test/elixir/test_helper.exs

@@ -1,4 +1,10 @@
-ExUnit.start [trace: "--trace" in System.argv]
+exclude =
+  case :erlang.system_info(:otp_release) do
+    '19' -> [otp19: false]
+    _    -> []
+  end
+
+ExUnit.start [exclude: exclude, trace: "--trace" in System.argv]
 
 # Beam files compiled on demand
 path = Path.expand("../../tmp/beams", __DIR__)

lib/elixir/test/erlang/string_test.erl

@@ -15,7 +15,6 @@ extract_interpolations(String) ->
       Z
    end.
 
-
 % Interpolations
 
 extract_interpolations_without_interpolation_test() ->

lib/elixir/test/erlang/tokenizer_test.erl

@@ -25,15 +25,16 @@ op_kw_test() ->
   [{atom, {1, 1, 5}, foo}, {dual_op, {1, 5, 6}, '+'}, {atom, {1, 6, 10}, bar}] = tokenize(":foo+:bar").
 
 scientific_test() ->
-  [{number, {1, 1, 7}, 0.1}] = tokenize("1.0e-1").
+  [{number, {1, 1, 7}, 0.1}] = tokenize("1.0e-1"),
+  [{number, {1, 1, 16}, 1.2345678e-7}] = tokenize("1_234.567_8e-10").
 
 hex_bin_octal_test() ->
   [{number, {1, 1, 5}, 255}] = tokenize("0xFF"),
-  [{number, {1, 1, 5}, 255}] = tokenize("0xF_F"),
+  [{number, {1, 1, 6}, 255}] = tokenize("0xF_F"),
   [{number, {1, 1, 5}, 63}] = tokenize("0o77"),
-  [{number, {1, 1, 5}, 63}] = tokenize("0o7_7"),
+  [{number, {1, 1, 6}, 63}] = tokenize("0o7_7"),
   [{number, {1, 1, 5}, 3}] = tokenize("0b11"),
-  [{number, {1, 1, 5}, 3}] = tokenize("0b1_1").
+  [{number, {1, 1, 6}, 3}] = tokenize("0b1_1").
 
 unquoted_atom_test() ->
   [{atom, {1, 1, 3}, '+'}] = tokenize(":+"),

lib/ex_unit/lib/ex_unit.ex

@@ -181,6 +181,9 @@ defmodule ExUnit do
       and print them on test failure. Can be overridden for individual tests via
       `@tag capture_log: false`. Defaults to `false`.
 
+    * `:case_load_timeout` - the timeout to be used when loading a test case.
+      Defaults to `60_000` milliseconds.
+
     * `:colors` - a keyword list of colors to be used by some formatters.
       The only option so far is `[enabled: boolean]` which defaults to `IO.ANSI.enabled?/0`
 

lib/ex_unit/lib/ex_unit/assertions.ex

@@ -379,7 +379,6 @@ defmodule ExUnit.Assertions do
                           <> ExUnit.Assertions.__mailbox__(self()))
         end
 
-      _ = unquote(vars) # Silence warnings
       received
     end
   end
@@ -428,6 +427,14 @@ defmodule ExUnit.Assertions do
     Enum.uniq_by(pins, &elem(&1, 0))
   end
 
+  defp collect_vars_from_pattern({:when, _, [left, right]}) do
+    pattern = collect_vars_from_pattern(left)
+    for {name, _, context} = var <- collect_vars_from_pattern(right),
+      Enum.any?(pattern, &match?({^name, _, ^context}, &1)),
+      into: pattern,
+      do: var
+  end
+
   defp collect_vars_from_pattern(expr) do
     Macro.prewalk(expr, [], fn
       {:::, _, [left, _]}, acc ->

lib/ex_unit/lib/ex_unit/capture_log.ex

@@ -37,7 +37,15 @@ defmodule ExUnit.CaptureLog do
   Returns the binary which is the captured output.
 
   This function mutes the `:console` backend
-  and captures any log messages sent to Logger.
+  and captures any log messages sent to Logger from the calling
+  processes. It is possible to ensure explicit log messages from other
+  processes are captured by waiting for their exit or monitor
+  signal. This does not guarantee to capture VM generated log messages
+  when a process that is spawned using a low level `Kernel` spawn function
+  (e.g. `Kernel.spawn/1`) raises an exception that isn't rescued or a throw
+  that isn't caught. A `Task`, or other OTP process, will send explicit logs
+  that are sent before its exit or monitor signal and will not cause VM
+  generated log messages.
 
   Note that when the `async` is set to `true`,
   the messages from another test might be captured.
@@ -59,6 +67,7 @@ defmodule ExUnit.CaptureLog do
     {:ok, string_io} = StringIO.open("")
 
     try do
+      _ = :gen_event.which_handlers(:error_logger)
       :ok = add_capture(string_io, opts)
       ref = ExUnit.CaptureServer.log_capture_on(self())
 

lib/ex_unit/lib/ex_unit/diff.ex

@@ -367,18 +367,29 @@ defmodule ExUnit.Diff do
       map_pair = format_key_value(key, val, keyword?)
       [[ins: ", ", ins: map_pair] | acc]
     end)
+    result =
+      if same == [] and altered == [] and missing != [] and surplus != [] do
+        [[_ | elem_diff] | rest] = result
+        [elem_diff | rest]
+      else
+        result
+      end
+
     result = Enum.reduce(surplus, result, fn({key, val}, acc) ->
       map_pair = format_key_value(key, val, keyword?)
       [[del: ", ", del: map_pair] | acc]
     end)
+
     result = Enum.reduce(altered, result, fn({key, {val1, val2}}, acc) ->
       value_diff = script_inner(val1, val2)
       [[{:eq, ", "}, {:eq, format_key(key, keyword?)}, value_diff] | acc]
     end)
+
     result = Enum.reduce(same, result, fn({key, val}, acc) ->
       map_pair = format_key_value(key, val, keyword?)
       [[eq: ", ", eq: map_pair] | acc]
     end)
+
     [[_ | elem_diff] | rest] = result
     [{:eq, "%" <> name <> "{"}, [elem_diff | rest], {:eq, "}"}]
   end

lib/ex_unit/lib/ex_unit/server.ex

@@ -1,6 +1,5 @@
 defmodule ExUnit.Server do
   @moduledoc false
-  @timeout 60_000
 
   use GenServer
 
@@ -21,11 +20,13 @@ defmodule ExUnit.Server do
   end
 
   def take_async_cases(count) do
-    GenServer.call(__MODULE__, {:take_async_cases, count}, @timeout)
+    timeout = Application.fetch_env!(:ex_unit, :case_load_timeout)
+    GenServer.call(__MODULE__, {:take_async_cases, count}, timeout)
   end
 
   def take_sync_cases() do
-    GenServer.call(__MODULE__, :take_sync_cases, @timeout)
+    timeout = Application.fetch_env!(:ex_unit, :case_load_timeout)
+    GenServer.call(__MODULE__, :take_sync_cases, timeout)
   end
 
   ## Callbacks

lib/ex_unit/mix.exs

@@ -18,6 +18,7 @@ defmodule ExUnit.Mixfile do
        assert_receive_timeout: 100,
        autorun: true,
        capture_log: false,
+       case_load_timeout: 60_000,
        colors: [],
        exclude: [],
        include: [],

lib/ex_unit/test/ex_unit/assertions_test.exs

@@ -266,6 +266,13 @@ defmodule ExUnit.AssertionsTest do
     :world = world
   end
 
+  test "assert received does not leak external variables used in guards" do
+    send self(), {:hello, :world}
+    guard_world = :world
+    assert_received {:hello, world} when world == guard_world
+    :world = world
+  end
+
   test "refute received does not wait" do
     false = refute_received :hello
   end

lib/ex_unit/test/ex_unit/diff_test.exs

@@ -242,6 +242,8 @@ defmodule ExUnit.DiffTest do
     assert script(map2, map1) == expected
     assert script(map1, %{}) == [{:eq, "%{"}, [[del: "baz: 12"]], {:eq, "}"}]
     assert script(%{}, map1) == [{:eq, "%{"}, [[ins: "baz: 12"]], {:eq, "}"}]
+    expected = [{:eq, "%{"}, [[del: "baz: 12"], [ins: "foo: 12"]], {:eq, "}"}]
+    assert script(map1, %{foo: 12}) == expected
 
     assert script(%{}, %{}) == [eq: "%{}"]
   end

lib/iex/lib/iex/autocomplete.ex

@@ -59,9 +59,12 @@ defmodule IEx.Autocomplete do
   defp reduce(expr) do
     Enum.reduce(' ([{', expr, fn token, acc ->
       hd(:string.tokens(acc, [token]))
-    end) |> Enum.reverse |> strip_ampersand
+    end) |> Enum.reverse |> strip_ampersand |> strip_percent
   end
 
+  defp strip_percent([?% | t]), do: t
+  defp strip_percent(expr), do: expr
+
   defp strip_ampersand([?& | t]), do: t
   defp strip_ampersand(expr), do: expr
 

lib/iex/lib/iex/helpers.ex

@@ -23,8 +23,8 @@ defmodule IEx.Helpers do
   There are many other helpers available:
 
     * `b/1`           - prints callbacks info and docs for a given module
-    * `c/1`           - compiles a file at the current directory
-    * `c/2`           - compiles a file at the given path
+    * `c/1`           - compiles a file into the current directory
+    * `c/2`           - compiles a file to the given path
     * `cd/1`          - changes the current directory
     * `clear/0`       - clears the screen
     * `flush/0`       - flushes all messages sent to the shell
@@ -115,6 +115,7 @@ defmodule IEx.Helpers do
 
       c "baz.ex"
       #=> [Baz]
+
   """
   def c(files, path \\ ".") when is_binary(path) do
     files = List.wrap(files)
@@ -123,10 +124,7 @@ defmodule IEx.Helpers do
       raise ArgumentError, "expected a binary or a list of binaries as argument"
     end
 
-    {found, not_found} =
-      files
-      |> Enum.map(&Path.expand(&1, path))
-      |> Enum.partition(&File.exists?/1)
+    {found, not_found} = Enum.partition(files, &File.exists?/1)
 
     unless Enum.empty?(not_found) do
       raise ArgumentError, "could not find files #{Enum.join(not_found, ", ")}"
@@ -535,6 +533,15 @@ defmodule IEx.Helpers do
     end
   end
 
+  @doc """
+  Similar to `import_file` but only imports it if it is available.
+  """
+  defmacro import_file_if_available(path) do
+    quote do
+      import_file(unquote(path), optional: true)
+    end
+  end
+
   @doc """
   Evaluates the contents of the file at `path` as if it were directly typed into
   the shell.
@@ -587,6 +594,25 @@ defmodule IEx.Helpers do
     raise ArgumentError, "import_file/1 expects a literal binary as its argument"
   end
 
+  @doc """
+  Calls `import/2` with the given arguments, but only if the module is available.
+
+  This lets you put imports in `.iex.exs` files (including `~/.iex.exs`) without
+  getting compile errors if you open a console where the module is not available.
+
+  ## Example
+
+      # In ~/.iex.exs
+      import_if_available Ecto.Query
+  """
+  defmacro import_if_available(quoted_module, opts \\ []) do
+    module = Macro.expand(quoted_module, __CALLER__)
+
+    if Code.ensure_loaded?(module) do
+      quote do: import unquote(quoted_module), unquote(opts)
+    end
+  end
+
   # Compiles and loads an Erlang source file, returns {module, binary}
   defp compile_erlang(source) do
     source = Path.relative_to_cwd(source) |> String.to_charlist

lib/iex/lib/iex/introspection.ex

@@ -108,7 +108,7 @@ defmodule IEx.Introspection do
     if docs = Code.get_docs(mod, :docs) do
       if doc = find_doc(docs, fun, arity) do
         if callback_module = is_nil(elem(doc, 4)) and callback_module(mod, fun, arity) do
-          filter = &match?({^fun, _}, elem(&1, 0))
+          filter = &match?({^fun, ^arity}, elem(&1, 0))
           print_callback_docs(callback_module, filter, &print_doc/2)
         else
           print_doc(doc)
@@ -139,6 +139,8 @@ defmodule IEx.Introspection do
     end)
   end
 
+  defp has_content?(nil),
+    do: false
   defp has_content?({_, _, _, _, false}),
     do: false
   defp has_content?({{name, _}, _, _, _, nil}),
@@ -147,14 +149,11 @@ defmodule IEx.Introspection do
     do: true
 
   defp callback_module(mod, fun, arity) do
+    filter = &match?({{^fun, ^arity}, _}, &1)
     mod.module_info(:attributes)
     |> Keyword.get_values(:behaviour)
     |> Stream.concat()
-    |> Enum.find(fn module ->
-      module.module_info(:attributes)
-      |> Enum.filter(&match?({:callback, _}, &1))
-      |> Enum.any?(&match?({_, [{{^fun, ^arity}, _} | _]}, &1))
-    end)
+    |> Enum.find(&Enum.any?(Typespec.beam_callbacks(&1), filter))
   end
 
   defp print_doc({{fun, _}, _line, kind, args, doc}) do

lib/iex/test/iex/autocomplete_test.exs

@@ -179,4 +179,13 @@ defmodule IEx.AutocompleteTest do
     :code.purge(Sample)
     :code.delete(Sample)
   end
+
+
+  defmodule MyStruct do
+    defstruct my_val: "val"
+  end
+
+  test "completion for structs" do
+    assert expand('%IEx.AutocompleteTest.MyStr') == {:yes, 'uct.', []}
+  end
 end

lib/iex/test/iex/helpers_test.exs

@@ -34,9 +34,11 @@ defmodule IEx.HelpersTest do
   test "h helper function" do
     pwd_h = "* def pwd()\n\nPrints the current working directory.\n\n"
     c_h   = "* def c(files, path \\\\ \".\")\n\nCompiles the given files."
+    eq_h  = "* def ==(left, right)\n\nReturns `true` if the two items are equal.\n\n"
 
     assert capture_io(fn -> h IEx.Helpers.pwd/0 end) =~ pwd_h
     assert capture_io(fn -> h IEx.Helpers.c/2 end) =~ c_h
+    assert capture_io(fn -> h ==/2 end) =~ eq_h
 
     assert capture_io(fn -> h IEx.Helpers.c/1 end) =~ c_h
     assert capture_io(fn -> h pwd end) =~ pwd_h
@@ -77,14 +79,17 @@ defmodule IEx.HelpersTest do
       @doc "Docs for MyBehaviour.first"
       @callback first(integer) :: integer
       @callback second(integer) :: integer
+      @callback second(integer, integer) :: integer
     end
     """
     impl = """
     defmodule Impl do
       @behaviour MyBehaviour
       def first(0), do: 0
-      @doc "Docs for Impl.second"
+      @doc "Docs for Impl.second/1"
       def second(0), do: 0
+      @doc "Docs for Impl.second/2"
+      def second(0, 0), do: 0
     end
     """
     files = ["my_behaviour.ex", "impl.ex"]
@@ -92,10 +97,11 @@ defmodule IEx.HelpersTest do
       assert c(files) |> Enum.sort == [Impl, MyBehaviour]
 
       assert capture_io(fn -> h Impl.first/1 end) == "* @callback first(integer()) :: integer()\n\nDocs for MyBehaviour.first\n"
-      assert capture_io(fn -> h Impl.second/1 end) == "* def second(int)\n\nDocs for Impl.second\n"
+      assert capture_io(fn -> h Impl.second/1 end) == "* def second(int)\n\nDocs for Impl.second/1\n"
+      assert capture_io(fn -> h Impl.second/2 end) == "* def second(int1, int2)\n\nDocs for Impl.second/2\n"
 
       assert capture_io(fn -> h Impl.first end) == "* @callback first(integer()) :: integer()\n\nDocs for MyBehaviour.first\n"
-      assert capture_io(fn -> h Impl.second end) == "* def second(int)\n\nDocs for Impl.second\n"
+      assert capture_io(fn -> h Impl.second end) == "* def second(int)\n\nDocs for Impl.second/1\n* def second(int1, int2)\n\nDocs for Impl.second/2\n"
     end
   after
     cleanup_modules([Impl, MyBehaviour])
@@ -255,11 +261,17 @@ defmodule IEx.HelpersTest do
   end
 
   test "import_file when the file is missing" do
-    assert "nil" == capture_iex("import_file \"nonexistent\", optional: true")
-
     failing = capture_iex("import_file \"nonexistent\"")
     assert "** (File.Error) could not read file" <> _ = failing
     assert failing =~ "no such file or directory"
+
+    assert "nil" == capture_iex("import_file_if_available \"nonexistent\"")
+  end
+
+  test "import_if_available helper" do
+    assert "nil" == capture_iex("import_if_available NoSuchModule")
+    assert "[1, 2, 3]" == capture_iex("import_if_available Integer; digits 123")
+    assert "[1, 2, 3]" == capture_iex("import_if_available Integer, only: [digits: 1]; digits 123")
   end
 
   test "c helper" do

lib/iex/test/test_helper.exs

@@ -70,4 +70,3 @@ defmodule IEx.Case do
     |> String.trim
   end
 end
-

lib/logger/lib/logger/backends/console.ex

@@ -86,7 +86,7 @@ defmodule Logger.Backends.Console do
     colors     = configure_colors(config)
     max_buffer = Keyword.get(config, :max_buffer, 32)
     %{state | format: format, metadata: Enum.reverse(metadata),
-      level: level, colors: colors, device: device, max_buffer: max_buffer}
+              level: level, colors: colors, device: device, max_buffer: max_buffer}
   end
 
   defp configure_merge(env, options) do
@@ -149,7 +149,7 @@ defmodule Logger.Backends.Console do
     %{format: format, metadata: keys, colors: colors} = state
     format
     |> Logger.Formatter.format(level, msg, ts, take_metadata(md, keys))
-    |> color_event(level, colors)
+    |> color_event(level, colors, md)
   end
 
   defp take_metadata(metadata, keys) do
@@ -161,10 +161,11 @@ defmodule Logger.Backends.Console do
     end
   end
 
-  defp color_event(data, _level, %{enabled: false}), do: data
+  defp color_event(data, _level, %{enabled: false}, _md), do: data
 
-  defp color_event(data, level, %{enabled: true} = colors) do
-    [IO.ANSI.format_fragment(Map.fetch!(colors, level), true), data | IO.ANSI.reset]
+  defp color_event(data, level, %{enabled: true} = colors, md) do
+    color = md[:ansi_color] || Map.fetch!(colors, level)
+    [IO.ANSI.format_fragment(color, true), data | IO.ANSI.reset]
   end
 
   defp log_buffer(%{buffer_size: 0, buffer: []} = state), do: state

lib/logger/test/logger/backends/console_test.exs

@@ -109,4 +109,12 @@ defmodule Logger.Backends.ConsoleTest do
       Logger.error("hello")
     end) == IO.ANSI.cyan() <> "hello" <> IO.ANSI.reset()
   end
+
+  test "can use colors from metadata" do
+    Logger.configure_backend(:console, [format: "$message", colors: [enabled: true]])
+
+    assert capture_log(fn ->
+      Logger.log(:error, "hello", ansi_color: :yellow)
+    end) == IO.ANSI.yellow() <> "hello" <> IO.ANSI.reset()
+  end
 end

lib/mix/lib/mix/compilers/elixir.ex

@@ -177,8 +177,6 @@ defmodule Mix.Compilers.Elixir do
   end
 
   defp each_module(pid, cwd, source, module, binary) do
-    beam = Atom.to_string(module) <> ".beam"
-
     {compile_references, runtime_references} = Kernel.LexicalTracker.remote_references(module)
 
     compile_references =
@@ -214,7 +212,7 @@ defmodule Mix.Compilers.Elixir do
         module: module,
         kind: kind,
         source: source,
-        beam: beam,
+        beam: nil, # They are calculated when writing the manifest
         binary: binary
       )
 
@@ -248,10 +246,7 @@ defmodule Mix.Compilers.Elixir do
 
   defp get_external_resources(module, cwd) do
     for file <- Module.get_attribute(module, :external_resource),
-        File.regular?(file),
-        relative = Path.relative_to(file, cwd),
-        Path.type(relative) == :relative,
-        do: relative
+        do: Path.relative_to(file, cwd)
   end
 
   defp each_file(source) do
@@ -265,8 +260,10 @@ defmodule Mix.Compilers.Elixir do
   ## Resolution
 
   defp update_stale_sources(sources, removed, changed) do
+    # Remove delete sources
     sources =
-      Enum.reject(sources, fn source(source: source) -> source in removed end)
+      Enum.reduce(removed, sources, &List.keydelete(&2, &1, source(:source)))
+    # Store empty sources for the changed ones as the compiler appends data
     sources =
       Enum.reduce(changed, sources, &List.keystore(&2, &1, source(:source), source(source: &1)))
     sources
@@ -394,13 +391,14 @@ defmodule Mix.Compilers.Elixir do
     File.mkdir_p!(Path.dirname(manifest))
 
     modules =
-      for module(beam: beam, binary: binary) = module <- modules do
+      for module(binary: binary, module: module) = entry <- modules do
+        beam = Atom.to_string(module) <> ".beam"
         if binary do
           beam_path = Path.join(compile_path, beam)
           File.write!(beam_path, binary)
           File.touch!(beam_path, timestamp)
         end
-        module(module, binary: nil)
+        module(entry, binary: nil, beam: beam)
       end
 
     manifest_data =

lib/mix/lib/mix/compilers/test.ex

@@ -264,9 +264,6 @@ defmodule Mix.Compilers.Test do
 
   defp get_external_resources(module, cwd) do
     for file <- Module.get_attribute(module, :external_resource),
-        File.regular?(file),
-        relative = Path.relative_to(file, cwd),
-        Path.type(relative) == :relative,
-        do: relative
+        do: Path.relative_to(file, cwd)
   end
 end

lib/mix/lib/mix/dep.ex

@@ -64,19 +64,22 @@ defmodule Mix.Dep do
   @doc """
   Returns loaded dependencies from the cache for the current environment.
 
-  Because the dependencies are cached during deps.check, their
+  Because the dependencies are cached during deps.loadpaths, their
   status may be outdated (for example, `:compile` did not
   yet become `:ok`). Therefore it is recommended to not rely
   on their status, also given they haven't been checked
   against the lock.
   """
   def cached do
-    if project = Mix.Project.get do
-      key = {:cached_deps, project}
-      Mix.ProjectStack.read_cache(key) ||
-        Mix.ProjectStack.write_cache(key, loaded(env: Mix.env))
-    else
-      loaded(env: Mix.env)
+    cond do
+      System.get_env("MIX_NO_DEPS") in ~w(1 true) ->
+        []
+      project = Mix.Project.get ->
+        key = {:cached_deps, project}
+        Mix.ProjectStack.read_cache(key) ||
+          Mix.ProjectStack.write_cache(key, loaded(env: Mix.env))
+      true ->
+        loaded(env: Mix.env)
     end
   end
 
@@ -225,16 +228,16 @@ defmodule Mix.Dep do
   def format_status(%Mix.Dep{app: app, status: {:divergedonly, other}} = dep) do
     recommendation =
       if Keyword.has_key?(other.opts, :only) do
-        "Ensure the parent dependency specifies a superset of the child one in"
+        "Ensure you specify at least the same environments in :only in your dep"
       else
-        "Remove the :only restriction from"
+        "Remove the :only restriction from your dep"
       end
 
-    "the dependency #{app}\n" <>
+    "the :only option for dependency #{app}\n" <>
     "#{dep_status(dep)}" <>
-    "\n  does not match the environments calculated for\n" <>
+    "\n  does not match the :only option calculated for\n" <>
     "#{dep_status(other)}" <>
-    "\n  #{recommendation} your dep"
+    "\n  #{recommendation}"
   end
 
   def format_status(%Mix.Dep{app: app, status: {:diverged, other}} = dep) do
@@ -264,7 +267,7 @@ defmodule Mix.Dep do
     do: "the dependency was built with another SCM, run \"#{mix_env_var}mix deps.compile\""
 
   defp dep_status(%Mix.Dep{app: app, requirement: req, manager: manager, opts: opts, from: from}) do
-    opts = Keyword.drop(opts, [:dest, :build, :lock, :manager])
+    opts = Keyword.drop(opts, [:dest, :build, :lock, :manager, :checkout])
     opts = opts ++ (if manager, do: [manager: manager], else: [])
     info = if req, do: {app, req, opts}, else: {app, opts}
     "\n  > In #{Path.relative_to_cwd(from)}:\n    #{inspect info}\n"

lib/mix/lib/mix/dep/converger.ex

@@ -84,7 +84,7 @@ defmodule Mix.Dep.Converger do
     if not diverged? && remote do
       # If there is a lock, it means we are doing a get/update
       # and we need to hit the remote converger which do external
-      # requests and what not. In case of deps.check, deps and so
+      # requests and what not. In case of deps.loadpaths, deps and so
       # on, there is no lock, so we won't hit this branch.
       lock = if lock_given?, do: remote.converge(deps, lock), else: lock
 

lib/mix/lib/mix/scm/git.ex

@@ -25,6 +25,11 @@ defmodule Mix.SCM.Git do
   end
 
   def accepts_options(_app, opts) do
+    opts =
+      opts
+      |> Keyword.put(:checkout, opts[:dest])
+      |> sparse_opts()
+
     cond do
       gh = opts[:github] ->
         opts
@@ -41,7 +46,9 @@ defmodule Mix.SCM.Git do
 
   def checked_out?(opts) do
     # Are we inside a Git repository?
-    File.regular?(Path.join(opts[:dest], ".git/HEAD"))
+    opts[:checkout]
+    |> Path.join(".git/HEAD")
+    |> File.regular?
   end
 
   def lock_status(opts) do
@@ -50,7 +57,7 @@ defmodule Mix.SCM.Git do
 
     cond do
       lock_rev = get_lock_rev(lock, opts) ->
-        File.cd!(opts[:dest], fn ->
+        File.cd!(opts[:checkout], fn ->
           %{origin: origin, rev: rev} = get_rev_info()
           if get_lock_repo(lock) == origin and lock_rev == rev do
             :ok
@@ -76,29 +83,75 @@ defmodule Mix.SCM.Git do
 
   def checkout(opts) do
     assert_git!()
-
-    path     = opts[:dest]
-    location = opts[:git]
-
-    _ = File.rm_rf!(path)
-    git!(~s(clone --no-checkout --progress "#{location}" "#{path}"))
-
-    File.cd! path, fn -> do_checkout(opts) end
+    path = opts[:checkout]
+    File.rm_rf!(path)
+    File.mkdir_p!(path)
+    File.cd!(path, fn ->
+      git!("init --quiet")
+      git!("--git-dir=.git remote add origin \"#{opts[:git]}\"")
+      checkout(path, opts)
+    end)
   end
 
   def update(opts) do
     assert_git!()
+    path = opts[:checkout]
+    File.cd! path, fn -> checkout(path, opts) end
+  end
 
-    File.cd! opts[:dest], fn ->
-      location = opts[:git]
-      update_origin(location)
+  defp checkout(_path, opts) do
+    # Set configuration
+    sparse_toggle(opts)
+    update_origin(opts[:git])
 
-      command = IO.iodata_to_binary(["--git-dir=.git fetch --force",
-                                     progress_switch(git_version()),
-                                     tags_switch(opts[:tag])])
+    # Fetch external data
+    command = IO.iodata_to_binary(["--git-dir=.git fetch --force --quiet",
+                                   progress_switch(git_version()),
+                                   tags_switch(opts[:tag])])
+    git!(command)
 
-      git!(command)
-      do_checkout(opts)
+    # Migrate the git repo
+    rev = get_lock_rev(opts[:lock], opts) || get_opts_rev(opts)
+    git!("--git-dir=.git checkout --quiet #{rev}")
+
+    if opts[:submodules] do
+      git!("--git-dir=.git submodule update --init --recursive")
+    end
+
+    # Get the new repo lock
+    get_lock(opts)
+  end
+
+  defp sparse_opts(opts) do
+    if opts[:sparse] do
+      dest = Path.join(opts[:dest], opts[:sparse])
+      Keyword.put(opts, :dest, dest)
+    else
+      opts
+    end
+  end
+
+  defp sparse_toggle(opts) do
+    cond do
+      sparse = opts[:sparse] ->
+        sparse_check(git_version())
+        git!("--git-dir=.git config core.sparsecheckout true")
+        File.write!(".git/info/sparse-checkout", sparse)
+      File.exists?(".git/info/sparse-checkout") ->
+        File.write!(".git/info/sparse-checkout", "*")
+        git!("--git-dir=.git read-tree -mu HEAD")
+        git!("--git-dir=.git config core.sparsecheckout false")
+        File.rm(".git/info/sparse-checkout")
+      true ->
+        :ok
+    end
+  end
+
+  defp sparse_check(version) do
+    unless {1, 7, 0} <= version do
+      version = version |> Tuple.to_list |> Enum.join(".")
+      Mix.raise "Git >= 1.7.0 is required to use sparse checkout. " <>
+                "You are running version #{version}"
     end
   end
 
@@ -111,24 +164,17 @@ defmodule Mix.SCM.Git do
   ## Helpers
 
   defp validate_git_options(opts) do
-    case Keyword.take(opts, [:branch, :ref, :tag]) do
-      []  -> opts
-      [_] -> opts
-      _   ->
-        Mix.raise "You should specify only one of branch, ref or tag, and only once. " <>
-                  "Error on Git dependency: #{opts[:git]}"
-    end
+    err = "You should specify only one of branch, ref or tag, and only once. " <>
+          "Error on Git dependency: #{opts[:git]}"
+    validate_single_uniq(opts, [:branch, :ref, :tag], err)
   end
 
-  defp do_checkout(opts) do
-    rev = get_lock_rev(opts[:lock], opts) || get_opts_rev(opts)
-    git!("--git-dir=.git checkout --quiet #{rev}")
-
-    if opts[:submodules] do
-      git!("--git-dir=.git submodule update --init --recursive")
+  defp validate_single_uniq(opts, take, error) do
+    case Keyword.take(opts, take) do
+      []  -> opts
+      [_] -> opts
+      _   -> Mix.raise error
     end
-
-    get_lock(opts)
   end
 
   defp get_lock(opts) do
@@ -146,7 +192,8 @@ defmodule Mix.SCM.Git do
   defp get_lock_rev(_, _), do: nil
 
   defp get_lock_opts(opts) do
-    lock_opts = Keyword.take(opts, [:branch, :ref, :tag])
+    lock_opts = Keyword.take(opts, [:branch, :ref, :tag, :sparse])
+
     if opts[:submodules] do
       lock_opts ++ [submodules: true]
     else

lib/mix/lib/mix/task.ex

@@ -275,7 +275,7 @@ defmodule Mix.Task do
     # 2. Otherwise we look for it in dependencies.
     # 3. Finally, we compile the current project in hope it is available.
     module =
-      get_task_or_run(proj, task, fn -> Mix.Task.run("deps.check") end) ||
+      get_task_or_run(proj, task, fn -> Mix.Task.run("deps.loadpaths") end) ||
       get_task_or_run(proj, task, fn -> Mix.Project.compile([]) end) ||
       get!(task)
 

lib/mix/lib/mix/tasks/app.start.ex

@@ -20,7 +20,7 @@ defmodule Mix.Tasks.App.Start do
       applications are started in permanent mode
 
     * `:consolidate_protocols` - when `true`, loads consolidated
-      protocols before start
+      protocols before start. The default value is `true`.
 
     * `:elixir` - matches the current elixir version against the
       given requirement

lib/mix/lib/mix/tasks/app.tree.ex

@@ -56,11 +56,18 @@ defmodule Mix.Tasks.App.Tree do
 
     if opts[:format] == "dot" do
       Mix.Utils.write_dot_graph!("app_tree.dot", "application tree",
-                                 {:normal, app}, callback, opts)
-      Mix.shell.info "Generated \"app_tree.dot\" in current directory.\n" <>
-                     "You can use http://www.graphviz.org/ to open it."
+                                 [{:normal, app}], callback, opts)
+      """
+      Generated "app_tree.dot" in the current directory. To generate a PNG:
+
+         dot -Tpng app_tree.dot -o app_tree.png
+
+      For more options see http://www.graphviz.org/.
+      """
+      |> String.trim_trailing
+      |> Mix.shell.info
     else
-      Mix.Utils.print_tree({:normal, app}, callback, opts)
+      Mix.Utils.print_tree([{:normal, app}], callback, opts)
     end
   end
 

lib/mix/lib/mix/tasks/clean.ex

@@ -47,7 +47,8 @@ defmodule Mix.Tasks.Clean do
 
   # Loadpaths without checks because compilers may be defined in deps.
   defp loadpaths! do
-    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check"]
+    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check", "--no-archives-check"]
     Mix.Task.reenable "loadpaths"
+    Mix.Task.reenable "deps.loadpaths"
   end
 end

lib/mix/lib/mix/tasks/compile.app.ex

@@ -90,14 +90,14 @@ defmodule Mix.Tasks.Compile.App do
         best_guess
       end
 
+      properties = ensure_correct_properties(app, config, properties)
+
       # Ensure we always prepend the standard application dependencies
-      core_apps = [:kernel, :stdlib] ++ language_app(config)
       properties = Keyword.update!(properties, :applications, fn apps ->
-        core_apps ++ apps
+        [:kernel, :stdlib] ++ language_app(config) ++ apps
       end)
 
-      properties = ensure_correct_properties(app, config, properties)
-      contents   = {:application, app, properties}
+      contents = {:application, app, properties}
 
       Mix.Project.ensure_structure()
       File.write!(target, :io_lib.format("~p.", [contents]), [:utf8])

lib/mix/lib/mix/tasks/compile.ex

@@ -11,10 +11,11 @@ defmodule Mix.Tasks.Compile do
   ## Configuration
 
     * `:compilers` - compilers to run, defaults to:
-      `[:yeec, :leex, :erlang, :elixir, :xref, :app]`
+      `[:yecc, :leex, :erlang, :elixir, :xref, :app]`
 
     * `:consolidate_protocols` - when `true`, runs protocol
-      consolidation via the `compile.protocols` task
+      consolidation via the `compile.protocols` task. The default
+      value is `true`.
 
     * `:build_embedded` - when `true`, activates protocol
       consolidation and does not generate symlinks in builds
@@ -93,8 +94,9 @@ defmodule Mix.Tasks.Compile do
 
   # Loadpaths without checks because compilers may be defined in deps.
   defp loadpaths! do
-    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check"]
+    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check", "--no-archives-check"]
     Mix.Task.reenable "loadpaths"
+    Mix.Task.reenable "deps.loadpaths"
   end
 
   defp consolidate_protocols? do

lib/mix/lib/mix/tasks/compile.protocols.ex

@@ -9,7 +9,8 @@ defmodule Mix.Tasks.Compile.Protocols do
 
   This task is automatically invoked whenever the project
   enables `:consolidate_protocols` or `:build_embedded` in
-  its configuration.
+  its configuration. As `:consolidate_protocols` is enabled
+  in the default configuration, this is the default case.
 
   ## Consolidation
 
@@ -44,10 +45,7 @@ defmodule Mix.Tasks.Compile.Protocols do
     output   = Mix.Project.consolidation_path(config)
     manifest = Path.join(output, @manifest)
 
-    protocols_and_impls =
-      unless Mix.Project.umbrella?(config) do
-        protocols_and_impls(config)
-      end
+    protocols_and_impls = protocols_and_impls(config)
 
     cond do
       opts[:force] || Mix.Utils.stale?(Mix.Project.config_files(), [manifest]) ->
@@ -79,10 +77,15 @@ defmodule Mix.Tasks.Compile.Protocols do
                not scm.fetchable?,
                do: opts[:build])
 
-    app = Mix.Project.app_path(config)
+    app =
+      if Mix.Project.umbrella?(config) do
+        []
+      else
+        [Mix.Project.app_path(config)]
+      end
 
     protocols_and_impls =
-      for path <- [app | deps] do
+      for path <- app ++ deps do
         manifest_path = Path.join(path, ".compile.elixir")
         compile_path = Path.join(path, "ebin")
         Mix.Compilers.Elixir.protocols_and_impls(manifest_path, compile_path)
@@ -120,10 +123,24 @@ defmodule Mix.Tasks.Compile.Protocols do
   defp consolidate(protocol, paths, output, opts) do
     impls = Protocol.extract_impls(protocol, paths)
     reload(protocol)
-    {:ok, binary} = Protocol.consolidate(protocol, impls)
-    File.write!(Path.join(output, "#{protocol}.beam"), binary)
-    if opts[:verbose] do
-      Mix.shell.info "Consolidated #{inspect protocol}"
+    case Protocol.consolidate(protocol, impls) do
+      {:ok, binary} ->
+        File.write!(Path.join(output, "#{protocol}.beam"), binary)
+        if opts[:verbose] do
+          Mix.shell.info "Consolidated #{inspect protocol}"
+        end
+
+      # If we remove a dependency and we have implemented one of its
+      # protocols locally, we will mark the protocol as needing to be
+      # reconsolidated when the implementation is removed even though
+      # the protocol no longer exists. Although most times removing a
+      # dependency will trigger a full recompilation, such won't happen
+      # in umbrella apps with shared build.
+      {:error, :no_beam_info} ->
+        remove_consolidated(protocol, output)
+        if opts[:verbose] do
+          Mix.shell.info "Unavailable #{inspect protocol}"
+        end
     end
   end
 

lib/mix/lib/mix/tasks/compile.xref.ex

@@ -36,7 +36,7 @@ defmodule Mix.Tasks.Compile.Xref do
   end
 
   defp run_xref do
-    Mix.Task.run("xref", ["--warnings"])
+    Mix.Task.run("xref", ["warnings"])
   end
 
   defp needs_xref?(opts) do

lib/mix/lib/mix/tasks/deps.compile.ex

@@ -33,7 +33,7 @@ defmodule Mix.Tasks.Deps.Compile do
   import Mix.Dep, only: [loaded: 1, available?: 1, loaded_by_name: 2,
                          make?: 1, mix?: 1]
 
-  @switches [include_children: :boolean]
+  @switches [include_children: :boolean, force: :boolean]
 
   @spec run(OptionParser.argv) :: :ok
   def run(args) do
@@ -44,18 +44,18 @@ defmodule Mix.Tasks.Deps.Compile do
     Mix.Project.get!
 
     case OptionParser.parse(args, switches: @switches) do
-      {_, [], _} ->
+      {opts, [], _} ->
         # Because this command may be invoked explicitly with
         # deps.compile, we simply try to compile any available
         # dependency.
-        compile(Enum.filter(loaded(env: Mix.env), &available?/1))
+        compile(Enum.filter(loaded(env: Mix.env), &available?/1), opts)
       {opts, tail, _} ->
-        compile(loaded_by_name(tail, [env: Mix.env] ++ opts))
+        compile(loaded_by_name(tail, [env: Mix.env] ++ opts), opts)
     end
   end
 
   @doc false
-  def compile(deps) do
+  def compile(deps, options \\ []) do
     shell  = Mix.shell
     config = Mix.Project.deps_config
 
@@ -65,6 +65,8 @@ defmodule Mix.Tasks.Deps.Compile do
       Enum.map(deps, fn %Mix.Dep{app: app, status: status, opts: opts, scm: scm} = dep ->
         check_unavailable!(app, status)
 
+        clean(app, options)
+
         compiled? = cond do
           not is_nil(opts[:compile]) ->
             do_compile dep, config
@@ -93,6 +95,12 @@ defmodule Mix.Tasks.Deps.Compile do
     if true in compiled, do: Mix.Dep.Lock.touch_manifest, else: :ok
   end
 
+  defp clean(app, opts) do
+    if Keyword.get(opts, :force, false) do
+      File.rm_rf! Path.join [Mix.Project.build_path, "lib", Atom.to_string(app)]
+    end
+  end
+
   defp touch_fetchable(scm, path) do
     if scm.fetchable? do
       File.mkdir_p!(path)
@@ -125,7 +133,8 @@ defmodule Mix.Tasks.Deps.Compile do
       end
 
       try do
-        res = Mix.Task.run("compile", ["--no-deps", "--no-elixir-version-check"])
+        res = Mix.Task.run("compile", ["--no-deps", "--no-archives-check",
+                                       "--no-elixir-version-check", "--no-warnings-as-errors"])
         :ok in List.wrap(res)
       catch
         kind, reason ->
@@ -176,13 +185,14 @@ defmodule Mix.Tasks.Deps.Compile do
       Mix.raise "\"#{manager}\" installation failed"
   end
 
-  defp do_make(dep, config) do
-    command = if match?({:win32, _}, :os.type) and File.regular?("Makefile.win") do
-      "nmake /F Makefile.win"
-    else
-      "make"
-    end
-    do_command(dep, config, command, true)
+  defp do_make(%{opts: opts} = dep, config) do
+    command =
+      if match?({:win32, _}, :os.type) and File.regular?(Path.join(opts[:dest], "Makefile.win")) do
+        "nmake /F Makefile.win"
+      else
+        "make"
+      end
+    do_command(dep, config, command, true, [{"IS_DEP", "1"}])
   end
 
   defp do_compile(%Mix.Dep{opts: opts} = dep, config) do

lib/mix/lib/mix/tasks/deps.loadpaths.ex

@@ -1,32 +1,26 @@
-defmodule Mix.Tasks.Deps.Check do
+defmodule Mix.Tasks.Deps.Loadpaths do
   use Mix.Task
 
   import Mix.Dep, only: [loaded_by_name: 2, format_dep: 1, ok?: 1,
                          format_status: 1, check_lock: 1]
 
   @moduledoc """
-  Checks if all dependencies are valid, loading them along
-  the way.
+  Checks and loads all dependencies along the way.
 
   If there is an invalid dependency, its status is printed
   before aborting.
 
-  This task is not shown in `mix help` but it is part
-  of the `mix` public API and can be depended on.
+  Although this task does not show up in `mix help`, it is
+  part of Mix public API and can be depended on.
 
   ## Command line options
 
-    * `--no-archives-check` - do not check archives
-    * `--no-deps-check` - do not check deps
+    * `--no-deps-check` - do not check or compile deps, only load available ones
     * `--no-compile` - do not compile dependencies
 
   """
   @spec run(OptionParser.argv) :: :ok
   def run(args) do
-    unless "--no-archives-check" in args do
-      Mix.Task.run "archive.check", args
-    end
-
     all = Enum.map(Mix.Dep.cached(), &check_lock/1)
 
     unless "--no-deps-check" in args do
@@ -39,7 +33,9 @@ defmodule Mix.Tasks.Deps.Check do
         path
       end
 
-    prune_deps(load_paths, "--no-deps-check" in args)
+    unless System.get_env("MIX_NO_DEPS") in ~w(1 true) do
+      prune_deps(load_paths, "--no-deps-check" in args)
+    end
   end
 
   # If the build is per environment, we should be able to look
@@ -95,11 +91,16 @@ defmodule Mix.Tasks.Deps.Check do
 
   defp partition([dep | deps], not_ok, compile) do
     cond do
-      from_umbrella?(dep)      -> partition(deps, not_ok, compile)
-      compilable?(dep)         -> partition(deps, not_ok, [dep | compile])
-      ok?(dep) and local?(dep) -> partition(deps, not_ok, [dep | compile])
-      ok?(dep)                 -> partition(deps, not_ok, compile)
-      true                     -> partition(deps, [dep | not_ok], compile)
+      compilable?(dep) ->
+        if from_umbrella?(dep) do
+          partition(deps, not_ok, compile)
+        else
+          partition(deps, not_ok, [dep | compile])
+        end
+      ok?(dep) ->
+        partition(deps, not_ok, compile)
+      true ->
+        partition(deps, [dep | not_ok], compile)
     end
   end
 
@@ -123,7 +124,7 @@ defmodule Mix.Tasks.Deps.Check do
   defp compilable?(%Mix.Dep{status: {:noappfile, _}}), do: true
   defp compilable?(%Mix.Dep{status: {:scmlock, _}}), do: true
   defp compilable?(%Mix.Dep{status: :compile}), do: true
-  defp compilable?(%Mix.Dep{}), do: false
+  defp compilable?(%Mix.Dep{} = dep), do: ok?(dep) and local?(dep)
 
   defp show_not_ok!([]) do
     :ok

lib/mix/lib/mix/tasks/deps.tree.ex

@@ -51,12 +51,19 @@ defmodule Mix.Tasks.Deps.Tree do
 
     if opts[:format] == "dot" do
       callback = callback(&format_dot/1, deps, opts)
-      Mix.Utils.write_dot_graph!("deps_tree.dot", "dependency tree", root, callback, opts)
-      Mix.shell.info "Generated \"deps_tree.dot\" in current directory.\n" <>
-                     "You can use http://www.graphviz.org/ to open it."
+      Mix.Utils.write_dot_graph!("deps_tree.dot", "dependency tree", [root], callback, opts)
+      """
+      Generated "deps_tree.dot" in the current directory. To generate a PNG:
+
+          dot -Tpng deps_tree.dot -o deps_tree.png
+
+      For more options see http://www.graphviz.org/.
+      """
+      |> String.trim_trailing
+      |> Mix.shell.info
     else
       callback = callback(&format_tree/1, deps, opts)
-      Mix.Utils.print_tree(root, callback, opts)
+      Mix.Utils.print_tree([root], callback, opts)
     end
   end
 

lib/mix/lib/mix/tasks/deps.unlock.ex

@@ -16,7 +16,7 @@ defmodule Mix.Tasks.Deps.Unlock do
 
   """
 
-  @switches [all: :boolean, unused: :boolean]
+  @switches [all: :boolean, unused: :boolean, filter: :string]
 
   @spec run(OptionParser.argv) :: :ok
   def run(args) do
@@ -29,6 +29,27 @@ defmodule Mix.Tasks.Deps.Unlock do
       opts[:unused] ->
         apps = Mix.Dep.loaded([]) |> Enum.map(& &1.app)
         Mix.Dep.Lock.read() |> Map.take(apps) |> Mix.Dep.Lock.write()
+      filter = opts[:filter] ->
+        lock = Mix.Dep.Lock.read
+        apps = Map.keys(lock)
+
+        unlock =
+          apps
+          |> Enum.filter(&String.contains?("#{&1}", filter))
+
+        if unlock == [] do
+          Mix.shell.error "warning: no dependencies were matched"
+        else
+          lock =
+            Enum.reject(lock, fn({app, _}) ->
+              app in unlock
+            end)
+          Mix.Dep.Lock.write(lock)
+          Mix.shell.info """
+          Unlocked deps:
+          * #{Enum.join(unlock, "\n* ")}
+          """
+        end
 
       apps != [] ->
         lock =

lib/mix/lib/mix/tasks/do.ex

@@ -6,6 +6,8 @@ defmodule Mix.Tasks.Do do
   @moduledoc """
   Executes the tasks separated by comma.
 
+  The comma should be followed by a space.
+
   ## Examples
 
   The example below prints the available compilers and
@@ -24,27 +26,21 @@ defmodule Mix.Tasks.Do do
 
   @doc false
   def gather_commands(args) do
-    gather_commands(args, [], [])
+    gather_commands args, [], []
   end
 
-  defp gather_commands([], current, commands) do
-    [current | commands]
-    |> Enum.reject(&(&1 == []))
-    |> Enum.map(&Enum.reverse(&1))
-    |> Enum.reverse
+  def gather_commands([head | rest], current, acc)
+      when binary_part(head, byte_size(head), -1) == "," do
+    part    = binary_part(head, 0, byte_size(head) - 1)
+    current = Enum.reverse([part | current])
+    gather_commands rest, [], [current | acc]
   end
 
-  defp gather_commands([arg | rest], current, commands) do
-    case String.split(arg, ",", parts: 2) do
-      [arg] ->
-        gather_commands(rest, [arg | current], commands)
-      [left, right] ->
-        rest    = append_unless_empty(right, rest)
-        current = append_unless_empty(left, current)
-        gather_commands(rest, [], [current | commands])
-    end
+  def gather_commands([head | rest], current, acc) do
+    gather_commands rest, [head | current], acc
   end
 
-  defp append_unless_empty("", list), do: list
-  defp append_unless_empty(h, list),  do: [h | list]
+  def gather_commands([], current, acc) do
+    Enum.reverse [Enum.reverse(current) | acc]
+  end
 end

lib/mix/lib/mix/tasks/escript.ex

@@ -1,7 +1,7 @@
 defmodule Mix.Tasks.Escript do
   use Mix.Task
 
-  @shortdoc "List installed escripts."
+  @shortdoc "Lists installed escripts"
 
   @moduledoc ~S"""
   Lists all installed escripts.

lib/mix/lib/mix/tasks/help.ex

@@ -105,8 +105,9 @@ defmodule Mix.Tasks.Help do
 
   # Loadpaths without checks because tasks may be defined in deps.
   defp loadpaths! do
-    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check"]
+    Mix.Task.run "loadpaths", ["--no-elixir-version-check", "--no-deps-check", "--no-archives-check"]
     Mix.Task.reenable "loadpaths"
+    Mix.Task.reenable "deps.loadpaths"
   end
 
   defp load_tasks() do

lib/mix/lib/mix/tasks/loadpaths.ex

@@ -12,12 +12,13 @@ defmodule Mix.Tasks.Loadpaths do
   ## Command line options
 
     * `--no-archives-check` - do not check archive
-    * `--no-deps-check` - do not check dependencies (also implies --no-archives-check)
+    * `--no-deps-check` - do not check dependencies
     * `--no-elixir-version-check` - do not check Elixir version
 
   """
 
   @spec run(OptionParser.argv) :: :ok
+
   def run(args) do
     config = Mix.Project.config
 
@@ -25,10 +26,14 @@ defmodule Mix.Tasks.Loadpaths do
       check_elixir_version(config, args)
     end
 
-    # --no-deps is used only internally. It has not purpose
-    # from Mix.CLI because the CLI itself already loads deps.
+    unless "--no-archives-check" in args do
+      Mix.Task.run "archive.check", args
+    end
+
+    # --no-deps is used only internally. It has no purpose
+    # from Mix.CLI because running a task may load deps.
     unless "--no-deps" in args do
-      Mix.Task.run "deps.check", args
+      Mix.Task.run "deps.loadpaths", args
     end
 
     if config[:app] do

lib/mix/lib/mix/tasks/new.ex

@@ -251,7 +251,7 @@ defmodule Mix.Tasks.New do
        elixir: "~> <%= @version %>",
        build_embedded: Mix.env == :prod,
        start_permanent: Mix.env == :prod,
-       deps: deps]
+       deps: deps()]
     end
 
     # Configuration for the OTP application

lib/mix/lib/mix/tasks/test.ex

@@ -3,7 +3,7 @@ defmodule Mix.Tasks.Test do
     @moduledoc false
 
     def start(compile_path, opts) do
-      Mix.shell.info "Cover compiling modules ... "
+      Mix.shell.info "Cover compiling modules ..."
       _ = :cover.start
 
       case :cover.compile_beam_directory(compile_path |> to_charlist) do
@@ -16,7 +16,7 @@ defmodule Mix.Tasks.Test do
       output = opts[:output]
 
       fn() ->
-        Mix.shell.info "\nGenerating cover results ... "
+        Mix.shell.info "\nGenerating cover results ..."
         File.mkdir_p!(output)
         Enum.each :cover.modules, fn(mod) ->
           {:ok, _} = :cover.analyse_to_file(mod, '#{output}/#{mod}.html', [:html])
@@ -67,6 +67,9 @@ defmodule Mix.Tasks.Test do
     * `--no-elixir-version-check` - do not check the Elixir version from mix.exs
     * `--stale` - run only tests which reference modules that changed since the
       last `test --stale`. You can read more about this option in the "Stale" section below.
+    * `--listen-on-stdin` - run tests, and then listen on stdin. Receiving a newline will
+      result in the tests being run again. Very useful when combined with `--stale` and
+      external commands which produce output on stdout upon file system modification.
 
   ## Filters
 
@@ -164,7 +167,7 @@ defmodule Mix.Tasks.Test do
              exclude: :keep, seed: :integer, only: :keep, compile: :boolean,
              start: :boolean, timeout: :integer, raise: :boolean,
              deps_check: :boolean, archives_check: :boolean, elixir_version_check: :boolean,
-             stale: :boolean]
+             stale: :boolean, listen_on_stdin: :boolean]
 
   @cover [output: "cover", tool: Cover]
 
@@ -172,6 +175,15 @@ defmodule Mix.Tasks.Test do
   def run(args) do
     {opts, files} = OptionParser.parse!(args, strict: @switches)
 
+    if opts[:listen_on_stdin] do
+      System.at_exit fn _ ->
+        IO.gets(:stdio, "")
+        Mix.shell.info "Restarting..."
+        :init.restart()
+        :timer.sleep(:infinity)
+      end
+    end
+
     unless System.get_env("MIX_ENV") || Mix.env == :test do
       Mix.raise "\"mix test\" is running on environment \"#{Mix.env}\". If you are " <>
                                 "running tests along another task, please set MIX_ENV explicitly"

lib/mix/lib/mix/tasks/xref.ex

@@ -2,7 +2,7 @@ defmodule Mix.Tasks.Xref do
   use Mix.Task
 
   alias Mix.Tasks.Compile.Elixir, as: E
-  import Mix.Compilers.Elixir, only: [read_manifest: 2, source: 1, source: 2]
+  import Mix.Compilers.Elixir, only: [read_manifest: 2, source: 1, source: 2, module: 1]
 
   @shortdoc "Performs cross reference checks"
   @recursive true
@@ -12,23 +12,82 @@ defmodule Mix.Tasks.Xref do
 
   ## Xref modes
 
-  The following options control the information xref can emit.
+  The `xref` task expects a mode as first argument:
 
-    * `--warnings` - prints warnings for violated cross reference checks
-    * `--unreachable` - prints all unreachable "file:line: module.function/arity" entries
-    * `--callers` - prints all references of given `Module`, `Module.function`, or
-      `Module.function/arity`
+      mix xref MODE
 
-  ## Command line options
+  All available modes are discussed below.
+
+  ### warnings
+
+  Prints warnings for violated cross reference checks:
+
+      mix xref warnings
+
+  This is the mode used by Mix during compilation.
+
+  ### unreachable
+
+  Prints all unreachable "file:line: module.function/arity" entries:
+
+      mix xref unreachable
+
+  The "file:line" represents the file and line a call to an unknown
+  "module.function/arity" is made.
+
+  ### callers CALLEE
+
+  Prints all callers of the given `CALLEE`, which can be one of: `Module`,
+  `Module.function`, or `Module.function/arity`. Examples:
+
+      mix xref callers MyMod
+      mix xref callers MyMod.fun
+      mix xref callers MyMod.fun/3
+
+  ### graph
+
+  Prints a file dependency graph where an edge from `A` to `B` indicates
+  that `A` depends on `B`.
+
+      mix xref graph --format dot
+
+  The following options are accepted:
+
+    * `--exclude` - paths to exclude
+
+    * `--source` - display all files that the given source file references (directly or indirectly)
+
+    * `--sink` - display all files that reference the given file (directly or indirectly)
+
+    * `--format` - can be set to one of:
+
+      * `pretty` - use Unicode codepoints for formatting the graph.
+        This is the default except on Windows
+
+      * `plain` - do not use Unicode codepoints for formatting the graph.
+        This is the default on Windows
+
+      * `dot` - produces a DOT graph description in `xref_graph.dot` in the
+        current directory. Warning: this will override any previously generated file
+
+  The `--source` and `--sink` options are particularly useful when trying to understand how
+  the modules in a particular file interact with the whole system.
+
+  ## Shared options
+
+  Those options are shared across all modes:
 
     * `--no-compile` - do not compile even if files require compilation
+
     * `--no-deps-check` - do not check dependencies
+
     * `--no-archives-check` - do not check archives
+
     * `--no-elixir-version-check` - do not check the Elixir version from mix.exs
 
   ## Configuration
 
-    All configuration for Xref should be placed under the key `:xref`.
+  All configuration for Xref should be placed under the key `:xref`.
 
     * `:exclude` - a list of modules and `{module, function, arity}` tuples to ignore when checking
       cross references. For example: `[MissingModule, {MissingModule2, :missing_func, 2}]`
@@ -36,31 +95,34 @@ defmodule Mix.Tasks.Xref do
   """
 
   @switches [compile: :boolean, deps_check: :boolean, archives_check: :boolean,
-             warnings: :boolean, unreachable: :boolean, elixir_version_check: :boolean,
-             callers: :string]
+             elixir_version_check: :boolean, exclude: :keep, format: :string,
+             source: :string, sink: :string]
 
   @doc """
   Runs this task.
   """
   @spec run(OptionParser.argv) :: :ok | :error
   def run(args) do
-    {opts, _} =
+    {opts, args} =
       OptionParser.parse!(args, strict: @switches)
 
+    Mix.Task.run("loadpaths")
+
     if Keyword.get(opts, :compile, true) do
       Mix.Task.run("compile")
     end
 
-    modes = [:warnings, :unreachable, :callers]
-    case Keyword.take(opts, modes) do
-      [warnings: true] ->
+    case args do
+      ["warnings"] ->
         warnings()
-      [unreachable: true] ->
+      ["unreachable"] ->
         unreachable()
-      [callers: callee] ->
+      ["callers", callee] ->
         callers(callee)
+      ["graph"] ->
+        graph(opts)
       _ ->
-        Mix.raise "xref expects exactly one of the following modes: --warnings, --unreachable, --callers"
+        Mix.raise "xref doesn't support this command, see mix help xref for more information"
     end
   end
 
@@ -90,11 +152,16 @@ defmodule Mix.Tasks.Xref do
     :ok
   end
 
+  defp graph(opts) do
+    write_graph(file_references(), excluded(opts), opts)
+
+    :ok
+  end
+
   ## Unreachable
 
   defp unreachable(pair_fun) do
     excludes = excludes()
-
     each_source_entries(&source_warnings(&1, excludes), pair_fun)
   end
 
@@ -102,20 +169,38 @@ defmodule Mix.Tasks.Xref do
     source(runtime_dispatches: runtime_dispatches) = source
 
     for {module, func_arity_lines} <- runtime_dispatches,
+        exports = load_exports(module),
         {{func, arity}, lines} <- func_arity_lines,
-        warning = unreachable_mfa(module, func, arity, lines, excludes),
+        warning = unreachable_mfa(exports, module, func, arity, lines, excludes),
         do: warning
   end
 
-  defp unreachable_mfa(module, func, arity, lines, excludes) do
+  defp load_exports(module) do
+    if :code.is_loaded(module) do
+      # If the module is loaded, we will use the faster function_exported?/3 check
+      module
+    else
+      # Otherwise we get all exports from :beam_lib to avoid loading modules
+      with file when is_list(file) <- :code.which(module),
+           {:ok, {^module, [exports: exports]}} <- :beam_lib.chunks(file, [:exports]) do
+        exports
+      else
+        _ -> :unknown_module
+      end
+    end
+  end
+
+  defp unreachable_mfa(exports, module, func, arity, lines, excludes) do
     cond do
       excluded?(module, func, arity, excludes) ->
         nil
       skip?(module, func, arity) ->
         nil
-      not Code.ensure_loaded?(module) ->
+      exports == :unknown_module ->
         {Enum.sort(lines), :unknown_module, module, func, arity}
-      not function_exported?(module, func, arity) ->
+      is_atom(exports) and not function_exported?(module, func, arity) ->
+        {Enum.sort(lines), :unknown_function, module, func, arity}
+      is_list(exports) and not {func, arity} in exports ->
         {Enum.sort(lines), :unknown_function, module, func, arity}
       true ->
         nil
@@ -210,7 +295,7 @@ defmodule Mix.Tasks.Xref do
   defp source_calls_for_filter(source, filter) do
     runtime_dispatches = source(source, :runtime_dispatches)
     compile_dispatches = source(source, :compile_dispatches)
-    dispatches = Stream.concat(runtime_dispatches, compile_dispatches)
+    dispatches = runtime_dispatches ++ compile_dispatches
 
     calls =
       for {module, func_arity_lines} <- dispatches,
@@ -279,12 +364,130 @@ defmodule Mix.Tasks.Xref do
 
   defp raise_invalid_callee(callee) do
     message =
-      "xref --callers expects Module, Module.function, or Module.function/arity, got: " <>
+      "xref callers CALLEE expects Module, Module.function, or Module.function/arity, got: " <>
       callee
 
     Mix.raise message
   end
 
+  ## Graph helpers
+
+  defp excluded(opts) do
+    Keyword.get_values(opts, :exclude)
+    |> Enum.flat_map(&[{&1, nil}, {&1, "(compile)"}, {&1, "(runtime)"}])
+  end
+
+  defp file_references() do
+    module_sources =
+      for manifest <- E.manifests(),
+          manifest_data = read_manifest(manifest, ""),
+          module(module: module, source: source) <- manifest_data,
+          source = Enum.find(manifest_data, &match?(source(source: ^source), &1)),
+          do: {module, source},
+          into: %{}
+
+    all_modules = MapSet.new(module_sources, &elem(&1, 0))
+
+    Map.new module_sources, fn {module, source} ->
+      source(runtime_references: runtime, compile_references: compile, source: file) = source
+      compile_references =
+        compile
+        |> MapSet.new()
+        |> MapSet.delete(module)
+        |> MapSet.intersection(all_modules)
+        |> Enum.filter(&module_sources[&1] != source)
+        |> Enum.map(&{source(module_sources[&1], :source), "(compile)"})
+
+      runtime_references =
+        runtime
+        |> MapSet.new()
+        |> MapSet.delete(module)
+        |> MapSet.intersection(all_modules)
+        |> Enum.filter(&module_sources[&1] != source)
+        |> Enum.map(&{source(module_sources[&1], :source), nil})
+
+      {file, compile_references ++ runtime_references}
+    end
+  end
+
+  defp write_graph(file_references, excluded, opts) do
+    {root, file_references} =
+      case {opts[:source], opts[:sink]} do
+        {nil, nil} ->
+          {Enum.map(file_references, &{elem(&1, 0), nil}) -- excluded, file_references}
+
+        {source, nil} ->
+          if file_references[source] do
+            {[{source, nil}], file_references}
+          else
+            Mix.raise "Source could not be found: #{source}"
+          end
+
+        {nil, sink} ->
+          if file_references[sink] do
+            file_references = filter_for_sink(file_references, sink)
+            roots =
+              file_references
+              |> Map.delete(sink)
+              |> Enum.map(&{elem(&1, 0), nil})
+            {roots -- excluded, file_references}
+          else
+            Mix.raise "Sink could not be found: #{sink}"
+          end
+
+        {_, _} ->
+          Mix.raise "mix xref graph expects only one of --source and --sink"
+      end
+
+    callback =
+      fn {file, type} ->
+        children = Map.get(file_references, file, [])
+        {{file, type}, children -- excluded}
+      end
+
+    if opts[:format] == "dot" do
+      Mix.Utils.write_dot_graph!("xref_graph.dot", "xref graph",
+                                 root, callback, opts)
+      """
+      Generated "xref_graph.dot" in the current directory. To generate a PNG:
+
+         dot -Tpng xref_graph.dot -o xref_graph.png
+
+      For more options see http://www.graphviz.org/.
+      """
+      |> String.trim_trailing()
+      |> Mix.shell.info()
+    else
+      Mix.Utils.print_tree(root, callback, opts)
+    end
+  end
+
+  defp filter_for_sink(file_references, sink) do
+    file_references
+    |> invert_references()
+    |> do_filter_for_sink([{sink, nil}], %{})
+    |> invert_references()
+  end
+
+  defp do_filter_for_sink(file_references, new_nodes, acc) do
+    Enum.reduce new_nodes, acc, fn {new_node_name, _type}, acc ->
+      new_nodes = file_references[new_node_name]
+      if acc[new_node_name] || !new_nodes do
+        acc
+      else
+        do_filter_for_sink(file_references, new_nodes, Map.put(acc, new_node_name, new_nodes))
+      end
+    end
+  end
+
+  defp invert_references(file_references) do
+    Enum.reduce file_references, %{}, fn {file, references}, acc ->
+      Enum.reduce references, acc, fn {reference, type}, acc ->
+        Map.update(acc, reference, [{file, type}], &[{file, type} | &1])
+      end
+    end
+  end
+
   ## Helpers
 
   defp each_source_entries(entries_fun, pair_fun) do