Release v1.9.4

Closes #9343.

CHANGELOG.md

@@ -58,7 +58,116 @@ Another important change related to configuration is that `mix new` will no long
 
 There are many other enhancements. The Elixir CLI got a handful of new options in order to best support releases. `Logger` now computes its sync/async/discard thresholds in a decentralized fashion, reducing contention. `EEx` templates support more complex expressions than before. Finally, there is a new `~U` sigil for working with UTC DateTimes as well as new functions in the `File`, `Registry`, and `System` modules.
 
-## v1.9.0-dev
+## v1.9.4 (2019-11-05)
+
+### 1. Bug fixes
+
+#### Mix
+
+  * [mix local.hex] Remove invalid deprecation warning on `mix local.hex` command
+
+## v1.9.3 (2019-11-05)
+
+Note this release deprecates the use of URLs on `mix archive.install`, `mix escript.install`, and `mix local.rebar`. Support for passing URLs to said commands will be fully removed on Elixir v1.10, as they are unsafe. Thanks to Bram Verburg for the report and for providing a fix.
+
+The alternative is straight-forward: you can simply download the artifact via the command line and then invoke the command with a file system path. For example, instead of:
+
+    $ mix archive.install https://example.org/installer.ez
+
+You can execute on Unix (Linux, MacOS X):
+
+    $ wget https://example.org/installer.ez
+    $ mix archive.install installer.ez
+
+or
+
+    $ curl -o installer.ez https://example.org/installer.ez
+    $ mix archive.install installer.ez
+
+On Windows (Win7 or later):
+
+    > powershell -Command "Invoke-WebRequest https://example.org/installer.ez -OutFile installer.ez"
+    > mix archive.install installer.ez
+
+or
+
+    > powershell -Command "(New-Object Net.WebClient).DownloadFile('https://example.org/installer.ez', 'installer.ez')"
+    > mix archive.install installer.ez
+
+Note that, if you are a library author, consider providing installable escripts and archives through Hex, such as Phoenix:
+
+    $ mix archive.install hex phx_new
+
+Installations through Hex are always safe and they come with version management and all other benefits from Hex too.
+
+### 1. Enhancements
+
+#### Mix
+
+  * [mix release] Add :tar option for releases to create a tarball
+
+### 2. Bug fixes
+
+#### Mix
+
+  * [mix release] Use `default_release` option when name is not given
+  * [mix release] Make release's boot script contents deterministic
+
+### 3. Deprecations
+
+#### Mix
+
+  * [mix archive.install] Warn when installing from URI
+  * [mix escript.install] Warn when installing from URI
+  * [mix local.rebar] Warn when installing from URI
+
+## v1.9.2 (2019-10-12)
+
+### 1. Enhancements
+
+#### Mix
+
+  * [mix release] Allow `{:from_app, app_name}` as a version for releases
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [Kernel] Ensure compilation works for a variable named `super`
+  * [Kernel] Ensure capture operator of a local function expands correctly inside a macro
+  * [Regex] Ensure dynamic recompilation of regexes considers options. This fixes an issue where parsing the protocol in `URI.parse/1` seemingly looked case sensitive when running Elixir precompiled on another machine
+
+#### Mix
+
+  * [mix release] Use `Base.encode32` when generating cookie to avoid unsafe chars
+  * [mix release] Fix `install` command on Windows
+  * [mix release] Quote executable path on Windows to ensure it works on directories with spaces
+
+## v1.9.1 (2019-07-18)
+
+### 1. Enhancements
+
+#### Mix
+
+  * [mix format] Print relative paths in `--check-formatted` output
+  * [mix release] Support included applications
+
+### 2. Bug fixes
+
+#### Elixir
+
+  * [Code] Fix formatter wrongly removing nested parens in nested calls
+
+#### Logger
+
+  * [Logger] Do not crash translator on poorly formatted supervisor names
+
+#### Mix
+
+  * [mix compile] Raise readable error for mismatched sources during compilation
+  * [mix release] Preserve UTF8 encoding in release config files
+
+## v1.9.0 (2019-06-24)
 
 ### 1. Enhancements
 
@@ -110,6 +219,7 @@ There are many other enhancements. The Elixir CLI got a handful of new options i
 
   * [Mix] Follow XDG base dir specification in Mix for temporary and configuration files
   * [Mix.Generator] Add `copy_file/3`, `copy_template/4`, and `overwite?/2`
+  * [Mix.Project] Add `preferred_cli_target` that works like `preferred_cli_env`
   * [mix archive.uninstall] Allow `mix archive.uninstall APP` to uninstall any installed version of APP
   * [mix new] No longer generate a `config/` directory for mix new
   * [mix release] Add support for releases
@@ -125,6 +235,7 @@ There are many other enhancements. The Elixir CLI got a handful of new options i
 #### Elixir
 
   * [Code] Quote `::` in `Code.format_string!/1` to avoid ambiguity
+  * [Code] Do not crash formatter on false positive sigils
   * [Enum] Ensure the first equal entry is returned by `Enum.min/2` and `Enum.max/2`
   * [Kernel] Improve error message when string interpolation is used in a guard
   * [Kernel] Properly merge and handle docs for callbacks with multiple clauses
@@ -133,6 +244,7 @@ There are many other enhancements. The Elixir CLI got a handful of new options i
   * [Kernel] Type variables starting with an underscore (`_foo`) should not raise compile error
   * [Kernel] Keep order of elements when macro `in/2` is expanded with a literal list on the right-hand side
   * [Kernel] Print proper location on undefined function error from dynamically generated functions
+  * [Kernel] **Potentially breaking** Do not leak aliases when nesting module definitions that are fully namespaced modules. If you defined `defmodule Elixir.Foo.Bar` inside `defmodule Foo`, previous Elixir versions would automatically define an alias, but fully namespaced modules such as `Elixir.Foo.Bar` should never define or require an alias. If you were accidentally relying on this broken behaviour, your code may no longer work
   * [System] Make sure `:init.get_status/0` is set to `{:started, :started}` once the system starts
   * [Path] Do not expand `~` in `Path.expand/2` when not followed by a path separator
   * [Protocol] Ensure `debug_info` is kept in protocols
@@ -157,6 +269,7 @@ There are many other enhancements. The Elixir CLI got a handful of new options i
   * [mix compile] Ensure Erlang-based Mix compilers (erlang, leex, yecc) set valid position on diagnostics
   * [mix compile] Ensure compilation halts in an umbrella project if one of the siblings fail to compile
   * [mix deps] Raise an error if the umbrella app's dir name and `mix.exs` app name don't match
+  * [mix deps.compile] Fix subcommand splitting bug in rebar3
   * [mix test] Do not consider modules that are no longer cover compiled when computing coverage report, which could lead to flawed reports
 
 ### 3. Soft-deprecations (no warnings emitted)

Makefile

@@ -1,7 +1,7 @@
 PREFIX ?= /usr/local
 SHARE_PREFIX ?= $(PREFIX)/share
 MAN_PREFIX ?= $(SHARE_PREFIX)/man
-CANONICAL := master/ # master/ or vMAJOR.MINOR/
+CANONICAL := v1.9/ # master/ or vMAJOR.MINOR/
 ELIXIRC := bin/elixirc --verbose --ignore-module-conflict $(ELIXIRC_OPTS)
 ERLC := erlc -I lib/elixir/include $(ERLC_OPTS)
 ERL := erl -I lib/elixir/include -noshell -pa lib/elixir/ebin

VERSION

@@ -1 +1 @@
-1.9.0-dev
+1.9.4

bin/elixir.bat

@@ -156,9 +156,9 @@ if not !runMode! == "iex" (
   set beforeExtra=-noshell -s elixir start_cli !beforeExtra!
 )
 if defined useWerl (
-  start !ERTS_BIN!werl.exe !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+  start "" "!ERTS_BIN!werl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
 ) else (
-  !ERTS_BIN!erl.exe !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
+  "!ERTS_BIN!erl.exe" !ext_libs! !ELIXIR_ERL_OPTIONS! !parsErlang! !beforeExtra! -extra !parsElixir!
 )
 :end
 endlocal

lib/elixir/docs.exs

@@ -26,7 +26,8 @@
       Time,
       Tuple,
       URI,
-      Version
+      Version,
+      Version.Requirement
     ],
     "Collections & Enumerables": [
       Access,

lib/elixir/lib/agent.ex

@@ -123,7 +123,6 @@ defmodule Agent do
   The generated `child_spec/1` can be customized with the following options:
 
     * `:id` - the child specification identifier, defaults to the current module
-    * `:start` - how to start the child process (defaults to calling `__MODULE__.start_link/1`)
     * `:restart` - when the child should be restarted, defaults to `:permanent`
     * `:shutdown` - how to shut down the child, either immediately or by giving it time to shut down
 

lib/elixir/lib/code.ex

@@ -963,6 +963,7 @@ defmodule Code do
 
   For compiling many files concurrently, see `Kernel.ParallelCompiler.compile/2`.
   """
+  @doc since: "1.7.0"
   @spec compile_file(binary, nil | binary) :: [{module, binary}]
   def compile_file(file, relative_to \\ nil) when is_binary(file) do
     :elixir_compiler.file(find_file(file, relative_to), fn _, _ -> :ok end)

lib/elixir/lib/code/formatter.ex

@@ -1075,7 +1075,7 @@ defmodule Code.Formatter do
   defp local_to_algebra(fun, meta, args, context, state) when is_atom(fun) do
     skip_parens =
       cond do
-        not Keyword.get(meta, :no_parens, false) -> :required
+        not Keyword.get(meta, :no_parens, false) -> :skip_if_only_do_end
         local_without_parens?(fun, args, state) -> :skip_unless_many_args
         true -> :skip_if_do_end
       end
@@ -1093,9 +1093,10 @@ defmodule Code.Formatter do
     {doc, state}
   end
 
-  # parens may one of:
+  # parens may be one of:
   #
   #   * :skip_unless_many_args - skips parens unless we are the argument context
+  #   * :skip_if_only_do_end - skip parens if we are do-end and the only arg
   #   * :skip_if_do_end - skip parens if we are do-end
   #   * :required - never skip parens
   #
@@ -1111,11 +1112,16 @@ defmodule Code.Formatter do
 
     if blocks = do_end_blocks(last, state) do
       {call_doc, state} =
-        if rest == [] do
-          {" do", state}
-        else
-          no_parens? = parens != :required
-          call_args_to_algebra_no_blocks(meta, rest, no_parens?, list_to_keyword?, " do", state)
+        case rest do
+          [] when parens == :required ->
+            {"() do", state}
+
+          [] ->
+            {" do", state}
+
+          _ ->
+            no_parens? = parens not in [:required, :skip_if_only_do_end]
+            call_args_to_algebra_no_blocks(meta, rest, no_parens?, list_to_keyword?, " do", state)
         end
 
       {blocks_doc, state} = do_end_blocks_to_algebra(blocks, state)
@@ -1156,7 +1162,7 @@ defmodule Code.Formatter do
         {left_doc, _join, state} =
           args_to_algebra_with_comments(
             left,
-            Keyword.delete(meta, :end),
+            Keyword.delete(meta, :closing),
             skip_parens?,
             :force_comma,
             join,
@@ -1365,26 +1371,26 @@ defmodule Code.Formatter do
   ## Sigils
 
   defp maybe_sigil_to_algebra(fun, meta, args, state) do
-    case {Atom.to_string(fun), args} do
-      {<<"sigil_", name>>, [{:<<>>, _, entries}, modifiers]} ->
-        opening_terminator = Keyword.fetch!(meta, :terminator)
-        doc = <<?~, name, opening_terminator::binary>>
+    with <<"sigil_", name>> <- Atom.to_string(fun),
+         [{:<<>>, _, entries}, modifiers] when is_list(modifiers) <- args,
+         opening_terminator when not is_nil(opening_terminator) <- Keyword.get(meta, :terminator) do
+      doc = <<?~, name, opening_terminator::binary>>
 
-        if opening_terminator in [@double_heredoc, @single_heredoc] do
-          closing_terminator = concat(opening_terminator, List.to_string(modifiers))
+      if opening_terminator in [@double_heredoc, @single_heredoc] do
+        closing_terminator = concat(opening_terminator, List.to_string(modifiers))
 
-          {doc, state} =
-            entries
-            |> prepend_heredoc_line()
-            |> interpolation_to_algebra(:heredoc, state, doc, closing_terminator)
-
-          {force_unfit(doc), state}
-        else
-          escape = closing_sigil_terminator(opening_terminator)
-          closing_terminator = concat(escape, List.to_string(modifiers))
-          interpolation_to_algebra(entries, escape, state, doc, closing_terminator)
-        end
+        {doc, state} =
+          entries
+          |> prepend_heredoc_line()
+          |> interpolation_to_algebra(:heredoc, state, doc, closing_terminator)
 
+        {force_unfit(doc), state}
+      else
+        escape = closing_sigil_terminator(opening_terminator)
+        closing_terminator = concat(escape, List.to_string(modifiers))
+        interpolation_to_algebra(entries, escape, state, doc, closing_terminator)
+      end
+    else
       _ ->
         :error
     end
@@ -1847,7 +1853,7 @@ defmodule Code.Formatter do
   end
 
   defp add_max_line_to_last_clause([{op, meta, args}], max_line) do
-    [{op, [end: [line: max_line]] ++ meta, args}]
+    [{op, [closing: [line: max_line]] ++ meta, args}]
   end
 
   defp add_max_line_to_last_clause([clause | clauses], max_line) do
@@ -2239,7 +2245,7 @@ defmodule Code.Formatter do
   end
 
   defp end_line(meta) do
-    meta[:end][:line] || @min_line
+    meta[:closing][:line] || @min_line
   end
 
   ## Algebra helpers

lib/elixir/lib/config.ex

@@ -4,8 +4,8 @@ defmodule Config do
 
   ## Example
 
-  This module is most commonly used to define application
-  configuration, typically in `config/config.exs`:
+  This module is most commonly used to define application configuration,
+  typically in `config/config.exs`:
 
       import Config
 
@@ -18,14 +18,20 @@ defmodule Config do
   `import Config` will import the functions `config/2`, `config/3`
   and `import_config/1` to help you manage your configuration.
 
-  Once Mix starts, it will automatically evaluate the configuration
-  file and persist it into `:some_app`'s application environment, which
-  can be accessed in as follows:
+  `config/2` and `config/3` are used to define key-value configuration
+  for a given application. Once Mix starts, it will automatically
+  evaluate the configuration file and persist the configuration above
+  into `:some_app`'s application environment, which can be accessed in
+  as follows:
 
       "value1" = Application.fetch_env!(:some_app, :key1)
 
-  See `Config.Reader` for evaluating and reading configuration
-  files.
+  Finally, the line `import_config "#{Mix.env()}.exs"` will import other
+  config files, based on the current Mix environment, such as
+  `config/dev.exs` and `config/test.exs`.
+
+  `Config` also provides a low-level API for evaluating and reading
+  configuration, under the `Config.Reader` module.
 
   **Important:** if you are writing a library to be used by other developers,
   it is generally recommended to avoid the application environment, as the
@@ -34,26 +40,34 @@ defmodule Config do
 
   ## Migrating from `use Mix.Config`
 
-  The `Config` module in Elixir was introduced in v1.9 as a
-  replacement to `Mix.Config`, which was specific to Mix and
-  has been deprecated.
+  The `Config` module in Elixir was introduced in v1.9 as a replacement to
+  `Mix.Config`, which was specific to Mix and has been deprecated.
 
-  You can leverage `Config` instead of `Mix.Config` in two steps.
-  The first step is to replace `use Mix.Config` at the top of
-  your config files by `import Config`.
+  You can leverage `Config` instead of `Mix.Config` in two steps. The first
+  step is to replace `use Mix.Config` at the top of your config files by
+  `import Config`.
 
-  The second is to make sure your `import_config/1` calls do
-  not have a wildcard character. If so, you need to perform
-  the wildcard lookup manually. For example, if you did:
+  The second is to make sure your `import_config/1` calls do not have a
+  wildcard character. If so, you need to perform the wildcard lookup
+  manually. For example, if you did:
 
       import_config "../apps/*/config/config.exs"
 
   It has to be replaced by:
 
-      for config <- "apps/*/config/config.exs" |> Path.expand() |> Path.wildcard() do
+      for config <- "../apps/*/config/config.exs" |> Path.expand(__DIR__) |> Path.wildcard() do
         import_config config
       end
 
+  ## config/releases.exs
+
+  If you are using releases, see `mix release`, there another configuration
+  file called `config/releases.exs`. While `config/config.exs` and friends
+  mentioned in the previous section are executed whenever you run a Mix
+  command, including when you assemble a release, `config/releases.exs` is
+  execute every time your production system boots. Since Mix is not available
+  in a production system, `config/releases.exs` must not use any of the
+  functions from Mix.
   """
 
   @config_key {__MODULE__, :config}
@@ -152,7 +166,7 @@ defmodule Config do
 
   """
   @doc since: "1.9.0"
-  def config(root_key, key, opts) when is_atom(root_key) do
+  def config(root_key, key, opts) when is_atom(root_key) and is_atom(key) do
     get_config!()
     |> __merge__([{root_key, [{key, opts}]}])
     |> put_config()

lib/elixir/lib/config/provider.ex

@@ -220,11 +220,9 @@ defmodule Config.Provider do
   end
 
   defp write_config!(config, path) do
-    {date, time} = :erlang.localtime()
-    args = [date, time, config]
-    contents = :io_lib.format("%% coding: utf-8~n%% config generated at ~p ~p~n~p.~n", args)
+    contents = :io_lib.format("%% coding: utf-8~n~tw.~n", [config])
 
-    case File.write(path, contents) do
+    case File.write(path, contents, [:utf8]) do
       :ok ->
         :ok
 

lib/elixir/lib/exception.ex

@@ -208,7 +208,7 @@ defmodule Exception do
          {_, kind, _, clauses} <- List.keyfind(defs, {function, arity}, 0) do
       clauses =
         for {meta, ex_args, guards, _block} <- clauses do
-          scope = :elixir_erl.scope(meta)
+          scope = :elixir_erl.scope(meta, true)
 
           {erl_args, scope} =
             :elixir_erl_clauses.match(&:elixir_erl_pass.translate_args/2, ex_args, scope)

lib/elixir/lib/gen_server.ex

@@ -142,7 +142,6 @@ defmodule GenServer do
   The generated `child_spec/1` can be customized with the following options:
 
     * `:id` - the child specification identifier, defaults to the current module
-    * `:start` - how to start the child process (defaults to calling `__MODULE__.start_link/1`)
     * `:restart` - when the child should be restarted, defaults to `:permanent`
     * `:shutdown` - how to shut down the child, either immediately or by giving it time to shut down
 

lib/elixir/lib/inspect/algebra.ex

@@ -33,7 +33,7 @@ defmodule Inspect.Opts do
 
     * `:printable_limit` - limits the number of characters that are inspected
       on printable strings and printable charlists. You can use `String.printable?/1`
-      and `List.ascii_printable?/1` to check if a a given string or charlist is
+      and `List.ascii_printable?/1` to check if a given string or charlist is
       printable. Defaults to 4096. If you don't want to limit the number of
       characters to a particular number, use `:infinity`.
 

lib/elixir/lib/kernel.ex

@@ -4433,14 +4433,18 @@ defmodule Kernel do
         defoverridable message: 1
 
         @impl true
-        def exception(msg) when is_binary(msg) do
+        def exception(msg) when Kernel.is_binary(msg) do
           exception(message: msg)
         end
       end
 
       # TODO: Change the implementation on v2.0 to simply call Kernel.struct!/2
+      # Calls to Kernel functions must be fully-qualified to ensure
+      # reproducible builds; otherwise, this macro will generate ASTs
+      # with different metadata (:import, :context) depending on if
+      # it is the bootstrapped version or not.
       @impl true
-      def exception(args) when is_list(args) do
+      def exception(args) when Kernel.is_list(args) do
         struct = __struct__()
         {valid, invalid} = Enum.split_with(args, fn {k, _} -> Map.has_key?(struct, k) end)
 
@@ -4451,9 +4455,9 @@ defmodule Kernel do
           _ ->
             IO.warn(
               "the following fields are unknown when raising " <>
-                "#{inspect(__MODULE__)}: #{inspect(invalid)}. " <>
+                "#{Kernel.inspect(__MODULE__)}: #{Kernel.inspect(invalid)}. " <>
                 "Please make sure to only give known fields when raising " <>
-                "or redefine #{inspect(__MODULE__)}.exception/1 to " <>
+                "or redefine #{Kernel.inspect(__MODULE__)}.exception/1 to " <>
                 "discard unknown fields. Future Elixir versions will raise on " <>
                 "unknown fields given to raise/2"
             )
@@ -4880,16 +4884,24 @@ defmodule Kernel do
   @doc ~S"""
   Handles the sigil `~S` for strings.
 
-  It simply returns a string without escaping characters and without
-  interpolations.
+  It returns a string without interpolations and without escape
+  characters, except for the escaping of the closing sigil character
+  itself.
 
   ## Examples
 
       iex> ~S(foo)
       "foo"
-
       iex> ~S(f#{o}o)
       "f\#{o}o"
+      iex> ~S(\o/)
+      "\\o/"
+
+  However, if you want to re-use the sigil character itself on
+  the string, you need to escape it:
+
+      iex> ~S((\))
+      "()"
 
   """
   defmacro sigil_S(term, modifiers)
@@ -4926,8 +4938,9 @@ defmodule Kernel do
   @doc ~S"""
   Handles the sigil `~C` for charlists.
 
-  It simply returns a charlist without escaping characters and without
-  interpolations.
+  It returns a charlist without interpolations and without escape
+  characters, except for the escaping of the closing sigil character
+  itself.
 
   ## Examples
 
@@ -5007,8 +5020,10 @@ defmodule Kernel do
   @doc ~S"""
   Handles the sigil `~R` for regular expressions.
 
-  It returns a regular expression pattern without escaping
-  nor interpreting interpolations.
+  It returns a regular expression pattern without interpolations and
+  without escape characters. Note it still supports escape of Regex
+  tokens (such as escaping `+` or `?`) and it also requires you to
+  escape the closing sigil character itself if it appears on the Regex.
 
   More information on regexes can be found in the `Regex` module.
 
@@ -5171,8 +5186,9 @@ defmodule Kernel do
   @doc ~S"""
   Handles the sigil `~W` for list of words.
 
-  It returns a list of "words" split by whitespace without escaping nor
-  interpreting interpolations.
+  It returns a list of "words" split by whitespace without interpolations
+  and without escape characters, except for the escaping of the closing
+  sigil character itself.
 
   ## Modifiers
 

lib/elixir/lib/kernel/lexical_tracker.ex

@@ -33,7 +33,7 @@ defmodule Kernel.LexicalTracker do
 
   @doc false
   def stop(pid) do
-    :gen_server.cast(pid, :stop)
+    :gen_server.call(pid, :stop)
   end
 
   @doc false
@@ -146,6 +146,10 @@ defmodule Kernel.LexicalTracker do
     {:reply, :maps.get(key, cache), state}
   end
 
+  def handle_call(:stop, _from, state) do
+    {:stop, :normal, :ok, state}
+  end
+
   def handle_cast({:write_cache, key, value}, %{cache: cache} = state) do
     {:noreply, %{state | cache: :maps.put(key, value, cache)}}
   end
@@ -206,10 +210,6 @@ defmodule Kernel.LexicalTracker do
     {:noreply, %{state | directives: add_directive(state.directives, module, line, warn, :alias)}}
   end
 
-  def handle_cast(:stop, state) do
-    {:stop, :normal, state}
-  end
-
   @doc false
   def handle_info(_msg, state) do
     {:noreply, state}

lib/elixir/lib/kernel/typespec.ex

@@ -557,7 +557,7 @@ defmodule Kernel.Typespec do
     types =
       :lists.map(
         fn {field, _} -> {field, Keyword.get(fields, field, quote(do: term()))} end,
-        struct
+        :lists.sort(struct)
       )
 
     fun = fn {field, _} ->

lib/elixir/lib/macro/env.ex

@@ -85,7 +85,7 @@ defmodule Macro.Env do
           aliases: aliases,
           functions: functions,
           macros: macros,
-          macro_aliases: aliases,
+          macro_aliases: macro_aliases,
           context_modules: context_modules,
           vars: vars,
           unused_vars: unused_vars,

lib/elixir/lib/regex.ex

@@ -461,13 +461,13 @@ defmodule Regex do
   end
 
   defp safe_run(
-         %Regex{re_pattern: compiled, source: source, re_version: version},
+         %Regex{re_pattern: compiled, source: source, re_version: version, opts: compile_opts},
          string,
          options
        ) do
     case version() do
       ^version -> :re.run(string, compiled, options)
-      _ -> :re.run(string, source, options)
+      _ -> :re.run(string, source, translate_options(compile_opts, options))
     end
   end
 

lib/elixir/lib/supervisor.ex

@@ -306,7 +306,6 @@ defmodule Supervisor do
   following options:
 
     * `:id` - the child specification identifier, defaults to the current module
-    * `:start` - how to start the child process (defaults to calling `__MODULE__.start_link/1`)
     * `:restart` - when the supervisor should be restarted, defaults to `:permanent`
 
   The `@doc` annotation immediately preceding `use Supervisor` will be

lib/elixir/lib/system.ex

@@ -910,10 +910,14 @@ defmodule System do
   `convert_time_unit/3` accepts an additional time unit (other than the
   ones in the `t:time_unit/0` type) called `:native`. `:native` is the time
   unit used by the Erlang runtime system. It's determined when the runtime
-  starts and stays the same until the runtime is stopped. To determine what
-  the `:native` unit amounts to in a system, you can call this function to
-  convert 1 second to the `:native` time unit (i.e.,
-  `System.convert_time_unit(1, :second, :native)`).
+  starts and stays the same until the runtime is stopped, but could differ
+  the next time the runtime is started on the same machine. For this reason,
+  you should use this function to convert `:native` time units to a predictable
+  unit before you display them to humans.
+
+  To determine how many seconds the `:native` unit represents in your current
+  runtime, you can can call this function to convert 1 second to the `:native`
+  time unit: `System.convert_time_unit(1, :second, :native)`.
   """
   @spec convert_time_unit(integer, time_unit | :native, time_unit | :native) :: integer
   def convert_time_unit(time, from_unit, to_unit) do

lib/elixir/lib/task.ex

@@ -97,7 +97,6 @@ defmodule Task do
   `child_spec/1` can be customized with the following options:
 
     * `:id` - the child specification identifier, defaults to the current module
-    * `:start` - how to start the child process (defaults to calling `__MODULE__.start_link/1`)
     * `:restart` - when the child should be restarted, defaults to `:temporary`
     * `:shutdown` - how to shut down the child, either immediately or by giving it time to shut down
 

lib/elixir/lib/tuple.ex

@@ -31,16 +31,17 @@ defmodule Tuple do
 
   The functions in this module that add and remove elements from tuples are
   rarely used in practice, as they typically imply tuples are being used as
-  collections. To append to a tuple, it is preferable to use pattern matching:
+  collections. To append to a tuple, it is preferable to extract the elements
+  from the old tuple with pattern matching, and then create a new tuple:
 
       tuple = {:ok, :example}
 
       # Avoid
-      Tuple.insert_at(tuple, 2, %{})
+      result = Tuple.insert_at(tuple, 2, %{})
 
       # Prefer
       {:ok, atom} = tuple
-      {:ok, atom, %{}}
+      result = {:ok, atom, %{}}
 
   """
 

lib/elixir/pages/Compatibility and Deprecations.md

@@ -158,4 +158,4 @@ Version | Deprecated feature                                  | Replaced by (ava
 [v1.6]: https://github.com/elixir-lang/elixir/blob/v1.6/CHANGELOG.md#4-deprecations
 [v1.7]: https://github.com/elixir-lang/elixir/blob/v1.7/CHANGELOG.md#4-hard-deprecations
 [v1.8]: https://github.com/elixir-lang/elixir/blob/v1.8/CHANGELOG.md#4-hard-deprecations
-[v1.9]: https://github.com/elixir-lang/elixir/blob/master/CHANGELOG.md#4-hard-deprecations
+[v1.9]: https://github.com/elixir-lang/elixir/blob/v1.9/CHANGELOG.md#4-hard-deprecations

lib/elixir/src/elixir.hrl

@@ -13,6 +13,7 @@
   backup_vars=nil,         %% a copy of vars to be used on ^var
   extra_guards=[],         %% extra guards from args expansion
   counter=#{},             %% a map counting the variables defined
+  expand_captures=false,   %% a boolean to control if captures should be expanded
   stacktrace=false         %% holds information about the stacktrace variable
 }).
 

lib/elixir/src/elixir_erl.erl

@@ -1,7 +1,7 @@
 %% Compiler backend to Erlang.
 -module(elixir_erl).
 -export([elixir_to_erl/1, definition_to_anonymous/4, compile/1, consolidate/3,
-         get_ann/1, debug_info/4, scope/1, format_error/1]).
+         get_ann/1, debug_info/4, scope/2, format_error/1]).
 -include("elixir.hrl").
 -define(typespecs, 'Elixir.Kernel.Typespec').
 
@@ -48,7 +48,7 @@ get_ann([], Gen, Line) -> erl_anno:set_generated(Gen, Line).
 %% Converts an Elixir definition to an anonymous function.
 
 definition_to_anonymous(Module, Kind, Meta, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Clause) || Clause <- Clauses],
+  ErlClauses = [translate_clause(Kind, Clause, true) || Clause <- Clauses],
   Fun = {'fun', ?ann(Meta), {clauses, ErlClauses}},
   LocalHandler = fun(LocalName, LocalArgs) -> invoke_local(Module, LocalName, LocalArgs) end,
   {value, Result, _Binding} = erl_eval:expr(Fun, [], {value, LocalHandler}),
@@ -117,8 +117,8 @@ elixir_to_erl_cons(T) -> elixir_to_erl(T).
 
 %% Returns a scope for translation.
 
-scope(_Meta) ->
-  #elixir_erl{}.
+scope(_Meta, ExpandCaptures) ->
+  #elixir_erl{expand_captures=ExpandCaptures}.
 
 %% Static compilation hook, used in protocol consolidation
 
@@ -220,15 +220,15 @@ add_definition(Meta, Body, {Head, Tail}) ->
   end.
 
 translate_definition(Kind, Meta, {Name, Arity}, Clauses) ->
-  ErlClauses = [translate_clause(Kind, Clause) || Clause <- Clauses],
+  ErlClauses = [translate_clause(Kind, Clause, false) || Clause <- Clauses],
 
   case is_macro(Kind) of
     true -> {function, ?ann(Meta), elixir_utils:macro_name(Name), Arity + 1, ErlClauses};
     false -> {function, ?ann(Meta), Name, Arity, ErlClauses}
   end.
 
-translate_clause(Kind, {Meta, Args, Guards, Body}) ->
-  S = scope(Meta),
+translate_clause(Kind, {Meta, Args, Guards, Body}, ExpandCaptures) ->
+  S = scope(Meta, ExpandCaptures),
 
   {TClause, TS} = elixir_erl_clauses:clause(Meta,
                     fun elixir_erl_pass:translate_args/2, Args, Body, Guards, S),

lib/elixir/src/elixir_erl_pass.erl

@@ -69,7 +69,7 @@ translate({'__CALLER__', Meta, Atom}, S) when is_atom(Atom) ->
 translate({'__STACKTRACE__', Meta, Atom}, S = #elixir_erl{stacktrace={Var, _}}) when is_atom(Atom) ->
   {{var, ?ann(Meta), Var}, S#elixir_erl{stacktrace={Var, true}}};
 
-translate({'super', Meta, Args}, S) ->
+translate({'super', Meta, Args}, S) when is_list(Args) ->
   %% In the expanded AST, super is used to invoke a function
   %% in the current module originated from a default clause
   %% or a super call.
@@ -96,7 +96,15 @@ translate({'&', Meta, [{'/', _, [{{'.', _, [Remote, Fun]}, _, []}, Arity]}]}, S)
   {{'fun', Ann, {function, TRemote, TFun, TArity}}, SR};
 translate({'&', Meta, [{'/', _, [{Fun, _, Atom}, Arity]}]}, S)
     when is_atom(Fun), is_atom(Atom), is_integer(Arity) ->
-  {{'fun', ?ann(Meta), {function, Fun, Arity}}, S};
+  case S of
+    #elixir_erl{expand_captures=true} ->
+      Vars = [{list_to_atom("arg" ++ integer_to_list(Counter)), [], ?MODULE}
+              || Counter <- tl(lists:seq(0, Arity))],
+      translate({'fn', Meta, [{'->', Meta, [Vars, {Fun, Meta, Vars}]}]}, S);
+
+    #elixir_erl{expand_captures=false} ->
+      {{'fun', ?ann(Meta), {function, Fun, Arity}}, S}
+  end;
 
 translate({fn, Meta, Clauses}, S) ->
   Transformer = fun({'->', CMeta, [ArgsWithGuards, Expr]}, Acc) ->
@@ -382,10 +390,11 @@ translate_with_else(Meta, [], S) ->
   Var = {var, Ann, VarName},
   {{clause, Ann, [Var], [], [Var]}, SC};
 translate_with_else(Meta, [{else, [{'->', _, [[{Var, VarMeta, Kind}], Clause]}]}], S) when is_atom(Var), is_atom(Kind) ->
+  Ann = ?ann(?generated(Meta)),
   Ctx = elixir_utils:var_context(VarMeta, Kind),
   {ElseVarErl, SV} = elixir_erl_var:assign(Meta, Var, Ctx, S),
   {TranslatedClause, SC} = elixir_erl_pass:translate(Clause, SV),
-  {{clause, Meta, [ElseVarErl], [], [TranslatedClause]}, SC};
+  {{clause, Ann, [ElseVarErl], [], [TranslatedClause]}, SC};
 translate_with_else(Meta, [{else, Else}], S) ->
   Generated = ?generated(Meta),
   ElseVarEx = {else, Generated, ?var_context},

lib/elixir/src/elixir_locals.erl

@@ -10,14 +10,21 @@
 
 -include("elixir.hrl").
 -define(cache, {elixir, cache_env}).
+-define(locals, {elixir, locals}).
 -define(tracker, 'Elixir.Module.LocalsTracker').
 
 setup({DataSet, _DataBag}) ->
   ets:insert(DataSet, {?cache, 0, nil}),
+
+  case elixir_config:get(bootstrap) of
+    false -> ets:insert(DataSet, {?locals, true});
+    true -> ok
+  end,
+
   ok.
 
-stop({_DataSet, _DataBag}) ->
-  ok.
+stop({DataSet, _DataBag}) ->
+  ets:delete(DataSet, ?locals).
 
 yank(Tuple, Module) ->
   if_tracker(Module, fun(Tracker) -> ?tracker:yank(Tracker, Tuple) end).
@@ -44,9 +51,12 @@ if_tracker(Module, Callback) ->
   if_tracker(Module, ok, Callback).
 
 if_tracker(Module, Default, Callback) ->
-  try {elixir_config:get(bootstrap), elixir_module:data_tables(Module)} of
-    {true, _Tracker} -> Default;
-    {false, Tracker} -> Callback(Tracker)
+  try
+    {DataSet, _} = Tables = elixir_module:data_tables(Module),
+    {ets:member(DataSet, ?locals), Tables}
+  of
+    {true, Tracker} -> Callback(Tracker);
+    {false, _} -> Default
   catch
     error:badarg -> Default
   end.

lib/elixir/src/elixir_parser.yrl

@@ -653,12 +653,12 @@ line_from_location({Line, _Column, _}) -> Line.
 is_eol({_, _, Eol}) -> is_integer(Eol) and (Eol > 0).
 
 end_meta(Token) ->
-  [{format, block}, {'end', meta_from_location(?location(Token))}].
+  [{format, block}, {closing, meta_from_location(?location(Token))}].
 
-meta_from_token_with_end(Begin, End) ->
+meta_from_token_with_closing(Begin, End) ->
   case ?formatter_metadata() of
     true ->
-      [{'end', meta_from_location(?location(End))} | meta_from_token(Begin)];
+      [{closing, meta_from_location(?location(End))} | meta_from_token(Begin)];
     false ->
       meta_from_token(Begin)
   end.
@@ -740,7 +740,7 @@ eol_pair(Left, Right) ->
     true ->
       [
         {eol, is_eol(?location(Left)) and is_eol(?location(Right))},
-        {'end', meta_from_location(?location(Right))}
+        {closing, meta_from_location(?location(Right))}
       ];
     false ->
       []
@@ -828,7 +828,7 @@ build_identifier({_, Location, Identifier}, Args) ->
 build_fn(Fn, Stab, End) ->
   case check_stab(Stab, none) of
     stab ->
-      Meta = eol_op(?location(Fn)) ++ meta_from_token_with_end(Fn, End),
+      Meta = eol_op(?location(Fn)) ++ meta_from_token_with_closing(Fn, End),
       {fn, Meta, collect_stab(Stab, [], [])};
     block ->
       return_error(meta_from_token(Fn), "expected anonymous functions to be defined with -> inside: ", "'fn'")
@@ -902,7 +902,7 @@ charlist_part({Begin, End, Tokens}) ->
   Form = string_tokens_parse(Tokens),
   Meta =
     case ?formatter_metadata() of
-      true -> [{'end', meta_from_location(End)} | meta_from_location(Begin)];
+      true -> [{closing, meta_from_location(End)} | meta_from_location(Begin)];
       false -> meta_from_location(Begin)
     end,
   {{'.', Meta, ['Elixir.Kernel', to_string]}, Meta, [Form]}.
@@ -915,7 +915,7 @@ string_part({Begin, End, Tokens}) ->
   Form = string_tokens_parse(Tokens),
   Meta =
     case ?formatter_metadata() of
-      true -> [{'end', meta_from_location(End)} | meta_from_location(Begin)];
+      true -> [{closing, meta_from_location(End)} | meta_from_location(Begin)];
       false -> meta_from_location(Begin)
     end,
   {'::', Meta, [{{'.', Meta, ['Elixir.Kernel', to_string]}, Meta, [Form]}, {binary, Meta, nil}]}.
@@ -943,7 +943,7 @@ build_stab(Stab) ->
 build_stab(Before, Stab, After) ->
   case build_stab(Stab) of
     {'__block__', Meta, Block} ->
-      {'__block__', Meta ++ meta_from_token_with_end(Before, After), Block};
+      {'__block__', Meta ++ meta_from_token_with_closing(Before, After), Block};
     Other ->
       Other
   end.

lib/elixir/test/elixir/code_formatter/calls_test.exs

@@ -449,6 +449,12 @@ defmodule Code.Formatter.CallsTest do
       assert_same "unquote(call)()"
       assert_same "unquote(call)(one, two)"
 
+      assert_same """
+      unquote(call)() do
+        :ok
+      end
+      """
+
       assert_same """
       unquote(call)(one, two) do
         :ok
@@ -668,6 +674,11 @@ defmodule Code.Formatter.CallsTest do
       assert_same "foo.bar(call)()"
       assert_same "foo.bar(call)(one, two)"
 
+      assert_same """
+      foo.bar(call)() do
+      end
+      """
+
       assert_same """
       foo.bar(call)(one, two) do
         :ok

lib/elixir/test/elixir/code_formatter/integration_test.exs

@@ -461,6 +461,14 @@ defmodule Code.Formatter.IntegrationTest do
     """
   end
 
+  test "false positive sigil" do
+    assert_same """
+    def sigil_d(<<year::2-bytes, "-", month::2-bytes, "-", day::2-bytes>>, calendar) do
+      ymd(year, month, day, calendar)
+    end
+    """
+  end
+
   test "capture with operators" do
     assert_same """
     "this works" |> (&String.upcase/1) |> (&String.downcase/1)

lib/elixir/test/elixir/code_test.exs

@@ -252,12 +252,14 @@ defmodule CodeTest do
       assert string_to_quoted.(":one") == {:__block__, [line: 1], [:one]}
 
       args = [[{:__block__, [original: '1', line: 1], [1]}]]
-      assert string_to_quoted.("[1]") == {:__block__, [eol: false, end: [line: 1], line: 1], args}
+
+      assert string_to_quoted.("[1]") ==
+               {:__block__, [eol: false, closing: [line: 1], line: 1], args}
 
       args = [{{:__block__, [line: 1], [:ok]}, {:__block__, [line: 1], [:test]}}]
 
       assert string_to_quoted.("{:ok, :test}") ==
-               {:__block__, [eol: false, end: [line: 1], line: 1], args}
+               {:__block__, [eol: false, closing: [line: 1], line: 1], args}
 
       assert string_to_quoted.(~s("""\nhello\n""")) ==
                {:__block__, [format: :bin_heredoc, line: 1], ["hello\n"]}
@@ -265,12 +267,12 @@ defmodule CodeTest do
       assert string_to_quoted.("'''\nhello\n'''") ==
                {:__block__, [format: :list_heredoc, line: 1], ['hello\n']}
 
-      left = {:__block__, [original: '1', line: 1, end: [line: 1], line: 1], [1]}
+      left = {:__block__, [original: '1', line: 1, closing: [line: 1], line: 1], [1]}
       right = {:__block__, [format: :string, line: 1], ["hello"]}
       args = [{:->, [line: 1], [[left], right]}]
 
       assert string_to_quoted.(~s[fn (1) -> "hello" end]) ==
-               {:fn, [end: [line: 1], line: 1], args}
+               {:fn, [closing: [line: 1], line: 1], args}
     end
 
     test "adds newlines information to blocks when :formatter_metadata (private) is given" do
@@ -285,11 +287,11 @@ defmodule CodeTest do
       """
 
       args = [
-        {:one, [eol: false, end: [line: 1], line: 1], []},
-        {:two, [newlines: 0, eol: false, end: [line: 1], line: 1], []},
-        {:three, [newlines: 1, eol: false, end: [line: 2], line: 2], []},
-        {:four, [newlines: 2, eol: false, end: [line: 4], line: 4], []},
-        {:five, [newlines: 3, eol: false, end: [line: 7], line: 7], []}
+        {:one, [eol: false, closing: [line: 1], line: 1], []},
+        {:two, [newlines: 0, eol: false, closing: [line: 1], line: 1], []},
+        {:three, [newlines: 1, eol: false, closing: [line: 2], line: 2], []},
+        {:four, [newlines: 2, eol: false, closing: [line: 4], line: 4], []},
+        {:five, [newlines: 3, eol: false, closing: [line: 7], line: 7], []}
       ]
 
       assert Code.string_to_quoted!(file, formatter_metadata: true) == {:__block__, [], args}

lib/elixir/test/elixir/config/provider_test.exs

@@ -18,7 +18,7 @@ defmodule Config.ProviderTest do
 
     File.rm_rf(@tmp_path)
     File.mkdir_p!(@tmp_path)
-    File.write!(@sys_config, :io_lib.format("~p.~n", [context[:sys_config] || []]))
+    File.write!(@sys_config, :io_lib.format("~tw.~n", [context[:sys_config] || []]), [:utf8])
 
     on_exit(fn ->
       Application.delete_env(@config_app, :config_providers)
@@ -50,14 +50,16 @@ defmodule Config.ProviderTest do
       assert config[@config_app] == [config_providers: :booted]
     end
 
-    test "writes extra config" do
-      init_and_assert_boot(extra_config: [my_app: [key: :old_value, sys_key: :sys_value]])
-      assert consult(@sys_config)[:my_app] == [sys_key: :sys_value, key: :value]
+    @tag sys_config: [my_app: [encoding: {:"£", "£", '£'}]]
+    test "writes sys_config with encoding" do
+      init_and_assert_boot()
+      config = consult(@sys_config)
+      assert config[:my_app][:encoding] == {:"£", "£", '£'}
     end
 
     @tag sys_config: [my_app: [key: :old_value, sys_key: :sys_value, extra_config: :old_value]]
-    test "overrides sys_config" do
-      init_and_assert_boot(extra_config: [my_app: [extra_config: :value]])
+    test "writes extra config with overrides" do
+      init_and_assert_boot(extra_config: [my_app: [key: :old_extra_value, extra_config: :value]])
 
       assert consult(@sys_config)[:my_app] ==
                [sys_key: :sys_value, extra_config: :value, key: :value]

lib/elixir/test/elixir/kernel/macros_test.exs

@@ -34,6 +34,10 @@ defmodule Kernel.MacrosTest do
     quote(do: 1 + unquote(value))
   end
 
+  defmacro my_macro_with_capture(value) do
+    Enum.map(value, &by_two/1)
+  end
+
   test "require" do
     assert Kernel.MacrosTest.Nested.value() == 1
   end
@@ -42,7 +46,7 @@ defmodule Kernel.MacrosTest do
     assert Nested.value() == 1
   end
 
-  test "local but private macro" do
+  test "local with private macro" do
     assert my_private_macro() == 4
   end
 
@@ -54,6 +58,10 @@ defmodule Kernel.MacrosTest do
     assert my_macro_with_local(4) == 17
   end
 
+  test "local with capture" do
+    assert my_macro_with_capture([1, 2, 3]) == [2, 4, 6]
+  end
+
   test "macros cannot be called dynamically" do
     x = Nested
     assert_raise UndefinedFunctionError, fn -> x.value end

lib/elixir/test/elixir/kernel/overridable_test.exs

@@ -254,6 +254,11 @@ defmodule Kernel.OverridableTest do
     assert Overridable.capture_super_macro(5) == [1, 2, 3, 4, 5, 1, 2, 3, 4, 5]
   end
 
+  test "super as a variable" do
+    super = :ok
+    assert super == :ok
+  end
+
   test "overridable with many clauses" do
     assert Overridable.many_clauses(0) == 11
     assert Overridable.many_clauses(1) == 13

lib/elixir/test/elixir/node_test.exs

@@ -5,7 +5,7 @@ defmodule NodeTest do
 
   doctest Node
 
-  @tag :unix
+  @tag :epmd
   test "start/3 and stop/0" do
     assert Node.stop() == {:error, :not_found}
     assert {:ok, _} = Node.start(:hello, :shortnames, 15000)

lib/elixir/test/elixir/regex_test.exs

@@ -194,6 +194,10 @@ defmodule RegexTest do
     assert Regex.run(@re_19_3_little, "abcd abce", capture: :all_names) == ["d"]
   end
 
+  test "run/3 with regexes with options compiled in different systems" do
+    assert Regex.run(%{~r/foo/i | re_version: "bad version"}, "FOO") == ["FOO"]
+  end
+
   test "scan/2" do
     assert Regex.scan(~r"c(d|e)", "abcd abce") == [["cd", "d"], ["ce", "e"]]
     assert Regex.scan(~r"c(?:d|e)", "abcd abce") == [["cd"], ["ce"]]
@@ -213,6 +217,10 @@ defmodule RegexTest do
     assert Regex.scan(@re_19_3_little, "abcd abce", capture: :all_names) == [["d"], ["e"]]
   end
 
+  test "scan/2 with regexes with options compiled in different systems" do
+    assert Regex.scan(%{~r/foo/i | re_version: "bad version"}, "FOO") == [["FOO"]]
+  end
+
   test "split/2,3" do
     assert Regex.split(~r",", "") == [""]
     assert Regex.split(~r",", "", trim: true) == []

lib/elixir/test/elixir/test_helper.exs

@@ -84,10 +84,11 @@ defmodule CodeFormatterHelpers do
 end
 
 assert_timeout = String.to_integer(System.get_env("ELIXIR_ASSERT_TIMEOUT") || "500")
-exclude = if PathHelpers.windows?(), do: [unix: true], else: [windows: true]
+epmd_exclude = if match?({_, 0}, System.cmd("epmd", ["-daemon"])), do: [], else: [epmd: true]
+os_exclude = if PathHelpers.windows?(), do: [unix: true], else: [windows: true]
 
 ExUnit.start(
   trace: "--trace" in System.argv(),
   assert_receive_timeout: assert_timeout,
-  exclude: exclude
+  exclude: epmd_exclude ++ os_exclude
 )

lib/elixir/test/elixir/typespec_test.exs

@@ -480,6 +480,23 @@ defmodule TypespecTest do
       assert {:type, _, :map_field_exact, [{:atom, _, :other}, {:type, _, :term, []}]} = arg2
     end
 
+    @fields Enum.map(10..42, &{:"f#{&1}", :ok})
+
+    test "@type with a large struct" do
+      bytecode =
+        test_module do
+          defstruct unquote(@fields)
+          @type my_type :: %TypespecSample{unquote_splicing(@fields)}
+        end
+
+      assert [type: {:my_type, type, []}] = types(bytecode)
+      assert {:type, _, :map, [struct, arg1, arg2 | _]} = type
+      assert {:type, _, :map_field_exact, struct_args} = struct
+      assert [{:atom, _, :__struct__}, {:atom, _, TypespecSample}] = struct_args
+      assert {:type, _, :map_field_exact, [{:atom, _, :f10}, {:atom, _, :ok}]} = arg1
+      assert {:type, _, :map_field_exact, [{:atom, _, :f11}, {:atom, _, :ok}]} = arg2
+    end
+
     test "@type with struct does not @enforce_keys" do
       bytecode =
         test_module do

lib/iex/lib/iex/introspection.ex

@@ -644,7 +644,9 @@ defmodule IEx.Introspection do
         types_not_found(inspect(module))
 
       {:ok, types} ->
-        Enum.each(types, &(&1 |> format_type() |> IO.puts()))
+        types
+        |> Enum.sort_by(fn {_, {name, _, args}} -> {name, length(args)} end)
+        |> Enum.each(&(&1 |> format_type() |> IO.puts()))
     end
 
     dont_display_result()
@@ -656,15 +658,19 @@ defmodule IEx.Introspection do
         no_beam(module)
 
       {:ok, types} ->
-        printed =
-          for {kind, {^type, _, args}} = typespec <- types,
-              kind != :typep do
-            type_doc(module, type, length(args), typespec)
-            |> print_typespec()
-          end
+        types
+        |> Enum.filter(&match?({kind, {^type, _, _}} when kind in [:type, :opaque], &1))
+        |> Enum.sort_by(fn {_, {name, _, args}} -> {name, length(args)} end)
+        |> case do
+          [] ->
+            types_not_found_or_private("#{inspect(module)}.#{type}")
 
-        if printed == [] do
-          types_not_found_or_private("#{inspect(module)}.#{type}")
+          types ->
+            Enum.map(types, fn {_, {_, _, args}} = typespec ->
+              module
+              |> type_doc(type, length(args), typespec)
+              |> print_typespec()
+            end)
         end
     end
 
@@ -677,16 +683,21 @@ defmodule IEx.Introspection do
         no_beam(module)
 
       {:ok, types} ->
-        printed =
-          for {kind, {^type, _, args}} = typespec <- types,
-              kind != :typep,
-              length(args) == arity do
-            type_doc(module, type, arity, typespec)
-            |> print_typespec()
-          end
+        types
+        |> Enum.find(
+          &match?(
+            {kind, {^type, _, args}} when kind in [:type, :opaque] and length(args) == arity,
+            &1
+          )
+        )
+        |> case do
+          nil ->
+            types_not_found_or_private("#{inspect(module)}.#{type}")
 
-        if printed == [] do
-          types_not_found_or_private("#{inspect(module)}.#{type}")
+          typespec ->
+            module
+            |> type_doc(type, arity, typespec)
+            |> print_typespec()
         end
     end
 

lib/iex/test/iex/helpers_test.exs

@@ -824,6 +824,10 @@ defmodule IEx.HelpersTest do
              end) >= 2
     end
 
+    test "prints private types" do
+      assert capture_io(fn -> t(Date.Range) end) =~ "@typep iso_days"
+    end
+
     test "prints type information" do
       assert "@type t() ::" <> _ = capture_io(fn -> t(Enum.t()) end)
       assert capture_io(fn -> t(Enum.t()) end) == capture_io(fn -> t(Enum.t() / 0) end)
@@ -831,6 +835,16 @@ defmodule IEx.HelpersTest do
       assert capture_io(fn -> t(URI.t()) end) == capture_io(fn -> t(URI.t() / 0) end)
     end
 
+    test "sorts types alphabetically" do
+      unsorted =
+        capture_io(fn -> t(Enum) end)
+        |> String.split("\n")
+        |> Enum.reject(&(&1 == ""))
+        |> Enum.map(&String.replace(&1, ~r/@(type|opaque) /, ""))
+
+      assert unsorted == Enum.sort(unsorted)
+    end
+
     test "prints type documentation" do
       content = """
       defmodule TypeSample do

lib/logger/lib/logger/translator.ex

@@ -332,6 +332,7 @@ defmodule Logger.Translator do
   defp sup_name({:global, name}), do: inspect(name)
   defp sup_name({:via, _mod, name}), do: inspect(name)
   defp sup_name({pid, mod}), do: [inspect(pid), " (", inspect(mod), ?)]
+  defp sup_name(unknown_name), do: inspect(unknown_name)
 
   defp sup_context(:start_error), do: "failed to start"
   defp sup_context(:child_terminated), do: "terminated"

lib/mix/lib/mix/cli.ex

@@ -29,7 +29,7 @@ defmodule Mix.CLI do
     load_mix_exs()
     {task, args} = get_task(args)
     ensure_hex(task)
-    change_env(task)
+    maybe_change_env_and_target(task)
     run_task(task, args)
   end
 
@@ -104,26 +104,39 @@ defmodule Mix.CLI do
     end
   end
 
-  defp change_env(task) do
-    if env = preferred_cli_env(task) do
-      Mix.env(env)
+  defp maybe_change_env_and_target(task) do
+    task = String.to_atom(task)
+    config = Mix.Project.config()
 
-      if project = Mix.Project.pop() do
-        %{name: name, file: file} = project
-        Mix.Project.push(name, file)
-      end
+    env = preferred_cli_env(task, config)
+    target = preferred_cli_target(task, config)
+    env && Mix.env(env)
+    target && Mix.target(target)
+
+    if env || target do
+      reload_project()
     end
   end
 
-  defp preferred_cli_env(task) do
+  defp reload_project() do
+    if project = Mix.Project.pop() do
+      %{name: name, file: file} = project
+      Mix.Project.push(name, file)
+    end
+  end
+
+  defp preferred_cli_env(task, config) do
     if System.get_env("MIX_ENV") do
       nil
     else
-      task = String.to_atom(task)
-      Mix.Project.config()[:preferred_cli_env][task] || Mix.Task.preferred_cli_env(task)
+      config[:preferred_cli_env][task] || Mix.Task.preferred_cli_env(task)
     end
   end
 
+  defp preferred_cli_target(task, config) do
+    config[:preferred_cli_target][task]
+  end
+
   defp load_dot_config do
     path = Path.join(Mix.Utils.mix_config(), "config.exs")
 

lib/mix/lib/mix/compilers/elixir.ex

@@ -264,7 +264,12 @@ defmodule Mix.Compilers.Elixir do
         nil -> {[file], false}
       end
 
-    {source, sources} = List.keytake(sources, file, source(:source))
+    {source, sources} =
+      List.keytake(sources, file, source(:source)) ||
+        Mix.raise(
+          "Could not find source for #{inspect(file)}. Make sure the :elixirc_paths configuration " <>
+            "is a list of relative paths to the current project or absolute paths to external directories"
+        )
 
     source =
       source(
@@ -515,10 +520,17 @@ defmodule Mix.Compilers.Elixir do
         split_manifest(data, compile_path)
 
       [v | data] when is_integer(v) ->
-        for module <- data, is_record(module, :module) do
-          File.rm(Path.join(compile_path, module(module, :beam)))
-          :code.purge(module(module, :module))
-          :code.delete(module(module, :module))
+        try do
+          for module <- data, is_record(module, :module) do
+            File.rm(Path.join(compile_path, module(module, :beam)))
+            :code.purge(module(module, :module))
+            :code.delete(module(module, :module))
+          end
+        rescue
+          _ ->
+            Mix.raise(
+              "Cannot clean-up stale manifest, please run \"mix clean --deps\" manually before proceeding"
+            )
         end
 
         {[], []}

lib/mix/lib/mix/local/installer.ex

@@ -107,7 +107,7 @@ defmodule Mix.Local.Installer do
           module.install(basename, binary, previous_files)
 
         :badpath ->
-          Mix.raise("Expected #{inspect(src)} to be a URL or a local file path")
+          Mix.raise("Expected #{inspect(src)} to be a local file path")
 
         {:local, message} ->
           Mix.raise(message)
@@ -164,7 +164,7 @@ defmodule Mix.Local.Installer do
     cond do
       local_path?(url_or_path) -> {:local, url_or_path}
       file_url?(url_or_path) -> {:url, url_or_path}
-      true -> {:error, "Expected #{inspect(url_or_path)} to be a URL or a local file path"}
+      true -> {:error, "Expected #{inspect(url_or_path)} to be a local file path"}
     end
   end
 

lib/mix/lib/mix/project.ex

@@ -67,9 +67,13 @@ defmodule Mix.Project do
     * `:preferred_cli_env` - a keyword list of `{task, env}` tuples where `task`
       is the task name as an atom (for example, `:"deps.get"`) and `env` is the
       preferred environment (for example, `:test`). This option overrides what
-      specified by the tasks with the `@preferred_cli_env` attribute (see the
+      is specified by the tasks with the `@preferred_cli_env` attribute (see the
       docs for `Mix.Task`). Defaults to `[]`.
 
+    * `:preferred_cli_target` - a keyword list of `{task, target}` tuples where
+      `task` is the task name as an atom (for example, `:test`) and `target`
+      is the preferred target (for example, `:host`). Defaults to `[]`.
+
   For more options, keep an eye on the documentation for single Mix tasks; good
   examples are the `Mix.Tasks.Compile` task and all the specific compiler tasks
   (such as `Mix.Tasks.Compile.Elixir` or `Mix.Tasks.Compile.Erlang`).

lib/mix/lib/mix/release.ex

@@ -7,7 +7,8 @@ defmodule Mix.Release do
   The Mix.Release struct has the following read-only fields:
 
     * `:name` - the name of the release as an atom
-    * `:version` - the version of the release as a string
+    * `:version` - the version of the release as a string or
+       `{:from_app, app_name}
     * `:path` - the path to the release root
     * `:version_path` - the path to the release version inside the release
     * `:applications` - a map of application with their definitions
@@ -24,7 +25,8 @@ defmodule Mix.Release do
       and `term` is the value given to it on `c:Config.Provider.init/1`
     * `:options` - a keyword list with all other user supplied release options
     * `:steps` - a list of functions that receive the release and returns a release.
-      Must also contain the atom `:assemble` which is the internal assembling step
+      Must also contain the atom `:assemble` which is the internal assembling step.
+      May also contain the atom `:tar` to create a tarball of the release.
 
   """
   defstruct [
@@ -47,7 +49,7 @@ defmodule Mix.Release do
           name: atom(),
           version: String.t(),
           path: String.t(),
-          version_path: String.t(),
+          version_path: String.t() | {:from_app, application()},
           applications: %{application() => keyword()},
           boot_scripts: %{atom() => [{application(), mode()}]},
           erts_version: charlist(),
@@ -83,14 +85,14 @@ defmodule Mix.Release do
     {include_erts, opts} = Keyword.pop(opts, :include_erts, true)
     {erts_source, erts_lib_dir, erts_version} = erts_data(include_erts)
 
-    loaded_apps = apps |> Keyword.keys() |> load_apps(%{}, erts_lib_dir)
+    loaded_apps = apps |> Keyword.keys() |> load_apps(%{}, erts_lib_dir, :maybe)
 
     # Make sure IEx is either an active part of the release or add it as none.
     {loaded_apps, apps} =
       if Map.has_key?(loaded_apps, :iex) do
         {loaded_apps, apps}
       else
-        {load_apps([:iex], loaded_apps, erts_lib_dir), apps ++ [iex: :none]}
+        {load_apps([:iex], loaded_apps, erts_lib_dir, :maybe), apps ++ [iex: :none]}
       end
 
     start_boot = build_start_boot(loaded_apps, apps)
@@ -112,6 +114,24 @@ defmodule Mix.Release do
           )
       end)
 
+    version =
+      case version do
+        {:from_app, app} ->
+          Application.load(app)
+          version = Application.spec(app, :vsn)
+
+          if !version do
+            Mix.raise(
+              "Could not find version for #{inspect(app)}, please make sure the application exists"
+            )
+          end
+
+          to_string(version)
+
+        _ ->
+          version
+      end
+
     {config_providers, opts} = Keyword.pop(opts, :config_providers, [])
     {steps, opts} = Keyword.pop(opts, :steps, [:assemble])
     validate_steps!(steps)
@@ -158,10 +178,16 @@ defmodule Mix.Release do
         {name, opts}
 
       [_ | _] ->
-        Mix.raise(
-          "\"mix release\" was invoked without a name but there are multiple releases. " <>
-            "Please call \"mix release NAME\" or set :default_release in your project configuration"
-        )
+        case Keyword.get(config, :default_release) do
+          nil ->
+            Mix.raise(
+              "\"mix release\" was invoked without a name but there are multiple releases. " <>
+                "Please call \"mix release NAME\" or set :default_release in your project configuration"
+            )
+
+          name ->
+            lookup_release(name, config)
+        end
     end
   end
 
@@ -228,36 +254,62 @@ defmodule Mix.Release do
     end
   end
 
-  defp load_apps(apps, seen, otp_root) do
-    for app <- apps,
-        not Map.has_key?(seen, app),
-        reduce: seen do
-      seen -> load_app(app, seen, otp_root)
+  defp load_apps(apps, seen, otp_root, included) do
+    for app <- apps, reduce: seen do
+      seen ->
+        if reentrant_seen = reentrant(seen, app, included) do
+          reentrant_seen
+        else
+          load_app(app, seen, otp_root, included)
+        end
     end
   end
 
-  defp load_app(app, seen, otp_root) do
+  defp reentrant(seen, app, included) do
+    properties = seen[app]
+
+    cond do
+      is_nil(properties) ->
+        nil
+
+      included != :maybe and properties[:included] != included ->
+        if properties[:included] == :maybe do
+          put_in(seen[app][:included], included)
+        else
+          Mix.raise(
+            "#{inspect(app)} is listed both as a regular application and as an included application"
+          )
+        end
+
+      true ->
+        seen
+    end
+  end
+
+  defp load_app(app, seen, otp_root, included) do
     path = Path.join(otp_root, "#{app}-*")
 
     case Path.wildcard(path) do
       [] ->
         case :code.lib_dir(app) do
           {:error, :bad_name} -> Mix.raise("Could not find application #{inspect(app)}")
-          path -> do_load_app(app, path, seen, otp_root, false)
+          path -> do_load_app(app, path, seen, otp_root, false, included)
         end
 
       paths ->
         path = paths |> Enum.sort() |> List.last()
-        do_load_app(app, to_charlist(path), seen, otp_root, true)
+        do_load_app(app, to_charlist(path), seen, otp_root, true, included)
     end
   end
 
-  defp do_load_app(app, path, seen, otp_root, otp_app?) do
+  defp do_load_app(app, path, seen, otp_root, otp_app?, included) do
     case :file.consult(Path.join(path, "ebin/#{app}.app")) do
       {:ok, terms} ->
         [{:application, ^app, properties}] = terms
-        seen = Map.put(seen, app, [path: path, otp_app?: otp_app?] ++ properties)
-        load_apps(Keyword.get(properties, :applications, []), seen, otp_root)
+        value = [path: path, otp_app?: otp_app?, included: included] ++ properties
+        seen = Map.put(seen, app, value)
+        seen = load_apps(Keyword.get(properties, :applications, []), seen, otp_root, false)
+        load_apps(Keyword.get(properties, :included_applications, []), seen, otp_root, true)
 
       {:error, reason} ->
         Mix.raise("Could not load #{app}.app. Reason: #{inspect(reason)}")
@@ -266,13 +318,19 @@ defmodule Mix.Release do
 
   defp build_start_boot(all_apps, specified_apps) do
     specified_apps ++
-      for(
-        {app, _props} <- all_apps,
-        not List.keymember?(specified_apps, app, 0),
-        do: {app, :permanent}
+      Enum.sort(
+        for(
+          {app, props} <- all_apps,
+          not List.keymember?(specified_apps, app, 0),
+          do: {app, default_mode(props)}
+        )
       )
   end
 
+  defp default_mode(props) do
+    if props[:included] == true, do: :load, else: :permanent
+  end
+
   defp build_start_clean_boot(boot) do
     for({app, _mode} <- boot, do: {app, :none})
     |> Keyword.put(:stdlib, :permanent)
@@ -280,12 +338,14 @@ defmodule Mix.Release do
   end
 
   defp validate_steps!(steps) do
-    if not is_list(steps) or Enum.any?(steps, &(&1 != :assemble and not is_function(&1, 1))) do
+    valid_atoms = [:assemble, :tar]
+
+    if not is_list(steps) or Enum.any?(steps, &(&1 not in valid_atoms and not is_function(&1, 1))) do
       Mix.raise("""
         The :steps option must be a list of:
 
         * anonymous function that receives one argument
-        * the atom :assemble
+        * the atom :assemble or :tar
 
       Got: #{inspect(steps)}
       """)
@@ -295,6 +355,14 @@ defmodule Mix.Release do
       Mix.raise("The :steps option must contain the atom :assemble once, got: #{inspect(steps)}")
     end
 
+    if :assemble in Enum.drop_while(steps, &(&1 != :tar)) do
+      Mix.raise("The :tar step must come after :assemble")
+    end
+
+    if Enum.count(steps, &(&1 == :tar)) > 1 do
+      Mix.raise("The :steps option can only contain the atom :tar once")
+    end
+
     :ok
   end
 
@@ -318,11 +386,10 @@ defmodule Mix.Release do
     {sys_config, runtime?} = merge_provider_config(release, sys_config, config_provider_path)
     path = Path.join(release.version_path, "sys.config")
 
-    {date, time} = :erlang.localtime()
-    args = [runtime?, date, time, sys_config]
-    format = "%% coding: utf-8~n%% RUNTIME_CONFIG=~s~n%% config generated at ~p ~p~n~p.~n"
+    args = [runtime?, sys_config]
+    format = "%% coding: utf-8~n%% RUNTIME_CONFIG=~s~n~tw.~n"
     File.mkdir_p!(Path.dirname(path))
-    File.write!(path, :io_lib.format(format, args))
+    File.write!(path, :io_lib.format(format, args), [:utf8])
 
     case :file.consult(path) do
       {:ok, _} ->
@@ -380,7 +447,7 @@ defmodule Mix.Release do
     end
   end
 
-  defp random_cookie, do: Base.url_encode64(:crypto.strong_rand_bytes(40))
+  defp random_cookie, do: Base.encode32(:crypto.strong_rand_bytes(32))
 
   @doc """
   Makes the start_erl.data file with the
@@ -407,7 +474,7 @@ defmodule Mix.Release do
           :ok | {:error, String.t()}
   def make_boot_script(release, path, modes, prepend_paths \\ []) do
     with {:ok, rel_spec} <- build_release_spec(release, modes) do
-      File.write!(path <> ".rel", consultable("rel", rel_spec))
+      File.write!(path <> ".rel", consultable(rel_spec), [:utf8])
 
       sys_path = String.to_charlist(path)
 
@@ -430,7 +497,7 @@ defmodule Mix.Release do
             |> prepend_paths_to_script(prepend_paths)
 
           script = {:script, rel_info, instructions}
-          File.write!(script_path, consultable("script", script))
+          File.write!(script_path, consultable(script), [:utf8])
           :ok = :systools.script2boot(sys_path)
 
         {:error, module, info} ->
@@ -546,10 +613,8 @@ defmodule Mix.Release do
     end)
   end
 
-  defp consultable(kind, term) do
-    {date, time} = :erlang.localtime()
-    args = [kind, date, time, term]
-    :io_lib.format("%% coding: utf-8~n%% ~ts generated at ~p ~p~n~p.~n", args)
+  defp consultable(term) do
+    :io_lib.format("%% coding: utf-8~n~tp.~n", [term])
   end
 
   @doc """

lib/mix/lib/mix/tasks/archive.install.ex

@@ -12,7 +12,7 @@ defmodule Mix.Tasks.Archive.Install do
 
       mix do archive.build, archive.install
 
-  If an argument is provided, it should be a local path or a URL to a
+  If an argument is provided, it should be a local path to a
   prebuilt archive, a Git repository, a GitHub repository, or a Hex
   package.
 
@@ -35,13 +35,12 @@ defmodule Mix.Tasks.Archive.Install do
       mix some_task
 
   Note that installing via Git, GitHub, or Hex fetches the source
-  of the archive and builds it, while using a URL or a local path
-  fetches a pre-built archive.
+  of the archive and builds it, while using local path uses a pre-built archive.
 
   ## Command line options
 
     * `--sha512` - checks the archive matches the given SHA-512 checksum. Only
-      applies to installations via URL or local path
+      applies to installations via a local path
 
     * `--force` - forces installation without a shell prompt; primarily
       intended for automation in build systems like Make
@@ -55,9 +54,6 @@ defmodule Mix.Tasks.Archive.Install do
     * `--organization` - specifies an organization to use if fetching the package
       from a private Hex repository
 
-    * `--timeout` - sets a request timeout in milliseconds for fetching
-      archives from URLs. Default is 60 seconds
-
   """
 
   @behaviour Mix.Local.Installer
@@ -77,17 +73,28 @@ defmodule Mix.Tasks.Archive.Install do
   end
 
   @impl true
-  def check_install_spec({local_or_url, path_or_url} = _install_spec, _opts)
-      when local_or_url in [:local, :url] do
-    if Path.extname(path_or_url) == ".ez" do
-      :ok
-    else
-      {:error, "Expected a local file path or a file URL ending in \".ez\"."}
+  def check_install_spec({:local, path} = _install_spec, _opts) do
+    check_extname(path)
+  end
+
+  def check_install_spec({:url, url} = _install_spec, opts) do
+    unless opts[:sha512] do
+      Mix.Utils.warn_install_over_http_deprecated("archive.install", url)
     end
+
+    check_extname(url)
   end
 
   def check_install_spec(_, _), do: :ok
 
+  defp check_extname(path_or_url) do
+    if Path.extname(path_or_url) == ".ez" do
+      :ok
+    else
+      {:error, "Expected a local file path ending in \".ez\"."}
+    end
+  end
+
   @impl true
   def find_previous_versions(src) do
     app =

lib/mix/lib/mix/tasks/deps.compile.ex

@@ -183,7 +183,7 @@ defmodule Mix.Tasks.Deps.Compile do
     lib_path = Path.join(config[:env_path], "lib/*/ebin")
 
     env = [{"REBAR_CONFIG", config_path}, {"TERM", "dumb"}]
-    cmd = "#{rebar_cmd(dep)} bare compile --paths #{inspect(lib_path)}"
+    cmd = "#{rebar_cmd(dep)} bare compile --paths=#{inspect(lib_path)}"
 
     File.mkdir_p!(dep_path)
     File.write!(config_path, rebar_config(dep))

lib/mix/lib/mix/tasks/escript.install.ex

@@ -12,12 +12,11 @@ defmodule Mix.Tasks.Escript.Install do
 
       mix do escript.build, escript.install
 
-  If an argument is provided, it should be a local path or a URL to a prebuilt escript,
+  If an argument is provided, it should be a local path to a prebuilt escript,
   a Git repository, a GitHub repository, or a Hex package.
 
       mix escript.install escript
       mix escript.install path/to/escript
-      mix escript.install https://example.com/my_escript
       mix escript.install git https://path/to/git/repo
       mix escript.install git https://path/to/git/repo branch git_branch
       mix escript.install git https://path/to/git/repo tag git_tag
@@ -40,7 +39,7 @@ defmodule Mix.Tasks.Escript.Install do
   ## Command line options
 
     * `--sha512` - checks the escript matches the given SHA-512 checksum. Only
-      applies to installations via URL or local path
+      applies to installations via a local path
 
     * `--force` - forces installation without a shell prompt; primarily
       intended for automation in build systems like Make
@@ -54,9 +53,6 @@ defmodule Mix.Tasks.Escript.Install do
     * `--organization` - specifies an organization to use if fetching the package
       from a private Hex repository
 
-    * `--timeout` - sets a request timeout in milliseconds for fetching
-      archives from URLs. Default is 60 seconds
-
   """
 
   @behaviour Mix.Local.Installer
@@ -79,6 +75,14 @@ defmodule Mix.Tasks.Escript.Install do
   end
 
   @impl true
+  def check_install_spec({:url, url} = _install_spec, opts) do
+    unless opts[:sha512] do
+      Mix.Utils.warn_install_over_http_deprecated("escript.install", url)
+    end
+
+    :ok
+  end
+
   def check_install_spec(_, _), do: :ok
 
   @impl true

lib/mix/lib/mix/tasks/format.ex

@@ -498,7 +498,7 @@ defmodule Mix.Tasks.Format do
   end
 
   defp to_bullet_list(files) do
-    Enum.map_join(files, "\n", &"  * #{&1}")
+    Enum.map_join(files, "\n", &"  * #{&1 |> to_string() |> Path.relative_to_cwd()}")
   end
 
   defp equivalent?(input, output) do

lib/mix/lib/mix/tasks/local.rebar.ex

@@ -25,9 +25,9 @@ defmodule Mix.Tasks.Local.Rebar do
 
   ## Command line options
 
-    * `rebar PATH` - specifies a path or URL for `rebar`
+    * `rebar PATH` - specifies a path for `rebar`
 
-    * `rebar3 PATH` - specifies a path or URL for `rebar3`
+    * `rebar3 PATH` - specifies a path for `rebar3`
 
     * `--sha512` - checks the archive matches the given SHA-512 checksum
 
@@ -48,9 +48,11 @@ defmodule Mix.Tasks.Local.Rebar do
 
     case argv do
       ["rebar", path | _] ->
+        if file_url?(path), do: warn_install_over_http_deprecated(:rebar, path)
         install_from_path(:rebar, path, opts)
 
       ["rebar3", path | _] ->
+        if file_url?(path), do: warn_install_over_http_deprecated(:rebar3, path)
         install_from_path(:rebar3, path, opts)
 
       [] ->
@@ -77,7 +79,7 @@ defmodule Mix.Tasks.Local.Rebar do
           Mix.shell().info([:green, "* creating ", :reset, Path.relative_to_cwd(local)])
 
         :badpath ->
-          Mix.raise("Expected #{inspect(path)} to be a URL or a local file path")
+          Mix.raise("Expected #{inspect(path)} to be a local file path")
 
         {:local, message} ->
           Mix.raise(message)
@@ -100,6 +102,10 @@ defmodule Mix.Tasks.Local.Rebar do
     true
   end
 
+  defp file_url?(url_or_path) do
+    URI.parse(url_or_path).scheme in ["http", "https"]
+  end
+
   defp install_from_s3(manager, list_url, escript_url, opts) do
     hex_mirror = Mix.Hex.mirror()
     list_url = hex_mirror <> list_url
@@ -114,4 +120,37 @@ defmodule Mix.Tasks.Local.Rebar do
 
     install_from_path(manager, url, Keyword.put(opts, :sha512, sha512))
   end
+
+  defp warn_install_over_http_deprecated(manager, url) do
+    shell = Mix.shell()
+    basename = Path.basename(url)
+
+    shell.error("""
+    Warning: the use of HTTP/HTTPS URLs with `mix local.rebar` is deprecated")
+
+    Run `mix help local.rebar` for details on installing #{manager} from Hex's CDN.
+    Alternatively you can fetch the file using an external HTTP client and then
+    install it locally:
+
+    Unix (Linux, MacOS X):
+
+        $ wget #{url}
+        $ mix local.rebar #{manager} #{basename}
+
+    or
+
+        $ curl -o #{basename} #{url}
+        $ mix local.rebar #{manager} #{basename}
+
+    Windows (Win7 or later):
+
+        > powershell -Command "Invoke-WebRequest #{url} -OutFile #{basename}"
+        > mix local.rebar #{manager} #{basename}
+
+    or
+
+        > powershell -Command "(New-Object Net.WebClient).DownloadFile('#{url}', '#{basename}')"
+        > mix local.rebar #{manager} #{basename}
+    """)
+  end
 end

lib/mix/lib/mix/tasks/release.ex

@@ -114,7 +114,15 @@ defmodule Mix.Tasks.Release do
   distribution. For example, if you need to do some prep work before
   running the actual system, like migrating your database, `eval` can
   be a good fit. Just keep in mind any application you may use during
-  eval has to be explicitly started.
+  eval has to be explicitly loaded and/or started.
+
+  You can start an application by calling `Application.ensure_all_started/1`.
+  However, if for some reason you cannot start an application, maybe
+  cause it will run other services you do not want, you must at least
+  load the application by calling `Application.load/1`. If you don't
+  load the application, any attempt at reading its environment or
+  configuration may fail. Note that if you start an application,
+  it is automatically loaded before started.
 
   Another way to run commands is with `rpc`, which will connect to the
   system currently running and instruct it to execute the given
@@ -133,7 +141,9 @@ defmodule Mix.Tasks.Release do
       defmodule MyApp.ReleaseTasks do
         def eval_purge_stale_data() do
           # Eval commands needs to start the app before
+          # Or Application.load(:my_app) if you can't start it
           Application.ensure_all_started(:my_app)
+
           # Code that purges stale data
           ...
         end
@@ -376,8 +386,10 @@ defmodule Mix.Tasks.Release do
     * `:path` - the path the release should be installed to.
       Defaults to `"_build/MIX_ENV/rel/RELEASE_NAME"`.
 
-    * `:version` - the release version as a string. Defaults to the current
-      application version.
+    * `:version` - the release version as a string or `{:from_app, app_name}`.
+      Defaults to the current application version. The `{:from_app, app_name}` format
+      can be used to easily reference the application version from another application.
+      This is particularly useful in umbrella applications.
 
     * `:quiet` - a boolean that controls if releases should write to the standard
       output its steps. Defaults to `false`.
@@ -443,11 +455,25 @@ defmodule Mix.Tasks.Release do
   to make sure the Erlang Distribution listens only on a given port
   known at runtime, you can set the following:
 
-      export ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min $BEAM_PORT inet_dist_listen_max $BEAM_PORT"
+      case $RELEASE_COMMAND in
+        start*|daemon*)
+          ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min $BEAM_PORT inet_dist_listen_max $BEAM_PORT"
+          export ELIXIR_ERL_OPTIONS
+          ;;
+        *)
+          ;;
+      esac
 
-  Or for Windows, in your `env.bat`:
+  Note we only set the port on start/daemon commands. If you also limit
+  the port on other commands, such as `rpc`, then you will be unable
+  to establish a remote connection as the port will already be in use
+  by the node.
 
-      set ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min %BEAM_PORT% inet_dist_listen_max %BEAM_PORT%"
+  On Windows, your `env.bat` would look like this:
+
+      IF NOT %RELEASE_COMMAND:start=%==%RELEASE_COMMAND% (
+        set ELIXIR_ERL_OPTIONS="-kernel inet_dist_listen_min %BEAM_PORT% inet_dist_listen_max %BEAM_PORT%"
+      )
 
   ### Steps
 
@@ -465,7 +491,9 @@ defmodule Mix.Tasks.Release do
   can pass anonymous functions before and after the `:assemble` to
   customize your release assembling pipeline. Those anonymous functions
   will receive a `Mix.Release` struct and must return the same or
-  an updated `Mix.Release` struct.
+  an updated `Mix.Release` struct. It is also possible to build a tarball
+  of the release by passing the `:tar` step anywhere after `:assemble`.
+  The tarball is created in `_build/MIX_ENV/rel/RELEASE_NAME-RELEASE_VSN.tar.gz`
 
   See `Mix.Release` for more documentation on the struct and which
   fields can be modified. Note that `:steps` field itself can be
@@ -667,6 +695,18 @@ defmodule Mix.Tasks.Release do
       not running on Erlang/OTP 22 or later, you must set `RELEASE_NODE`
       to `RELEASE_NAME@127.0.0.1` with an IP or a known host
 
+    * `RELEASE_BOOT_SCRIPT` - the name of the boot script to use when starting
+      the release. This script is used when running commands such as `start` and
+      `daemon`. The boot script is expected to be located at the
+      path `releases/RELEASE_VSN/RELEASE_BOOT_SCRIPT.boot`. Defaults to `start`
+
+    * `RELEASE_BOOT_SCRIPT_CLEAN` - the name of the boot script used when
+      starting the release clean, without your application or its dependencies.
+      This script is used by commands such as `eval`, `rpc`, and `remote`.
+      The boot script is expected to be located at the path
+      `releases/RELEASE_VSN/RELEASE_BOOT_SCRIPT_CLEAN.boot`. Defaults
+      to `start_clean`
+
   ## Umbrellas
 
   Releases are well integrated with umbrella projects, allowing you to
@@ -915,6 +955,10 @@ defmodule Mix.Tasks.Release do
     end
   end
 
+  defp run_steps(%{steps: [:tar | steps]} = release) do
+    %{release | steps: steps} |> make_tar() |> run_steps()
+  end
+
   defp run_steps(%{steps: [:assemble | steps]} = release) do
     %{release | steps: steps} |> assemble() |> run_steps()
   end
@@ -968,6 +1012,34 @@ defmodule Mix.Tasks.Release do
     release
   end
 
+  defp make_tar(release) do
+    tar_filename = "#{release.name}-#{release.version}.tar.gz"
+    out_path = Path.join([release.path, "..", "..", tar_filename]) |> Path.expand()
+    info(release, [:green, "* building ", :reset, out_path])
+
+    lib_dirs =
+      Enum.reduce(release.applications, [], fn {name, app_config}, acc ->
+        vsn = Keyword.fetch!(app_config, :vsn)
+        [Path.join("lib", "#{name}-#{vsn}") | acc]
+      end)
+
+    release_files =
+      for basename <- File.ls!(Path.join(release.path, "releases")),
+          not File.dir?(Path.join([release.path, "releases", basename])),
+          do: Path.join("releases", basename)
+
+    dirs =
+      ["bin", Path.join("releases", release.version), "erts-#{release.erts_version}"] ++
+        lib_dirs ++ release_files
+
+    files =
+      Enum.map(dirs, &{String.to_charlist(&1), String.to_charlist(Path.join(release.path, &1))})
+
+    File.rm(out_path)
+    :ok = :erl_tar.create(String.to_charlist(out_path), files, [:dereference, :compressed])
+    release
+  end
+
   # build_rel
 
   defp build_rel(release, config) do

lib/mix/lib/mix/tasks/release.init.ex

@@ -60,13 +60,19 @@ defmodule Mix.Tasks.Release.Init do
     #!/bin/sh
 
     # Sets and enables heart (recommended only in daemon mode)
-    # if [ "$RELEASE_COMMAND" = "daemon" ] || [ "$RELEASE_COMMAND" = "daemon_iex" ]; then
-    #   HEART_COMMAND="$RELEASE_ROOT/bin/$RELEASE_NAME $RELEASE_COMMAND"
-    #   export HEART_COMMAND
-    #   export ELIXIR_ERL_OPTIONS="-heart"
-    # fi
+    # case $RELEASE_COMMAND in
+    #   daemon*)
+    #     HEART_COMMAND="$RELEASE_ROOT/bin/$RELEASE_NAME $RELEASE_COMMAND"
+    #     export HEART_COMMAND
+    #     export ELIXIR_ERL_OPTIONS="-heart"
+    #     ;;
+    #   *)
+    #     ;;
+    # esac
 
-    # Set the release to work across nodes
+    # Set the release to work across nodes. If using the long name format like
+    # the one below (my_app@127.0.0.1), you need to also uncomment the
+    # RELEASE_DISTRIBUTION variable below.
     # export RELEASE_DISTRIBUTION=name
     # export RELEASE_NODE=<%= @release.name %>@127.0.0.1
     """
@@ -94,6 +100,8 @@ defmodule Mix.Tasks.Release.Init do
     export RELEASE_TMP="${RELEASE_TMP:-"$RELEASE_ROOT/tmp"}"
     export RELEASE_VM_ARGS="${RELEASE_VM_ARGS:-"$REL_VSN_DIR/vm.args"}"
     export RELEASE_DISTRIBUTION="${RELEASE_DISTRIBUTION:-"sname"}"
+    export RELEASE_BOOT_SCRIPT="${RELEASE_BOOT_SCRIPT:-"start"}"
+    export RELEASE_BOOT_SCRIPT_CLEAN="${RELEASE_BOOT_SCRIPT_CLEAN:-"start_clean"}"
 
     rand () {
       od -t xS -N 2 -A n /dev/urandom | tr -d " \n"
@@ -103,7 +111,7 @@ defmodule Mix.Tasks.Release.Init do
       exec "$REL_VSN_DIR/elixir" \
            --hidden --cookie "$RELEASE_COOKIE" \
            --$RELEASE_DISTRIBUTION "rpc-$(rand)-$RELEASE_NODE" \
-           --boot "$REL_VSN_DIR/start_clean" \
+           --boot "$REL_VSN_DIR/$RELEASE_BOOT_SCRIPT_CLEAN" \
            --boot-var RELEASE_LIB "$RELEASE_ROOT/lib" \
            --rpc-eval "$RELEASE_NODE" "$1"
     }
@@ -117,7 +125,7 @@ defmodule Mix.Tasks.Release.Init do
            --$RELEASE_DISTRIBUTION "$RELEASE_NODE" \
            --erl "-mode $RELEASE_MODE" \
            --erl-config "$RELEASE_SYS_CONFIG" \
-           --boot "$REL_VSN_DIR/start" \
+           --boot "$REL_VSN_DIR/$RELEASE_BOOT_SCRIPT" \
            --boot-var RELEASE_LIB "$RELEASE_ROOT/lib" \
            --vm-args "$RELEASE_VM_ARGS" "$@"
     }
@@ -164,7 +172,7 @@ defmodule Mix.Tasks.Release.Init do
         exec "$REL_VSN_DIR/elixir" \
            --cookie "$RELEASE_COOKIE" \
            --erl-config "$RELEASE_SYS_CONFIG" \
-           --boot "$REL_VSN_DIR/start_clean" \
+           --boot "$REL_VSN_DIR/$RELEASE_BOOT_SCRIPT_CLEAN" \
            --boot-var RELEASE_LIB "$RELEASE_ROOT/lib" \
            --vm-args "$RELEASE_VM_ARGS" --eval "$2"
         ;;
@@ -173,7 +181,7 @@ defmodule Mix.Tasks.Release.Init do
         exec "$REL_VSN_DIR/iex" \
              --werl --hidden --cookie "$RELEASE_COOKIE" \
              --$RELEASE_DISTRIBUTION "rem-$(rand)-$RELEASE_NODE" \
-             --boot "$REL_VSN_DIR/start_clean" \
+             --boot "$REL_VSN_DIR/$RELEASE_BOOT_SCRIPT_CLEAN" \
              --boot-var RELEASE_LIB "$RELEASE_ROOT/lib" \
              --remsh "$RELEASE_NODE"
         ;;
@@ -228,7 +236,9 @@ defmodule Mix.Tasks.Release.Init do
   def env_bat_text,
     do: ~S"""
     @echo off
-    rem Set the release to work across nodes
+    rem Set the release to work across nodes. If using the long name format like
+    rem the one below (my_app@127.0.0.1), you need to also uncomment the
+    rem RELEASE_DISTRIBUTION variable below.
     rem set RELEASE_DISTRIBUTION=name
     rem set RELEASE_NODE=<%= @release.name %>@127.0.0.1
     """
@@ -256,6 +266,8 @@ defmodule Mix.Tasks.Release.Init do
     if not defined RELEASE_TMP (set RELEASE_TMP=!RELEASE_ROOT!\tmp)
     if not defined RELEASE_VM_ARGS (set RELEASE_VM_ARGS=!REL_VSN_DIR!\vm.args)
     if not defined RELEASE_DISTRIBUTION (set RELEASE_DISTRIBUTION=sname)
+    if not defined RELEASE_BOOT_SCRIPT (set RELEASE_BOOT_SCRIPT=start)
+    if not defined RELEASE_BOOT_SCRIPT_CLEAN (set RELEASE_BOOT_SCRIPT_CLEAN=start_clean)
     set RELEASE_SYS_CONFIG=!REL_VSN_DIR!\sys
 
     if "%~1" == "start" (set "REL_EXEC=elixir" && set "REL_EXTRA=--no-halt" && set "REL_GOTO=start")
@@ -270,11 +282,11 @@ defmodule Mix.Tasks.Release.Init do
     )
 
     if not "!REL_GOTO!" == "" (
-      findstr "RUNTIME_CONFIG=true" "!RELEASE_SYS_CONFIG!.config" >nil 2>&1 && (
+      findstr "RUNTIME_CONFIG=true" "!RELEASE_SYS_CONFIG!.config" >nul 2>&1 && (
         for /f "skip=1" %%X in ('wmic os get localdatetime') do if not defined TIMESTAMP set TIMESTAMP=%%X
         set RELEASE_SYS_CONFIG=!RELEASE_TMP!\!RELEASE_NAME!-!RELEASE_VSN!-!TIMESTAMP:~0,11!-!RANDOM!.runtime
-        mkdir "!RELEASE_TMP!" >nil
-        copy /y "!REL_VSN_DIR!\sys.config" "!RELEASE_SYS_CONFIG!.config" >nil || (
+        mkdir "!RELEASE_TMP!" >nul 2>&1
+        copy /y "!REL_VSN_DIR!\sys.config" "!RELEASE_SYS_CONFIG!.config" >nul || (
           echo Cannot start release because it could not write to "!RELEASE_SYS_CONFIG!.config"
           goto end
         )
@@ -321,7 +333,7 @@ defmodule Mix.Tasks.Release.Init do
       --!RELEASE_DISTRIBUTION! "!RELEASE_NODE!" ^
       --erl "-mode !RELEASE_MODE!" ^
       --erl-config "!RELEASE_SYS_CONFIG!" ^
-      --boot "!REL_VSN_DIR!\start" ^
+      --boot "!REL_VSN_DIR!\!RELEASE_BOOT_SCRIPT!" ^
       --boot-var RELEASE_LIB "!RELEASE_ROOT!\lib" ^
       --vm-args "!RELEASE_VM_ARGS!"
     goto end
@@ -331,7 +343,7 @@ defmodule Mix.Tasks.Release.Init do
       --eval "%~2" ^
       --cookie "!RELEASE_COOKIE!" ^
       --erl-config "!RELEASE_SYS_CONFIG!" ^
-      --boot "!REL_VSN_DIR!\start_clean" ^
+      --boot "!REL_VSN_DIR!\!RELEASE_BOOT_SCRIPT_CLEAN!" ^
       --boot-var RELEASE_LIB "!RELEASE_ROOT!\lib" ^
       --vm-args "!RELEASE_VM_ARGS!"
     goto end
@@ -340,7 +352,7 @@ defmodule Mix.Tasks.Release.Init do
     "!REL_VSN_DIR!\iex.bat" ^
       --werl --hidden --cookie "!RELEASE_COOKIE!" ^
       --!RELEASE_DISTRIBUTION! "rem-!RANDOM!-!RELEASE_NODE!" ^
-      --boot "!REL_VSN_DIR!\start_clean" ^
+      --boot "!REL_VSN_DIR!\!RELEASE_BOOT_SCRIPT_CLEAN!" ^
       --boot-var RELEASE_LIB "!RELEASE_ROOT!\lib" ^
       --remsh "!RELEASE_NODE!"
     goto end
@@ -349,7 +361,7 @@ defmodule Mix.Tasks.Release.Init do
     "!REL_VSN_DIR!\elixir.bat" ^
       --hidden --cookie "!RELEASE_COOKIE!" ^
       --!RELEASE_DISTRIBUTION! "rpc-!RANDOM!-!RELEASE_NODE!" ^
-      --boot "!REL_VSN_DIR!\start_clean" ^
+      --boot "!REL_VSN_DIR!\!RELEASE_BOOT_SCRIPT_CLEAN!" ^
       --boot-var RELEASE_LIB "!RELEASE_ROOT!\lib" ^
       --rpc-eval "!RELEASE_NODE!" "!REL_RPC!"
     goto end
@@ -366,7 +378,8 @@ defmodule Mix.Tasks.Release.Init do
     )
 
     !ERLSRV! add !RELEASE_NAME!_!RELEASE_NAME! ^
-      -name "!RELEASE_NODE!" ^
+      -!RELEASE_DISTRIBUTION! "!RELEASE_NODE!" ^
+      -env RELEASE_ROOT=!RELEASE_ROOT! -env RELEASE_NAME=!RELEASE_NAME! -env RELEASE_VSN=!RELEASE_VSN! -env RELEASE_COOKIE=!RELEASE_COOKIE! -env RELEASE_NODE=!RELEASE_NODE! -env RELEASE_VM_ARGS=!RELEASE_VM_ARGS! -env RELEASE_TMP=!RELEASE_TMP! -env RELEASE_SYS_CONFIG=!RELEASE_SYS_CONFIG! ^
       -args "-setcookie !RELEASE_COOKIE! -config !RELEASE_SYS_CONFIG! -mode !RELEASE_MODE! -boot !REL_VSN_DIR!\start -boot_var RELEASE_LIB !RELEASE_ROOT!\lib -args_file !REL_VSN_DIR!\vm.args"
 
     if %ERRORLEVEL% EQU 0 (

lib/mix/lib/mix/utils.ex

@@ -676,4 +676,38 @@ defmodule Mix.Utils do
 
     [proxy_auth: {user, pass}]
   end
+
+  def warn_install_over_http_deprecated(task_name, url) do
+    basename = Path.basename(url)
+
+    shell = Mix.shell()
+
+    shell.error("""
+    Warning: the use of HTTP/HTTPS URLs with `mix #{task_name}` is deprecated
+
+    Run `mix help #{task_name}` for details on the available alternatives, using
+    hex, git or github. Alternatively you can fetch the file using an external HTTP
+    client and then install it locally:
+
+    Unix (Linux, MacOS X):
+
+        $ wget #{url}
+        $ mix #{task_name} #{basename}
+
+    or
+
+        $ curl -o #{basename} #{url}
+        $ mix #{task_name} #{basename}
+
+    Windows (Win7 or later):
+
+        > powershell -Command "Invoke-WebRequest #{url} -OutFile #{basename}"
+        > mix #{task_name} #{basename}
+
+    or
+
+        > powershell -Command "(New-Object Net.WebClient).DownloadFile('#{url}', '#{basename}')"
+        > mix #{task_name} #{basename}
+    """)
+  end
 end

lib/mix/test/fixtures/release_test/config/config.exs

@@ -1,3 +1,4 @@
 use Mix.Config
 
 config :release_test, :static, :was_set
+config :release_test, :encoding, {:"£", "£", '£'}

lib/mix/test/fixtures/release_test/lib/release_test.ex

@@ -19,11 +19,12 @@ defmodule ReleaseTest do
       static_config: Application.fetch_env(:release_test, :static),
       runtime_config: Application.fetch_env(:release_test, :runtime),
       sys_config_init: to_string(sys_config),
-      sys_config_env: System.get_env("RELEASE_SYS_CONFIG")
+      sys_config_env: System.get_env("RELEASE_SYS_CONFIG"),
+      encoding: Application.get_env(:release_test, :encoding)
     }
 
     path = Path.join(System.get_env("RELEASE_ROOT"), "RELEASE_BOOTED")
-    File.write!(path, inspect(info))
+    File.write!(path, :erlang.term_to_binary(info))
 
     if System.get_env("RELEASE_NAME") =~ "permanent" do
       Supervisor.start_link([], strategy: :one_for_one)

lib/mix/test/mix/cli_test.exs

@@ -143,6 +143,32 @@ defmodule Mix.CLITest do
     System.delete_env("MIX_EXS")
   end
 
+  test "target config defaults to the user's preferred cli target", context do
+    in_tmp(context.test, fn ->
+      File.write!("custom.exs", """
+      defmodule P do
+        use Mix.Project
+        def project, do: [app: :p, version: "0.1.0", preferred_cli_target: [test_task: :other]]
+      end
+
+      defmodule Mix.Tasks.TestTask do
+        use Mix.Task
+
+        def run(args) do
+          IO.inspect {Mix.target, args}
+        end
+      end
+      """)
+
+      System.put_env("MIX_EXS", "custom.exs")
+
+      output = mix(["test_task", "a", "b", "c"])
+      assert output =~ ~s({:other, ["a", "b", "c"]})
+    end)
+  after
+    System.delete_env("MIX_EXS")
+  end
+
   test "new with tests and cover" do
     in_tmp("new_with_tests", fn ->
       output = mix(~w[new .])

lib/mix/test/mix/release_test.exs

@@ -50,6 +50,21 @@ defmodule Mix.ReleaseTest do
       assert release.options[:quiet]
     end
 
+    test "allows specifying the version from an application" do
+      overrides = [version: {:from_app, :elixir}]
+      release = from_config!(nil, config(), overrides)
+
+      assert release.version == to_string(Application.spec(:elixir, :vsn))
+    end
+
+    test "raises when :from_app is used with an app that doesn't exist" do
+      overrides = [version: {:from_app, :not_valid}]
+
+      assert_raise Mix.Error,
+                   ~r"Could not find version for :not_valid, please make sure the application exists",
+                   fn -> from_config!(nil, config(), overrides) end
+    end
+
     test "includes applications" do
       release = from_config!(nil, config(), [])
       assert release.applications.mix[:path] == to_charlist(Application.app_dir(:mix))
@@ -80,7 +95,7 @@ defmodule Mix.ReleaseTest do
     test "uses chosen release via the default_release" do
       release =
         from_config!(
-          :bar,
+          nil,
           config(
             default_release: :bar,
             releases: [foo: [version: "0.2.0"], bar: [version: "0.3.0"]]
@@ -153,6 +168,14 @@ defmodule Mix.ReleaseTest do
                    ~r"The :steps option must contain the atom :assemble once",
                    fn -> release(steps: [:assemble, :assemble]) end
 
+      assert_raise Mix.Error,
+                   ~r"The :tar step must come after :assemble",
+                   fn -> release(steps: [:tar, :assemble]) end
+
+      assert_raise Mix.Error,
+                   ~r"The :steps option can only contain the atom :tar once",
+                   fn -> release(steps: [:assemble, :tar, :tar]) end
+
       assert_raise Mix.Error,
                    ~r"The :steps option must be",
                    fn -> release(steps: [:foo]) end
@@ -437,6 +460,14 @@ defmodule Mix.ReleaseTest do
       assert contents =~ "[{foo,[{bar,baz}]}]."
     end
 
+    test "writes sys_config with encoding" do
+      assert make_sys_config(release([]), [encoding: {:"£", "£", '£'}], "unused/runtime/path") ==
+               :ok
+
+      {:ok, contents} = :file.consult(@sys_config)
+      assert contents == [[encoding: {:"£", "£", '£'}]]
+    end
+
     test "writes the given sys_config with config providers" do
       release = release(config_providers: @providers)
       assert make_sys_config(release, [kernel: [key: :value]], "/foo/bar/bat") == :ok
@@ -569,6 +600,54 @@ defmodule Mix.ReleaseTest do
     end
   end
 
+  describe "included applications" do
+    test "are included in the release", context do
+      in_tmp(context.test, fn ->
+        app =
+          {:application, :my_sample1,
+           applications: [:kernel, :stdlib, :elixir],
+           description: 'my_sample1',
+           modules: [],
+           vsn: '1.0.0',
+           included_applications: [:runtime_tools]}
+
+        File.mkdir_p!("my_sample1/ebin")
+        Code.prepend_path("my_sample1/ebin")
+        format = :io_lib.format("%% coding: utf-8~n~p.~n", [app])
+        File.write!("my_sample1/ebin/my_sample1.app", format)
+
+        release = release(applications: [my_sample1: :permanent])
+        assert release.applications.runtime_tools[:included]
+        assert release.boot_scripts.start[:runtime_tools] == :load
+
+        release = release(applications: [my_sample1: :permanent, runtime_tools: :none])
+        assert release.applications.runtime_tools[:included]
+        assert release.boot_scripts.start[:runtime_tools] == :none
+      end)
+    end
+
+    test "raise on conflict", context do
+      in_tmp(context.test, fn ->
+        app =
+          {:application, :my_sample2,
+           applications: [:kernel, :stdlib, :elixir, :runtime_tools],
+           description: 'my_sample',
+           modules: [],
+           vsn: '1.0.0',
+           included_applications: [:runtime_tools]}
+
+        File.mkdir_p!("my_sample2/ebin")
+        Code.prepend_path("my_sample2/ebin")
+        format = :io_lib.format("%% coding: utf-8~n~p.~n", [app])
+        File.write!("my_sample2/ebin/my_sample2.app", format)
+
+        assert_raise Mix.Error,
+                     ":runtime_tools is listed both as a regular application and as an included application",
+                     fn -> release(applications: [my_sample2: :permanent]) end
+      end)
+    end
+  end
+
   defp size!(path) do
     File.stat!(path).size
   end

lib/mix/test/mix/tasks/app.start_test.exs

@@ -19,7 +19,6 @@ defmodule Mix.Tasks.App.StartTest do
     end
   end
 
-  @tag apps: [:app_start_sample]
   test "compiles and starts the project" do
     Mix.Project.push(AppStartSample)
 
@@ -50,7 +49,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:app_start_sample, :app_loaded_sample]
   test "start checks for invalid configuration", context do
     Mix.Project.push(AppStartSample)
 
@@ -77,7 +75,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:error]
   test "validates Elixir version requirement", context do
     Mix.ProjectStack.post_config(elixir: "~> ~> 0.8.1")
     Mix.Project.push(WrongElixirProject)
@@ -89,7 +86,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:error]
   test "validates the Elixir version with requirement", context do
     Mix.Project.push(WrongElixirProject)
 
@@ -100,7 +96,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:error]
   test "does not validate the Elixir version with requirement when disabled", context do
     Mix.Project.push(WrongElixirProject)
 
@@ -136,7 +131,6 @@ defmodule Mix.Tasks.App.StartTest do
     def start(_type, return), do: return
   end
 
-  @tag apps: [:return_sample]
   test "start points to report on error", context do
     Mix.Project.push(ReturnSample)
 
@@ -155,7 +149,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:return_sample]
   test "start points to report on exception error", context do
     Mix.Project.push(ReturnSample)
 
@@ -177,7 +170,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:return_sample]
   test "start points to report on bad return", context do
     Mix.Project.push(ReturnSample)
 
@@ -212,7 +204,6 @@ defmodule Mix.Tasks.App.StartTest do
     def start(_type, reason), do: exit(reason)
   end
 
-  @tag apps: [:exit_sample]
   test "start points to report on exit", context do
     Mix.Project.push(ExitSample)
 
@@ -230,7 +221,6 @@ defmodule Mix.Tasks.App.StartTest do
     end)
   end
 
-  @tag apps: [:exit_sample]
   test "start points to report on normal exit", context do
     Mix.Project.push(ExitSample)
 

lib/mix/test/mix/tasks/archive_test.exs

@@ -82,6 +82,8 @@ defmodule Mix.Tasks.ArchiveTest do
       # Try to override it with URL
       send(self(), {:mix_shell_input, :yes?, false})
       Mix.Tasks.Archive.Install.run(["https://example.com/archive-0.1.0?hello.ez"])
+
+      assert_received {:mix_shell, :error, ["Warning: the use of HTTP/HTTPS URLs" <> _]}
       assert_received {:mix_shell, :yes?, ["Found existing entry: " <> _]}
 
       # Loading the archive should emit warning again
@@ -116,7 +118,7 @@ defmodule Mix.Tasks.ArchiveTest do
   end
 
   test "archive install missing file" do
-    message = ~r[Expected "./unlikely-to-exist-0.1.0.ez" to be a URL or a local file path]
+    message = ~r[Expected "./unlikely-to-exist-0.1.0.ez" to be a local file path]
 
     assert_raise Mix.Error, message, fn ->
       Mix.Tasks.Archive.Install.run(["./unlikely-to-exist-0.1.0.ez"])

lib/mix/test/mix/tasks/escript_test.exs

@@ -242,6 +242,13 @@ defmodule Mix.Tasks.EscriptTest do
       assert_received {:mix_shell, :info, ["* escript_test"]}
       refute_received {:mix_shell, :info, ["* escript_test.bat"]}
 
+      # Try to override it with URL
+      send(self(), {:mix_shell_input, :yes?, false})
+      Mix.Tasks.Escript.Install.run(["https://example.com/escript_test"])
+
+      assert_received {:mix_shell, :error, ["Warning: the use of HTTP/HTTPS URLs" <> _]}
+      assert_received {:mix_shell, :yes?, ["Found existing entry: " <> _]}
+
       # check uninstall confirmation
       send(self(), {:mix_shell_input, :yes?, false})
       Mix.Tasks.Escript.Uninstall.run(["escript_test"])

lib/mix/test/mix/tasks/profile.cprof_test.exs

@@ -4,10 +4,8 @@ defmodule Mix.Tasks.Profile.CprofTest do
   use MixTest.Case
 
   import ExUnit.CaptureIO
-
   alias Mix.Tasks.Profile.Cprof
 
-  @moduletag apps: [:sample]
   @expr "Enum.each(1..5, &String.Chars.Integer.to_string/1)"
 
   test "profiles evaluated expression", context do

lib/mix/test/mix/tasks/profile.eprof_test.exs

@@ -4,10 +4,8 @@ defmodule Mix.Tasks.Profile.EprofTest do
   use MixTest.Case
 
   import ExUnit.CaptureIO
-
   alias Mix.Tasks.Profile.Eprof
 
-  @moduletag apps: [:sample]
   @expr "Enum.each(1..5, &String.Chars.Integer.to_string/1)"
 
   test "profiles evaluated expression", context do

lib/mix/test/mix/tasks/profile.fprof_test.exs

@@ -4,9 +4,7 @@ defmodule Mix.Tasks.Profile.FprofTest do
   use MixTest.Case
 
   import ExUnit.CaptureIO
-
   alias Mix.Tasks.Profile.Fprof
-  @moduletag apps: [:sample]
 
   test "profiles evaluated expression", context do
     in_tmp(context.test, fn ->

lib/mix/test/mix/tasks/release_test.exs

@@ -36,6 +36,49 @@ defmodule Mix.Tasks.ReleaseTest do
       end)
     end
 
+    test "tar" do
+      in_fixture("release_test", fn ->
+        config = [releases: [demo: [steps: [:assemble, :tar]]]]
+
+        Mix.Project.in_project(:release_test, ".", config, fn _ ->
+          root = Path.absname("_build/#{Mix.env()}/rel/demo")
+
+          ignored_app_path = Path.join([root, "lib", "ignored_app-0.1.0", "ebin"])
+          File.mkdir_p!(ignored_app_path)
+          File.touch(Path.join(ignored_app_path, "ignored_app.app"))
+
+          ignored_release_path = Path.join([root, "releases", "ignored_dir"])
+          File.mkdir_p!(ignored_release_path)
+          File.touch(Path.join(ignored_release_path, "ignored"))
+
+          Mix.Task.run("release")
+          tar_path = Path.expand(Path.join([root, "..", "..", "demo-0.1.0.tar.gz"]))
+          message = "* building #{tar_path}"
+          assert_received {:mix_shell, :info, [^message]}
+          assert File.exists?(tar_path)
+
+          {:ok, files} = String.to_charlist(tar_path) |> :erl_tar.table([:compressed])
+
+          files = Enum.map(files, &to_string/1)
+          files_with_versions = File.ls!(Path.join(root, "lib"))
+
+          assert "bin/demo" in files
+          assert "releases/0.1.0/sys.config" in files
+          assert "releases/0.1.0/vm.args" in files
+          assert "releases/COOKIE" in files
+          assert "releases/start_erl.data" in files
+
+          for dir <- files_with_versions -- ["ignored_app-0.1.0"] do
+            [name | _] = String.split(dir, "-")
+            assert "lib/#{dir}/ebin/#{name}.app" in files
+          end
+
+          refute "lib/ignored_app-0.1.0/ebin/ignored_app.app" in files
+          refute "releases/ignored_dir/ignored" in files
+        end)
+      end)
+    end
+
     test "steps" do
       in_fixture("release_test", fn ->
         last_step = fn release ->
@@ -140,8 +183,9 @@ defmodule Mix.Tasks.ReleaseTest do
                  static_config: {:ok, :was_set},
                  runtime_config: :error,
                  sys_config_env: sys_config_env,
-                 sys_config_init: sys_config_init
-               } = wait_until_evaled(Path.join(root, "RELEASE_BOOTED"))
+                 sys_config_init: sys_config_init,
+                 encoding: {:"£", "£", '£'}
+               } = wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
 
         if match?({:win32, _}, :os.type()) do
           # `RELEAS~1` is the DOS path name (8 character) for the `release_test` directory
@@ -169,6 +213,7 @@ defmodule Mix.Tasks.ReleaseTest do
         File.write!("config/releases.exs", """
         import Config
         config :release_test, :runtime, :was_set
+        config :release_test, :encoding, {:runtime, :"£", "£", '£'}
         """)
 
         root = Path.absname("_build/dev/rel/runtime_config")
@@ -198,8 +243,9 @@ defmodule Mix.Tasks.ReleaseTest do
                  static_config: {:ok, :was_set},
                  runtime_config: {:ok, :was_set},
                  sys_config_env: sys_config_env,
-                 sys_config_init: sys_config_init
-               } = wait_until_evaled(Path.join(root, "RELEASE_BOOTED"))
+                 sys_config_init: sys_config_init,
+                 encoding: {:runtime, :"£", "£", '£'}
+               } = wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
 
         if match?({:win32, _}, :os.type()) do
           assert sys_config_env =~ "tmp\\runtime_config-0.1.0"
@@ -250,7 +296,7 @@ defmodule Mix.Tasks.ReleaseTest do
                  root_dir: root_dir,
                  static_config: {:ok, :was_set},
                  runtime_config: :error
-               } = wait_until_evaled(Path.join(root, "RELEASE_BOOTED"))
+               } = wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
 
         if match?({:win32, _}, :os.type()) do
           assert String.ends_with?(app_dir, "demo/lib/release_test-0.1.0")
@@ -265,6 +311,7 @@ defmodule Mix.Tasks.ReleaseTest do
     end)
   end
 
+  @tag :epmd
   test "executes rpc instructions" do
     in_fixture("release_test", fn ->
       config = [releases: [permanent1: [include_erts: false]]]
@@ -275,12 +322,13 @@ defmodule Mix.Tasks.ReleaseTest do
         script = Path.join(root, "bin/permanent1")
 
         open_port(script, ['start'])
-        wait_until_evaled(Path.join(root, "RELEASE_BOOTED"))
+        wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
         assert System.cmd(script, ["rpc", "ReleaseTest.hello_world"]) == {"hello world\n", 0}
-        assert System.cmd(script, ["stop"]) == {"", 0}
 
         assert {pid, 0} = System.cmd(script, ["pid"])
         assert pid != "\n"
+
+        assert System.cmd(script, ["stop"]) == {"", 0}
       end)
     end)
   end
@@ -324,7 +372,7 @@ defmodule Mix.Tasks.ReleaseTest do
                  release_vsn: "0.1.0",
                  static_config: {:ok, :was_set},
                  runtime_config: {:ok, :was_set}
-               } = wait_until_evaled(Path.join(root, "RELEASE_BOOTED"))
+               } = wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
       end)
     end)
   end
@@ -341,22 +389,27 @@ defmodule Mix.Tasks.ReleaseTest do
         script = Path.join(root, "bin/permanent2")
         open_port(script, ['daemon_iex'])
 
-        assert wait_until_evaled(Path.join(root, "RELEASE_BOOTED")) == %{
-                 app_dir: Path.join(root, "lib/release_test-0.1.0"),
+        assert %{
+                 app_dir: app_dir,
                  cookie_env: "abcdefghij",
                  mode: :embedded,
-                 node: :"permanent2@#{@hostname}",
+                 node: release_node("permanent2"),
                  protocols_consolidated?: true,
                  release_name: "permanent2",
                  release_node: "permanent2",
-                 release_root: root,
+                 release_root: ^root,
                  release_vsn: "0.1.0",
-                 root_dir: :code.root_dir() |> to_string(),
+                 root_dir: root_dir,
                  static_config: {:ok, :was_set},
                  runtime_config: :error,
-                 sys_config_env: Path.join(root, "releases/0.1.0/sys"),
-                 sys_config_init: Path.join(root, "releases/0.1.0/sys")
-               }
+                 sys_config_env: sys_config_env,
+                 sys_config_init: sys_config_init
+               } = wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
+
+        assert app_dir == Path.join(root, "lib/release_test-0.1.0")
+        assert root_dir == :code.root_dir() |> to_string()
+        assert sys_config_env == Path.join(root, "releases/0.1.0/sys")
+        assert sys_config_init == Path.join(root, "releases/0.1.0/sys")
 
         assert wait_until(fn ->
                  File.read!(Path.join(root, "tmp/log/erlang.log.1")) =~
@@ -402,8 +455,13 @@ defmodule Mix.Tasks.ReleaseTest do
     Port.open({:spawn_executable, to_charlist(command)}, [:hide, args: args])
   end
 
-  defp wait_until_evaled(file) do
-    wait_until(fn -> File.exists?(file) && Code.eval_file(file) |> elem(0) end)
+  defp wait_until_decoded(file) do
+    wait_until(fn ->
+      case File.read(file) do
+        {:ok, bin} when byte_size(bin) > 0 -> :erlang.binary_to_term(bin)
+        _ -> nil
+      end
+    end)
   end
 
   defp wait_until(fun) do

lib/mix/test/mix/tasks/run_test.exs

@@ -5,8 +5,6 @@ defmodule Mix.Tasks.RunTest do
 
   import ExUnit.CaptureIO
 
-  @moduletag apps: [:sample]
-
   setup do
     Mix.Project.push(MixTest.Case.Sample)
   end

lib/mix/test/mix/umbrella_test.exs

@@ -3,8 +3,6 @@ Code.require_file("../test_helper.exs", __DIR__)
 defmodule Mix.UmbrellaTest do
   use MixTest.Case
 
-  @moduletag apps: [:foo, :bar]
-
   test "apps_paths" do
     in_fixture("umbrella_dep/deps/umbrella", fn ->
       assert Mix.Project.apps_paths() == nil

lib/mix/test/test_helper.exs

@@ -2,8 +2,12 @@ Mix.start()
 Mix.shell(Mix.Shell.Process)
 Application.put_env(:mix, :colors, enabled: false)
 
-exclude = if match?({:win32, _}, :os.type()), do: [unix: true], else: [windows: true]
-ExUnit.start(trace: "--trace" in System.argv(), exclude: exclude)
+Logger.remove_backend(:console)
+Application.put_env(:logger, :backends, [])
+
+os_exclude = if match?({:win32, _}, :os.type()), do: [unix: true], else: [windows: true]
+epmd_exclude = if match?({_, 0}, System.cmd("epmd", ["-daemon"])), do: [], else: [epmd: true]
+ExUnit.start(trace: "--trace" in System.argv(), exclude: epmd_exclude ++ os_exclude)
 
 unless {1, 7, 4} <= Mix.SCM.Git.git_version() do
   IO.puts(:stderr, "Skipping tests with git sparse checkouts...")
@@ -32,12 +36,9 @@ defmodule MixTest.Case do
     end
   end
 
-  setup config do
-    if apps = config[:apps] do
-      Logger.remove_backend(:console)
-      Application.put_env(:logger, :backends, [])
-    end
+  @apps Enum.map(Application.loaded_applications(), &elem(&1, 0))
 
+  setup do
     on_exit(fn ->
       Application.start(:logger)
       Mix.env(:dev)
@@ -48,14 +49,9 @@ defmodule MixTest.Case do
       Mix.ProjectStack.clear_stack()
       delete_tmp_paths()
 
-      if apps do
-        for app <- apps do
-          Application.stop(app)
-          Application.unload(app)
-        end
-
-        Logger.add_backend(:console, flush: true)
-        Application.put_env(:logger, :backends, [:console])
+      for {app, _, _} <- Application.loaded_applications(), app not in @apps do
+        Application.stop(app)
+        Application.unload(app)
       end
     end)