Before this patch, if you invoked:
mix test test/foo_test.exs:13 test/bar_test.exs
We would include a location filter only for foo_test.exs
and no tests in bar_test.exs would run. We now ensure
that the full location of files with line filters are
included as tags, as long as any file is included.
Closes#14445.
When a process that traps exits used ExUnit's log capture functionality,
it would receive stray exits from the Tasks started inside it that could
lead to issues if not handled.
In order to have more deterministic test runs when using `--max-cases 1`
and `--max-requires 1` (https://github.com/elixir-lang/elixir/pull/13635)
(see also https://github.com/elixir-lang/elixir/pull/13589), we need to
run tests in compilation order (FIFO).
In the past, ExUnit.Server appended new tests to the front of a list,
which would result in the most recently added test to be run first.
Let's quickly demonstrate the problem this causes for deterministic runs
with a simple example:
Imagine a test (let's call if FooTest) that takes a non-deterministic
amount of time to run. For now let's assume that it sometimes takes 1
second and sometimes up to 5. And as async tests execute in parallel with
compilation of other test files, we could have the following scenario:
FooTest is compiled and because it's async it is immediately started.
It takes 1 second to run.
In this 1 second two more tests are compiled. First BarTest is prepended
to the list, then BazTest.
The order of test runs now is:
FooTest, BazTest, then whatever is last compiled while BazTest runs, ...
Now another run, FooTest takes 5 seconds to run.
While FooTest runs, more than two other tests are compiled.
The order of test runs is:
FooTest, LastCompiledTest, SecondLastCompiledTest, ..., BazTest, BarTest
This can be fixed either by appending new test modules to the end of the
list, or - and that's what this commit does - by using a `:queue`
instead.
Sometimes you want to run the same tests but with different parameters.
In ExUnit, it is possible to do so by passing a `:parameterize` key to
`ExUnit.Case`. The value must be a list of maps which will be the
parameters merged into the test context.
For example, Elixir has a module called `Registry`, which can have type
`:unique` or `:duplicate`, and can control its concurrency factor using
the `:partitions` option. If you have a number of tests that *behave the
same* across all of those values, you can parameterize those tests with:
use ExUnit.Case,
async: true,
parameterize:
for(kind <- [:unique, :duplicate],
partitions <- [1, 8],
do: %{kind: kind, partitions: partitions})
Then, in your tests, you can access the parameters as part of the context:
test "starts a registry", %{kind: kind, partitions: partitions} do
...
end
Use parameterized tests with care:
* Although parameterized tests run concurrently when `async: true` is also given,
abuse of parameterized tests may make your test suite slower
* If you use parameterized tests and then find yourself adding conditionals
in your tests to deal with different parameters, then parameterized tests
may be the wrong solution to your problem. Consider creating separated
tests and sharing logic between them using regular functions
Some CI environments may report the wrong number
of cores to the Erlang VM. So we choose to print
the number of max cases along side the seed
(both which affect randomness of tests) at the
top of each suite.
José Valim
Rudolf Manusadzhian
Jonatan Männchen
Steffen Deusch
Jean Klingler
Daniel Kukula
eeff
Marcus Kruse
Kieran
Dino Kovač