The idea is to provide them as alternative to bat scripts and potentially
replace all bat scripts in the future. One benefit of using PowerShell is
that we can also run it on Linux systems, facilitating dev and testing.
In order to have more deterministic test runs when using `--max-cases 1`
and `--max-requires 1` (https://github.com/elixir-lang/elixir/pull/13635)
(see also https://github.com/elixir-lang/elixir/pull/13589), we need to
run tests in compilation order (FIFO).
In the past, ExUnit.Server appended new tests to the front of a list,
which would result in the most recently added test to be run first.
Let's quickly demonstrate the problem this causes for deterministic runs
with a simple example:
Imagine a test (let's call if FooTest) that takes a non-deterministic
amount of time to run. For now let's assume that it sometimes takes 1
second and sometimes up to 5. And as async tests execute in parallel with
compilation of other test files, we could have the following scenario:
FooTest is compiled and because it's async it is immediately started.
It takes 1 second to run.
In this 1 second two more tests are compiled. First BarTest is prepended
to the list, then BazTest.
The order of test runs now is:
FooTest, BazTest, then whatever is last compiled while BazTest runs, ...
Now another run, FooTest takes 5 seconds to run.
While FooTest runs, more than two other tests are compiled.
The order of test runs is:
FooTest, LastCompiledTest, SecondLastCompiledTest, ..., BazTest, BarTest
This can be fixed either by appending new test modules to the end of the
list, or - and that's what this commit does - by using a `:queue`
instead.
Sometimes you want to run the same tests but with different parameters.
In ExUnit, it is possible to do so by passing a `:parameterize` key to
`ExUnit.Case`. The value must be a list of maps which will be the
parameters merged into the test context.
For example, Elixir has a module called `Registry`, which can have type
`:unique` or `:duplicate`, and can control its concurrency factor using
the `:partitions` option. If you have a number of tests that *behave the
same* across all of those values, you can parameterize those tests with:
use ExUnit.Case,
async: true,
parameterize:
for(kind <- [:unique, :duplicate],
partitions <- [1, 8],
do: %{kind: kind, partitions: partitions})
Then, in your tests, you can access the parameters as part of the context:
test "starts a registry", %{kind: kind, partitions: partitions} do
...
end
Use parameterized tests with care:
* Although parameterized tests run concurrently when `async: true` is also given,
abuse of parameterized tests may make your test suite slower
* If you use parameterized tests and then find yourself adding conditionals
in your tests to deal with different parameters, then parameterized tests
may be the wrong solution to your problem. Consider creating separated
tests and sharing logic between them using regular functions
Jean Klingler
José Valim
Ben Murden
Philip Sampaio
Maksym Verbovyi
Lukasz Samson
Andrea Leopardi
Vsevolog Grigoriev
Theodor Fiedler
Steffen Deusch
liamwhite
Daniel Kukula
Tangui