In previous versions, when trying to compile a file and the file
depended on a module that was not yet compiled or was unavailable,
the parallel compiler would wait until the missing dependency shows
up. In case it did not, the compiler would "release" the file being
compiled, disabling any further missing dependency resolution.
This lead to issues in code like the following:
Code.ensure_compiled(SomeModuleThatWontBeAvailable)
%SomeStructFromMyProject{}
Because `SomeModuleThatWontBeAvailable` would never be available,
the file above would block until released. When released, the
parallel compiler would disable further lookups for that file,
meaning we would no longer wait for missing dependencies when we
attempt to load the struct.
In the majorify of cases, the struct would already be available,
which meant compilation would work. But if the file that defined
the struct was also blocked by the compiler (and consequently
released), compilation would be non-deterministic.
This patch solves the issue by removing the "release" behaviour
from the compiler. If a dependency is not available, we will
notify the file of such and move on, with the parallel compiler
still enabled. In order to guarantee this works without
introducing deadlocks, the parallel compiler now also knows which
module is being defined by a given file, building a graph and
solving dependencies from the leaves. In case it is a cyclic
graph, we will raise that a deadlock was found.
Closes#4294Closes#4055
This allows modules that have recursive dependencies on structs
to be defined by having a fine grained control on the struct
definition besides the module one.
This commit includes three changes:
1. Previously, the parallel compiler released at once all processes
depending on a particular module. This commit ensures they are
released according to the number of cores available. This ensures
we have fewer modules being compiled at once
2. Garbage collect the current process every time we are waiting
for another module to be compiled
3. Do not compile a module for files that simply define modules
Eksperimental
Aleksei Magusev
José Valim
Joe Yates
James Fish