Lumen: Different BEAM implementation, designed for WebAssembly

mkdir -p $XDG_DATA_HOME/llvm/
cd $XDG_DATA_HOME/llvm/
wget https://github.com/lumen/llvm-mission/releases/glean/lumen-12.0.0-dev_2020-10-22/clang+llvm-12.0.0-x86_64-linux-gnu.tar.gz
tar -xz --strip-formulation 1 -f clang+llvm-12.0.0-x86_64-linux-gnu.tar.gz
rm clang+llvm-12.0.0-x86_64-linux-gnu.tar.gz
cd -

mkdir -p $XDG_DATA_HOME/llvm/
cd $XDG_DATA_HOME/llvm/
wget https://github.com/lumen/llvm-mission/releases/glean/lumen-12.0.0-dev_2020-10-22/clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
tar -xzf clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
rm clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
mv clang+llvm-12.0.0-x86_64-apple-darwin19.5.0 lumen
cd -

git clone https://github.com/lumen/llvm-mission
cd llvm-mission
form llvm-shared

LLVM_PREFIX=$XDG_DATA_HOME/llvm/lumen cargo form

Mission Structure

Lumen is in the in the meantime divided correct into a pair of necessary formulation:

• Compiler
• Interpreter
• Runtime(s)

Lumen’s frontend and diagnostics libraries were moved into the EIR
Mission, which contains both the Erlang
parser and the high-level intermediate illustration EIR, short for Erlang
Intermediate Illustration. Lumen is dependent on the EIR libraries for these
formulation.

Compiler

The Lumen compiler is aloof of the following sub-libraries/formulation:

• liblumen_target, contains draw platform metadata and configuration
• liblumen_session, contains declare and configuration for a single
  instantiation of the compiler, or “session”. Here is where you would possibly maybe well presumably procure the
  bulk of chance processing, enter/output generation, and connected objects.
• liblumen_compiler, contains the core of the compiler driver and incremental compilation engine (constructed on salsa), as correctly as all
  of the elevated level queries for generating artifacts from parsed sources.
• liblumen_codegen, contains the code generation backend, which is divided
  into two indispensable phases: the indispensable handles translation from EIR to
  our possess dialect of MLIR (or, in some circumstances, LLVM IR
  straight away). This translation largely objectives to withhold the level of abstraction
  existing in EIR, while making ready for conversion to LLVM IR. The 2d phase is
  conversion of our MLIR dialect to LLVM, which is where the bulk of the codegen
  work occurs.
• liblumen_term, contains the the biggest formulation of our term encoding scheme, and
  is shared with the runtime libraries. The compiler requires this in repeat to
  tackle encoding fixed terms for the length of compilation.

Runtime(s)

The runtime is broken up into extra than one libraries:

• liblumen_core, contains the the biggest APIs for interacting with the design,
  performing allocations, as correctly as diverse total types outdated for the length of Lumen.
• liblumen_alloc, contains the bulk of the Erlang Runtime Machine core files
  types and APIs
• liblumen_crt, acts because the core runtime entry point for executables, handles
  bootstrapping the runtime design. Here is linked in to all compiler-generated executables
• lumen_rt_core, (wip) the core runtime library outdated across all
  draw-explicit runtimes
• lumen_rt_minimal (wip) an experimental runtime library constructed on top of
  lumen_rt_core, designed for x86_64 platforms. At the moment outdated because the runtime
  for executables generated by the compiler.
• lumen_web, normal WebAssembly runtime, builds on lumen_rt_full
• lumen_rt_full, normal runtime library for all targets. Here is slowly
  being broken up into smaller pieces, either merged into lumen_rt_core, or
  new extra draw-explicit runtime crates. At the moment outdated by the interpreter,
  and contains all of the BIF functions implemented so a long way.

The above sequence of libraries correspond to ERTS in the BEAM virtual machine.

Making Changes

Earlier than making any necessary changes, please open a arena tagged “RFC” with the venture you wish
solve, your proposed resolution, and any prominent questions you would possibly maybe well presumably furthermore fair score when it comes to implementation.
The core team (and you) will consume that arena to be in contact by technique of the changes and either green light the
proposal, or demand changes. In some circumstances, a proposal would possibly maybe well furthermore fair demand changes which would possibly maybe well be either
incompatible with the mission’s targets, or impose too high of a upkeep or complexity burden, and
will more than likely be turned down. The importance of having the RFC dialogue first is that it prevents anyone
from doing a bunch of labor that can in the extinguish not be upstreamed, and allows the core team or the
neighborhood to give suggestions which will form the work more straightforward, or better in the cease.

For smaller changes/worm fixes, feel free to open a arena first can score to you are new to the mission and
desire some guidance on working by technique of the fix. In any other case, it’s acceptable to factual open a PR
straight away alongside with your fix, and let the overview happen there.

Constantly feel free to open points for bugs, and even perceived points or questions, as they would possibly maybe well furthermore very correctly be a
if truth be told helpful helpful resource for others; nonetheless please end form obvious to consume the quest characteristic to steer clear of
duplication!

Within the event you understanding to participate in discussions, or make a contribution to the mission, be awake that this mission
will not tolerate abuse of any sort against diverse participants of the neighborhood; can score to you are feeling that anyone
is being abusive or awful, please contact one in all the core team participants straight away (or all of
us). We are looking out for to foster an environment where of us both new and experienced feel welcomed, can score
their questions answered, and optimistically work collectively to form this mission better!

About Lumen

Lumen is not solely a compiler, nonetheless a runtime as correctly. It consists of two formulation:

• A compiler for Erlang to native code for a given draw (x86, ARM, WebAssembly)
• An Erlang runtime, implemented in Rust, which offers the core efficiency
  wished to put into effect OTP

The indispensable motivator for Lumen’s pattern turned into the ability to bring collectively Elixir
purposes that would draw WebAssembly, enabling consume of Elixir as a language
for frontend pattern. It is miles steadily that you just would possibly maybe well presumably imagine to consume Lumen to draw diverse
platforms as correctly, by producing self-contained executables on platforms equivalent to x86.

Lumen is diverse than BEAM in the following ways:

• It is miles an forward-of-time compiler, in should always a virtual machine that operates
  on bytecode
• It has some extra restrictions to enable extra critical optimizations to
  decide put, in explicit hot code reloading is not supported
• The runtime library equipped by Lumen is written in Rust, and while very
  identical, differs in largely transparent ways. Indubitably one of many targets is to give a
  better foundation for finding out how the runtime is implemented, and to decide
  motivate of Rust’s extra critical static diagnosis to settle on bugs early.
• It has strengthen for focusing on WebAssembly, as correctly as diverse targets.

The of compiling a BEAM utility by technique of Lumen is a static executable. This differs
vastly from how deployment on the BEAM works on the present time (i.e. by technique of OTP releases). Whereas we
sacrifice the ability to contrivance hot upgrades/downgrades, we form mountainous gains in nasty-platform
compatibility, and ease of consume. Merely tumble the executable on a correctly matched platform, and speed it, no
instruments required, or particular concerns for the length of builds. This works the identical formulation that building Rust
or Plod purposes works on the present time.

Wishes

• Lend a hand WebAssembly as a form draw
• Fetch easy-to-deploy static executables as form artifacts
• Combine with tooling equipped by BEAM languages
• Extra efficient execution by taking out the necessity for an interpreter at runtime
• Characteristic parity with mainline OTP (with exception of the non-targets listed below)

Non-Wishes

• Lend a hand for warm upgrades/downgrades
• Lend a hand for dynamic code loading

Lumen is an different implementation of Erlang/OTP, in repeat a result it’s not as battle examined, or necessarily
as performant because the BEAM itself. Except we have a chance to speed some benchmarks, it’s exhausting to know
what the adaptation between the 2 when it comes to efficiency if truth be told is.

Lumen is not intended to change BEAM at the present restrict. At a minimum, the stated non-targets
of this mission imply that for not lower than some share of projects, some required efficiency would
be missing. On the different hand, it’s intended to be a tumble-in replacement for purposes which would possibly maybe well be better served
by its feature living.

Architecture

Compiler

The compiler frontend accepts Erlang source files. Here is parsed into an
summary syntax tree, lowered into EIR (Erlang Intermediate Illustration),
then at final lowered to LLVM IR where codegen is performed.

Internally, the compiler represents Erlang/Elixir code in a make very equivalent to
continuation-passing vogue. Continuations are an spectacular create that enable
easy implementations of non-local returns/exceptions, green
threading, and extra. Optimizations are essentially performed on this
illustration, earlier than decreasing to LLVM IR. Gape
eirproject/eir for added files on the
compiler frontend and EIR itself.

One day of decreasing to LLVM IR, the continuation illustration is stripped away,
and platform-explicit ideas for implementing diverse constructs are generated.
As an instance, on x86_64, hand-written assembly is outdated to contrivance extraordinarily low sign
stack switching by the scheduler, and to give dynamic characteristic utility
facilities for the implementation of prepare. At the moment, the C++-vogue zero-sign
exception mannequin is outdated for implementing exceptions. There are some future
proposals in growth for WebAssembly which will enable us to consume continuations for
exceptions, nonetheless that is not yet stabilized or implemented in browsers.

The compiler produces object files, and handles linking objects
collectively into an executable. It would possibly maybe well furthermore dump all of the intermediate artifacts,
such because the AST, EIR, MLIR in diverse kinds, LLVM IR, LLVM bitcode, and unsightly assembly.

Runtime

The runtime contrivance is basically the identical as OTP, nonetheless we’re not running an interpreter, as an different the
code is forward-of-time compiled:

• The entry point devices up the atmosphere, and begins the scheduler
• The scheduler is aloof of 1 scheduler per thread
• Every scheduler can settle on work from diverse schedulers if it’s short on work
• Processes are spawned on the identical scheduler because the assignment they’re spawned from,
  nonetheless a scheduler is ready to settle on them away to load steadiness
• I/O is asynchronous, with devoted threads and an tournament loop for dispatch

The initial model will more than likely be quite spartan, nonetheless right here’s so we are able to focal point on getting the runtime
habits rock solid sooner than we circle support to add in extra capabilities.

NIFs

NIFs will more than likely be ready to be defined in any language with C FFI, and can score to nonetheless can score to nonetheless be compiled to object
files after which handed by technique of linker flags to the compiler. The compiler will then make sure that that the NIFs
are linked into the executable.

The contrivance of the FFI is nonetheless up in the air – we are able to likely score a compatibility layer which is ready to
mimic the existing erl_nif.h interface, nonetheless since the runtime is diverse, there would possibly maybe well be
alternatives to give extra grunt hooks to formulation of the design.

License

Apache 2.0