Exile is an alternative to ports for running external programs. It provides back-pressure, non-blocking io, and tries to fix ports issues.
Exile is built around the idea of having demand-driven, asynchronous interaction with external process. Think of streaming a video through ffmpeg to serve a web request. Exile internally uses NIF. See Rationale for details. It also provides stream abstraction for interacting with an external program. For example, getting audio out of a stream is as simple as
Exile.stream!(~w(ffmpeg -i pipe:0 -f mp3 pipe:1), input: File.stream!("music_video.mkv", [], 65535))
|> Stream.into(File.stream!("music.mp3"))
|> Stream.run()Exile.stream! is a convenience wrapper around Exile.Process. If you want more control over stdin, stdout, and os process use Exile.Process directly.
Exile requires OTP v22.1 and above.
Exile is experimental and it is still work-in-progress. Exile is based on NIF, please know the implications of it before using it
Existing approaches
Port is the default way of executing external commands. This is okay when you have control over the external program's implementation and the interaction is minimal. Port has several important issues.
- it can end up creating zombie process
- cannot selectively close stdin. This is required when the external programs act on EOF from stdin
- it sends command output as a message to the beam process. This does not put back pressure on the external program and leads exhausting VM memory
Libraries such as Porcelain, Erlexec, Rambo, etc. solves the first two issues associated with ports - zombie process and selectively closing STDIN. But not the third issue - having back-pressure. At a high level, these libraries solve port issues by spawning an external middleware program which in turn spawns the program we want to run. Internally uses port for reading the output and writing input. Note that these libraries are solving a different subset of issues and have different functionality, please check the relevant project page for details.
- no back-pressure
- additional os process (middleware) for every execution of your program
- in few cases such as porcelain user has to install this external program explicitly
- might not be suitable when the program requires constant communication between beam process and external program
On the plus side, unlike Exile, bugs in the implementation does not bring down whole beam VM.
This is my other stab at solving back pressure on the external program issue. It implements a demand-driven protocol using odu to solve this. Since ExCmd is also a port based solution, concerns previously mentioned applies to ExCmd too.
Internally Exile uses non-blocking asynchronous system calls to interact with the external process. It does not use port's message based communication instead does raw stdio using NIF. Uses asynchronous system calls for IO. Most of the system calls are non-blocking, so it should not block the beam schedulers. Makes use of dirty-schedulers for IO.
Highlights
- Back pressure
- it does not use any middleware program
- no additional os process. no performance/resource cost
- no need to install any external command
- tries to handle zombie process by attempting to cleanup external process. But as there is no middleware involved with exile so it is still possbile to endup with zombie process
- stream abstraction
If you are running executing huge number of external programs concurrently (more than few hundred) you might have to increase open file descriptors limit (ulimit -n)
Non-blocking io can be used for other interesting things. Such as reading named pipe (FIFO) files. Exile.stream!(~w(cat data.pipe)) does not block schedulers so you can open hundreds of fifo files unlike default file based io.
- add benchmarks results
As with any NIF based solution, bugs or issues in Exile implementation can bring down the beam VM. But NIF implementation is comparatively small and mostly uses POSIX system calls. Also, spawned external processes are still completely isolated at OS level.
If all you want is to run a command with no communication, then just sticking with System.cmd is a better option.
Copyright (c) 2020 Akash Hiremath.
Exile source code is released under Apache License 2.0. Check LICENSE for more information.