Liquidsoap 2.3.0

[toots]

Liquidsoap 2.3.0

⚠️ Wanna help us know our users and plan the next release cycle? Fill up this form ⚠️

We are stoked to announce the release of liquidsoap 2.3.0! 🎉 ✨

This release brings a lot of exciting changes. As usual, we recommend to first test it in a staging environment before deploying it to production. Migration notes for this release can be found here.

A lot of hard engineering work went into this release! The largest changes should that happened during this cycle were under the hood:

Complete re-implementation of the streaming cycle and clocks

This change allowed us to prepare the codebase to use OCaml's new multicore support in future releases and get rid of design decisions that were 20 years old now. Back then, we had no idea how things would evolve! In particular, we had to account for new form of content such as raw video and encoded content.

Instead of filling a partial frame during the streaming cycle, we are now manipulating chunks of immutable content, concatenating them to build a frame on each streaming cycle. This is much more inline with the logic underlying video and encoded content where pieces of content can happen at random time, according to their presentation timestamp (PTS) or decoder timestamp (DTS).

Abstracting content this way has allowed us to finally get rid of frame breaks doubling as both track marks and partial filling marks. With the new content model, track marks and metadata are first-class content and part of the frame generation.

If you have ever encountered any of the infamous #get_frame didn't add exactly one break! fatal error, you know what we mean. This early design decision had plagued us for a while and it's wonderful to see it gone!

Along with these changes, we also reimplemented from scratch our notion of clocks, the abstract API that runs the main streaming cycles.

While clocks are still a pretty complex topic (self-sync, active vs. passive etc), we hope that this rewrite will clarify things. It was a good opportunity to clean up our implementation and use the experience gained about it in the recent years.

With these changes, clocks are now surfaced to the user via a clock method on each source. This makes it possible to directly access source clocks, animate them in your script, etc. We've also started to support synchronous streaming scripts, which should make it possible to use liquidsoap as a CLI for offline processing.

Script caching and memory optimization

Another important new feature with this release is the ability to cache the standard library and user scripts. This allows liquidsoap to run much faster. More details are available in this blog post

Incidentally, script caching also has very important benefits for memory usage. This is documented here.

This feature is probably the most important one of this release for most users!

Internal autocue

Autocue, the method for computing the perfect crossfade has also been polished in this release. We have now switched to our internal implementation, which is available when enabling our ffmpeg support.

You should be able to enable it by adding this to your script:

enable_autocue_metadata()

With this, all requests should resolve with the appropriate metadata so that any crossfade produces perfect transitions!

Remember that this feature requires CPU-heavy computations on each track!

A lot of testing went into this feature so we are really thankful to our users for the support and patience!

Other new features

Here's a list of other notable changes in this release:

• Frame duration was dropped to 0.02s. This should enhance streams latency and was made possible thanks to the new streaming implementation. See Change default frame duration to 0.02 #4033 for a discussion about the trade-offs involved with this setting.
• New support for video size-independent placement of video elements. This makes it possible to design video streams with different elements whose placement is independent of the final video resolution. See this blog post for an example of it.
• Support for NDI output. We always try to extend the scope of applications that can be interfaced with liquidsoap. In the future, we would like to add support for NDI inputs and extend the supported formats.

The rest of the changelog is included below.

As with each release, we want to thank all our users for their patience, early testing and collaboration to make this release possible. It's a testament to their help that we are able to ship this release with such a level of confidence despite all the major changes that happened in the code base.

Special thanks go to: @vitoyucepi, @gAlleb, @RM-FM, @Moonbase59, @Russsgithub, @gilou, Xogium, the whole AzuraCast and Radio France team and everyone who helped reporting, fixing and confirming bugs.

Thanks y'all!

2.3.0 (2024-11-27)

New:

• Rewrote the streaming API to work with immutable frame content. This
  should greatly impact impredictable side-effect of the previous models w.r.t.
  track marks, content sharing and more. This also impacts multiple operators
  behavior. Mostly, things should be roughly the same with differences around
  behaviors related to track marks (source.on_track and etc). (New streaming model #3577)
• Added script caching layer for faster script startup time. See: https://www.liquidsoap.info/blog/2024-06-13-a-faster-liquidsoap/ for details (Cache terms #3924, Cache stdlib #3949, Cache plugins #3959 and Set proper cache dir for posix build #3977)
• Rewrote the clock/streaming loop layer. This prepares our streaming system to
  support multicore when the OCaml compiler is mature enough to allow it. Clocks
  are now attached to sources via their clock methods. Returned value is a stripped
  down clock variable. Users can use the clock function to retrieve the full
  methods, e.g. s = sine(); c = clock(s.clock). This value has advanced functions
  for clock control such as start/stop, ticks and self_sync to check for
  self-sync. (Reclock #3781)
• Allow frames duration shorter than one video frames, typically values under 0.04s.
  Smaller frames means less latency and memory consumption at the expense of
  a higher CPU usage (Smaller frames #3607)
• Change default frame duration to 0.02s (Change default frame duration to 0.02 #4033)
• Optimized runtime (No extensible type for types. #3927, Native ground types #3928, Parse script in one pass #3919)
• Added NDI output support (Experimental NDI output #4181)
• Added finally to execute code regardless of whether or not an exception is raised
  (see: Add finally. #3895 for more details).
• Added support for Spinitron submission API (Add spinitron API functions. #4158)
• Removed gstreamer support. Gstreamer's architecture was never a good fit for us
  and created a huge maintenance and debugging burden and it had been marked as
  deprecated for a while. Most, if not all of its features should be available using
  ffmpeg. (Remove gstreamer. #4036)
• Removed taglib support. It is superseded by the internal ocaml-metadata module
  and taglib, with its dependency on the C++ runtime library, has been causing issues
  with binary builds portability and crashes with the (not yet supported) OCaml 5
  compiler. (Remove taglib, add more tags build config. #4087)
• Added video.canvas to make it possible to position video elements independently
  of the rendered video size (#3656, blog post)
• Added cover manager from an original code by @vitoyucepi (Add cover manager #3651)
• Added non-interleaved API to %ffmpeg encoder, enabled by default when only
  one stream is encoded.
• Allow trailing commas in record definition (Allow comma after the last argument in records #3300).
• Added metadata.getter.source.float (Add metadata.getter.source.float. #3356).
• BREAKING: Added duration and ticks to metadata available when computing HLS segment names (Expand HLS segment name metadata #4135)
• Added optional main_playlist_writer to output.file.hls and
  derivated operator (Add scripted main playlist writer for HLS output. #3484)
• Added is_nan, is_infinite, ceil, floor, sign and round (Math function floor, ceil, sign and round #3407)
• Added %track.drop to the %ffmpeg encoder to allow partial encoding
  of a source's available tracks (Add support for %*.drop tracks in ffmpeg encoder. #3480)
• Added let { foo? } = ... pattern matching (Nullable record pattern #3481)
• Added metadata.replaygain method to extract unified replay gain value from metadata (feat!: Unify ReplayGain handling and calculation #3438).
• Added metadata.parse.amplify to manually parse amplify override metadata.
• Added compute parameter to file.replaygain to control gain calculation (feat!: Unify ReplayGain handling and calculation #3438).
• Added compute parameter to enable_replaygain_metadata to control replay gain calculation (feat!: Unify ReplayGain handling and calculation #3438).
• Added copy: protocol (Add copy: protocol. #3506)
• Added file.touch.
• Added support for sqlite databases (Add support for sqlite #3575).
• Added string.of_int and string.spaces.
• Added list.assoc.nullable.
• Added source.cue (Add operation to generate CUE files. #3620).
• Added string.chars (Take encoding into account in string functions. #4111)
• Added atomic file write operations.
• Added new macos_say speech synthesis protocol. Make it the default implementation for the say:
  protocol on macos.
• Added settings.request.timeout to set the request timeout globally.

Changed:

• Reimplemented request.once, single and more using source.dynamic. Removed experiment
  flag on source.dynamic. The operator is considered stable enough to define advanced sources
  but the user should be careful when using it.
• Mute SDL startup messages (Add SDL log verbosity conf. #2913).
• int can optionally raises an error when passing nan or infinity, int(infinity)
  now returns max_int and int(-infinity) returns min_int. (Math function floor, ceil, sign and round #3407)
• Made default font a setting (Make default font a setting. #3507)
• Changed internal metadata format to be immutable (Immutable metadata #3297).
• Removed source.dump and source.drop in favor of safer request.dump and request.drop.
  source.{dump, drop} can still be implemented manually when needed and with the proper
  knowledge of what's going on.
• Allow a getter for the offset of on_offset and dropped the metadata
  mechanism for updating it (Allow a float getter for the offset of on_offset. #3355).
• string.length and string.sub now default to utf8 encoding (String functions UTF-8 issue #4109)
• Disable output paging when TERM environment variable is not set.
• Allow running as root user inside docker container by default (Liquidsoap interactive in container with user root #3406).
• Run check_next before playlist's requests resolutions (check_next is evaluated after request resolution #3625)
• Set force to true by default in file.copy to make operator behave
  as expected.
• BREAKING: Float comparison now follows the expected specs, in particular: nan == x is always false and
  nan != x is always true. Use float.is_nan to test if a float is nan.
• BREAKING: replaygain no longer takes ebu_r128 parameter (feat!: Unify ReplayGain handling and calculation #3438).
• BREAKING: assume replaygain_track_gain always stores volume in dB (feat!: Unify ReplayGain handling and calculation #3438).
• BREAKING: protocols can now check for nested static uri. Typically, this means
  that requests for an uri of the form: annotate:key="value",...:/path/to/file.mp3
  is now considered infallible if /path/to/file.mp3 can be decoded.
• Added parents option of file.mkdir (file.mkdir to support "recursive" dir creation / parent dir creation #3600, Add parents option to file.mkdir. #3601).
• Added forced_major_collections record field to the result of runtime.gc.stat() and
  runtime.gc.quick_stat() (feat: Add forced_major_collections back to the GC stat record #3783).
• Changed the port for the built-in Prometheus exporter to 9599 (refactor(prometheus): Change prometheus port #3801).
• Set segments_overheader in HLS outputs to disable segments cleanup altogether.
• Added support for caching LV2 and LADSPA plugins (Cache plugins #3959).
• Pulseaudio input and output now restart on pulseaudio errors (Make pulseaudio input and output resilient to errors. #4174).

Fixed:

• Fixed type generalization on values returned from function applications. Most notably,
  this should help with HTTP endpoint registration (Different http handlers may cause typing errors #3303, fixed in Generalize function applications #4030)

---

This discussion was created from the release Liquidsoap 2.3.0.

[gAlleb]

Hurray! Thank you, @toots!

[Russsgithub]

Whoop whoop