New Release Cycle for td-agent (Fluentd LTS)

[ashie]

New Release Cycle for td-agent (Fluentd LTS)

Introduction

We are planning to change the release cycle of td-agent.
Our mission is to keep maintaining Fluentd/td-agent ecosystem healthy, by
minimizing friction around deployment and upgrading.

Previously, we didn't present a clear release policy for td-agent,
which resulted in a constant flow of questions about the support
policy. We hope presenting it clears things up.

In addition, we are also planning to change the package name to a more
clear name like Fluentd LTS (fluentd-lts) on this occasion, to indicate
clearly that the package is Fluentd.

We would like to hear from users on these ideas and brush up them.

New Release Cycle

Overview

Starting with td-agent v5.0.0 in Q1 of 2023, we plan to apply the
following release policy:

• We release a new patch version every 3 months. Each patch version
  contains security/bug fixes.
• We release a new minor version every year. Each minor version
  contains the latest snapshot of Fluentd.
• Each minor version reaches EOL once the next minor version is
  released.

In other words:

• td-agent will be officially maintained for a year, with patch
  releases every three months.
• This should make things simpler for enterprise users. Previously
  any release can break compatibility, so it was hard to update td-agent.

Some notes:

• We'll publish a point (interlim) release whenever any critical security
  bugs are found.
• We'll also maintain Fluentd (and other plugins we maintain) based on
  this release cycle.

New Package Name

For the past years, td-agent has been the "de facto" official
distribution of Fluentd.

However, since the package maintenance has been transferred from
Treasure Data to the community last year, the package name
("td-agent") is really not accurate anymore.

To avoid the confusion about the nature of the distribution, we are
discussing to change the package name as follows:

• Current name: "td-agent"
• Candidate name: "Fluentd LTS" (From v5.0.0)
  • LTS means "Long Term Support"

Notes on compatibility:

• We're planning to make the new release fully-compatible with
  td-agent 4.x (regardless of the name change).
• For example, the install path will be "/opt/td-agent" even the
  package name is "fluentd-lts".
• This is mostly to make the transition as smooth as possible for
  existing users.

Related Links

• Rebranding td-agent-builder fluent-package-builder#311

Reference: example of plugins packaged in td-agent

Projects managed by GitHub fluentd organization

• fluentd
• fluent-plugin-kafka
• fluent-plugin-prometheus
• fluent-plugin-prometheus_pushgateway
• fluent-plugin-s3
• fluent-plugin-webhdfs
• fluent-plugin-rewrite-tag-filter
• fluent-plugin-sd-dns
• fluent-plugin-parser-winevt_xml
• fluent-plugin-windows-eventlog
• fluent-plugin-windows-exporter
• fluent-plugin-systemd
• fluent-plugin-utmpx
• fluent-plugin-opensearch

Plugins managed by the 3rd party

• fluent-plugin-td (Tresure Data)
• fluent-plugin-record-modifier (repeatedly)
• fluent-plugin-flowcounter-simple (sonots)
• fluent-plugin-metrics-cmetrics (Calyptia)
• fluent-plugin-calyptia-monitoring (Calyptia)
• fluent-plugin-elasticsearch (uken)
• ...

other libraries in td-agent

• yajl
• Oj
• nokogiri
• rdkafka
• elasticsearch
• aws-*
• ...

[shenningz]

• regarding minor releases: decoupling from the fluentd minor release cycle and staying on a minor release for one year would mean that you would miss features and bug fixes in subsequent fluentd minor releases that are published in-between your yearly release cycles. In your example, you would miss fluentd 1.18, 1.19, 1.20. A one-year LTS cycle is fine but should not be the only option. For users that rely on the latest features and CVE fixes, there should be some kind of "rolling release" that packages the latest fluentd snapshot. If this option does not exist, we would consider to consume the fluentd packages directy instead of using td-agent.
• regarding patch releases: generally this seems ok
• regarding interim releases: will these be based on fluentd patch releases ? Will they be released as td-agent patch releases, e.g. v5.0.4, just without the 3-month cycle ? Or as v5.0.3.1 ?
• regarding security fixes: are these backported into past fluentd minor releases (as patch releases) ? If yes, is this done directly in fluentd, or would you do the backporting/patching when picking up a fluentd patch release ? In your example: imagine a serious CVE is found and fixed in fluentd 1.18.1 on 2023-07-29. When will the fix become available in which td-agent release ?
• your write: "We'll also maintain Fluentd (and other plugins we maintain) based on this release cycle."
  Does that mean that in the future the fluentd and td-agent (aka fluentd-lts) release cycles (major/minor/patch) will be synchronized/aligned ?

[drbugfinder-work]

A one-year LTS cycle is fine but should not be the only option. For users that rely on the latest features and CVE fixes, there should be some kind of "rolling release" that packages the latest fluentd snapshot.

Then the naming schema would befluentd-lts and maybe fluentd-rolling, but we would suggest to rename td-agent project to something more generic, e.g. fluentd-bundle

[fujimotos]

staying on a minor release for one year would mean that you would
miss features and bug fixes in subsequent fluentd minor releases
that are published in-between your yearly release cycles

We'll backport bug fixes to patch releases. So that part will not be
your issue.

And yes, you'll miss new features. But you can actually easily upgrade
Fluentd to latest by running:

$ sudo td-agent-gem update fluentd

Anyway, the motivation here is that we often got complaints about the
old cycle being too fast. So we wanted to evolve the model from "just
push everything every 3 month" to "keep compatiblity for a year".

imagine a serious CVE is found and fixed in fluentd 1.18.1 on 2023-07-29.
When will the fix become available in which td-agent release ?

If it's a really serious bug (something like Heartbleed v2), we'll
publish a point release on 2023-07-29.

We'll also maintain Fluentd (and other plugins we maintain) based on this
release cycle." Does that mean that in the future the fluentd and td-agent
(aka fluentd-lts) release cycles

We plan to maintain two Fluentd branches: "master" and "stable".

The "stable" contains the bug fixes and security fixes, and will be
released every 3 month. The "master" is the same as the current one.
It contains all the PR accepted and gets released more often.

[drbugfinder-work]

And yes, you'll miss new features. But you can actually easily upgrade Fluentd to latest by running:

$ sudo td-agent-gem update fluentd

That's what we mean to be covered by a fluentd-rolling release. So for all minor (or even patch) releases of fluentd, a new fluentd-rolling release would be packaged - to make installation cleaner and determinable (not just installing and updating to latest but install specific releases as packages).

So our suggestion would be to have a real LTS release (fluentd-lts) as you describe it, but also have a second release channel fluentd-rolling, which updates on every fluentd (minor or patch) update.

To stay in your example from the image above, there could be the following releases:
fluentd-lts:
v5.0.0 (fluentd v1.17.1)
v5.0.1 (fluentd v1.17.2)
v5.0.2 (fluentd v1.17.3)
v5.0.3 (fluentd v1.17.4)
v6.0.0 (fluentd 1.21.0)

and

fluentd-rolling (in this example also with fluentd patch versions):
v5.0.0 (fluentd v1.17.0)
v5.0.1 (fluentd v1.17.1)
v5.1.0 (fluentd v1.18.0)
v5.1.1 (fluentd v1.18.1)
v5.1.2 (fluentd v1.18.2)
v5.2.0 (fluentd v1.19.0)
v5.3.0 (fluentd v1.20.0)
v6.0.0 (fluentd 1.21.0)

[fujimotos]

So our suggestion would be to have a real LTS release (fluentd-lts)
as you describe it, but also have a second release channel fluentd-rolling,
which updates on every fluentd (minor or patch) update.

@drbugfinder-work The major point is our human-power. Sadly, we don't
have enough people to maintain two channels, much less rolling one.

I'm planning to do some clean up on infra so that we can publish releases
more efficiently, and once that's done, maybe we can maintain parallel
releases lines, but we are not quite getting there yet!

Also I want to ask: Can you elaborate what do you use fluentd-rolling for?
I've never heard from anyone (well except you and @shenningz) who want
more frequent releases, so I want to understand the context here.

[fujimotos]

Technical note: The current td-agent is actually not a rolling release. It's
some kind of an odd ball between LTS and rolling release. Here is the
recent version history:

date	td-agent	Fluentd
2022-07-29	v4.4.0	v1.15.1
2022-06-28	v4.3.2	v1.14.6
2022-04-29	v4.3.1	v1.14.6
2021-12-06	v4.3.0	v1.14.3
2021-07-29	v4.2.0	v1.13.3

It followed roughly "release every three month, and make point releases on
serious bugs", and as you can see, each release almost always skipped
some Fluentd versions.

[drbugfinder-work]

Is there also a test environment for td-agent already existing or planned for the future? I can imagine, that larger steps in fluentd updates after a year cycle could bring up compatibility issues.

[fujimotos]

Is there also a test environment for td-agent already existing or planned for the future?

We build packages on https://github.com/fluent/fluent-package-builder.

The repository automatically builds on every push and you can download
pre-release RPM/DEB packages (etc.) from there. For example:

https://github.com/fluent/fluent-package-builder/actions/runs/2758958173

[drbugfinder-work]

@fujimotos I mean a test environment, so if the bundles libraries and Ruby package, etc. are fully working together. For example to ensure functionality after the switch to OpenSSL3 or the switch to Ruby 3 (fluent/fluent-package-builder#402 (comment)).
So for all td-agent/fluentd-lts/fluentd-rolling builds on the various platforms and OS Versions (e.g. Ubuntu 22.04 on x86_64) programmatic tests would be executed to ensure all components are working as expected.

Is something like this already existing or planned?

[ashie]

We already have tests in there GitHub Actions (current status is far from perfect though).
For example, we test kafka connection like this: https://github.com/fluent/fluent-package-builder/runs/7574381383?check_suite_focus=true#step:8:3398

Contributions to improve it is welcome 🙂

[kou]

I propose "fluent-package" as a new package name instead of "Fluentd LTS".
"Fluentd LTS 5.0.0" ships "Fluentd 1.17.1" will confuses users.

I propose aligning version of "fluent-package" and "fluentd".
It means that "fluent-package 1.17.0" ships "fluentd 1.17.0". If we need to release a new version that has only package related changes (e.g. packaging bug fix), we use "fluent-package 1.17.0+001", "fluent-package 1.17.0+002" and so on. "+XXX" is used by SemVer.

I propose updating bundled Ruby in Windows package to the latest release or the second newest release per year.
I propose updating Ruby module in RHEL packages to the latest release or the second newest release per year.
FYI: Ruby releases a new major or minor version per year.

[daipom]

I agree.
Our concern is that the "fluent-package" versions will not be sequential, right?
I think it would be better to allow for that and align versions of "fluent-package" and "fluentd".

[ashie]

In fact there was various discussion internally for versioning issue.
The original idea I described in the first comment is keeping full compatibility with td-agent.

We're planning to make the new release fully-compatible with
td-agent 4.x (regardless of the name change).

This meant that neither the name nor the versioning would change significantly.
The aim of this idea is reducing migration cost for both Fluentd developers and users.

On the other hand I'm not stuck with that, other ideas are also welcome for me as long as compatibility is kept.
But I don't yet confirm that other ideas can keep compatibility gracefully on all platforms.

[ashie]

(compatibility means upgrading from td-agent.)

[kou]

For .deb/.rpm, we can implement smooth upgrade by "transitional package":

• https://wiki.debian.org/PackageTransition
• https://fedoraproject.org/wiki/Upgrade_paths_%E2%80%94_renaming_or_splitting_packages

For Windows and macOS, I need to check the current upgrade path. (Manual operation?)

[ashie]

I think we don't need to prepare safe upgrade path for macOS, it's not supported well yet and less users.
For Windows, keeping upgrade code might be enough but I don't yet confirm it.

[kou]

@kenhys and I are still evaluating Open Build System as the new package delivery service.
So we can't release a new "Fluentd LTS" package in this month.
Sorry.

I'll share the current status later again.

[kenhys]

OBS itself is a great service, but there is no AmazonLinux support yet. As of that, we can't migrate to it as the new package delivery service.

[Houlistonm]

Any updates on the new release schedule?
(Our security team is asking about our use of ruby 2.7)

[kenhys]

In the end of Aug, Fluentd LTS will be available.

[ashie]

We released new major version of fluent-package (td-agent) on August 29, 2023.

Refs:

• Release Note for RC (Release Candidate)
• Release Note for GA (General Availability)
• Upgrade to fluent-package v5
• Scheduled support lifecycle announcement about Fluent Package
• Drop schedule announcement about EOL of Treasure Agent (td-agent) 4

I summarize the final answers for some discussions in this topic.

Package Name

• td-agent has been renamed to fluent-package

Version Number

• The version number is continued from td-agent, the new major version is v5.

Release Cycle

• Please see https://www.fluentd.org/blog/fluent-package-scheduled-lifecycle

Release Channels

From fluent-package v5 we introduced 2 release channels:

• Normal release channel
• LTS (Long Term Support) channel

Please see https://www.fluentd.org/blog/fluent-package-scheduled-lifecycle for more detail.

Package Delivery Site

• Thanks to Treasure Data, we'll continue using https://td-agent-package-browser.herokuapp.com/ for a while.
• On the other hand we are asking CNCF to prepare a new site, we need some more time on this.

Compatibility against td-agent

• Some file/directory names are changed from td-agent to fluent or fluentd
• Existing configuration files are mostly migrated automatically on upgrading but there are some cases where you need to migrate them manually. So you need to confirm carefuly whether you have configuration which is needed to be migrate manually before upgrading production environment.
• Please see https://www.fluentd.org/blog/upgrade-td-agent-v4-to-v5 for the migration steps.