RFC: Kibana preboot lifecycle stage.

[azasypkin]

Summary

This RFC proposes a new preboot lifecycle stage. At this stage Kibana only initializes a bare minimum of the core services and a limited set of special-purpose plugins. It's assumed that Kibana can change and reload its own configuration at this stage and may require administrator involvement before it can proceed to the setup and start stages.

The preboot lifecycle stage is a prerequisite for the Kibana interactive setup mode.

View rendered RFC

The commenting period will last two weeks, ending on June 23, 2021.

You can find the proof-of-concept with proposed changes here: azasypkin/kibana@master...issue-xxx-preboot-poc

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[azasypkin]

Hey @elastic/kibana-core and @elastic/kibana-security,

This RFC is ready for the initial feedback, I'd really appreciate if you can take a look whenever you have time. Let me know if a quick Zoom-walk-through would be helpful as well.

Thanks!

[legrego]

What happens if one of these promises rejects? Do we abort the process, pretend it didn't reject, or something else?

[azasypkin]

The plan was to shut Kibana down as we do when setup or start rejects. I'll call it out here, thanks.

[legrego]

question what do we intend to use isBootOnHold() for? Would it be worthwhile adding a bit more information here, so help explain why the boot is on hold? e.g. "setup is preventing boot because ..."

[azasypkin]

The intention was to have a method that can tell you if Kibana needs to hold the boot before you block on waiting. Here is the example from the PoC where I wanted to log a message that Kibana is about to hold the boot:

const { preboot } = await root.preboot();

if (preboot.isBootOnHold()) {
  root.logger.get().info('Holding setup until pre-boot phase is completed.');
  const { shouldReloadConfig } = await preboot.waitUntilCanBoot();
  if (shouldReloadConfig) {
    await reloadConfiguration('pre-boot request');
  }
}

await root.setup();

I feel that we'll need this method eventually, but I don't have a good use case in mind to justify it right now. Would you prefer to drop it until we know we absolutely need it?

Would it be worthwhile adding a bit more information here, so help explain why the boot is on hold? e.g. "setup is preventing boot because ..."

That's a great idea, would having something like this in the log help?

// Consumers of `holdBootUntilResolved` are obliged to provide a user-friendly `reason`.
[2021-06-09T09:36:17.931+02:00][INFO][preboot] `setup` plugin is preventing boot: {reason}

[legrego]

Yeah I think an entry in the log would be helpful

[legrego]

I'm trying to decide if we want to run through the preboot lifecycle again following a config reload. Could we get into a scenario where the written config was valid preboot time, but then something within the environment causes the config to become invalid? Having preboot run again would give us the opportunity to correct a misconfigiured instance, or at least show a status page.

On the other hand, a bug in the preboot lifecycle could cause us to get into an infinite loop where we preboot forever without ever reaching the setup phase.

I'm torn on which way to go, so I'll leave this here as a discussion topic for now.

[azasypkin]

Could we get into a scenario where the written config was valid preboot time, but then something within the environment causes the config to become invalid?

We can, but I see only two reasons when this can happen:

• Preboot plugins produce invalid config due to a bug
• User changes config outside of Kibana for some reason, while Kibana is at the preboot stage

These look like unrecoverable issues to me, and I'd prefer Kibana to crash as it does today when config validation fails. For example, if HTTP config fails validation we won't be able to even start preboot HTTP server.

Having preboot run again would give us the opportunity to correct a misconfigiured instance, or at least show a status page.

That would be a great feature for Kibana for sure, but I feel like it's a bit out of scope for now. The near future plan is to make very few config changes and every change should be extensively validated.

Having said that, you're raising a very important point that seems to be outside of this RFC, but we'll need to take care of it as soon as we start working on the setup plugin: preboot plugins should have access to schemas (or validation methods) for the Core configuration they may change. Same goes for security configuration (encryption keys, auth providers etc.)

[legrego]

The preboot phase could imply the existence of a boot phase, but I don't have any better suggestions. presetup is the only other option that comes to mind, but I don't like that any more than preboot.

[azasypkin]

The preboot phase could imply the existence of a boot phase

I was thinking of setup -> start as a two-step boot phase, but yeah I don't really have a good alternative either. If anyone has a strong opinion on some specific name, I'll happily take it.

[legrego]

In the interest of simplicity, this seems tolerable. This feels like something we could improve in subsequent releases without undoing any of the work required to implement an interactive setup mode. In a perfect world we would have a well defined contract for the preboot phase on the front-end, but I'm not opposed to a pragmatic approach here.

If we move forward, perhaps it'd be worthwhile opening an issue to align the client-side preboot experience with the server-side one?

[azasypkin]

If we move forward, perhaps it'd be worthwhile opening an issue to align the client-side preboot experience with the server-side one?

Yep, definitely agree! As soon as we all agree on the approach for the server-side preboot stage, I'll file an issue to sync client-side with the server-side as well.

[pgayvallet]

FWIW, when we discussed it with @azasypkin, I did agree that the pragmatic approach looked acceptable, at least initially.

[joshdover]

I agree with the approach to keep setup and start on the client. In the browser, setup and start occur once in every user window that is opened, so the semantics here are slightly different than the server. Keeping this consistent makes sense.

That said, I think the only client-side CoreStart services that need to unavailable are the CoreStart.savedObjects and CoreStart.uiSettings services. Could we expose a PrebootPlugin interface on the client that is identical but Omits these two services from PrebootCoreStart?

[azasypkin]

That said, I think the only client-side CoreStart services that need to unavailable are the CoreStart.savedObjects and CoreStart.uiSettings services. Could we expose a PrebootPlugin interface on the client that is identical but Omits these two services from PrebootCoreStart?

Yeah, we can definitely do that. Would "compile-time guarantee" we'll get with the dedicated client-side PrebootPlugin interface be enough or you think we should "physically" remove these two services from the start contract (I tried to avoid such client-side changes until now, but can investigate how difficult it would be if necessary)?

[pgayvallet]

Will probably not be a surprise, but I'm very mixed on all this.

If we do take the 'we need a preliminary UI for Kibana, from the same server, blocking SO migration and other plugins to boot (well, basically any ES interaction) until some user action is performed' as a mandatory statement/requirement, then the implementation looks mostly fine to me. We're reusing core's code and services as much as possible to avoid duplication. At least I don't really see a better implementation (but here again, maybe some team members do?).

If we do accept that, my main concerns is regarding the other potential use case of this preboot lifecycle we identified for core: The interactive migration, where we would also have some kind of blocking UI to allow users to have more control over the migration (see #100685 (comment)). In that case, however, we would still need all existing plugins to have at least be running their setup phase, as we need to pass that stage to be able to perform the migration with SO types registered, which is conflicting with the current RFC's proposed design.

On the other hand, if we instead accept that this 'we need a preliminary UI for Kibana, from the same server, blocking other plugins to boot until some user action is performed' statement can be challenged, it really feels like we're introducing a lot of changes and complexity within core for just one very specific feature (I don't remember a single feature requiring that much core changes tbh), and I wonder if the conclusion of the preliminary POC shouldn't just be: it's not worth the complexity, let's just go with a CLI tool instead.

Don't get me wrong, a CLI tool is a worse user experience, and we wouldn't be able to 'force' the user to go though this interactive setup mode, he would have to explicitly choose to do so by running the CLI tool, however:

• (correct me if wrong) we're not targeting Cloud here, as the instances are already properly configured, so a CLI tool seems indeed like a possible option
• we already extracted core's config service to a package. @kbn/cli-dev-mode, for instance, already manually parse and read the config the exact same way core's server does (just need to duplicate the required schema parts atm, as we judged that it was not worth factorizing yet, but this can change)
• updating the config file could be done the exact same way as it's currently done in the POC
• I think the interactive setup workflow is simple enough to be done from the command line

Note that one downside of the CLI tool approach is that the interactive migration could not be achieved that way, as this is a feature we do also need for cloud. However, with the RFC current design, we wouldn't be able to leverage this new API for the interactive migration anyway due to the setup invocation requirement explained previously.

[pgayvallet]

Will setup be the actual name of the plugin?

[azasypkin]

Yeah, that was the plan, do you feel like it'd be confusing?

[joshdover]

I think something like userSetup might be more clear IMO

[pgayvallet]

I initially though the main reason to have an UI for this was for Cloud, as we can't provide a CLI tool for cloud customers. However, AFAIK, cloud instances are not really impacted by this interactive setup, as instances should already be properly configured.

Given that, isn't the possibility to provide a cli tool another potential alternative?

[azasypkin]

Answered in #99318 (comment)

[pgayvallet]

FWIW, when we discussed it with @azasypkin, I did agree that the pragmatic approach looked acceptable, at least initially.

[pgayvallet]

I never really understood the real value of this feature tbh. One option could be, when isBootOnHold returns true, to send this SERVER_LISTENING message when the preboot server is up.

Anyhow, this is only impacting local dev mode, so I don't think the impact should be significant, regardless of the solution we choose.

[azasypkin]

Anyhow, this is only impacting local dev mode, so I don't think the impact should be significant, regardless of the solution we choose.

Yep, agree, mentioned this just to have a more accurate picture on the amount of changes we'll need to make.

[lukeelmers]

If we do take the 'we need a preliminary UI for Kibana, from the same server, blocking SO migration and other plugins to boot (well, basically any ES interaction) until some user action is performed' as a mandatory statement/requirement, then the implementation looks mostly fine to me.

On the other hand, if we instead accept that this 'we need a preliminary UI for Kibana, from the same server, blocking other plugins to boot until some user action is performed' statement can be challenged, it really feels like we're introducing a lot of changes and complexity within core for just one very specific feature

++ Agreed on this. If from a product perspective we feel a UI vs CLI justifies this significant investment, then overall the approach here makes sense to me.

Aside from the questions on interactive migrations, I'm also thinking about how this RFC would play with the multi-process plan proposed in #94057

It seems that, if the current proposal remained unchanged, the only implication would be that we'd need to make sure preboot only writes updated configs to disk from the main process, so that there's no risk of concurrency issues (though multiple users writing to the config simultaneously during preboot is admittedly quite unlikely).

Based on what's proposed here, it sounds like Larry's idea from the original issue of running this on a separate process/port has been totally ruled out at this point, correct?

[mshustov]

Once the preboot stage is complete,

What means complete in this case? A Kibana user opens preboot UI and closes it? What if UI is closed without configuring the connection to Elasticsearch?

[azasypkin]

Sorry, used a poor wording, will improve. I should have said once none of the preboot plugins holds the boot/setup anymore, Kibana might need .... In the scope of interactive setup mode it will happen when user finishes configuring of the ES connection and clicks on something like Open/Start/whatever Kibana (TBD).

[mshustov]

considering that users can adjust a sub-set of plugin settings (according to #89287 (comment)) we will have to duplicate config schema between "pre-boot" and "primary" plugins. But it's an error-prone approach as it's easy to forget to update a "pre-boot" plugin once you touch its "primary" couterpart.

[azasypkin]

Yes, I agree. I intentionally omitted plugin-specific configuration in the RFC since we don't have a good solution for this yet. The required core schemas can be provided by the core at the preboot stage in one way or another, and when/if we decide to change plugin specific configuration we'll figure this out.

[mshustov]

Kibana Plugins register their ui settings during the setup phase. Thus, we won't be able to have access to plugin ui_settings defaults in the "preboot" phase. Thus, users can specify overrides for Core and read Core defaults only.

[azasypkin]

Correct, I'll make it clear that we're talking about default Core UI settings, thanks.

[mshustov]

We may need to expose a full or portion of Elasticsearch config to the preboot plugins for them to check if the user has already configured Elasticsearch connection.

Motivation says we run preboot phase only if it detects that user hasn't explicitly configured their own connection to Elasticsearch. Does it mean that we run preboot whenever ES configuration is present, but Kibana cannot establish a network connection?

[azasypkin]

That's a good question.

Motivation says we run preboot phase only if it detects that user hasn't explicitly configured their own connection to Elasticsearch.

The preboot will still run, but if interactive setup mode plugin detects that user has already configured connection to ES it 1) will not hold the boot, and 2) will instead render a "replacement" for the static Kibana server is not ready yet string (Design Team will help us to display something user-friendlier and that will automatically redirect user to the original URL when the primary server is up).

Does it mean that we run preboot whenever ES configuration is present, but Kibana cannot establish a network connection?

Yes, if a user has already configured ES connection, we assume they know what they are doing no matter if ES is available or not. But it'd easy to improve experience in the future if we feel we need it.

[joshdover]

Design Team will help us to display something user-friendlier and that will automatically redirect user to the original URL when the primary server is up

I strongly feel we should be showing some limited status information on this UI to indicate what's happening inside Kibana. Historically, this information has required authentication to access (/api/status), but I think we could safely expose just the overall status level and summary text without requiring auth.

[azasypkin]

I strongly feel we should be showing some limited status information on this UI to indicate what's happening inside Kibana. Historically, this information has required authentication to access (/api/status), but I think we could safely expose just the overall status level and summary text without requiring auth.

I agree. We may not have enough time to cover this in the initial implementation though, but I'll make sure we have everything in place to do this in the follow-up, if we want to.

[mshustov]

This would potentially allow us to switch locale during interactive setup mode if there is such a need in the future.

Agree, that we shouldn't implement it from the very beginning. Preboot phase is used by admin, we don't translate any log messages for them.

[azasypkin]

Preboot phase is used by admin, we don't translate any log messages for them.

We don't translate log messages, but we still localize the interactive setup mode plugin (I can see us or partners distributing Kibana binaries with the predefined non-English locale). But every byte counts, so preboot HTTP server will only be serving translations required by Core and preboot plugins.

[mshustov]

What if we have a requirement to support configuration changes from Stack Management UI? We will have to remove preboot lifecycle support?

[azasypkin]

Can you please elaborate on this a bit? I'm not sure I'm following.

[mshustov]

I meant that if we had a requirement to support configuration changes from Stack Management UI, Kibana wouldn't leverage the preboot phase logic. However, as mentioned, the ES connection setup is not the only use case for the interactive mode, so you can ignore the question.

[azasypkin]

Ah, I see now, thanks. Yeah, to be able to handle configuration changes from the Stack Management UI "at run-time" we'll likely need to widely adopt that reactive approach Josh mentioned in #99318 (review). Hopefully we can get there at some point.

[mshustov]

Another alternative could be to avoid blocking Kibana start until the connection to Elasticsearch is established. @pgayvallet would it be covered by #96626? Of course, it's not something we'll add tomorrow, but if we think it's the right approach, we can temporarily limit ourselves to any quick hack-ish solution.

[pgayvallet]

would it be covered by #96626

No, for multiple reasons

1. Eventually consistent saved object/data index migrations #96626 does not plan to force 'delayed' migration for all SO types, it would more be an opt-in when type owners thinks their types can safely be migrated at a later time.
2. independently from SO migration, some plugins retrieves an ES and/or SO client during start and keep its reference (e.g to use within services). atm the es clients do not dynamically reload their configuration, meaning that not blocking the other plugins initialization would just break their ES connectivity.

[azasypkin]

Thanks for the feedback, everyone!

Sorry, I feel like I didn’t do a good job of providing you with enough context for this RFC. Let me fix that before addressing your questions and concerns:

As you already know, the core motivation behind the proposed change is enabling security features across Elastic Stack by default. For everyone, unless they explicitly decide to opt out. To reduce the scope I’ll refer to only TLS (Kibana ↔ Elasticsearch) and authentication as those security features we want to enable by default, and Kibana + Elasticsearch as a part of the Elastic Stack we’re concentrating on at the initial stage.

By Default is the first key requirement here. It implies that whenever a user runs a fresh Elasticsearch instance it will automatically enable both TLS on HTTP/transport level and authentication. When the user eventually runs a fresh instance of Kibana it should be able to connect to the corresponding Elasticsearch by default and require an absolute minimum of actions and specialized knowledge from the user to do so.

This first experience should be as easy from the operational standpoint as possible, no matter how Elasticsearch and Kibana are run: directly from the distributive or as a Docker image. By the way, we figured out that Docker is super popular among our audience and hence became one of the top scenarios we commit to support out of the box.

Having a separate, non-default, "entry path" in the form of a dedicated application (e.g. CLI tool) introduces unnecessary friction, contradicts by default requirement, and introduces a non-trivial level of complexity to support setups like Docker.

We could theoretically make this CLI as a default “entry point” that will run Kibana eventually, but then conceptually it'd not be much different from the "native" preboot stage and, at the same time, would have all the drawbacks of a separate tool.

Enabling security by default is technically a breaking change, a necessary evil, if you will. This leads us to the second requirement - provide as good UX as possible. We have to reduce user’s pain because of a new flow, not drive them away with something suboptimal and implicitly force them to opt-out of our security features and a plethora of solutions that depend on them.

Now, let me cover some technical aspects of the proposed changes that seem to be the most confusing. On the diagram below you can see the lifetime of the preboot plugins and preboot HTTP server (same as lifetime of the Not Ready server today): these are only stopped when the primary server is finally ready to start so that there is a very minimal gap when nothing is serving user requests.

The diagram also includes holdSetup and holdStart as the extension points for the preboot plugins. I didn’t include both of them in this RFC as interactive setup needs only holdSetup (or holdBootUntilResolved as I named it), but if we ever need anything more granular (e.g. to support SO migrations UI via preboot plugins, after setup of primary plugins) I don’t see any reason why we cannot achieve that.

The complexity and maintenance cost is also something we assessed at some point. Dealing with the config is just one aspect of the interactive setup mode. There are many more things we'll need to do:

• deal with the Elasticsearch APIs, respecting default elasticsearch.* configuration (e.g. request timeouts) and behavior of the Elasticsearch service (e.g. X-Opaque-Id)
• expose HTTP endpoints with the validation (e.g. schemas) and security (e.g. prototype pollution guards) guarantees we have in Kibana today
• leverage the same server-side logging sub-system (format, rolling, censoring, tags etc.) to be able to rely on the same diagnostics tooling and techniques we have in Kibana to troubleshoot interactive setup mode in production
• build a UI that’s visually and behaviorally consistent with the Kibana (e.g. client side HTTP service, notifications, I18n, error screens)

The list can go on, but even with those mentioned the maintenance burden for the separate tool feels unrealistically high to me. If the core becomes fully modular in the future I can see this as a feasible alternative, but I don’t think we’re there yet.

And last but not least, we see the interactive setup mode not just as a one-off feature to support Security On By Default, but more as an essential part of the platform that will eventually allow us to reduce the amount of frustration the ordinary Kibana user has today when they try to configure something that’s a bit beyond a trivial setup (👋 TLS and anonymous access).

I certainly don’t want to inflate the scope of this RFC, but a hacky or a very limited-value Core functionality isn’t my goal either. Assuming I’m not overlooking any critical fundamental flaws, I’m happy to iterate on this with you all as much as needed to have a solid feature that’d cover our immediate and mid-term needs.

Does that additional context make any sense to you and alleviate some of your concerns?

---

If we do take the 'we need a preliminary UI for Kibana, from the same server, blocking SO migration and other plugins to boot (well, basically any ES interaction) until some user action is performed' as a mandatory statement/requirement, then the implementation looks mostly fine to me. We're reusing core's code and services as much as possible to avoid duplication. At least I don't really see a better implementation (but here again, maybe some team members do?).

I tend to treat this as a mandatory requirement, but I don’t see any reason why we cannot try to challenge this, that's what RFC is for!

If we do accept that, my main concerns is regarding the other potential use case of this preboot lifecycle we identified for core: The interactive migration, where we would also have some kind of blocking UI to allow users to have more control over the migration (see #100685 (comment)). In that case, however, we would still need all existing plugins to have at least be running their setup phase, as we need to pass that stage to be able to perform the migration with SO types registered, which is conflicting with the current RFC's proposed design.

I tried to expand on this a bit more using the diagram above. Do you still feel it’s not enough to fulfill this requirement?

On the other hand, if we instead accept that this 'we need a preliminary UI for Kibana, from the same server, blocking other plugins to boot until some user action is performed' statement can be challenged, it really feels like we're introducing a lot of changes and complexity within core for just one very specific feature (I don't remember a single feature requiring that much core changes tbh), and I wonder if the conclusion of the preliminary POC shouldn't just be: it's not worth the complexity, let's just go with a CLI tool instead.

I’m definitely biased when I compare the complexity of these two solutions above, but the complexity and maintenance burden of the separate tool still outweighs the complexity of the proposed Core changes for me. Do you feel like I’m overestimating?

Don't get me wrong, a CLI tool is a worse user experience, and we wouldn't be able to 'force' the user to go though this interactive setup mode, he would have to explicitly choose to do so by running the CLI tool, however:

Yeah, I see where you’re coming from for sure, but the worse UX and, more importantly, violated “defaultiness” requirement sound like a deal breaker for me. @legrego , I’d appreciate it if you can share your thoughts on that.

(correct me if wrong) we're not targeting Cloud here, as the instances are already properly configured, so a CLI tool seems indeed like a possible option

You’re right that interactive setup mode isn’t targeting Cloud at this point, but as I mentioned above preboot HTTP server and plugins can still be useful in Cloud as well.

we already extracted core's config service to a package. @kbn/cli-dev-mode, for instance, already manually parse and read the config the exact same way core's server does (just need to duplicate the required schema parts atm, as we judged that it was not worth factorizing yet, but this can change)
updating the config file could be done the exact same way as it's currently done in the POC
I think the interactive setup workflow is simple enough to be done from the command line

Well, I tried to list some of the functionality beyond the config manipulation interactive setup mode is going to need, so it may not be that simple as it seems. But yeah, it’s a judgment call. Let’s discuss.

Note that one downside of the CLI tool approach is that the interactive migration could not be achieved that way, as this is a feature we do also need for cloud. However, with the RFC current design, we wouldn't be able to leverage this new API for the interactive migration anyway due to the setup invocation requirement explained previously.

Again, if I’m not overlooking anything then we should be able to leverage proposed changes for the interactive SO migration if we want to.

Aside from the questions on interactive migrations, I'm also thinking about how this RFC would play with the multi-process plan proposed in #94057
It seems that, if the current proposal remained unchanged, the only implication would be that we'd need to make sure preboot only writes updated configs to disk from the main process, so that there's no risk of concurrency issues (though multiple users writing to the config simultaneously during preboot is admittedly quite unlikely).

Thanks for bringing this up! Yeah, if I understand your proposal correctly, with the exposed isMainWorker API we can achieve that.

Based on what's proposed here, it sounds like Larry's idea from the original issue of running this on a separate process/port has been totally ruled out at this point, correct?

I can see why we ruled it out, but my memory is hazy here, @legrego would you mind commenting on this one?

[joshdover]

I share similar concerns about the number of changes to Core this would require, but the tl;dr of my conclusions on this are: If we need to support this requirement, I see no viable alternative.

I completely agree that we need a strong default experience for new users that exposes all the goodies that the Stack has to offer. An optional CLI on a separate codepath would not achieve that goal and a default CLI for doing this setup mode would likely require most of the same changes proposed by this RFC, however it would allow us to defer on the client-side changes needed to support preboot plugins in the browser. Has this been considered as a intermediate phase towards the end-goal of this project? It may be a "good enough" option that buys us time in other areas.

@azasypkin Thanks for the detailed visual of where the preboot phase fits into the setup and start lifecycles. I think it'd be good to include that image in the RFC itself for posterity. As I understand it, we should be able to support an interactive Migrations UI (#100685 (comment)) using the preboot APIs for registering routes from the SavedObjectsService prior to completing the migrations process. I don't believe we'd even need an additional API for blocking HTTP start since the SavedObjectsService.start() already blocks for migrations to complete.

IMO the most plausible alternative is the one suggested by @mshustov where we delay running Saved Object migrations until after start. In theory, such an approach would allow any normal plugin to offer the same UX without so many changes to Core. However, it would likely require significant changes to many plugins to handle the case where Elasticsearch and SavedObjects are not yet available (or even worse, available but may be reconfigured at some future point). I fear the number of bugs and issues that could come from this requirement to be quite high.

The original motivation for Kibana Platform plugin lifecycles was to provide key points in time where certain guarantees about the application state could be assumed. The most significant of those is the availability of Saved Objects. It's not impossible that we move to a more reactive approach and require that plugins adapt to changes in core configuration like these, but it would require a significant re-education and refactoring of many plugins. Becoming more reactive would be quite a shift in architectural direction, one we actually discussed at GAH as likely not being an improvement worth the cost.

[joshdover]

Design Team will help us to display something user-friendlier and that will automatically redirect user to the original URL when the primary server is up

I strongly feel we should be showing some limited status information on this UI to indicate what's happening inside Kibana. Historically, this information has required authentication to access (/api/status), but I think we could safely expose just the overall status level and summary text without requiring auth.

[joshdover]

It seems to me that this special API interaction is probably unnecessary? Is there anything else that could be returned from root.preboot() that we would need to access from bootstrap? It seems we could simplify this as:

const { shouldReloadConfig } = await root.preboot();

[azasypkin]

Is there anything else that could be returned from root.preboot() that we would need to access from bootstrap?

I had two possible cases in mind:

• To support additional isBootOnHold-like property I mentioned here https://github.com/elastic/kibana/pull/99318/files#r648047582
• I see we have quite a few integration tests in the core that use coreSetup contract returned from the root.setup() and assumed that I'll likely have to do something similar for root.preboot() to be consistent with the existing tests.

[joshdover]

I think something like userSetup might be more clear IMO

[joshdover]

I agree with the approach to keep setup and start on the client. In the browser, setup and start occur once in every user window that is opened, so the semantics here are slightly different than the server. Keeping this consistent makes sense.

That said, I think the only client-side CoreStart services that need to unavailable are the CoreStart.savedObjects and CoreStart.uiSettings services. Could we expose a PrebootPlugin interface on the client that is identical but Omits these two services from PrebootCoreStart?

[mshustov]

Thanks for the detailed visual of where the preboot phase fits into the setup and start lifecycles. I think it'd be good to include that image in the RFC itself for posterity.

Absolutely, It helped me to finally understand how we can integrate a complex async flow with Kibana's setup and start phases.

I don't believe we'd even need an additional API for blocking HTTP start since the SavedObjectsService.start() already blocks for migrations to complete.

We might need it in the next stages when preboot won't be limited by ES connection setup and SO migration. We can document it in the RFC but postpone adding an implementation until necessary.

If we need to support this requirement, I see no viable alternative.

Yeah, I can see the point that an alternative implementation might require at least a comparable amount of effort, so my ✅ to move with the current proposal.

[azasypkin]

and a default CLI for doing this setup mode would likely require most of the same changes proposed by this RFC, however it would allow us to defer on the client-side changes needed to support preboot plugins in the browser. Has this been considered as a intermediate phase towards the end-goal of this project? It may be a "good enough" option that buys us time in other areas.

Just for the records, we discussed this with @joshdover over Slack and agreed that the client-side changes isn't a concern here: one of our base assumptions, preliminary backed by the PoC, is that implementation of this RFC will require only very minor changes in the client-side part of the Core, if any at all. If that changes, we'll go back to the drawing board.

@azasypkin Thanks for the detailed visual of where the preboot phase fits into the setup and start lifecycles. I think it'd be good to include that image in the RFC itself for posterity.

Yep, will do!

IMO the most plausible alternative is the one suggested by @mshustov where we delay running Saved Object migrations until after start. In theory, such an approach would allow any normal plugin to offer the same UX without so many changes to Core. However, it would likely require significant changes to many plugins to handle the case where Elasticsearch and SavedObjects are not yet available (or even worse, available but may be reconfigured at some future point). I fear the number of bugs and issues that could come from this requirement to be quite high.

The original motivation for Kibana Platform plugin lifecycles was to provide key points in time where certain guarantees about the application state could be assumed. The most significant of those is the availability of Saved Objects. It's not impossible that we move to a more reactive approach and require that plugins adapt to changes in core configuration like these, but it would require a significant re-education and refactoring of many plugins. Becoming more reactive would be quite a shift in architectural direction, one we actually discussed at GAH as likely not being an improvement worth the cost.

Yeah, totally agree. We considered similar options in the past, but abandoned most of them as they would have required changes in all or most of the plugins we have today making the overall risk a way too high.

---

Absolutely, It helped me to finally understand how we can integrate a complex async flow with Kibana's setup and start phases.

Sorry for not including it earlier, will push it with the next set of changes.

We might need it in the next stages when preboot won't be limited by ES connection setup and SO migration. We can document it in the RFC but postpone adding an implementation until necessary.

Yeah, that makes sense to me, will document it here. That would also mean I'll rename holdBootUntilResolved to holdSetupUntilResolved to be consistent with the holdStartUntilResolved we may want to implement in the future.

Yeah, I can see the point that an alternative implementation might require at least a comparable amount of effort, so my ✅ to move with the current proposal.

Great, thanks 👍

[azasypkin]

Hey all,

I believe I've addressed the feedback you shared so far, and RFC should be ready for another review pass.

To timebox this effort, the commenting period will last for one more week, ending on June 23, 2021.

Thanks

[thomheymann]

I've been away for a bit so have probably missed a few key conversations around this. Apologies if the below comments have already been considered.

Are there any plans to build another preboot plugin at this point in time? Appreciate that there might be other use cases but until we fully understand them we're probably not going to come up with the right abstraction here and it might be easier to hook into the existing not ready server code.

If we envision multiple preboot plugins in Kibana then I think each plugin itself should define its own conditions that trigger preboot mode. (e.g. A migration assistant might want to launch based on different conditions than the interactive setup mode).

I'm also not sure how we'd reconcile multiple preboot plugins wanting to start their own preboot UI. There can only be one active at a time so would we wait until e.g. interactive setup mode is completed and then launch into the migration assistant UI? What would that mean to the user experience?

Is it desired to turn this into a first class citizen in Kibana were plugin developers can stop Kibana from starting and present their own UI or should this be a tightly controlled exception?

With this RFC, will the not ready server UI ("Kibana is not ready") then also be served by a preboot plugin moving forwards?

[azasypkin]

I've been away for a bit so have probably missed a few key conversations around this. Apologies if the below comments have already been considered.

No worries, let me know if Zoom sync would help.

Appreciate that there might be other use cases but until we fully understand them we're probably not going to come up with the right abstraction here and it might be easier to hook into the existing not ready server code.

I wish it would have been easier and non-hacky, but after dealing with multiple PoCs, I honestly don't see how that would be possible. Do you have any concrete ideas? I'd be happy to discuss!

If we envision multiple preboot plugins in Kibana then I think each plugin itself should define its own conditions that trigger preboot mode. (e.g. A migration assistant might want to launch based on different conditions than the interactive setup mode).

Well, preboot stage isn't really triggered, it's a permanent part of the Kibana boot flow that exists (and technically already existed from the moment we introduced not-ready-server) independently from preboot plugins. This RFC does allow preboot plugins to define their own conditions to temporarily suspend the boot flow if they need to.

I'm also not sure how we'd reconcile multiple preboot plugins wanting to start their own preboot UI. There can only be one active at a time so would we wait until e.g. interactive setup mode is completed and then launch into the migration assistant UI? What would that mean to the user experience?

The way preboot plugins function isn't that different from how standard plugins function. If I'm not missing anything, there can be only one active "app" at a time whether it's preboot one or not, and the default "app" is always configured somewhere (e.g via uiSettings.overrides.defaultRoute == /app/home or / for the preboot stage).

What would that mean to the user experience?

Exact UX and cross-preboot-plugin interaction flows are a bit outside of the scope of the RFC. The goal is not to mandate one single way at this point, but to give enough basic building blocks to build the flow we may need. Having said that, there are a plenty of options, e.g. userSetup is an entry-point, like /app/home in fully booted Kibana. There we can give user options where to go to next, or make a decision for them based on Kibana status, current config, interactions with other preboot plugins, or anything along these lines.

Is it desired to turn this into a first class citizen in Kibana were plugin developers can stop Kibana from starting and present their own UI or should this be a tightly controlled exception?

It's desired to turn this into a first class citizen to leverage existing Core functionality without band-aid-like hacks here and there, but it's also not meant to be used by non-platform plugin developers. Preboot stage is a "service" stage, and preboot plugins are "service" plugins that are meant to improve Kibana getting started (and potentially after-upgrade) experience from the operational stand point. If you ask me, changes in the "service" plugins as well as in any Core/Kernel level code should always be treated as tightly controlled exceptions.

With this RFC, will the not ready server UI ("Kibana is not ready") then also be served by a preboot plugin moving forwards?

Yes, that's the plan.

[jportner]

I haven't been away, but I was otherwise preoccupied! Apologies for being late to the party 😄

I had the same questions/concerns that @thomheymann already outlined above. Reading the motivation it seems like we have a need for one plugin to consume this (userSetup), and as mentioned in other comments, there are some vague assumptions that these lifecycle hooks may be useful for Cloud at some point in the future.

The goal is not to mandate one single way at this point, but to give enough basic building blocks to build the flow we may need. Having said that, there are a plenty of options, e.g. userSetup is an entry-point, like /app/home in fully booted Kibana. There we can give user options where to go to next, or make a decision for them based on Kibana status, current config, interactions with other preboot plugins, or anything along these lines.

I wonder if it wouldn't be simpler to build this into Core, instead of exposing the pluggable interface for a single plugin to take advantage of. But in the interest of avoiding bike-shedding, this justification sounds agreeable to me. I don't have any other concerns, this seems very well-thought-out approach 👏

[azasypkin]

I wonder if it wouldn't be simpler to build this into Core, instead of exposing the pluggable interface for a single plugin to take advantage of

Yes, that's a reasonable point, building user setup mode entirely inside of the Core would certainly be possible too. The reason for preferring preboot plugins to built-in Core functionality was that most of the changes still have to be done anyway (support for rendering using the preboot HTTP server, suspending of the boot flow etc.), and we'll have to modify the client-side of the Core in this case that would likely lead to a similar level of effort in the end.

I agree that the benefits of clearer separation and leveraging of the plugin infrastructure are purely theoretical at this point, and are weak arguments here even though I like to think that these are important 🙂

[joshdover]

I'm comfortable moving forward with this. Thanks @azasypkin for all of your very detailed explanations 💯

I do share similar concerns to @thomheymann about exposing this prematurely to all plugin developers but I'd also like to avoid as much churn in the Core codebase as possible. Core changes are higher risk in nature, though the same could be said for any plugins that interact with the preboot extension point.

If you ask me, changes in the "service" plugins as well as in any Core/Kernel level code should always be treated as tightly controlled exceptions.

I absolutely agree, however I don't think we yet have a good way of auditing or monitoring this. Of course I expect we'll have exhaustive functional tests of this functionality, but it may also be wise to add a mechanism for the Core & Security teams to be notified when a new preboot plugin is added. We do something similar for Telemetry schema & mapping changes where CI will fail if a static file is not updated with the schema changes and that file is under Core's CODEOWNERS so we're pinged on all PRs. It's actually been really helpful for catching things before they're merged. Could we do something similar in scope of this project?

A similar idea came up in a discussion recently for monitoring new plugins in general, so maybe we do this for all new plugins? (cc @stacey-gammon @kobelb @tylersmalley)

[azasypkin]

but it may also be wise to add a mechanism for the Core & Security teams to be notified when a new preboot plugin is added. We do something similar for Telemetry schema & mapping changes where CI will fail if a static file is not updated with the schema changes and that file is under Core's CODEOWNERS so we're pinged on all PRs. It's actually been really helpful for catching things before they're merged. Could we do something similar in scope of this project?

A similar idea came up in a discussion recently for monitoring new plugins in general, so maybe we do this for all new plugins?

Yeah, that's a great idea. I think that pinging the entire Platform group (COS - Core, Operations, and Security) whenever a new non-test/non-example kibana.json is added would be the best option - we all should be aware of new plugins to do a basic soundness check anyway (proper required/optional dependencies, reasonable tsconfig.json and jest.config.js, completed security audit etc.).

I'll sync with @elastic/kibana-operations on that once the time-sensitive work is done then.

[azasypkin]

Thanks everyone for review and feedback! I'm going to merge this RFC now and immediately proceed to the implementation to hopefully meet the 8.0.0-alpha1 deadline (July 20th).

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 99318 or prevent reminders by adding the backport:skip label.