[Dataset quality] State management (elastic#174906)

## 📝 Summary

This PR contains `dataset-quality` plugin state management, we have
decided to go with xstate.
The general idea, and following
elastic#170200 as inspiration, we wanted
to detach `dataset-quality` plugin state from its consumers, in this way
our plugin wouldn't perform side effect, such as updating routes,
outside its scope.

The flow of the information now looks like

<img width="543" alt="image"
src="https://github.com/elastic/kibana/assets/1313018/6a723f5b-047b-4f81-aef5-dbfce1ac0181">


Current internal state of the plugin is fairly simple but it's
envisioned to grow in a near future with for example filters, flyout
options, etc.

## 💡 Notes for Logs UX reviewers

### Dataset Quality plugin

#### Goals

**Decoupling from global state**: The primary goal is to decouple the
`<DatasetQuality>` component from direct dependencies on the URL and
other page-wide side effects. Decoupling from global state enables its
use in other applications without interfering with their page state.

**Stable and Strictly Typed Public API**: Introduce a public API for the
`<DatasetQuality>` component. This API provides consumers the ability to
subscribe to the component's state and/or initialize the state from the
outside.

#### Architecture

The architecture of the `<DatasetQuality>` plugin has been designed to
provide a modular structure, with a focus on robust state management.
This structure is primarily composed of the uncontrolled
`<DatasetQuality>` component and a separately instantiable controller.

**Uncontrolled `<DatasetQuality>` component**: The `<DatasetQuality>` is
designed as an uncontrolled component. All the logic around this
component is handled by hooks, such as `useDatasetQualityTable` with the
ability to change or replace the underlying dependencies.

**Separately instantiable controller**: A controller is introduced to
encapsulate and manage the business logic associated with the
`<DatasetQuality>` component. The controller centralizes the business
logic, separating it from the UI layer. This provides flexibility in
managing different instances of the `<DatasetQuality>` and reusability
from different consumers.

#### App statechart

<img width="1041" alt="image"
src="https://github.com/elastic/kibana/assets/1313018/7da9e424-8577-4a5e-8c3e-46fb1df83948">

### Observability Logs Explorer App

#### Goals

**URL persistence**: Implement a versioned data structure for URL
persistence, this give us the flexibility to extend or change the app
state without workarounds. The general idea of having the public state
of the dataset quality plugin stored in the URL of this consumer is the
ability to share an exact state with colleagues, customers, etc.

#### Changes

**Introduction of versioned URL schema**: This new schema will
standarize how URL-based state is managed, providing a clear and
consistent mechanism for encoding and decoding state information in the
URL. The versioning will allow us to evolve the data structure in a
backwards-compatible way incorporate features added or changed in the
future.

**Page-level statechart implementation**: This introduces a page-level
statechart to orchestrate the initialization and instantiation of the
`<DatasetQuality>` controller.

#### App statechart

<img width="936" alt="image"
src="https://github.com/elastic/kibana/assets/1313018/e3f1c61b-322b-4054-a30e-157eadb6de6b">

---------

Co-authored-by: kibanamachine <[email protected]>

x-pack/plugins/dataset_quality/common/api_types.ts

@@ -54,8 +54,9 @@ export const degradedDocsRt = rt.type({
 
 export type DegradedDocs = rt.TypeOf<typeof degradedDocsRt>;
 
-export const dataStreamDetailsRt = rt.type({
+export const dataStreamDetailsRt = rt.partial({
   createdOn: rt.number,
+  lastActivity: rt.number,
 });
 
 export type DataStreamDetails = rt.TypeOf<typeof dataStreamDetailsRt>;

x-pack/plugins/dataset_quality/common/constants.ts

@@ -10,3 +10,5 @@ export const DEFAULT_DATASET_TYPE = 'logs';
 
 export const POOR_QUALITY_MINIMUM_PERCENTAGE = 3;
 export const DEGRADED_QUALITY_MINIMUM_PERCENTAGE = 0;
+export const DEFAULT_SORT_FIELD = 'title';
+export const DEFAULT_SORT_DIRECTION = 'asc';

x-pack/plugins/dataset_quality/common/data_streams_stats/integration.ts

@@ -9,8 +9,8 @@ import { IntegrationType } from './types';
 
 export class Integration {
   name: IntegrationType['name'];
-  title: IntegrationType['title'];
-  version: IntegrationType['version'];
+  title: string;
+  version: string;
   icons?: IntegrationType['icons'];
 
   private constructor(integration: Integration) {

x-pack/plugins/dataset_quality/common/data_streams_stats/types.ts

@@ -33,4 +33,4 @@ export type GetDataStreamDetailsResponse =
   APIReturnType<`GET /internal/dataset_quality/data_streams/{dataStream}/details`>;
 
 export type { DataStreamStat } from './data_stream_stat';
-export type { DataStreamDetails } from './data_stream_details';
+export type { DataStreamDetails } from '../api_types';

x-pack/plugins/dataset_quality/public/components/dataset_quality/columns.tsx

@@ -22,7 +22,7 @@ import { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
 import { ES_FIELD_TYPES, KBN_FIELD_TYPES } from '@kbn/field-types';
 import { i18n } from '@kbn/i18n';
 import { FormattedMessage } from '@kbn/i18n-react';
-import React, { Dispatch, SetStateAction } from 'react';
+import React from 'react';
 import { css } from '@emotion/react';
 import {
   DEGRADED_QUALITY_MINIMUM_PERCENTAGE,
@@ -32,6 +32,7 @@ import { DataStreamStat } from '../../../common/data_streams_stats/data_stream_s
 import { QualityIndicator, QualityPercentageIndicator } from '../quality_indicator';
 import { IntegrationIcon } from '../common';
 import { useLinkToLogExplorer } from '../../hooks';
+import { FlyoutDataset } from '../../state_machines/dataset_quality_controller';
 
 const expandDatasetAriaLabel = i18n.translate('xpack.datasetQuality.expandLabel', {
   defaultMessage: 'Expand',
@@ -108,25 +109,25 @@ const degradedDocsColumnTooltip = (
 export const getDatasetQualityTableColumns = ({
   fieldFormats,
   selectedDataset,
-  setSelectedDataset,
+  openFlyout,
   loadingDegradedStats,
 }: {
   fieldFormats: FieldFormatsStart;
-  selectedDataset?: DataStreamStat;
+  selectedDataset?: FlyoutDataset;
   loadingDegradedStats?: boolean;
-  setSelectedDataset: Dispatch<SetStateAction<DataStreamStat | undefined>>;
+  openFlyout: (selectedDataset: FlyoutDataset) => void;
 }): Array<EuiBasicTableColumn<DataStreamStat>> => {
   return [
     {
       name: '',
       render: (dataStreamStat: DataStreamStat) => {
-        const isExpanded = dataStreamStat === selectedDataset;
+        const isExpanded = dataStreamStat.rawName === selectedDataset?.rawName;
 
         return (
           <EuiButtonIcon
             size="m"
             color="text"
-            onClick={() => setSelectedDataset(isExpanded ? undefined : dataStreamStat)}
+            onClick={() => openFlyout(dataStreamStat as FlyoutDataset)}
             iconType={isExpanded ? 'minimize' : 'expand'}
             title={!isExpanded ? expandDatasetAriaLabel : collapseDatasetAriaLabel}
             aria-label={!isExpanded ? expandDatasetAriaLabel : collapseDatasetAriaLabel}

x-pack/plugins/dataset_quality/public/components/dataset_quality/context.ts

@@ -5,10 +5,10 @@
  * 2.0.
  */
 import { createContext, useContext } from 'react';
-import { IDataStreamsStatsClient } from '../../services/data_streams_stats';
+import { DatasetQualityControllerStateService } from '../../state_machines/dataset_quality_controller';
 
 export interface DatasetQualityContextValue {
-  dataStreamsStatsServiceClient: IDataStreamsStatsClient;
+  service: DatasetQualityControllerStateService;
 }
 
 export const DatasetQualityContext = createContext({} as DatasetQualityContextValue);

x-pack/plugins/dataset_quality/public/components/dataset_quality/dataset_quality.tsx

@@ -4,32 +4,34 @@
  * 2.0; you may not use this file except in compliance with the Elastic License
  * 2.0.
  */
-import React from 'react';
+import React, { useMemo } from 'react';
 import { CoreStart } from '@kbn/core/public';
 import { EuiFlexGroup, EuiFlexItem } from '@elastic/eui';
-import { DataStreamsStatsService } from '../../services/data_streams_stats/data_streams_stats_service';
+import { dynamic } from '@kbn/shared-ux-utility';
 import { DatasetQualityContext, DatasetQualityContextValue } from './context';
 import { useKibanaContextForPluginProvider } from '../../utils';
 import { DatasetQualityStartDeps } from '../../types';
-import { Header } from './header';
-import { Table } from './table';
+import { DatasetQualityController } from '../../controller';
+
+export interface DatasetQualityProps {
+  controller: DatasetQualityController;
+}
 
 export interface CreateDatasetQualityArgs {
   core: CoreStart;
   plugins: DatasetQualityStartDeps;
 }
 
 export const createDatasetQuality = ({ core, plugins }: CreateDatasetQualityArgs) => {
-  return () => {
+  return ({ controller }: DatasetQualityProps) => {
     const KibanaContextProviderForPlugin = useKibanaContextForPluginProvider(core, plugins);
 
-    const dataStreamsStatsServiceClient = new DataStreamsStatsService().start({
-      http: core.http,
-    }).client;
-
-    const datasetQualityProviderValue: DatasetQualityContextValue = {
-      dataStreamsStatsServiceClient,
-    };
+    const datasetQualityProviderValue: DatasetQualityContextValue = useMemo(
+      () => ({
+        service: controller.service,
+      }),
+      [controller.service]
+    );
 
     return (
       <DatasetQualityContext.Provider value={datasetQualityProviderValue}>
@@ -41,6 +43,9 @@ export const createDatasetQuality = ({ core, plugins }: CreateDatasetQualityArgs
   };
 };
 
+const Header = dynamic(() => import('./header'));
+const Table = dynamic(() => import('./table'));
+
 function DatasetQuality() {
   return (
     <EuiFlexGroup direction="column" gutterSize="m">

x-pack/plugins/dataset_quality/public/components/dataset_quality/header.tsx

@@ -9,6 +9,8 @@ import { EuiPageHeader } from '@elastic/eui';
 import React from 'react';
 import { datasetQualityAppTitle } from '../../../common/translations';
 
-export function Header() {
+// Allow for lazy loading
+// eslint-disable-next-line import/no-default-export
+export default function Header() {
   return <EuiPageHeader bottomBorder pageTitle={datasetQualityAppTitle} />;
 }

x-pack/plugins/dataset_quality/public/components/dataset_quality/table.tsx

@@ -8,9 +8,11 @@
 import React from 'react';
 import { EuiBasicTable, EuiHorizontalRule, EuiSpacer, EuiText, EuiEmptyPrompt } from '@elastic/eui';
 import { FormattedMessage } from '@kbn/i18n-react';
+import { dynamic } from '@kbn/shared-ux-utility';
 import { loadingDatasetsText, noDatasetsTitle } from '../../../common/translations';
 import { useDatasetQualityTable } from '../../hooks';
-import { Flyout } from '../flyout';
+
+const Flyout = dynamic(() => import('../flyout/flyout'));
 
 export const Table = () => {
   const {
@@ -69,3 +71,7 @@ export const Table = () => {
     </>
   );
 };
+
+// Allow for lazy loading
+// eslint-disable-next-line import/no-default-export
+export default Table;

x-pack/plugins/dataset_quality/public/components/flyout/dataset_summary.tsx

@@ -8,29 +8,26 @@
 import React from 'react';
 import { FieldFormatsStart } from '@kbn/field-formats-plugin/public';
 import { ES_FIELD_TYPES, KBN_FIELD_TYPES } from '@kbn/field-types';
+import { DataStreamDetails } from '../../../common/data_streams_stats';
 import {
   flyoutDatasetCreatedOnText,
   flyoutDatasetDetailsText,
   flyoutDatasetLastActivityText,
 } from '../../../common/translations';
-import { DataStreamStat, DataStreamDetails } from '../../../common/data_streams_stats';
 import { FieldsList, FieldsListLoading } from './fields_list';
 
 interface DatasetSummaryProps {
   fieldFormats: FieldFormatsStart;
   dataStreamDetails?: DataStreamDetails;
-  dataStreamStat: DataStreamStat;
 }
 
-export function DatasetSummary({
-  dataStreamStat,
-  dataStreamDetails,
-  fieldFormats,
-}: DatasetSummaryProps) {
+export function DatasetSummary({ dataStreamDetails, fieldFormats }: DatasetSummaryProps) {
   const dataFormatter = fieldFormats.getDefaultInstance(KBN_FIELD_TYPES.DATE, [
     ES_FIELD_TYPES.DATE,
   ]);
-  const formattedLastActivity = dataFormatter.convert(dataStreamStat.lastActivity);
+  const formattedLastActivity = dataStreamDetails?.lastActivity
+    ? dataFormatter.convert(dataStreamDetails?.lastActivity)
+    : '-';
   const formattedCreatedOn = dataStreamDetails?.createdOn
     ? dataFormatter.convert(dataStreamDetails.createdOn)
     : '-';

x-pack/plugins/dataset_quality/public/components/flyout/flyout.tsx

@@ -15,46 +15,29 @@ import {
   EuiSpacer,
 } from '@elastic/eui';
 import React, { Fragment } from 'react';
-import { DEFAULT_DATASET_TYPE } from '../../../common/constants';
-import { DataStreamStat } from '../../../common/data_streams_stats/data_stream_stat';
 import { flyoutCancelText } from '../../../common/translations';
 import { useDatasetQualityFlyout } from '../../hooks';
 import { DatasetSummary, DatasetSummaryLoading } from './dataset_summary';
 import { Header } from './header';
 import { IntegrationSummary } from './integration_summary';
+import { FlyoutProps } from './types';
 
-interface FlyoutProps {
-  dataset: DataStreamStat;
-  closeFlyout: () => void;
-}
-
-export function Flyout({ dataset, closeFlyout }: FlyoutProps) {
-  const {
-    dataStreamStat,
-    dataStreamDetails,
-    dataStreamStatLoading,
-    dataStreamDetailsLoading,
-    fieldFormats,
-  } = useDatasetQualityFlyout({
-    type: DEFAULT_DATASET_TYPE,
-    dataset: dataset.name,
-    namespace: dataset.namespace,
-  });
+// Allow for lazy loading
+// eslint-disable-next-line import/no-default-export
+export default function Flyout({ dataset, closeFlyout }: FlyoutProps) {
+  const { dataStreamStat, dataStreamDetails, dataStreamDetailsLoading, fieldFormats } =
+    useDatasetQualityFlyout();
 
   return (
     <EuiFlyout onClose={closeFlyout} ownFocus={false} data-component-name={'datasetQualityFlyout'}>
       <>
         <Header dataStreamStat={dataset} />
         <EuiFlyoutBody>
-          {dataStreamStatLoading || dataStreamDetailsLoading ? (
+          {dataStreamDetailsLoading ? (
             <DatasetSummaryLoading />
           ) : dataStreamStat ? (
             <Fragment>
-              <DatasetSummary
-                dataStreamStat={dataStreamStat}
-                dataStreamDetails={dataStreamDetails}
-                fieldFormats={fieldFormats}
-              />
+              <DatasetSummary dataStreamDetails={dataStreamDetails} fieldFormats={fieldFormats} />
               <EuiSpacer />
               {dataStreamStat.integration && (
                 <IntegrationSummary integration={dataStreamStat.integration} />

x-pack/plugins/dataset_quality/public/components/flyout/header.tsx

@@ -17,11 +17,11 @@ import {
 import { css } from '@emotion/react';
 import React from 'react';
 import { flyoutOpenInLogExplorerText } from '../../../common/translations';
-import { DataStreamStat } from '../../../common/data_streams_stats/data_stream_stat';
 import { useLinkToLogExplorer } from '../../hooks';
+import { FlyoutDataset } from '../../state_machines/dataset_quality_controller';
 import { IntegrationIcon } from '../common';
 
-export function Header({ dataStreamStat }: { dataStreamStat: DataStreamStat }) {
+export function Header({ dataStreamStat }: { dataStreamStat: FlyoutDataset }) {
   const { integration, title } = dataStreamStat;
   const euiShadow = useEuiShadow('s');
   const { euiTheme } = useEuiTheme();

x-pack/plugins/dataset_quality/public/components/flyout/types.ts

@@ -0,0 +1,13 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { FlyoutDataset } from '../../state_machines/dataset_quality_controller';
+
+export interface FlyoutProps {
+  dataset: FlyoutDataset;
+  closeFlyout: () => void;
+}

x-pack/plugins/dataset_quality/public/controller/create_controller.ts

@@ -0,0 +1,66 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { CoreStart } from '@kbn/core/public';
+import { getDevToolsOptions } from '@kbn/xstate-utils';
+import equal from 'fast-deep-equal';
+import { distinctUntilChanged, from, map } from 'rxjs';
+import { interpret } from 'xstate';
+import { DataStreamsStatsService } from '../services/data_streams_stats';
+import {
+  createDatasetQualityControllerStateMachine,
+  DEFAULT_CONTEXT,
+} from '../state_machines/dataset_quality_controller';
+import { DatasetQualityStartDeps } from '../types';
+import { getContextFromPublicState, getPublicStateFromContext } from './public_state';
+import { DatasetQualityController, DatasetQualityPublicStateUpdate } from './types';
+
+type InitialState = DatasetQualityPublicStateUpdate;
+
+interface Dependencies {
+  core: CoreStart;
+  plugins: DatasetQualityStartDeps;
+}
+
+export const createDatasetQualityControllerFactory =
+  ({ core }: Dependencies) =>
+  async ({
+    initialState = DEFAULT_CONTEXT,
+  }: {
+    initialState?: InitialState;
+  }): Promise<DatasetQualityController> => {
+    const initialContext = getContextFromPublicState(initialState ?? {});
+
+    const dataStreamStatsClient = new DataStreamsStatsService().start({
+      http: core.http,
+    }).client;
+
+    const machine = createDatasetQualityControllerStateMachine({
+      initialContext,
+      toasts: core.notifications.toasts,
+      dataStreamStatsClient,
+    });
+
+    const service = interpret(machine, {
+      devTools: getDevToolsOptions(),
+    });
+
+    const state$ = from(service).pipe(
+      map(({ context }) => getPublicStateFromContext(context)),
+      distinctUntilChanged(equal)
+    );
+
+    return {
+      state$,
+      service,
+    };
+  };
+
+export type CreateDatasetQualityControllerFactory = typeof createDatasetQualityControllerFactory;
+export type CreateDatasetQualityController = ReturnType<
+  typeof createDatasetQualityControllerFactory
+>;

x-pack/plugins/dataset_quality/public/controller/index.ts

@@ -0,0 +1,10 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export * from './create_controller';
+export * from './provider';
+export * from './types';

x-pack/plugins/dataset_quality/public/controller/lazy_create_controller.ts

@@ -0,0 +1,15 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import type { CreateDatasetQualityControllerFactory } from './create_controller';
+
+export const createDatasetQualityControllerLazyFactory: CreateDatasetQualityControllerFactory =
+  (dependencies) => async (args) => {
+    const { createDatasetQualityControllerFactory } = await import('./create_controller');
+
+    return createDatasetQualityControllerFactory(dependencies)(args);
+  };