Merge branch 'main' of https://github.com/ScottLogic/finos-vuu into S…

…LVUU-102-simplify-get-application-layout-contract

docs/getting_started/adding.md

@@ -10,13 +10,13 @@ You can add them to your pom by referencing the parent pom directly.
 
 ```
     <dependency>
-        <groupId>org.finos</groupId>
+        <groupId>org.finos.vuu</groupId>
         <artifactId>vuu-parent</artifactId>
         <version>{check the latest version}</version>
     </dependency>
 
         <dependency>
-        <groupId>org.finos</groupId>
+        <groupId>org.finos.vuu</groupId>
         <artifactId>vuu-ui</artifactId>
         <version>{check the latest version}</version>
     </dependency>

docs/getting_started/using_vuu_from_java.md

@@ -4,6 +4,6 @@ import { SvgDottySeparator } from "@site/src/components/SvgDottySeparator";
 
 <SvgDottySeparator style={{marginBottom: 32}}/>
 
-We have a sample Maven project on the Vuu Github site:
+We have a sample Maven module within the code repo:
 
-[Getting Started in Java](https://github.com/venuu-io/vuu-getting-started)
+[Getting Started in Java](https://github.com/finos/vuu/tree/main/example/main-java)

docs/introduction/how_does_it_work.md

@@ -2,7 +2,7 @@
 
 Let's start with a diagram.
 
-![](./diagrams-server-internals.png)
+![](./diagrams-server-internals.svg)
 
 This shows the basic flow through the system, there are components which have been omitted for clarity.
 
@@ -20,7 +20,7 @@ out of band on a separate thread to the update path (Filter and Sort Thread.) Th
 table you have asked for in your viewport and applies the filters and sorts you've requested. The resulting array of
 primary keys is then pushed into your viewport.
 
-THe update path, by comparison, is different. If you have a particular key in the visible range of your viewport already
+The update path, by comparison, is different. If you have a particular key in the visible range of your viewport already
 and it is updated, it follows the tick() path, so it will progress through the system as an event, on the same thread
 as the provider itself.
 

docs/perf/indices.md

@@ -1,6 +1,6 @@
 # Indices
 
-Indices are a mechansim for filter a tables keys quickly. They are defined as part of a table  and under the hood they are implemented as skip lists. 
+Indices are a mechanism for filter a tables keys quickly. They are defined as part of a table  and under the hood they are implemented as skip lists. 
 Currently, there is no query planner for indices, as a more advanced SQL database might have. They simply will shortcut the filtering process if an index exists on a field. 
 
 Adding indices to a field dramatically reduces the cost of filter on that field, at the memory expense of maintaining an extra data structure and slight processing overhead.  

docs/providers_tables_viewports/lifecycle.md

@@ -56,6 +56,6 @@ When Vuuu starts up it evaluates components from the furthest node out back to t
 
 ## What does the Vui lifecycle look like on startup?
 
-![Lifecycle Vuu](./vuu.svg)
+![Lifecycle Vuu](./lifecycle-startup.svg)
 
 As you can see from this graph. The server has as well defined startup and shutdown sequencer, controlled by its lifecycle.

docs/providers_tables_viewports/providers.md

@@ -5,7 +5,7 @@ import { SvgDottySeparator } from "@site/src/components/SvgDottySeparator";
 <SvgDottySeparator style={{marginBottom: 32}}/>
 
 Providers are classes which receive data from a particlar location (network, file, in-process lib) and format that data into a map which matches the shape of the table
-that the provider is populating. THey have a very simple interface:
+that the provider is populating. They have a very simple interface:
 
 Included below is an example of the metrics provider.
 

docs/providers_tables_viewports/tables.md

@@ -44,14 +44,14 @@ processUpdate functions like an upsert in a SQL data (i.e. it is used for both a
 # (Simple) Table
 
 Simple Tables (the default table type, defined by using the TableDef() class) are sinks for data. They are wrappers around
-a concurrent map and are mechanisms to propogate update or delete events to join tables and view ports.
+a concurrent map and are mechanisms to propagate update or delete events to join tables and view ports.
 
 Currently simple tables are limited to having strings as the key. This is likely to change in future.
 
 # Join Tables
 
 Join tables represent the logical joining of two separate tables into a single merged table. In practice they are mappings of
-keys from one table to keys from one or more other tables. When data is realised (i.e. sent down to a user's ui via the websocket)
+keys from one table to keys from one or more other tables. When data is realized (i.e. sent down to a user's ui via the websocket)
 the relevant rows are realized by dragging the data from the underlying simple tables.
 
 # AutoSubscribe Table
@@ -69,7 +69,7 @@ Session tables are specific types of tables that live only during the users conn
 
 ### Tree Session Tables
 
-Tree Session tables are created dyanmically whenever there is a request to tree an underlying flat table. THe reason for this is that Tree's are a view ontop of
+Tree Session tables are created dynamically whenever there is a request to tree an underlying flat table. THe reason for this is that Tree's are a view on top of
 and underlying raw table. When we create a tree, we are generating a tree data structure in memory whose leaves are keys that point back to the original rows
 in the underlying table. When your session is closed, the server cleans up these tree tables, freeing up resources.
 

docs/providers_tables_viewports/viewports.md

@@ -8,7 +8,7 @@ A Viewport is a specific client's view onto an underlying table. It has knowledg
 the window of data that a client currently has displayed on her screen. It contains information on any sorts, or filters that a specific client
 has requested on the data, on columns that the client has asked to display as well as information such as which rows are currently selected on the clients grid.
 
-As well as this viewports contain references to immutable arrays of the keys of the underlying table. These arrays are sorted and filtered based on the clients
+As well as the above, viewports contain references to immutable arrays of the keys of the underlying table. These arrays are sorted and filtered based on the clients
 requested sort of the data.
 
 When a user opens a viewport on a table from the client, a thread in the server will asynchronously populate the keys based on the viewports parameters (sorts, filters, etc..) into an immutable array
@@ -17,8 +17,8 @@ and will pass that array to the viewport. This thread will then continuously rec
 The row that is sent to a user is only realized in the viewport at the point the row becomes visible in the client (or part of the pre-post fetch.) This occurs by dragging the fields from the underlying tables
 when an update needs to be sent to the client.
 
-![Viewport](./diagrams-view-ports.png)
+![Viewport](./viewports.svg)
 
 And in the context of the wider Vuu server.
 
-![Viewport](./diagrams-server-internals.png)
+![Viewport](./server-internals.svg)

docs/ui/calculated_columns.md

@@ -46,13 +46,13 @@ The calculated column would at minimum need to contain:
 though datatype might be surplus or could be inferred.
 
 **Question:** How would we infer datatype? How would we cater for nasty expressions like "= price _ clientName"? possible we could error. We could use
-precendence in the datatypes, i.e. first column sets return value.? Or we could look for widest type in the event it was int _ double or int \* long.
-It would likely be porr UX to ask the user to define the return type.
+precedence in the datatypes, i.e. first column sets return value.? Or we could look for widest type in the event it was int _ double or int \* long.
+It would likely be poor UX to ask the user to define the return type.
 
 ### Use in Tree'd Viewports
 
 By default calculated columns would work the same in tree'd viewports as in non-tree'd viewports. THe only caveat to that
-would be when the calculated column would be a branch in the tree. In that case the column values would have to be calcuated in the
+would be when the calculated column would be a branch in the tree. In that case the column values would have to be calculated in the
 tree building function, which may slow down tree generation for specific viewports.
 
 ### Implementation

docs/ui/vuu_ui_features.md

@@ -0,0 +1,61 @@
+# How Features currently work in Vuu
+
+# Vuu Application = Vuu Shell + Feature(s)
+
+A Vuu ui is an instance of the Vuu Shell, which loads content dynamically at runtime. The Shell renders the outermost chrome of the application:
+
+- the App Header
+- the left nav
+- the Context panel
+- a container for the main content
+
+The shell creates the WebWorker from which all communication with the server will be handled. It provides the layout system, a persistence service and other application level features. COmponents implementing the business functionality of a Vuu application will be loaded dymanically (by the Shell) and rendered within the main content area.
+
+Feature is the term used in Vuu to describe a UI component that can be loaded into the application to provide some specific functionality. This can be as simple as a single data table that displays data from a remote Vuu server. Equally, it can be a complex component with multiple tabbed pages that orchestrates data from multiple Vuu data tables. The Feature might occupy the full content area of the app, or it may be one of multiple features assembled within a `layout`. These are decisions that can be enforced by the application developer or made available to the end user.
+
+## Loading Features at runtime
+
+Technically, a feature is an ES module with a default export which must be a React component. The module will be loaded at runtime, using standard ES module loading. In other words, the module is known to the application by a url, which will be used to load it via a dynamic import statement.
+
+The component that actually implements the dynamic loading is `Feature`, which can be found in the `vuu-shell` package.
+
+Here are props expected/supported by `Feature`
+
+```TypeScript
+export interface FeatureProps<P extends object | undefined = any> {
+  ComponentProps?: P;
+  css?: string;
+  height?: number;
+  title?: string;
+  url: string;
+  width?: number;
+}
+```
+
+The only required prop is `url`. That is the url for the JavaScript bundle that exports the featured component. Most features will also ship a css bundle. By default, features will be rendered within the Vuu UI with a header. This will dsplay the feature `title`, if provided. `ComponentProps` will be passed to the component actually rendered (i.e. the component exported by the feature bundle), so these should appropriate to that component. Vuu provide a mechanism for features to persist state between sessions. This can include props, which will be injected back into a loaded feature at construction time.
+
+## How are Feature bundles built - current state
+
+Right now, both the bundle exporting the Vuu Shell and all feature bundles are built together in a single build task. ESBuild is used for this and the code splitting features of ESBuild determine the exact breakdown of code into multiple bundles. The main application will output a bundle. Each feature is defined as an entrypoint to the build, so each feature will also output a bundle. These are the feature bundles that will be loaded dynamically. Any number of additional bundles may be created as dependencies, as ESBuild identifies opportunities for code sharing across bundles.
+
+## How will Feature bundles be built - future state
+
+It is not ideal that all bundles must be created, together with the runtime shell, in a single build. If one feature gets an update, the entire app must be rebuilt and redeployed to make this update available. Vuu is going to move to a more dynamic module system, whereby feature bundles can be built and published independently. The challenge here is managing shared depedencies. The current plan is to use Vite based `Module Federation` to achieve this. It is a system designed for exactly this scenario. This will be implemented alongside a runtime discovery mechanism, so that newly published or republished feature bundles can be indentified and surfaced in a running application.
+
+## What is a Vuu `View` and what is the relationship between a `View` and a `Feature`
+
+## How does a Feature manage data communication with the Vuu Server ?
+
+## How does a Vuu app know which Feature(s) to load ?
+
+## Getting started - how do I create a Feature ?
+
+## Does the Vuu Showcase support Features ?
+
+Yes it does. The Vuu showcase is a developer tool that allows components to be rendered in isolation, with hot module reloading for a convenient developer experience. Features are a little more complex to render here because of the injection of props by the Vuu Shell
+and the fact that many features will create Vuu datasources. When running in the Showcase, we might want to use local datasources with no requirement to have a Vuu server instance running. All of this can be achieved and existing Showcase examples demonstrate how this is done.
+
+### dataSource creation/injection in Showcase features
+
+Most features will create Vuu dataSource(s) internally. However there is a pattern for dataSource creation, descibed above. Features should use the session state service provided by the Vuu shell to manage any dataSources created. The Showcase can take advantage of this pattern by pre-populating the session store with dataSources. The Feature, when loaded will then use these dataSources rather than creating.
+In the existing Showcase examples, there is a `features` folder. The components in here are wrappers round actual VuuFeatures implemented in sample-apps. These render the actual underlying feature but only after creating local versions of the dataSource(s) required by those features and storing them in session state. WHen these features are rendered in SHowcase examples, they will be using the local test data.

vuu-ui/packages/vuu-data-test/src/TickingArrayDataSource.ts

@@ -11,6 +11,7 @@ import { DataSourceRow } from "@finos/vuu-data-types";
 import {
   ClientToServerEditRpc,
   ClientToServerMenuRPC,
+  ClientToServerViewportRpcCall,
   VuuMenu,
   VuuRange,
   VuuRowDataItemType,
@@ -28,18 +29,21 @@ export interface TickingArrayDataSourceConstructorProps
   extends Omit<ArrayDataSourceConstructorProps, "data"> {
   data?: Array<VuuRowDataItemType[]>;
   menu?: VuuMenu;
+  menuRpcServices?: RpcService[];
   rpcServices?: RpcService[];
   table?: Table;
   updateGenerator?: UpdateGenerator;
 }
 
 export class TickingArrayDataSource extends ArrayDataSource {
+  #menuRpcServices: RpcService[] | undefined;
   #rpcServices: RpcService[] | undefined;
   #updateGenerator: UpdateGenerator | undefined;
   #table?: Table;
 
   constructor({
     data,
+    menuRpcServices,
     rpcServices,
     table,
     updateGenerator,
@@ -54,6 +58,7 @@ export class TickingArrayDataSource extends ArrayDataSource {
       data: data ?? table?.data ?? [],
     });
     this._menu = menu;
+    this.#menuRpcServices = menuRpcServices;
     this.#rpcServices = rpcServices;
     this.#updateGenerator = updateGenerator;
     this.#table = table;
@@ -153,6 +158,23 @@ export class TickingArrayDataSource extends ArrayDataSource {
     return Promise.resolve(true);
   }
 
+  async rpcCall<T extends RpcResponse = RpcResponse>(
+    rpcRequest: Omit<ClientToServerViewportRpcCall, "vpId">
+  ) {
+    const rpcService = this.#rpcServices?.find(
+      (service) =>
+        service.rpcName ===
+        (rpcRequest as ClientToServerViewportRpcCall).rpcName
+    );
+    if (rpcService) {
+      return rpcService.service({
+        ...rpcRequest,
+      });
+    } else {
+      console.log(`no implementation for PRC service ${rpcRequest.rpcName}`);
+    }
+  }
+
   async menuRpcCall(
     rpcRequest: Omit<ClientToServerMenuRPC, "vpId"> | ClientToServerEditRpc
   ): Promise<

vuu-ui/packages/vuu-data-test/src/basket/basket-module.ts

@@ -7,7 +7,11 @@ import ftse from "./reference-data/ftse100";
 import nasdaq from "./reference-data/nasdaq100";
 import sp500 from "./reference-data/sp500";
 import hsi from "./reference-data/hsi";
-import { VuuMenu, VuuRowDataItemType } from "@finos/vuu-protocol-types";
+import {
+  ClientToServerViewportRpcCall,
+  VuuMenu,
+  VuuRowDataItemType,
+} from "@finos/vuu-protocol-types";
 import { Table } from "../Table";
 
 // This is a 'local' columnMap
@@ -220,13 +224,11 @@ function createTradingBasket(basketId: string, basketName: string) {
   });
 }
 
-async function createNewBasket(rpcRequest: any) {
-  const { basketName, selectedRows } = rpcRequest;
-  if (selectedRows.length === 1) {
-    const [row] = selectedRows;
-    const basketId = row[KEY];
-    createTradingBasket(basketId, basketName);
-  }
+async function createNewBasket(rpcRequest: ClientToServerViewportRpcCall) {
+  const {
+    params: [basketId, basketName],
+  } = rpcRequest;
+  createTradingBasket(basketId, basketName);
 }
 
 //-------------------
@@ -298,7 +300,7 @@ const services: Record<BasketsTableName, RpcService[] | undefined> = {
   algoType: undefined,
   basket: [
     {
-      rpcName: "CREATE_NEW_BASKET",
+      rpcName: "createBasket",
       service: createNewBasket,
     },
   ],

vuu-ui/packages/vuu-data-test/src/basket/basket-schemas.ts

@@ -34,7 +34,6 @@ export const schemas: Readonly<
     columns: [
       { name: "basketId", serverDataType: "string" },
       { name: "change", serverDataType: "string" },
-      // this column doesn't exist on Vuu server
       { name: "description", serverDataType: "string" },
       { name: "lastTrade", serverDataType: "string" },
       { name: "ric", serverDataType: "string" },

vuu-ui/packages/vuu-data-test/src/simul/simul-module.ts

@@ -52,7 +52,11 @@ export const populateArray = (tableName: SimulTableName, count: number) => {
 
 const getColumnDescriptors = (tableName: SimulTableName) => {
   const schema = schemas[tableName];
-  return schema.columns;
+  if (schema) {
+    return schema.columns;
+  } else {
+    console.error(`simul-module no schema found for table SIMUL ${tableName}`);
+  }
 };
 
 const createDataSource = (tableName: SimulTableName) => {

vuu-ui/packages/vuu-data/src/array-data-source/array-data-source.ts

@@ -101,7 +101,6 @@ export class ArrayDataSource
   private groupMap: undefined | GroupMap;
   /** the index of key field within raw data row */
   private key: number;
-  private tableSchema: TableSchema;
   private lastRangeServed: VuuRange = { from: 0, to: 0 };
   private rangeChangeRowset: "delta" | "full";
   private openTreeNodes: string[] = [];
@@ -120,6 +119,7 @@ export class ArrayDataSource
   protected _menu: VuuMenu | undefined;
   protected selectedRows: Selection = [];
 
+  public tableSchema: TableSchema;
   public viewport: string;
 
   private keys = new KeySet(this.#range);

vuu-ui/packages/vuu-data/src/data-source.ts

@@ -24,6 +24,7 @@ import { EventEmitter } from "@finos/vuu-utils";
 import { TableSchema } from "./message-utils";
 import {
   MenuRpcResponse,
+  ViewportRpcResponse,
   VuuUIMessageInRPCEditReject,
   VuuUIMessageInRPCEditResponse,
 } from "./vuuUIMessageTypes";
@@ -495,7 +496,8 @@ export type DataSourceInsertHandler = (
 export type RpcResponse =
   | MenuRpcResponse
   | VuuUIMessageInRPCEditReject
-  | VuuUIMessageInRPCEditResponse;
+  | VuuUIMessageInRPCEditResponse
+  | ViewportRpcResponse;
 
 export type RpcResponseHandler = (response: RpcResponse) => boolean;
 
@@ -556,9 +558,9 @@ export interface DataSource extends EventEmitter<DataSourceEvents> {
   menuRpcCall: (
     rpcRequest: Omit<ClientToServerMenuRPC, "vpId"> | ClientToServerEditRpc
   ) => Promise<RpcResponse | undefined>;
-  rpcCall?: (
+  rpcCall?: <T extends RpcResponse = RpcResponse>(
     message: Omit<ClientToServerViewportRpcCall, "vpId">
-  ) => Promise<RpcResponse | undefined>;
+  ) => Promise<T | undefined>;
   openTreeNode: (key: string) => void;
   range: VuuRange;
   select: SelectionChangeHandler;