From 4a61f34cc6f8e72657d23f5c3271805a44b6c3b1 Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Mon, 4 Jul 2022 18:43:20 +0530 Subject: [PATCH 1/9] Wip --- how-sync-works.ftd | 184 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 how-sync-works.ftd diff --git a/how-sync-works.ftd b/how-sync-works.ftd new file mode 100644 index 000000000..308226fc5 --- /dev/null +++ b/how-sync-works.ftd @@ -0,0 +1,184 @@ +-- ft.page: How sync works + + +-- ft.h1: `.fpm/workspace.ftd` + +After fpm clone, the .fpm/workspace.ftd created. This contains entry for all the files in the file system. +It is hashmap of filename to version. + +The only time the workspace is updated is when we do a sync or resolve a conflict. + + +-- ft.code: +lang: ftd + +\-- fpm.workspace-file: +timestamp: + + + +-- ft.h1: `.history/history.ftd` + +This contains the history related information for the files + + +-- ft.code: +lang: ftd + + +\-- fpm.history-file: + +\--- action.add: +timestamp: + + + + +-- ft.h1: Global sync behaviour + +1. history.ftd + + + + + +-- ft.h1: `new-file-added` + +Before sync: + +Create file in file system + +After sync: +> .history/index.v1.ftd +> Create entry in .latest.ftd +> Create entry .history/history.ftd + +```ftd + +-- fpm.history: index.ftd + +--- fpm.status: Added +timestamp: v1 + +``` + +> Create entry in `.fpm/workspace.ftd` + + +-- ft.h1: `new-file-added-after-already-added` + +Before sync: + +Create file in file system + +After sync: +> Return conflict (index.v1.ftd: content and version) +> Create entry in .fpm/workspace.ftd + +```ftd +-- fpm.workspace: index.ftd +status: conflicted +version: v1 +``` + +> Don't modify index.ftd in fs + + + +-- ft.h1: `file-deleted` + +Before sync: + +Delete file in file system + +After sync: +> Remove entry from .history/.latest.ftd +> Create entry in .history/history.ftd + +```ftd + +-- fpm.history: index.ftd + +--- fpm.status: Deleted +timestamp: v1 +``` + +> Remove entry from .fpm/workspace.ftd + + + + +-- ft.h1: `file-deleted-after-already-modified` + +Before sync: + +Delete the file + +After sync: +> Return conflict (index.v1.ftd: content and version) +> Create entry in .fpm/workspace.ftd + +```ftd +-- fpm.workspace: index.ftd +version: v2 +``` +> It won't restore the file in fs + + + + + +-- ft.h1: `file-deleted-after-already-deleted` + + +Before sync: + +Delete file in file system + +After sync: + +Nothing + +> Remove entry from `.fpm/workspace.ftd` + + + + +-- ft.h1: `file-modified` + +Before sync: + +modify the file + +After sync: + +> Create new snapshot .history/index.v2.ftd +> Create entry in .latest.ftd +> Create entry in .history/history.ftd + + + +-- ft.h1: `file-modified-after-already-deleted` + +Before sync: + +modify the file + +After sync: + +Nothing + + + +-- ft.h1: `file-modified-after-already-modified` + +Before sync: + +modify the file + +After sync: + +Nothing + + + From e52cb156db0bf2d6bd26dfdc3fe0d9af6503ce7a Mon Sep 17 00:00:00 2001 From: Amit Upadhyay Date: Mon, 4 Jul 2022 19:38:32 +0530 Subject: [PATCH 2/9] more design --- how-sync-works.ftd | 159 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 151 insertions(+), 8 deletions(-) diff --git a/how-sync-works.ftd b/how-sync-works.ftd index 308226fc5..89a4986f3 100644 --- a/how-sync-works.ftd +++ b/how-sync-works.ftd @@ -1,10 +1,158 @@ --- ft.page: How sync works +-- ft.page: How sync & CR works + + +-- ft.h1: State + +State is represented by what is with the server and what is with the client. + +-- ft.h2: Server State + +Server state is composed of history of all files, and the tracking information +about all files. + +-- ft.code: +lang: ftd + +\-- record server-state: +file-history list history: +file-tracking list tracks: + +-- ft.markdown: + +File history captures every modification to a file. It also included information +about changes that happened due to merges, if an action happened to "main +branch" via merge from a CR, we store the CR where the change came from in the +`src-cr` field. + +-- ft.code: +lang: ftd + +\-- record file-history: +caption filename: +file-edit list history: + +\-- or-type file-edit: +\--- added: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- updated: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- deleted: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +-- ft.markdown: + +Actual content is stored in the file system based on +`..`. + +Also `version` starts with 1 for every file, and is incremented by one for every +change in that file. Delete also increments the version number. + +A file can track one or more files. Any file that is tracking another file has +an entry in `server-state.track`, represented by `file-tracking`. + +-- ft.code: +lang: ftd + +\-- record file-tracking: +caption filename: +tracking-info list tracks: + +-- ft.markdown: + +Every file that is being tracked is recorded using `tracking-info`. + +-- ft.code: +lang: ftd + +\-- record tracking-info: +caption filename: +integer version: +option integer self-version: + +-- ft.markdown: + +Say `a.ftd` is tracking `b.ftd`. We have some operation that mark `a.ftd` up to +date with `b.ftd`. This can happen when a we sync with remote, say if a was +local and b was remote version of same file. This can happen when the author +performs `fpm mark-up-to-date` action if a is tracking b in an ad-hoc fashion, +or if a is a translation of b. This can also happen when a merge happens from +CR, if a was CR version of b, and was being merged into b. + +`version` represents the version of `b.ftd` , and `self-version` +represents the version of `a.ftd`. + +-- ft.h1: Client State + +On the client the state is largely two: workspace content and the unsynced +tracking information. + +-- ft.code: +lang: ftd + +\-- record client-state: +workspace-entry list workspace: +file-tracking list unsynced-tracks: + +\-- record workspace-entry: +caption filename: +optional boolean deleted: +integer version: + + +-- ft.markdown: + +Client state is stored in `.client-state` folder. Workspace information is +stored in `.client-state/workspace.ftd` and unsynced changes are stored in +`.client-state/unsynced.tracks.ftd`. + +-- ft.h1: Operations + +Given current server and client state, a bunch of operations are there, which +either print some output, or update either the client or server state. + +-- ft.h2: `fpm clone` + +This command starts with remote state and no client state, and fetches client +state and lays things out. + +It also stores the server state in `.server-state` folder. This folder contains +`history.ftd`, `tracks.ftd` and `index.1.ftd` etc. + +-- ft.h2: `fpm start-tracking a.ftd b.ftd` + + + +-- ft.h2: `fpm status` + +This shows the changes on local. + +-- ft.h2: `fpm sync` + +On client, some files are updated, if there is a mergable change on server, +along with their entries in workspace file. + +On every sync we also copy and update the `server-state` to `.server-state` +folder. -- ft.h1: `.fpm/workspace.ftd` -After fpm clone, the .fpm/workspace.ftd created. This contains entry for all the files in the file system. -It is hashmap of filename to version. +After fpm clone, the .fpm/workspace.ftd created. This contains entry for all the +files in the file system. It is hashmap of filename to version. The only time the workspace is updated is when we do a sync or resolve a conflict. @@ -34,11 +182,6 @@ timestamp: --- ft.h1: Global sync behaviour - -1. history.ftd - - From 456a9e387551d837563efd65a3ca5d6368e3d163 Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Mon, 11 Jul 2022 11:36:36 +0530 Subject: [PATCH 3/9] Some changes in how-sync-work --- how-sync-works.ftd | 72 ++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 31 deletions(-) diff --git a/how-sync-works.ftd b/how-sync-works.ftd index 89a4986f3..c8a97f953 100644 --- a/how-sync-works.ftd +++ b/how-sync-works.ftd @@ -192,20 +192,21 @@ Before sync: Create file in file system After sync: -> .history/index.v1.ftd -> Create entry in .latest.ftd -> Create entry .history/history.ftd +- .history/index.v1.ftd +- Create entry in .latest.ftd +- Create entry .history/history.ftd -```ftd +-- ft.code: +lang: ftd --- fpm.history: index.ftd +\-- fpm.history: index.ftd ---- fpm.status: Added +\--- fpm.status: Added timestamp: v1 -``` +-- ft.markdown: -> Create entry in `.fpm/workspace.ftd` +- Create entry in `.fpm/workspace.ftd` -- ft.h1: `new-file-added-after-already-added` @@ -215,16 +216,19 @@ Before sync: Create file in file system After sync: -> Return conflict (index.v1.ftd: content and version) -> Create entry in .fpm/workspace.ftd +- Return conflict (index.v1.ftd: content and version) +- Create entry in .fpm/workspace.ftd -```ftd --- fpm.workspace: index.ftd +-- ft.code: +lang: ftd + +\-- fpm.workspace: index.ftd status: conflicted version: v1 -``` -> Don't modify index.ftd in fs +-- ft.markdown: + +- Don't modify index.ftd in fs @@ -235,18 +239,20 @@ Before sync: Delete file in file system After sync: -> Remove entry from .history/.latest.ftd -> Create entry in .history/history.ftd +- Remove entry from .history/.latest.ftd +- Create entry in .history/history.ftd -```ftd +-- ft.code: +lang: ftd --- fpm.history: index.ftd +\-- fpm.history: index.ftd ---- fpm.status: Deleted +\--- fpm.status: Deleted timestamp: v1 -``` -> Remove entry from .fpm/workspace.ftd +-- ft.markdown: + +- Remove entry from .fpm/workspace.ftd @@ -258,14 +264,18 @@ Before sync: Delete the file After sync: -> Return conflict (index.v1.ftd: content and version) -> Create entry in .fpm/workspace.ftd +- Return conflict (index.v1.ftd: content and version) +- Create entry in .fpm/workspace.ftd -```ftd --- fpm.workspace: index.ftd +-- ft.code: +lang: ftd + +\-- fpm.workspace: index.ftd version: v2 -``` -> It won't restore the file in fs + +-- ft.markdown: + +- It won't restore the file in fs @@ -282,7 +292,7 @@ After sync: Nothing -> Remove entry from `.fpm/workspace.ftd` +- Remove entry from `.fpm/workspace.ftd` @@ -295,9 +305,9 @@ modify the file After sync: -> Create new snapshot .history/index.v2.ftd -> Create entry in .latest.ftd -> Create entry in .history/history.ftd +- Create new snapshot .history/index.v2.ftd +- Create entry in .latest.ftd +- Create entry in .history/history.ftd From bb73401724613eb9f1e666566e5a62116a3a3175 Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Fri, 15 Jul 2022 02:36:02 +0530 Subject: [PATCH 4/9] wip: draft cr --- draft-cr.ftd | 288 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 288 insertions(+) create mode 100644 draft-cr.ftd diff --git a/draft-cr.ftd b/draft-cr.ftd new file mode 100644 index 000000000..016c914af --- /dev/null +++ b/draft-cr.ftd @@ -0,0 +1,288 @@ +-- ft.page: CR + + +We have two states + +-- ft.h1: Server state + +The server state is stored inside the `.server-state` folder. It comprises all +files' history and tracking information. + +-- ft.code: +lang: ftd + +\-- record server-state: +file-history list history: +file-tracking list tracks: + +-- ft.markdown: + +History-related information is stored in the `.server-state/history.ftd` and +track-related information is stored in the `server-state/tracks.ftd` + + + + + +-- ft.h2: History + +File history captures every modification to a file. It also included information +about changes that happened due to merges, if an action happened to the “main branch” +via merge from a CR, we store the CR where the change came from in the `src-cr` field. + + +-- ft.code: +lang: ftd + +\-- record file-history: +caption filename: +file-edit list history: + +\-- or-type file-edit: +\--- added: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- updated: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- deleted: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + + +-- ft.markdown: + +Or we can have file-edit as a record + + +-- ft.code: +lang: ftd + +\-- record file-edit: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: +string operation: + +-- ft.markdown: + +Actual content is stored in the file system based on +`.server-state/..`. + +Also, the version number starts with 1 for every file, and is incremented by +one for every change in that file. Delete also increments the version number. + + + +-- ft.h2: Tracks + +A file can track one or more files. Any file that is tracking another file has +an entry in `server-state.track`, represented by file-tracking. + + +-- ft.code: +lang: ftd + +\-- record file-tracking: +caption filename: +tracking-info list tracks: + + +-- ft.markdown: + +Every file that is being tracked is recorded using `tracking-info`. + +-- ft.code: +lang: ftd + +\-- record tracking-info: +caption filename: +integer version: +option integer self-version: + + +-- ft.code: In `fpm.ftd` +lang: ftd + +\-- file-tracking list tracker: + +-- ft.markdown: + +Say `a.ftd` is tracking `b.ftd`. We have some operations that mark `a.ftd` up +to date with `b.ftd`. This can happen when we sync with remote, say if `a` was +local and `b` was a remote version of the same file. This can happen when the +author performs fpm mark-up-to-date action if `a` is tracking `b` in an ad-hoc +fashion, or if `a` is a translation of `b`. This can also happen when a merge +happens from CR, if `a` was the CR version of `b`, and was being merged into `b`. + +`version` represents the version of `b.ftd` , and `self-version` represents the +version of `a.ftd`. + + + +-- ft.h1: Client state + + +On the client, the state is largely two: workspace content and the unsynced +tracking information. + + +-- ft.code: +lang: ftd + +\-- record client-state: +workspace-entry list workspace: +file-tracking list unsynced-tracks: + +\-- record workspace-entry: +caption filename: +optional boolean deleted: +integer version: + +-- ft.markdown: + +Client state is stored in `.client-state` folder. Workspace information is +stored in `.client-state/workspace.ftd` and unsynced changes are stored in +`.client-state/unsynced.tracks.ftd`. + + +-- ft.h1: Operations + +Given the current server and client state, a bunch of operations are there, +which either print some output or update either the client or server state. + + +-- ft.h2: `fpm clone` + +This command starts with a remote state and no client state, and fetches the +client state and lays things out. + +It also stores the server state in `.server-state` folder. This folder contains +`history.ftd`, `tracks.ftd` and `index.1.ftd` etc. + + +-- ft.h2: `fpm start-tracking a.ftd --target b.ftd` + +With this command, `a.ftd` starts tracking `b.ftd`. In case of server or any new +package, this creates entry in `.server-state/tracks.ftd`. In case of client, +this commands creates the entry in `.client-state/tracks.ftd` + +In case, if the files (`a.ftd` and `b.ftd`) are not synced i.e. latest snapshot is +not created or is not in sync with the workspace, this command does nothing, +instead, it throws a warning asking to sync first. Let's say the `a.ftd` version is +1 and the `b.ftd` version is 2 then the entry looks something like this. + + + +-- ft.code: +lang: ftd + +\-- fpm.tracker : a.ftd + +\-- tracks: b.ftd +version: 2 +self-version: 1 + + + +-- ft.h2: `fpm status` + +This shows the changes in local. Sync-related and track-related information + + + +-- ft.h2: `fpm sync` + +On the client, some files are updated, if there is a mergeable change on the server, +along with their entries in the workspace file. + +On every sync, we also copy and update the server-state to the `.server-state` folder. + + +-- ft.h2: `fpm create-cr --title --description ` + +This command creates a new CR. In case, if the package is not a client (not a clone +package) then this does nothing, rather gives a warning that this command is not +supported. It achieves this in the following steps: + +- It calls the server/remote post API `create-cr/` with title and description, +both are optional fields. +- The server acquires the lock and reads the file `.server-state/cr`. This file contains +the current CR number. It then increments the CR number by one and then releases the lock. +- The server then creates a file `-//about.ftd`. This file stores the title and +description of the CR. In case the title is not specified, then the CR number becomes the title + +-- ft.code: +lang: ftd + +\-- record cr-about-data: +caption title: +optional body description: + +-- ft.markdown: + +- Afterwards, the server returns the cr-number. (Probably also the file +`-//about.ftd` content. Not sure though). +- On the client side, the file `-//about.ftd` is created with +data similar to the server. + + +-- ft.h2: `fpm cr ` + +The operations available are `add` and `delete` + +-- ft.h3: `add` + +This operation adds a new file or existing file (with some modification) in CR +(let’s say the CR number is 1). + +In case of adding a new file, let's say `foo.ftd`, just create a new file in a +current file system, do some modification if needed and then run this command +`fpm cr 1 add foo.ftd`. + +Similarly, in the case of adding an existing file with some modification in CR, +just do the modification and run the command. + +On running this command, it first checks if the file is different from the latest +version in the history or if it’s a new file altogether, else this fails with +a warning that no changes detected. It then clones the file from the current file +system to the `-//`. + +(Should it also creates the entry in `.client-state/-//workspace.ftd` file??) + + +-- ft.h3: `delete` + +This operation deletes the existing file in the CR. It first checks if the file exists +in the latest history else this fails with the warning that the file doesn’t exist. +It creates an entry in the `.client-state/-//workspace.ftd`. + +-- ft.code: +lang: ftd + +\-- workspace-entry list entry: + +\-- entry: +deleted: true +version: 1 + + + + + + + + From 73457271d9a4dcb26d89b8ef66c72d499f3113de Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Fri, 15 Jul 2022 17:30:22 +0530 Subject: [PATCH 5/9] doc: cr, track, sync, and some commands --- FPM.ftd | 3 + cli/add.ftd | 19 +++++ cli/edit.ftd | 17 ++++ cli/rm.ftd | 16 ++++ dev/cr.ftd | 166 ++++++++++++++++++++++++++++++++++++++ dev/sync.ftd | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++ dev/track.ftd | 54 +++++++++++++ draft-cr.ftd | 74 ++++++++++------- 8 files changed, 540 insertions(+), 28 deletions(-) create mode 100644 cli/add.ftd create mode 100644 cli/edit.ftd create mode 100644 cli/rm.ftd create mode 100644 dev/cr.ftd create mode 100644 dev/sync.ftd create mode 100644 dev/track.ftd diff --git a/FPM.ftd b/FPM.ftd index 151c1e7f6..d8ca62593 100644 --- a/FPM.ftd +++ b/FPM.ftd @@ -55,6 +55,9 @@ implements: fifthtry.github.io/theme - `fpm diff`: /diff/ - `fpm update`: /update/ - `fpm serve`: /serve/ + - `fpm add`: /cli/add/ + - `fpm rm`: /cli/rm/ + - `fpm edit`: /cli/edit/ diff --git a/cli/add.ftd b/cli/add.ftd new file mode 100644 index 000000000..b253aec76 --- /dev/null +++ b/cli/add.ftd @@ -0,0 +1,19 @@ +-- ft.page: `fpm add ` + + +This would create entry in workspace.ftd and only the files added would be +consider for sync. + +-- ft.h1: `fpm add --cr ` + +This command will create a file in cr `-//` + +-- ft.h1: Error handling + +Adding a file that exists in latest history would suggest to use `fpm edit` +instead. + + + + + diff --git a/cli/edit.ftd b/cli/edit.ftd new file mode 100644 index 000000000..c5028b409 --- /dev/null +++ b/cli/edit.ftd @@ -0,0 +1,17 @@ +-- ft.page: `fpm edit --cr ` + +This command works with cr number only, else it fails +This would create entry in workspace.ftd and only the files added would be +consider for sync. + +This command will clone the file in cr directory `-/` and also +start tracking with the latest version. + + +-- ft.h1: Error handling + +If the file doesn't exist in the latest history then throw error + + + + diff --git a/cli/rm.ftd b/cli/rm.ftd new file mode 100644 index 000000000..cdfe24f59 --- /dev/null +++ b/cli/rm.ftd @@ -0,0 +1,16 @@ +-- ft.page: `fpm rm ` + + +This would create entry in workspace.ftd and this file is consider for sync + +-- ft.h1: `fpm rm --cr ` + +This would add an entry in `-//-/delete.ftd` + +-- ft.h1: Error Handling + +If the file doesn't exist in the latest history then throw error + + + + diff --git a/dev/cr.ftd b/dev/cr.ftd new file mode 100644 index 000000000..c30f3282a --- /dev/null +++ b/dev/cr.ftd @@ -0,0 +1,166 @@ +-- ft.page: How CR Works + +-- ft.h1: Reserved CR IDs + +Each clone has a unique id, stored in `.client-state/clone-id.ftd`. The id is +created by the server. + +On clone we create `.client-state/reserved-crs.ftd`, and store the 5 CR numbers +we get from server. + +Server will have `.server-state/reserved-crs.ftd` which will mention the CRs +handed out to each "clone". + +-- ft.code: `.server-state/reserved-crs.ftd` +lang: ftd + +\-- client: +reserved-cr: 1 +reserved-cr: 2 +reserved-cr: 3 +reserved-cr: 4 +reserved-cr: 5 + +\-- client: +reserved-cr: 6 +reserved-cr: 7 +reserved-cr: 8 +reserved-cr: 9 +reserved-cr: 10 + +-- ft.markdown: + +On every sync, we replenish our reserved CR pool. + +-- ft.h1: Create A New CR + +`fpm create-cr` can be used to create a CR. It will invoke `$EDITOR` with: + +-- ft.code: +lang: ftd + +\-- cr.title: +id: 1 + + + +-- ft.markdown: + +We find the smallest CR number from reserved CR, and create +`/-//-/about.ftd`. + +On next `fpm sync`, the `about.ftd` file would be synced and everyone will have +access to it. Server will also replenish the reserved cr pool as we have +consumed one CR number from there. + +-- ft.h1: Add a new file in CR + +`fpm cr --add ` + +To add a new file, create the file `/-//`, and let user edit +the file. + +On sync the file would be synced. From sync's perspective this is just a normal +file. Before sync `workspace.ftd` will not have a version number for this file, +and post sync it will. + +-- ft.h1: Edit an existing file in CR + +`fpm cr --add ` + +If we want to edit `foo.ftd` in CR 1, we have to create `-/1/foo.ftd` and make +it [start tracking](/dev/track/) `foo.ftd`. + +On the sync the two files will be synced with server. One every update server +will update the `self-version` in the `.track/-/1/foo.ftd`. + +-- ft.h1: Editing a file both on client and server + +Say you started editing the file, but someone else also started editing the +file at the same time and their sync landed first. If we handle this as regular +conflict (both sides added a file) we ask end users to do a lot. We can make +users life better by implementing a special case for this scenario, and use the +smaller version number from both tracking files, and do a three way sync. If the +sync succeeds there is no conflict. + +-- ft.h1: Delete a file in CR + +`fpm rm --cr ` + +We will have a file called `/-/1/-/deleted.ftd` which stores all the files and +their versions that were deleted by CR number 1. + +-- ft.code: +lang: ftd + +\-- fpm.deleted: file/name.ftd +version: v1 + +-- ft.markdown: + +This file gets synced with server as normal file. + +-- ft.h1: Detect conflict in CR + +We can detect if a file `/-/1/foo.ftd` in CR 1 is conflicted with corresponding +`foo.ftd` in main by looking at `.track/-/1/foo.ftd.track` to get the version +number, and finding the content of that version of `foo.ftd` from +`.server-state/foo..ftd`, and using it as "base", and content of +`foo.ftd` as "theirs" and content of `/-/1/foo.ftd` as ours and do three way +merge. + +If `/-/1/foo.ftd` does not exist, we check for `/-/1/-/deleted.ftd`, and see if +the version for `foo.ftd` in that file is different than the latest version of +`foo.ftd`. + +-- ft.h1: Merge changes in main to CR + +`fpm merge --src=main --dest=1 foo.ftd` # --src=main is default, if no +file/folder name, merge all + + +Merge only happens on server for now + +We perform the three way merge as described in previous section. We use content +of `foo.ftd` as "theirs" and content of `/-/1/foo.ftd` as "ours". If there is +no conflict we update `/-/1/foo.ftd` with the merged content, and update the +tracking information corresponding to `/-/1/foo.ftd` against `foo.ftd` (store +the version of `foo.ftd` we just merged in `.track/-/1/foo.ftd`. + +-- ft.h1: Merge CR changes to main + + +`fpm merge --src=1 --dest=main foo.ftd` + +Merge only happens on server for now + +We perform the three way merge as described in previous section. We use content +of `foo.ftd` as "ours" and content of `/-/1/foo.ftd` as "theirs". If there is +no conflict we update `foo.ftd` with the merged content, delete `/-/1/foo.ftd`, +and `.track/-/1/foo.ftd` and increment the version number of all three files in +`history.ftd`. + +-- ft.h1: Resolve conflict in CR when merging from main Or from CR into main + +In both these scenarios, we will create a temporary file with conflict markers +as described in previous two steps and launch `$EDITOR`, and once user has +resolved conflict we will do what is described in previous two steps. + +-- ft.h1: Close a CR + +To close a CR we update the `about.ftd` and set `status: closed`. + +`fpm close-cr ` + +This sets the `close` flag in `-//-/about.ftd`. + + +-- ft.h1: Update CR About + + +-- ft.h1: `fpm status --cr=1` +-- ft.h1: `fpm status` (sync status) +-- ft.h1: `fpm status --versions` +-- ft.h1: `fpm status --translation` + + diff --git a/dev/sync.ftd b/dev/sync.ftd new file mode 100644 index 000000000..ecd413819 --- /dev/null +++ b/dev/sync.ftd @@ -0,0 +1,219 @@ +-- ft.page: How Sync Works + +We have two states, "server state" and "client state". + +-- ft.h1: Server state + +The server state is stored inside the `.server-state` folder. It comprises all +files' history. + +-- ft.code: `.server-state/history.ftd` +lang: ftd + +\-- record server-state: +file-history list history: + + + + +-- ft.h2: History + +File history captures every modification to a file. It also included information +about changes that happened due to merges, if an action happened to the "main +branch" via merge from a CR, we store the CR where the change came from in the +`src-cr` field. + + +-- ft.code: +lang: ftd + +\-- record file-history: +caption filename: +file-edit list history: + +\-- or-type file-edit: +\--- added: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- updated: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + +\--- deleted: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: + + +-- ft.markdown: + +Or we can have file-edit as a record + + +-- ft.code: +lang: ftd + +\-- record file-edit: +caption or body message: +integer timestamp: +integer version: +string author: +option integer src-cr: +string operation: + +-- ft.markdown: + +Actual content is stored in the file system based on +`.server-state/..`. + +Also, the version number starts with 1 for every file, and is incremented by +one for every change in that file. Delete also increments the version number. + + + + + +-- ft.h1: Client state + +On the client, the state is largely two: workspace content and the unsynced +tracking information. + + +-- ft.code: +lang: ftd + +\-- record client-state: +workspace-entry list workspace: + +\-- record workspace-entry: +caption filename: +optional boolean deleted: +integer version: + +-- ft.markdown: + +Client state is stored in `.client-state` folder. Workspace information is +stored in `.client-state/workspace.ftd`. + + +-- ft.h1: Sync Scenarios + +Given the current server and client state, a bunch of operations are there, +which either print some output or update either the client or server state. + +-- ft.h2: A file was updated locally, no changes on remote + +We detect a file is updated locally by finding the version for every file in +`workspace.ftd`, and then comparing the local file with the +`.history/..`. If the content is different file is +considered modified. + +Say file is file.ftd, and its current version on server is v1. On sync server +will get file.ftd, v1 and new content of file. Server sees their version is also +v1, so server computes v2, and stored the new content of the file. + +Server returns latest `file.v2.ftd` content, and client stores it in +`.server-state`, and updates the `.client-state/workspace.ftd` and `file.ftd` +in workspace. + +-- ft.h2: A file was updated locally, with non conflicting changes on server + +If `file.ftd` was modified by someone else in the meanwhile, and the server +state has already moved to v2, server does a three way merge between v1 version +of file.ftd as "base", and v2 as "theirs" and the current content as "ours". + +In non conflicting state the file content is simply updated. Server computes +a v3, and returns the merged content to client, client adds it to .server-state +folder, and updates the workspace.ftd and file.ftd's content in the workspace. + +-- ft.h2: A file was updated locally with conflicting changes on server + +If on server, v2 and our changes do not merge cleanly with v1 version as based, +server notifies the client that the sync for that file failed. Server sends the +v2 version of file which client stored in `.server-state` folder. `workspace.ftd` +and `file.ftd` in workspace are not modified. + +-- ft.h2: Only a file with no known conflicts are sent to server + +Before sending a file we check if the file has known conflict. To do this we +use the version in `workspace.ftd` and the content of `file.ftd` for that +version as "base", and the latest version of `file.ftd` in `.server-state` as +"theirs", and the content of `file.ftd` in the workspace as "ours" and perform +three way merge, and if the merge fails, we know there is conflict. + +-- ft.h2: Conflict Resolution + +if there is any conflict, we have a local command `fpm resolve-conflict +file.ftd`, which does the three way merge, and generates the merge with conflict +marker string, and launches `$EDITOR`. User has to manually resolve the conflict, +and when they confirm they have done it by saving it, and we do not find any +remaining conflict markers, we update the `workspace.ftd` with the latest +version number, and the content of `file.ftd` with the content in EDITOR. + +We also have `fpm resolve-conflict --pick-ours file.ftd` and `fpm resolve-conflict +--pick-theirs file.ftd` commands, these commands will not launch `$EDITOR`. + +The resolve-conflict will update the version number to latest in `workspace.ftd` +in the end. + +-- ft.h2: `fpm revert file.ftd` + +We will have this command which can be used to discard any local changes done to +a file. This command will find the latest version of `file.ftd` from the +`.server-state` and update both `.client-state/workspace.ftd` and `file.ftd` in +the workspace with the latest version. + +-- ft.h2: Both sides have added a file conflict + +In this case we have a conflict, and server will detect it, and send the new +file added on server. And `fpm sync` will show conflict message to user. + +On next `fpm sync` we will not pick `file.ftd` as our conflict detector will +now see the file has a version in `.server-state`, but in `workspace.ftd` there +is no entry for that file, so we are adding a file that already exists, and will +show conflict message. + +-- ft.h2: Deleting a file, no edits on server case + +If a file was deleted from workspace, and since it already has an entry in +`workspace.ftd` we know that file is being deleted by us, we send this +information to server on `fpm sync`, and our version number. Server will see our +version numbers match, so server will create a new version number to mark +deletion of the file, and update the `.server-state/history.ftd`. And the client +will delete the entry from `.client-state/workspace.ftd`. + +-- ft.h2: Deleted by us, edited on server + +If the file we deleted has been updated on server, server will see different +version number, and return a conflict message along with the latest content of +the file. Our conflict detector will spot that we have deleted a file +corresponding to a version number in workspace.ftd that is no longer the latest +version of that file, and treat is as conflict and not send it to server on next +fpm sync. + +In the meanwhile user will have to review the diff of changes from their version +till the latest version of that file and decide if the file should be still +deleted (using `fpm resolve-conflict file.ftd`) or the file should be reverted +(using `fpm revert-changes file.ftd`). + + +-- ft.h2: Edited by us, deleted on server + +Server will again see two different version numbers, on delete we increase the +version number, and reject the changes, as delete and edit are not cleanly +merge-abe using three way merge operation. + +Client will show the conflict message and user will have to `fpm +resolve-conflict` or `fpm revert-changes`. + + + diff --git a/dev/track.ftd b/dev/track.ftd new file mode 100644 index 000000000..7b51404e3 --- /dev/null +++ b/dev/track.ftd @@ -0,0 +1,54 @@ +-- ft.page: How track works + +Say `a.ftd` is tracking `b.ftd`. We have some operations that mark `a.ftd` up +to date with `b.ftd`. This can happen when the author performs fpm +mark-up-to-date action if `a` is tracking `b` in an ad-hoc fashion, or if `a` is +a translation of `b`. This can also happen when a merge happens from CR, if `a` +was the CR version of `b`, and was being merged into `b`. + +`version` represents the version of `b.ftd` , and `self-version` represents the +version of `a.ftd`. + +Every file that is tracking something has a corresponding file: +`.tracks/.track`, which contains a list of `tracking-info`. + +-- ft.code: +lang: ftd + +\-- record tracking-info: +caption filename: +integer version: +option integer self-version: + +-- ft.markdown: + +Sample track file: + +-- ft.code: `.tracks/a.ftd.track` +lang: ftd + +\-- import: fpm +\-- fpm.tracks: b.ftd +version: 2 +self-version: 1 + +-- ft.h1: To Start Tracking A File + +Tracking is initialised by creating a `.tracks/.track` file. + +`self-version` may be missing if the file is being added and tracked at the +same time. + +The job of `fpm sync` is to also populate the missing `self-version` in every +`.track` file on successfully syncing the corresponding file. This will be done +by server. + +If `a.ftd` wants to track `b.ftd`, `b.ftd` must be first synced with the server +else we do not have version number. + +`fpm start-tracking a.ftd --target b.ftd` + + +-- ft.h1: `fpm stop-tracking a.ftd --target b.ftd` + +-- ft.h1: `fpm mark-up-to-date a.ftd --target b.ftd` \ No newline at end of file diff --git a/draft-cr.ftd b/draft-cr.ftd index 016c914af..4f1985779 100644 --- a/draft-cr.ftd +++ b/draft-cr.ftd @@ -1,7 +1,6 @@ --- ft.page: CR +-- ft.page: How Sync And CR Works? - -We have two states +We have two states, "server state" and "client state". -- ft.h1: Server state @@ -27,8 +26,9 @@ track-related information is stored in the `server-state/tracks.ftd` -- ft.h2: History File history captures every modification to a file. It also included information -about changes that happened due to merges, if an action happened to the “main branch” -via merge from a CR, we store the CR where the change came from in the `src-cr` field. +about changes that happened due to merges, if an action happened to the "main +branch" via merge from a CR, we store the CR where the change came from in the +`src-cr` field. -- ft.code: @@ -100,6 +100,8 @@ lang: ftd caption filename: tracking-info list tracks: +TODO: split one big track file into one track file per file. filename which is +caption here, will become `.track`. -- ft.markdown: @@ -164,6 +166,18 @@ stored in `.client-state/workspace.ftd` and unsynced changes are stored in Given the current server and client state, a bunch of operations are there, which either print some output or update either the client or server state. +-- ft.h2: Sync Vs Tracking + +The state we keep are of two kinds, history, which is used for syncing. + +We should be able to do syncing with just the content of the `history.ftd` and +the `workspace.ftd` files only. + +Track files from syncing perspective are just another files in the package. Since +track files are generated by us, we may be able to do a better job of helping the +users with conflict resolutions, beyond what we do for other regular files. +Special UI, or maybe some heuristic which can decide the conflict is not a real +conflict, maybe earlier or later version wins. -- ft.h2: `fpm clone` @@ -176,14 +190,14 @@ It also stores the server state in `.server-state` folder. This folder contains -- ft.h2: `fpm start-tracking a.ftd --target b.ftd` -With this command, `a.ftd` starts tracking `b.ftd`. In case of server or any new -package, this creates entry in `.server-state/tracks.ftd`. In case of client, -this commands creates the entry in `.client-state/tracks.ftd` +With this command, `a.ftd` starts tracking `b.ftd`. If command is executed +directly on server, this creates an entry in `.server-state/tracks.ftd`. In case +of client, this commands creates the entry in `.client-state/tracks.ftd` -In case, if the files (`a.ftd` and `b.ftd`) are not synced i.e. latest snapshot is -not created or is not in sync with the workspace, this command does nothing, -instead, it throws a warning asking to sync first. Let's say the `a.ftd` version is -1 and the `b.ftd` version is 2 then the entry looks something like this. +In case, if the files (`a.ftd` and `b.ftd`) are not synced i.e. latest snapshot +is not created or is not in sync with the workspace, this command does nothing, +instead, it throws a warning asking to sync first. Let's say the `a.ftd` version +is 1 and the `b.ftd` version is 2 then the entry looks something like this. @@ -206,24 +220,27 @@ This shows the changes in local. Sync-related and track-related information -- ft.h2: `fpm sync` -On the client, some files are updated, if there is a mergeable change on the server, -along with their entries in the workspace file. +On the client, some files are updated, if there is a merge-able change on the +server, along with their entries in the workspace file. -On every sync, we also copy and update the server-state to the `.server-state` folder. +On every sync, we also copy and update the server-state to the `.server-state` +folder. -- ft.h2: `fpm create-cr --title --description ` -This command creates a new CR. In case, if the package is not a client (not a clone -package) then this does nothing, rather gives a warning that this command is not -supported. It achieves this in the following steps: +This command creates a new CR. In case, if the package is not a client (not a +clone package) then this does nothing, rather gives a warning that this command +is not supported. It achieves this in the following steps: - It calls the server/remote post API `create-cr/` with title and description, -both are optional fields. -- The server acquires the lock and reads the file `.server-state/cr`. This file contains -the current CR number. It then increments the CR number by one and then releases the lock. -- The server then creates a file `-//about.ftd`. This file stores the title and -description of the CR. In case the title is not specified, then the CR number becomes the title + both are optional fields. +- The server acquires the lock and reads the file `.server-state/cr`. This file + contains the current CR number. It then increments the CR number by one and + then releases the lock. +- The server then creates a file `-//about.ftd`. This file stores the + title and description of the CR. In case the title is not specified, then the + CR number becomes the title -- ft.code: lang: ftd @@ -256,12 +273,13 @@ current file system, do some modification if needed and then run this command Similarly, in the case of adding an existing file with some modification in CR, just do the modification and run the command. -On running this command, it first checks if the file is different from the latest -version in the history or if it’s a new file altogether, else this fails with -a warning that no changes detected. It then clones the file from the current file -system to the `-//`. +On running this command, it first checks if the file is different from the +latest version in the history or if it’s a new file altogether, else this fails +with a warning that no changes detected. It then clones the file from the +current file system to the `-//`. -(Should it also creates the entry in `.client-state/-//workspace.ftd` file??) +Q: Should it also creates the entry in `.client-state/-//workspace.ftd` +file? -- ft.h3: `delete` From ddcd9b65526b6a43b5451bd37996e296566713bf Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Thu, 21 Jul 2022 11:53:52 +0530 Subject: [PATCH 6/9] minor changes --- dev/sync.ftd | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dev/sync.ftd b/dev/sync.ftd index ecd413819..880761fa9 100644 --- a/dev/sync.ftd +++ b/dev/sync.ftd @@ -36,21 +36,21 @@ file-edit list history: caption or body message: integer timestamp: integer version: -string author: +optional string author: option integer src-cr: \--- updated: caption or body message: integer timestamp: integer version: -string author: +optional string author: option integer src-cr: \--- deleted: caption or body message: integer timestamp: integer version: -string author: +optional string author: option integer src-cr: @@ -66,7 +66,7 @@ lang: ftd caption or body message: integer timestamp: integer version: -string author: +optional string author: option integer src-cr: string operation: From 019a31b5886362fc4a06720e8ee5ed4cfc830473 Mon Sep 17 00:00:00 2001 From: Arpita-Jaiswal Date: Mon, 15 Aug 2022 20:46:28 +0530 Subject: [PATCH 7/9] Added more details in commands --- cli/add.ftd | 14 ++++++++++++-- cli/edit.ftd | 17 ++++++++++++++++- cli/rm.ftd | 32 +++++++++++++++++++++++++++++--- 3 files changed, 57 insertions(+), 6 deletions(-) diff --git a/cli/add.ftd b/cli/add.ftd index b253aec76..ef7d9fc00 100644 --- a/cli/add.ftd +++ b/cli/add.ftd @@ -4,9 +4,16 @@ This would create entry in workspace.ftd and only the files added would be consider for sync. --- ft.h1: `fpm add --cr ` +-- ft.h1: `fpm add -//` + +This command should check that a file in cr `-//` exists. + +-- ft.code: `.client-workspace/workspace.ftd` +lang: ftd + +\-- fpm.client-workspace: +cr: -This command will create a file in cr `-//` -- ft.h1: Error handling @@ -17,3 +24,6 @@ instead. + + + diff --git a/cli/edit.ftd b/cli/edit.ftd index c5028b409..e3e58f710 100644 --- a/cli/edit.ftd +++ b/cli/edit.ftd @@ -4,12 +4,27 @@ This command works with cr number only, else it fails This would create entry in workspace.ftd and only the files added would be consider for sync. -This command will clone the file in cr directory `-/` and also +This command will copy the file in cr directory `-/` and also start tracking with the latest version. +-- ft.code: `.client-workspace/workspace.ftd` +lang: ftd + +\-- fpm.client-workspace: -// +version: + + +-- ft.code: `.tracks/-//` +lang: ftd + +\-- fpm.tracks: +version: + + -- ft.h1: Error handling +If file is already in `-/` directory, throw error If the file doesn't exist in the latest history then throw error diff --git a/cli/rm.ftd b/cli/rm.ftd index cdfe24f59..9278a30d6 100644 --- a/cli/rm.ftd +++ b/cli/rm.ftd @@ -1,11 +1,37 @@ --- ft.page: `fpm rm ` +-- ft.page: `fpm rm` -This would create entry in workspace.ftd and this file is consider for sync +-- ft.h1: `fpm rm ` + +This would mark the file deleted in workspace.ftd. +This would delete the file. + +-- ft.code: `.client-workspace/workspace.ftd` +lang: ftd + +\-- fpm.client-workspace: +version: +cr: +deleted: true -- ft.h1: `fpm rm --cr ` -This would add an entry in `-//-/delete.ftd` +This would add an entry in -//-/delete.ftd + +-- ft.code: `-//-/delete.ftd` +lang: ftd + +\-- fpm.deleted: +version: + + +-- ft.code: `.client-workspace/workspace.ftd` +lang: ftd + +\-- fpm.client-workspace: -//-/delete.ftd +version: +cr: + -- ft.h1: Error Handling From c1db96fa18a4a3be94535b4c5395b8b69d1b9968 Mon Sep 17 00:00:00 2001 From: Amit Upadhyay Date: Tue, 16 Aug 2022 10:14:37 +0530 Subject: [PATCH 8/9] Update sync.ftd --- dev/sync.ftd | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/dev/sync.ftd b/dev/sync.ftd index 880761fa9..9ea40571f 100644 --- a/dev/sync.ftd +++ b/dev/sync.ftd @@ -75,8 +75,37 @@ string operation: Actual content is stored in the file system based on `.server-state/..`. -Also, the version number starts with 1 for every file, and is incremented by -one for every change in that file. Delete also increments the version number. + +-- ft.h2: Global Versioning + +For a FPM project, there is a single version number that represents current state of +the server. This number is incremented by one on every successful sync. + +-- ft.h2: Consistent Read + +Since sync operation modifies multiple files in the package, if we do not take +care we can indavertantly read inconsistent state from file system. We have to +ensure reads are always consistent, even in the middle of a large sync. + +-- ft.h3: Locked `version-number` + +First we use global read-write lock mutex, with single writer, and multiple +readers. Since fpm server is single server, in the server's memory there exists +a variable called `version-number`, which is can be read by any number of threads +but can be updated by only one thread. All fpm server's read operations take a +read lock to read the `version-number`, and sync operation takes a write lock and +updates `version-number`. + +Most important thing is that the `version-number` is only updated after successful +sync operation, at the very end, after everything else has happened. + +-- ft.h3: Read From History + +Till the `version-number` has been updated, all reads read operations do not read +from the "workspace" on the server, but from the history folder. If current version +number is `v1`, and sync operation is going on, some `v2` files may exist in history +folder. But the readers ignore any file with version greater than current value of +`version-number`. From 630f9fd2bd59ff00a975bc574a787a9944841255 Mon Sep 17 00:00:00 2001 From: Arpita Jaiswal Date: Tue, 23 Aug 2022 12:47:27 +0530 Subject: [PATCH 9/9] Update cr.ftd --- dev/cr.ftd | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/dev/cr.ftd b/dev/cr.ftd index c30f3282a..36190cbde 100644 --- a/dev/cr.ftd +++ b/dev/cr.ftd @@ -34,25 +34,47 @@ On every sync, we replenish our reserved CR pool. -- ft.h1: Create A New CR -`fpm create-cr` can be used to create a CR. It will invoke `$EDITOR` with: +`fpm create-cr ""` can be used to create a CR. + + +-- ft.markdown: + +We find the smallest CR number from reserved CR, and create +`/-/<cr-id>/-/meta.ftd`: -- ft.code: lang: ftd -\-- cr.title: <emtpy> -id: 1 +\-- import: fpm -<description> +\-- fpm.cr: <title> +status: open -- ft.markdown: -We find the smallest CR number from reserved CR, and create -`/-/<cr-id>/-/about.ftd`. - -On next `fpm sync`, the `about.ftd` file would be synced and everyone will have +On next `fpm sync`, the `meta.ftd` file would be synced and everyone will have access to it. Server will also replenish the reserved cr pool as we have consumed one CR number from there. +-- ft.h1: Edit CR Description + +The description of a CR is stored in `/-/<cr-id>/-/about.ftd`. + +`fpm cr <cr-number> edit` launched `$EDITOR` with the content of `about.ftd`. + +If the file does not exist it will be initialised with `FPM/default-cr-about.ftd` if +present, else, `THEME/default-cr-about.ftd`. + + +-- ft.h1: Edit CR Title + +`fpm cr <cr-number> edit-title` will launch `$EDITOR` with current title. + +Note: Only the first line would be used as CR title. Content before the first +new line character is first line. + + + -- ft.h1: Add a new file in CR `fpm cr <cr-number> --add <filename>`