Google Drive Synchronization

[rumplestilzken]

We discussed briefly the possibility of Google Drive synchronization in an issue I posted about getting novelWriter working on a Chromebook #1676

Are you still interested in that collaboration? I would very much like to implement this feature if you are on board. A set of guidelines for how you would like things to work would be a good start so i'm not going totally AWOL. But I'm willing to do the grunt work. rclone can do synchronization with Google Drive, i'm not sure if you would want it to work like it does or if you would just like it to work with the filesystem that gets created when you mount a Google Drive using rclone or similar software. Both have their pros and cons.

[vkbo]

Hi!

Depends on what it entails. I don't want to add features that interact with APIs. At least not at this point. But if there is a way to interact with the file system or mount that works with Google drive, it's maybe worth considering. Especially if it opens up a new platform.

File I/O is handled in multiple places in novelWriter, but most of it is passed through a small set of core classes, so if we're talking about adding a layer of abstraction, that should be fine. There's already some abstraction for storage, but it handles it on the file path level, not the I/O stream itself.

[vkbo]

I haven't decided how to cache open projects yet. I'm open to ideas.

My initial idea was to use the data location, which is ~/.local/share/novelwriter/, create a projects folder there, and for each project, create a folder that is the sha1 hash of the absolute storage path. This is the simplest way to associate the cache folder directly with the corresponding storage location so that we can detect if the project was not properly closed last time (in which case the sha hash folder will already exist, and we can trigger a recovery process).

[rumplestilzken]

I haven't decided how to cache open projects yet. I'm open to ideas.

My initial idea was to use the data location, which is ~/.local/share/novelwriter/, create a projects folder there, and for each project, create a folder that is the sha1 hash of the absolute storage path. This is the simplest way to associate the cache folder directly with the corresponding storage location so that we can detect if the project was not properly closed last time (in which case the sha hash folder will already exist, and we can trigger a recovery process).

By placing the cached open projects in the same folder as the projects exist, but in a hidden folder, you gain the possibility of syncing the unfinished files to the cloud as well. The caching and syncing will automatically be handled by the sync provider(rclone here). Then, when closing the application, or on auto save, we rebuild the archive, the sync provider sees that the file has changed, moves it to its own cache location, syncs, and deletes the leftovers. Then we remove the hidden application cached project as long as we know nothing else has been modified since last archive and it all just kinda fits together. We can leverage the syncing remote caching system to make sure our local application project cache is always synced to the cloud for maximum recovery potential as well, until archiving and then archive and clean up.

I can work on a simple archiving process that can be swapped out to follow this methodology and give you something to stand on and you can do what you'd like with it if that interests you. Just build a small framework that would handle the actual archiving on save and autosave in a way that will allow you to expand or change how you see fit, after i have reviewed the code you mentioned earlier.

[rumplestilzken]

In addition to the loading of the new archive if it presents itself. I don't think putting something small together would be too much of an effort and i generally aim for maintenance and reusability/configurability so you wouldn't have to worry about not being able to plop your own solution in when the time comes.

[vkbo]

Extracting the archive next to itself is also an idea I've considered. I may end up doing that in the end, for the sake of transparency. The principle of novelWriter is to ensure that text is saved frequently and safely. I wouldn't necessarily have the folder hidden even. Just make sure the name makes it clear it is temporary. If not for this, I could easily just cache everything in memory since it isn't much data even for a large project.

[vkbo]

I may start working on this solution soon actually. I'm just taking a break from developing new features right now as I've spent a lot of time on the last two releases, and I want to do other things for a bit.

[rumplestilzken]

It just occurred to me that if you package it all up, you then have to keep track of revision progress as well.

I think if you want to implement the archived project, you should automatically have it backed up into a reposity. Git, svn, whatever. I think making the application archive every x amount of minutes, including the back end local repository would be ideal. And it presents another opportunity to read through your writing and quickly compare revisions.

This appoach stops zip archives from being loaded into the SCM which is going to bloat.

Also having access to a graphical interface for the projects history using SCM would help a lot.

Tldr; SCM integration before zip container work. Then revisions. Then special support for each popular SCM that would need to be supprt. The containerize the given project.