XBVR Database Backup/Restore

[toshski]

Backup/Restore
When I first started using XBVR about a month ago, there were a few things for backup and restore that I wanted to do. e.g.

• Having gotten a full dump of VRHush, I wanted to create a backup so I wouldn’t lose it, other sites were scrapping fine, so I wasn’t so worried about those
• To backup up data not from the scrapping, e.g. Favorites, Star Ratings and most importantly Cue points. I love Cue points but to set them up would take some effort and didn’t want to lose those
• Easily set up test environments with data
• After doing a couple of pull requests, I also wanted to be able to copy test data between MySql and SQLite

A standard database which probably is all a lot of people may not, doesn’t provide a finer level of control that I needed.
The existing Import/Export looked promising, stating “If you already have scraped scene data, you can export it below”. But on investigating that’s not really right, it doesn’t look at the data you have scrapped, it runs a new scrap and exports that data.
I have developed a Backup/Restore model on the existing Import/Export but sourcing data from data in the database.
It is fairly flexible with options to Backup/Restore

• Only New Records or to New and Overwrite Existing data
• Storage Paths
• Saved Searches
• Scene Data with separate options for
  • Scene master data
  • Scene/File links (to save file links made manually)
  • Cue points
  • Watch History
  • Edits (actions table)
• Choose All Sites or only Enabled Sites for the scene related data options

The process is based on the structs used for the database rather than the ScrapedScene which enables data that wasn't scraped, e.g. Favorite Flag, Star Ratings, manually added tags etc., and also non-scene data i.e. Saved Search definitions and Storage Paths
However, I have been a bit hesitant to submit a PR, for a few reason

• Users may be confused with the difference between the existing Export/Import and the Backup/Restore
• The content bundles created by the existing Export/Import and the Backup/Restore are not compatible (more confusion)
  -I was brand new to XBVR, GO, Vue, pretty much anything, so am I doing things the right way
• Making everything optional also caused the changes to grow a bit big.
• I needed to add a new Github package json-iterator/go to specify new tags to control the json, the existing json tags on the structures were not always what I needed for the json file.

I have found it particularly useful for managing data for testing, but with recent Discord conversations about getting VRHush data made me think maybe I should do a PR after all.

Do people think this would be particularly worthwhile, if so I’ll look at putting forward a PR

[theRealKLH]

this could potentially replace the current import/export functionality.. As you (and I over on Discord) have mentioned, current export actually performs a scrape of website, it's not an export of your local library. Looks like you've done a lot of work. Want to show it off to the world? Submit that PR my friend!

[toshski]

Will sort out a PR.
FYI: just added backwards compatibility to the old/existing bundle Json format.
So now you can

• scrape and export data (current scraper based functionality)
• backup database with extended data choices in new format 2
• backup database data using old format 1, i.e. just scene data excluding things like Favorites Flag, Star Ratings
• restore data in either format (drops the "Import scene data" from the screen as the new restore handles both)

I was mindful, of dropping the existing function. Having both is a bit confusing, but removing it breaks compatibility as you couldn't load a bundle in the current format with the new restore. I know some others have built their own imports on the existing bundle format.

With the compatibility change I just done, you can now export from the database using the old format. That would still be useful for sharing scene data with someone without setting their favorites and ratings values to what was in your system. Useful for those "my scrappers broken, can you send me your data" questions, but that probably never happens (tongue in cheek).

[a18project]

This is very nice to have. Looking forward to it.