generated from noaa-nwfsc/NWFSC-Template
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
5cd2676
commit d53a09f
Showing
1 changed file
with
26 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
## Copying data from Google Drive to Google Cloud Storage | ||
|
||
*This guide was adapted from [Eric Anderson’s chapter](https://eriqande.github.io/eca-bioinf-handbook/working-on-remote-servers.html)* | ||
Install Rclone locally. | ||
`mamba install rclone -y ` | ||
Or download from [here](https://rclone.org/downloads/) | ||
Create a new config | ||
`rclone config` | ||
You need to create a config for both | ||
‘Drive’: https://rclone.org/drive/ | ||
Google cloud storage : https://rclone.org/googlecloudstorage/ | ||
Make sure you authenticate from your browser. | ||
|
||
Optional, [Encrypt your config](https://eriqande.github.io/eca-bioinf-handbook/working-on-remote-servers.html#encrypting-your-config-file) | ||
You now have a config file at `~/.config/rclone/rclone.conf`. | ||
You can open and edit this file, or copy it to a remote server as desired. | ||
To speed up the process, create a section of your config for each root folder you want to copy using the parameter `root_folder_id=’ and set it to the folder ID of your drive folder. This is the string at the end of the folder URL : https://drive.google.com/drive/u/2/folders/<folder id here> | ||
|
||
You can then copy one config section and change the name and folder ID. | ||
Update your .bashrc (then run ‘source .bashrc’). | ||
Now test your installation | ||
rclone lsd gsc: | ||
rclone copy verily-rockfish: gcs:BUCKETNAME/verily-rockfish –dry-run | ||
Optional: Copy your config to remote location and then copy data | ||
`rclone copy verily-rockfish: gcs:BUCKETNAME/verily-rockfish` | ||
|