Helper class - possible to add directory GET

[stephenmcconnachie]

No description provided.

[stephenmcconnachie]

It would be really useful if a 'GET whole directory' helper could be developed, for cases where a bucket has a sub directory structure, and the user needs to GET all files from a directory, including subfolders and their files.

[berkeleysquare]

It would be really useful if a 'GET whole directory' helper could be developed, for cases where a bucket has a sub directory structure, and the user needs to GET all files from a directory, including subfolders and their files.

Stephen -- I might have misunderstood tyour request, but have you seen the Prefix attribute? If you get_bucket with the prefix "my/folder/" you would get a listing of all the files "under" that folder. It's just a string match, but it does what I think you're looking for.

ds3_python3_sdk/samples/getBucket.py

Line 22 in 623f299

prefix = None

[stephenmcconnachie]

Thanks John!

I was thinking of a GET all objects rather than get list of objects, so equivalent to this getting_all_objects_in_bucket.py - https://github.com/SpectraLogic/ds3_python3_sdk/blob/623f2997121d850a8b9b989fcbc8ecae45d9a2b6/samples/getting_all_objects_in_bucket.py - but with ability to specify the folder only. So imagine the prefix function from getBucket.py, but to get the objects themselves not just a list of them.

The CLI has a method to do this - get_bulk - https://developer.spectralogic.com/java-command-line-interface-cli-reference-and-examples/ - but it would be great to have it in the Python sdk.

[stephenmcconnachie]

This is an example of how to achieve this in the CLI (with an output to txt file for a log) - it works perfectly there, but would be great to have similar in the Python...

/mnt/path/to/CLI/bin/ds3_java_cli -e 111.11.11.11 -a key -k key -c get_bulk -b backup_bucket -p "Hard Drive 12/folder_7" -d /mnt/path/to/stephen_bp_cli_test --http --verbose >> /mnt/path/to/stephen_bp_cli_test/folder_7.txt

[gtrossell]

@stephenmcconnachie we are looking into this and working out where in our current development cycles we can slot this feature in.

[stephenmcconnachie]

Thanks very much @gtrossell

[berkeleysquare]

@stephenmcconnachie -- just merged a PR to do this. The get_all_files_in_bucket now takes a prefix string.

[gtrossell]

@stephenmcconnachie - we have added in a fix to master, if you want to pull it down

[stephenmcconnachie]

That's fantastic, thank you!