Application breaks when run on linux and it tries to create links on NTFS-like target FS

[mucst]

I'm running the application in docker (today's latest version: 3.1.2), git revision a631449. Host Device is a Raspberry PI 2, attached an exFat filesystem external HDD.
When I try to run the application, it fails with:

docker run --rm -v /home/pi/mediacenter/gphotos/config:/root/.config/gphotos-sync -v /media/myHdd/gphotos:/storage -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync --album MY_ALBUM --start-date 2023-04-01 /storage
# Note: I tried to add --ntfs option, did not help.

04-08 20:59:15 WARNING  gphotos-sync 3.1.2 2023-04-08 20:59:15.194045
04-08 20:59:15 ERROR    Symbolic links not supported
04-08 20:59:15 ERROR    Albums are not going to be synced - requires symlinks
04-08 20:59:16 WARNING  Downloading Photos ...
04-08 20:59:16 WARNING  Downloaded 0 Items, Failed 0, Already Downloaded 2
04-08 20:59:16 WARNING  Indexing Shared (titled) Albums ...
04-08 20:59:17 WARNING  Indexed 3 Shared (titled) Albums
04-08 20:59:17 WARNING  Indexing Albums ...
04-08 20:59:18 WARNING  Indexed 2 Albums
04-08 20:59:18 WARNING  Downloading Photos ...
04-08 20:59:18 WARNING  Downloaded 0 Items, Failed 0, Already Downloaded 2
04-08 20:59:18 WARNING  Creating album folder links to media ...
04-08 20:59:18 ERROR
Process failed.
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 507, in main
    self.start(args)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 447, in start
    self.do_sync(args)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 436, in do_sync
    self.google_albums_sync.create_album_content_links()
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/GoogleAlbumsSync.py", line 329, in create_album_content_links
    os.symlink(relative_filename, link_file)
PermissionError: [Errno 1] Operation not permitted: '../../../photos/2023/04/IMG_8133.MOV' -> '/storage/albums/2023/0405 MY_ALBUM/0000_IMG_8133.MOV'
04-08 20:59:18 WARNING  Done.

Apparently it tried to create a symlink on the storage which failed. Is there a chance to avoid this error by for example omitting symlink creation? I don't need it for my usecase anyway. I however want to synchronize a specific album.

[gilesknap]

--use-hardlinks will use hardlinks instead of symlinks

I'm not convinced these will help though. If you are using Windows I say to
enable symlinks or not sync albums. So unless you can support some kind of links in your FS I don't think gphotos-sync will work for albums.

[gilesknap]

I just checked in the code and it looks like using --album MyAlbumName --skip-albums will:

• Download files filtered by the single album name
• not try to create a symlink copy for any albums

[gilesknap]

@mucst does the above work for your use case?

[mucst]

Hey @gilesknap , thanks for the reply.

I don't think hard links could ever help since windows does not have anything similar.

As a first quick solution, I "patched" the script by replacing line 436 with a pass (custom Dockerfile FROM your image). Later on however I reformatted my HDD with ext4 filesystem ( I had other issues not related to this project).
Now I tried the solution that you suggested (with non-patched Main.py). Unfortunately the --skip-albums did not do the trick because links were eventually created.
The situation in my opinion is:

• The application needs a proper, dedicated option for skipping link creation (purpose of --skip-albums is unclear to me at least).
• We also need to consider the scenario when (the host machine is linux and) the target FS is NTFS like. Looking at:

  gphotos-sync/src/gphotos_sync/GoogleAlbumsSync.py

  Line 329 in a631449

  os.symlink(relative_filename, link_file)

  I am not sure this is how one creates Windows-like shortcuts with python (this is what we want to achieve, right?). Googling up "How to create windows shortcut python", I got various other crazy solutions but none of them is as trivial an os.symlink(...). Perhaps as first quick fix, if target FS is ntfs-like then skip link creation altogether. Same if OS is windows (because on windows there is no scenario where would ever write to an ext FS that supports symlinks or hard links.

So on my part this issue is solved, however as project owner you gotta decide how you tackle the above issues, if you want to in the first place.

Final thought: with current lineup, how can this application run on Windows (if not containerized)? 🤔 Because as I said, according to google shortcuts are not created with a plain os.symlink(...) there.

PS. Because I am lazy to experiment or research properly, I asked ChatGPT:

What happens if in python I call os.symlink on a Windows machine?

Answer:
If you call os.symlink() on a Windows machine running Python, it will most likely raise a NotImplementedError exception, as symbolic links are not fully supported on Windows.

Windows does support a type of symbolic link called "junction points", but they have different semantics than Unix-style symbolic links, and they are not created using the os.symlink() method. Instead, you can create junction points using the os.system() method to run the mklink command from the Windows command prompt.

Here's an example of creating a junction point using mklink:

import os
target = "C:\\Users\\UserName\\Documents\\target_folder"
link = "C:\\Users\\UserName\\Desktop\\link_name"
os.system('mklink /J "{}" "{}"'.format(link, target))

In the example above, target is the folder you want to link to, and link is the path of the junction point you want to create. The /J flag indicates that you want to create a junction point. You'll need to replace UserName with your actual Windows user name.

[gilesknap]

Thanks for the detailed report.

The intention of skip-albums is entirely not creating symlinks. So either it's completely broken or using --album with it causes an issue.

Re symlinks on Windows. I do have windows users of this tool and it does work if you enable symlinks in the policy manager as documented here https://gilesknap.github.io/gphotos-sync/main/how-to/windows.html#symlinks. Also the CI tests in Github actions run against a WIndows VM (but I don't know for sure that it uses NTFS).

I assume therefore that os.symlink does work on modern Windows versions at least. Can't remember the journey to make this tool work on Windows but I did testing on my linux/Windows dual boot.

[mucst]

Right. I'm surprised about the ability to enable symlinks on windows, but that explains. I'm sure that the VMs run NTFS or alike filesystem because windows does not support ext fs of any kind. But ok, let's leave it at that. Thanks for the clarification.

[gilesknap]

leaving this open - TODO look at behaviour of --skip-albums with and without --album

[gilesknap]

closing as I'm tracking this in #428