Enhancement: thumbnails support

[stokito]

I have a router with connected SSD. And there installed lighttpd + webdav. Now I wish to use the webdav share instead of Google Photos for my family. I've put the webdav.js there and now my wife can see list of photos just from her iPhone. But ideally she also needs to see small miniatures i.e. thumbnails.
There is a spec for "external thumbnails" where there are stored into a hidden folder .sh_thumbnails. You may generate it with the script https://github.com/stokito/genthumbs/tree/posix
It's not widely supported yet because of chicked-egg problem but I believe that the webdav-js can do this easily.
If the .sh_thumbnails folder exists then the webdav-js may try to load a thumbnail from it,
Will you accept a PR with the feature?
I may not have enough time so if you are interested in the feature you may mark it as "help needed" and maybe someone else will be interested too,

[dom111]

This sounds like a possibility for a feature. I might consider looking at a plugin mechanism and taking this on. Thanks for taking an interest!

As an aside, I've just started work on a lighttpd example too: https://github.com/dom111/webdav-js/tree/feature/lighttpd-example

[stokito]

I already see that next plugin will be a media player :)
Recently I figured out that basically I don't want to just have a browserable view of WebDAV share but to have a full JS file manager. I had such with NextCloud but I did't liked it.

I searched for any existing file managers and looks like there is a plenty of them https://dev.to/plazarev/10-javascript-file-manager-libraries-to-consider-1nd5
So eventually the webdav-js, if continue to grow ,will end up to become a full File Manager from a browser. And it will just use the WebDAV as a client-to-server protocol.

And now I think that maybe it should be done in another way: try to add the WebDAV protocol support to existing file managers.

For example here is a Backend PHP script for mootools-filemanager: https://github.com/cpojer/mootools-filemanager/tree/master/Assets/Connector

How do you think, will it be better to go from the other side?
Maybe the WebDAV protocol interaction may be extracted as js a library and added to an existing file managers.
I guess we may find existing JS wrappers over WebDAV on NPM .

Still there is a question how it will be comfortable or not to use the WebDAV because of it's limitations. For example Delete file or Move to Trash? What about folder sorting? And many other questions.

The NextCloud for example uses API which is a WebDAV based but with many their own extensions.

[gstrauss]

I happened across this interesting project. :) I am the lighttpd developer who wrote the current version of lighttpd mod_webdav. If you have suggestions for lighttpd mod_webdav (and those suggestions conform to the WebDAV spec), please ping me.

(In the past, I have tried to get NextCloud and OwnCloud to add X-Sendfile support, but they both balked. lighttpd mod_webdav can be much faster due to not having the PHP overhead of NextCloud or OwnCloud.)

[stokito]

Hi @gstrauss thank you for lighttpd. Speaking about the thumbnails generation discussed here it would be great to have hooks on file upload. Maybe inotify can be used for this but I don't khow how to deal with it and not sure if it may work on OpenWrt.
Could you please tell your opinion how to better implement the generation of thumbnails?
From the lighttpd plugin api it looks like it's possibe to make a hook. Maybe it will be better to to make a plugin that calls shell scripts from some folder.
Thank you

[gstrauss]

Yes, inotify is an option in the application. Yes, inotify works on local filesystems under OpenWRT.

My first impression to your comment is that adding shell-outs to a high performance server (lighttpd) is not the best idea. Instead, you can implement your own "triggers" using lighttpd mod_magnet and a few lines of custom lua. If the lua script detects an image upload -- e.g. when lighttpd is sending a 201 Created or 204 No Content response to a PUT request which successfully uploaded an image -- then the lua could send a request to a local daemon on the system to do additional processing (or the lua could fork) for thumbnail generation.

[stokito]

Thank you. Looks like I have to create a hook in magnet.attract-response-start-to but from docs the hook blocks a request.
The lua script is really hard to write and debug. In fact the lua script must call a shell script that will check if thumbnail exists and call graphicsmagick or imagemagick to generate a thumbnail.
Even if this is possible for me this seems like not a good solution.
So I'll try to use inotify.
Anyway thank you for your tip, maybe I'll need to use the magnet for something else

[gstrauss]

I think that you might be trying to do too much inside lighttpd.

From the lighttpd plugin api it looks like it's possibe to make a hook. Maybe it will be better to to make a plugin that calls shell scripts from some folder.

I responded that you can implement the "hook" using lighttpd mod_magnet. I did not mean to suggest that you should implement EVERYTHING in the lua script. Here is (untested) lua code to sketch that out, which should be workable in lighttpd 1.4.60 or later. This is much more customizable to your application, works in lighttpd today, and is faster than executing a shell script (and for which I would have to write a new generic hook inside lighttpd mod_webdav).
lighttpd.conf: magnet.attract-response-start-to = "/path/to/thumb-trigger.lua"

-- thumb-trigger.lua for lighttpd mod_magnet

local r = lighty.r
local req_attr = r.req_attr

-- check for successful PUT request with HTTP status 201 or 204
local http_status = req_attr["response.http-status"] -- lighttpd 1.4.60 - 1.4.65 (to be removed in 1.4.66)
--local http_status = r.req_item.http_status    -- lighttpd 1.4.65 and later (lighttpd 1.4.65 is next release)
if (not http_status == 201 and not http_status == 204) then return 0 end
if (not req_attr["request.method"] == "PUT") then return 0 end

-- get file path components
local path = req_attr["physical.path"]
local dir, file, ext = string.match(path, "^(.*/)([^/]+)([.][%a%d]+)$")
if (not ext) then return 0 end

-- TODO:
-- check for .jpg (obviously, this should be expanded to all valid extensions)
if (not ext == ".jpg") then return 0 end

-- [Edit: just-uploaded image always needs thumbnail regenerated, so should skip to bottom to send trigger]

-- TODO: 
-- string manipulation on path to get path to thumbnail
local thumb = dir .. "thumbs/" .. file .. ext

-- check if thumbnail exists
if (lighty.c.stat(thumb)) then return 0 end

-- thumbnail does not exist; trigger thumbnail creation

-- TODO: fork and exec shell script to handle the rest
--       (shell script should background itself so that lua can wait
--        for the pid and continue execution without pausing too long)
--
-- A quick search in your favorite search engine: (YMMV)
-- https://stackoverflow.com/questions/13228200/lua-fork-concurrent-processes

return 0

All the code up to checking whether or not the thumbnail exists runs inside lighttpd and is very, very fast -- much faster than running a shell script only to find that the thumbnail already exists. Once it is determined that the thumbnail needs to be created, then a trigger or signal is sent, either to a standalone daemon to create the thumbnail, or by forking and execing a script to create the thumbnail. (If you're going to be running a daemon to monitor filesystem paths with inotify, then that daemon instead could listen for trigger signals (socket connection and request) from the lua script, rather than forking and execing from inside lighttpd.)

[gstrauss]

Reviewing the above, if thumb-trigger.lua detects a successful PUT request (201 or 204) for an image file, then there is no need to check if the thumbnail already exists because the thumbnail needs to be regenerated for the newly uploaded file. Therefore, the trigger should always be sent after a successful PUT for an image file for which you want thumbnails.

[stokito]

I finally found a problem: the req_attr["http-status"] returns nil.
lighttpd 1.4.63-2

I'll try latter. Thank you anyway.

For now I wrote a script that detects a new file upload:

#!/bin/sh
inotifywait -e MOVED_TO /srv/disk/dav/|
while read path action file;
do
   echo "new $path $action $file"
done

[gstrauss]

Another alternative is to generate the thumbnail upon first access with something like:

$HTTP["url"] =~ "/thumbs/" {
    url.rewrite-if-not-file = ( "" => "/cgi-bin/thumb-generator.py" )
}
```
and configure CGI to run "/cgi-bin/thumb-generator.py" (not provided above)

[gstrauss]

Oh, by the way: my mistake: req_attr["http-status"] should be req_attr["response.http-status"]. I've edited to fix that in my comment above.

[gstrauss]

@stokito FYI: I wrote a wiki page with code describing how to use lighttpd mod_magnet and lua to write trigger functions, e.g. how to generate a thumbnail after a successful image upload.

[stokito]

Thank you, I'll read.

I'm not sure if it worth to use it here. The inotifytools takes 20kb of space which is not critical but still quite a lot. And it's an additional dependency.
But as an advantage it works on filesystem level e.g. I may upload an image with SFTP and it will be captured.

In the same time a hook on web server level gives more flexibility. For example on FS level we see only moved_to event instead of create. And we don't have headers like content type, user, etc. This makes the inotify solution much complicated.

I feel that it's not a good idea to use Lua scripts and especially that are specific to the Lighttpd only.

In ideal world I need some kind of CGI interceptor that works for the whole path. Such interceptor may, for example, add headers and perform some custom authorization.
This is something that might make old CGI technology more powerful.

I'll think about this latter.

[gstrauss]

In ideal world I need some kind of CGI interceptor that works for the whole path. Such interceptor may, for example, add headers and perform some custom authorization.
This is something that might make old CGI technology more powerful.

All that CGI info is available to and can be manipulated with lua in lighttpd mod_magnet, but yes, it would be a lighttpd-specific enhancement to webdav-js.

As you found, the inotifytools is relatively small and seems like a better fit to work independently with multiple webservers and with multiple alternative upload methods (e.g. SFTP)

If that is not good enough, then before writing your own CGI, you might look at other heavyweight server-side solutions: owncloud or nextcloud might be worth a look. I think both use SabreDAV.

[dom111]

Hey @stokito,

I'm considering breaking out the current list view into an optional package (included by default in the bookmarklet/examples) and making a tile view (which would easily allow thumbnail support when combined with something like the above script.

I'll update here when I've made a start!

Appreciate your input on this too @gstrauss!

Thanks both,

Dom