"quality" settings gives a hint that my photo will be reconverted? #14

vitaly-zdanevich · 2024-05-21T01:00:40Z

But I want to upload the original quality.

trougnouf · 2024-05-21T09:26:46Z

This uses existing darktable export file formats / output options, the output and options are exactly the same as if you were to set the target storage to "file on disk", but only allowing what's supported on Wikimedia Commons. It works off the raw/input file and applies the processing pipeline, there is no quality loss inherent to this plugin.

Darktable does not have an option to output a copy of a (JPEG) file without going through its processing pipeline.

vitaly-zdanevich · 2024-05-21T10:26:01Z

My jpg was 2.1 mb, on Commons I found that it size if 2.6 mb - so it was reencoded :(

This is bad, I cannot use such great plugin because of this :(

Should I create a ticket about this in Darktable itself?

trougnouf · 2024-05-21T15:24:41Z

The issue you are describing is more relevant to darktable, yes.

I don't understand what is your use-case.

Any image you open through darktable will go through its processing pipeline.

Are you using darktable to develop your (ideally RAW) images? If yes then you will get exactly the same image out whether you chose "file on disk" or "Wikimedia Commons" on the darktable target storage. Otherwise, if you have JPEG images that you want to upload without doing any processing then there are other tools that do just that.

If you developed an image in darktable and saved it to disk (so that you have both the (ideally RAW) original image and the JPEG output), then make sure you are still working off the original RAW image rather than the already processed JPEG image (or darktable will process it again with whatever is in your default pipeline).

ralfbrown · 2024-05-21T16:21:29Z

Allowing ext=="" in register_storage_supported should enable the "copy" format to skip the reencoding.

trougnouf · 2024-05-21T18:07:38Z

Allowing ext=="" in register_storage_supported should enable the "copy" format to skip the reencoding.

Done in b21c2f2 . I had to add an additional extension check because register_storage_supported is no longer sufficient to check if a file format is supported.
For some reason exporting an image which fails that check lead to a crash (free(): double free detected in tcache 2\n Aborted (core dumped)), I don't know if that's related to darktable or the plugin.
I did not test whether this works, please let me know if it's useful.

vitaly-zdanevich · 2024-05-21T20:49:00Z

Updated the plugin, now I see the copy. Tried to upload my photos - crash:

$ darktable -d lua
darktable 4.6.0
Copyright (C) 2012-2023 Johannes Hanika and other contributors.

Compile options:
  Bit depth              -> 64 bit
  Debug                  -> DISABLED
  SSE2 optimizations     -> ENABLED
  OpenMP                 -> ENABLED
  OpenCL                 -> DISABLED
  Lua                    -> ENABLED  - API version 9.2.0
  Colord                 -> DISABLED
  gPhoto2                -> DISABLED
  GMIC                   -> DISABLED - Compressed LUTs are NOT supported
  GraphicsMagick         -> DISABLED
  ImageMagick            -> DISABLED
  libavif                -> ENABLED
  libheif                -> ENABLED
  libjxl                 -> DISABLED
  OpenJPEG               -> DISABLED
  OpenEXR                -> DISABLED
  WebP                   -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils.lua 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils.lua is already in search path, returning... 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/debug.lua 
     2.4936 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/debug.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/file.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/file.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/log.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/log.lua is already in search path, returning... 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/string.lua 
     2.4937 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/string.lua is already in search path, returning... 
     2.4938 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 516: line is /home/vitaly/.config/darktable/lua/lib/dtutils/system.lua 
     2.4938 LUA DEBUG: script_manager.lua: ensure_lib_in_search_path: 519: /home/vitaly/.config/darktable/lua/lib/dtutils/system.lua is already in search path, returning... 
Credentials: main-account
Performing HTTP request
  Path:	https://commons.wikimedia.org/w/api.php
  Request body:	action=query&format=json&meta=userinfo
  Result status:	200
  Result body:	{"batchcomplete":"","query":{"userinfo":{"id":0,"name":"188.169.244.17","anon":""}}}
Not logged in, need to login
Performing HTTP request
  Path:	https://commons.wikimedia.org/w/api.php
  Request body:	action=query&format=json&meta=tokens&type=login
  Result status:	200
  Result body:	{"batchcomplete":"","query":{"tokens":{"logintoken=***}}}
Performing HTTP request
  Path:	https://commons.wikimedia.org/w/api.php
  Request body:	action=clientlogin&loginreturnurl=https%3A%2F%2Fwww.mediawiki.org&password=***&format=json&username=Vitaly+Zdanevich&logintoken=***
  Result status:	200
  Result body:	{"clientlogin":{"status":"PASS","username":"Vitaly Zdanevich"}}
   278.7411 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:426: attempt to call a nil value (field 'ggettest')
stack traceback:
	[C]: in field 'ggettest'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:426: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:414>
PANIC: unprotected error in call to Lua API (bad argument #-1 (dt_lua_image_t expected))
Aborted

trougnouf · 2024-05-21T20:54:24Z

I made a crash causing typo, sorry. It's fixed in b68b1e2

vitaly-zdanevich · 2024-05-21T20:56:49Z

Related on that - looks like avif is supported by MediaWiki - would you like to add that too?

vitaly-zdanevich · 2024-05-21T20:57:42Z

I made a crash causing typo, sorry. It's fixed in b68b1e2

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

trougnouf · 2024-05-21T21:02:12Z

Related on that - looks like avif is supported by MediaWiki - would you like to add that too?

It's not yet supported on Wikimedia Commons per https://phabricator.wikimedia.org/T257719

trougnouf · 2024-05-21T21:02:36Z

I made a crash causing typo, sorry. It's fixed in b68b1e2

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

What type of file are you trying to upload?

vitaly-zdanevich · 2024-05-21T21:03:15Z

xxx.JPG

trougnouf · 2024-05-21T21:09:06Z

xxx.JPG

I added case insensitivity in 3c4738a

vitaly-zdanevich · 2024-05-21T21:28:07Z

Updated, tried again:

dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
dtMediaWiki.register_storage_initialize warning: using copy mode from an unsupported image format is known to cause a crash.
Error: /home/vitaly/photos/2023/july/gonio/gonio has an unsupported extension, won't be exported to Wikimedia Commons in copy mode
exported 0/7 images to Wikimedia Commons
free(): double free detected in tcache 2
Aborted

trougnouf · 2024-05-21T22:04:54Z

I added a fix in cc7f446 , I don't know if it will work with copy but at least the error message will be slightly more useful.

vitaly-zdanevich · 2024-05-21T22:39:58Z

register_storage_store: exporting the following image:
/tmp/IMG_0332.(Gonio_Fortress__red_hair_man)
    11.9038 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0335.(Gonio Fortress: red hair man sitting)
    11.9119 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0338.(Gonio Fortress: inside)
    11.9155 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0340.(Gonio Fortress: inside 2)
    11.9194 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0344.(Gonio Fortress: red hair man in entrance)
    11.9242 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0347.(Gonio Fortress: looking into hole)
    11.9277 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
register_storage_store: exporting the following image:
/tmp/IMG_0389.(Gonio Fortress: red hair man staying)
    11.9325 LUA ERROR : .../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: attempt to concatenate a nil value (local 'ext')
stack traceback:
	[C]: in metamethod 'concat'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:178: in upvalue 'make_image_name'
	.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:392: in function <.../share/darktable/lua/contrib/dtMediaWiki/dtMediaWiki.lua:383>
exported 7/7 images to Wikimedia Commons

trougnouf · 2024-05-21T22:48:23Z

I don't know why uploading image copies yields no extension. I normally get something like

register_storage_store: exporting the following image:
/tmp/DSC_4474.jpg(Trougnouf's sandbox)

image.path has the original file's extension so it could be taken from there in make_image_name and _register_storage_store_final_format_check if the temporary file's is empty.

vitaly-zdanevich · 2024-05-24T23:42:01Z

Hi, no insights about how to solve this?

trougnouf · 2024-05-24T23:56:18Z

Hi, no insights about how to solve this?

Yes, the solution is

image.path has the original file's extension so it could be taken from there in make_image_name and _register_storage_store_final_format_check if the temporary file's is empty.

I don't have time to work on it at the moment (PRs welcome)

vitaly-zdanevich · 2024-05-25T04:01:46Z

Thanks, I tried - but I found that image.path has full path without the extension.

trougnouf · 2024-05-25T08:32:36Z

Thanks, I tried - but I found that image.path has full path without the extension.

That's unfortunate. Thanks for trying. If we have no way to check that the copied file format is compatible then I guess the next best thing is to make copy an activable flag in the settings with a warning in the tooltip.

vitaly-zdanevich · 2024-05-26T01:06:54Z

register_storage_store() receives tmp_exp_path without file extension, so I tried

tmp_exp_path = '/tmp/' .. image.filename

In logs I see

Performing HTTP request
  Path:	https://commons.wikimedia.org/w/api.php
  Request body:	type=csrf&meta=tokens&action=query&format=json
  Result status:	200
  Result body:	{"batchcomplete":"","query":{"tokens":{"csrftoken=***}}}
Failed to export Gonio_Fortress__red_hair_man IMG_0332.JPG
exported 0/1 images to Wikimedia Commons

Why? Is it possible to print error details?

trougnouf closed this as completed May 21, 2024

vitaly-zdanevich mentioned this issue May 21, 2024

Export: please add ability to export without reencoding darktable-org/darktable#16846

Closed

trougnouf mentioned this issue May 21, 2024

README.md: add sad line saying that images always will be reencoded #18

Closed

trougnouf referenced this issue May 21, 2024

add support for copy format without processing

b21c2f2

trougnouf reopened this May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"quality" settings gives a hint that my photo will be reconverted? #14

"quality" settings gives a hint that my photo will be reconverted? #14

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 •

edited

Loading

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 •

edited

Loading

ralfbrown commented May 21, 2024

trougnouf commented May 21, 2024 •

edited

Loading

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 •

edited

Loading

vitaly-zdanevich commented May 24, 2024

trougnouf commented May 24, 2024

vitaly-zdanevich commented May 25, 2024

trougnouf commented May 25, 2024

vitaly-zdanevich commented May 26, 2024

"quality" settings gives a hint that my photo will be reconverted? #14

"quality" settings gives a hint that my photo will be reconverted? #14

Comments

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 • edited Loading

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 • edited Loading

ralfbrown commented May 21, 2024

trougnouf commented May 21, 2024 • edited Loading

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024

vitaly-zdanevich commented May 21, 2024

trougnouf commented May 21, 2024 • edited Loading

vitaly-zdanevich commented May 24, 2024

trougnouf commented May 24, 2024

vitaly-zdanevich commented May 25, 2024

trougnouf commented May 25, 2024

vitaly-zdanevich commented May 26, 2024

trougnouf commented May 21, 2024 •

edited

Loading

trougnouf commented May 21, 2024 •

edited

Loading

trougnouf commented May 21, 2024 •

edited

Loading

trougnouf commented May 21, 2024 •

edited

Loading