jdownloader2: init at 2.0 #136998

remgodow · 2021-09-07T16:06:18Z

Motivation for this change

JDownloader is a popular file downloader. Nix package downloads only bootstraping file and creates a wrapper for it. Inspired by JDownloader flatpak. Unfortunately there is no way to package it "the nix way" due to how the app is designed (mandatory selfupdate + writes configs/logs in its dir, and no way to change that). The upside is that this package should not require any updates.
Resolves: #41321 and #76568

Things done

bobby285271

Thanks for your contribution! Builds and runs for me :-)

pkgs/applications/networking/jdownloader2/default.nix

bobby285271 · 2021-09-08T00:30:49Z

pkgs/top-level/all-packages.nix

@@ -25471,6 +25471,8 @@ with pkgs;

  dupd = callPackage ../tools/misc/dupd { };

+  jdownloader2 = callPackage ../applications/networking/jdownloader2 {};


Nit: one more space here will be nice.

Suggested change

jdownloader2 = callPackage ../applications/networking/jdownloader2 {};

jdownloader2 = callPackage ../applications/networking/jdownloader2 { };

Kranzes · 2021-09-08T18:49:10Z

I tried to this built this, and it builds fine but jdownloader just shows a white, blank screen.

bobby285271 · 2021-09-09T00:05:26Z

Hmmm... That's weird, I can see this interface :-/

Kranzes · 2021-09-09T11:49:53Z

Hmmm... That's weird, I can see this interface :-/

I Figured it out, it was something with bspwm.

remgodow · 2021-09-09T11:53:45Z

@bobby285271 I've implemented your suggestions, it should be ready now.

Kranzes · 2021-09-09T11:56:35Z

The meta set is incorrect.

pkgs/applications/networking/jdownloader2/default.nix

SuperSandro2000

Please follow the contributing guide when doing your commits.

pkgs/applications/networking/jdownloader2/default.nix

SuperSandro2000 · 2021-09-09T13:37:15Z

pkgs/applications/networking/jdownloader2/default.nix

+  version = "2.0";
+
+  src = fetchurl {
+    url = "http://installer.jdownloader.org/JDownloader.jar";


Suggested change

url = "http://installer.jdownloader.org/JDownloader.jar";

url = "https://installer.jdownloader.org/JDownloader.jar";

Please upload this to web.archive.org.

https://web.archive.org/web/2020*/https://installer.jdownloader.org/JDownloader.jar seems like its already there

Then please update the URL with that link.

SuperSandro2000 · 2021-09-09T13:38:40Z

pkgs/applications/networking/jdownloader2/default.nix

+  wrapper = writeScript "jdownloader2.sh" ''
+    #! ${stdenv.shell}
+    PATH=${lib.makeBinPath [ jre coreutils ]}
+
+    JDDIR=''${XDG_DATA_HOME:-$HOME/.local/share}/jdownloader
+    JDJAR=''${JDDIR}/JDownloader.jar
+    dir=`dirname "$0"`
+
+    if [ ! -f ''${JDJAR} ]; then
+        install -Dm755 $dir/JDownloader.jar ''${JDJAR}
+    fi
+
+    ${jre}/bin/java -jar ''${JDDIR}/JDownloader.jar "''${@}"


Can we use makeWrapper instead?

I could not find a way, even with help from NixOS discord channel. The problem is that JDDIR value must remain unchanged/unsubstituted, so that bash can do the substitution on runtime.

pkgs/applications/networking/jdownloader2/default.nix

Kranzes · 2021-09-09T18:44:15Z

Once you're done with writing the expression could you rebase so we only have 2 commits, $package init and maintainers

remgodow · 2021-09-09T18:45:56Z

OK, but wouldnt squash merge do more or less the same thing?

Kranzes · 2021-09-09T18:47:00Z

Might be, im not very good with git.

pkgs/applications/networking/jdownloader2/default.nix

milahu · 2021-10-01T07:33:45Z

nitpick: rename to jdownloader? no one will want jdownloader1, compared to python2, nodejs-10_x, ...

milahu · 2021-10-01T07:41:04Z

pkgs/applications/networking/jdownloader2/default.nix

+      name = "JDownloader 2";
+      exec = wrapper;
+      icon = "jdownloader";
+      comment = "Free, open-source download management tool.";


imho, "FOSS" is not relevant here. i would just say "Download manager"
(you can download all kinds of files with jd, also youtube videos ...)

I took that from the official website, the FOSS refers to JDownloader not the stuff you can download, I think. Sure thing, I can change that.

As for the package name, I can change that as well. For quite some time there were jdownloader and jdownloader2, the jdownloader is obsolete by now, but few people still use the jdownloader2 name as you can see on the repology https://repology.org/projects/?search=jdownloader

milahu · 2021-10-01T07:47:34Z

pkgs/applications/networking/jdownloader2/default.nix

+  wrapper = writeScript "jdownloader" ''
+    #! ${stdenv.shell}
+    PATH=${lib.makeBinPath [ jre coreutils ]}
+    JDJAR=''${XDG_DATA_HOME:-$HOME/.local/share}/jdownloader/JDownloader.jar


personally, i am NOT happy with auto-updates. should be "default off, enable on demand"

also, should be built from source. there is a mirror at https://github.com/mirror/jdownloader
the official repo is password-protected

that "source" contains 250 MByte build artifacts (*.jar files) and 50 MByte sources (*.java files in src/)
ideally, we would download only the src/ folder, like github-downloader (via stackoverflow) wontfix

So to begin with, JDownloader supports many websites via its plugin system, and those are always autoupdated. Main app does not update automatically, only checks for updates, here is more info
https://support.jdownloader.org/Knowledgebase/Article/View/updates-update-behavior

I did not try to build from source, and I'm not sure if it is possible without access to the main repo, for example they use proprietary libraries, and I believe there are closed source parts as well (kinda like VSCode). That would require some tinkering, as for ex. AUR packages are using the bootstrapper as well.

Here is an issue from the mirror about missing dependencies from password-protected official repo. mirror/jdownloader#47

Here is an issue from the mirror about missing dependencies

the svn repos listed in https://jdownloader.org/knowledge/wiki/development/get-started are public, the issue is only a request to mirror these repos

they use proprietary libraries, and I believe there are closed source parts as well

im afraid yes ... im trying to build from source with headless eclipse, but at the end of the day, there will still be some binary blobs (*.jar files with closed source), so jdownloader has license unfree

→ use containers for better security?

pyload is missing : /

$ nix-shell -p pyload error: pyload has been removed from nixpkgs, as it was unmaintained.

→ use containers for better security?

https://github.com/mviereck/x11docker (via)

x11docker

https://github.com/jlesage/docker-jdownloader-2 has invented this wheel already : )

at the end of the day, there will still be some binary blobs (*.jar files with closed source), so jdownloader has license unfree

yepp

Compiling 4610 source files to /build/jdownloader/bin /build/jdownloader/src/jd/plugins/AccountInfo.java:267: error: cannot find symbol final long serverTime = br.getCurrentServerTime(-1); ^ symbol: method getCurrentServerTime(int) location: variable br of type Browser

jdownloader requires closed sources:

JDClosed JDownloaderServerOptions UpdateClientV2 JDownloaderUpdater

per jdownloader/build/newBuild/build.xml

i want to patch the myjdownloaderclient ...
in headless mode, jdownloader requires a valid account for myjdownloader

jd ignores json config

i have not-yet found a way to use a self-hosted api server, or to disable this completely ...
so much for "untrusted" software

relevant configkey is serverhost (default api.jdownloader.org) in cfg/org.jdownloader.api.myjdownloader.MyJDownloaderSettings.json
but jd always says

|---------------------------Headless Information------------------------------- | MyJDownloader | Your 'My JDownloader' logins are not correct. | Please check username/email and password! | Enter y -> Enter Logins | Enter n -> Exit JDownloader

so im back to binary patching ... or pyLoad

binary patching - its simple, its fun

let jd update, to download jar files
get Core.jar from approot
open recaf
open Core.jar
classes > org > jd > api > myjd > myjdcontroller
Ctrl+F for: boolean isAlwaysConnectRequired

edit the assembler code

DEFINE PUBLIC FINAL isAlwaysConnectRequired()Z A: LINE A 105 -INVOKESTATIC org/appwork/utils/Application.isHeadless()Z -IRETURN +EXPR return false; B:

Ctrl+S, file > export > export to Core.patched.jar

in jd approot, replace Core.jar with Core.patched.jar

now, jd starts in headless mode, without requiring myjd login

vs49688 · 2021-10-03T14:19:55Z

Just thought I'd mention I have one here that I'm using:
https://github.com/vs49688/nix-repo/blob/master/pkgs/jdownloader/default.nix

With custom patches to make it use the XDG_* dirs instead of trying to write to its local directory.
I've also been in contact with Daniel (jiaz) from JDownloader in order to get something proper implemented in regards to this (JD Ticket #ADF-935-28292). He explicitly asked to hold off on packaging until he can get proper handling in place, but can't give an ETA:

Thanks again for detailed feedback :)
I will setup a VM and try to get familiar with it, so I can test my changes :)
But I can't promise when I will find time, how fast I will be.

I will ping you when there is progress/are questions, okay?

daniel


Ticket Details
Ticket ID: ADF-935-28292
Department: Development
Type: Issue
Status: Todo
Priority: Low

Support Center: https://support.jdownloader.org

AndersonTorres · 2021-11-02T22:00:03Z

Drafted until news.

stale · 2022-05-02T07:24:05Z

I marked this as stale due to inactivity. → More info

AndersonTorres · 2022-05-05T04:41:12Z

Closed

vs49688 · 2022-05-11T13:12:33Z

Quick update. I've been in contact with upstream, and they're adding a build-time option to enable using XDG_* and friends, and are going to provide an endpoint to query the stable SVN revisions for updating.

Will report back once I hear more.

milahu · 2022-05-11T14:14:07Z

using ~/.local/share/jdownloader instead of ~/.jd is nitpickery

jdownloader practically needs autoupdate, as it breaks often

the elephant in the room is JDClosed in build.xml

	<property name="dep.jdc" value="../JDClosed" />

which is a known wontfix

"jdownloader requires closed sources"
-> Yes, it is true that our official build also contains closed source parts

Repository: JDClosed
-> plugin sources that are not public due to the sites looking for plugin changes to break them again and again and again.
of course we can provide an up2date zip of the sources or even provide access to it but we don't want those parts to be public as
this just will cause the plugins to break (very) fast and end up in cat/mice game.

→ im running jd in docker #41321 (comment)

vs49688 · 2022-05-11T14:51:34Z

using ~/.local/share/jdownloader instead of ~/.jd is nitpickery

Perhaps.

jdownloader practically needs autoupdate, as it breaks often

That's why they're providing an endpoint to get build.json for the latest stable version.
build.json contains SVN revision numbers, of which it's to modify my update script to use, then it's nixpkgs-update's problem.

the elephant in the room is JDClosed in build.xml
	<property name="dep.jdc" value="../JDClosed" />
which is a known wontfix

"jdownloader requires closed sources"
-> Yes, it is true that our official build also contains closed source parts
Repository: JDClosed
-> plugin sources that are not public due to the sites looking for plugin changes to break them again and again and again.
of course we can provide an up2date zip of the sources or even provide access to it but we don't want those parts to be public as
this just will cause the plugins to break (very) fast and end up in cat/mice game.

→ im running jd in docker #41321 (comment)

The continue to do so by all means. But for "casual" users like myself, who don't download from "hostile" sites very often, the open-source version is fine.

milahu · 2022-05-11T15:01:08Z

the open-source version

wait, there is an open-source version?

when i try to compile jd without JDClosed.jar (etc) i get the error

Compiling 4610 source files to /build/jdownloader/bin
/build/jdownloader/src/jd/plugins/AccountInfo.java:267: error: cannot find symbol
        final long serverTime = br.getCurrentServerTime(-1);
                                  ^
  symbol:   method getCurrentServerTime(int)
  location: variable br of type Browser

vs49688 · 2022-05-11T15:05:05Z

the open-source version

wait, there is an open-source version?

when i try to compile jd without JDClosed.jar (etc) i get the error

Compiling 4610 source files to /build/jdownloader/bin
/build/jdownloader/src/jd/plugins/AccountInfo.java:267: error: cannot find symbol
        final long serverTime = br.getCurrentServerTime(-1);
                                  ^
  symbol:   method getCurrentServerTime(int)
  location: variable br of type Browser

Yup, I've got a (slightly outdated) one here: https://github.com/vs49688/nix-repo/tree/master/pkgs/jdownloader

Once upstream's finished with their changes, I planned to submit it here for inclusion.

I think the main difference is that I'm using build_standalone.xml instead of build.xml.

milahu · 2022-05-11T15:13:33Z

$ find trunk/ browser/ MyJDownloaderClient/ -name '*.jar' | wc -l 
76

still lots of room to hide malicious binaries. this is not "open source"

dependencies should be fetched from maven (or similar)

other binaries:

$ git ls-tree -r -t -l --full-name HEAD | sort -n -k 4 -r >large-files.txt
$ head -n 100 large-files.txt | cut -d$'\t' -f2 | grep -v -e 'zip$' -e 'png$' -e '\.js$' -e 'psd$' | head 
trunk/ressourcen/tools/linux/ffmpeg/x64/ffmpeg
trunk/ressourcen/tools/linux/ffmpeg/x64/ffprobe
trunk/ressourcen/tools/mac/ffmpeg_10.10+/libavcodec.58.dylib
trunk/ressourcen/tools/linux/ffmpeg/i386/ffmpeg
trunk/ressourcen/tools/linux/ffmpeg/i386/ffprobe
trunk/ressourcen/tools/Windows/ffmpeg/x64/avcodec-58.dll
trunk/ressourcen/tools/Windows/ffmpeg/i386/avcodec-58.dll
trunk/ressourcen/tools/mac/ffmpeg_10.5.x-/ffmpeg
trunk/ressourcen/tools/mac/ffmpeg_10.5.x-/ffprobe
trunk/ressourcen/tools/mac/ffmpeg_10.6+/ffmpeg

similar:

Linux-libre is a modified version of the Linux kernel that contains no binary blobs, obfuscated code, or code released under proprietary licenses.

vs49688 · 2022-05-11T15:25:11Z

Baby steps. Upstream's been receptive to changes so far, I don't want to demand too much of them at once.

milahu · 2022-05-12T11:55:08Z

for "casual" users like myself, who don't download from "hostile" sites very often, the open-source version is fine.

did you try pyload?

vs49688 · 2022-05-12T14:49:18Z

other binaries:

$ git ls-tree -r -t -l --full-name HEAD | sort -n -k 4 -r >large-files.txt
$ head -n 100 large-files.txt | cut -d$'\t' -f2 | grep -v -e 'zip$' -e 'png$' -e '\.js$' -e 'psd$' | head 
trunk/ressourcen/tools/linux/ffmpeg/x64/ffmpeg
trunk/ressourcen/tools/linux/ffmpeg/x64/ffprobe
trunk/ressourcen/tools/mac/ffmpeg_10.10+/libavcodec.58.dylib
trunk/ressourcen/tools/linux/ffmpeg/i386/ffmpeg
trunk/ressourcen/tools/linux/ffmpeg/i386/ffprobe
trunk/ressourcen/tools/Windows/ffmpeg/x64/avcodec-58.dll
trunk/ressourcen/tools/Windows/ffmpeg/i386/avcodec-58.dll
trunk/ressourcen/tools/mac/ffmpeg_10.5.x-/ffmpeg
trunk/ressourcen/tools/mac/ffmpeg_10.5.x-/ffprobe
trunk/ressourcen/tools/mac/ffmpeg_10.6+/ffmpeg

Yup, they're going to make it search $PATH for ffmpeg and friends, so these can just be rm'd for our purposes.

for "casual" users like myself, who don't download from "hostile" sites very often, the open-source version is fine.

did you try pyload?

I've never heard of it, I'll have a look.

bobby285271 reviewed Sep 8, 2021

View reviewed changes

bobby285271 reviewed Sep 9, 2021

View reviewed changes

pkgs/applications/networking/jdownloader2/default.nix Outdated Show resolved Hide resolved

Kranzes reviewed Sep 9, 2021

View reviewed changes

pkgs/applications/networking/jdownloader2/default.nix Outdated Show resolved Hide resolved

Kranzes reviewed Sep 9, 2021

View reviewed changes

pkgs/applications/networking/jdownloader2/default.nix Outdated Show resolved Hide resolved

SuperSandro2000 requested changes Sep 9, 2021

View reviewed changes

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Sep 9, 2021

remgodow force-pushed the jdownloader2 branch from c0e513c to 9a0f404 Compare September 9, 2021 19:56

SuperSandro2000 reviewed Sep 11, 2021

View reviewed changes

pkgs/applications/networking/jdownloader2/default.nix Outdated Show resolved Hide resolved

remgodow force-pushed the jdownloader2 branch from 9a0f404 to ad58647 Compare September 12, 2021 19:59

remgodow added 2 commits September 13, 2021 19:58

jdownloader2: init at 2.0

7d075d3

maintainers: add remgodow

29d4a33

remgodow force-pushed the jdownloader2 branch from ad58647 to 29d4a33 Compare September 13, 2021 17:58

milahu reviewed Oct 1, 2021

View reviewed changes

AndersonTorres marked this pull request as draft November 2, 2021 21:59

stale bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label May 2, 2022

AndersonTorres closed this May 5, 2022

		@@ -25471,6 +25471,8 @@ with pkgs;

		dupd = callPackage ../tools/misc/dupd { };

		jdownloader2 = callPackage ../applications/networking/jdownloader2 {};

	url = "http://installer.jdownloader.org/JDownloader.jar";
	url = "https://installer.jdownloader.org/JDownloader.jar";

jdownloader2: init at 2.0 #136998

jdownloader2: init at 2.0 #136998

Conversation

remgodow commented Sep 7, 2021 • edited Loading

Motivation for this change

Things done

bobby285271 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kranzes commented Sep 8, 2021

bobby285271 commented Sep 9, 2021

Kranzes commented Sep 9, 2021

remgodow commented Sep 9, 2021

Kranzes commented Sep 9, 2021

SuperSandro2000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kranzes commented Sep 9, 2021

remgodow commented Sep 9, 2021

Kranzes commented Sep 9, 2021

milahu commented Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milahu Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

remgodow Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milahu Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milahu Oct 3, 2021 • edited Loading

Choose a reason for hiding this comment

vs49688 commented Oct 3, 2021

AndersonTorres commented Nov 2, 2021

stale bot commented May 2, 2022

AndersonTorres commented May 5, 2022

vs49688 commented May 11, 2022

milahu commented May 11, 2022

vs49688 commented May 11, 2022

milahu commented May 11, 2022

vs49688 commented May 11, 2022 • edited Loading

milahu commented May 11, 2022 • edited Loading

vs49688 commented May 11, 2022

milahu commented May 12, 2022

vs49688 commented May 12, 2022

remgodow commented Sep 7, 2021 •

edited

Loading

milahu commented Oct 1, 2021 •

edited

Loading

milahu Oct 1, 2021 •

edited

Loading

remgodow Oct 1, 2021 •

edited

Loading

milahu Oct 1, 2021 •

edited

Loading

milahu Oct 3, 2021 •

edited

Loading

vs49688 commented May 11, 2022 •

edited

Loading

milahu commented May 11, 2022 •

edited

Loading