Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kemonoparty][error] An unexpected error occurred: TypeError - argument of type 'NoneType' is not iterable. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues . #6415

Closed
RJFAC opened this issue Nov 3, 2024 · 23 comments

Comments

@RJFAC
Copy link

RJFAC commented Nov 3, 2024

[gallery-dl][debug] Version 1.27.7 - Executable (stable/windows)
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.22000
[gallery-dl][debug] requests 2.32.3 - urllib3 2.2.3
[gallery-dl][debug] Configuration Files ['%USERPROFILE%\gallery-dl\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://kemono.su/fanbox/user/2060482'
[kemonoparty][debug] Using KemonopartyUserExtractor for 'https://kemono.su/fanbox/user/2060482'
[kemonoparty][debug] Loading cookies from 'F:/cookies-kemono-party.txt'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): kemono.su:443
[urllib3.connectionpool][debug] https://kemono.su:443 "GET /fanbox/user/2060482 HTTP/11" 200 1020
[kemonoparty][error] An unexpected error occurred: TypeError - argument of type 'NoneType' is not iterable. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemonoparty][debug]
Traceback (most recent call last):
File "gallery_dl\job.pyc", line 151, in run
File "gallery_dl\extractor\kemonoparty.pyc", line 66, in items
File "html_init_.pyc", line 130, in unescape
TypeError: argument of type 'NoneType' is not iterable

@mikf
Copy link
Owner

mikf commented Nov 3, 2024

For now, you can prevent this error by disabling metadata.


I thought kemono is shutting down in less than a month. Then why is the owner changing the site and adding new API endpoints like https://kemono.su/api/v1/fanbox/user/2060482/profile ?

@RJFAC
Copy link
Author

RJFAC commented Nov 3, 2024

I thought kemono is shutting down in less than a month.

Never heard of that, the dev can still be contact on Matrix.
https://kemono.su/matrix

Then why is the owner changing the site and adding new API endpoints like https://kemono.su/api/v1/fanbox/user/2060482/profile ?

Hours ago the dev uploaded the new api documents, both kemono and coomer.
https://kemono.su/documentation/api

@valdearg
Copy link

valdearg commented Nov 3, 2024

For now, you can prevent this error by disabling metadata.

I thought kemono is shutting down in less than a month. Then why is the owner changing the site and adding new API endpoints like https://kemono.su/api/v1/fanbox/user/2060482/profile ?

Nah, bit of fear mongering and people not understanding what the admin said. What's actually happening is just that the current server provider is unreliable and they're switching out the servers.

@mikf
Copy link
Owner

mikf commented Nov 3, 2024

I thought kemono is shutting down in less than a month.

Never heard of that

#6096 (comment)
https://chan.kemono.party/kemono/res/46156.html#46779

@tezrilet
Copy link

tezrilet commented Nov 3, 2024

Apologies, I might have misinterpreted that comment, though it was worded a bit vaguely. The admin issued an update later on: https://chan.kemono.party/kemono/res/46907.html#46908

It sounds like they just got fed up with the old server host and will be colocating with dedicated hardware instead, not shutting down the site entirely.

@mo-han
Copy link
Contributor

mo-han commented Nov 4, 2024

For now, you can prevent this error by disabling metadata.

with -o metadata=false, another error:

[kemonoparty][error] An unexpected error occurred: KeyError - 'service'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[kemonoparty][debug]
Traceback (most recent call last):
  File "C:\Users\mo-han\AppData\Local\Programs\Python\Python38\lib\site-packages\gallery_dl\job.py", line 151, in run
    for msg in extractor:
  File "C:\Users\mo-han\AppData\Local\Programs\Python\Python38\lib\site-packages\gallery_dl\extractor\kemonoparty.py", line 82, in items
    self.root, post["service"], post["user"], post["id"])
KeyError: 'service'

@AlttiRi
Copy link

AlttiRi commented Nov 4, 2024

[urllib3.connectionpool][debug] https://kemono.su:443 "GET /fanbox/user/2060482 HTTP/11" 200 1020

They no longer render HTML on the server side, they now just return an HTML file with minimal content, so the site is now an SPA.

All data should be taken from the API, not from HTML.

@mikf mikf pinned this issue Nov 4, 2024
@AlttiRi
Copy link

AlttiRi commented Nov 11, 2024

I thought kemono is shutting down in less than a month.

It looks they have some server issues. And they are going to switch the servers to another ones.

@mikf
Copy link
Owner

mikf commented Nov 15, 2024

Extracting username by enabling metadata seems to be working again.

@AlttiRi
Copy link

AlttiRi commented Nov 16, 2024

They added SSR (Server-Side Rendering).

Seems, they just returned the old web-server's code.
Temporally? Technically, the SPA (Single Page Application) uses fewer server resources.

@mikf mikf unpinned this issue Nov 18, 2024
@Aroy-Art
Copy link

It looks like the new SPA is live again
docs can be found here
https://kemono.su/documentation/api

@valdearg
Copy link

We're back on the new version! Will it stay, will it be reverted again? Who knows!

@valdearg
Copy link

OK, the Admin for Kemono has confirmed no roll back this time, only a couple of small display issues so nothing enough to need reverting.

@mikf mikf pinned this issue Nov 26, 2024
mikf added a commit that referenced this issue Nov 26, 2024
(#6415, #6503, #6528, #6530, #6536)

… at least for the most part. Favorites are still broken, but the rest
should be functional again.
@mikf
Copy link
Owner

mikf commented Nov 26, 2024

Everything except favorites should be working again: 74d855c

To update:

  • pip (Windows)
    py -3 -m pip install -U --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
    
  • pip
    python3 -m pip install -U --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
    
  • Executable
    gallery-dl --update-to dev
    

@valdearg
Copy link

Thank you so much! One thing I've just noticed, if comments are enabled within the config, if there are no comments on a page, the response gives a 404 response.

This means that gallery-dl seems to exit.:

        "comments": true,

Gives:

[kemonoparty][error] HttpError: '404 NOT FOUND' for 'https://kemono.su/api/v1/gumroad/user/4316243074436/post/cfqjw/comments'

That response for the page is actually:

{
"error": "No comments found."
}

@a84r7a3rga76fg
Copy link

@valdearg Could you share your config for saving comments? Long time ago, I failed with setting up a config that saved comments to a text file and didn't save the same comments twice. I'm not sure if I did anything wrong or if comments can't be added to an archive.

@valdearg
Copy link

@valdearg Could you share your config for saving comments? Long time ago, I failed with setting up a config that saved comments to a text file and didn't save the same comments twice. I'm not sure if I did anything wrong or if comments can't be added to an archive.

I have the following set:

{
    "extractor": {
        "kemonoparty": {
            "metadata": true,
            "comments": true,
        },
   }
}

I've disabled the comments part for the moment as I don't require it.

@Triramama
Copy link

Triramama commented Nov 27, 2024

[config][error] JSONDecodeError when loading 'config.json': Expecting ',' delimiter: line 6 column 13 (char 153)
[kemonoparty][error] An unexpected error occurred: KeyError - 'service'
this is all i get when trying to download somebody help
this is how my config.json looks, i want to scrape with filename date+id+title+number and i want to save comments too, what am i doing wrong
image

@valdearg
Copy link

[config][error] JSONDecodeError when loading 'config.json': Expecting ',' delimiter: line 6 column 13 (char 153) [kemonoparty][error] An unexpected error occurred: KeyError - 'service' this is all i get when trying to download somebody help this is how my config.json looks, i want to scrape with filename date+id+title+number and i want to save comments too, what am i doing wrong image

Your sleep-request line should be an array, similar to the line below.

Here's mine:
"sleep-request": [2.0, 3.0],

@Triramama
Copy link

Triramama commented Nov 27, 2024

ok apparently the problem is with kemono screwing up their api and no scraper works at all, except for the "kemono-scraper" fork for some reason huh

@mikf
Copy link
Owner

mikf commented Nov 27, 2024

Your sleep-request line should be an array, similar to the line below.
Here's mine: "sleep-request": [2.0, 3.0],

Both are valid. See https://gdl-org.github.io/docs/configuration.html#duration

@Triramama
Update to -dev: #6415 (comment)

@Triramama
Copy link

` name enclosed in double quotes: line 55 column 9 (char 2212)
[update][info] Updating from 1.27.0 to 2024.11.26
C:\Users*\Downloads\gallery-dl.exe

[kemonoparty][error] An unexpected error occurred: KeyError - 'service'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .`

still broken for me

@mikf
Copy link
Owner

mikf commented Nov 28, 2024

@valdearg
Empty comments are fixed with 2162fa7

Logging in to retrieve favorites is fixed as well: 7c7b8a2
(At the time of writing, login is still broken on the kemono website itself)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants