Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine #92

Open
mileyvirus opened this issue Jan 14, 2022 · 29 comments

Comments

@mileyvirus
Copy link

Hello there, excellent work with the agent. Unfortunately it seems there's an issue with the channel metadata and thumbnail not appearing but episode metadata and thumbnails work fine.

  • My output template is --output "/Media/YouTube/%(uploader)s [%(channel_id)s]/%(upload_date>%Y-%m-%d)s - %(title)s [%(id)s].%(ext)s" which resolves to /Media/YouTube/belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]/2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].mkv
  • I'm using my own API key
  • Using latest YouTube Agent and ASS Scanner on latest PMS on Windows 11 (latest as today)
  • Logs attached (tried multiple wipes, reinstalls and restarts)

Screenshot of "Channel metadata and thumbnail not appearing"

chrome_WhkDbpUtpg


Some channel metadata gets scraped apparently

chrome_Rx73LSphpz


Screenshot of "episode metadata and thumbnails working fine"

chrome_bBSgB3WEXn


Log file with API key redacted

com.plexapp.agents.youtube.log

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 15, 2022

No search log in logs provided
No Plex token ID file
Language xn for library = Chinese?

INFO (logkit:16) - Place correct Plex token in X-Plex-Token.id file in logs folder or in PLEX_LIBRARY_URL variable to have a log per library - https://support.plex.tv/hc/en-us/articles/204059436-Finding-your-account-token-X-Plex-TokenHTTP Error 401: Unauthorized
[ ] dir: "D:\Media\YouTube\belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]"
INFO (logkit:16) - No GUID so random folder

(https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py#L533)

UCXvKUavCtDOlA8bT1i2tI3w is 22 chars after UC and match description

Missing search log in logs so cannot check guide assignment but missing token ID file...

@mileyvirus
Copy link
Author

mileyvirus commented Jan 15, 2022

Sorry and thanks for your time. Here's both logs after:

  • Adding X-Plex-Token.id got from here to this path ~\AppData\Local\Plex Media Server\Logs
  • Deleting all Plex logs leaving folders intact
  • Restarting Plex
  • Updating all metadata for the YouTube library

It still isn't working unfortunately…

com.plexapp.agents.youtube.log
com.plexapp.system.log


Language xn for library = Chinese?

Language is set to None in Library settings, unsure why the logs state "xn"


If it's any help I'm using yt-dlp 2021.12.27.


Library file tree

.
├── belle delphine [UCXvKUavCtDOlA8bT1i2tI3w]
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].description
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].info.json
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].mkv
│  ├── 2020-06-17 - I'M BACK -belle delphine [TL470fJMi7w].webp
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].description
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].info.json
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].mkv
│  ├── 2021-02-14 - my room tour (worlds largest sex toy collection idk) [cfLX_s8bsfE].webp
│  ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].description
│  ├── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].info.json
│  └── NA - belle delphine - Videos [UCXvKUavCtDOlA8bT1i2tI3w].jpg
└── yotobi [UCIPsK5xspHC3-ZFNPTx2X_w]
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].description
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].info.json
   ├── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].mkv
   └── 2007-02-12 - Sad moment from scrubs [Lv36agD7Tfo].webp

I tried an old version of YouTube-Agent.bundle I had on an iMac. That one downloads metadata and images for the channels but with wrong image and it sets the title to the filepath.

This old version: YouTube-Agent.bundle.zip

chrome_V3V1eCoZjS

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 15, 2022

Can access library file now but issue persists.
No search was done on that series in logs...

@mileyvirus
Copy link
Author

I'm sorry I don't understand if I should search something on Plex or you're referring to the agent not searching?

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 15, 2022

I need you to match manually the series so I have that in logs, the agent search() funct89n assign a unique I'd and it seems it is what doesn't work here

@mileyvirus
Copy link
Author

mileyvirus commented Jan 17, 2022

Hello again, hope you have a good week.

If you meant "Fix match" and then manually matching the YouTube channel, I did just that and the correct metadata for the channel got scraped BUT now the episodes metadata don't get collected.

Here are the logs:

com.plexapp.agents.youtube.log
com.plexapp.system.log

@cbartondock
Copy link

Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.

@mileyvirus
Copy link
Author

Just wanted to chime in that I'm also seeing this issue - episode metadata is loading fine but series metadata no longer is. No images for the series and the [RANDOM LETTERS AND NUMBERS] are still in the title of the series. Let me know if you'd like me to get logs also.

@cbartondock please do! That way ZeroQI can see your logs together with mine as soon as they have time. I'm pasting the instructions to attach the logs if it's of any help:

  • Update to the latest Absolute Series Scanner, Youtube-Agent
  • Delete all Plex logs leaving folders intact
  • Restart Plex
  • Update the series Metadata
  • Include all the following logs: (location)
    • [...]/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.agents.Youtube-Agent.log (Agent logs)
    • [...]/Plex Media Server/Logs/PMS Plugin Logs/com.plexapp.system.log (show why the agent cannot launch)
  • Attach some screenshots to illustrate if helpful.

@cbartondock
Copy link

Same error as you looks like, the key line being Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'.

snip

com.plexapp.agents.youtube.log
com.plexapp.system.log

@cbartondock
Copy link

@mileyvirus Does the X-Plex-Token.id file just have the token string and nothing else in it? Doesn't seem to be working for me.

@mileyvirus
Copy link
Author

mileyvirus commented Jan 25, 2022

@cbartondock yes, just the token string and nothing else and no line breaks either. I put mine in two places since I didn't figure which was the right path.

C:\Users\<username>\AppData\Local\Plex Media Server\X-Plex-Token.id
C:\Users\<username>\AppData\Local\Plex Media Server\Logs\X-Plex-Token.id

The content of X-Plex-Token.id looks like this (I anonymized mine for posting it here)

HEYXX9aT8UxW7igoDuCk

@cbartondock
Copy link

Okay yeah, I just worked out by looking over __index.py__ that actually the correct path is the first one C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server. Now my logs look better, but still no dice. Going to try debugging __index.py__ further.
com.plexapp.agents.youtube.log
com.plexapp.system.log
.

@cbartondock
Copy link

It seems to be grabbing the guid from a random episode of the channel rather than for the channel itself.

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 25, 2022

"No GUID so random folder" so it considers the folder doesn't have a channel ID indicated...
You should output the metadata.id in logs to see if the search part got the channel id, and if not, you might have used the Plex scanner that would delete all in square brackets which is the channel id...

@cbartondock
Copy link

Pretty sure this is a bug. It is setting the channel guid to the guid of a random episode. metadata.id is youtube|aPhrTOg1RUk|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA] when I believe it should be youtube|UCNvsIonJdJ5E4EXMa65VYpA|Contrapoints [UCNvsIonJdJ5E4EXMa65VYpA]. Here is the directory
snip3
And here is the Info.xml file that is being set in C:\Users\Chris Barton Dock\AppData\Local\Plex Media Server\Metadata\TV Shows\e\73abc82969112e63ba6300258b3a9a97c05c7a2.bundle\Contents\com.plexapp.agents.youtube:
snip2

@cbartondock
Copy link

I believe this is occurring as early as media.filename in the Search(...) function.

@cbartondock
Copy link

The reason it says No GUID is because the GUID is an episode GUID, which it doesn't recognize since it doesn't start with UC or HC

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 25, 2022

ASS scanner let the title with square brackets with channel I'd pass and the agent search function assign a forced guid. Plex scanner will drop square brackets and would cause that exact behaviour, so is ASS the scanner?
If so, un-assign, re-assign the series and attach the agents logs so they include the search part.

Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py line 120 should be display_name instead of filename

@cbartondock
Copy link

cbartondock commented Jan 25, 2022

Ah I see, I don't think I am using ASS and I see now that it is a requirement for YouTube-Agent. Weird that it was working before. If you don't mind my asking, the documentation for ASS (what it does and how to use it for YouTube-Agent) is pretty opaque, how do I set-up ASS?

If not no worries, I got it working in the end without ASS by patching __init__.py with the following (added after line 109):

  a_list=[x for x in os.listdir(dir) if x.endswith('.info.json') and re.search(r'\[(.*)\]',x).group(1)[0:2] in ('PL', 'UU', 'FL', 'LP', 'RD','UC','HC')]
  if len(a_list)>0:
    good_guid = re.search(r'\[(.*)\]',a_list[0]).group(1)
    try:     json_video_details = JSON.ObjectFromString(Core.storage.load(json_filename))  #with open(json_filename) as f:  json_video_details = JSON.ObjectFromString(f.read())
    except:  pass
    else:
      results.Append( MetadataSearchResult( id='youtube|{}|{}'.format(good_guid, os.path.basename(dir)), name=displayname, year=Datetime.ParseDate(Dict(json_video_details, 'upload_date')).year, score=100, lang=lang ) )
      Log(u''.ljust(157, '='))
      return

Basically it just forcibly gets the good guid from the correct .info.json

@mileyvirus
Copy link
Author

mileyvirus commented Jan 26, 2022

Edit: Please test https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py line 120 should be display_name instead of filename

@ZeroQI I tried that change but it still won't work for me. I have the latest ASS.

If not no worries, I got it working in the end without ASS by patching init.py with the following (added after line 109):

@cbartondock I tried your change but it won't work for me.


I'm a Python 3 dev but opening the Agent's code it looks like it's all Python 2 and it has limited libraries. I'm going to look more into how Plex agents work. I tried the other YouTube agent and that works but with some other issues…

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 26, 2022

ASS is a requirement for playlist and channel guide in title functionality, as well as numbering playlist correctly...

My agent also read metadata from local Json file if present like the other agent...

Where are the search agent logs after the change?
The change will take the ass title with brackets, will make no change with Plex default scanner...

@ZeroQI
Copy link
Owner

ZeroQI commented Jan 28, 2022

We need code working within Json then working with local Json file so it benefits everybody. I unfortunately don't use Plex anymore, so don't reproduce libraries, and so need the log to do the coding on the phone most of times

@sourcecodemage
Copy link

I was able to recreate this issue and resolve it ( at least for myself). Initially I was using the TV scanner, and I inserted a "air date" into the file name to make that scanner happy. I also configured youtube-dl to create an "info.json" file for each episode.

The fix was to change the scanner to the ASS scanner, and remove the "info.json" files from the series folder.

To verify the .json file was the issue, I recreated the library with 2 "series", one with the info.json files and one without. The series without the json file has the correct series title, the other has "foldername + channel id" in the name.

One think I could not do was remove the json file and rescan the library to correct the series name. Perhaps I am being impatient, or I'm not performing the correct steps to fix the series name. Ideas?

@ZeroQI
Copy link
Owner

ZeroQI commented Apr 26, 2022

Without Jason it works
With Jason it gives "foldername + channel id" in the name.
When you remove the Json file and rescan (after Plex dance), does it changes anything?
Would need agent logs for both situations

@sourcecodemage
Copy link

sourcecodemage commented May 20, 2022

I can't attach files here. I'll open another issue. with logs and additional details.

edit 1: it looks like github doesn't let you attach logs to issues. I'm looking for an alternative way to share the logs.

edit 2: it accepts .log but not archive file like .tgz. I'll post logs and details in another post.

@ZeroQI
Copy link
Owner

ZeroQI commented May 20, 2022

It does but depends the extension, try txt

@sourcecodemage
Copy link

sourcecodemage commented May 20, 2022

Issue

Channel metadata inc poster missing. episode name/metadata is correct. Channel name is the channel folder name.

*Example: *

Folder is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], and channel name is Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].

Notable Observations

  • The ASS scanner *.hama directory does not exist.
  • ASS Scanner logs are empty.
  • Error decoding with simplejson warning.
  • The resourceHashes critical error.
  • The cause for the "No GUID" is probably what I need to solve.
  • I haven't looked at the Agent code, so I don't know the logic it uses to determine the Channel ID.

Environment

  • OS: Debian v11.3
  • Plex Media Server: v1.26.2.5797
  • ASS Version: commit id 11c303b.
  • Youtube Agent: I don't have the version. But I downloaded and installed it on "Apr 25 06:02"

Plex Token ID File: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/X-Plex-Token.id

Log Excerpts

Plex token and json from top part of com.plex.agent.youtube.log:

2022-05-20 14:06:57,086 (7f162ad1c808) :  INFO (__init__:549) - 'X-Plex-Token.id' file present
2022-05-20 14:06:57,100 (7f1626e3bb38) :  WARNING (data:179) - Error decoding with simplejson, using demjson instead (this will cause a performance hit) - Expecting value: line 5 column 1 (char 518)

Exception getting hosted resource hashes from com.plex.agent.youtube.log:

2022-05-20 12:53:17,364 (7fc7e8ad8808) :  DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,367 (7fc7e8ad8808) :  ERROR (networking:196) - Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2022-05-20 12:53:17,369 (7fc7e8ad8808) :  CRITICAL (runtime:1299) - Exception getting hosted resource hashes (most recent call last):
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/runtime.py", line 1293, in get_resource_hashes
    json = self._core.networking.http_request("http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes", timeout=10).content
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 242, in content
    return self.__str__()
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 220, in __str__
    self.load()
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-5bd057d2b/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 158, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/plexmediaserver/Resources/Python/python27.zip/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found

Data for single channel from com.plex.agent.youtube.log:

2022-05-20 14:10:27,045 (7f1626e3bb38) :  DEBUG (networking:143) - Requesting 'http://127.0.0.1:32400/library/metadata/226165/tree'
2022-05-20 14:10:27,053 (7f1626e3bb38) :  INFO (__init__:165) - === update(lang=xn, force=True, movie=False) ===
2022-05-20 14:10:27,053 (7f1626e3bb38) :  INFO (__init__:40) - /mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]/The_Press_vs_Trump_-_Volume_Three_The_Charlottesville_Lie [4jTf3CMbHrk].mp4
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:509) - =============================================================================================================================================================
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:252) - [ ] library:    "YT"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:253) - [ ] root:       "/mnt/smb/youtube2"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:254) - [ ] path:       "Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:255) - [ ] dir:        "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,054 (7f1626e3bb38) :  INFO (__init__:260) - [ ] series_root_folder: "/mnt/smb/youtube2/Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]"
2022-05-20 14:10:27,056 (7f1626e3bb38) :  INFO (__init__:263) - [ ] subfolder_count:    "0"
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:378) - Grouping folder not found or single folder, root: /mnt/smb/youtube2, path: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], Grouping folder: Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA], subdirs: 0, reverse_path: [u'Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA]']
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:400) - No GUID so random folder
2022-05-20 14:10:27,057 (7f1626e3bb38) :  INFO (__init__:410) - =============================================================================================================================================================

Log file attachments

Plex Media Scanner.log

com.plexapp.agents.youtube.log

ASS Scanner Logs for channel I'm focusing on were empty so I can't attach them. Here are their names.

  • Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].filelist.log
  • Don_t_Walk_Run_Productions [UCwpDBy43upJR7LZLrgnKkPA].scanner.log

@sourcecodemage
Copy link

sourcecodemage commented May 22, 2022

ok.I redownloaded my library again with better results. Since I started, I made the following changes

  1. I added the Plex Token file
  2. (yt-dlp) added --restrict-filename to only allow ascii filenames.
  3. (yt-dlp) included the channel name in the filename to remove non-ascii chars from that as well. My channel name had some accented chars in them.

Channel Names , episode names, metadata, and images are all correct now. ( yay!) .

However, I found a new problem. Youtube episodes released on the same day are absent. I had a case where 5 of them had the same modified date and none of them were discovered by the scanner., The Plex scanner log has the write number of episodes, but doesn't add them.

This is my file naming convention, is correct? Or should I change it? I don't use playlist so there isn't a playlist id.
-o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s"

The overall script looks like this
${ytdlp_path} \ --match-filter "is_live != true & was_live != true" \ ${PROXY_CMD} \ --abort-on-error \ --no-warnings \ --restrict-filenames \ --no-progress \ --add-metadata \ --ffmpeg-location "${ffmpeg_path}" \ --recode "mp4" \ --batch-file="${batch_file}" \ --cookies "${cookies_file}" \ --download-archive "${download_archive}" \ -o "%(channel)s [%(channel_id)s]/%(title)s [%(id)s].%(ext)s" \ --exec post_process:"./scripts/move_video_file.sh -f '%(filepath)s' -r '${root_path}'"

I made the mistake of touching all of 24 file in a channel folder, and not only one is shown in Plex. So the scanner seems to be using the modified date and only allowing one episode per day.

I tried a few things, but what got me results was adding MMDDhhmm to the file name. The scanner picks up these files and I see them in Plex.
But the youtube agent doesn't fetch the metadata then, so the episode title /etc are wrong. The filename becomes the title.

So I need a file naming format that makes both ASS and Youtube Agent happy.

@sourcecodemage
Copy link

I resolved the episode issue with 2 changes

  1. change the youtube type to "youtube2" by changing the channel ID field to [youtube2-id] as per the documentation.
  2. (tricky part) I scripted adding a randomized hour and minute to the "release_date" value from yt-dlp. The value is applied to the video file after copying to the destination folder.
# generate random hour/minute
HH=$(( 1 + $RANDOM % 23))
MM=$((1 + $RANDOM % 60))
# change modified date using touch command 
touch -m -t "${date_string}${HH}${MM}" "${DESTPATH}${FILENAME}"

ASS will process all of the video files include those with identical release dates.

I'm creating a dev environment to see if I come up with a way to have ASS handle identical release dates on it's own. Moving responsibility for this from the end user to the application.

Btw, ASS/Youtube-Agent are great tools. I threw a donation the dev's way and recommend others do the same.

I believe the root causes for the issue I ran into were the missing Plex token, the non-ascii characters, and duplicate release_dates on video files in the same series.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants