Field Report - Fedora 41 / Root-less Podman

[antler5]

So, first I ran into something the looked like containers/podman#12715. It's fairly widely reported across the internet, but I was seeing huge ids, like 3495090557:18285364, and had a hard time figuring out what my system's maximum was -- whatever I set by subids, I always got either the same error (with new numbers) or overflow errors upon restarting podman. I starting raising my subids up to what was requested, only to get more errors with higher ids, and it had to download 800mb each time.... but seeing no other option, I wrote up a bash script that raised my subids to what was requested, on a loop, until it worked. I needed up to 4286438922.

Then I ran into bad parameter: Link is not supported, so I removed the link blocks and renamed all the containers in docker-compose.yml.

At this point the containers finally spun up, but I got an error from the mysql container about some issue with creating it's unix socket:

Can't start server : Bind on unix socket: Value too large for defined data type

There are some mentions of similar errors online, but nothing matching my situation (unix socket, btrfs-backed overlays, no bind mount on the socket-dir). I added a volume for the socket-dir and the error went away (but may be responsible for present issues).

At this point I could access web ui's! (even from other local machines.) Great! Took a diversion to build a custom client for Guix (my other machine), tried to sign in with admin/admin (not from localhost) -- swear I saw Host '10.?.?.?' is not allowed to connect to this MariaDB server in the logs at this point but haven't been able to replicate it -- then tried to run create-user.sh. I had to remove the sudo's (rootless), and got the same same error. This time I know the IP was 10.5.5.8, which makes sense, that's the dataserver. Came back to the docs, pre- and post- fork, and tried bin/init.sh -- same DB allow-list errors.

# From the other end, probably when trying to sign in (rather then running init or create_user)
mysql-1                | 2024-11-05 18:28:38 3 [Warning] Aborted connection 3 to db:
 'unconnected' user: 'unauthenticated' host: '10.5.5.8' (This connection closed norm
ally without authentication)

And that's where I'm at right now! Usually when I give reports like this I work all the way through first, clean them up, and take the time to polish my prose for both upstream and posterity, but my SO is urging me to wrap it up and I really wanted to get this out in case someone more familiar with the platform can help me solve my remaining issues (so, uh, sorry about the rambling structure >u<). I think it's obvious that I'm glad to do some leg-work, but a trained eye might know what's "not right", and any context or even pointers to what logs / config to be looking at would be helpful.

I have a few Q's, like what my user's library id should be (when I get there), and whether this is expected:

zotprime-dataserver-1  | AH00558: httpd: Could not reliably determine the server's f
ully qualified domain name, using 10.5.5.8. Set the 'ServerName' directive globally
to suppress this message

Oh, while looking through the logs for anything I want to mention (too much to share, literally a day's worth of scrollback, maybe after a reboot and clean-slate-ing tomorrow), and found this:

elasticsearch-1        | Exception in thread "main" java.nio.file.FileSystemExceptio
n: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp: Value too large for d
efined data type

Bet I can take there of that with a volume at that location tomorrow.
Don't think the DB auth issues will disappear when I'm done.

Thanks for making this possible to run!

oh and here are the changes i've made

[uniuuu]

Hi @antler5
Thank you for the report.
I've never used podman but docker CE. In their guide you would delete any other docker alternative provided by distro before installing CE Install Docker Engine on Fedora
This will resolve all the issues you mentioned as I'm running Fedora 41 too.

I'd recommend to use MicroK8s instead of docker compose to get all flexibility it gives.

Do remove containers and volumes with docker compose down -v, remove repository and clone it again as per Dependencies and source code

Oh, while looking through the logs for anything I want to mention (too much to share, literally a day's worth of scrollback, maybe after a reboot and clean-slate-ing tomorrow), and found this:

For sure debugging can be off once all settled.
You may stop http debug in stack/dataserver/entrypoint.sh comment out 78 and uncomment 79 line respectively.

Please let me know if it was helpful.
Let's continue it under discussion.

[uniuuu]

Created discussion here.

[uniuuu]

Hi @antler5
Good day.
Please let me know if the issue has been resolved.

[antler5]

oh, well, "use docker" isn't really a solution but you're welcome to close this as out-of-scope and i'll happily keep hacking it together myself when i get back to it

[uniuuu]

Hi @antler5
Yes, initially it's out of scope as focus was and is to move ZotPrime production to Kubernetes. So I just got it working on Docker CE. However for production can't recommend it due to lacking of features K8s provides.

I might look into Podman but cannot promise to do it soon. I'll look into groups support followed by ARM support first.