Error 500 when trying to run a task

[loweehahn]

After upgrading to v2.9.45 from v2.9.37, I'm unable to run any new task. I would get level=error msg="illegal base64 data at input byte 40" error="Cannot write new event to database". Any idea why is this happening to me?

[andreas-marschke]

Can you see if you can get more detailed logs and possibly a stack trace from the logs?
This report is lacking a lot of details:

• Are you using the UI or API?
• Are you using an environment?
• Are you using any special chars in your data?
• What is the task you're attempting to run?

[loweehahn]

Hi @andreas-marschke

I'm not sure how to get more detailed logs. Every time when I try to start a task, that single line of error appears. I forgot to mention that I'm running Semaphore via Docker. Not sure if this additional info helps. I got the logs via docker logs <container name>. Meanwhile, to answer your questions:

• I was using the UI.
• I'm not using any environment.
• Do you mean special character in my playbook? No.
• Every task suddenly failed to run. Even tasks as simple as curl.

[andreas-marschke]

What database do you use? Do you use a Dockerfile to build on the semaphore image or have a configured invocation? Can you share your config.json (with redaction if necessary)?
Do you use any special characters in your username or display name?

[loweehahn]

Hi Andreas,

I use Docker Compose. Below is my file:

version: '3'
services:
mysql:
restart: unless-stopped
ports:
- 3306:3306
image: mysql:8.0
hostname: mysql
volumes:
- semaphore-mysql:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: 1234
MYSQL_DATABASE: semaphore
MYSQL_USER: semaphore_user
MYSQL_PASSWORD: 1234

semaphore:
restart: unless-stopped
ports:
- 3000:3000
image: semaphoreui/semaphore:latest
volumes:
- /root/
environment:
TZ: Asia/Singapore
ANSIBLE_HOST_KEY_CHECKING: 'False'
SEMAPHORE_DB_USER: semaphore_user
SEMAPHORE_DB_PASS: 1234
SEMAPHORE_DB_HOST: mysql
SEMAPHORE_DB_PORT: 3306
SEMAPHORE_DB_DIALECT: mysql
SEMAPHORE_DB: semaphore
SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
SEMAPHORE_ADMIN_PASSWORD: 1234
SEMAPHORE_ADMIN_NAME: admin
SEMAPHORE_ADMIN_EMAIL: [email protected]
SEMAPHORE_ADMIN: admin
SEMAPHORE_ACCESS_KEY_ENCRYPTION: 1234=
SEMAPHORE_LDAP_ACTIVATED: 'no'
SEMAPHORE_LDAP_HOST: dc01.local.example.com
SEMAPHORE_LDAP_PORT: '636'
SEMAPHORE_LDAP_NEEDTLS: 'yes'
SEMAPHORE_LDAP_DN_BIND: 'uid=bind_user,cn=users,cn=accounts,dc=local,dc=shiftsystems,dc=net'
SEMAPHORE_LDAP_PASSWORD: 'ldap_bind_account_password'
SEMAPHORE_LDAP_DN_SEARCH: 'dc=local,dc=example,dc=com'
SEMAPHORE_LDAP_SEARCH_FILTER: "(\u0026(uid=%s)(memberOf=cn=ipausers,cn=groups,cn=accounts,dc=local,dc=example,dc=com))"
depends_on:
- mysql

[fiftin]

Hi @loweehahn

SEMAPHORE_ACCESS_KEY_ENCRYPTION: 1234= is incorrect.

You should use head -c32 /dev/urandom | base64 to generate key.

Docs: https://docs.semui.co/administration-guide/security#database-encryption

[fiftin]

I agree, error message absolutely uninformative.

TODO: change error message for encryption key errors.

[loweehahn]

Hi @fiftin

Below is the output of the container's log after I adopted your suggestion and updated to the latest version:

Attempting to connect to database semaphore on mysql:3306 with user semaphore_user ...
Generating /tmp/semaphore/config.stdin ...
1
mysql:3306
semaphore_user
<password>
semaphore
/tmp/semaphore

no
no
no
no
no
no
/etc/semaphore
admin
[email protected]
admin
<password>

Hello! You will now be guided through a setup to:

1. Set up configuration for a MySQL/MariaDB database
2. Set up a path for your playbooks (auto-created)
3. Run database Migrations
4. Set up initial semaphore user & password

What database to use:
   1 - MySQL
   2 - BoltDB
   3 - PostgreSQL
 (default 1): 
db Hostname (default 127.0.0.1:3306): 
db User (default root): 
db Password: 
db Name (default semaphore): 
Playbook path (default /tmp/semaphore): 
Public URL (optional, example: https://example.com/semaphore): 
Enable email alerts? (yes/no) (default no): 
Enable telegram alerts? (yes/no) (default no): 
Enable slack alerts? (yes/no) (default no): 
Enable Rocket.Chat alerts? (yes/no) (default no): 
Enable Microsoft Team Channel alerts? (yes/no) (default no): 
Enable LDAP authentication? (yes/no) (default no): 
Config output directory (default /home/semaphore): 
Running: mkdir -p /etc/semaphore..
Configuration written to /etc/semaphore/config.json..
 Pinging db..
Running db Migrations..
Executing migration v2.9.46 (at 2024-03-25 16:43:43.45625795 +0800 +08 m=+0.126023237)...
 [2/0]]
Executing migration v2.9.60 (at 2024-03-25 16:43:43.517085643 +0800 +08 m=+0.186850914)...
 [5/0]6]
Executing migration v2.9.61 (at 2024-03-25 16:43:44.095198406 +0800 +08 m=+0.764963693)...
 [9/0]4]
Executing migration v2.9.62 (at 2024-03-25 16:43:44.400630149 +0800 +08 m=+1.070395420)...
 [5/0]4]
Migrations Finished


 > Username:  > Email: 
 Welcome back, Admin! (a user with this username/email is already set up..)

 Re-launch this program pointing to the configuration file

./semaphore server --config /etc/semaphore/config.json

 To run as daemon:

nohup ./semaphore server --config /etc/semaphore/config.json &

 You can login with [email protected] or admin.
Run Semaphore with semaphore server --config /etc/semaphore/config.json
Loading config
Validating config
MySQL semaphore_user@mysql:3306 semaphore
Tmp Path (projects home) /tmp/semaphore
Semaphore v2.9.58
Interface 
Port :3000
Server is running
time="2024-03-25T16:45:02+08:00" level=error msg="illegal base64 data at input byte 4" error="Cannot write new event to database"

The error at the last line is very similar to the one I got before.

Below is my updated Docker Compose file:

semaphore:
    restart: unless-stopped
    ports:
      - 3000:3000
    image: semaphoreui/semaphore:latest
    volumes:
      - /root/
    environment:
      TZ: Asia/Singapore
      ANSIBLE_HOST_KEY_CHECKING: 'False'
      SEMAPHORE_DB_USER: semaphore_user
      SEMAPHORE_DB_PASS: <password>
      SEMAPHORE_DB_HOST: mysql
      SEMAPHORE_DB_PORT: 3306
      SEMAPHORE_DB_DIALECT: mysql
      SEMAPHORE_DB: semaphore
      SEMAPHORE_PLAYBOOK_PATH: /tmp/semaphore/
      SEMAPHORE_ADMIN_PASSWORD: <password>
      SEMAPHORE_ADMIN_NAME: admin
      SEMAPHORE_ADMIN_EMAIL: [email protected]
      SEMAPHORE_ADMIN: admin
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: head -c32 /dev/urandom | base64
      SEMAPHORE_LDAP_ACTIVATED: 'no'
      SEMAPHORE_LDAP_HOST: dc01.local.example.com
      SEMAPHORE_LDAP_PORT: '636'
      SEMAPHORE_LDAP_NEEDTLS: 'yes'
      SEMAPHORE_LDAP_DN_BIND: 'uid=bind_user,cn=users,cn=accounts,dc=local,dc=shiftsystems,dc=net'
      SEMAPHORE_LDAP_PASSWORD: 'ldap_bind_account_password'
      SEMAPHORE_LDAP_DN_SEARCH: 'dc=local,dc=example,dc=com'
      SEMAPHORE_LDAP_SEARCH_FILTER: "(\u0026(uid=%s)(memberOf=cn=ipausers,cn=groups,cn=accounts,dc=local,dc=example,dc=com))"
    depends_on:
      - mysql

[fiftin]

Put result of the command head -c32 /dev/urandom | base64 it docker compose file.

[loweehahn]

Hi @fiftin

I followed your suggestion and got this error instead: level=error msg="cannot decrypt access key, perhaps encryption key was changed" error="Cannot write new event to database"

Then I recalled that actually 1234= was just me masking the actual value. I already had one which is working in v2.9.37. I cloned the server to troubleshoot this issue. Sorry for the confusion.

Anyway when I reverted to the current access key encryption value, I'm getting the same error message as my first post.

[fiftin]

Looks like your secrets are not encrypted. This was not necessary in older versions. Please try following command:

semaphore vault rekey

If it is not helped, you need to update all your secrets.

[loweehahn]

I'm still getting the illegal base64 error:

/etc/semaphore $ semaphore vault rekey --old-key <Value from SEMAPHORE_ACCESS_KEY_ENCRYPTION>
Loading config
Validating config
panic: illegal base64 data at input byte 40

goroutine 1 [running]:
github.com/ansible-semaphore/semaphore/cli/cmd.glob..func15(0x40000f1400?, {0x7b27ac?, 0x4?, 0x7b27b0?})
	/go/src/github.com/ansible-semaphore/semaphore/cli/cmd/vault_rekey.go:26 +0xb0
github.com/spf13/cobra.(*Command).execute(0x1e778e0, {0x40000e9cc0, 0x2, 0x2})
	/go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/spf13/cobra/command.go:987 +0x828
github.com/spf13/cobra.(*Command).ExecuteC(0x1e75380)
	/go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/spf13/cobra/command.go:1115 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/spf13/cobra/command.go:1039
github.com/ansible-semaphore/semaphore/cli/cmd.Execute()
	/go/src/github.com/ansible-semaphore/semaphore/cli/cmd/root.go:37 +0x5c
main.main()
	/go/src/github.com/ansible-semaphore/semaphore/cli/main.go:8 +0x1c

Am I right to use the value from SEMAPHORE_ACCESS_KEY_ENCRYPTION? If yes, how should I update all my secrets?

[fiftin]

1. you should add option access_key_encryption to config file or env var SEMAPHORE_ACCESS_KEY_ENCRYPTION. Value must be result of head -c32 /dev/urandom | base64.

2. If it is possible, check what format has data in the database table access_key column secret. If this data are base64 strings, you need old encryption key and use command semaphore vault rekey --old-key old_key. Otherwise you don't need old encryption key, just use semaphore vault rekey.

3. if you have can't access to DB, just call semaphore vault rekey and check result.

[loweehahn]

I used the following query to determine if the data are base64 strings:

SELECT column_name
FROM your_table
WHERE column_name REGEXP '^[A-Za-z0-9+/]+={0,2}$';

It returned all rows. This means the data is indeed in base64 strings, therefore I need to run semaphore vault rekey --old-key old_key, but in my previous reply, you can see the error I got when running the command.

May I confirm what is the value of the old_key?