Default database variable names in .env file

[maikschneider]

Hey,

First off, very good work! I really enjoy switching from my custom deployment scripts to your deployer configuration.

Since it's possible to use the environment variables in AdditionalConfiguration.php, it would be nice to have a consistent syntax for variables for additional configuration. I would prefer TYPO3_CONF_VARS__DB__Connections__Default__dbname instead of the hardcoded TYPO3__DB__Connections__Default__dbname. This way, you can use this AdditionalConfiguration as shared and configure all other TYPO3 settings in the same style.
Setting new db_databases with the other prefix would override the cool db_default values. What do you think about it?

[kszymukowicz]

First. Nice that you like deployer-extended-*

About your idea. I am not sure :)

My "hardcoded" version is something that was used by Helmut Humel a few years ago. So at this time you could say it was some "standard". Of course I have also automatic "reader" into TYPO3_CONF_VARS global var similar to the gist you sent.

So what's more the gist you sent is that it allows you to choose what other "namespaces" can populate despite the standard  TYPO3_CONF_VARS. But can you give me some examples when you want to overwrite something other than TYPO3_CONF_VARS?

The second thing is that I found this way of overwriting vars from .env has more cons than pros.
The Pros is of course that it's fast.

The cons are:

1. When you upgrade TYPO3 the vars names can change. It was just the case for databases and also GFX settings. It brings additional effort.

2. It's harder to make a nice deployment when the application is working on some values which are out of git and it must be changed during deployment. You would need to change it manually during deployment and yet remember that you need to change it. If there are more developers and one is doing a feature and the other is deploying then it's 99% that the deploying developer will not know/remember that he should change something in the .env file. The better way here is to have separate PHP files that are in git and are read depending on the instance. Then you can put the values which differ on instances directly into git! The golden rule is to have as little outside git as possible

3. Taking point 2. If you will implement reading PHP instance based files (which are in git) then the only vars in .env should be:
     a) passwords (database / smtp / encryptionKey)
     b) the instance name (based on which you will read proper php file)

Then also \SourceBroker\DeployerExtendedTypo3\Drivers\Typo3EnvDriver would need to be changed because it no longer can have all database data from the .env file. I was thinking here to use typo3cms configuration:show DB.

So if I was to change this part today:

1. I would totally give the responsibility to get database data to TYPO3 (by using typo3cms configuration:show DB)
2. I would store only the instance name and secrets in .env file.

[maikschneider]

Thanks for you explanation.

I wouldn't override other settings than TYPO3_CONF_VARS. I totally agree with you that there should be as less as possible entries in the .env file, because of the reasons you mentioned.
This is my .env in production:

TYPO3_CONTEXT='Production'
INSTANCE='production'

TYPO3_CONF_VARS__DB__Connections__Default__dbname='usr_xxxxx'
TYPO3_CONF_VARS__DB__Connections__Default__host='dbxx.xxx.com'
TYPO3_CONF_VARS__DB__Connections__Default__password='xxxxxxx'
TYPO3_CONF_VARS__DB__Connections__Default__port='xxxx'
TYPO3_CONF_VARS__DB__Connections__Default__user='xxxxxx'

# Copy configuration for deployer
TYPO3__DB__Connections__Default__dbname=$TYPO3_CONF_VARS__DB__Connections__Default__dbname
TYPO3__DB__Connections__Default__host=$TYPO3_CONF_VARS__DB__Connections__Default__host
TYPO3__DB__Connections__Default__password=$TYPO3_CONF_VARS__DB__Connections__Default__password
TYPO3__DB__Connections__Default__port=$TYPO3_CONF_VARS__DB__Connections__Default__port
TYPO3__DB__Connections__Default__user=$TYPO3_CONF_VARS__DB__Connections__Default__user

My local .env file looks like this and is inside git:

TYPO3_CONTEXT='Development/Local'
INSTANCE='local'

TYPO3_CONF_VARS__DB__Connections__Default__dbname='db'
TYPO3_CONF_VARS__DB__Connections__Default__host='db'
TYPO3_CONF_VARS__DB__Connections__Default__password='db'
TYPO3_CONF_VARS__DB__Connections__Default__port='3306'
TYPO3_CONF_VARS__DB__Connections__Default__user='db'

TYPO3_CONF_VARS__BE__debug='true'
TYPO3_CONF_VARS__BE__compressionLevel=0

TYPO3_CONF_VARS__FE__debug='true'
TYPO3_CONF_VARS__FE__compressionLevel=0
TYPO3_CONF_VARS__FE__processor_path='/usr/bin'
TYPO3_CONF_VARS__FE__processor_path_lzw='/usr/bin'

TYPO3_CONF_VARS__MAIL__transport='smtp'
TYPO3_CONF_VARS__MAIL__transport_smtp_server='localhost:1025'
TYPO3_CONF_VARS__MAIL__transport_sendmail_command='/usr/local/bin/mailhog sendmail [email protected] --smtp-addr 127.0.0.1:1025'

TYPO3_CONF_VARS__GFX__processor='/usr/bin/'
TYPO3_CONF_VARS__GFX__processor='/usr/bin/'

TYPO3_CONF_VARS__SYS__trustedHostsPattern='.*.*'
TYPO3_CONF_VARS__SYS__devIPmask='*'
TYPO3_CONF_VARS__SYS__displayErrors=1
TYPO3_CONF_VARS__SYS__exceptionalErrors=12290
TYPO3_CONF_VARS__SYS__sitename='SiteXY DDEV'

The LocalConfiguration.php is optimized for Production and inside git, too. If a team member does an upgrade or change to TYPO3_CONF_VARS, no manual adjustments are necessary. You see, that I do the development overrides only via environment variables. I guess, you have a custom AdditionalConfiguration.php? I replaced the one created by ddev with the one from the gist to have the file inside git as well as on every host.

My suggestion would be to make both configurations possible. Maybe something like that inside Typo3EnvDriver:

    foreach ($dbMappingFields as $key => $dbMappingField) {
        $envSetting = $env->get($dbMappingField) ?? $env->get(str_replace('TYPO3__', 'TYPO3_CONF_VARS__', $dbMappingField));
        $dbSettings[$key] = $envSetting;
    }

It would be just a minimal advantage for my setup, since I wouldn't need to write the config twice inside the remote env files.

[kszymukowicz]

Ok.
And how your .env looks like for staging instance? Its not so clean as for production and you have it out of git, right ?

[maikschneider]

Yes, it's out of git. But it looks as clean as the production .env - there is just one additional MAIL override to use an external mailhog. I want my staging to behave like production - there is not really a need for adjustments. Also most of the differences are handled via ApplicationContext.

[kszymukowicz]

@maikschneider
If you want you can test branch t311 (ddev composer req sourcebroker/deployer-extended-typo3:dev-t311)
Its called t311 but it should work also on other TYPO3 versions.

What's new in this branch is that it use ./vendor/bin/typo3cms configuration:showactive DB to read the database config for dep:db commands.

So you can remove:

# Copy configuration for deployer
TYPO3__DB__Connections__Default__dbname=$TYPO3_CONF_VARS__DB__Connections__Default__dbname
TYPO3__DB__Connections__Default__host=$TYPO3_CONF_VARS__DB__Connections__Default__host
TYPO3__DB__Connections__Default__password=$TYPO3_CONF_VARS__DB__Connections__Default__password
TYPO3__DB__Connections__Default__port=$TYPO3_CONF_VARS__DB__Connections__Default__port
TYPO3__DB__Connections__Default__user=$TYPO3_CONF_VARS__DB__Connections__Default__user

from your config.

Remember that to activate new driver you need to set:

set('driver_typo3cms', true);

I tested it on TYPO3 10 and TYPO3 11 and its working ok. (deploy and database pull/copy)

[kszymukowicz]

or wait a bit - there is a bug ;)

[kszymukowicz]

looks like all is working OK now
plz test

[kszymukowicz]

New driver based on typo3cms command ./vendor/bin/typo3cms configuration:showactive DB --json has been added in version 16.1.0.

To activate it use:
set('driver_typo3cms', true);

This way deployer-extended-typo3 no longer force you to use specific way for setting database creds so your problem is solved.