Mautic 4 stateless container #198

thinkl33t · 2021-05-27T14:58:11Z

This update makes a few changes:

Moves local.php, cache, logs, media and spool into a volume
at /data
Makes sure the above exist and are owned by www-data on startup
Drops mautic into /var/www/html and removes the copy on initial
bringup

Should close at least #2, #90, #184, #203

iBobik · 2021-05-31T02:28:04Z

Tried this but installation was not successful.

I have not filled env vars MAUTIC_URL, MAUTIC_ADMIN_EMAIL, MAUTIC_ADMIN_PASSWORD because it is not mentioned in README.

When run it behaves like I filled them:

2021-05-31T02:26:01.520939806Z 
2021-05-31T02:26:01.520982383Z Ensuring Mautic database is present
2021-05-31T02:26:01.529807392Z 
2021-05-31T02:26:01.529834392Z MySQL Database Created
2021-05-31T02:26:01.537833883Z ========================================================================
2021-05-31T02:26:01.537861930Z This server is now configured to run Mautic!
2021-05-31T02:26:01.537868255Z The following information will be prefilled into the installer (keep password field empty):
2021-05-31T02:26:01.537910546Z Host Name: srv-captain--shared-maria-db
2021-05-31T02:26:01.537915447Z Database Name: mautic
2021-05-31T02:26:01.537918818Z Database Username: mautic
2021-05-31T02:26:01.537922247Z Database Password: xxx
2021-05-31T02:26:01.537925821Z ========================================================================
2021-05-31T02:26:01.537930169Z 
2021-05-31T02:26:01.537933441Z 
2021-05-31T02:26:01.537936722Z ========================================================================
2021-05-31T02:26:01.572933227Z Updating existing mautic config
2021-05-31T02:26:01.580907815Z ========================================================================
2021-05-31T02:26:01.580931061Z 
2021-05-31T02:26:01.580934982Z 
2021-05-31T02:26:01.610898760Z ========================================================================
2021-05-31T02:26:01.610924296Z Mautic not currently installed (no secret_key in local.php)
2021-05-31T02:26:01.610928723Z 
2021-05-31T02:26:01.610932252Z URL & Admin credentials supplied, attempting automated mautic installation.
2021-05-31T02:26:01.610936264Z 
2021-05-31T02:26:01.961278843Z Mautic Install
2021-05-31T02:26:01.961406612Z ==============
2021-05-31T02:26:01.961465415Z 
2021-05-31T02:26:01.961513857Z Parsing options and arguments...
2021-05-31T02:26:01.961689912Z 0 - Checking installation requirements...
2021-05-31T02:26:01.963124404Z Missing optional settings:
2021-05-31T02:26:01.963193814Z   - [0] It is recommended to secure your installation with an SSL certificate (https). Starting February 2020 Google Chrome will stop sending third-party cookies in cross-site requests unless the cookies are secure. Tracking will stop working for Mautic instances running on HTTP. Use HTTPS.
2021-05-31T02:26:01.973076893Z Continue with install anyway? [yes/no]========================================================================
2021-05-31T02:26:01.973161497Z 
2021-05-31T02:26:01.973254359Z 
2021-05-31T02:26:02.291497854Z ========================================================================
2021-05-31T02:26:02.291620632Z Applying any needed database migrations
2021-05-31T02:26:02.291689755Z 
2021-05-31T02:26:02.595702014Z 
2021-05-31T02:26:02.595895462Z [2021-05-31 02:26:02] ++ adding generated column generated_sent_date
2021-05-31T02:26:02.595977942Z [2021-05-31 02:26:02] -> ALTER TABLE email_stats ADD generated_sent_date DATE AS (CONCAT(YEAR(date_sent), '-', LPAD(MONTH(date_sent), 2, '0'), '-', LPAD(DAY(date_sent), 2, '0'))) COMMENT '(DC2Type:generated)';
2021-05-31T02:26:02.595987948Z             ALTER TABLE email_stats ADD INDEX `generated_sent_date_email_id`(generated_sent_date, email_id)
2021-05-31T02:26:02.598631416Z 
2021-05-31T02:26:02.599178109Z In AbstractMySQLDriver.php line 61:
2021-05-31T02:26:02.599246743Z                                                                                
2021-05-31T02:26:02.599322415Z   An exception occurred while executing 'ALTER TABLE email_stats ADD generate  
2021-05-31T02:26:02.599382891Z   d_sent_date DATE AS (CONCAT(YEAR(date_sent), '-', LPAD(MONTH(date_sent), 2,  
2021-05-31T02:26:02.599431637Z    '0'), '-', LPAD(DAY(date_sent), 2, '0'))) COMMENT '(DC2Type:generated)';    
2021-05-31T02:26:02.599504599Z               ALTER TABLE email_stats ADD INDEX `generated_sent_date_email_id  
2021-05-31T02:26:02.599561535Z   `(generated_sent_date, email_id)':                                           
2021-05-31T02:26:02.599609809Z                                                                                
2021-05-31T02:26:02.599681716Z   SQLSTATE[42S02]: Base table or view not found: 1146 Table 'mautic.email_sta  
2021-05-31T02:26:02.599738052Z   ts' doesn't exist                                                            
2021-05-31T02:26:02.599786285Z                                                                                
2021-05-31T02:26:02.599846379Z 
2021-05-31T02:26:02.600032136Z In Exception.php line 18:
2021-05-31T02:26:02.600087224Z                                                                                
2021-05-31T02:26:02.600134784Z   SQLSTATE[42S02]: Base table or view not found: 1146 Table 'mautic.email_sta  
2021-05-31T02:26:02.600217710Z   ts' doesn't exist                                                            
2021-05-31T02:26:02.600268852Z                                                                                
2021-05-31T02:26:02.600310870Z 
2021-05-31T02:26:02.600493447Z In PDOConnection.php line 132:
2021-05-31T02:26:02.600546725Z                                                                                
2021-05-31T02:26:02.600593965Z   SQLSTATE[42S02]: Base table or view not found: 1146 Table 'mautic.email_sta  
2021-05-31T02:26:02.600678256Z   ts' doesn't exist                                                            
2021-05-31T02:26:02.600729267Z                                                                                
2021-05-31T02:26:02.600768600Z 
2021-05-31T02:26:02.600972796Z doctrine:migrations:migrate [--write-sql [WRITE-SQL]] [--dry-run] [--query-time] [--allow-no-migration] [--all-or-nothing [ALL-OR-NOTHING]] [--configuration [CONFIGURATION]] [--db-configuration [DB-CONFIGURATION]] [--db DB] [--em EM] [--shard SHARD] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--] <command> [<version>]
2021-05-31T02:26:02.601032628Z

iBobik · 2021-05-31T02:29:12Z

For anyone who wants to test it, I published this build here: https://hub.docker.com/r/bobik/mautic (v4-apache)

iBobik · 2021-05-31T02:36:39Z

When I filled variables it installed.

Btw, installation should work also without HTTPS because it ofter takes time before Let's Encrypt creates certificate.

iBobik · 2021-05-31T02:41:33Z

There are "PHP options" twice in the README.

An effort has been made in the past to combine any dockerfiles into a central location, but wasn't finalized (i assume due to docker not allowing includes from directories higher up). This completes this work, and documents how to run the Dockerfiles to produce working containers.

This changes the environment variable config so that any environment variable changes are re-applied to local.php on container startup. It is part of some work I am doing to help make the mautic containers more devops friendly.

This commit will automatically perform a command line installation on first start, assuming mautic hasn't already been installed. You need to supply MAUTIC_URL, MAUTIC_ADMIN_EMAIL and MAUTIC_ADMIN_PASSWORD, or these will default to localhost, [email protected] and password. If you are installing on a non-https host, or have another reason why the install might need to be forced, you can set the `MAUTIC_INSTALL_FORCE` environment variable to force this install.

clear cache, apply migrations and grab iplookup database on startup

We currently apply all migrations at startup - this is not needed for some users who are only running small parts of mautic. Add an environment variable that can be set to disable this behaviour.

This update makes a few changes: - Moves `local.php`, `cache`, `logs`, `media` and `spool` into a volume at `/data` - Makes sure the above exist and are owned by www-data on startup - Drops mautic into /var/www/html and removes the copy on initial bringup

thinkl33t · 2021-08-13T10:20:08Z

I'm now happy this is working - i've been running a slightly tweaked version of it (built using gitlab CI) for around month.

Thanks for the review @iBobik:

I have removed the 'default' username / password / url in Mautic 4 container updates #196 and rebased so install doesn't happen every time anymore
I'm pretty sure you can set your MAUTIC_URL to be HTTPS and install will complete fine, your instance just won't be accessible until letsencrypt has generated your certificate :)

thinkl33t · 2021-08-13T10:58:34Z

Added a commit that clears the cache fully on container startup, as mautic's built in cache clear was missing files and causing failures on upgrade from 4.0.0-beta to 4.0.0-rc

luizeof · 2021-08-27T10:39:19Z

@thinkl33t
thank you very much for your work and time.. this is a very interesting resource, but i ask for a little patience so that we can finish the common image of mautic, then we dedicate time to this pull..

user592965 · 2021-09-22T18:05:26Z

@iBobik I just tested your image which works well on my side. There is just the MAUTIC_VERSION env variable in docker-compose that doesn't seem to be taken into account because when I change it to MAUTIC_VERSION=4.0.0 it installs version 4.0.0-rc
Maybe with: ENV MAUTIC_VERSION=${MAUTIC_VERSION:-4.0.0} in the Dockerfile ?

The only thing missing would be the possibility to run Mautic with a slave mysql database in locked state to be able to do failover and high availability on a cluster, currently I get an error message when I run mautic on lock tables.

iBobik · 2021-09-23T09:03:22Z

Hello, I will not continue on this for now, because my project was cancelled, so I will not test it. Honza Pobořil

…

22. 9. 2021 v 20:05, user592965 ***@***.***>: @iBobik I just tested your image which works well on my side. There is just the MAUTIC_VERSION variable that doesn't seem to be taken into account because when I change it to MAUTIC_VERSION=4.0.0 it installs version 4.0.0-rc The only thing missing would be the possibility to run Mautic with a slave mysql database in locked state to be able to do failover and high availability on a cluster, currently I get an error message when I run mautic on lock tables. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

jvlatzko · 2022-01-14T12:15:50Z

Any news on getting this merged? In my tests it worked fine? Codebase divergence is a thing though

dsp76 · 2022-01-18T14:05:11Z

Pretty useful changes, as it looks like. The addressed issues are a main issue currently when deciding to go docker with Mautic or not.

ivanmonteiro · 2022-03-02T19:03:03Z

Hello, I will not continue on this for now, because my project was cancelled, so I will not test it. Honza Pobořil

@iBobik Could you elaborate on which tasks need to be done?
Any broken changes?
This feature seems to be really useful and it would be a shame to see it wasted

iBobik · 2022-03-02T19:14:33Z

Check by fork I deployed into production: https://github.com/ibobik/docker-mautic

I have not tested it extensively, so not sure if everything works, but it seemed ok.

luizeof · 2022-03-02T19:22:06Z

thanks @iBobik ... I will check this pull this week ... many thanks.

kimtiago · 2022-06-17T01:51:37Z

any news about this?

ivanmonteiro · 2022-07-27T21:29:46Z

common/docker-entrypoint.sh

+fi
+
+# clear mautic cache
+rm -rf /data/cache/*


I found a couple of problems here:

It deletes the Maxmind's IP database on startup (/data/cache/ip_data)
It would be better to to keep the IP database between restarts (for my use case)

I can't even mount a volume for the IP database (/data/cache/ip_data)
The container fails to start with the following error message: rm: cannot remove '/data/cache/ip_data': Device or resource busy if I mount a volume to persist the ip database

I can't mount any volumes inside /data/cache/ otherwise the container will not start with the same error above.

By the commit logs I think @thinkl33t run into cache problems when doing version upgrades

Possible solutions:

Delete only /data/cache/prod either running rm -rf /data/cache/prod/* or php bin/console cache:clear

Other possible solution would be to add an environment variable (MAUTIC_CLEAR_CACHE) and only clear the cache if it is true.

RCheesley · 2022-07-28T10:28:05Z

Thanks for the review and testing @ivanmonteiro - much appreciated! @thinkl33t @wget @luizeof could you take a look at the feedback and decide where we want to go with this feature?

luizeof · 2022-07-28T10:59:01Z

@RCheesley I'm currently working on it.

valeu @ivanmonteiro ... muito bom ter brasileiros por aqui!

ivanmonteiro · 2022-07-28T18:59:30Z

@RCheesley I'm currently working on it.

valeu @ivanmonteiro ... muito bom ter brasileiros por aqui!

Apart from the cache delete problem I commented above, I really liked the lean folder structure. With only a couple of folders and configuration files instead of the whole mautic code.

@luizeof Se precisar de ajuda para testar me avisa e farei o possível para ajuda-lo

francoisauclair911 · 2022-11-15T15:56:22Z

Hi guys just checking if this is moving forward and If something else needs to be done (maybe I can help)

cla-bot bot added the cla-signed label May 27, 2021

thinkl33t marked this pull request as draft May 28, 2021 12:47

thinkl33t changed the title ~~Draft: Mautic 4 stateless container~~ Mautic 4 stateless container May 28, 2021

thinkl33t force-pushed the mautic4-volumes branch from 260026e to bb792ba Compare July 5, 2021 08:50

This was referenced Jul 31, 2021

Mautic 4 container updates #196

Merged

FatalThrowableError: "Attempted to load class "OAuthRequestListener" on upgrade #203

Open

thinkl33t added 8 commits August 13, 2021 09:46

Allow additional config to be applied on startup.

c97b64d

This changes the environment variable config so that any environment variable changes are re-applied to local.php on container startup. It is part of some work I am doing to help make the mautic containers more devops friendly.

Add default mysql port 3306

b450107

Initial work to allow for updates by changing out software

f732c0c

clear cache, apply migrations and grab iplookup database on startup

Remove old mautic2 files

a3954da

Add ability to disable automatically running migrations

443fe33

We currently apply all migrations at startup - this is not needed for some users who are only running small parts of mautic. Add an environment variable that can be set to disable this behaviour.

thinkl33t force-pushed the mautic4-volumes branch from bb792ba to e1ecffd Compare August 13, 2021 10:10

thinkl33t marked this pull request as ready for review August 13, 2021 10:11

clear cache manually, as mautic's built in cache cear command isn't

811488b

This was linked to issues Aug 13, 2021

Documentation on how this should be used with volumes #2

Closed

/var/www/html data volume? #90

Open

Mautic need to become stateless in the Docker image #184

Open

FatalThrowableError: "Attempted to load class "OAuthRequestListener" on upgrade #203

Open

RCheesley requested a review from luizeof August 13, 2021 13:45

luizeof mentioned this pull request Sep 22, 2021

images for 3.3.4 and 4.0.0 not yet on hub.docker.com #209

Closed

ivanmonteiro reviewed Jul 27, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mautic 4 stateless container #198

Mautic 4 stateless container #198

thinkl33t commented May 27, 2021 •

edited

Loading

iBobik commented May 31, 2021

iBobik commented May 31, 2021 •

edited

Loading

iBobik commented May 31, 2021

iBobik commented May 31, 2021

thinkl33t commented Aug 13, 2021

thinkl33t commented Aug 13, 2021

luizeof commented Aug 27, 2021

user592965 commented Sep 22, 2021 •

edited

Loading

iBobik commented Sep 23, 2021 via email

jvlatzko commented Jan 14, 2022

dsp76 commented Jan 18, 2022

ivanmonteiro commented Mar 2, 2022 •

edited

Loading

iBobik commented Mar 2, 2022

luizeof commented Mar 2, 2022

kimtiago commented Jun 17, 2022

ivanmonteiro Jul 27, 2022 •

edited

Loading

RCheesley commented Jul 28, 2022

luizeof commented Jul 28, 2022

ivanmonteiro commented Jul 28, 2022

francoisauclair911 commented Nov 15, 2022

Mautic 4 stateless container #198

Are you sure you want to change the base?

Mautic 4 stateless container #198

Conversation

thinkl33t commented May 27, 2021 • edited Loading

iBobik commented May 31, 2021

iBobik commented May 31, 2021 • edited Loading

iBobik commented May 31, 2021

iBobik commented May 31, 2021

thinkl33t commented Aug 13, 2021

thinkl33t commented Aug 13, 2021

luizeof commented Aug 27, 2021

user592965 commented Sep 22, 2021 • edited Loading

iBobik commented Sep 23, 2021 via email

jvlatzko commented Jan 14, 2022

dsp76 commented Jan 18, 2022

ivanmonteiro commented Mar 2, 2022 • edited Loading

iBobik commented Mar 2, 2022

luizeof commented Mar 2, 2022

kimtiago commented Jun 17, 2022

ivanmonteiro Jul 27, 2022 • edited Loading

Choose a reason for hiding this comment

RCheesley commented Jul 28, 2022

luizeof commented Jul 28, 2022

ivanmonteiro commented Jul 28, 2022

francoisauclair911 commented Nov 15, 2022

thinkl33t commented May 27, 2021 •

edited

Loading

iBobik commented May 31, 2021 •

edited

Loading

user592965 commented Sep 22, 2021 •

edited

Loading

ivanmonteiro commented Mar 2, 2022 •

edited

Loading

ivanmonteiro Jul 27, 2022 •

edited

Loading