modsecurity-crs-docker integration

[TafkaMax]

I am trying to create a superduper all-in-one package, that has:

1. OIDC support as the webserver as the relying party
2. Modsecurity

I have found out that nginx-plus supports oidc, but its nginx-plus.

So I found lots of guides how to use Openresty with its luarocks-oidc plugin.

https://developers.redhat.com/blog/2018/10/08/configuring-nginx-keycloak-oauth-oidc#installation_instructions
https://kevalnagda.github.io/configure-nginx-and-keycloak-to-enable-sso-for-proxied-applications

I am using https://github.com/coreruleset/modsecurity-crs-docker and I have forked it: https://github.com/TafkaMax/modsecurity-crs-docker

I have tried to combine the OpenResty and ModsecCRS docker builds, but in the end I get this weird error after docker build is finished and i launch the container: Error relocating /usr/local/openresty/bin/openresty: SSL_magic_pending_session_ptr: symbol not found

Any thoughts?

[TafkaMax]

coreruleset/modsecurity-crs-docker#117

Adding a similar thing, that I wrote in modsec-crs github page.

If you two could collab it would be awesome.

[neomantra]

I can’t dive deep in that tooling right now, but cool to see it all. Either an earlier compile step or that link phase is not using the SSL installation these images create. So check some ld flags.
For example, this closed issue had that problem:
openresty/openresty#658

[neomantra]

I see another potential issue -- OpenResty uses PCRE 1, not PCRE2. So there might be linking issues related to that which pop up after this is resolved.

[TafkaMax]

I see another potential issue -- OpenResty uses PCRE 1, not PCRE2. So there might be linking issues related to that which pop up after this is resolved.

Do you know, if the lua-nginx-module needs to use PCRE 1 ?

With newer versions coming out, do you think openresty will move to PCRE2?

[neomantra]

See this issue: openresty/lua-nginx-module#1984

That seems to show it is an Nginx version limitation. I think there's gonna be an OpenResty release sometime supporting newer versions -- I don't work on that, but I see there's been dependency upgrade commits lately.

If modsecurity works with older nginx, then it is probably not a problem. I was also wondering if modsecurity worked out of process, in which case the SSL won't be an issue either; just a build bug to figure out.

[TafkaMax]

I did find an Openresty reference in modsecurity documentation: https://github.com/SpiderLabs/ModSecurity/wiki/Compilation-recipes-for-v3.x#nginx-connector-openresty-1

[TafkaMax]

https://forum.openresty.us/d/6481-building-dynamic-modules-for-openresty

[neomantra]

I spent some time understanding this. As you are pointing with those, you need to build the module against the OpenResty tree instead of the nginx tree.

To progress from where you are might be as simple as this diff?

@@ -56,9 +57,9 @@ RUN set -eux; \
 # We use master
 RUN set -eux; \
     git clone -b master --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git; \
-    curl -sSL https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz -o nginx-${NGINX_VERSION}.tar.gz; \
-    tar -xzf nginx-${NGINX_VERSION}.tar.gz; \
-    cd ./nginx-${NGINX_VERSION}; \
+    curl -fSL https://openresty.org/download/openresty-${OPENRESTY_VERSION}.tar.gz -o openresty-${OPENRESTY_VERSION}.tar.gz; \
+    tar xzf openresty-${OPENRESTY_VERSION}.tar.gz; \
+    cd ./openresty-${OPENRESTY_VERSION}; \

I'm building it, but won't be able to check it for a while.

EDIT: fixed to match the style of that Dockerfile

[TafkaMax]

I figured this out aswell. I am also trying to build it in a similar way.

[neomantra]

So that didn't work out well for me.

Another option would be to build it in the build-from-source image process. You would build the Nginx Connector by using environment variables. This issue discusses it #79. You'd want to do like RESTY_CONFIG_OPTIONS_MORE="--add-dynamic-module=../ModSecurity-nginx" and adding the download in other steps. You can also just copy and edit that Dockerfile, rather than using the injection methods.

Then you would use that image in the FROM of the mod security-crs-docker integration, and you can skip this section we patched where the module is being built. What's unique about this is that there's a bunch of scaffolding around the module -- so you can prepare the nginx install for it separately.

[TafkaMax]

https://github.com/TafkaMax/modsecurity-crs-docker/blob/develop/openresty/Dockerfile-alpine#L55

I have pushed the latest testing version here.

As you can see I have

1. Download openresty src
2. download modsec-nginx
3. Set env variables
4. get the build options that the current container version of openresty used
5. build the module against the nginx that is bundled inside the openresty src

Seems straight forward, but my build fails because it cant find LuaJIT. Maybe I added some env variable 'wrong'?

#9 9.551 checking for LuaJIT library in  and ../LuaJIT/LuaJIT-2.1-20220411/src/ (specified by the LUAJIT_LIB and LUAJIT_INC env, with -ldl) ... found
#9 9.618 checking for LuaJIT 2.x ... not found
#9 9.658     ./configure: error: unsupported LuaJIT version; ngx_http_lua_module requires LuaJIT 2.x.

[TafkaMax]

I have gotten it to build. But I seem to be facing the same issue as in the initial post. I will try to look at the build logs and the one similar issue from a few years ago.

[TafkaMax]

#9 2.127 + COMPILEOPTIONS=' --prefix=/usr/local/openresty/nginx --with-cc-opt='"'"'-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include'"'"' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='"'"'-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib'"'"' --with-pcre --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads --with-pcre-jit --with-stream --with-stream_ssl_preread_module'
#9 2.127 + eval ./configure '--prefix=/usr/local/openresty/nginx' '--with-cc-opt='"'"'-O2' -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include '-I/usr/local/openresty/openssl/include'"'" '--add-module=../ngx_devel_kit-0.3.1' '--add-module=../echo-nginx-module-0.62' '--add-module=../xss-nginx-module-0.06' '--add-module=../ngx_coolkit-0.2' '--add-module=../set-misc-nginx-module-0.33' '--add-module=../form-input-nginx-module-0.12' '--add-module=../encrypted-session-nginx-module-0.09' '--add-module=../srcache-nginx-module-0.32' '--add-module=../ngx_lua-0.10.21' '--add-module=../ngx_lua_upstream-0.07' '--add-module=../headers-more-nginx-module-0.33' '--add-module=../array-var-nginx-module-0.05' '--add-module=../memc-nginx-module-0.19' '--add-module=../redis2-nginx-module-0.15' '--add-module=../redis-nginx-module-0.3.9' '--add-module=../rds-json-nginx-module-0.15' '--add-module=../rds-csv-nginx-module-0.09' '--add-module=../ngx_stream_lua-0.0.11' '--with-ld-opt='"'"'-Wl,-rpath,/usr/local/openresty/luajit/lib' -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib '-Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib'"'" --with-pcre --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module '--with-http_geoip_module=dynamic' --with-http_gunzip_module --with-http_gzip_static_module '--with-http_image_filter_module=dynamic' --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module '--with-http_xslt_module=dynamic' --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads --with-pcre-jit --with-stream --with-stream_ssl_preread_module '--add-dynamic-module=../../../ModSecurity-nginx'
#9 2.127 + ./configure '--prefix=/usr/local/openresty/nginx' '--with-cc-opt=-O2 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' '--add-module=../ngx_devel_kit-0.3.1' '--add-module=../echo-nginx-module-0.62' '--add-module=../xss-nginx-module-0.06' '--add-module=../ngx_coolkit-0.2' '--add-module=../set-misc-nginx-module-0.33' '--add-module=../form-input-nginx-module-0.12' '--add-module=../encrypted-session-nginx-module-0.09' '--add-module=../srcache-nginx-module-0.32' '--add-module=../ngx_lua-0.10.21' '--add-module=../ngx_lua_upstream-0.07' '--add-module=../headers-more-nginx-module-0.33' '--add-module=../array-var-nginx-module-0.05' '--add-module=../memc-nginx-module-0.19' '--add-module=../redis2-nginx-module-0.15' '--add-module=../redis-nginx-module-0.3.9' '--add-module=../rds-json-nginx-module-0.15' '--add-module=../rds-csv-nginx-module-0.09' '--add-module=../ngx_stream_lua-0.0.11' '--with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' --with-pcre --with-compat --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module '--with-http_geoip_module=dynamic' --with-http_gunzip_module --with-http_gzip_static_module '--with-http_image_filter_module=dynamic' --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module '--with-http_xslt_module=dynamic' --with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-sha1-asm --with-stream --with-stream_ssl_module --with-threads --with-pcre-jit --with-stream --with-stream_ssl_preread_module '--add-dynamic-module=../../../ModSecurity-nginx'

My build options. that come from the Openresty Dockerfile version of openresty using COMPILEOPTIONS=$(openresty -V 2>&1| grep -i "arguments"|cut -d ":" -f2-); \

EDIT: My --with-ld-opt does not contain /usr/lib

[TafkaMax]

So that didn't work out well for me.

Another option would be to build it in the build-from-source image process. You would build the Nginx Connector by using environment variables. This issue discusses it #79. You'd want to do like RESTY_CONFIG_OPTIONS_MORE="--add-dynamic-module=../ModSecurity-nginx" and adding the download in other steps. You can also just copy and edit that Dockerfile, rather than using the injection methods.

Then you would use that image in the FROM of the mod security-crs-docker integration, and you can skip this section we patched where the module is being built. What's unique about this is that there's a bunch of scaffolding around the module -- so you can prepare the nginx install for it separately.

Hmm... to implement this flow

1. Modify the base Dockerfile to my liking https://github.com/openresty/docker-openresty/blob/master/alpine/Dockerfile
2. Make the fat version use my modified base image https://github.com/openresty/docker-openresty/blob/master/alpine/Dockerfile.fat
3. in the modsecurity Dockerfile use my custom image alpine-fat image and remove the nginx-connector build part.

[TafkaMax]

I guess the bigger problem is that Nginx base image uses docker-entrypoint.sh scripting...

[TafkaMax]

I found the issue. It was similar to the linked issue before.

The modsec-crs uses this ENV variable: https://github.com/coreruleset/modsecurity-crs-docker/blob/develop/nginx/Dockerfile-alpine#L158

I will now try to docker-entrypointify all of this.

[neomantra]

Great progress! And yep, that line would mess things up at runtime for sure! It's telling the dynamic library loader to use a whole different set of libraries.

Using a different entrypoint system should be OK. In the end, the normal installation just runs the nginx binary against a prescribed set of config files. Just to clear away some magic, the openresty binary is the nginx binary:

/ # ls -l /usr/local/openresty/bin/openresty
lrwxrwxrwx    1 root     root            37 Feb 16 12:15 /usr/local/openresty/bin/openresty -> /usr/local/openresty/nginx/sbin/nginx

[TafkaMax]

It seems to be working.

I pushed the modifications to my fork of modsec-docker-crs and added a "new" webserver openresty.

I also modified your openresty setup to use docker-entrypoint like the official Nginx docker image.

I will write a longer post soon.