Include Production WSGI (Gunicorn) to replace Flask Default Server #263

KaiyiLiu1234 · 2024-06-13T22:45:11Z

To Address: #259

A side effect of using Gunicorn is that it will replace the address with default address and default port (8000). This can be resolved by binding (ex. binding model_server to 0.0.0.0:8100). Binding with 0.0.0.0 is acceptable but according to Flask, it is better to also introduce nginx server to act as a reverse proxy. Will this be necessary to include?

Also, since model_server AND offline_trainer (and in future online_trainer) expects to use their own Flask Apps, supervisord is needed to run multiple CMD for model_server and offline_trainer in Dockerfile.

Currently, this changes here works for metal docker compose so long as python3.8 -u src/server/model_server.py is replaced by gunicorn -b -0.0.0.0.8100 -src.server.model_server:app. However, there is more functionality that can be introduced depending on what is acceptable or not.

Add Gunicorn to Dockerfile Signed-off-by: Kaiyi <[email protected]>

sthaha · 2024-06-14T01:11:14Z

@KaiyiLiu1234 https://developers.redhat.com/articles/2023/08/17/how-deploy-flask-application-python-gunicorn#containerization is good reference to follow

Binding with 0.0.0.0 is acceptable but according to Flask, it is better to also introduce nginx server to act as a reverse proxy. Will this be necessary to include?

I am not sure if the recommendation applies to containers / k8s.

Gunicorn should not be run as root because it would cause your application code to run as root, which is not secure. However, this means it will not be possible to bind to port 80 or 443. Instead, a reverse proxy such as nginx or Apache httpd should be used in front of Gunicorn.

The part applies to containers is don't run as root. However, binding to port 80/443 isn't needed for containers.

You can bind to all external IPs on a non-privileged port using the -b 0.0.0.0 option. Don’t do this when using a reverse proxy setup, otherwise it will be possible to bypass the proxy.

Again, this is fine for a containerized env. I feel following the article linked should be good enough to fix our issue.

Included Gunicorn Instances for Model Server, Online Trainer (inactive), and Offline Trainer. All three of these applications make use of one Flask App, so three Gunicorn Instances should be set up to manage them all. Signed-off-by: Kaiyi <[email protected]>

Incorporate Gunicorn Server setup for Model Server and Offline Trainer. The setup is currently placed in a dockerfile test as changes need to be made in using the Kepler Model Server image with gunicorn rather than with python. Signed-off-by: Kaiyi <[email protected]>

KaiyiLiu1234 · 2024-06-26T02:51:10Z

@sthaha @sunya-ch I incorporated the Gunicorn changes into a test dockerfile for now. How we interact with the image will change now in docker composes and kubernetes resources (for instance command should not be python model_server.py - it should instead use gunicorn or no command as the CMD for the dockerfile can handle launching the gunicorn servers for model server, offline trainer and online trainer). I can also include in the makefile commands to run the new gunicorn test dockerfile in this PR or in a future PR?

Added Makefile Tests which will easily build, run, and clean Model Server with Gunicorn Servers for Model Server and Offline Trainer. Signed-off-by: Kaiyi <[email protected]>

sthaha · 2024-06-26T23:07:22Z

Makefile

@@ -51,27 +55,42 @@ run-model-server:
 $(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8100:8100 --name model-server $(TEST_IMAGE) /bin/bash -c "python3.8 tests/http_server.py & sleep 10 && python3.8 src/server/model_server.py"
 while ! docker logs model-server | grep -q Serving; do echo "waiting for model-server to serve"; sleep 5; done

+run-model-server-gunicorn-complete:


Suggested change

run-model-server-gunicorn-complete:

run-model-server-prod:

sthaha · 2024-06-26T23:08:19Z

Makefile

@@ -51,27 +55,42 @@ run-model-server:
 $(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8100:8100 --name model-server $(TEST_IMAGE) /bin/bash -c "python3.8 tests/http_server.py & sleep 10 && python3.8 src/server/model_server.py"
 while ! docker logs model-server | grep -q Serving; do echo "waiting for model-server to serve"; sleep 5; done

+run-model-server-gunicorn-complete:
+ $(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8105:8105 -p 9109:9109 --name model-server-gunicorn-complete $(GUNICORN_TEST_IMAGE)


Suggested change

$(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8105:8105 -p 9109:9109 --name model-server-gunicorn-complete $(GUNICORN_TEST_IMAGE)

$(CTR_CMD) run -d \

--platform linux/amd64 \

-e "MODEL_TOPURL=http://localhost:8110" \

-v ${MODEL_PATH}:/mnt/models \

-p 8105:8105 -p 9109:9109 \

--name model-server-prod \

$(GUNICORN_TEST_IMAGE)

sthaha · 2024-06-26T23:09:40Z

cmd/base_gunicorn_config.py

+workers = int(os.environ.get('GUNICORN_PROCESSES', '2'))
+
+threads = int(os.environ.get('GUNICORN_THREADS', '4'))
+
+port = os.environ.get('GUNICORN_PORT', '8100')
+
+bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port)


Suggested change

workers = int(os.environ.get('GUNICORN_PROCESSES', '2'))

threads = int(os.environ.get('GUNICORN_THREADS', '4'))

port = os.environ.get('GUNICORN_PORT', '8100')

bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port)

workers = int(os.environ.get('GUNICORN_PROCESSES', '2'))

threads = int(os.environ.get('GUNICORN_THREADS', '4'))

port = os.environ.get('GUNICORN_PORT', '8100')

bind = os.environ.get('GUNICORN_BIND', '0.0.0.0:' + port)

sthaha · 2024-06-26T23:10:54Z

cmd/base_gunicorn_config.py

how about we call it gunicorn_config.py ?

lets keep config under a config dir

sthaha · 2024-06-27T02:10:11Z

dockerfiles/Dockerfile.test-gunicorn

+# ENTRYPOINT ["python3.8", "cmd/main.py"]
+#ENTRYPOINT [ "python3.8", "-u", "src/server/model_server.py" ]


Suggested change

# ENTRYPOINT ["python3.8", "cmd/main.py"]

#ENTRYPOINT [ "python3.8", "-u", "src/server/model_server.py" ]

sthaha · 2024-06-27T02:10:32Z

dockerfiles/Dockerfile.test-gunicorn

+COPY src/estimate src/estimate
+COPY src/server src/server
+COPY src/train src/train
+COPY src/util src/util
+COPY cmd cmd


why not copy the entire src and cmd ?

I think it is my bad that when we run the test the built ./model will be inside the src folder. We need to fix that first then the src will be cleaned enough for entire copy.

I can check it out.

sthaha · 2024-06-27T02:10:59Z

dockerfiles/Dockerfile

@@ -18,4 +18,5 @@ EXPOSE 8101
 # port for Offline Trainer
 EXPOSE 8102

+


unrelated change?

Fix code errors requested by sunil. Signed-off-by: Kaiyi <[email protected]>

sthaha · 2024-07-06T01:18:20Z

cmd/gunicorn.sh

+echo "Starting Model Server"
+export GUNICORN_PORT=$PORT_MODEL_SERVER
+gunicorn --config config/gunicorn_config.py src.server.model_server:app &
+echo "Starting Offline Trainer"
+export GUNICORN_PORT=$PORT_OFFLINE_TRAINER
+gunicorn --config config/gunicorn_config.py src.train.offline_trainer:app &
+wait


Can we create separate containers for model-server and trainer?
This removes the issue of handling (unexpected) termination.

sthaha · 2024-07-06T01:19:31Z

dockerfiles/Dockerfile.test-gunicorn

+# port for Offline Trainer
+EXPOSE ${PORT_OFFLINE_TRAINER}
+
+CMD ["sh", "cmd/gunicorn.sh"]


Is the idea to replace current Dockerfile with this?

Separated Gunicorn Offline Trainer and Model Server into separate containers. Signed-off-by: Kaiyi <[email protected]>

WIP Include Production WSGI to replace Flask Default Server

57e478b

Add Gunicorn to Dockerfile Signed-off-by: Kaiyi <[email protected]>

KaiyiLiu1234 requested a review from sunya-ch June 13, 2024 22:45

KaiyiLiu1234 added 2 commits June 20, 2024 15:44

KaiyiLiu1234 changed the title ~~WIP Include Production WSGI (Gunicorn) to replace Flask Default Server~~ Include Production WSGI (Gunicorn) to replace Flask Default Server Jun 26, 2024

KaiyiLiu1234 requested a review from sthaha June 26, 2024 13:03

Add Makefile Tests for Gunicorn Server Tests

c784c1c

Added Makefile Tests which will easily build, run, and clean Model Server with Gunicorn Servers for Model Server and Offline Trainer. Signed-off-by: Kaiyi <[email protected]>

sthaha reviewed Jun 26, 2024

View reviewed changes

sthaha reviewed Jun 27, 2024

View reviewed changes

dockerfiles/Dockerfile

@@ -18,4 +18,5 @@ EXPOSE 8101

# port for Offline Trainer

EXPOSE 8102

Copy link

Contributor

sthaha Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated change?

Fix code errors

f945cf9

Fix code errors requested by sunil. Signed-off-by: Kaiyi <[email protected]>

KaiyiLiu1234 requested a review from sthaha July 6, 2024 01:17

sthaha reviewed Jul 6, 2024

View reviewed changes

KaiyiLiu1234 changed the title ~~Include Production WSGI (Gunicorn) to replace Flask Default Server~~ WIP:Include Production WSGI (Gunicorn) to replace Flask Default Server Jul 13, 2024

feat(dockerfile) Separate Gunicorn Trainer and Model Server

b77d722

Separated Gunicorn Offline Trainer and Model Server into separate containers. Signed-off-by: Kaiyi <[email protected]>

KaiyiLiu1234 changed the title ~~WIP:Include Production WSGI (Gunicorn) to replace Flask Default Server~~ Include Production WSGI (Gunicorn) to replace Flask Default Server Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include Production WSGI (Gunicorn) to replace Flask Default Server #263

Include Production WSGI (Gunicorn) to replace Flask Default Server #263

KaiyiLiu1234 commented Jun 13, 2024

sthaha commented Jun 14, 2024

KaiyiLiu1234 commented Jun 26, 2024

sthaha Jun 26, 2024

sthaha Jun 26, 2024

sthaha Jun 26, 2024

sthaha Jun 26, 2024

sthaha Jun 26, 2024

sthaha Jun 27, 2024

sthaha Jun 27, 2024

sunya-ch Jun 27, 2024

KaiyiLiu1234 Jun 27, 2024

sthaha Jun 27, 2024

sthaha Jul 6, 2024

sthaha Jul 6, 2024 •

edited

Loading

- $(CTR_CMD) run -d --platform linux/amd64 -e "MODEL_TOPURL=http://localhost:8110" -v ${MODEL_PATH}:/mnt/models -p 8105:8105 -p 9109:9109 --name model-server-gunicorn-complete $(GUNICORN_TEST_IMAGE)
+ $(CTR_CMD) run -d \
+ --platform linux/amd64 \
+ -e "MODEL_TOPURL=http://localhost:8110" \
+ -v ${MODEL_PATH}:/mnt/models \
+ -p 8105:8105 -p 9109:9109 \
+ --name model-server-prod \
+ $(GUNICORN_TEST_IMAGE)

		# ENTRYPOINT ["python3.8", "cmd/main.py"]
		#ENTRYPOINT [ "python3.8", "-u", "src/server/model_server.py" ]

		@@ -18,4 +18,5 @@ EXPOSE 8101
		# port for Offline Trainer
		EXPOSE 8102

Include Production WSGI (Gunicorn) to replace Flask Default Server #263

Are you sure you want to change the base?

Include Production WSGI (Gunicorn) to replace Flask Default Server #263

Conversation

KaiyiLiu1234 commented Jun 13, 2024

sthaha commented Jun 14, 2024

KaiyiLiu1234 commented Jun 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sthaha Jul 6, 2024 • edited Loading

Choose a reason for hiding this comment

sthaha Jul 6, 2024 •

edited

Loading