How does one configure the execution backend for maptasks

[palchicz]

Hi Flyte crew,

I was wondering if I could get some insight into the backend execution of maptasks and how configurable that is. My current understanding is that maptasks use a container_array type node as opposed to a python-task type node, and that seems to impact how they are run. As far as I can tell, python-task nodes look like they are run on a k8 pod whereas I'm not entirely sure how container_array nodes get executed (it doesn't seem like a k8 pod though). I also see that one can run maptasks on aws batch, but it is unclear from the docs how one sets that up.

So I guess my questions are:

1. What is the default execution backend for maptasks.
2. Can it be changed?
3. What can it be changed to (e.g. k8 or aws batch) and how?

Thanks ahead of time for your response!

[EngHabu]

Hi @palchicz,

As you observed, there are two plugins to run maptasks that ship as part of flyteplugins; k8s array and aws array.

Which plugin to use is controlled through this config

1. By default, as you can see in sandbox, it uses k8s-array plugin. The way this executes array tasks is by launching a pod for every instance in the array. It's a simple and straightforward implementation but has some scale/performance drawbacks that we have some open issues to address (issue) but the big benefit it, it works out of the box.
2. You can implement your own plugin that handles that task type and configure the system to use it instead.
3. k8s-array and aws_array are the names of these two plugins. If you change the config, flyte will attempt to use that one to launch array jobs.

The AWS Array Plugin needs some additional configuration to make work which we are in the process of documenting as well. On top of my head, they are:

• You need to configure the AWS Role flytepropeller assumes to be able to PassRole all the roles you will use to run jobs in AWS Batch. That requires setting up Trust Relationship
• All workflows/launchplans need to be configured with an assumable_iam_role (example) that flyte will pass to AWS Batch Service to run the jobs as.
• You need to configure AWS Batch Queues and configure FlyteAdmin (example config) to choose which projects/domains/workflows go to which AWS Batch Queus. I strongly recommend, at the very least, separating your development and production domains into separate queues. There is a limitation of the number of queues in AWS Batch so you can't create a queue for every project/domain combination.

cc @migueltol22 as you are going through that process at the moment in case I missed anything?
cc @katrogan, @anandswaminathan if you can correct me or augment my answer if I missed anything?

[kumare3]

@palchicz please do and also, join the slack channel where we can help faster

[EngHabu]

@palchicz We have now published docs on how to setup AWS Batch integration. Would love if you give it a shot and let us know...

[SandraGH5]

@EngHabu do we need more documentation for this issue?

[EngHabu]

This should be all we need.. until customers try it out and give us feedback :-)

[SandraGH5]

Sounds good 👍🏻