Skip to content

Examples to run Hadoop/Spark clusters locally with docker-compose.

Notifications You must be signed in to change notification settings

flokkr/runtime-compose

Repository files navigation

Docker compose based hadoop/spark cluster

This repository contains docker-compose files to demonstrate usage of the flokkr docker images with simple docker-compose files.

To start the containers go to a subdirectory and start the containers with

docker-compose up -d

To scale services

docker-compose scale datanode=1

Attributes

Topic Solution
Configuration management
Source of config files: docker-compose external environment variable file
Configuration preprocessing: envtoconf (Convert environment variables to configuration formats
Automatic restart on config change: Not supported, docker-compose up is required
Provisioning and scheduling
Multihost support NO
Requirements on the hosts docker daemon and docker-compose
Definition of the containers per host N/A, one docker-compose file for the local host
Scheduling (find hosts with available resource) NO, localhost only
Failover on host crash NO
Scale up/down: Easy with docker-compose scale datanode=3
Multi tenancy (multiple cluster) Partial (from multiple checkout directory, after port adjustment)
Network
Network between containers dedicated network per docker-compose file
DNS YES, handled by the docker network
Service discovery NO (DNS based)
Data locality NO
Availability of the ports Published according to the docker-compose files

About

Examples to run Hadoop/Spark clusters locally with docker-compose.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published