Info

Build Hadoop cluster in OpenStack with Terraform.

Primary goal of this project is to build Hadoop cluster. But the most part is generic - Hadoop deployment can be replaced by implementing different deployment type (providing user-data and optionally implementing plugin for orchestration).

Requirements

Locally installed:

• Terraform
• Ansible

Hadoop image

To setup Hadoop on single machine, launch:

/usr/local/sbin/hadoop-setup.sh

Hadoop image can be used also to build Hadoop cluster. It contains pre-downloaded and pre-installed Hadoop packages and dependencies, so this will speed things up.

Cluster

Build cluster:

1. check variables.tf

2. launch setup script

   ./launch.sh

The launch.sh script is doing this:

./terraform apply
./terraform output -json > config.json
./orchestrate.py

The orchestration script has multiple steps and dry-run option. See ./orchestrate.py --help.