@@ -9,18 +9,17 @@ Primary goal of this project is to build Hadoop cluster. But the most part is ge
...
@@ -9,18 +9,17 @@ Primary goal of this project is to build Hadoop cluster. But the most part is ge
Locally installed:
Locally installed:
*[Terraform](https://www.terraform.io/)
*[Terraform](https://www.terraform.io/)
*[Ansible](https://www.ansible.com/)
Configuration:
Configuration:
* public part of ssh key uploaded to OpenStack
* public part of ssh key uploaded to OpenStack
* ssh-agent with ssh key
* ssh-agent with ssh key
* configured access to OpenStack (for example downloaded *cloud.yaml* file or the environment set)
* configured access to OpenStack, see [Cloud Documentation](https://docs.cloud.muni.cz/cloud/cli/#getting-credentials)(either downloaded *cloud.yaml* file or the environment set)
* floating IP created
* floating IP created
# Hadoop image
# Hadoop image
To setup Hadoop on single machine, launch:
To setup Hadoop on single machine using Hadoop image, launch:
/usr/local/sbin/hadoop-setup.sh
/usr/local/sbin/hadoop-setup.sh
...
@@ -40,6 +39,7 @@ For example (check also the other values used in *variables.tf*):
...
@@ -40,6 +39,7 @@ For example (check also the other values used in *variables.tf*):
flavor = "standard.large" # >4GB memory needed
flavor = "standard.large" # >4GB memory needed
EOF
EOF
terraform init
terraform apply
terraform apply
# Build cluster
# Build cluster
...
@@ -62,6 +62,7 @@ For example (check also the other values used in *variables.tf*):
...
@@ -62,6 +62,7 @@ For example (check also the other values used in *variables.tf*):
#
#
# 2. launch the setup
# 2. launch the setup
#
#
terraform init
terraform apply
terraform apply
# Destroy cluster
# Destroy cluster
...
@@ -94,7 +95,7 @@ The public IP is in the *public_hosts* file or *inventory* file.
...
@@ -94,7 +95,7 @@ The public IP is in the *public_hosts* file or *inventory* file.
On the terraform client machine:
On the terraform client machine:
# decrease number of nodes in terraform
# increase number of nodes in terraform
vim *.auto.tfvars
vim *.auto.tfvars
# check the output
# check the output
...
@@ -115,7 +116,7 @@ Data must be migrated from the removed nodes first in the Hadoop cluster. Theore
...
@@ -115,7 +116,7 @@ Data must be migrated from the removed nodes first in the Hadoop cluster. Theore
On the master machine:
On the master machine:
# add nodes to remove (it must be the nodes with the highest numbers), for example:
# nodes to remove (it must be the nodes with the highest numbers), for example:
echo node3.terra >> /etc/hadoop/conf/excludes
echo node3.terra >> /etc/hadoop/conf/excludes
# refresh configuration
# refresh configuration
...
@@ -151,8 +152,10 @@ On the master machine:
...
@@ -151,8 +152,10 @@ On the master machine:
Launch */usr/local/sbin/hadoop-adduser.sh USER_NAME* in the whole cluster.
Launch */usr/local/sbin/hadoop-adduser.sh USER_NAME* in the whole cluster.
For example using Ansible (replace *$USER\_NAME* by the user name):
For example using Ansible from the master machine (replace *$USER\_NAME* by the new user name):
sudo su -l deployadm
cd ~/terraform
ansible -i ./inventory -m command -a "/usr/local/sbin/hadoop-adduser.sh $USER_NAME" all
ansible -i ./inventory -m command -a "/usr/local/sbin/hadoop-adduser.sh $USER_NAME" all
The generated password is written on the output and stored in the home directory.
The generated password is written on the output and stored in the home directory.