add more hadoop versions

hadoop/2.7.1/README.md

@@ -1,87 +1,115 @@
-# based on sequenceiq/hadoop-docker
+# Deploy one Hadoop Cluster with docker
 
-## Create a hadoop cluster in swarm mode
+## Start Master
 
-`--hostname` needs 1.13 or higher
-
-```
+```bash
 docker service create \
---name hadoop-master \
---network swarm-net \
---hostname hadoop-master \
---replicas 1 \
---endpoint-mode dnsrr \
-newnius/hadoop
+  --name hadoop-master \
+  --hostname hadoop-master \
+  --network swarm-net \
+  --replicas 1 \
+  --detach=true \
+  --endpoint-mode dnsrr \
+  --mount type=bind,source=/etc/localtime,target=/etc/localtime \
+  newnius/hadoop:2.7.1
 ```
 
-```
+## Start slaves
+
+```bash
 docker service create \
---name hadoop-slave1 \
---network swarm-net \
---hostname hadoop-slave1 \
---replicas 1 \
---endpoint-mode dnsrr \
-newnius/hadoop
+  --name hadoop-slave1 \
+  --hostname hadoop-slave1 \
+  --network swarm-net \
+  --replicas 1 \
+  --detach=true \
+  --endpoint-mode dnsrr \
+  --mount type=bind,source=/etc/localtime,target=/etc/localtime \
+  newnius/hadoop:2.7.1
 ```
 
-```
+```bash
 docker service create \
---name hadoop-slave2 \
---network swarm-net \
---hostname hadoop-slave2 \
---replicas 1 \
---endpoint-mode dnsrr \
-newnius/hadoop
+  --name hadoop-slave2 \
+  --network swarm-net \
+  --hostname hadoop-slave2 \
+  --replicas 1 \
+  --detach=true \
+  --endpoint-mode dnsrr \
+  --mount type=bind,source=/etc/localtime,target=/etc/localtime \
+  newnius/hadoop:2.7.1
 ```
 
-```
+```bash
 docker service create \
---name hadoop-slave3 \
---network swarm-net \
---hostname hadoop-slave3 \
---replicas 1 \
---endpoint-mode dnsrr \
-newnius/hadoop
+  --name hadoop-slave3 \
+  --hostname hadoop-slave3 \
+  --network swarm-net \
+  --replicas 1 \
+  --detach=true \
+  --endpoint-mode dnsrr \
+  --mount type=bind,source=/etc/localtime,target=/etc/localtime \
+  newnius/hadoop:2.7.1
 ```
 
-## Init && Test
+## Init for the first time
 
-In the first deploy, format dfs first
+#### format dfs first
+Run these commands on the master node.
 
-### stop cluster (in master)
-`sbin/stop-yarn.sh`
-`sbin/stop-dfs.sh`
+```bash
+# stop HDFS services
+sbin/stop-dfs.sh
 
-### remove previous data (in all nodes)
-clear all data in /tmp in all nodes
-
-### format hdfs (in master)
-```
+# format HDFS meta data
 bin/hadoop namenode -format
+
+# restart HDFS services
+sbin/start-dfs.sh
 ```
 
-### start cluster (in master)
-`sbin/start-dfs.sh`
-`sbin/start-yarn.sh`
+## Run a test job
+To make sure youui have successfully setup the Hadoop cluster, just run the floowing commands to see if it is executed well.
 
-### Run a test job
-
-bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
-
-Note: there is no such dir default, you can add data by
-
-```
+```bash
+# prepare input data
 bin/hadoop dfs -mkdir -p /user/root/input
-```
-and
-```
+
+# copy files to input path
 bin/hadoop dfs -put etc/hadoop/* /user/root/input
+
+# submit the job
+bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.4.jar grep input output 'dfs[a-z.]+'
 ```
 
-### monitor cluster in browser
+## Browse the web UI
+You can expose the ports in the script, but I'd rather not since the slaves shoule occupy the same ports.
 
-YARN: hadoop-master:8088
+To access the web UI, deploy another (socks5) proxy to route the traffic.
 
-HDFS: hadoop-master:50070
+If you don't one, try [newnius/docker-proxy](https://hub.docker.com/r/newnius/docker-proxy/), it is rather easy to use.
 
-_Proxy needed, e.g. newnius/docker-proxy_
+Visit [hadoop-master:8088](hadoop-master:8088) fo YARN pages.
+
+Visit [hadoop-master:50070](hadoop-master:50070) fo YARN pages.
+
+## Custom configuration
+
+To persist data or modify the conf files, refer to the following script.
+
+The `/config/hadoop` path is where new conf files to be replaces, you don't have to put all the files.
+
+```bash
+docker service create \
+  --name hadoop-master \
+  --hostname hadoop-master \
+  --network swarm-net \
+  --replicas 1 \
+  --detach=true \
+  --endpoint-mode dnsrr \
+  --mount type=bind,source=/etc/localtime,target=/etc/localtime \
+  --mount type=bind,source=/data/hadoop/config,target=/config/hadoop \
+  --mount type=bind,source=/data/hadoop/hdfs/master,target=/tmp/hadoop-root \
+  --mount type=bind,source=/data/hadoop/logs/master,target=/usr/local/hadoop/logs \
+  newnius/hadoop:2.7.1
+```