newnius/Dockerfiles
1
0
Fork 0
mirror of https://github.com/newnius/Dockerfiles.git synced 2025-12-15 10:26:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

add hadoop

Browse Source
This commit is contained in:
newnius
2017-02-26 15:16:53 +08:00
parent dea8e39309
commit ca6ff3ecac
8 changed files with 301 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
hadoop/2.7.1/README Normal file
View File

@@ -0,0 +1,97 @@
# based on sequenceiq/hadoop-docker
## create a hadoop cluster in swarm mode
`--hostname` need 1.13 or higher
```
docker service create \
--name hadoop-master \
--network swarm-net \
--hostname hadoop-master \
--replicas 1 \
--mount type=bind,source=/mnt/data/hadoop/hdfs/master,target=/tmp/hadoop-root \
--mount type=bind,source=/mnt/data/hadoop/logs/master,target=/usr/local/hadoop/logs \
--mount type=bind,source=/mnt/data/hadoop/config,target=/mnt/hadoop-config \
--endpoint-mode dnsrr \
newnius/hadoop
```
```
docker service create \
--name hadoop-slave1 \
--network swarm-net \
--hostname hadoop-slave1 \
--replicas 1 \
--mount type=bind,source=/mnt/data/hadoop/hdfs/slave1,target=/tmp/hadoop-root \
--mount type=bind,source=/mnt/data/hadoop/logs/slave1,target=/usr/local/hadoop/logs \
--mount type=bind,source=/mnt/data/hadoop/config,target=/mnt/hadoop-config \
--endpoint-mode dnsrr \
newnius/hadoop
```
```
docker service create \
--name hadoop-slave2 \
--network swarm-net \
--hostname hadoop-slave2 \
--replicas 1 \
--mount type=bind,source=/mnt/data/hadoop/hdfs/slave2,target=/tmp/hadoop-root \
--mount type=bind,source=/mnt/data/hadoop/logs/slave2,target=/usr/local/hadoop/logs \
--mount type=bind,source=/mnt/data/hadoop/config,target=/mnt/hadoop-config \
--endpoint-mode dnsrr \
newnius/hadoop
```
```
docker service create \
--name hadoop-slave3 \
--network swarm-net \
--hostname hadoop-slave3 \
--replicas 1 \
--mount type=bind,source=/mnt/data/hadoop/hdfs/slave3,target=/tmp/hadoop-root \
--mount type=bind,source=/mnt/data/hadoop/logs/slave3,target=/usr/local/hadoop/logs \
--mount type=bind,source=/mnt/data/hadoop/config,target=/mnt/hadoop-config \
--endpoint-mode dnsrr \
newnius/hadoop
```
## Init && Test
In the first deploy, format dfs first
### stop cluster (in master)
`sbin/stop-yarn.sh`
`sbin/stop-dfs.sh`
### remove previous data (in all nodes)
clear all data in /tmp in all nodes
### format hdfs (in master)
```
bin/hadoop namenode -format
```
### start cluster (in master)
`sbin/start-dfs.sh`
`sbin/start-yarn.sh`
### Run a test job
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
Note: there is no such dir default, you can add data by
```
bin/hadoop dfs -mkdir -p /user/root/input
```
and
```
bin/hadoop dfs -put etc/hadoop/* /user/root/input
```
### monitor cluster in browser
YARN: hadoop-master:8088
HDFS: hadoop-master:50070