Testing multi machine hadoop cluster

 So I was testing setup of a hadoop cluster and running map reduce.

There are preliminary notes in first post.

One key, I ran all commands under openssh as that user

Setup the user

Main machine:


sudo usermod --shell /bin/bash mainhdfs


Start openssh

  openssh

  su - mainhdfs

  sudo apt-get install openssh-server

  sudo systemctl enable ssh

  sudo systemctl enable ssh --now

  sudo systemctl start ssh

  su - mainhdfs

  sudo mkdir /usr/local/hadoop

  sudo mv hadoop-3.3.6 /usr/local/hadoop/

  sudo chown own -R mainhdfs:hadoop /usr/local/hadoop

  sudo chown -R mainhdfs:hadoop /usr/local/hadoop


Setup more on ssh:

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

chmod 700 ~/.ssh


Write the following .bashrc and .bash_profile for that user


export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.6

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64


And run hdfs commands and yarn:


   hdfs namenode -format

   start-dfs.sh

   start-yarn.sh

   hdfs dfsadmin -report

   hdfs dfs -mkdir /usr/test

   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/output

   hdfs dfs -mkdir /user/test

   hdfs dfs -mkdir /user/

   hdfs dfs -mkdir /user/test

   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/output

   hdfs dfs -mkdir /user/test

   hdfs dfs -mkdir /user/test/input

   hdfs dfs -mkdir /user/test/output

   hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/output

  hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/outputx

  hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/outputx

  hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/test/input /user/test/outputy 


Comments

Post a Comment

Popular posts from this blog

Random thoughts on AI

 Random AI Thoughts Posts - things I am looking at: scale ai Pinecone Perplexity Notion Hebbia Character AI --- web server web client stand alone -    limited html and http - validates your html and shit   very strict     -- tomcat based      -- my own ML-From-Scratch Description: This repository provides implementations of various machine learning algorithms from scratch in Python, without using libraries like Scikit-learn. It's educational and shows how algorithms like k-means clustering, decision trees, and linear regression work under the hood. Repository: ML-From-Scratch on GitHub You-Get Description: A simple and powerful tool for downloading media files from the web. It uses machine learning for analyzing and categorizing the downloaded media. Features: Supports various media sites, custom download options, and automated media organization. 3. OpenAI Gym Description: A toolkit for developing and comparing reinforcement learning algorit...
Read more

Getting Numenta htm.java to run

Image
 OK, that didn't take long, the fork ran for htm.java numenta nupic So if you want to run with latest Java, Eclipse have it build, run the unit tests and run an integration, try the fork here: mvn compile exec:java https://github.com/berlinbrown/htm.java/blob/master/integration/testing/reports/BASIC-INTEGRATION-RUN.txt
Read more