A coder in the attic

Scripts from a coder

Install and Configure Hadoop on Ubuntu 14.04 LTS

Following steps were copied from a blog post and kept here for reference purpose only.

Download Hadoop 2.4.0

1
2
3
4
5
6

$ cd ~
$ wget http://apache.osuosl.org/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz
$ sudo tar vxzf hadoop-2.4.0.tar.gz -C /usr/local
$ cd /usr/local
$ sudo mv hadoop-2.4.0 hadoop
$ sudo chown -R hduser:hadoop hadoop

Setup Hadoop Environment Variables

1
2

$cd ~
$vi .bashrc

paste following to the end of the file

1
2
3
4
5
6
7
8
9
10
11
12

#Hadoop variables
export JAVA_HOME=/usr/lib/jvm/jdk/
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_INSTALL}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
###end of paste
1
2
3
4
5
6
7
8
9
10
11
12
13

$ cd /usr/local/hadoop/etc/hadoop
$ vi hadoop-env.sh

#modify JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/jdk/
Re-login into Ubuntu using hdser and check hadoop version
$ hadoop version
Hadoop 2.4.0
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1583262
Compiled by jenkins on 2014-03-31T08:29Z
Compiled with protoc 2.5.0
From source with checksum 375b2832a6641759c6eaf6e3e998147
This command was run using /usr/local/hadoop-2.4.0/share/hadoop/common/hadoop-common-2.4.0.jar

At this point, hadoop is installed.

Configuring Hadoop

based on script of ericduq at https://github.com/ericduq/hadoop-scripts/blob/master/make-single-node.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

$ cd /usr/local/hadoop/etc/hadoop
# Edit configuration files
$ sudo -u hduser sed -i.bak 's=<configuration>=<configuration>\<property>\<name>fs\.default\.name\</name>\<value>hdfs://localhost:9000\</value>\</property>=g' core-site.xml 
$ sudo -u hduser sed -i.bak 's=<configuration>=<configuration>\<property>\<name>yarn\.nodemanager\.aux-services</name>\<value>mapreduce_shuffle</value>\</property>\<property>\<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>\<value>org\.apache\.hadoop\.mapred\.ShuffleHandler</value>\</property>=g' yarn-site.xml
  
$ sudo -u hduser cp mapred-site.xml.template mapred-site.xml
$ sudo -u hduser sed -i.bak 's=<configuration>=<configuration>\<property>\<name>mapreduce\.framework\.name</name>\<value>yarn</value>\</property>=g' mapred-site.xml
 
$ cd ~
$ mkdir -p mydata/hdfs/namenode
$ mkdir -p mydata/hdfs/datanode

$ cd /usr/local/hadoop/etc/hadoop
$ sudo -u hduser sed -i.bak 's=<configuration>=<configuration>\<property>\<name>dfs\.replication</name>\<value>1\</value>\</property>\<property>\<name>dfs\.namenode\.name\.dir</name>\<value>file:/home/hduser/mydata/hdfs/namenode</value>\</property>\<property>\<name>dfs\.datanode\.data\.dir</name>\<value>file:/home/hduser/mydata/hdfs/datanode</value>\</property>=g' hdfs-site.xml


# Format Namenode
# hdfs namenode -format

# Start Hadoop Service
# sudo -u hduser start-dfs.sh
# sudo -u hduser start-yarn.sh

# Check status
# sudo -u hduser jps

# Example
# sudo -u hduser cd /usr/local/hadoop
# sudo -u hduser hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

Configuring Ubuntu Server 14.04 LTS

Objective: Install and configure

*   SSH Server
*   Java JDK
  1. Install and configure SSH
1

$ sudo apt-get install openssh-server

Setup SSH certificate

1

scp ~/.ssh/id_rsa.pub <username>@<host>:~/.ssh/authorized_keys
  1. Install and configure Java JDK
1
2
3
4
5
6
7

$ sudo apt-get install openjdk-7-jdk
$ cd /usr/lib/jvm
$ sudo ln -s java-7-openjdk-amd64 jdk
$ java -version
java version "1.7.0_51"
OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Installing Ubuntu Server 14.04 LTS

There is nothing fancy by installing new OS. It’s dead simple to install Ubuntu server for learning purpose.

  1. First, go to here to download ubuntu-14.04-server-amd64.iso file.
  2. Second, mount iso file to Virtual Box/Parallel Desktop and install OS from there.
  3. Third, wait and see.

Screenshots: installing server's ready

Delay function call with ReactiveCocoa

Delay a function call is possible with ReactiveCocoa with –[RACSignal delay:]

1
2
3
4
5

- (void)delayLogString:(NSString *)string timeInterval:(NSTimeInterval)timeInterval {
  [[[RACSignal empty] delay:timeInterval] subscribeComplete:^{
      NSLog(@"This log message is printed out after %f seconds", timeInterval);
  }];
}

Thinking of solutions for remote blogging with Octopress

After done setting up blogging solution with Octopress, which is this blog is now powered by, I’m thinking of finding a solution to write blog post from iPhone. The first site came up from Google results is this one. The solution seems very neat, it involves setting up a shared folder on Dropbox and a SSH server to run the Octopress flows. Setup a shared folder in Dropbox for holding git repository is easy part. The harder part is getting a always-on SSH server.

Currently I think of three solutions:

  1. Make use of existing SmartTV box which is now running Android 4.2 to be a SSH server
  2. Purchase a minimal VPS instance.
  3. Purchase a Raspberry Pi and make it a SSH server

First solution is free because I already have one SmartTV box. However making Android box running RVM seems to be a difficult task. Further search, there is an attempt to run a Linux distro on Android. It looks promising and worth a try.

Second solution will cost at least 5$/month for a VPS instance. Which is not a good way to go at the moment.

Third solution will cost about 40$ and I’ll got a Raspberry Pi for something else.

In conclusion, first solution goes first. If it is impossible to run RVM on Android box and come third solution.

A Year in review

Today is last day of Snake’s Year. It’s time to review what has been done, what has been left.

  1. Learnt new language, ie. Objective-C, and worked with it for a year.
  2. Released some versions of two apps on AppStore.
  3. One major project is still not in good form, which means, it is moved to new year.
  4. PM is still far from real.

Year of Hourse is coming and there are some works to do. Let’s rock and roll.