Back to Hadoop section

SlipStream Recipe

OS customisation

Compilation of several studies:
Red Hat, Exploring the next generation of Big Data solutions with Hadoop 2 (2014).
AMD, Hadoop Perf. Tuning Guide (2012).

• Automatic Tuning Daemon Tuned:

$ tuned-adm profile enterprise-storage

• Max. number of open file descriptors:

echo "hdfs          -    nofile          32768" >> /etc/security/limits.conf 
echo "mapred        -    nofile          32768 " >> /etc/security/limits.conf 

• Max. number of "backlogged sockets":

echo "net.core.somaxconn = 1024" >> /etc/sysctl.conf

• Ext4 formatting:

mkfs.ext4 /dev/vdx

• Disk metadata logging disabled:

mount -t ext4 -o defaults,noatime /dev/vdx /grid01

• CFQ IO scheduler:

echo cfq > /sys/block/vdd/queue/scheduler

• Disk read ahead factor:

blockdev --setra 8196 /dev/vdx