facebook-flashcache

1. download facebook-flashcache

wget --no-check-certificate https://github.com/facebook/flashcache/tarball/master
tar xvfpz facebook-flashcache-1.0-135-gaa462a4.tar.gz
cd facebook-flashcache-aa462a4

2. install facebook-flashcache

[root@zenoss facebook-flashcache-aa462a4]# make
make: git: Command not found
make -C src KERNEL_TREE=/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.7.1.el5.x86_64 PWD=/root/facebook-flashcache-aa462a4/src all
make[1]: Entering directory `/root/facebook-flashcache-aa462a4/src'
rpmbuild -bp --target=`uname -m` /usr/src/redhat/SPECS/kernel.spec 2>&1 | tee `dirname /usr/src/redhat/SPECS/kernel.spec`/prep.log
error: failed to stat /usr/src/redhat/SPECS/kernel.spec: No such file or directory
Building target platforms: x86_64
Building for target x86_64
make -C /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.7.1.el5.x86_64 oldconfig ; \
make -C /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.7.1.el5.x86_64 prepare modules_prepare
make: Entering an unknown directory
make: *** /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.7.1.el5.x86_64: No such file or directory. Stop.
make: Leaving an unknown directory
make: Entering an unknown directory
make: *** /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.7.1.el5.x86_64: No such file or directory. Stop.
make: Leaving an unknown directory
make[1]: *** [rhel5-setup] Error 2
make[1]: Leaving directory `/root/facebook-flashcache-aa462a4/src'
make: *** [all] Error 2

뭔가 제대로 안되는데, ㅋ 이게 핵심인듯한 느낌이옴.. " make: git: Command not found "

헉! 근데 yum install git 여기서 설치가 안됨

http://www.phpntips.com/mysql-uncorrelated-subqueries-poor-performances-2-2012-02/

에서 힌트를 얻어

rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
yum install --enablerepo=webtatic git-all

했더니.. 뭔가 되는듯 하다가

Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test


Transaction Check Error:
file /usr/share/emacs/site-lisp/psvn.el from install of subversion-1.6.13-0.1.el5.rf.x86_64 conflicts with file from package subversion-1.6.11-7.el5_6.4.i386
file /usr/share/locale/zh_CN/LC_MESSAGES/subversion.mo from install of subversion-1.6.13-0.1.el5.rf.x86_64 conflicts with file from package subversion-1.6.11-7.el5_6.4.i386
file /usr/share/xemacs/site-packages/lisp/psvn.el from install of subversion-1.6.13-0.1.el5.rf.x86_64 conflicts with file from package subversion-1.6.11-7.el5_6.4.i386

이래 떨어짐

[root@zenoss kernels]# rpm -qa |grep subver
subversion-1.6.11-7.el5_6.4
subversion-1.6.11-7.el5_6.4
[root@zenoss kernels]# rpm -e subversion-1.6.11-7.el5_6.4
error: "subversion-1.6.11-7.el5_6.4" specifies multiple packages

http://articlesfavs.blogspot.com/2011/10/remove-packages-with-this-rpm-error.html?z#!/2011/10/remove-packages-with-this-rpm-error.html

에서 힌트를 얻어

rpm -ev --allmatches subversion-1.6.11-7.el5_6.4

때려서 패키지 삭제하고 다시

yum install --enablerepo=webtatic git-all

하여 드뎌 git 패키지 설치 완료.

[root@zenoss facebook-flashcache-aa462a4]# rpm -qa |grep git
git-email-1.7.7.3-1.w5
git-gui-1.7.7.3-1.w5
emacs-git-1.7.7.3-1.w5
gitk-1.7.7.3-1.w5
git-svn-1.7.7.3-1.w5
git-cvs-1.7.7.3-1.w5
git-1.7.7.3-1.w5
git-all-1.7.7.3-1.w5

git init 때려 .git 생성(http://tranbot.net/ALA/get-started-with-git/, http://namhyung.springnote.com/pages/3132772)

[root@zenoss facebook-flashcache-aa462a4]# git --version
git version 1.7.7.3
[root@zenoss facebook-flashcache-aa462a4]# git init
Initialized empty Git repository in /root/facebook-flashcache-aa462a4/.git/
[root@zenoss facebook-flashcache-aa462a4]# git status
# On branch master
#
# Initial commit
#
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# LICENSE
# Makefile
# Makefile.dkms
# README
# README-CentOS5
# README-DKMS
# doc/
# flashcache-wt/
# src/
# utils/
nothing added to commit but untracked files present (use "git add" to track)

다시 flashcache make 시도 했으나.. 아래와 같이 커널 소스 설치해줘야 한다고 해서..(https://wiki.archlinux.org/index.php/Flashcache)

make: *** /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64: No such file or directory. Stop.

아래와 같이 받아서 rpm -ivh 로 설치해줌.

wget http://vault.centos.org/5.7/os/SRPMS/kernel-2.6.18-274.el5.src.rpm

ㅋ

make 성공

CC [M] /root/facebook-flashcache-aa462a4/src/flashcache_subr.o
CC [M] /root/facebook-flashcache-aa462a4/src/flashcache_ioctl.o
CC [M] /root/facebook-flashcache-aa462a4/src/flashcache_procfs.o
LD [M] /root/facebook-flashcache-aa462a4/src/flashcache.o
Building modules, stage 2.
MODPOST
CC /root/facebook-flashcache-aa462a4/src/flashcache.mod.o
LD [M] /root/facebook-flashcache-aa462a4/src/flashcache.ko
make[2]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
make -C utils all
make[2]: Entering directory `/root/facebook-flashcache-aa462a4/src/utils'
cc -c -I.. -I. -DCOMMIT_REV="\"\"" -g flashcache_create.c -o flashcache_create.o
cc flashcache_create.o -o flashcache_create
cc -c -I.. -I. -DCOMMIT_REV="\"\"" -g flashcache_destroy.c -o flashcache_destroy.o
cc flashcache_destroy.o -o flashcache_destroy
cc -c -I.. -I. -DCOMMIT_REV="\"\"" -g flashcache_load.c -o flashcache_load.o
cc flashcache_load.o -o flashcache_load
make[2]: Leaving directory `/root/facebook-flashcache-aa462a4/src/utils'
make[1]: Leaving directory `/root/facebook-flashcache-aa462a4/src'
[root@zenoss facebook-flashcache-aa462a4]#

내킨김에 make install 까지 성공

# configuration written to .config
#
make[2]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
make[2]: Entering directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
scripts/kconfig/conf -s arch/x86_64/Kconfig
make[2]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
make[2]: Entering directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
CHK include/linux/version.h
CHK include/linux/utsrelease.h
make[2]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
ln -s -f /usr/src/kernels/2.6.18-274.17.1.el5-x86_64/Module.symvers /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64/Module.symvers
make -C /usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64 M=/root/facebook-flashcache-aa462a4/src modules V=0
make[2]: Entering directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
Building modules, stage 2.
MODPOST
make[2]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18-274.17.1.el5.x86_64'
install -o root -g root -m 0755 -d /lib/modules/2.6.18-274.17.1.el5/extra/flashcache/
install -o root -g root -m 0755 flashcache.ko /lib/modules/2.6.18-274.17.1.el5/extra/flashcache/
depmod -a
make -C utils all
make[2]: Entering directory `/root/facebook-flashcache-aa462a4/src/utils'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/root/facebook-flashcache-aa462a4/src/utils'
make -C utils install
make[2]: Entering directory `/root/facebook-flashcache-aa462a4/src/utils'
install -d -m 755 /sbin/
install -m 755 flashcache_create flashcache_destroy flashcache_load /sbin/
make[2]: Leaving directory `/root/facebook-flashcache-aa462a4/src/utils'
make -C ocf install
make[2]: Entering directory `/root/facebook-flashcache-aa462a4/src/ocf'
install -d -m 755 /usr/lib/ocf/resource.d/flashcache
install -m 755 flashcache /usr/lib/ocf/resource.d/flashcache
make[2]: Leaving directory `/root/facebook-flashcache-aa462a4/src/ocf'
make[1]: Leaving directory `/root/facebook-flashcache-aa462a4/src'
[root@zenoss facebook-flashcache-aa462a4]#

flashcache 바이너리 생성된것을 아래와 같이 확인함.

[root@zenoss facebook-flashcache-aa462a4]# type flashcache_create
flashcache_create is /sbin/flashcache_create

3. create device

[root@zenoss ~]# flashcache_create -v -p thru -s 1g -b 4k cachedev /dev/sdb7 /dev/sda7
cachedev cachedev, ssd_devname /dev/sdb7, disk_devname /dev/sda7 cache mode WRITE_THROUGH
block_size 8, cache_size 2097152
Flashcache metadata will use 4MB of your 2001MB main memory
Flashcache Module already loaded
version string "git commit:
"
Creating FlashCache Volume : "echo 0 97225317 flashcache /dev/sda7 /dev/sdb7 cachedev 2 2 8 2097152 47154445943296 8 | dmsetup create cachedev"
[root@zenoss ~]#
[root@zenoss ~]# flashcache_load /
[root@zenoss ~]# dmsetup table cachedev
0 97225317 flashcache conf:
ssd dev (/dev/sdb7), disk dev (/dev/sda7) cache mode(WRITE_THROUGH)
capacity(1024M), associativity(512), data block size(4K)
skip sequential thresh(0K)
total blocks(262144), cached blocks(0), cache percent(0)
nr_queued(0)
Size Hist:
[root@zenoss ~]# ll /dev/mapper/cachedev
brw-rw---- 1 root disk 253, 0 Feb 24 15:15 /dev/mapper/cachedev

4. mount cachedev
[root@zenoss ~]# mount -t ext3 -o defaults,noatime,nodiratime /dev/mapper/cachedev /home
[root@zenoss ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 2.8G 2.4G 240M 92% /
/dev/sda1 190M 119M 62M 66% /boot
none 1001M 0 1001M 0% /dev/shm
/dev/sda5 4.6G 764M 3.7G 18% /var
/dev/sda6 19G 2.9G 15G 17% /usr
/dev/mapper/cachedev 46G 36G 7.9G 82% /home

5. 퍼포먼스 테스트
=> SATA + SATA flash cache
[IOTEST REPORT]
./iotest.sh: line 39: /root/info: No such file or directory
IOTEST 시작시간: 2012-02-24,15:26:40
IOTEST 종료시간: 2012-02-24,15:40:44
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zenoss.simple 2G:1k 34678 47 43847 17 21142 8 34862 42 53086 4 802.3 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100 956 1 +++++ +++ 661 1 1019 2 +++++ +++ 652 1
zenoss.simplexi.com,2G:1k,34678,47,43847,17,21142,8,34862,42,53086,4,802.3,0,100,956,1,+++++,+++,661,1,1019,2,+++++,+++,652,1

=> SATA + SSD flash cache write_through mode(순차 입력이 SATA에 비해 2배)
[IOTEST REPORT]
./iotest.sh: line 39: /root/info: No such file or directory
IOTEST 시작시간: 2012-02-24,16:04:27
IOTEST 종료시간: 2012-02-24,16:17:06
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zenoss.simple 2G:1k 32404 44 41780 15 22782 8 38009 45 94651 7 785.3 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100 1435 3 +++++ +++ 659 1 1323 2 +++++ +++ 665 1
zenoss.simplexi.com,2G:1k,32404,44,41780,15,22782,8,38009,45,94651,7,785.3,0,100,1435,3,+++++,+++,659,1,1323,2,+++++,+++,665,1


=> SATA + SSD flash cache writeback mode(결과가 좋지 않음..)
[IOTEST REPORT]
./iotest.sh: line 39: /root/info: No such file or directory
IOTEST 시작시간: 2012-02-24,16:57:35
IOTEST 종료시간: 2012-02-24,17:45:14
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zenoss.simple 2G:1k 40201 55 59847 23 26894 11 39764 50 69357 6 898.2 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100 159 0 +++++ +++ 158 0 158 0 +++++ +++ 153 0
zenoss.simplexi.com,2G:1k,40201,55,59847,23,26894,11,39764,50,69357,6,898.2,0,100,159,0,+++++,+++,158,0,158,0,+++++,+++,153,0


=> SSD(순차 입력이 SATA에 비해 8배)
[IOTEST REPORT]
./iotest.sh: line 39: /root/info: No such file or directory
IOTEST 시작시간: 2012-02-24,15:53:56
IOTEST 종료시간: 2012-02-24,15:58:51
Version 1.03e ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
zenoss.simple 2G:1k 73081 99 239850 80 125930 36 82447 97 419719 32 +++++ +++
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
100 2803 5 +++++ +++ 1675 3 2757 5 +++++ +++ 1489 3
zenoss.simplexi.com,2G:1k,73081,99,239850,80,125930,36,82447,97,419719,32,+++++,+++,100,2803,5,+++++,+++,1675,3,2757,5,+++++,+++,1489,3

 

* 성능에 관련된 링크
http://www.mysqlperformanceblog.com/2010/05/10/flashcache-first-experiments/

 

* 설명서

https://raw.github.com/facebook/flashcache/master/doc/flashcache-sa-guide.txt