Page cache and pdflush: Theory of operation and tuning for write-heavy loads

For write-heavy option in system, some kernel option must be adjusted. Here are link to article about some kernel options that might help.

Please refer to : http://www.westnet.com/~gsmith/content/linux-pdflush.htm

find command with -exec options

find command is quite a handy tool for file searching. It can also include an command to execute for matched file with

1) -exec command ;

2) -exec command {} +

Both are different.

Let say you have 3 files, a b c in the directory.

#find . -exec echo '{}' >> tmp1 \;

output in tmp1 will be :

.

./b

./a

./c

./tmp1

#find . -exec echo '{}' >> tmp2 \+;

output in tmp2 will be :

a b c

with + all matched file will be append and execute once.

with ; all matched file will be execute for each matching.

Install windows font in linux(ubuntu)

Linux user cannot avoid living with windows user. That make linux users to adjust ourselves to read file from windows including fonts they using.

(1) using font package

#sudo apt-get install msttcorefonts




#sudo fc-cache -fv







(2) copying font files from windows

copy *.ttf font to /usr/share/fonts/truetype/thai

#fc-cache -fv

reference

1. ms-font package installation [here]

The linux documentation project

Linux newbie might be looking for reliable document for linux.
Here is a good link that worth a look.

Link:
http://tldp.org/guides.html

gentoo SNMP

Installing SNMP in gentoo for system monitoring.


#emerge net-snmp

edit /etc/snmp/snmpd.conf add the following
com2sec Mybox localhost public
com2sec cacti 192.168.1.0/24 public

group RWGroup v2c Mybox
group ROGroup v1 cacti
group ROGroup v2c cacti

view all included .1 80
view system included system fe

access ROGroup "" any noauth exact all none none
access RWGroup "" v2c noauth exact all all all

#rc-update add snmpd default

Nginx / lighttpd light weight web server.

Apache is a standard for web server. But, you can help apache and utilize computing resource by using lighttpd and nginx as a helper for static content/load balancer for your website.

nginx :
http://wiki.nginx.org/NginxModules
lighttpd
http://redmine.lighttpd.net/wiki/lighttpd

User & group management : uid, gid user

If you share file about variety distro of linux, you might experience with permission error.

The following command help to change uid, gid of user apache group apache to 81 :
#usermod -u 81 apache
#groupmod -g 81 apache
#id apache

Read more:

user & group management : http://www.linux-mag.com/cache/7759/1.html

How to monitor harddisk with smartmontools

Now let's install the smartmontools package:

# emerge -av smartmontools

Finally, you have to check if your hard disk(s) support SMART:

# smartctl -i /dev/hda

For SATA drives:

# smartctl -i -d ata /dev/sda

Using smartctl

[edit]

SMART Health Status

Let's check the SMART Health Status:

# smartctl -H /dev/hda

If you read PASSED it's ok, but if you read FAILED you have to backup your data now: the disk has already failed or it's predicted to fail within 24 hours!

To recover from bad block :

http://smartmontools.sourceforge.net/badblockhowto.html

smartmontool document:

http://sourceforge.net/apps/trac/smartmontools/wiki/TocDoc

swappiness tuning linux

vm.swappiness is a tunable kernel parameter that controls how much the kernel favors swap over RAM. At the source code level, it’s also defined as the tendency to steal mapped memory. A high swappiness value means that the kernel will be more apt to unmap mapped pages. A low swappiness value means the opposite, the kernel will be less apt to unmap mapped pages. In other words, the higher the vm.swappiness value, the more the system will swap.

More to read : http://www.linuxvox.com/2009/10/what-is-the-linux-kernel-parameter-vm-swappiness/

The following is the test. It seems that the higher swapiness the slower i/o rate. More to read : http://lwn.net/Articles/100978/

linux memory management

Overview of memory management

Traditional Unix tools like 'top' often report a surprisingly small amount of free memory after a system has been running for a while. For instance, after about 3 hours of uptime, the machine I'm writing this on reports under 60 MB of free memory, even though I have 512 MB of RAM on the system. Where does it all go?

The biggest place it's being used is in the disk cache, which is currently over 290 MB. This is reported by top as "cached". Cached memory is essentially free, in that it can be replaced quickly if a running (or newly starting) program needs the memory.

The reason Linux uses so much memory for disk cache is because the RAM is wasted if it isn't used. Keeping the cache means that if something needs the same data again, there's a good chance it will still be in the cache in memory. Fetching the information from there is around 1,000 times quicker than getting it from the hard disk. If it's not found in the cache, the hard disk needs to be read anyway, but in that case nothing has been lost in time.

To see a better estimation of how much memory is really free for applications to use, run the command free -m:

Code: free -m

total used free shared buffers cached Mem: 503 451 52 0 14 293 -/+ buffers/cache: 143 360 Swap: 1027 0 1027

The -/+ buffers/cache line shows how much memory is used and free from the perspective of the applications. Generally speaking, if little swap is being used, memory usage isn't impacting performance at all.

Notice that I have 512 MB of memory in my machine, but only 52 is listed as available by free. This is mainly because the kernel can't be swapped out, so the memory it occupies could never be freed. There may also be regions of memory reserved for/by the hardware for other purposes as well, depending on the system architecture. However, 360M are free for application consumption.

The difference among VIRT, RES, and SHR in top output

VIRT stands for the virtual size of a process, which is the sum of memory it is actually using, memory it has mapped into itself (for instance the video card's RAM for the X server), files on disk that have been mapped into it (most notably shared libraries), and memory shared with other processes. VIRT represents how much memory the program is able to access at the present moment.

RES stands for the resident size, which is an accurate representation of how much actual physical memory a process is consuming. (This also corresponds directly to the %MEM column.) This will virtually always be less than the VIRT size (there are only very rare cases that they will be equal, and RES will never be more than VIRT).

SHR indicates how much of the VIRT size is actually sharable (memory or libraries). In the case of libraries, it does not necessarily mean that the entire library is resident. For example, if a program only uses a few functions in a library, the whole library is mapped and will be counted in VIRT and SHR, but only the parts of the library file containing the functions being used will actually be loaded in and be counted under RES.

The difference between buffers and cache

Buffers are allocated by various processes to use as input queues, etc. Most of the time, buffers are some processes' output, and they are file buffers. A simplistic explanation of buffers is that they allow processes to temporarily store input in memory until the process can deal with it.

Cache is typically frequently requested disk I/O. If multiple processes are accessing the same files, much of those files will be cached to improve performance (RAM being so much faster than hard drives), it's disk cache.

More to read : http://www.gentoo-wiki.info/FAQ_Linux_Memory_Management

http://forums.gentoo.org/viewtopic-t-175419-postdays-0-postorder-asc-start-0.html

Hardisk upgrade/Replace

เปลี่ยน ฮาร์ดดิส ให้ลิกนุกซ์
http://www.losurs.org/docs/LDP/HOWTO/Hard-Disk-Upgrade/index.html

Ubuntu handy configuration and command after installation + apache PHP

Ubuntu

configuration after setup.

startup interface/fix ip on startup
> /etc/network/interfaces.
iface eth1 inet static
address 192.168.0.2
netmask 255.255.255.0
gateway 192.168.0.1

set name server
> /etc/resolv.conf
nameserver 204.11.126.131


install ssh for remote login
> openssh
sudo apt-get install openssh-client openssh-server


>adduser
sudo adduser username


>user / add/ super user (group admin) sodoer
sudo adduser xeon
sudo adduser xeon admin


>php5 + mysql
sudo apt-get install

• php5
• php5-cgi
• php5-cli
• php5-mysql
• php5-memcache
• php5-gd
• php5-imagick

>apache
sudo apt-get install apache2-mpm-worker


>remove service from startup
update-rc.d -f apache2 remove

>add service
update-rc.d apache2 defaults

>more reading on services
http://www.debuntu.org/how-to-manage-services-with-update-rc.d


>NFS server
sudo apt-get install nfs-kernel-server
sudo /etc/init.d/nfs-kernel-server start

>NFS client
sudo apt-get install nfs-common
For more info about nfs : https://help.ubuntu.com/community/SettingUpNFSHowTo


>/etc/exports
/ubuntu *(ro,sync,no_root_squash)
/home *(rw,sync,no_root_squash)

>package management-install
#apt-get install packagename
-see update current version

# apt-cache policy apache2
apache2:
Installed: 2.2.8-1ubuntu0.3
Candidate: 2.2.8-1ubuntu0.14
Version table:
2.2.8-1ubuntu0.14 0
500 http://th.archive.ubuntu.com hardy-updates/main Packages
500 http://security.ubuntu.com hardy-security/main Packages
*** 2.2.8-1ubuntu0.3 0
100 /var/lib/dpkg/status
2.2.8-1 0
500 http://th.archive.ubuntu.com hardy/main Packages

Remark
*apache worker mpm not work with libapach2-php
using fcgid mod and php cgi

*NFS add host to /etc/hosts.allow
*snmpd change init opt /etc/default/snmpd
*all rp_filter need to be disable

How to Monitor/manager software raid

# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb5[2] sda5[0]
9767424 blocks [2/1] [U_]
[=====>...............] recovery = 25.8% (2528512/9767424) finish=4.4min speed=27019K/sec

md2 : active raid1 sdb6[1]
19534912 blocks [2/1] [_U]

md3 : active raid1 sdb8[1] sda8[0]
44756992 blocks [2/2] [UU]

md0 : active raid1 sdb1[1] sda1[0]
64128 blocks [2/2] [UU]

# mdadm --detail /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Sat Jul 9 21:46:05 2005
Raid Level : raid1
Array Size : 9767424 (9.31 GiB 10.00 GB)
Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Tue Dec 8 12:59:37 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 5757d121:b65d61e8:0f50c7af:cbb83343
Events : 0.367292852

Number Major Minor RaidDevice State
0 8 5 0 active sync /dev/sda5
1 0 0 1 removed


Rebuild raid :
# mdadm --add /dev/md1 /dev/sdb5
mdadm: re-added /dev/sdb5

Mark device as faulty
#mdadm --manage /dev/md0 --fail /dev/sda2

Remove device from array
#mdadm --remove /dev/md3 /dev/sb3

Stop & remove raid

*please unmount all file system before stop / remove raid.

#mdadm --stop /dev/md2

#mdadm --zero-superblock /dev/sda2

----

Reference :
http://tldp.org/HOWTO/Software-RAID-0.4x-HOWTO.html#toc4
http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html
http://www.sanitarium.net/golug/Linux_Software_RAID.html

http://blogama.org/node/1825

How to : Software Raid migration

Now use sfdisk to duplicate partitions from old drive to new drive:

#sfdisk -d /dev/sda | sfdisk /dev/sdb

Now use mdadm to create the raid arrays. We mark the first drive (sda) as "missing" so it doesn't wipe out our existing data:

#mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb1

Now copy the remaining partitions. Be careful to match your md devices with your filesystem layout. This example is for my particular setup

mount /dev/md1 /mnt/var
cp -dpRx /var /mnt
... repeat for remaining folder.

Setting fstab, grub to boot from mdX

Reboot

At this point, you have all of your original data on the new drive, so we can safely add the original drive to the raid volume.

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2
... repeat for remaining partitions.

Thanks to :
http://www.debian-administration.org/articles/238

How to : software raid GRUB + RAID

How to : software raid GRUB + RAID

In the Software RAID howto it is mentioned that it is not known how
to set up GRUB to boot off RAID. Here is how I did it:
**Follow at your own risk. If you break something it's your fault.**
==================================================================
Configuration:
- /dev/hda (Pri. Master) 60 GB Seagate HDD (blank)
- /dev/hdc (Sec. Master) 60 GB Seagate HDD (blank)
- /dev/hdd (Sec. Slave) CDROM Drive

Setup Goals:
- /boot as /dev/md0: RAID1 of /dev/hda1 & /dev/hdc1 for redundancy
- / as /dev/md1: RAID1 of /dev/hda2 & /dev/hdc2 for redundancy
- swap*2 with equal priority: /dev/hda3 & /dev/hdc3 for more speed
- GRUB installed in boot records of /dev/hda and /dev/hdc so either
drive can fail but system still boot.

Tools:
- mdadm (http://www.cse.unsw.edu.au/~neilb/source/mdadm/)
(I used 1.2.0, but notice that as of 20030729 1.3.0 is available)

1. Boot up off rescue/installation CD/disk/HDD/whatever with mdadm
tools installed.

2. Partitioning of hard drives:
(I won't show you how to do this. See: # man fdisk ; man sfdisk )
But here's how stuff was arranged:


# sfdisk -l /dev/hda

Disk /dev/hda: 7297 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting
from 0

Device Boot Start End #cyls #blocks Id System
/dev/hda1 * 0+ 16 17- 136521 fd Linux raid autodetect
/dev/hda2 17 7219 7203 57858097+ fd Linux raid autodetect
/dev/hda3 7220 7296 77 618502+ 82 Linux swap
/dev/hda4 0 - 0 0 0 Empty
------------------------------------------------------------------
To make /dev/hdc the same:
------------------------------------------------------------------
# sfdisk -d /dev/hda | sfdisk /dev/hdc
------------------------------------------------------------------
/dev/hd[ac]1 for /dev/md0 for /boot
/dev/hd[ac]2 for /dev/md1 for /
/dev/hd[ac]3 for 2*swap
It is important to make md-to-be partitions with ID 0xFD, not 0x83.

3. Set up md devices: (both are RAID1 [mirrors])
------------------------------------------------------------------
# mdadm --create /dev/md0 --level=1 \
--raid-devices=2 /dev/hda1 /dev/hdc1
# mdadm --create /dev/md1 --level=1 \
--raid-devices=2 /dev/hda2 /dev/hdc2
------------------------------------------------------------------

• สร้าง metadevice ด้วยคำสั่ง cd /dev && MAKEDEV md
(หรือทำเอง mknod /dev/md1 b 9 1 )
• กำหนด raid ด้วยคำสั่ง
◦ mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
◦ mdadm --create --verbose /dev/md5 --level=1 --raid-devices=2 /dev/sda5 /dev/sdb5
◦ mdadm --create --verbose /dev/md6 --level=1 --raid-devices=2 /dev/sda6 /dev/sdb6

รอให้สร้าง raid เสร็จก่อนด้วย คำสั่ง cat /proc/mdstat

หมายเหตุ ถ้ามีการ create แล้วให้ใช้ mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1

4. Make filesystems:
------------------------------------------------------------------
# mke2fs /dev/md0
# mkreiserfs /dev/md1
# mkswap /dev/hda3
# mkswap /dev/hdc3
------------------------------------------------------------------

5. Install Your distribution:
Simply treat /dev/md0 and /dev/md1 as the partitions to install on,
and install the way your normally do.

Here're the relevant entries /etc/fstab for the newly created
partitions:
------------------------------------------------------------------
/dev/md0 /boot ext2 noauto,noatime 1 1
/dev/md1 / reiserfs noatime 1 1
/dev/hda3 none swap sw,pri=1 0 0
/dev/hdc3 none swap sw,pri=1 0 0
------------------------------------------------------------------
The "pri=1" for each of the swap partitions makes them the same
priority so the kernel does striping and that speeds up vm. Of
course, this means that if a disk dies then the system may crash,
needing a reboot. Perhaps it would be wiser to make hd[ac]3 a RAID1
array too, and just use /dev/md2 as swap.

6. Setting up GRUB: (assuming you've already installed it)
------------------------------------------------------------------
# grub
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 16 sectors are
embedded.
succeeded
Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p
(hd0,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.

grub> root (hd1,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd1)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd1)"... 16 sectors are
embedded.
succeeded
Running "install /boot/grub/stage1 (hd1) (hd1)1+16 p
(hd1,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.

grub> quit
------------------------------------------------------------------
Here is how /boot/grub/grub.conf is: (/dev/md0 mounted as /boot)
(Assuming kernel is installed as /boot/bzImage, and RAID1 support
compiled into the kernel).
------------------------------------------------------------------
# Boot automatically after 30 secs.
timeout 30

# By default, boot the first entry.
default 0

# Fallback to the second entry.
fallback 1

# For booting with disc 0 kernel
title GNU/Linux (hd0,0)
kernel (hd0,0)/bzImage root=/dev/md1

# For booting with disc 1 kernel, if (hd0,0)/bzImage is unreadable
title GNU/Linux (hd1,0)
kernel (hd1,0)/bzImage root=/dev/md1



http://www.linuxsa.org.au/mailing-list/2003-07/1270.html
http://en.gentoo-wiki.com/wiki/RAID/Software
http://www.geisterstunde.org/drupal/?q=raid1
http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml