Replace failure harddisk on SW raid 1

In example, Server build software raid 1 with two harddisk, sda and sdb. In this case, sdb fail. Below step show how to replace new harddisk on mdadm raid configuration.

1. Check sdb partition member on raid group  
$ cat /proc/mdstat

####################
 root@raid:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb6[2] sda6[0]
      6917056 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdb1[2] sda1[0]
      487104 blocks super 1.2 [2/2] [UU]

unused devices:
####################
You will see sdb1 was a member of md0 and sdb6 was a member of md1

2. Mark sdb partition to failure
$ mdadm --manage /dev/md0 --fail /dev/sdb1
$ mdadm --manage /dev/md1 --fail /dev/sdb6

####################
root@raid:/# mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0

root@raid:/# mdadm --manage /dev/md1 --fail /dev/sdb6
mdadm: set /dev/sdb6 faulty in /dev/md1

root@raid:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdb6[2](F) sda6[0]
      6917056 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sdb1[2](F) sda1[0]
      487104 blocks super 1.2 [2/1] [U_]

unused devices:

root@raid:/#
####################
You will see the member of md0 and md1 change to [2/1] [U_]

3. Remove sdb1 and sdb6 from raid group
$ mdadm --manage /dev/md0 --remove /dev/sda1
$ mdadm --manage /dev/md1 --remove /dev/sdb6

####################
root@raid:/# mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
root@raid:/# mdadm --manage /dev/md1 --remove /dev/sdb6
mdadm: hot removed /dev/sdb6 from /dev/md1

root@raid:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda6[0]
      6917056 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0]
      487104 blocks super 1.2 [2/1] [U_]

unused devices:

####################
You will see partition sdb1 and sdb6 already remove from raid group

4. Shutdown and change new harddisk

4.1 If new harddisk have old data you need to clear data first
$ dd if=/dev/zero bs=1M count=1 of=/dev/sdb; sync

5. After turn on check you new harddisk already install completely 
$ fdisk -l /dev/sdb

####################
root@raid:/# fdisk -l /dev/sdb

Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
root@raid:/#
####################

6. Before we add new harddisk to raid member group we need to clone partition from sda and confirm partition must be same as sda
$ sfdisk -d /dev/sda | sfdisk /dev/sdb

####################
root@raid:/# sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
OK....
...
root@raid:/# fdisk -l /dev/sdb

Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders, total 16777216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048      976895      487424   fd  Linux raid autodetect
/dev/sdb2          978942    16775167     7898113    5  Extended
/dev/sdb5          978944     2930687      975872   82  Linux swap / Solaris
/dev/sdb6         2932736    16775167     6921216   fd  Linux raid autodetect
####################

7. Add sdb partition to raid group
$ mdadm --manage /dev/md0 --add /dev/sdb1
$ mdadm --manage /dev/md1 --add /dev/sdb6

####################
root@raid:/# mdadm --manage /dev/md0 --add /dev/sdb1
mdadm: added /dev/sdb1
root@raid:/# mdadm --manage /dev/md1 --add /dev/sdb6
mdadm: added /dev/sdb6
root@raid:/# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[2] sda1[0]
      487104 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sdb6[2] sda6[0]
      6917056 blocks super 1.2 [2/1] [U_]
      [>....................]  recovery =  1.8% (130816/6917056) finish=0.8min speed=130816K/sec

unused devices:
####################

8. Install grub on sdb partition 
$ grub-install /dev/sdb

####################
root@raid:/# grub-install /dev/sdb
Installation finished. No error reported.
root@raid:/#
####################

Remark: You can uninstall grub by:
$ dd if=/dev/zero of=/dev/sdb bs=446 count=1

9. For confirm grub is install on partition, it have two ways
- by command 
$ dd if=/dev/sdb  bs=512  count=1 2>/dev/null | hexdump -C

the result you will found GRUB string on output, it mean you grub already install on sdb correctly

- by script "Boot Info Script", is develop by ghulselmans, you can download on URL

http://sourceforge.net/projects/bootinfoscript/

after download and extract just run ./bootinfoscript it will show grub information in file RESULTS.txt

10. That all for change new harddisk but wait It still have SWAP partition remain, In my case I make SWAP partition 1 GB on sda5 and sdb5 but after change disk found SWAP partition of sdb5 not activate.

####################

root@raid:/# free -m

             total       used       free     shared    buffers     cached

Mem:           495         89        406          0         10         39

-/+ buffers/cache:         40        455

Swap:          952          0        952

root@raid:/#

####################

11. Because new harddisk drive the UUID of drive will change we need to make SWAP and change the UUID in fstab file

$ mkswap /dev/sdb5

####################

root@raid:/# mkswap /dev/sdb5

Setting up swapspace version 1, size = 975868 KiB

no label, UUID=815d087f-0875-494a-a947-2220fd14a12d

root@raid:/#

####################

12. Copy you new UUID replace in /etc/fstab 

In my case change from 

# swap was on /dev/sdb5 during installation

UUID=0f049d6d-506c-4bdb-9fec-1a6b808fbb65 none            swap    sw              0       0

to

# swap was on /dev/sdb5 during installation

UUID=815d087f-0875-494a-a947-2220fd14a12d none            swap    sw              0       0

13. Activate swap partition sdb5

$ swapon /dev/sdb5

####################

root@raid:/# swapon /dev/sdb5

root@raid:/# free -m

             total       used       free     shared    buffers     cached

Mem:           495         93        402          0         10         42

-/+ buffers/cache:         40        455

Swap:         1905          0       1905

root@raid:/#

####################



From :
http://www.liberiangeek.net/2012/07/how-to-find-where-grub-is-installed-in-ubuntu/
http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array

perc H200 hardware raid utility in linux

I just bought server coming with hardware raid PERC H200 (Dell).

By the time of this blog, I understand that it use chipset from LSI here are out put from

#lspci --n

LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)

Unfortunately, the monitoring software for this h/w raid is not open. You have to download sas2ircu bin code. The program will display raid status.

I've downloaded from this link :

http://www.natecarlson.com/2010/08/23/lsi-command-line-utility-for-sas2-non-raid-controllers/#comments

Please visit for usage:

http://hwraid.le-vert.net/wiki/LSIFusionMPTSAS2#a2.Linuxkerneldrivers

Gentoo migration to raid

At some point, high availability become more essential to your server. Raid1(mirror) can help server to be able to continue even the disk go wrong. This blog is intend to be a complete just-type-guide for system admin. I hope that you guy succeeded in this migration

Background my system is gentoo on sda. The new disk will be sdb. After finishing copying data to sdb. The another new disk will be replace as sda. sda and sdb will be raid1 mirroring each other.

#0 move running service to another system

- make sure that the system can be down for maintainance

#1 shutdown system

shutdown -h now

#2 install new hdd as sdb

some vendor disable port B (for sdb) please make sure that it is enable so it can be visible from linux.

#3 boot system with live cd

modprobe raid1

#4 format disk and create raid on sdb

available 500GB

-> sdb1 boot 1GB ext2

-> sdb5 root entire ext3

-> sdb6 swap 4GB swap

cfdisk /dev/sdb

cd /dev && MAKEDEV md

mdadm --create /dev/md1 --level=1 --raid-devices=2 missing /dev/sdb1

mdadm --create /dev/md5 --level=1 --raid-devices=2 missing /dev/sdb5

mke2fs /dev/md1

mke2fs -j /dev/md5

mkswap /dev/sdb6

swapon /dev/sdb6

#5 copy data from sda to raid

mkdir /mnt/gentoo1

mount /dev/sda5 /mnt/gentoo1

mount /dev/md5 /mnt/gentoo

mount /dev/sda1 /mnt/gentoo1/boot

mkdir /mnt/gentoo/boot

mount /dev/md1 /mnt/gentoo/boot

cp -dpRx /mnt/gentoo1/* /mnt/gentoo/

#6 install boot loader to raid

mount -t proc none /mnt/gentoo/proc

mount -o bind /dev /mnt/gentoo/dev

chroot /mnt/gentoo /bin/bash

env-update

source /etc/profile

export PS1=”(chroot) $PS1”

#Edit grub.conf change root=/dev/sdaxx to mdxx

nano /boot/grub/grub.conf

#install boot loader to the disk

grub –no-floppy

- device (hd0) /dev/sdb

- root (hd0,0)

- setup (hd0)

#7 edit fstab to mount disk on raid instead // or just copy from preedit @ /home/username/fstab

nano /etc/fstab

#8 shutdown & replace sda with new disk

exit

cd /

umount /mnt/gentoo/boot /mnt/gentoo/proc /mnt/gentoo/dev /mnt/gentoo

umount /mnt/gentoo1/boot /mnt/gentoo1

shutdown -h now

-replace new disk

#9 sync disk and install boot loader

-change bios priority to start boot from sdb

sfdisk -d /dev/sdb | sfdisk /dev/sda

mdadm --add /dev/md1 /dev/sda1

mdadm --add /dev/md5 /dev/sda5

#edit fstab add sda as swap

nano /etc/fstab

mkswap /dev/sda6

swapon /dev/sda6

grub –no-floppy

- device (hd0) /dev/sda

- root (hd0,0)

- setup (hd0)

#wait until sync finish

- reboot test

- restore bios boot priority

- restore sevice back to the server.

Hardisk upgrade/Replace

เปลี่ยน ฮาร์ดดิส ให้ลิกนุกซ์
http://www.losurs.org/docs/LDP/HOWTO/Hard-Disk-Upgrade/index.html

How to Monitor/manager software raid

# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb5[2] sda5[0]
9767424 blocks [2/1] [U_]
[=====>...............] recovery = 25.8% (2528512/9767424) finish=4.4min speed=27019K/sec

md2 : active raid1 sdb6[1]
19534912 blocks [2/1] [_U]

md3 : active raid1 sdb8[1] sda8[0]
44756992 blocks [2/2] [UU]

md0 : active raid1 sdb1[1] sda1[0]
64128 blocks [2/2] [UU]

# mdadm --detail /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Sat Jul 9 21:46:05 2005
Raid Level : raid1
Array Size : 9767424 (9.31 GiB 10.00 GB)
Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
Raid Devices : 2
Total Devices : 1
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Tue Dec 8 12:59:37 2009
State : clean, degraded
Active Devices : 1
Working Devices : 1
Failed Devices : 0
Spare Devices : 0

UUID : 5757d121:b65d61e8:0f50c7af:cbb83343
Events : 0.367292852

Number Major Minor RaidDevice State
0 8 5 0 active sync /dev/sda5
1 0 0 1 removed


Rebuild raid :
# mdadm --add /dev/md1 /dev/sdb5
mdadm: re-added /dev/sdb5

Mark device as faulty
#mdadm --manage /dev/md0 --fail /dev/sda2

Remove device from array
#mdadm --remove /dev/md3 /dev/sb3

Stop & remove raid

*please unmount all file system before stop / remove raid.

#mdadm --stop /dev/md2

#mdadm --zero-superblock /dev/sda2

----

Reference :
http://tldp.org/HOWTO/Software-RAID-0.4x-HOWTO.html#toc4
http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html
http://www.sanitarium.net/golug/Linux_Software_RAID.html

http://blogama.org/node/1825

How to : Software Raid migration

Now use sfdisk to duplicate partitions from old drive to new drive:

#sfdisk -d /dev/sda | sfdisk /dev/sdb

Now use mdadm to create the raid arrays. We mark the first drive (sda) as "missing" so it doesn't wipe out our existing data:

#mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb1

Now copy the remaining partitions. Be careful to match your md devices with your filesystem layout. This example is for my particular setup

mount /dev/md1 /mnt/var
cp -dpRx /var /mnt
... repeat for remaining folder.

Setting fstab, grub to boot from mdX

Reboot

At this point, you have all of your original data on the new drive, so we can safely add the original drive to the raid volume.

mdadm --add /dev/md0 /dev/sda1
mdadm --add /dev/md1 /dev/sda2
... repeat for remaining partitions.

Thanks to :
http://www.debian-administration.org/articles/238

How to : software raid GRUB + RAID

How to : software raid GRUB + RAID

In the Software RAID howto it is mentioned that it is not known how
to set up GRUB to boot off RAID. Here is how I did it:
**Follow at your own risk. If you break something it's your fault.**
==================================================================
Configuration:
- /dev/hda (Pri. Master) 60 GB Seagate HDD (blank)
- /dev/hdc (Sec. Master) 60 GB Seagate HDD (blank)
- /dev/hdd (Sec. Slave) CDROM Drive

Setup Goals:
- /boot as /dev/md0: RAID1 of /dev/hda1 & /dev/hdc1 for redundancy
- / as /dev/md1: RAID1 of /dev/hda2 & /dev/hdc2 for redundancy
- swap*2 with equal priority: /dev/hda3 & /dev/hdc3 for more speed
- GRUB installed in boot records of /dev/hda and /dev/hdc so either
drive can fail but system still boot.

Tools:
- mdadm (http://www.cse.unsw.edu.au/~neilb/source/mdadm/)
(I used 1.2.0, but notice that as of 20030729 1.3.0 is available)

1. Boot up off rescue/installation CD/disk/HDD/whatever with mdadm
tools installed.

2. Partitioning of hard drives:
(I won't show you how to do this. See: # man fdisk ; man sfdisk )
But here's how stuff was arranged:


# sfdisk -l /dev/hda

Disk /dev/hda: 7297 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting
from 0

Device Boot Start End #cyls #blocks Id System
/dev/hda1 * 0+ 16 17- 136521 fd Linux raid autodetect
/dev/hda2 17 7219 7203 57858097+ fd Linux raid autodetect
/dev/hda3 7220 7296 77 618502+ 82 Linux swap
/dev/hda4 0 - 0 0 0 Empty
------------------------------------------------------------------
To make /dev/hdc the same:
------------------------------------------------------------------
# sfdisk -d /dev/hda | sfdisk /dev/hdc
------------------------------------------------------------------
/dev/hd[ac]1 for /dev/md0 for /boot
/dev/hd[ac]2 for /dev/md1 for /
/dev/hd[ac]3 for 2*swap
It is important to make md-to-be partitions with ID 0xFD, not 0x83.

3. Set up md devices: (both are RAID1 [mirrors])
------------------------------------------------------------------
# mdadm --create /dev/md0 --level=1 \
--raid-devices=2 /dev/hda1 /dev/hdc1
# mdadm --create /dev/md1 --level=1 \
--raid-devices=2 /dev/hda2 /dev/hdc2
------------------------------------------------------------------

• สร้าง metadevice ด้วยคำสั่ง cd /dev && MAKEDEV md
(หรือทำเอง mknod /dev/md1 b 9 1 )
• กำหนด raid ด้วยคำสั่ง
◦ mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
◦ mdadm --create --verbose /dev/md5 --level=1 --raid-devices=2 /dev/sda5 /dev/sdb5
◦ mdadm --create --verbose /dev/md6 --level=1 --raid-devices=2 /dev/sda6 /dev/sdb6

รอให้สร้าง raid เสร็จก่อนด้วย คำสั่ง cat /proc/mdstat

หมายเหตุ ถ้ามีการ create แล้วให้ใช้ mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1

4. Make filesystems:
------------------------------------------------------------------
# mke2fs /dev/md0
# mkreiserfs /dev/md1
# mkswap /dev/hda3
# mkswap /dev/hdc3
------------------------------------------------------------------

5. Install Your distribution:
Simply treat /dev/md0 and /dev/md1 as the partitions to install on,
and install the way your normally do.

Here're the relevant entries /etc/fstab for the newly created
partitions:
------------------------------------------------------------------
/dev/md0 /boot ext2 noauto,noatime 1 1
/dev/md1 / reiserfs noatime 1 1
/dev/hda3 none swap sw,pri=1 0 0
/dev/hdc3 none swap sw,pri=1 0 0
------------------------------------------------------------------
The "pri=1" for each of the swap partitions makes them the same
priority so the kernel does striping and that speeds up vm. Of
course, this means that if a disk dies then the system may crash,
needing a reboot. Perhaps it would be wiser to make hd[ac]3 a RAID1
array too, and just use /dev/md2 as swap.

6. Setting up GRUB: (assuming you've already installed it)
------------------------------------------------------------------
# grub
grub> root (hd0,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 16 sectors are
embedded.
succeeded
Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p
(hd0,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.

grub> root (hd1,0)
Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd1)
Checking if "/boot/grub/stage1" exists... yes
Checking if "/boot/grub/stage2" exists... yes
Checking if "/boot/grub/e2fs_stage1_5" exists... yes
Running "embed /boot/grub/e2fs_stage1_5 (hd1)"... 16 sectors are
embedded.
succeeded
Running "install /boot/grub/stage1 (hd1) (hd1)1+16 p
(hd1,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.

grub> quit
------------------------------------------------------------------
Here is how /boot/grub/grub.conf is: (/dev/md0 mounted as /boot)
(Assuming kernel is installed as /boot/bzImage, and RAID1 support
compiled into the kernel).
------------------------------------------------------------------
# Boot automatically after 30 secs.
timeout 30

# By default, boot the first entry.
default 0

# Fallback to the second entry.
fallback 1

# For booting with disc 0 kernel
title GNU/Linux (hd0,0)
kernel (hd0,0)/bzImage root=/dev/md1

# For booting with disc 1 kernel, if (hd0,0)/bzImage is unreadable
title GNU/Linux (hd1,0)
kernel (hd1,0)/bzImage root=/dev/md1



http://www.linuxsa.org.au/mailing-list/2003-07/1270.html
http://en.gentoo-wiki.com/wiki/RAID/Software
http://www.geisterstunde.org/drupal/?q=raid1
http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml