3Ware RAID controllers are dependable, reliable controllers,capable of decent throughput, which are supported by many versions of Linixi.For IDE drives, the 7000 series controllers are often available in thediscount/auction market for very little money, especially the 7006-2. ForSATA drives, the 9500 series controllers are likewise available for discountprices, especially the 9500S-LP. These notes are geared towards working withthose cards, although they are extensible to other models of the 3Warecontrollers.
Broadcom Inc. Connecting Everything. VMware ESX 4.0 Driver CD for AMCC 3ware RAID Controllers. This driver CD release includes support for version 2.26.08.035vm40 of the AMCC 3w-9xxx driver on ESX 4.0. The 3w-9xxx driver supports products based on the AMCC 3ware 9650SE and 9690SA SAS and SATA RAID controllers.
The steps to physically setting up a RAID array using the 3Warecontrollers are covered quite well in their instructions. These areadditional notes that address topics that aren't covered by the traditionaldocumentation.
Most versions of Linux come with device drivers built-in,for the 3Ware cards. All that remains to be installed is the 3Ware managementprogram which can be installed to monitor and manage the RAID arrays on asystem using either commands or a Web-based UI.
To install this package, get the latest version of it fromthe 3Ware Web site (you can use the 3DM2 supplied for the 9000 seriescontrollers, with the 7000 series controllers):
To get the proper software, one can pick the 9690SA (or9690SA-4I) series of controllers and then the combined 3DM2 and CLI package.In the past, the version that we have downloaded is the Linux 32-bit version:
For this version, unpack the file as Joe User:
tar -xvzf 3DM2_CLI-Linux_x86-126.96.36.199.tgz
Run the resulting binary as root:
You'll love answering the quesions using only the number keys.We pick /usr/local/3Ware as the install directory (mainly because we can'tremember who AMCC is, apart from the local movie theatre). Some versionsof the installer always want to use a TLD of AMCC so they will actually putthings in /usr/local/3Ware/AMCC, which makes no nevermind to us. But, ifyou want one less directory level, you could pick /usr/local/AMCC.
We usually want to use 'root' as the recipient of the localemail messages but the 'Wizard' won't allow it (only addresses of the [email protected]). But, after it is done 'configuring' your system, you can go to/etc/3dm2/3dm2.conf and simply put in 'root'.
Currently, we are using the 10.2.1 Linux version:
For this version, and despite its being meant for Linux, theclever guys at LSI Corp (the new home for 3Ware) have decided to packagethe 3DM2 CLI as a zip file. So, you're going to need something (like WindozeZip) that can unpack the zip file. We unpack it on a Windoze box, copy it toa Linux box, and then make a new tar file, which we can then use as normal.This is completely up to you. You could just unzip the package and copy thefiles to your Linux system.
Once you have the package unpacked, run the resulting shellscript as root:
Answer 'Y' to the 'agree to the license' prompt and thenpick '0' to install both the CLI and Web interface. The install script doeseverything else for you. There are no other questions to answer.
If you wish to tune 3DM2, particularly if you want it to usea Web port other than 888, you can directly edit /etc/3dm2/3dm2.conf. Weprefer to use port 7080. After editing /etc/3dm2/3dm2.conf, 3DM2 needs tobe recycled (see below).
Once the install is done, 3DM2 should start automagically.In addition, it will start itself whenever the system boots. If you want tostart/stop it manually, its in /etc/init.d, called tdm2 (that's intuitive):
To view the controller's staus from the Web, you need to useHTTPS, not HTTP. The first time around, you need to accept the boguscertificate from the system running 3DM2 (pay no attention to all of thewhining that FireFox or IE does). If you picked port 7080 to run it under,try this:
After a fresh install, the secret passwords are '3ware'. Youcan change them, if you want.
If you receive a connection reset error when you connect tothe 3DM2 Web page, you should check out KB16625 for a fix:
If the disks used in a RAID array that is attached to a3Ware controller are SMART enabled, they can be monitored by the SMARTdaemon smartd.
To monitor the disks in the array, install smartd in theusual manner (either by selecting it from your operating system's packagemanager or installing it from the source directly). Once this is done, hackthe sample configuration file (/etc/smartd.conf) that comes with smartd asshown here, in this case to monitor 3Ware-controlled SATA disks:
If your array uses older, IDE disks, hack the sampleconfiguration file (/etc/smartd.conf) that comes with smartd as shown here:
You may have noticed, from inspection of the two config filesnippets shown above, that the name of the 3Ware device depends on whichdevice driver is employed. The older 3w-xxxx driver names the array device/dev/twen (where 'n' is the device number). The newer 3w-9xxx driver namesthe array device /dev/twan (where 'n' is, once again, the device number).
In either case, the entire array appears as a single device.It might also be named /dev/sda to appear transparent to existing softwareand packages but SMART always uses the twa or twe device names.
To monitor and test individual devices in the array, the'-d 3ware,...' parameter is used to tell SMART that the array is a 3Warearray and to select which device is of interest. The devices in the arrayare numbered starting from zero, up to n-1 where n is the number ofphysical disks in the array.
This naming convention can also be used with the smartctlcommand to list or set the SMART information associated with an specificdisk. For example, to list all SMART information for the first IDE drive ina RAID array, use this command:
/usr/sbin/smartctl -a -d 3ware,0 /dev/twe0
To list all SMART information for the second SATA drive ina different RAID array, use this command:
/usr/sbin/smartctl -a -d 3ware,1 /dev/twa0
When one disk in a 2-disk array goes bad, the following stepsshould allow you to recover the array (these steps also work for creating abrand new array from a cloned drive):
dd if=/dev/zero of=/dev/hdc bs=512 count=1024
In the event that an array did not get rebuilt properly or ifyou are reusing a disk that was previously in an array, you can follow thesesteps to delete the failed/previous array information from the disk and startover (or see the procedure using dd, above).
Should an array fail to rebuild properly, you can try thefollowing steps to get it to do so, despite the occasional I/O error.
If you'd like to clone a disk for use in another system or withanother controller card, this is the preferred method for doing so (althoughyou will be facing a short period of time, while the array is being rebuilt,when you only have one copy of the disk). You can proceed as outlined abovein 'Recovering From A Disk Failure' to make a duplicate of a good drive from aworking array. Start with the original drive in Slot 0 and the clone drive inSlot 1. Follow the steps for recovering from a degraded array.
Once the array with the new drive has been rebuilt, you can shutdown the machine and remove the new drive. You should be able to move the newdrive to another controller card and proceed to build a working array from itwith a second new drive.
Meanwhile, when you insert the original Slot 1 drive back intothe system, the RAID controller will degrade the array and you will have torebuild it again. You should first delete the prior array information from theoriginal Slot 1 drive, as outlined above, either by removing the Slot 0 driveand then deleting the array or by using dd to write zeros over the first fewtracks.
If you are having problems with an array and want a spare copyof the good drive (e.g. before you do anything to the failing array) withoutresorting to the controller-assisted cloning method described above or youjust want to make a copy of a good drive without exposing yourself duringthat short period when there's only one good drive in your array, you canmake a copy of a drive as outlined below.
Once you've done this, in the case of a failing drive you cando your blasting on the copied drive and save the old drive, just in caseyour recovery efforts don't succeed and you need to go back. For juststraight copies, you can put the original drive back in the array and itshould start up without the need to rebuild.
To do the copy, run:
Note that the disk image may be too big for sg_dd or dd to copy all in one operation, in which case the copy will end short of the whole disk. If that happens, you can restart the copy simply by running sg_dd or dd, beginning from just before where copying gave up. For example:
sg_dd skip=1234567 seek=1234567 bs=...
If you want to put the disk into the existing RAID array, useit to build a totally new array for a duplicate machine, or for some otherprocedure that requires it be connected to a 3Ware controller, you will firstneed to alter the disk serial number in the first track of the drive.
When the 3Ware controller builds the array, it puts the serialnumbers of the drives in the array into a control block that it writes to thefirst track of each of the array's drives. The act of copying the drive makesthe serial numbers be mismatched (i.e. the copy contains the original drive'sserial number but the drive it is copied to has a new, different serialnumber).
To make the copy acceptable to a 3Ware controller, you need touse your favorite disk editing program to locate the original disk's serialnumber and update it to the new disk's serial number. At that point, youshould be able to proceed, using the new disk, as outlined above in'Recovering From A Disk Failure'.
Expanding an array's size is complicated principally by thefact that the only thing that will touch the array's drives is the 3Warecontroller. You could try putting two controller cards in one system, asthere is no physical reason preventing it. However, the crap operating systemsoftware (especially Linux) out there may not be too keen on booting up thisconfiguration.
Better to use a separate IDE drive as a temporary copy buffer,as outlined in the following steps (incidentally, these steps asume you'reexpanding a Linux drive, not Windows):
dd bs-65536 if=/dev/sda1 of=/dev/hda1
dd bs-65536 if=/dev/sda3 of=/dev/hda3
Note that we're assuming it was a Linux system with three partitions, one of which (/dev/hda2) was the swap partition.
dd bs=65536 if=/dev/hda1 of=/dev/sda1
dd bs=65536 if=/dev/hda1 of=/dev/sda1
mount /dev/sda3 /mnt/out
rm -rf /mnt/out/*
mount /dev/hda3 /mnt/in
cp -rfp /mnt/in/* /mnt/out
> find /grub/stage1
or, if that fails
> find /boot/grub/stage1
> root (hd0,0) (use the correct address from the find command)
> setup (hd0)
If you wish to migrate the existing SATA or IDE drive ordrives of a Linux system to a RAID array you can use whatever method youchoose to copy the drive(s) to the array.
The array must be set up, connected to a controller andinstalled in a system. It can be the existing system or it might be a new,upgraded system that will eventually run the array.
One way to do the copy is to install the original disk andthe controller/array in the same system. The array is built and then oneboots up a standalone OS (such as Knoppix), from a CD or USB stick, that canbe used to partition the array, copy partiton data from the original driveto the array, expand array partitions, etc.
Another way to do the copy is to install the controller/arrayin a separate system that has a network connection, build the array and thenboot up a standalone OS (such as Knoppix), from a CD or USB stick, that canonce again be used to partition the array and run lftp. Once the array ispartitioned, lftp is used to mirror the file system over the network, onetop-level directory at a time. For example:
mount /dev/sda3 /media/sda3
lftp 192.168.1.1 -u root
mirror -a /bin bin
mirror -a /etc etc
mirror -a /home home
You probably only need to create a top-level directory for/boot, if your system mounts a separate partition (e.g. /dev/sda1) on/boot. In that case, you'll need to copy the /boot partition separatelybut, otherwise, you should copy the /boot sub-directory too. Either do:
mirror -a /boot boot
You should definitely not copy the /dev and /proc directories,instead just create their place holders (since the OS builds their contenton the fly):
We have successfully used this technique to migrate a runningversion of Linux to a new RAID array on a separate system. The onlydifficulty occurred when the /var directory was mirrored. Some of the lockfiles in /var/lock and /var/run could not be copied by FTP. The solutionwas to mirror the directory on the original system, in a user's homedirectory, and the copy the mirror. On the original system:
cp --preserve -R /var /home/jblow/var_mirror
On the system with the array, under lftp:
mirror -a /home/jblow/var_mirror var
Note that, for FTP to be able to copy all of the systemfiles, you may have to allow root to log in to FTP on the original system.If the original system is running VSFTPD, you need to hack the files:/etc/vsftpd/ftpusers and /etc/vsftpd/user_list and comment out the linewith 'root' on it, in each of the files. Then, restart VSFTPD:
If you need to set up a swap partiton, you can do so likethis:
Once all of the partitions have been created, you mustinstall the boot loader (Linux uses GRUB) on the disk. As above, you canavoid the trouble of setting up the boot sector with the boot loader byjust copying the first few tracks of the source disk directly with dd.However, since we've already set up the partition table, we must do thecopy very carefully to avoid wiping it out. First begin by copying theboot loader from the original system to a file:
dd bs=512 count=63 if=/dev/hda of=/home/joeblow/BootLoader.dump
Now, on the system with the RAID array:
lftp 192.168.1.1 -u root
dd bs=446 count=1 if=BootLoader.dump of=/dev/sda
dd bs=2 skip=255 seek=255 count=1 if=BootLoader.dump of=/dev/sda
dd bs=512 skip=1 seek=1 count=62 if=/BootLoader.dump of=/dev/sda
This copies the first 446 bytes of the first sector (a.k.a.the MBR), which contains the boot loader code. It skips over the next 64bytes, which contain the partition table. Then, it copies the two-bytemagic cookie that defines it as the MBR. Finally, it copies the next 62sectors (a.k.a. the DOS Compat Space), which contain the GRUB 1.5 code.Once that is done, you should have a bootable disk that is partitioned theway you want it.
You might want to test that the system can boot the disk(admittedly, it might not get far but you should at least see GRUB Stage1.5 running), before you proceed any further.
You should now go and tune up the grub.conf (or menu.lst)file in the /boot/grub directory, the /etc/fstab file, any networking filesthat will be affected by a change in the NIC (if you're using a new system),and any other files that may be need to get the sytem running.
And, if your system experiences a kernel panic, that says theroot and other file systems cannot be found, when the system is booted, itis probably due to the initrd image not containing the 3Ware device drivers.These drivers are included among the kernel modules by default, for all laterversions of the Linux kernel, but the initrd image from the original systemprobably didn't include them. This can be fixed by going back to theoriginal system and building a new initrd image (this must be done on arunning system) that includes the appropriate drivers. For the 7000-seriesRAID controllers, you should do:
/sbin/mkinitrd -v -f --with=3w-xxxx --with=scsi_mod --with=sd_mod
/boot/initrd-`uname -r`.img.new `uname -r`
For the 9500-series RAID controllers, you should do:
/sbin/mkinitrd -v -f --with=3w-9xxx --with=scsi_mod --with=sd_mod
/boot/initrd-`uname -r`.img.new `uname -r`
The new initrd image should then be FTPed to the newsystem with the RAID array and renamed to replace the existing initrd image.Your system should then boot with no problems.
As was mentioned above, the only thing that will touch thedisks in the array is the RAID controller. This can present a problem,should you ever wish to have a non-RAID-managed copy of the data (e.g.you'd like a regular, bootable disk that doesn't need the RAID controllerto run).
The procedure for doing so is basically outlined above in'Expanding An Array's Size'. Essentially, you cofigure a system with aRAID controller, with the array connected, plus a non-RAID (i.e. IDE orSATA) drive.
You boot up the system with your favorite copy software andproceed to create new partitions on the non-RAID drive. Then, you do apartition-by-partition copy, instead of copying the drive directly on atrack-by-track basis.
Once the copying is done, you may need to reinstall the bootloader, if the disk is supposed to be bootable. All the steps aredescribed above.