Generally, I like to tell a story when I make a post. Today's story is about running out of space on our production mail server. Here is the background. We're running Novell Groupwise on a SLES10, OES2 server serving a small business. There are about 70 mailboxes in the system. We're also highly virtualized, so the server is a virtual machine running as a guest on VMware ESX.
There are two virtual hard drives. The 1st contains the OS, and the 2nd is in a Logical Volume Group. Turns out that was a lucky choice. The 2nd drive was originally provisioned as 110 GB. I knew the drive was getting full, but the last time I checked it had a few GBs free. Yesterday, some urgent system messages were sent to the administrator email address, informing me that the Quick Index couldn't run because the remaining space was 0000 MB. Oops.
I shut down GW after notifying all the employees, then I shutdown the server. While the server was down, I made a clone just in case everything went badly. I also increased the size of the 2nd hard drive to 180GB, while thinking hard about imposing mailbox limits next week...
Now, I mentioned that the whole hard drive /dev/sdb was in the volume group (VG). This made expanding the size a piece of cake and made me really appreciate the use of LVM.
I restarted the server. Once it came up, I stopped Groupwise. Then I took the 1st steps toward increasing my storage.
gw8:~ # rcgrpwise stop
gw8:~ # pvresize /dev/sdb
gw8:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/sdb system lvm2 a- 180.00G 70.09G
gw8:~ # lvextend -L +70G /dev/system/data
Extending logical volume data to 179.90 GB
Logical volume data successfully resized
gw8:~ # resize2fs /dev/system/data
resize2fs 1.38 (30-Jun-2005)
/dev/system/data is mounted; can't resize a mounted filesystem!
gw8:~ # umount /dev/system/data
umount: /opt/novell/groupwise: device is busy
umount: /opt/novell/groupwise: device is busy
gw8:~ # fuser -m /dev/system/data
/dev/system/data: 4977 10649e
gw8:~ # ps -ef | grep 4977
root 4977 1 0 17:44 ? 00:00:01
/opt/novell/eDirectory/sbin/ndsdroot 16007 10972 0 17:58 pts/1
00:00:00 grep 4977
gw8:~ # rcndsd stop
Executing customized settings before stopping the Novell eDirectory
server...
Stopping Novell eDirectory server...
......................
done
Executing customized settings after stopping the Novell eDirectory
server...
gw8:~ # ps -ef | grep 10649
root 10649 1 0 17:49 ? 00:00:00
/opt/novell/groupwise/agents/bin/gwmon
@/opt/novell/groupwise/agents/share/monitor.xml
root 16806 10972 0 17:59 pts/1 00:00:00 grep 10649
gw8:~ # rcgrpwise-ma stop
Shutting down GroupWise Monitor Agent
done
gw8:~ # umount /dev/system/data
gw8:~ # resize2fs /dev/system/data
resize2fs 1.38 (30-Jun-2005)
Please run 'e2fsck -f /dev/system/data' first.
gw8:~ # e2fsck -f /dev/system/data
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/system/data: 630763/14417920 files (22.5% non-contiguous),
27427038/28810240 blocks
gw8:~ # resize2fs /dev/system/data
resize2fs 1.38 (30-Jun-2005)
Resizing the filesystem on /dev/system/data to 47160320 (4k) blocks.
The filesystem on /dev/system/data is now 47160320 blocks long.
gw8:~ # mount /dev/system/data
gw8:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 14G 7.2G 6.4G 53% /
udev 503M 136K 502M 1% /dev
/dev/mapper/system-data
178G 103G 66G 62% /opt/novell/groupwise
Here's what was supposed to happen. I used pvresize to tell LVM that the physical volume had grown. Then I extended the logical volume by 70GB to take advantage of the larger hard drive. Then I tried to extend the filesystem over the new space. That didn't work immediately.
resize2fs gave me a warning that it couldn't work on a mounted drive, so I tried to unmount the drive. The command complained that the device was busy. I knew Groupwise was down, so I used fuser to figure out what processes were using /dev/system/data. Once I identified the processes as eDirectory (NDS daemon) and Groupwise Monitor, I shut them down and then was able to unmount the volume.
Again I tried resize2fs, but this time it warned me to run e2fsck to repair any damage to the file system. Ok, ok. So I ran e2fsck as instructed, and then finally resize2fs.
Remounted the volume and checked to make sure that the new space was really there... And I left for home with a nice successful feeling.
Comments