[GRLUG] Raid, LVM, and Cheap Storage

[Ben DeMott]

Hey GRLUG I wanted to open a discussion on storage, and share my thoughts,
opinions, and observations about the storage world, and get some input.
I hope you find my observations somewhat thought provoking, and I look
forward to hearing different opinons and experiences.

*Introduction:*
Disk storage, disks themselves are rather cheap.  In Linux this is furthered
by the fact we have LVM2 and Software Raid support in our Kernels.
This combined with Hardware Raid makes the combinations of disk / storage
setups just about endless.

I'm curious to know about other peoples experience with LVM2 / Virtual File
Systems, and Raid and how they have combined these technologies together to
serve a particular storage environments needs and purpose.

This isn't a forum about 'boxed solutions' - If you feel you must share on
how IBM or HP does it fine - but keep it brief.
This is about actual managed solutions, that you don't depend on a vendor
for.

Some of my biggest problems with the commercial storage world:
They oversell performance in applications where its not needed, sacrificing
overall storage space, or simplicity.
They adopt new disk technologies (SAS) too quickly and too widely when it is
NOT the best solution for everyone simply to keep their profit magins higher
(suprise, suprise)

*My Experience:*
I'm all about saving money, if I can buy 3 servers for the cost of 1, at the
cost of more time, I'll do that - solutions that cost half of my salary
can't ever justify themselves for being 'managed' - I'm sorry that argument
just doesn't apply to most medium sized businesses. I call that the 'lazy IT
department argument'
In my consulting work I've discovered that most IT managers often don't know
that much, or don't force themselves to keep up in technology, I can't tell
you how many places I've been to that had IBM or HP or Dell servers that
were setup by someone else - that weren't even configured for RAID!!!!!! -
If you do it yourself you know it's done right, now obviously this can't
always be the case but it's worked for me so far.

When I started at my last employer they were paying $1,200 a month to
offsite 500gb of data a week. If this is the best solution you can come up
with you just lack an imagination...
I could be throwing away hard drives in the dumpster after each use and
still have a better/cost effective solution.

Layering Abstraction technologies:
Scenario -
 Raid 1+0
 LVM2
 ISCSI Partition
 Contains a Virtual File System.

The only situation in which this is a good idea, is if you are mirroring the
data to another Raid array.
So much can go wrong to corrupt data in this scenario that the complexity
can outweigh the benefits very quickly.

*Why Mirror:*
It's true, mirrors lack the performance of other raid striping methods,
however:
GMTD - "Give me the drive" is a philosophy that says the physical disk drive
must always have value nomatter what, mirroring supports this.
When you go to a striping methodology during a read more drives are being
accessed simultaneously, this has consequences of its own.


*Raid Controllers:*
My experiences with raid controllers and their interfaces vary.  Many raid
controllers have batteries on the controller itself, so in the event of a
hardware failure the controllers buffer can be cleared and the raid array
will not be damaged by incomplete writes.
My experiences with rebuilding Raid arrays or restoring data from a crashed
raid array are all painful.
Does anyone have a good story or know of technologies that allow you to move
hardware or a portion of a raid array to restore data on a different piece
of non-identical hardware in the event of a server failure?

*Raid Thoughts:*
Hardware RAID has become a standard amongst servers, for redundancy,
scalability, and uptime reliability.
In my journeys as an Engineer I've encountered more failures due to human
error, improper setup, bad hardware, not verifying data itself, the list
goes on.
In the end I've come to realize the more ideal solution is being hardware
agnostic, and the best way I've accomplished this is through a combination
of Software Raid, DRBD, and LVM - no physical raid.
At the end of the day a raid array can not guarantee anything about data
integrity, it is only a small piece of the puzzle, and often a convoluted
one when the day comes that you need to rely on it.
Instead we need to focus on the data at the data level more than we do, we
need to use checksums and data comparison strategies while backing up and
restoring data, this is often completely overlooked at many organizations.
Corrupt data being mirrored is no more valuable than no data at all.
And keep in mind our great Linux kernel caches/buffers disk transactions -
if you have 36GB of memory on a server the majority of that will be consumed
with buffered data, this offsets many performance concerns, and adds to the
longevity of hardware.
*
Preventative Maintenance:*
These are the things I encounter most that amaze me about IT departments:
1.) The Vendors hardware monitoring software is not installed.
2.) The only way of detecting a disk failure at a remote site is for someone
to notice there is a 'red' or 'amber' light on.
3.) No one monitors system voltages
4.) No one monitors system temperatures.
5.) No one performs read / access tests or write tests or head tests - if
this is just done once a year you can almost always predict a drive failure.
6.) There is too much confidence in hardware

*Managing Storage:*
Storage isn't just about picking good hardware to put data on - it's about
weighing the uses and cost of your data and having a ready made solution
that is bundled around what that data is for.
Is the data application data
Is the data user data
is the data Corporate data
is the Data for a hardware system (operating system)

Each type of data has it's own unique needs -
-archival importance (how far back to revision, this type of data can be
compressed, and is less important for FAST restoration)
-security importance (should the data be encrypted when transported to a
remote site, should you limit departmental access)
-User data (The data can be split up, and managed seperately from other
forms of data, an individual user being affected is less important than a
server going down)
-application data (matters more about availability and restorability than
archival)
-Hardware data  (once again, restorability and place in time backups /hot
standy are more important than continual backup and archival)

Many organizations don't recognize these different requirements of their
data, and fail to develop different storage plans around the purpose of
data.

Now I've pointed out all of the short sightedness I've experienced in my
journeys - I'm sure the Linux IT community being as informed as it is will
not be among these organizations I speak of - so I'll be interested to hear
about your experiences and setups and uses of linux technology to come up
with unique and cost effective storage environments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shinobu.grlug.org/pipermail/grlug/attachments/20081013/357aa374/attachment-0001.htm