[GRLUG] VMWare ESXi Was: Raid, LVM, and Cheap Storage

Tue Oct 14 16:18:56 EDT 2008

Let me explain my experiences in some more detail for you Bruce :)

*Network Stack Corruption:*
The network stack is corrupted during the conversion only on the Virtual
Machine, the host is not affected permantly; HOWEVER when a guest OS runs
that has this problem it can affect (believe it or not) the hosts network
stack, and cause strange timeouts, once the guest is shut off the host
problems go away.
The only solution to this problem is to rebuild the GUEST os from scratch,
just creating a NEW VM.
Also, only Windows XP and 2003 GUESTS are affected that I have experienced.
*
Exchange 2003, and 2000 servers:*

*Exchange 2000:* EM64T/AM64 Hardware as the host is not detected properly by
the supported patches, processor instruction sets are not detected correctly
by the windows installation, causing problems when certain processor
built-in instructions are called such as ARCTAN (this happens when running
both 64bit/32bit)
My suggestion is to avoid all version 2000 anything for virtualization, I
have had too many problems to recommend it to anyone.

*Exchange 2003:* Prior to exchange 2003 SP2 the Mail Store Service attempted
to detect machine state with an API call, the API call used the CMOS Timer
in such a way that there was a time offset caused by vmware's emulated cmos
- this time offset caused the Mail Store to (in some cases) stop, this would
happen occasionally throughout the day.
This problem was fixed in Exchange Service Pack 2.
Exchange Mail Store Traffic over The Network:
Mail Store Traffic uses a non-routable, encapsulated packet. This packet is
encapsulated in another network layer packet to make it routable. For an
unknown reason after virtualizing the exchange server, the cisco firewall
started filtering local traffic that it was told not to filter (only mail
store traffic over VPN links was affected) - still no idea why, but we fixed
it by placing explicit rules about stateful inspection for the Ports /
Protocols the Mail Store used.
*
Sql 2000:* Sql 2000 uses a slow and ancient database engine that others
would say is awesome ... but besides that this database engine attempts to
write to memory in such a way that both EM64T hardware doesn't like, (which
microsoft has a work around for - they ship you a custom version of SQL 2000
after a month or so of bugging them) - But this problem is exaggerated with
the VM Abstraction layer - I've had continual problems with stored
procedures, and procedures built into dll's that will just NOT run - it
could be how the dll was compiled, but either way I would avoid virtualizing
SQL 2000 servers.
*
Software Virtualization Applications:*
Many individuals don't know this but all of the large companies out there
that have software virtualization engines or any form of Just in Time
compiling creates a problem for virtualization.  Java, .Net and Zend
Framework all had to add support for virtualization - they were just smart
enough to do it way before the technology was needed.  I am telling you this
because there are many smaller companies that produce Virtualized Compiling
Environments. Our overpriced and obfuscated accounting system is written for
this awesome APP: (I hope you can hear my hours of blood and sweat thats
culminated into this amount of sarcasm)
http://en.wikipedia.org/wiki/PROIV

This application just runs badly, really badly on any virtual platform, I've
tried them all - and there's lots others out there so beware.

*Domain Controllers:* (I will go into some detail here seeing you are
considering this)
Windows Domain Controllers states cannot be restored, they cannot be left
offline, snapshots cannot be used - any of this can cause a USN Rollback -
which is bad, and basically makes the server useless, untrusted by the
remaining domain controllers, and unsyncable.
This can happen if you leave the server connected to the network AFTER
performing your Conversion.
When you convert a DC (which I don't advise, it's better to rebuild it,
transfer Host Master Roles, and retire the old one just because of how
insanely tempermental AD really is.)

So when you convert a Domain Controller (always convert the dc's without
Host Master Roles first) follow these steps:
1.) Force one last replication with Active Directory.
2.) Unplug the server from the network
3.) Begin the conversion.
((If you ever intend to use this converted VM you can NEVER connect the
physical box to the network ever again until you format it, do not attempt
to unjoin it from the domain, just format it.
This is a no-going back process. ))

4.) Once the conversion is complete move the converted VM to a TEST server,
that is not connected to the domain.
Make sure the server starts up, check its logs, attempt to fix any drive
problems, and test its FILE performance. (copy some files around make sure
everything seems right.)
Pause the machine, start the machine, restart the machine etc....

5.) Once you verify all is well connect the host, to your network, let the
guest sync with Active Directory, and check for replication errors.
If you don't see any replication errors you are good.
If you don't have the server 2003 support tools installed, install them, and
run dcdiag after replication and look for errors.
Also run repadmin /showreps (make sure everything says SUCCESS)
Note: YOU MUST start the virtual machine within literally HOURS of
converting it, if a DC is left offline in a multi-site multi-dc environment
Microsoft states that problems can start in as little as 8 hours.
Note, if at this point the original server is ever allowed to connect to
Active Directory it will cause extremely annoying problems with Active
Directory. (I have been through this hell with several clients ...)

Info on USN Rollback:
http://support.microsoft.com/kb/875495  ( I have this KB framed )  haha
http://utools.com/help/UsnRollback.asp

On Tue, Oct 14, 2008 at 3:31 PM, Bruce Smith <blubdog at gmail.com> wrote:

> > For those that have not gone down the virtualization road -
> > Beware Exchange 2000 servers, Exchange Mail Stores without SP2, Domain
> > Controllers, and Conversion!
>
> Is related to the network stack problem below, or are you taking about
> different problems?  If so, what problems?
>
> I ask mainly because I was thinking of running converter on a couple
> Active Directory boxes (Win2003).
>
> > VMware Converter had a bug prior to version 3.01 - in which it corrupted
> the
> > network stack/winsock.
>
> Where was the corruption?
>  In the Windows box running the converter program?
>  In the resulting VM?
>  Or in the VMware server itself?
>
> > Once the conversion was done the damage could not be reversed - so
> beware.
> > The problem shows up more on Linux hosts, your network adapter (only when
> > loaded) will appear to go from maximum throughput to ~10kb/s over and
> over
> > again several times a second.
>
> And the fix was ... ?
> Did you have to rebuild the corrupt VM, or reinstall VMware server
> from scratch, or ... ?
>
>  - BS
> _______________________________________________
> grlug mailing list
> grlug at grlug.org
> http://shinobu.grlug.org/cgi-bin/mailman/listinfo/grlug
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shinobu.grlug.org/pipermail/grlug/attachments/20081014/93c1ae2b/attachment-0001.htm 