Hai

when a live migration of a VM happens from one host to another host assume that many services are running in that VM without stopping the services how the live Migration takes place What is the background process done while the vmotion takes place.

Please any geneous can answer this.

Need technical advise to rightly size the memory for the VM's.

Currently i have 7 * ESXi Hosts with total 1008GB RAM and 202GHz on 56CPU's

Inventory - 193 Virtual Machines with a total of 349 vCPUs and 1029 GB of vRAM.

As you can see the memory is over committed and below is the actual current memory usage% which is nothing but memory shared / installed to the VM's. Please correct me if i am wrong.

Host 1 - 75% , Host 2 - 71% , Host 3 - 91%, Host 4 - 79%, Host 5 - 74%, Host 6 - 88% , Host 7 - 80%.

Questions to clarify ---

> Do i need to reclaim the unused VM memory to keep the cluster stable ?

> With the current setup without memory reclaim - Can i add more VM's , Is it safe?

> What best solution to keep the cluster stable and future growth?

-------- Should i add one more hosts?

---------- Should i increase the memory in the ESX servers?

Hi,

when doing a vMotion for a high CPU usage / high RAM usage VMs, could it really happen that there is a downtime (while the VM is paused) of 5-10 minutes during the vMotion process of moving the VM from host A to B?

I am aware that there needs to be a short pause to copy the final memory delta, the CPU registers and to resume the VM, but normally that should take seconds, at most. I was attending a discussion and some people told me they frequently experience outages of 5-10 minutes when vMotioning very large VMs. The ESXi hosts in discussion have 2 CPU sockets, 8 cores each, 396 GB RAM, 2x 1 GBit dedicated user LAN NICs, 1x 1GBit dedicated for vMotion LAN.

Can anybody comment / share their experience on a) if downtimes of 5-10 minutes can really happen b) if there are common misconfigurations?

Thanks.

Best regards,

Andreas

Sorry if the title is vague, I am unsure how to properly word my question down to a single line. Anyways, on to my question, if I have two 16 core processors in a server, but I have 1 virtual machine that is setup with 4 cores, will the vmware only use 4 of the physical cores, or will the processing be distributed among all the cores? If searched the documentation but was unable to find much information about this specific situation. The reason I have it set to 4 cores instead of more is because the application I am trying to run only uses up to 4 core and if I can use vSphere to distribute out that processing it would greatly improve the performance. Thank you in advance.

Hi,

I have seen various other similar examples but so far not a solution for the problem I have.

I have several servers all connected to a distributed switch.  Normally they seem to work ok and do vmotions etc... without problems.

In order to ensure a consistant configuration, all my ip addresses, switches, interfaces, vmk are the same number on each server

My vmotion vmk is vmk3 on every server and has a vlan id of say 123 again on every server.

The uplinks are again the same on every server ie vmnic0 is dvuplink1 on each server etc...

vmotions have and do work but occasionally they fail at the 9% mark with the error

If I then SSH onto the servers I can ping the local vmotion vmk ie if on .11 I can ping .11 but can't ping .15 and vice versa.  If I delete the vmk from the server and recreate it with the exact same details it starts working again and then at some point in the future may fail in the same way.

We are running 5.0 u1 on each server, all built from same iso.

How can i change vSphere Web Client Default Port from 9443 to any other port in Windows Based vmware web client server after the installation of web client and in vCenter appliance as well?

Hello together,

i have a big problem with my migration function on our ESX 5.1 Cluster. We have two ESX-Hosts which are connected by iSCSI and using two SAN. Also there are two switches, so we have an redundancy on all parts of the cluster. The virtual machines working on the esxhost-A and have their vmdk and so on stored on the SAN-A. A replication makes different replications to the second host (esxhost-B). Now i had to increase the RAM in both of the esx host without have a downtime. My idea was to shut down esxhost-B at first (there are no virtual machines working) and increase the RAM. Than start esxhost-B and migrate all machines from esxhost-A to esxhost-B. All works fine and fast. After that i increased the RAM in esxhost-A and started it up. Now i wanted to migrate all the machines back to esxhost-A. And this doesnt work

If i try to migrate a machine back to esxhost-A there pop up an error after 65% of migration:

I cant increase the switchover time because i have to shut down the VMs and for that and thats no possibility for me.

Can anybody help me out?

Sincerely,

Thomas G.

-------------

edit 21.08.2013: I added an attachement with a summary of the hostd.log. It is the time within the error occures.

edit 22.08.2013: added new attachement with the vmware.log Nobody can help me?

I have two servers 1) Dell 710 Quad core Xeon processor 5500 or 5600 series 2) Dell 720 Intel Xeon Processor E5- 2600 working in cluster , but i am about to buy another server Dell 730 Intel Xeon Processor E5 2600 v3 family that  need to add to the cluster. is there any form to know if the processor is compatible for the cluster (EVC). Thanks

I'm experiencing an issue with a VM that once lived on the VSAN, but was moved to another VMFS datastore and now when I attempt to move it back I recieve the error "Operation failed, diagnostics report: Hostsvc::osfs::CreateDirectory : Failed to create directory "virtual machine name" (Cannot Create File)".  When looking at the Cluster/ Monitor/ Virtual SAN/ Virtual Disks, I can see that the VM still resides in the list.  But when I click to see what host and drive the "home" and "Hard disk 1" lives on, there is no information.  I'm assuming that I cannot migrate this VM to the VSAN because of some residual folders or files from when it once lived there.  

Is there a way to find this VMs location as it once existed in VSAN and manually remove it?

Hi,

We're trying to use L3 vMotion in a 5.5 environment (our sandbox). Besides adding a static route on each ESXi host for the other vMotion subnets (on the vmk port assigned to vMotion). What else do we need to do?

We're able to vmkping (with the vMotion vmk port) to the gateway on the other subnet... But not able to vMotion between them?

what are all the possible negative impacts of using mandatory vm to host affinity rules?

i.e.

Can't put host in maintenance mode without powering off vm

others?

when i'm migrating a vm from shared storage ("change host" option) it's fine. all the vmnics are working

when i'm migrating a vm from host to host, from local storage to local storage, only NIC0 is working in both ends (change host and storage option via web gui)

the system is a trial version of esxi 6 (both hosts) and also vcenter 6 trial

anything I can do about my problem? or it's a fact that I have to live with?

thank you

Hey everyone. I'm new to a company that has 2 vCenter servers, one in each datacenter, and we have some standa-alone servers (not clustered) in some locations.

All the hosts are licensed with Enterprise plus. Let me draw the locations so you'll have an Idea:

VCenter1

-Datacenter1 (remote dc 01)

** Host1

...

** Host15

VCenter 2

-Datacenter2 (remote dc 01)

** Host16

...

** Host30

-Datacenter 3 (Local network)

**HostLocal1

**HostLocal2

That make sense, as I have 15 hosts on Datacenter 1 and 2. We use VCenter 2 to manage HostLocal1 and HostLocal2, but we don't cluster them together and we don't vMotion among them, because all the vMotion traffic would have to go to vCenter and Then to the second host, is that right? Is there a way to prevent that?

Let me rephrase that: The VMotion traffic would be like this:

HostLocal1 --> VCenter 2 ---> HostLocal2

Or:

HostLocal1 --> HostLocal2.

Writing this question down make me realize that my assumption is incorrect, vMotion traffic has nothing to do with VCenter, traffig flows from those 2 servers. Can anyone confirm?

Thanks a lot for your time.

We have a Bladcenter with management IPs of 10.1.42.x and vMotion IPs of 10.1.47.x with a 10.1.42 gateway.

Then we have standalone hosts with management IPs of 10.1.43.x and vMotion IPs of 10.1.48.x with a 10.1.43 gateway.

To make it easier to see (hopefully)

Bladecenter hosts                             Standalone hosts

Mgmt .42                                            Mgmt .43       

vMotion .47                                         vMotion .48

Gateway .42                                       Gateway .43

I know I can’t vMotion between the 2 because they have to be on the same subnet but when I change a Bladecenter host vMotion IP from .47 to .48 it still doesn’t work. Is there something else that needs to be changed? I can vMotion a VM while it’s off between the 2 without changing any IP settings but not while it’s running.

Could anyone please describe the roles and responsibilities of Level 2 and Level 3 VMware administrators ??

My organization has a vSphere environment with 6 clusters and a variety of VMs (Windows 2K3+, Linux, Solaris etc)

I have been tasked (asked) to answer a seemingly simple question: is there enough capacity to upgrade the Windows 2003/2008 virtual machines to Server 2012 R2?

I'm no architect, but I've been given a list with the VMs that they want to upgrade and a report with oversized VMs (from vRealize Operations Manager). Looking at said report there are many oversized VMs, and some are extremely oversized (eg. 8-core CPU when 1 CPU might be enough). So just from this report I think there is no problem to go ahead with the upgrades, because we can easily reclaim tons of resources.

However, because this is a big decision, I can't just say "yeah I think its doable". I need to provide a serious analysis.

But how do I go about it? Initially I was thinking about doing some calculations from a "GHz" perspective, as in how many extra GHz do I need and how many I have "free" (or am able to free through resource reclamation with vROPS).
But then, CPU usage might not change that much since the applications remain the same (right?). Maybe I should look at how many cores each VM uses? However it seems to me that Win Server 2012 R2 has some pretty low minimum requirements, 1x 64bit CPU and 512MB RAM whereas all the VMs have much more than that..

And what about RAM? Should I assume maybe +512MB when upgrading from Windows 2008 machines and +1GB for Windows 2003 ? Would that make sense?

What do you guys think? Any suggestions would be appreciated, including tools, scripts etc.

I'm having some trouble to figure out vMotion (without EVC) is possible between hosts with X5600 and E7-2800 CPU's. Both CPU's are from the Westmere Generation. I've read the vMotion CPU compatibility guide and both CPU's are in group E. They have the same CPU features according to Intel spec sheets:

Intel® Xeon® Processor 5600 Series

Intel® Xeon® Processor E7-2800 Product Family

I'm thinking that vMotion is possible without EVC because they are from the same Westmere generation (EP and EX). Our supplier however tells me that i need the same CPU series for vMotion, in this case a machine with an E7 CPU to replace the X5600 host.

So is my supplier right or wrong? Do i need to buy a second server with an E7 CPU for vMotion compatibility or can i keep the servers with X5600 CPU's buzzing?

Hi there.

I have 5 esxi nodes. All with 5.1 update 2.

I'm having issues with NUMA scheduler. Poor balance.

Let me go directly there.

I'm playing (doing tests) now with one node. Just one.

Dell r910, 4 opteron --> 4 numa nodes: each --> 6 cores + 64 GB ram. Total: 24 cores + 256 GB ram.

10 VMs. VMs cores: 4,4,4,4,2,2,1,1,1,1, respectively. Very well sized. Each one of them use 80-90% of those vcores.

No under or oversized situations. Memory, between 1 and 16 GB. No problem with memory. The issue is strictly CPU related.

Ok. ESXTOP + m + f. Numa statistics.

Numa nodes and cores (VMs)

0          4,4,1            9 cores (!!). Terribly balanced. The 3 VMs have high CPU ready times.

1          2,2,1,1          Completing numa core count. 6 cores. Ok here.

2          4,1                5 cores. Ok here.

3          4                  4 cores. Ok here.

So. Why?

I waited for an entire day and the VMs stay there. No new rebalance. Nothing.

So, i fix it manually. I move the VMs between nodes using resource settings --> advanced memory / cpu (specifying numa node and cpu affinity*).

* Fact : I've read on the official documentation that specifying only the numa node on advanced memory, does not work. You need to specify the CPU affinity too.

So, for example, for numa node 1, cpu's are: 6,7,8,9,10,11. I specified: 6-11, which is the same

The VMs move instantly.

Result on esxtop:

0          4,1,1

1          4,1,1

2          4,2

3          4,2

Excellent. That's balance. VMs on each numa node; completing the 6 cores per node.

Yes, of course: memory locality 97-100%. Every time. No problem there, like i remarked at the beginning.

CPU ready time dropped to 0-50 ms on every VM. Perfect. Before, we were talking about 2000 ms - 5000 ms (!!!).

Another fact:

I've read that once a VM is part of a new ESXi host (by automatic vmotion for example), the scheduler considers the VM memory.

It puts the VM on the first numa node which has enough free memory to hold the VM. That's all.

It does not care about the core amount. That can deliver poor CPU performance in the short term.

Now, after one hour, i removed, from each VM, every setting from advanced settings (affinity matters).

After another hour, i checked CPU ready times on each VM. All doing fine, but 2.

I went to ESXTOP. AGAIN. NUMA nodes IMBALANCED.

One of the numa nodes had enough VMs for 7 cores and another, 8 cores.

Why !!!

So. What i'm doing right now and from now on?

I do manual balance and then --> ESXi host --> software --> advanced settings --> numa.RebalanceEnable=0.

The VMs stayed right on the numa node i put them.

Excellent CPU ready times up to now.

2 questions:

1) Is there a way to fix this using one or more of the NUMA advanced attributes? I want the VMs to be positioned on each of the NUMA nodes, taking as reference / considering each VM core count, too, not only the memory!! It' s obvious and essential !!! Otherwise, you experience the obvious bridge cross (that's how i call it) between physical cores; adding latency. Instant latency. I want each VM to stay on one numa node. No remote memory or remote CPU !!

2) Is this, in some way, totally fixed on vSphere 5.5 ? Is numa balancer/scheduler better? is quite frightening.

Thanks !!!!!!!

ps: The "again" on the subject is version related. I've seen NUMA poor balancing issues on other discussion threads, for vSphere 4.1 and 5.0.

My Customer has existing cluster with 4 blade servers (Intel Xeon E5-2670 2.6 Ghz 8 Core Processor) and want to add one more server in the same cluster. A partner has proposed Intel Xeon E7-2880v2 2.5 Ghz 15 Core Processor. Customer wanted to know: Is the vMotion possible between the 2 processor generations ? Will there be any compatibility issue having the 2 servers in same cluster ?

Thanks

I’m hoping someone in the community is running a cluster with Intel v2 and v3 processors and can duplicate this issue.

The issue:

Virtual machine workload is impacted in performance by approximately 40% when vmotioned between a VMware ESXi 5.5 host with an Intel v2 processor and one with an Intel v3 processor.  The performance is normal if the vm powers up and remains on a v2 or v3 Intel host.  As soon as the vm migrates to a different Intel family (v2/v3) performance is impacted.  For example if the vm powers up on a v2 and then migrates to a v3 performance is impacted.  If the vm powers up on a v3 and then migrates to a v2 performance is impacted.

How we discovered  this issue:

We recently purchased Dell M630 PowerEdge hosts with the E5-2697 v3 processors.  We placed the new hosts in a cluster set for EVC mode of Ivy-Bridge with existing M620 hosts running E5-2690 v2 processors.  We discovered that SQL workloads migrating to the new v3 hosts were be impacted by about a 40% loss in performance when they moved from the v2 they were powered up on to the v3 hosts.  As soon as we vmotioned the workload back to the v2 processor host the performance was fine again.  We could also power cycle the vm on the v3 host and performance was fine until we vmotioned the vm to a v2 host.
 

How we duplicated this issue:

We made a clone copy of our Windows 2008 vCenter database server running Windows 2008 SQL R2.  We then disabled the NIC and created the query below. Timing the query revealed a 40% loss in performance when we vmotioned the VM from a v2 to a v3 Intel host. Power cycling the VM on the v3 host put the query back to the same performance level as it was when it powered up on the v2 Intel host.

This is the first time we have encountered issues moving a vm between different Intel processor families.  The impact is so severe that our only choice is to build a new cluster of Intel v3 VMware hosts and cold migrate any workloads to this new cluster.

DECLARE @i int= 0

WHILE @i < 50 BEGIN

SET @i = @i + 1

IFOBJECT_ID('[ServerVCDB].[dbo].A_HIST_STAT1')ISNOTNULL

DROPTABLE [ServerVCDB].[dbo].[A_HIST_STAT1]

SELECT*

INTO [ServerVCDB].[dbo].[A_HIST_STAT1]

FROM [ServerVCDB].[dbo].[VPX_HIST_STAT1]

IFOBJECT_ID('[ServerVCDB].[dbo].A_HIST_STAT10')ISNOTNULL

DROPTABLE [ServerVCDB].[dbo].[A_HIST_STAT10]

SELECT*

INTO [ServerVCDB].[dbo].[A_HIST_STAT10]

FROM [ServerVCDB].[dbo].[VPX_HIST_STAT1_10]

END