Measure the Performance of the Hyper V Using Performance Monitor

When we receive call from users about the slowness of the server, we always look into Task Manager.This is  not applicable when you measure the performance of the Virtual machine.
Therefore i remind visitor do not use "Task Manager".

In order for you to correctly measure the Virtual Machine, you can use Performance Monitor which available on Hyper V Host machine. Configure Data Collector Set and enter below counter to collect the report.

i) CPU Utilization:-

a) For Physical Server
Hyper-V HyperVisor Logical Processor (*)\% Total Run Time
Sample of a Hyper V physical host:-

b) For Virtual Machine
Hyper-V HyperVisor Virtual Processor\% Guest Run Time
- select vcpu of the virtual machine.
Sample of a virtual machine with 2vcpu:-

Sample of DPM VM with 4vcpu:-

ii) Memory Availability
a) \Memory\Available Mbytes

iii) Disk Utilization
a) \Logical Disk (*)\Avg Disk Sec /Read
b) \Logical Disk (*)\Avg Disk Sec /Write

Sample of Virtualization host:- memory and disk utilization

Look nice the graphic but sometimes not 100% accurate. Refer to the average value and below benchmark to interpret the result of the performance collection.

CPU Utilization

Healthy : Less than 75%
Warning : Greater than 75%
Critical : Greater than 85% (need to investigate)

Memory Availability
Healthy : Greater than 10% of RAM
Warning : Less than 10% of RAM
Critical : Less than 100MB

Disk utilization
Healthy : Less than 10 ms (0.010)
Warning : Greater than 15 ms (0.015)
Critical : Greater than 25 ms (0.025)

[Updated]- 18 Oct 2011
Network monitoring
Use  \Network Interface (*)\OutputQueue Length
Health :- Less than 1 on average
Warning :- Above 1 on average
Critical :- 2 or more on average
If you would like to learn more, please refer to http://technet.microsoft.com/en-us/library/cc768535(BTS.10).aspx

Get Support from Microsoft PSS for Web Server Issue

Last few days ago, i have migrated a web server from physical server to Hyper V. The specification of the existing server is 4 CPU - Intel Quad Core with 32GB RAM. According to MAP, the CPU utilization is less than 10%. Then we decided to move this server to Virtual Machine. Once migrated this web server to Virtual, the CPU utilization spike to 90-100% most of the time. Due to web server is running on Windows Server 2003, i can only set to 2 vcpu as supported by Microsoft.

Nothing much to do on my side and application team could not identified the problem. So i have decided to lodge a case with Microsoft PSS to find the root cause. The purpose of this article is to share some step that you should take if you would like to report issue to PSS on Web Server issue.

1. Go to https://partner.microsoft.com/global/supportsecurity/40032273 and call the toll free number according to your country.
2. Fill up incident template which send by PSS and a SR number will be given
3. Collect IIS hang dump. To do this follow below instruction

a)Download and install the Microsoft Debuggers
Download and install the MS Debugging Tool from http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.11.1.404.msi
NOTE: You do not have to install them on your server, but can install on a workstation and then XCOPY the directory to your server. They will work on your server after you have done this without rebooting the server. This debug tool work fine when running on IIS 6.

b) Run it when the high CPU issue occurs.
Run AutoDump Plus from a command line (cmd.exe). In the debuggers directory (Default is c:\Program Files\Debugging Tools For Windows) run the command (changing the path where you would like the files created in):

cscript.exe adplus.vbs -hang -iis -o c:\outdirhang –quiet

Please ignore any warning and collect the hang dump for 3 times with 5 minutes interval between each time.

4. Send event log.
a)Go to Run and type “eventvwr”
b)Right click the Application (Security, System) and click Save Log File As…The Save as type is .evt.

5. Send IIS log
a) Go to IIS manager MMC.
b) Right click the problematic web site and click Properties.
c) Under Web Site tab, click Properties and find the “Log file directory”

6. Compress the event viewer, IIS Log and Dump file as the size is quite huge. Microsoft PSS will send you the workspace to upload the file.
 
7. Once they receive the log and dump file, they will begin to analyse it.
 
There are two Severity when report a case, by default
a) Severity A - Support after 6pm
b) Severity B - Support  from 9am to 6pm.
 
You should raise the severity if require. Once the severity is raise from B to A, you need to pay for the support.
 
Finding from PSS:-
a) Application coding issue especially on string operation issue which causing the CPU to high spike.

Now the ball is pass back to Application team to resolve the issue on the coding side.
On my side, i just perform fine tune on the application pool to Recycle every 1 hour.
At this moment, with 400+ people access to the 2vcpu and 5GB Memory Virtual Machine running on Hyper V platform, the user experience is still good. The Average CPU utilization is 60%.
Not bad when you convert 4 Quad Core (16 cores) CPU and 32GB of RAM into 2 vCPU with 5 GB RAM.

Cheer for Hyper V! 

In my next article, i will share some performance counter which i use to monitor.

Stay Tuned.

Should I Enable Hyper Threading in Hyper V?

Recently i have deployed a Hyper V R2 on Dell R710 with 2 Quad Core CPU. That's mean i will get
2 X 4 cores = 8 cores. Then i decided to enable Hyper Threading on the server. Guess what! now i got 16 cores after enable Hyper Threading. Well, not a big deal. That's what HT is for: it double up your CPU cores.

The next question that pop out from my head is

"Will Hyper Threading affect Hyper V performance?"

After surfing for a while on this question, i managed to find a page which discuss about this question, please refer to http://www.windowsitpro.com/article/virtualization/q-does-hyper-threading-affect-hyper-v-.aspx

As a summary, Hyper Threading will never hurt performance but aid performance. So just go to BIOS and enable Hyper Threading to enjoy extra cores and performance.

FYI, Hyper-V R2 now supports up to 64 logical processors (cores or hyper-threaded CPU's).

Understanding Microsoft Virtualization Solutions (second edition)-Ebook

Found this free ebook while surfing: "Understanding Microsoft Virtualization Solutions: Second edition" which updated for Windows Server 2008 R2.

The ebook included with six chapters:-
Chapter 1: - Why Virtualization
Chapter 2:- Server Virtualization - Hyper V
Chapter 3:- Local desktop virtualization - App-V, MED-V, XP Mode
Chapter 4:- Remote desktop virtualization- App-V on RDS, VDI, RDS
Chapter 5:- Virtualization Management - VMM
Chapter 6:- Cloud Computing

480 pages is worth to have. Please go to http://blogs.msdn.com/b/microsoft_press/archive/2010/02/16/free-ebook-understanding-microsoft-virtualization-r2-solutions.aspx to get the ebook in xps or pdf format.

Enjoy!

Event ID 1584 -A backup application initiated a VSS snapshot on Cluster Shared Volume without properly preparing the volume for snapshot

You will see this error when you run DPM 2010 to backup CSV. On the DPM 2010 console, the backup is successful and can restore VHD backup. However on the Event Viewer, you will see Event ID 1584 is logged.

The reason why i saw this is because
a)i did not purchase premium key for MD3000 (shared storage) to support snapshot
b) do not have additional hard disk space to allocate for snapshot. It require 20% of free space on the storage
c) Besides that, i do not install hardware VSS provider to support parallel backup. (default:- 3) or no hardware VSS provider is provided.

To solve this problem, you need to enable serialization backup of CSV node.Thanks to Robert which blog about this issue. (You can view the solution from here as well:- http://robertanddpm.blogspot.com/2010/07/enabling-serialized-backup-of-hyper-v.html )

Here is the summary to resolve this issue:-

a) On DPM server, modify the registry (change data from 3 to 1)
Key : HKLM\Software\Microsoft\Microsoft Data Protection Manager\2.0\Configuration\MaxAllowedParallelBackups
Value: Microsoft Hyper-V
Data :1
Type: DWORD

This will ensure that only one backup job will run at a time on a Hyper-V host.

b) Share the folder C:\Program Files\Microsoft DPM\DPM\Config with the name DPMConfig. Assign domain account as Full right.
 
c) Save the following script as "DSConfig.ps1" and dump to C:\Scriptps (create this folder on the cluster node). Just modify the DPM FQDN.
Note:- You can get the script from here:- http://www.hodgkins.net.au/wp-content/uploads/downloads/2010/03/ListVMsOnClusterForDPM.zip
 

# List Hyper-V Virtual Machines On The Cluster

# Original Script From DPM2010BetaProtectingHyperV.docx
# Updated and Improved by Matthew Hodgkins
# http://www.hodgkins.net.au/

# Please Specify The FQDN or IP address of your DPM Server
$DPMServerName = "DPM FQDN"

# DSConfig.ps1
$infoText = "This script will generate the DatasourceGroups.xml file in the current path. Once this file is created merge it with the same file name under %programfiles%\Microsoft DPM\DPM\Config directory on the DPM server. Read the documentation for more details."
echo $infoText
$header = " `n ""
$footer = "
import-module -name FailoverClusters
$dir = [guid]::NewGuid()
md $dir
$cluster = get-Cluster
$FQDN = $cluster.Name + "." + $cluster.Domain
$res = get-clusterresource
where-object { $_.ResourceType.Name -eq "Virtual Machine Configuration"}
foreach ($r in $res)
{
$VmObj = Get-ClusterParameter -inputobject $r
where {$_.Name -eq "VmStoreRootPath"} # Identifies the CSV volume on which the VM is hosted.
$VmName = Get-ClusterParameter -inputobject $r
where {$_.Name -eq "VmId"}
$vol = $vmobj.Value.Split("\")[2] # $vol will return to us the Volume of the CSV on which the VM resides.
$line = ""
echo $line >> $dir\$vol # File VolumeX will contain entries for all VMs hosted on CSV VolumeX
}

echo $header > DataSourceGroups.xml
$filelist = dir $dir\Volume*
$GroupEndString = ""
foreach ($file in $filelist)
{
$GroupBeginString = "" # Group name is kept VolumeX itself
echo $GroupBeginString >> DataSourceGroups.xml
type $file >> DataSourceGroups.xml # Consolidating groups pertaining to all the volumes.
echo $GroupEndString >> DataSourceGroups.xml
}

Remove-Item -Force -Recurse $dir
echo $footer >> DataSourceGroups.xml
# Copy The DataSourceGroups.xml file to the DPM Server
copy DataSourceGroups.xml \\$DPMServerName"\DPMConfig"

d) On one of the CSV node, use powershell to execute

set-executionpolicy remotesigned
Type - Y
./DSConfig.ps1 (to execute the script)

e) On the DPM server, check and make sure file DataSourceGroups.XML is listed in C:\Program Files\Microsoft DPM\DPM\Config. If you edit the file "DataSourceGroups.XML", you can see all the Virtual Machine on your cluster is listed in the XML file.

f) Finally change the protection group to backup one VM at a time.

p/s:-If your shared storage can support hardware VSS provider, remember to revert the above changes.

That's all. Cheer, Lai

Create a Faster Base image using WIM2VHD and VHDTool

Most of the time we need to create a base image when setup virtual machine. How i normally do this is using Hyper V, install an operating system and run sysprep command? But this process is taking quite a long time especially the duration to setup Windows Server 2008 R2 will take around 30 minute.
Now the question is, how to do it fast within 5 minute?

In this article, i will use Wim2vhd tool. In order to use Wim2VHD, you need to install WAIK for Windows 7. You can get WAIK from this link:- http://www.microsoft.com/downloads/details.aspx?FamilyID=696dd665-9f76-4177-a811-39c26d3b3b34&displaylang=en

Once you have install WAIK, download WIM2VHD from here:- http://code.msdn.microsoft.com/wim2vhd/Release/ProjectReleases.aspx?ReleaseId=2603
It is a small 103K file. Then put this into a folder.

Step 1:-
To execute, go to WAIK- Deployment tool Command prompt.

Step 2:-
Enter the following command to view the parameter. Just like a help syntax
"cscript WIN2VHD.wsf"

Example:- Create a base image Windows Server 2008 R2 Standard Edition

/wim:h:\sources\install.wim
Is refer to the location of the install.wim which located in DVDROM

/sku:serverstandard
Is refer to Windows Server 2008 r2 Standard Edition

Additional /sku parameter:-

Here is the process of the snapshot:-

The tool will create a dynamic disk with default 40GB size. If you want, you can use additional parameter such as
a) /size:51200
To create 50GB partition

b) /disktype:fixed
To create fixed disk vhd. The option available is dynamic, fixed and fast fixed.

Example:- To create a fixed disk of Windows Server 2008 R2 Standard Edition with a size 50GB
C:\WIM>cscript WIM2VHD.wsf /wim:h:\sources\install.wim /sku:serverstandard /disktype:fixed /size:51200

The above command will take almost 15minute. In order to speed up the creation of 50GB fixed disk, i download VHDTool (Download from:-http://code.msdn.microsoft.com/vhdtool). Put this tool inside the WIM2VHD folder and execute below command

C:\WIM>cscript WIM2VHD.wsf /wim:h:\sources\install.wim /sku:serverstandard /disktype:fastfixed /size:51200

Now, you can create 50GB fixed disk in 5 minute. Quite a useful tool and it's FREE!

Check out both tool to explore more cool stuff.

Free HyperV Performance Monitor Tool

I'm looking around for Hyper V Performance Monitor Tool. Found a tool provided by ManageEngine.

It allow you to monitor

• CPU Utilization
• Memory Utilization
• Disk read/Write
• Network Rx / Tx

for physical Hyper V host and Guest Virtual machine. Guess what - It 's Free and allow you to monitor up to 2 Hyper V host. Just download 1.2MB file and install into another machine.

Once install, enter Host 1 and Host 2 details. Set CPU threshold and refresh interval that you want the tool to gather. Once done, click Connect. You will be able to see each Hyper V Host Utilization and Guest Utilization.

The tool will change the Computer icon color when it reach warning level (yellow) and Critical (red).
Warning level:-

Critical level:-

To download:- http://www.manageengine.com/free-hyperv-performance-monitor/download.html

Enjoy!