Hi everyone, Rajeet Nair here, and today I’d like to talk about troubleshooting Service Deployment in SCVMM 2012. As you probably know, one of the key new features in VMM 2012 is Service Deployment. Alaks Sevugan has posted blog entries that provide a good overview of Service feature set so if you haven’t reviewed these entries, now would be a good time to review them. In this post I will go over some troubleshooting tips to help diagnose service deployment failures.
Service deployment is a complex multi-step operation. The good thing is that when service deployment operation fails, VMM provides a lot of information about the exact state of the service and virtual machines, that can help guide a user through the next steps. Understanding the steps involved in service deployment will help diagnose the root cause of failures and move past them. So, we will take a look at the workflow involved in deploying a service. Figure 1 below shows a high level sequence of steps in service deployment.
figure 1
In most cases when service deployment fails, simply retrying the job should help you proceed further. The sections below will help you determine how far VMM got along in service deployment, so you can take any actions if needed before retrying the job.
Step 1: Create group of VMs
This step is responsible for creation of all virtual machines that are part of the service and customizing their operating systems. Internally this step performs multiple “new vm from template” operations in parallel. If this step fails, it indicates that VMM could not create or customize one or more vms for the service. Note that since all vm creation tasks go on in parallel, there could also be some vms which were successfully created. The vms that failed creation/customization will have their Status field either set to CreationFailed or Customization failed. If the cause of failure is incorrect parameter specification on Template, you can fix the issue out of band and then retry the job. If VMM can detect that the VM has been customized successfully, it will proceed further with deployment. If the VM is in a state that cannot be fixed out of band, simply delete the VM through VMM and then retry the job. In this case VMM will go ahead and deploy the vm from scratch.
Step 2: Wait For VM Guest To Boot:
This step is responsible for ensuring that the VMs are up and ready state for application deployment. Here VMM tries to connect to the VM guest operating system using Windows Management Instrumentation (WMI). If this step takes too long (more than 20 minutes) it indicates that VMM server is unable to connect to vms. In this case first thing to check is if the VM was actually customized correctly and is joined to the domain. If not, this is a problem and user should find the root cause of the issue, for example
1. Is the vm connected to the correct network switch? If no, check the LogicalNetwork property on VirtualNetworkAdapter on source VM Template and on physical network adapters on host.
2. Was the correct credentials specified for joining domain?
Note that if VMM is unable to connect to the VM in 60 minutes, the job will timeout and cause the deployment job to fail.
Common Errors in this step include:
1. RPCServerUnavailable: This indicates VMM server is unable to connect to the VM guest. The above mentioned causes are the most common reason for this error. This error could also happen if there is delay in DNS replication, so VMM server is unable to connect to VM. Try pinging the VM using its FQDN and verify if the VM is reachable and correct IP is being shown by ping results. If no, you need to just wait for DNS replication to happen.
2. AccessDenied: This error indicates that VMM Server is unable to connect to the VM using the specified local administrator credentials. The most common reason for this error is that Administrator password specified in Guest OS profile does not match the administrator password in VM guest (the admin password was not cleared out from boot vhd while sysprepping it).
Again if the failure is due to environment issues, retrying the job will help move service deployment along to the next steps. If the error is due to incorrect parameters being specified in the template or Service Configuration, user will need to delete the service and redeploy after fixing the parameters.
Step 3: Deploy Applications to Computer:
This step is responsible for deploying applications into the vm guest. The steps involved here are:
3.1 Install VM Guest Agent:
To facilitate application deployment or roles/features installation, VMM deploys a guest agent inside the VM. If VMM fails to deploy guest agent inside a VM, it puts the VM.ApplicationDeploymentState to DeploymentFailed. VM.ServiceDeploymentErrorMessage will provide information on why the guest agent installation failed. If the error looks like it could be due to network/AD glitch, retrying the job should help move along to next steps of service deployment.
Common Errors in this step include:
1. Another installation is in progress: If there are patches or updates being installed on the VM (through automatic updates) guest agent installation may fail with error indicating another installation is going on. In such cases, waiting for some time and then retrying the job should help proceed through service deployment.
2. VMM Server unable to communicate with VM using WinRM: This error shows up when patches have been installed on the vm and it requires a reboot of vm for WinRM communication to work. In this case user must reboot the vm and then retry the job.
3.2 Install Server Roles and Features
If server roles and features were specified in the Guest OS Profile, this step will go ahead and attempt to enable the roles and features.
3.3 Deploy SQL Profile
If SQL Server installation fails for some reason, user must go and fix the issue out of band. This is because on retry VMM will skip the failed SQL server installation step and proceed with next step in deployment.
3.4 Deploy Applications
In this step VMM server copies the application packages into the VM and then:
1. If Application Profile contains pre install GCE, execute the GCE.
2. Installs each application that was specified in the Application Profile
a. If preinstall GCE was specified for an application, execute the GCE.
b. Install application
c. If postinstall GCE was specified for application, execute the GCE.
3. If Application Profile contains post install GCE, execute the GCE.
Common Errors in this step include:
1. GCE execution failed: If the GCE execution failed, user must examine GCE output and error logs to determine the cause of failure. Note that VMM does not rerun the failed GCE when job is retried so user must fix the issue manually or run the GCE himself before retrying the job.
2. Application Deployment Failed: If this step fails, user should examine application deployment logs inside VM guest and determine the reason for failure. If the failure was due to incorrect parameters being passed for application deployment, then:
a. Failed service instance and service configuration must be deleted
b. Incorrect parameter values must be fixed
c. Deploy a new service instance.
VM state fields:
Any time service deployment fails, VMM identifies the VMs that failed to deploy and marks their state so users can easily find them. VMM uses three fields on a VM to specify what state a VM is in and the cause for errors.
1. VM.Status field: If VMM fails to create a new vm or customize it, it will mark the VM status to CreationFailed or Customization Failed.
2. VM.ApplicationDeploymentState: If VMM fails to provision the VM guest for application deployment (e.g. Guest agent install, install roles and features, install SQL Profile, application installation, GCE execution) this field will be set to DeploymentFailed.
3. VM.ServiceDeploymentError: If a guest provisioning step fails for a VM, VMM will populate this field with the exact cause for failure of the step.
Wrapping Up:
I hope this post helps in providing better understanding of steps involved in service deployment. We welcome feedback on this process on VMM TechNet forum. Let us know if you encountered a problem that is not covered here, or if it helped you solve a problem.
Thanks!
Rajeet Nair | Senior Development Lead
The App-V Team blog: http://blogs.technet.com/appv/
The WSUS Support Team blog: http://blogs.technet.com/sus/
The SCMDM Support Team blog: http://blogs.technet.com/mdm/
The ConfigMgr Support Team blog: http://blogs.technet.com/configurationmgr/
The SCOM 2007 Support Team blog: http://blogs.technet.com/operationsmgr/
The SCVMM Team blog: http://blogs.technet.com/scvmm/
The MED-V Team blog: http://blogs.technet.com/medv/
The DPM Team blog: http://blogs.technet.com/dpm/
The OOB Support Team blog: http://blogs.technet.com/oob/
The Opalis Team blog: http://blogs.technet.com/opalis
The Service Manager Team blog: http: http://blogs.technet.com/b/servicemanager
The AVIcode Team blog: http: http://blogs.technet.com/b/avicode
The System Center Essentials Team blog: http: http://blogs.technet.com/b/systemcenteressentials
The Server App-V Team blog: http: http://blogs.technet.com/b/serverappv
