vRealize Operation Troubleshooting

I decided to share some common troubleshooting steps I’ve had to do with vRealize Operations at customers.

Troubleshooting steps Enabling SSH on the appliance:

  1. Log in to the vRealize Operations Manager (formerly known as VMware vCenter Operations Manager (vApp)) virtual machine console as root using ALT-F1.Note: By default there is no root password configured.
  2. Start the SSH service by running the command:service sshd start
  3. To configure SSH to start automatically run this command:chkconfig sshd on

Troubleshooting tip #1: Pairing the vRealize Operations Management for Horizon Agent fails.

  1. Uninstall V4H agent on connection broker
  2. Delete/Rename ‘C:\Programdata\vmware\vRealize Operations for Horizon’
  3. Delete registry key:

Follow the following KB and install the hotfix on the broker. Horizon 6.4 Hot Fix Download: https://www.dropbox.com/s/n9virk2iuc70k7a/VMware-v4vbrokeragent-x86_64-6.4.0-5099242.zip?dl=0 Additionally, check the following KB 2140844 ‘VMware vRealize Operations Manager for Horizon 6.2 Broker Agent fails to pair with the Horizon adapter’ link: https://kb.vmware.com/s/article/2140844 Troubleshooting tip #2:

  • On the Connection Server on which the V4H Agent is installed, move this file to another directory:install_directory\VMware\VMware View\Server\sslgateway\conf\locked.properties   
  • Restart the View Connection Server service or security server service for the changes to take effect.
  • Restart the V4H Broker Agent service.

Troubleshooting tip #3: There is a rare chance the appliance log partition filled up. To see if the log partition is filled execute ‘df -h’ at the command-line Before executing this next step, verify no historical data needs to be saved because this will wipe it. On the appliances in question execute: find /storage/log/ -mount -type f -iname “*log*” -exec truncate -cs 0 {} \; This will empty the log partition.

 

vSphere Syslog Script

I was working with a client recently who needed to route all their vSphere host’s syslogs to their Log Insight Manager server. I decided to write a script to automate the process. Hope this helps! You will need a machine with VMware’s PowerCLI installed <Tested on 6.5 Release 1 (Build 4624819)> to execute this script with the following parameters:

Script Syntax:

SysLogCfg <vCenter Server> <Admin Username> <Password> <vCenter Cluster Name> <SysLog Server IP>

Script:

# This script will set a Syslog Server all all ESXi hosts within a vCenter once connected.
# Seems to run a little more cleaner with this cmdlet and doesn't ask for confirmation
# Created by Jeremy Wheeler
# 01/17/2018

param($Server,$User,$Password,$ClusterN,$SysLogN)
Connect-VIserver -Server $Server -User $User -Password $Password
$sys = 'udp://' + $SysLogN + ':514'
$cl = get-cluster $ClusterN| get-vmhost

foreach($h in $cl){
  Set-VMHostSysLogServer -VMHost $h.name -SysLogServer $sys
  Write-Host 'vSphere Host: ' $h.name
  $syslog = Get-VMHostFirewallException -name ‘syslog’ -vmhost $h.name
  $syslog | Set-VMHostFirewallException -Enabled:$true
}

You can validate the script worked by checking one of the hosts in the cluster:

Script download link: SysLogCfg

Steps for monitoring App Volumes with Log Insight Manager

 

The first step in this process is to install Log Insight Agent on ONE App Volumes Server per instance.

1. From the Log Insight Manager web portal navigate to the top-corner of the page selecting the three lines

2. Select ‘Administration’

3. Select ‘Agents’

4. Select the dropdown under ‘Agents’

5. Select ‘+ NEW GROUP’

6. Fill out the group name, i.e. ‘App Volumes Managers’

7. Select ‘New Group’

8. Select the dropdown under ‘Agents’ again and this time select your new Agent Group

9. From the dropdown select ‘IP Address’ (default)

10. Select ‘matches’

11. Manually enter the IP address for your App Volumes Manager instances. In this example, I have two sites aka two instances. The IP I am providing is the Load-balanced IP. You can also just put the direct broker IP for testing but we recommend App Volumes Managers be fronted with a load balancer.

12. Select ‘Refresh’

13. After selecting ‘Refresh’ you will see the App Volume Managers that you added in step eleven. If your servers do NOT populate continue until the end of the steps outlined.

14. Enable auto-update for all agents

15. Select ‘Edit’

16. Copy-past the following content into the dialog:

[filelog|Production_Logs]
directory=C:\Program Files (x86)\CloudVolumes\Manager\log
include=*.log
exclude=svmanager*
[filelog|Server_Logs]
directory=C:\Program Files (x86)\CloudVolumes\Manager\log
include=*.log
exclude=production*

17. Select ‘Save Agent Group’

18. Finally, select ‘Refresh’ to validate your servers are populated

If you have completed all the above steps and still do NOT see your App Volume Managers in the agent list after selecting ‘refresh’ you could have something blocking traffic from the Managers to the Log Insight Collector. You do not need to manually edit any files local to the App Volumes Managers. When validating if data is collecting from dashboards you should wait a minimum of five to ten minutes after following the steps in this blog.

Multiple Horizon View Clients, One Workstation

With as much traveling as I do around the country one thing I sometimes need is being able to access multiple Horizon View environments. To do this on a Mac, you can simply clone the Horizon View Client. Adding those multiple View Clients to your Dock can be a little tricky considering it’s the same icon for every client session. One workaround I did was customized the icons:

 

 

 

Download link for ICNS files:
https://www.dropbox.com/s/nqkhlqc8nly38ss/HorizonView-XiCONS.zip?dl=0

1. Right-click ‘Applications’
2. Open ‘Applications’ folder
3. Find the multiple Horizon View Clients
4. Right-click on the 1st instance of your Horizon View Client and select ‘Show Package Contents’
5. This will open a new FINDER window showing the View Client package contents
6. Rename ‘view1_2017-05-31.icns’ to ‘view.icns’
7. Copy ‘view.icns’ (Right-click on the file and select copy)
8. Past the file into the folder ‘Contents’
9. Drag ‘view.icns’ file (The one you just copied) into the ‘Resources’ folder
10. Click ‘Yes’ to overwrite the existing ‘view.icns’
11. That’s it! Now re-open your Applications folder and the icon should be updated.
Repeat this process for the remaining View Clients.

NOTE: Pasting the new view.icns file directly into the ‘Resources’ folder doesn’t have the same behavior as the steps outlined above. You need to move the file into the File Contents and then move the file into Resources.

AppVolumes 2.12 and SQL AlwaysOn Migration

Hey all, I wanted to highlight an excellent blog post Mark Ma did about Migrating from a single SQL database with App Volumes to an AlwaysOn solution. With the recent release of App Volumes 2.12, we officially support Microsoft SQL Server AlwaysOn Availability Groups. SQL AlwaysOn Availability Groups is a great way to provide high availability and disaster recovery because live copies of your databases reside on secondary servers. By integrating SQL AlwaysOn with App Volumes, we ensure the most popular application layering product can be enjoyed by users in any situation. Uninstall 2.11 then Run setup wizard for 2.12

To accelerate your migration process, follow the steps below to migrate App Volumes from a single SQL database to SQL AlwaysOn Availability Groups (SQL 2014 Service Pack 1):

1. Launch the VMware App Volumes 2.12 Installation Wizard, and click Next.

2. Accept user agreement

3.Install App Volumes Manager

4. Launch App Volumes Manager Wizard.

5. Connect to an existing SQL Server Database (Pre-created)

6. Choose the single SQL server with the pre-created AppVolume database.

7. Choose https for secure connection.

8. Choose installation directory.

9. Install.

10. Finish.

11. Launch Manager Console.

12. Verify all services is working.

13. Stop App Volumes Manager Services

14. Backup AppVolume database.

15. Add AppVolume Database to SQL AlwaysON Availability Groups.

16. After verify Database is replicated in SQL AlwaysOn Availability Group change ODBC settings.

17. Edit 64 Bit ODBC settings.

18. Change SQL server from single SQL server to SQL AlwaysOn Availability Group Licenser.

19. Start App Volume Manager services.

20. Verify App Volumes Manager is up and running by launch the console.

I hope this post was valuable in helping you learn how to migrate App Volumes from single SQL Server database to SQL AlwaysOn Availability Groups (SQL 2014 Service Pack 2).

App Volumes and Blocked Ports

When installing a fresh App Volumes Manager, you might receive the error that HTTP port is in-use. Verify services such as Microsoft’s IIS is not running, if it is, remove it. To check what application is using what port on a Windows system execute the following from a command-line:

Syntax: Netstat<space>-anob

Netstat –anob

This will list all ACTIVE connections; example:

Optional:

Syntax: Netstat<space>-anob<space>|<space>findstr<space>:<port>

Netstat –anob | findstr :80

Additional services you can check would be:

Service System Service Name Port(s)
SharePoint Server 80, 443
Windows Media Services WMServer 80
World Wide Web Publishing Service W32SVC 80, 443
SQL Reporting Service ReportServer 80
Sync Share Services SyncShareSvc 80
Web Deployment Agent Service MsDepSvc 80
Internet Information Server WAS, IISADMIN 80

 

HTTP (HTTP.SYS) Hidden Driver/Service

Windows Server 2003/2008/2012 and Windows XP(SP2)/Vista/7/8/10 comes with an HTTP front-end proxy service who’s job is to parse and forward incoming HTTP requests to other Services.

Values in URL “http://hostname:port/virtual_url_or_dir” are registered with it, and when an HTTP request comes in that matches on those values, that request gets routed to the other application or service (which itself is running on a different port).

HTTP.SYS is usually started “on demand” by other services (Windows Remote Management, Print Spooler, etc), and is not usually listening on port 80 until some other application registers a HOST (127.0.0.1) + PORT (80) + virtual URL/DIR with it. HTTP.SYS runs under PID 4 (NT Kernel).

On some Windows systems, oftentimes port 80 is already taken by HTTP.SYS for use.

Show Reserved URLs:

netsh http show urlacl

 

Show active Registered URLs:

netsh http show servicestate

 

To Disable HTTP.SYS:

  • Control Panel > Device Manager
  • In menu View, select: Show hidden devices
  • Open tree: Non-plug and Play Drivers
  • Double-click: HTTP
  • Tab Driver – Group Startup
  • Switch from: Demand to Disabled

Or run this from the administrative privileged command-line (right click cmd.exe, select – run as admin):

  • net stop http /y
  • sc config http start= disabled

Windows Work Folders

Under Windows Server 2012 R2 and Windows 8, Microsoft has introduced a new feature called “Work Folders”, that synchronizes files/folders between different machines.

By default, “Work Folders” uses ports 80 and 443!

There are 3 options to get around this, from simplest to more difficult…

A) Disable the Windows ‘Sync Share Service’, named “SyncShareSvc”.

B) Remove/ “Work Folders” Server Role / Windows Feature:

  • Launch Server Manager. Click “Add roles and features”.
  • Server Roles -> File and Storage Services -> File and iSCSI Services -> Work Folders

C) Or change the ports “Work Folders” use:

Edit file:
C:\Windows\System32\SyncShareSvc.config

Change ports from 80 to 11180 and 443 to 11443 (or something else)…

<sites>

<bindings>
<binding protocol=”http” bindingInformation=”*:80:” />

<binding protocol=”https” bindingInformation=”*:443:” sslFlags=”0″ />

 

Then from a permissions-elevated command-line (right click cmd.exe, Run as admin), run:

Netsh http add urlacl url=http://*:11180/ user=”NT Authority\LOCAL SERVICE”
Netsh http add urlacl url=https://*:11443/ user=”NT Authority\LOCAL SERVICE”

 

Then from a permissions-elevated command-line (right click cmd.exe, Run as admin), run:

You’ll also need to follow more instructions here:

Horizon 7.0.2, What’s New?

Blast Improvements

  • Further enhancements to the protocol
  • Improvements in the GPU-encode/decode that significantly lower bandwidth and latency
  • Improvements in the JPG/PNG codec to reduce bandwidth utilization by 6x
  • vRealize Operations integration with Blast Extreme.  I can now see Blast statistics in the vROPs console
  • UEM Smart Policies Integration with Blast.  I can now use the same PCoIP smart policies to control the Blast protocol.  This enhancement also allows administrators to set per-device policies so I can set different policies for Windows, Mac, Android, and IOS.
  • A Raspberry Pi client

3D Graphics

  • NVIDIA M10 support for high-density graphics acceleration use cases
  • Intel vDGA support on the Skylake platform using 1:1 PCI-E passthru

Horizon RDSH

VMware has continued to close the feature gap with Citrix XenApp, and the latest release checks off a few more boxes.    The main features in this release are:

  • Real-time Audio/Video support for RDSH
  • USB Redirection for RDSH on servers running Windows Server 2012 R2
  • Parameter Passthrough to RDSH Apps – this allows administrators to create custom links that pass parameters through to the application, such as command-line switches or authentication tokens, on launch.

Remote Experience

  • Expanded Windows OS support, including support for Windows 10 LTSB, Anniversary Update, and Pro virtual desktops
  • Flash redirection is now GA.  This allows flash content to be redirected to the local endpoint for rendering for a better experience.
  • Windows Media Redirection support for Windows 10 and Server 2016
  • Windows Media MMR support for Linux-based thin clients
  • Client Drive Redirection is now supported on port 443.  Enhancements have also been made to improve performance on high-latency networks and to speed up file and folder listings
  • DPI synchronization on native Windows clients to ensure crisp rendering of remote session
  • Enhanced clipboard with support for Microsoft Word and Excel
  • Clipboard size increased to 10 MB
  • Ability to link one smart card to multiple accounts

HTML Access Improvements

  • Time Zone Sync
  • File transfer between remote desktop and endpoint using web client
  • RTAV support for desktops and apps

Horizon View 7 Agent and RDP

Working with a customer and also conducting some testing in my lab I discovered that with Horizon 7 Instant Clones I wasn’t able to RDP into them. I verified my firewall settings and also that the ‘Allow connections from computers’ piece was enabled. After various tests I discovered once installing the Horizon 7 Agent it disables TLS 1.1 and 1.2. I resolved this issue by installing a patch from Microsoft on my VDI image to add RDS support for TLS 1.1 and TLS 1.2 (Microsoft KB3080079). Additionally, my endpoint needed the RDP 8.0 update (Microsoft KB2592687) to also enable TLS 1.2. Once putting these two pieces in place I was able to RDP into my Instant Clones with no issues.

KB Reference links to support for TLS 1.1 and TLS 1.2

VDI Desktop: https://support.microsoft.com/en-us/kb/3080079

Endpoint: https://support.microsoft.com/en-us/kb/2592687

 

Configure App Volumes log rolling

App Volumes Manager logs are growing continuously, after a long while taking up substantial amounts of disk space. App Volumes can be configured to roll the logs after a specified size on disk has been reached.

 

On the manager server:

1) Open C:\Program Files (x86)\CloudVolumes\Manager\config\log4r.yml

2) Find the section output_templates under which standard_output section exists.

3) Change parameter CV_ROLL_LOGS to 1

4) To configure the size of each log before it is rolled change the maxsize attribute in the same section. The default is 20971520 bytes (20mb)

5) You can change the amount of files to keep using the max_backups attribute. The default is 3.

 

NOTE: Always keep as many logs as possible, as they may be required for problem analysis. If older logs do not exist, it may be more complicated or impossible to troubleshoot a future problem.