vRealize Operation Troubleshooting

I decided to share some common troubleshooting steps I’ve had to do with vRealize Operations at customers.

Troubleshooting steps Enabling SSH on the appliance:

  1. Log in to the vRealize Operations Manager (formerly known as VMware vCenter Operations Manager (vApp)) virtual machine console as root using ALT-F1.Note: By default there is no root password configured.
  2. Start the SSH service by running the command:service sshd start
  3. To configure SSH to start automatically run this command:chkconfig sshd on

Troubleshooting tip #1: Pairing the vRealize Operations Management for Horizon Agent fails.

  1. Uninstall V4H agent on connection broker
  2. Delete/Rename ‘C:\Programdata\vmware\vRealize Operations for Horizon’
  3. Delete registry key:

Follow the following KB and install the hotfix on the broker. Horizon 6.4 Hot Fix Download: https://www.dropbox.com/s/n9virk2iuc70k7a/VMware-v4vbrokeragent-x86_64-6.4.0-5099242.zip?dl=0 Additionally, check the following KB 2140844 ‘VMware vRealize Operations Manager for Horizon 6.2 Broker Agent fails to pair with the Horizon adapter’ link: https://kb.vmware.com/s/article/2140844 Troubleshooting tip #2:

  • On the Connection Server on which the V4H Agent is installed, move this file to another directory:install_directory\VMware\VMware View\Server\sslgateway\conf\locked.properties   
  • Restart the View Connection Server service or security server service for the changes to take effect.
  • Restart the V4H Broker Agent service.

Troubleshooting tip #3: There is a rare chance the appliance log partition filled up. To see if the log partition is filled execute ‘df -h’ at the command-line Before executing this next step, verify no historical data needs to be saved because this will wipe it. On the appliances in question execute: find /storage/log/ -mount -type f -iname “*log*” -exec truncate -cs 0 {} \; This will empty the log partition.

 

vROPS Python Adapter – A Hidden Treasure

Even more power comes out of VMware vRealize Operations when enabling the vRealize Operations Python Adapter, adding additional intelligent monitoring and action capabilities. To do this, execute the following steps:  

Image1:

 

  1. Select ‘Solutions’
  2. Select ‘VMware vSphere’
  3. Select ‘vCenter Python Adapter’
  4. Add your vCenters, match what you configured under the ‘vCenter Adapter’ section above #3 in image1.

What does this do for me?

When viewing the default dashboard ‘Recommendations’ you might see something such as the following in your ‘Top Risk Alerts For Descendants’  

Image2:

 

Selecting the alert you will be presented with another dialog to dig into which object we should inspect:  

Image3:

After I select ‘View Details’ it will present me with the object details of the VM, ‘av_prov1’.  

Image4:

Without python adapters configured you will not see the ‘Set Memory for VM’ button, with it configured it will be visible under the Recommendations section.  

Image5:

 

After selecting ‘Set Memory for VM’ you will presented with a new dialog (Image5). Here we can see what the new memory recommendation would be and adjust or apply it. Additionally, if you want the changes to happen now, you can select Power-Off/Snapshot. Without powering off the VM, vROPS will attempt to hot-add the additional memory if the OS will support it.  

Image6:

 

Once you select ‘Begin Action’ you will see the dialog in Image6.

 

Note: ENSURE that permissions for users are limited as in the wrong hands the python corrections could create some havoc also ensure you have a good 3-6 months of collected data before using it or u will get bad results.

vROps 6, Supermetrics, and editing XML

How to find the metric description you can use for metric configuration in XML files. As with vROps 6 the way to retrieve the metric names from v5 doesn’t work anymore. This document explains how to utilize the REST API (Suite API) to achieve what some starting to struggle with. The word ‘Lab:’ in this document reference what steps I performed in my lab. The following flow-chart explains the high-level steps involved to present your Super Metric onto a Dashboard:

Lab Setup: Custom Group

In my lab I created the following:  

  • Custom Group
    • Name: Lab Desktops
    • Group Type: Environment
    • Policy: Default Policy
    • Membership criteria:
      • Criteria: VDI Desktop Pool


Lab Setup: Super Metric

Objective: For my testing I am determining the capacity used for a View Pool


  Super Metric creation and testing steps:

  • In section (1) of this picture you will enter your Super Metric formula or build your Super Metric based on the Adapter Type, Objects, Object Types, Metrics, Attribute types.
  • Once you complete your formula select (2) (blue play icon) to test your formula.
  • After testing your formula’s syntax window (3) will populate with the short-form formula.
  • Select (4) (the graph icon) to test the Super Metric. Note: You may need to select a timeframe to see results (see picture on next page for an example.)

 

 

 

My formula: sum(${this, metric=desktop_vms|pool_capacity_used})   Example of executing a Super Metric through the graph icon: 

Lab Setup: New Policy

My next step is to create new or modify a policy to make sure the Super Metric is active for the objects you want it to be assigned for.

Note: After assigning a Super Metric to a policy it should take 2 cycles (10 minutes) maximum till the Super Metric is visible and accessible via the “All Metrics” tab below troubleshooting. Assuming the related metrics are collected in 5min interval either.

  • Name: Lab Policy
  • Base Policy: vSphere Solution’s Default Policy
  • Override Attributes
    • Super Metric|vdi_pool_capacity
      • Type: Super Metric
      • Adapter Type: All Adapter Types
      • Object Type: All Object Types
      • State: Local
      • KPI: Inherited
    • Super Metric|vdi_pool_capacity
      • Type: Super Metric
      • Adapter Type: View Adapter
      • Object Type: VDI Desktop Pool
      • State: Local
      • KPI: Inherited
  • Apply policy to Group
    • Lab Desktops

Creating a custom XML file

  Create a copy of the appropriate XML file; do NOT edit canned XML files.

  • Copy XML process
    • Select the canned XML file
    • Select the pencil to edit
    • Select all contents (Ctrl-A) and copy into clipboard (Ctrl-C)
    • Click Cancel to stop editing the canned XML
    • Open your web browser to the following URL
      • http://codebeautify.org/xmlviewer
      • Past the XML contents into the XML input field and select the middle button ‘Beautify’
      • Select the two layered notebooks in the upper right or copy the contents in the Result dialog to clipboard
    • Select the green ‘+’ to add a new XML file
    • Enter the name of your new XML

Lab: In my lab we will call this file lab_desktop.xml

  • Past the newly formatted contents into the XML file

  Note: The next steps outline the process to inject a Super Metric into an XML. One requirement that should be identified is the internal name for the metric (Super metric) is required. There is a 2-way process.

  • Retrieve resource with resource identifier.
  • Use the resource identifier to retrieve resource metrics and their names

Retrieving your object’s identifier ID (API Call #1)

  Navigate to the following location:

  • From Home select Environment
  • Select
    • Inventory Tree that pertains to your Super Metric, or
    • All Objects and drill into the needed object or
    • Your custom group
  • Select your object under ‘Related Objects’
  • Click the ‘Troubleshooting’ tab
  • Select ‘All Metrics’
  • Find your needed object

Lab: “Lab Desktops” icon, i.e. our object

  • Open a new tab/browser window and navigate to the following URL:
    • https://<your vrops server>/suite-api/api/resources?regex=.*<Your object>.*

Lab: https://172.16.0.19/suite-api/api/resources?regex=.*Lab%20Desktops.*

 

Note: You can use the complete name or just a fraction. With “.*” at the beginning/end we make sure that we have not missed something. You should see a similar output dump to the following screenshot:

  • Verify the name of the object is correct

Lab: ‘Lab Desktops’.

  • Copy the identifier ID, you will use this for the next step

Retrieving your metric description (API Call #2)

Open a new tab/browser window and navigate to the following URL:

  • https://<your vrops server>/suite-api/api/resources/<your object identifier ID>/statkeys

 

Lab: https://172.16.0.19/suite-api/api/resources/31522d45-63d4-4d8c-b97a-8bae419a9fd5/statkeys  

 

You should see a similar output dump to the following screenshot:  Scroll until you find your Super Metric identifier ID. This is the name of the metric we are going to use for the interactive widget and the xml definitions.   Navigate to:

  • Administration
  • Manage Metric Config (two down from Super Metrics in the left panel)
  • Expand the tree ReskndMetric

Modify your XML file

  Modify your new XML file to reflect your Super Metric:

  • Find the section that has the ResourceKind that pertains to your Super Metric.

Lab: In our example we need ‘ViewPool’

  • Copy an existing line and past it, example shown below:

 

Modify your new string to match your requirements for the super metric. Lab:

  • <Metric red=”90″
  • orange=”85″
  • yellow=”75″
  • unit=”%”
  • label=”vdi_latency” (common name of your Super Metric goes here)
  • attrkey=”Super Metric|sm_b896a92d-657f-4cb2-b550-8ed4ff1526b6″ /> (past your Super Metric identifier ID in this section.)

Select ‘Save’   Verify your policy has the Super Metric setup in the overrides section.

Update your widget

  Navigate to:

  • Home
  • Dashboard List drop-down
  • Select your custom dashboard

  Edit Object Metric Charts widget

  • In the ‘Metric Configuration’ section
    • Use the drop-down and select your custom XML file from the list

Lab: ‘lab_desktop’

  • Select Save

Validate your XML changes

  Navigate to:

  • Home
  • Dashboard List drop-down
  • Select your custom dashboard

Lab: ‘Lab Dashboard’

  • In your View Sessions widget, select a VDI session

Lab: my session is mydomain\myuseraccnt

  • In your Session Related Objects widget, select a desktop pool

Lab: admin_vm pool   Your Object Metric Charts widget should update with the new Super Metric. You may need to wait a few cycles before the data is updated.

vROps Change Master Node IP

Changing Master Node IP address

In some cases I have found customers want to change their Master Node IP when moving from a Pilot to Production for example. This is supported from VMware but isn’t an easy task. Before we dig into this I also want to make sure it’s clear this isn’t changing the Master Node NAME. Changing the Master Node name isn’t supported and cannot be achieved by simply editing the CaSa script, * IT WILL BREAK YOUR CLUSTER *. There mentioning of changing the Master Node Name through an API call, but this is something you will need to call VMware’s GSS about. In a nutshell, it’s not support and you are better off following my Master Node Name change blog entry.

Two different methods exist where you could change the Master Node IP, both are complicated. You can’t simply login to the appliance and change the IP or change them from the properties window. roleState, Gemfire, and Persistence all need this new IP information updated. If you just need to change the IP address this blog entry will assist you in accomplishing such task (the first method). The second method I spoke about is following my blog entry regarding changing the Master Node Name. This entry talks about turning your single node cluster into an HA federation and have the Master Replica reflect the new IP and New node name you want and invoke a failure.

  • Master Node IP preparation steps

 

Special Note:

It’s always better to be safe than sorry so don’t be a hero and take a snapshot on vROps virtual machine before you start.

 

The first thing you need to do is to make sure you bring vROps cluster offline. This can be done via vROps admin UI at https://<your Master Node IP>/admin/

As a result of taking the cluster offline, the Master Node will be brought offline as well.

 

 

 

  • Change IP from the C# Client

 

Shutdown the Master Node VM and change its IP address. In vROps, this configuration is part of the VM properties.

 

Before powering on the VM, make sure to change any DNS A record you might have pointing to the old IP address.

Once the VM is up and running, open SSH console session.

 

  • Configuring the Master Node from console

 

 

Changing the IP in the VM properties is not enough, we also need to make sure the GemFire and the vROps cluster service are also aware of the change.

 

Note

Press Alt-F1 to get into the shell of the virtual appliance.

 

  • CaSA service changes

 

  1. Stop the CaSA service

    service vmware-casa stop

  2. Edit the following file and update the old IP to the new IP:

    /storage/db/casa/webapp/hsqldb/casa.db.script

 

Note

If the IPs do not exist yet in casa.db.script, start then stop the CaSA service.

 

  1. Start the CaSA service

    service vmware-casa start

 

  • ConfigureRoles script

 

Navigate to: /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/bin

 

Run the following script:

./vcopsConfigureRoles.py –-adminCS <Your New Master Node IP>

Note

The above script command is two dashes, not one dash, i.e. <dash><dash>adminCS.

 

 

  • Manually changing Script files

 

If you feel like manual process is your thing (or if you just want to validate the script did its job), edit the IP address in the following 3 configuration files:

 

/usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/data/roleState.properties

/usr/lib/vmware-vcops/user/conf/gemfire.properties

/usr/lib/vmware-vcops/user/conf/persistence/persistence.properties

 

roleState.properties modifications

 

roleState.properties

 

gemfire.properties modifications

 

gemfire.properties

 

persistence.properties modifications

 

persistence.properties

 

After the change has been made, all that is left is to login to the admin UI again (using the new IP address of course) and bring the cluster (and the Master Node for that matter) online again.

 


This should complete the steps needed to re-IP the appliance.

vROps Licensing

vRealize Operations Manager Licensing

You must associate View objects with your vRealize Operations for Horizon license key by editing license groups in vRealize Operations Manager. A license group is a way to gather certain objects, called license group members, under a particular license key. By default, the vRealize Operations Manager and vRealize Operations for Horizon license groups both include all host, virtual machine, and datastore objects. Because these objects are members of both license groups, they are covered by both your vRealize Operations Manager license and your vRealize Operations for Horizon license. Each license group includes membership criteria that you can use to filter the objects that are members of the license group. By editing the membership criteria for the vRealize Operations Manager and vRealize Operations for Horizon license groups, you can specify that certain objects are covered only under your vRealize Operations for Horizon license key.

Prerequisites:

Add your vRealize Operations for Horizon and vRealize Operations Manager license key(s).

 

Licensing vRealize Operations Manager

Note: The license group for vRealize Operations Manager is called Product Licensing.

 

Navigate to the following section:

  1. Administration > Licensing
  2. License Groups
  3. Product Licensing (Shown above)
  4. Select the pencil to edit ‘Product Licensing


  1. Select the license key under ‘vRealize Operations Manager‘, so the background turns blue.
  2. Select ‘Next’


 

Your end-result of licensing vRealize Operations Manager should look like the above picture. Do not change any of the settings with the red X areas, ignore these sections. Note: If you see a different object criteria in any of these object types, do not change them. Only change or append the membership criteria. Example, if Object Type #1 was named Datastore, skip that until you find ‘Host System’, etc. Object Type #1

  • Name criteria: “Host System”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is not
  • Object property #4, “All Hosts”

Object Type #2

  • Name criteria: “Virtual Machine”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is not
  • Object property #4, “All Desktop VMs”

Object Type #3

  • Name criteria: “Datastore”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is not
  • Object property #4, “All Storage”

 


 

Select ‘Next’ when finished entering the object type data shown above.

Verify the License Key(s) selected are specific to your vRealize Operations Manager license and select ‘Finish’.

 

 

 

 

Licensing vROps for Horizon View (V4V)

Note: The license group for vRealize Operations for Horizon is called VMware Horizon Solution Licensing


Navigate to the following section:

  1. Administration > Licensing
  2. License Groups
  3. VMware Horizon Solution Licensing (Shown above)
  4. Select the pencil to edit ‘VMware Horizon Solution Licensing


  1. Select the license key under ‘VMware Horizon Solution Licensing‘, so the background turns blue.
  2. Select ‘Next’


 

Your end-result of licensing VMware Horizon Solution Licensing’ should look like the above picture. Do not change any of the settings with the red X areas, ignore these sections. Note: If you see a different object criteria in any of these object types, do not change them. Only change or append the membership criteria. Example, if the top object was named “View Pod”, skip that until you find ‘Host System’, etc.

 

Object Type #1

  • Name criteria: “Host System”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is
  • Object property #4, “All Hosts”

Object Type #2

  • Name criteria: “Virtual Machine”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is
  • Object property #4, “All Desktop VMs”

Object Type #3

  • Name criteria: “Datastore”
  • Object property #1, “Relationship”
  • Object property #2, “Descendant of”
  • Object property #3, “is
  • Object property #4, “All Storage”


 

Select ‘Next’ when finished entering the object type data shown above.

Verify the License Key(s) selected are specific to your VMware Horizon Solution Licensing license and select ‘Finish’.