vRealize Automation Center

vRealize Automation bulk update lease on Virtual Machines

I wrote a workflow that takes some inputs and uses an ODATA query to list all virtual machines where their lease is expiring within a certain time frame.

It uses the API to get all the virtual machines and then you can bulk update these lease days by changing the value for the numberOfDaysToAdd variable.

You will need to add the vCACCAFE:vCACHOST value for the host attribute and also make sure that the plugin user configure has the right privileges and entitlements.


var operationName = "Change Lease";
var numberOfDaysToAdd = 30;
var leaseEndDate = "2017-08-15T00:00:00.000Z";
var pageSizeLimit = "20"
var endpoint = "com.vmware.csp.core.cafe.catalog.api";
var restClient = host.createRestClient(endpoint);

var pages = parseJson(api());
System.log("Number of pages found from API query: " + pages.metadata.totalPages);

for (i=1; i-1 < pages.metadata.totalPages; i++){

var json = parseJson(api(i));
if (i >1) {
System.log("Processing page " + i + " of pages." + metadata.totalPages + " . Sleeping for 60 seconds whilst vRA processes change lease operations.");
System.sleep(60000);
}

for each (deployment in json.content){
System.log("Found Deployment: " + deployment.name);
var catalogResource = vCACCAFEEntitiesFinder.findCatalogResources(host , deployment.name)[0];

System.log("Found catalog Resource " + catalogResource.name);

var newLeaseDate = new Date();
newLeaseDate.setDate(newLeaseDate.getDate() + numberOfDaysToAdd);

System.log("Setting new lease date to " + newLeaseDate);

var myvCACCAFEDateTimeLiteral = new vCACCAFEDateTimeLiteral(newLeaseDate) ;

//Get all the operations that are available on the deployment
var operations = catalogResource.getOperations();
var resourceActionNotFound = false;

//Locate the deployment specific operations and store it
for each (operation in operations){
System.log(workflow.currentWorkflow.name + ": " + operation.getName() + " operation is available on this catalog resource")
if (operation.getName().toLowerCase() == operationName.toLowerCase()){
System.log(workflow.currentWorkflow.name + ": " + operation.getName() + " operation identified and selected");
resourceActionNotFound = true;
break;
}
}
if (resourceActionNotFound == false) {continue};

var inputs = new Properties();
inputs.put("provider-ExpirationDate", myvCACCAFEDateTimeLiteral);

System.getModule("com.vmware.library.vcaccafe.request").requestResourceAction(operation,inputs)

}
}

function api(page){

var url = "consumer/resources?&limit=" + pageSizeLimit;
if (page) url = url + "&page=" + page.toString();
var catItemsUrl = url + "&$filter=resourceType/id eq 'composition.resource.type.deployment' and lease/end lt '" + leaseEndDate + "'";
var catItems = restClient.get(catItemsUrl);
return catItems.getBodyAsString();

}

function parseJson(res){

return JSON.parse(res.replace(/\\/g, '').replace(/\\t/g, '').replace(/\\r/g, '').replace(/\\n/g, ''));
}

vRealize Automation – Changing a Virtual Machine’s Business Group in 7.2 and 6.x

Changing business groups in both vRealize Automation v6 and v7 is a common task and below demonstrates how to automate the process. Luckily, VMware provides an out of the box vRealize Orchestrator workflow solution in both vRealize 6.x and 7.2, however, the workflow you need to use in both versions is different. Let’s have a look:

vRealize Automation 7.2

As of v7.2, we have a new workflow that changes the business group of the IaaS virtual machine entity as well as the CAFE deployment entity. The virtual machine entity is still part of the IaaS model, however, the deployment is a CAFE entity and therefore there are a few things you need to be aware of before changing the business group. We’ll cover those later. However, here is a screenshot of the workflow that you need to use:

The workflow you need to run to change business group can be found in the following location, noting this is under the Library folder:

screen-shot-2017-01-28-at-16-55-52

You need to provide the following inputs:

Screen Shot 2017-01-28 at 16.56.39.png

There are a few things to be considered.

1 – The owner of the virtual machine must also be a member of the target business group. The business group is determined based on the reservation you select. If the user is not a member of the target business group, the virtual machine IaaS entity business group will get updated, however, the deployment CAFE entity business group entity will not get updated, leaving you in a position where the 2 entities that are out of sync. If this is the case, your my items will be out of sync as the deployment will still appear under the original business group but the virtual machine will appear under the new business group.

2 – You may have more than one Virtual Machine within the deployment so make sure you update all virtual machine entities within the deployment.

3 – This workflow is not available prior to version vRealize Automation 7.2, although if you need it, contact VMware GSS support as they may be able to help you get your hands on a fix. Or, better still upgrade to vRealize 7.2.

Based on the above, one way I would implement this solution would be to create a day 2 operation on the deployment CAFE entity. Then you can do a couple of things as follows:

1 – Check that the owner of the deployment is a member of the target business group. If not, flag this on the day 2 XaaS form.

2 – Get all virtual machines that are part of the deployment so you update all virtual machines which are part of the deployment.

That’s it but you can enhance the custom day 2 resource operation if you need to.

vRealize Automation 6.x

In my opinion, it is slightly easier to update the virtual machine business group as we don’t have to worry about the CAFE deployment as the entity doesn’t exist in vRA 6.x. We do have MMBP, but this is part of the IaaS model which can be updated. This post just concentrates on single machine blueprints. The workflow you need to run to change business groups can be found in the following location, noting this is under the Library folder:

Screen Shot 2017-01-28 at 17.12.29.png

You can just run the import vCAC Virtual Machine and you can rerun this over an over again. It won’t register the machine multiple times but just update the business group based on the following inputs:

screen-shot-2017-01-28-at-17-14-02

You can see there are few more options to supply in vRA 6.x. You can also make this a day 2 operation by wrapping this workflow around another workflow as you can see the actual code that changes the business group is this:

vm.register()

Where vm is a vCAC:VirtualMachine type. You can see this in the ‘start register workflow‘ scripting element.

You can see the requirements parameters using the API library, but here’s a screenshot of what you need:

screen-shot-2017-01-28-at-17-18-32

That’s pretty much it. Happy Business Group Automating.

Updating vRA ASD Service Blueprint workflows

There isn’t an easy way to find out what vRealize Orchestrator workflow is associated to a vRealize Automation Service Blueprint. You can query the vPostgres database to find out the workflow ID as follows:

1 – Log in to vPostgres and connect to the vcac database

2 – Run the following SQL command

vcac=# \x on;
Expanded display is on.
vcac=# select * from asd_serviceblueprint;

This should output something similar to this:

-[ RECORD 1 ]------------------------------------------------
id | 850b9e17-024e-44f7-9831-a5651a5d6e0f
description | This is a workflow description.
name | My XAAS workflow
status | 1
tenant | vsphere.local
workflowid | 1eea8b02-15e8-41b8-992e-3df1cd8e4b99
baseform |
outputparameter | ea001bf4-d43f-4c74-a010-b6027c7ecbdf
catalogrequestinfohidden | t
version | 1.0.0
-----------------------------------------------------------

You can change the vRealize Orchestrator workflow that gets called by a vRealize Automation service blueprint pretty easily, by changing the workflow ID value in the vPostgres database

vcac=# update asd_serviceblueprint 
set workflowid='ad9e2cc2-2efa-44c1-a574-6a02bec2f998'
where id = '9d6510ad-24a6-4f0c-adc8-736cc3e99d77';

This will change what workflow gets executed without redeploying the ASD form. It’s handy when you want to clone the ASD service blueprint and actually change the underlying workflow to test something new, without having to rebuild the ASD form.

You can also run a vRealize Orchestrator workflow using the below javascript to find out what the name is using the workflow ID

wfId = '9d6510ad-24a6-4f0c-adc8-736cc3e99d77'
System.log("Workflow name: " + Server.getWorkflowWithId(wfId).name);

 

All of this can be managed by creating your own service blueprint and publishing as a catalog item..

Adding a vNIC to a virtual machine in vRealize Automation Center

I was asked the other day to add 2 nics to a virtual machine. 1 nic had to be an adapter type of E1000 and the other had to be a VMXNET3. The reason behind this was because the VM was being built by PXE and until VMtools was installed as part of the build, the VMXNET3 adapter wouldn’t have been recognised by the build process. I thought that I could just specify the custom properties “VMware.Network.Type”, however this is a global property and applies to all vm nics added at request time and you cannot change the apadter type after you have added a NIC to the vm. So I had to add the additional vNIC at the Machine Provisioned vRA stub workflow stage. I also needed to match the virtual wire that was submitted at request time. That was the challenge and this was how I went around it.

First of all I had to get the virtual wire requested. In this post, I am going to use an example of ‘virtual-wire-001’. I then had to find that virtual wire on the logical switch, or in old ESX terms, the dvSwitch. To do this, I got he host the VM was allocated on, searched for this host in the clusters available, then got the dvSwitch so I could get the dvPortGroup object type.

// get host that is running VM
var vmHost = vcVm.runtime.host.name;

// example name for the virtual wire to look for.
var vNic1Network = "virtual-wire-001 ";

// prepare sdk connection
var sdk = Server.findAllForType("VC:SdkConnection");

// loop through SDK connections
for (var con=0; con < sdk.length; con++){

    // create object of allClusterComputeResources
    var clusters = sdk[con].getAllClusterComputeResources();

     // loop through Clusters in allClusterComputeResources

     for each (var cluster in clusters){
    
        // loop through all hosts in cluster
        for each (var host in cluster.host){
        
            // match vm host to host in cluster
            if (vmHost == host.name){
            
                // get object for cluster
                var vmCluster = clusters;

                // get network object in cluster and match on network name requested
                for each (var dvPortGroup in cluster.network){
                
                    if (dvPortGroup.name.indexOf(vNic1Network) = -1){
                        System.log("Found dvPortGroup object to connect the vNIC to: " + dvPortGroup)};
                        break;
                }
            }
        }
    }
}

What this gave me was a dvPortGroup object that I could use to connect the vNIC to on the virtual machine. Next was to create the vNIC on the virtual machine. Initially, I found this next bit tricky to get my head round, but once I understood the code, it opens up a lot of configuration options not just for adding a vNIC, but also adding other virtual machine configuration specifications:


// creating a config spec
var configSpec = new VcVirtualMachineConfigSpec();
var deviceConfigSpecs = new Array();
var deviceConfigSpec;

// create operation for device config spec
var deviceConfigSpec = new VcVirtualDeviceConfigSpec();
deviceConfigSpec.operation = VcVirtualDeviceConfigSpecOperation.add;

// Create connect info for dvPortgroup
var connectInfo = new VcVirtualDeviceConnectInfo();
connectInfo.allowGuestControl = true;
connectInfo.connected = true;
connectInfo.startConnected = true;

// Create Network BackingInfo
var dvsPortConnection = new VcDistributedVirtualSwitchPortConnection();
var dvSwitch = VcPlugin.convertToVimManagedObject(dvPortGroup, dvPortGroup.config.distributedVirtualSwitch);
dvsPortConnection.switchUuid = dvSwitch.uuid;
dvsPortConnection.portgroupKey = dvPortGroup.key;

var netBackingInfo = new VcVirtualEthernetCardDistributedVirtualPortBackingInfo() ;
netBackingInfo.port = dvsPortConnection;

// Create VirtualNetwork
var vNetwork = new VcVirtualVmxnet3();
vNetwork.backing = netBackingInfo;
vNetwork.unitNumber = 0;
vNetwork.addressType = "Generated";
vNetwork.wakeOnLanEnabled = true;
vNetwork.connectable = connectInfo;

// Creating devicespecs configured in vNetwork object
deviceConfigSpec.device = vNetwork;
deviceConfigSpecs[0] = deviceConfigSpec;

// List of devices
configSpec.deviceChange = deviceConfigSpecs;

// Launch the reconfigVM task
vcVmTask = vcVm.reconfigVM_Task( configSpec );

The key areas in this script are as follows:

// List of devices
configSpec.deviceChange = deviceConfigSpecs;

// Launch the reconfigVM task
vcVmTask = vcVm.reconfigVM_Task( configSpec );

This creates an object that is used to submit the configuration specifications

var configSpec = new VcVirtualMachineConfigSpec();

We then build up this object with various configuration objects that are required. Things like connection information, dvsPortConnection, adapter type settings. Here’s each bit in more info.

For the connection settings, these are quite obvious.

var connectInfo = new VcVirtualDeviceConnectInfo();
connectInfo.allowGuestControl = true;
connectInfo.connected = true;
connectInfo.startConnected = true;

We are basically telling the NIC to be connected at start up and connected by default.

The next was to create the dvsPortConnection settings. Here is where we pass the dvPortGroup object found in the cluster the virtual machine was allocated in.

var dvsPortConnection = new VcDistributedVirtualSwitchPortConnection();
var dvSwitch = VcPlugin.convertToVimManagedObject(dvPortGroup, dvPortGroup.config.distributedVirtualSwitch);
dvsPortConnection.switchUuid = dvSwitch.uuid;
dvsPortConnection.portgroupKey = dvPortGroup.key;

You need to pass in the dvPortGroup object to get the uuid and the key for the dvSiwtch. These values are effectively required to know how to ‘attach’ the vNIC to the dvswitch. If you don’t have these settings, the portgroup will be blank and you will get an invalid backing message, and hence no network connectivity. Once you create this object, you need to add this to the VcVirtualEthernetCardDistributedVirtualPortBackingInfo() object

var netBackingInfo = new VcVirtualEthernetCardDistributedVirtualPortBackingInfo() ;
netBackingInfo.port = dvsPortConnection;

The next section is required to create the VcVirtualVmxnet3 settings. There are different object types for different vNICs. As I needed the vmxnet3 vNIC, I used the VcVirtualVmxnet3() object type. You also have VcVirtualE1000, VcVirtualE1000e, VcVirtualVmxnet2 etc.

var vNetwork = new VcVirtualVmxnet3();
vNetwork.backing = netBackingInfo;
vNetwork.unitNumber = 0;
vNetwork.addressType = "Generated";
vNetwork.wakeOnLanEnabled = true;
vNetwork.connectable = connectInfo;

What we are doing now is adding some of the previously created objects to the VcVirtualVmxnet3 object, for example the netBackingInfo, connectInfo etc. This is making up the profile of the vNIC.

Once we have created the vNIC profile, we just need to execute the object on the vNIC.

// Creating devicespecs configured in vNetwork object
deviceConfigSpec.device = vNetwork;
deviceConfigSpecs[0] = deviceConfigSpec;

// List of devices
configSpec.deviceChange = deviceConfigSpecs;

// Launch the reconfigVM task
vcVmTask = vcVm.reconfigVM_Task( configSpec );

In my vRO workflow, the vcVm is the VC:VirtualMachine object passed in to the stub workflow.

Here’s a look at the workflow:

Screen Shot 2015-02-17 at 20.35.50 1

and here are the input parameters:

Screen Shot 2015-02-17 at 20.36.54

Hope this helps

Using ASD to deploy vRealize Automation Center blueprints

Deploying vRealize Automation blueprints using ASD is a useful technique. It gives you flexibility by creating drop down lists and interactive forms. What is also very handy, is building up request logic based on human readable inputs and passing this in to the request. As you might know, some request values are only applicable when making the request, typically allocation (vrm.datacenter.location and __reservationPolicyID for example).

You can achieve this by first creating a vRO workflow that calls the catalog item. Here is an example workflow that I have written to do this.

Screen Shot 2015-02-17 at 14.24.51

The below script shows an example of the javascript to request a catalog items


System.log(&Getting catalog item & + item.getName() + ' request form...);
var form = vCACCAFERequestsHelper.getRequestFormForCatalogItem(item);

System.log(Expected inputs:);
var fields = vCACCAFERequestsHelper.getFormKeys(form);
for (var i = 0; i < 0; fields.length; i++) {
    System.log( " - " + fields[i]);
}

System.log("Filling in catalog item request...");
form = vCACCAFERequestsHelper.setFormValues(form, inputs);

System.log("Sending catalog item request...");
var request = vCACCAFERequestsHelper.requestCatalogItem(item, form);

System.log("Catalog item request " + request.getId() + " sent!");

Running through this, we can see a couple of key lines of code. The item is actually the blueprint you are calling.

var form = vCACCAFERequestsHelper.getRequestFormForCatalogItem(item);

You can actually hard code this as an attribute or determine this on the fly depending on the options that user has selected.

The next key line is creating the form object. The form object is submitted as part of the request, along with the item.

form = vCACCAFERequestsHelper.setFormValues(form, inputs);

The form is what information you are submitting as part of the request.

And now here is the request.

var request = vCACCAFERequestsHelper.requestCatalogItem(item, form);

Once you have this bit of code ready, you need to create the form values. You can actually get these using firbug and capture the POST values when submitting a catalogue item. Here is an example:

//Build up the request form for the new VM
properties = new Properties();

properties.put("provider-provisioningGroupId", provisioningGroupId);
properties.put("provider-Cafe.Shim.VirtualMachine.NumberOfInstances", new vCACCAFEIntegerLiteral(numberOfInstances).getValue());
properties.put("provider-Cafe.Shim.VirtualMachine.TotalStorageSize", new vCACCAFEDecimalLiteral(totalStorageSize).getValue());
properties.put("provider-VirtualMachine.LeaseDays", new vCACCAFEIntegerLiteral(leaseDays).getValue());
properties.put("provider-VirtualMachine.CPU.Count", new vCACCAFEIntegerLiteral(cpuCount).getValue());
properties.put("provider-VirtualMachine.Memory.Size", new vCACCAFEIntegerLiteral(memorySize).getValue());
properties.put("provider-VirtualMachine.Disk0.Size", disk0Size);
properties.put("provider-VirtualMachine.Disk0.IsClone", disk0IsClone ? true : false);
properties.put("provider-VMware.VirtualCenter.OperatingSystem", vmOS);
properties.put("provider-VirtualMachine.Network0.Name", fullNetworkName);

The provider inputs are the required inputs for a generic catalogue request. You can dynamically populate this form based on ASD input parameters. However you can also add to this form whatever values you wish by passing in a properties object. For example:

customProperties.put("virtualdevops.custom.props.one", value1);
customProperties.put("virtualdevops.custom.props.two", value2);
customProperties.put("virtualdevops.custom.props.three", value3);
customProperties.put("virtualdevops.custom.props.four&", value4);

Then you can add these custom properties to the properties object you pass into the request

for each (var k in customProperties.keys) {properties.put(&provider-&+k,customProperties.get(k));};

Now, you can pass the properties object in to the default action as part of the request., or use the script above taken from the action item.

actionResult = System.getModule(com.vmware.library.vcaccafe.request).requestCatalogItem(item, properties)

Now you can use any blueprint and build up a property object that consists of request values adding to your blueprint profile. There is a caveat around this. If you use vRA to request catalogue items, the owner of the resources will be the user configured when you added the plugin. But that’s ok as we can change the owner to the VM as part of the ASD workflow, which is next up!

Changing the owner of a resource in vRealize Automation Center

A previous post, I mentioned the power of deploying blueprints via ASD and I said at the end of the request that you need to change the owner of the virtual machine. This blog shows an example of how to do just that.

The first key thing to mention is when submitted a service blueprint in vRA, certain request information is being sent through as part of the request. Here are some of the values being passed in.


__asd_requestedBy = System.getContext().getParameter(__asd_requestedBy);
__asd_catalogRequestId = System.getContext().getParameter(__asd_catalogRequestId);
__asd_tenantRef = System.getContext().getParameter(__asd_tenantRef);
__asd_subtenantRef = System.getContext().getParameter(__asd_subtenantRef);

 

 There are other ways you can get these key values, but the one to look out for is the __asd_requestedBy value. This comes in the format of the logged on user who submitted the request and is the user you can use to change the owner ship of the resources to when deploying blueprints via ASD.

As part of the solution, you need to get the catalog resource object. Now typically, this is returned when you submit a catalog request.

You need to do is wait for the catalog item to be ready. Use this OOTB workflow, “wait for catalog item”:

Screen Shot 2015-02-17 at 15.09.45

Waiting for the the catalog item to appear means the resource will be ready to then run the reconfigure resource action against it. What this does is change the owner – the same as if you were to click on the reconfigure action and change the owner.

You also need to get the vCAC VM entity. You can get this using the VCACCAFE bindingid and matching the binding ID on the vCAC:VirtualMachine

Here’s a great blog from Sergio https://communities.vmware.com/blogs/sergioatvmware/2014/07

And the bit of code you are interesting in is:


System.log("Provider binding is: " + catalogResource.providerBinding.getBindingId());
bindingId = catalogResource.providerBinding.getBindingId();
&nbsp;&nbsp; &nbsp;
var vm = Server.findForType('vCAC:VirtualMachine', bindingId);
var virtualMachineID = vm.virtualMachineID;

You also need to get the vCAC User. The vRA entity you are changing ownership of is actually the vCAC IaaS user. Here is a way of getting that user:

var model = ManagementModelEntities.svc;
var entitySetName = Users;
var userProperty = new Properties();
userProperty.put(UserName,userToChangeTo);
var IaaSUser = vCACEntityManager.createModelEntity(vCACHost.id , model, entitySetName, userProperty, null, null)
userEntity = System.getModule(com.vmware.library.vcac).getUserEntityByName(vCACHost,userToChangeTo);

You can get the operation ID is the reconfigure resource action. You can get this ID as follows:

var resourceActions = VM.operations;
var count = 0;
while(resourceActions.length &lt; count)
    {System.log("Action available on this catalog resource is:"+ resourceActions[count].name); &nbsp;&nbsp; &nbsp;
    if(resourceActions[count].name == Operation){
         System.log("Reconfigure action available on this catalog resource");
         var vmOperation = vCACCAFEEntitiesFinder.getCatalogResourceAction(VM,resourceActions[count].Id);
    }
count++; } 

Once you have the vCAC VM and the VCAC User and the reconfigure resource action ID, you can then create the form that is required to change the owner. You can get the inputs by using firebug.

// Build input list for the form

form var FormInputs = new Properties();
FormInputs.put("provider-operationId", VcacConsumerResourceOperation.getBindingId());
FormInputs.put("provider-machineId", VcacVM.virtualMachineID);
FormInputs.put("provider-TARGET_MACHINE_NAME", VcacVM.displayName);
FormInputs.put("provider-Cafe.Shim.VirtualMachine.Owner", VcacUserEntity.getProperty(UserID));

Once you have all of this, you can then use the requestResourceAction from list of default actions. The code example is below and takes the operation vCACCAFE:ConsumerResourceOperation and the FormInputs properties object you ave built from above:

System.log("Getting resource action " + operation.getName() + " request form... ");
var form = vCACCAFERequestsHelper.getRequestFormForResourceAction(operation);

System.log("Expected inputs: ");
var fields = vCACCAFERequestsHelper.getFormKeys(form);
for (var i = 0; i &lt; 0; fields.length; i++) {
&nbsp;&nbsp; &nbsp;System.log(" - " + fields[i]);
}

System.log("Filling in resource action request..");
form = vCACCAFERequestsHelper.setFormValues(form, FormInputs);

System.log("Sending resource action request...");
var request = vCACCAFERequestsHelper.requestResourceAction(operation, FormInputs);

System.log("Resource action request " + request.getId() + " sent!");

This is just an explanation and you should be able to build your own workflows around this example. Feel free to leave a comment or any questions.

Restarting INFOBLOX Services via the REST API via vRealize Orchestrator

I was recently working with Infoblox where I had to restart some services in order to registering the information I was configuring. Here is a quick over view of the workflow I wrote to achieve this.

Screen Shot 2015-02-17 at 22.07.48

You would need to register aVRO REST host that points to the infoblox grid. Then we need to first find the GRIB _ref. The _ref returns a URI that we append to the REST host that does something. It is like an HREF in XML.

Here is the first script.


operationUrl = "grid"
requestType = "GET";

We then pass this in to the Invoke REST host workflow. This workflow accepts a requestType and the opertionUrl. Once this has been submitted, we then look at the contenResponse and get the _ref for the infoblox grid.


// Parsing JSON response
var grid = JSON.parse(restOutput.contentAsString);

// Setting and logging next available IP information
System.log("Creating the grid reference " + grid[0]._ref);

// Setting the REST operation URL
var operationUrl = grid[0]._ref + "?_function=restartservices";

// Creating request JSON content - using strinfiy to convert form object to JSON string.

var requestContent = JSON.stringify({
"restart_option" : "RESTART_IF_NEEDED",
"service_option" : "DHCP",
"member_order" : "SEQUENTIALLY",
"sequential_delay" : 10
});

requestType = "POST";

The script above parses the content response and looks for the _ref value in the returned object (grid[0]._ref).

Once we have that _ref URI, we append that onto the end of the REST host. We also pass in the following JSON post payload.


"restart_option" : "RESTART_IF_NEEDED",
"service_option" : "DHCP",
"member_order" : "SEQUENTIALLY",
"sequential_delay" : 10

I was updating DHCP options and I was able to specify the service I wanted to restart. I could have specified other options such as ALL and DNS. I set the sequential_delay to 10 which means that it will wait 10 seconds before restarted in the DHCP service in each GRID master.

Some other really helpful info can be found here:

https://community.infoblox.com/forum/ddi/definitive-list-rest-examples

Hope that helps!