Recently, due to the subnet decommission, had to change IP addresses of two vCenter servers.

After changing the IP addresses, the ESXI servers under each vCenter servers were disconnected. I thought this was a normal behaviour after the modification. Thus, I re-connected the servers and it was successful.

Few minutes later, ESXi servers were disconnected again. While Googling, found an KB article to this issue (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1001493). The problem was that the vpxa.cfg (vCenter Server Agent configuration file) file under /etc/opt/vmware/vpxa/vpxa.cfg in ESXi server stored the old IP address of the vCenter server.

To fix the issue, the KB article above suggested two methods (please refer to the article for the detailed steps):

1. vpxa.cfg modification
2. Reinstallation of vCenter Server Agent

Since I am a CLI person, applied the first method and ESXi servers were connected back to the vCenter servers. However, after having a cup of coffee, found out they were disconnected again! Checked the vpxa.cfg file and it was still pointing to the old vCenter server IP address. Thought second method should work. I tried it but still, same consequence.

While investigating into the vCenter settings, I found that under Runtime Settings, vCenter Server Managed IP was set to the old IP address!

After changing it to the new IP address, I tried the first method and it fixed the issue.

In summary, make sure you change the Managed IP Address under Runtime Settings before applying any methods suggested on the KB article, otherwise, you will see the same thing over and over!

Hope this helps.

Introduction

In our vSphere environment there are number of clusters consuming 200+ SCSI IDs, i.e. 200+ VMFS volumes and due to this, even creating a single VMFS volume takes forever. It gets worse if there are many ESXi servers in the cluster when deploying more than one VMFS volumes because of the storage rescan filter (refer to Duncan’s blog for further explanation what storage rescan filter is, http://www.yellow-bricks.com/2010/08/11/storage-filters/).

While I was studying for VCAP-DCA certificate, I got to read an KB article “Manually creating a VMFS volume using vmkfstools -C” (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1009829). When I tried it myself, I was amazed how fast a VMFS volume was created. Since I knew that Python (programming language) was embedded to ESXi server, I decided to write up a script to automate the above.

Assumption

• ESXi with 5.x version
• Python is very restrict to the indentation, do it right
• Input file to be in the right format (an example will be given below) and placed in a same directory with the script

The format of the input file is:

“UID”,”Name of the VMFS volume”

An example:

naa.60050768018180732000000000000001,VMFS1
naa.60050768018180732000000000000002,VMFS2
naa.60050768018180732000000000000003,VMFS3
naa.60050768018180732000000000000004,VMFS4

Script

import subprocess
ask = True
vmfs_version = ""

while ((vmfs_version != "VMFS3" or vmfs_version != "VMFS5") and ask == True):
    vmfs_version = raw_input("Do you want to create VMFS3 or VMFS5 (for VMFS3, the default block size is 8MB and for VMFS5 is 1MB): ")
    if (vmfs_version == "VMFS5"):
        vmfs_version = "vmfs5"
        block_size = "1m"
        partition_format = "gpt"
        ask = False
        type = "AA31E02A400F11DB9590000C2911D1B8"
    elif (vmfs_version == "VMFS3"):
        vmfs_version = "vmfs3"
        block_size = "8m"
        partition_format = "msdos"
        ask = False
        type = 251

f = open('list')
for line in f.readlines():
    uid=line.split(',')[0]
    vmfs=line.split(',')[1].replace('\n', '')
    partition=subprocess.Popen(["partedUtil","get","/vmfs/devices/disks/"+str(uid)],stdout=subprocess.PIPE).stdout.read().replace('\n','')
    end_sector = int(partition.split(' ')[0]) * int(partition.split(' ')[1]) * int(partition.split(' ')[2]) - 1
    arguement = "1 2048 " + str(end_sector) + " " + str(type) + " 0"

    print ""
    print "Partition Detail: "
    print partition
    print ""

    print "End Sector: "
    print end_sector
    print ""

    print "Creating a partition on /vmfs/devices/disks/"+str(uid)
    subprocess.call(["partedUtil","setptbl","/vmfs/devices/disks/"+str(uid),partition_format,str(arguement)])
    print ""

    print "Creating a VMFS Volume: " + str(vmfs)
    subprocess.call(["vmkfstools","-C",vmfs_version,"-b",block_size,"-S",vmfs,"/vmfs/devices/disks/"+str(uid)+":1"])
    print ""
f.close

print "Rescanning all HBAs"
subprocess.call(["esxcli","storage","core","adapter","rescan","-a"])
print "Done"
print ""

Steps

• SCP both input and script files to the ESXi server (I used /tmp folder)
• Locate your directory to tmp, cd /tmp
• Run the script (I named the script file as create_vmfs.py and input file as list), python create.py
• The script will ask which VMFS volume to be created, VMFS3 or VMFS5
  • One thing to note here is that when VMFS3 is chosen, by default, it will create a VMFS volume with block size of 8mb. Obviously, VMFS5 with 1mb of block size.
• Choose the one required.
• Rescan for datastores on a cluster level.

Test and Result

The following input list file is used:

naa.60050768018180732000000000000ed4,TMP_5
naa.60050768018180732000000000000ed5,TMP_6

• Check the existing VMFS volumes (this could be checked on the vSphere client):
  • esxcli storage vmfs extent list

Volume Name       VMFS UUID                            Extent Number  Device Name                           Partition
----------------  -----------------------------------  -------------  ------------------------------------  ---------
TMP_1             51c3d7e9-2b04a674-358f-001a6435b02e              0  naa.60050768018180732000000000000e74          1
TMP_2             51c90e55-b2ac1066-5119-001a6435b05a              0  naa.60050768018180732000000000000e79          1
TMP_3             521aa8f4-397b9299-b068-001a6435b02e              0  naa.60050768018180732000000000000ed9          1
TMP_4             51ee0a76-b1e515e7-4908-0015177af2a1              0  naa.60050768018180732000000000000e97          1
datastore1        51c3afef-a11bc9f8-fa00-001a6435b02e              0  naa.6005076b060c39401005d5c40c298f43          3

• Run the script and choose the VMFS version
  • python create_vmfs.py

/tmp # python create.py 
Do you want to create VMFS3 or VMFS5 (for VMFS3, the default block size is 8MB and for VMFS5 is 1MB): VMFS5

Partition Detail: 
130 255 63 2097152

End Sector: 
2088449

Creating a partition on /vmfs/devices/disks/naa.60050768018180732000000000000ed4
gpt
0 0 0 0
1 2048 2088449 AA31E02A400F11DB9590000C2911D1B8 0

Creating a VMFS Volume: TMP_5
create fs deviceName:'/vmfs/devices/disks/naa.60050768018180732000000000000ed4:1', fsShortName:'vmfs5', fsName:'TMP_5'
deviceFullPath:/dev/disks/naa.60050768018180732000000000000ed4:1 deviceFile:naa.60050768018180732000000000000ed4:1
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs5 file system on "naa.60050768018180732000000000000ed4:1" with blockSize 1048576 and volume label "TMP_5".
Successfully created new volume: 52c65b61-91d23030-907f-001a6435b02c

Partition Detail: 
130 255 63 2097152

End Sector: 
2088449

Creating a partition on /vmfs/devices/disks/naa.60050768018180732000000000000ed5
gpt
0 0 0 0
1 2048 2088449 AA31E02A400F11DB9590000C2911D1B8 0

Creating a VMFS Volume: TMP_6
create fs deviceName:'/vmfs/devices/disks/naa.60050768018180732000000000000ed5:1', fsShortName:'vmfs5', fsName:'TMP_6'
deviceFullPath:/dev/disks/naa.60050768018180732000000000000ed5:1 deviceFile:naa.60050768018180732000000000000ed5:1
Checking if remote hosts are using this device as a valid file system. This may take a few seconds...
Creating vmfs5 file system on "naa.60050768018180732000000000000ed5:1" with blockSize 1048576 and volume label "TMP_6".
Successfully created new volume: 52c65b65-b3574c56-89d0-001a6435b02c

Rescanning all HBAs
Done

• Make sure the VMFS volumes are created

Volume Name       VMFS UUID                            Extent Number  Device Name                           Partition
----------------  -----------------------------------  -------------  ------------------------------------  ---------
TMP_1             51c3d7e9-2b04a674-358f-001a6435b02e              0  naa.60050768018180732000000000000e74          1
TMP_2             51c90e55-b2ac1066-5119-001a6435b05a              0  naa.60050768018180732000000000000e79          1
TMP_3             521aa8f4-397b9299-b068-001a6435b02e              0  naa.60050768018180732000000000000ed9          1
TMP_4             51ee0a76-b1e515e7-4908-0015177af2a1              0  naa.60050768018180732000000000000e97          1
TMP_5             52c65b61-91d23030-907f-001a6435b02c              0  naa.60050768018180732000000000000ed4          1
TMP_6             52c65b65-b3574c56-89d0-001a6435b02c              0  naa.60050768018180732000000000000ed5          1
datastore1        51c3afef-a11bc9f8-fa00-001a6435b02e              0  naa.6005076b060c39401005d5c40c298f43          3

Conclusion

Compared to the normal way of creating VMFS volumes using the vSphere client, benefits of creating VMFS volumes using esxcli, partedUtil and vmkfstools are:

1. Faster deployment
2. Can avoid storage rescan filter rescanning the cluster every time a VMFS volume is created
3. Most importantly, it’s an automated way which means tons of VMFS volumes could be created by running one script

Hope you enjoy.