VMware vCenter Inventory Service Fails To Start 5.5

Introduction

Recently, due to a network issue, vCenter server was failed over from Primary to Secondary. vCenter Heartbeat was keep reporting warnings and as there was already an issue with vCenter Profile-Drive Storage service, thought the warnings were related to it. However, the next day, vCenter users were complaining that web-client wasn’t  operating. Checked the services on Windows where vCenter server 5.5 was installed and found that vCenter Inventory Service was stopped.

Log Analysis

Analysed the ds.log under C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\ and found a Fatal error message, attached below:

WrapperListener_start_runner  FATAL com.vmware.vim.dataservices.WrapperListenerImpl] Error starting service: 
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'instanceUUID' defined in class path resource [server/config/server-config.xml]: Cannot resolve reference to bean 'store' while setting bean property 'targetObject'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storeFactory' defined in class path resource [server/config/query-server-config.xml]: Invocation of init method failed; nested exception is com.xhive.error.XhiveException: INTERNAL_ERROR: Corrupt log record at lsn 362168822267912, Original message: Page 2000054f02 in extent queue contains incorrect magic number
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:275)
 at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:104)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1244)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1008)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:470)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
 at java.security.AccessController.doPrivileged(Native Method)
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
 at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:423)
 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
 at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
 at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
 at com.vmware.vim.dataservices.DataService.getService(DataService.java:213)
 at com.vmware.vim.dataservices.WrapperListenerImpl.start(WrapperListenerImpl.java:52)
 at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:3560)

Issues and Solution

Luckily, there was already an KB article regarding this issue http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2017370. The issue was that the database for vCenter Inventory service was unusable or corrupted. To fix this, I had to either:

• Restore the database from back-up
• Reset the database

Since there was no back-up taken to restore the database, had to reset it. The steps of resetting vCenter Inventory Service database can be found here: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2042200

Followed steps and everything was smooth. However, faced an issue on running register-is.bat file. It thrown an error message that it cannot authenticate SSL certificate files. The error is attached below:

The SSL certificate of STS service was successfully verified against the list of client-trusted certificates SOAP fault javax.xml.ws.soap.SOAPFaultException: Invalid credentials
at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(Unknow n Source)
at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(Unknow n Source)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(Unknown Source)
at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(Unknown S ource)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindin gImpl.java:131)
at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindin gImpl.java:82)
at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestRespon seProcessor.sendRequest(SecurityTokenServiceImpl.java:677) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestRespon seProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:611) at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenB yCertificate(SecurityTokenServiceImpl.java:377)
at com.vmware.vim.dataservices.vcregtool.RegisterVC.acquireSamlToken(Reg isterVC.java:635)
at com.vmware.vim.dataservices.vcregtool.RegisterVC.register(RegisterVC. java:211)
at com.vmware.vim.dataservices.vcregtool.RegisterVC.doRegistration(Regis terVC.java:1253)
at com.vmware.vim.dataservices.vcregtool.RegisterVC.main(RegisterVC.java :1332) Creating SoapFault Processing fault: ns0:FailedAuthentication: Invalid credentials Provided credentials are not valid. opId=6eacd52b-fc8d-401b-bbda-08af92ec2e21 END operation Token request rejected by STS Service com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided cred entials are not valid.

Raised a ticket to VMware support and they modified a line on register-is.bat file. The modification is shown below in bold:

set COMMAND="%~dp0vcregtool.bat" -action register 
-vcurl %1 
-isurl %2 
-lookupserviceurl %3 -vccert "%DATA_DIR%\SSL\sso.crt" 
-vcprivkey "%DATA_DIR%\SSL\sso.key"
-vcinstancecfg "%DATA_DIR%\instance.cfg" -vcendpointsdir  "%PROGRAM_DIR%\endpoints" -vcextensionsdir "%PROGRAM_DIR%\extensions" -vcforceregister true

Ran register-is.bat file again and was successful.

Still not quite sure why he pointed to sso.crt and sso.key instead of rui.crt and rui.key. Probably this is the behaviour changed since vCenter server 5.5.

Summary

In summary, when vCenter Inventory Service stops functioning, first thing is to check the ds.log located under C:\ProgramData\VMware\Infrastructure\Inventory Service\Logs\. If the fatal error message is similar to the attached above, the root cause is that the database is corrupted or unusable. KB articles mentioned up this post explains the steps well but there is a file, register-is.bat to be tweaked. This seems to be the behaviour change and if so, KB article has to be updated accordingly.

Hope this helps to people who are facing issues with vCenter Inventory Service failing to start.