Intro

One of my customers had problems with their vRLI environment, the most obvious one was that the webgui wasn’t available or even started. The following post is a more general approach of how I troubleshoot vRLI. In this case I had to apply 2 workarounds / solutions to make it work again. In this case I fixed the keystore & and restored the config of the master node.

Troubleshooting

I used the Runtime.Log to find the issue of the vRLI appliance.

The log is placed in the folder:

/storage/var/loginsight/

There I saw a clear issue with the key Store

[2020-09-21 09:39:48.867+0000] ["main"/10.35.9.19 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Begin start tomcat web server at 6191 ms after launch]

[2020-09-21 09:39:48.908+0000] ["main"/10.35.9.19 ERROR] [com.vmware.loginsight.services.EmbeddedService] [Error when starting service com.vmware.loginsight.daemon.shared.TomcatController]

java.io.IOException: Keystore was tampered with, or password was incorrect

at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source)

at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source)

at sun.security.provider.KeyStoreDelegator.engineLoad(Unknown Source)

at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(Unknown Source)

at java.security.KeyStore.load(Unknown Source)

at com.vmware.loginsight.daemon.shared.TomcatController.configureTomcatServerXml(TomcatController.java:87)

at com.vmware.loginsight.daemon.shared.TomcatController.start(TomcatController.java:238)

at com.vmware.loginsight.services.EmbeddedService.startService(EmbeddedService.java:125)

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:767)

at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startTomcatServer$12(LogInsightDaemon.java:593)

at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:738)

at com.vmware.loginsight.daemon.LogInsightDaemon.startTomcatServer(LogInsightDaemon.java:589)

at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:331)

at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1040)

Caused by: java.security.UnrecoverableKeyException: Password verification failed

... 14 more

[2020-09-21 09:39:48.909+0000] ["main"/10.35.9.19 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Failed to start Tomcat: Keystore was tampered with, or password was incorrect]

[2020-09-21 09:39:48.910+0000] ["main"/10.35.9.19 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Exception during start tomcat web server]

[2020-09-21 09:39:48.910+0000] ["main"/10.35.9.19 INFO] [com.vmware.loginsight.daemon.LogInsightDaemon] [Ended start tomcat web server at 6235 ms after launch, took 44 ms]

[2020-09-21 09:39:48.914+0000] ["main"/10.35.9.19 FATAL] [com.vmware.loginsight.daemon.LogInsightDaemon] [Error starting services]

com.vmware.loginsight.daemon.LogInsightDaemon$StartupFailedException: Daemon startup failed: Failed to start Tomcat: Keystore was tampered with, or password was incorrect.

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:786)

at com.vmware.loginsight.daemon.LogInsightDaemon.lambda$startTomcatServer$12(LogInsightDaemon.java:593)

at com.vmware.loginsight.daemon.LogInsightDaemon.runAndTime(LogInsightDaemon.java:738)

at com.vmware.loginsight.daemon.LogInsightDaemon.startTomcatServer(LogInsightDaemon.java:589)

at com.vmware.loginsight.daemon.LogInsightDaemon.run(LogInsightDaemon.java:331)

at com.vmware.loginsight.daemon.LogInsightDaemon.main(LogInsightDaemon.java:1040)

Caused by: java.lang.Exception: Keystore was tampered with, or password was incorrect

at com.vmware.loginsight.daemon.LogInsightDaemon.startService(LogInsightDaemon.java:775)

So the issue “Keystore was tampered with or Password was incorrect” is a known error. The kb article for that is:

https://kb.vmware.com/s/article/66993

vRealize Check if Daemon is running

2^nd, I checked if the vRealize Daemon was running, which was not the case.
The command for that is:

service loginsight status

Because the service wasn’t running I was curious if there more issues. So when I wanted to check the config of the vRLI Master appliance, I noticed that its config file and folder was gone. Which was also the case for the Restore folder which contains different RPMs.

While the other 2 vRLI appliances do have these folders

So in my case, I first made a snapshot of all the vRLI machines as a fallback & then copied both the “config” as the “restore” folder from the 2th appliances towards the Master Appliance. Since the data is identical within these folders, this can be easily done. You can always check with vi if there is some specific settings for the appliances like IP address & DNS Name, but this shouldn’t be the case for the config. Do note there are 2 config files, the one that we copied right now is a simple config. The other one is at another location but it is also the master config if remember correct. That one was still intact so need to tamper with it.

So copy / pasting the 2 folders of the other appliances solved 1 issue, but there was one bigger issue and that was that my keystore was not working. Which unfortunately was also the case on all the nodes. Otherwise I could have copied that to the master node too.

So in this case I generate a new key.

Before you start do make sure you have a snapshot on all the vRLI appliances or have a backup. Then, note down the highest apache directory number.

Which in my case was:  apache-tomcat-8.5.38

Now it is time to remove the keystore on all the nodes.
With the command:

rm /usr/lib/loginsight/application/3rd_party/apache-tomcat-8.5.38/conf/keystore

grep syslog-ssl-keystore-password /storage/core/loginsight/config/loginsight-config.xml#36 | gawk -F'"' '{print $2}'

Then Run the command to generate the Keystore (With the your own apache version ofcourse)

/usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /usr/lib/loginsight/application/3rd_party/apache_ver/conf/keystore -keypass keystore_password -storepass keystore_password -dname "CN=VMware vCenter Log Insight, OU=vCenter Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US"

*Note. The command only worked for me when I moved into the /usr/java/default/bin folder, and then run the keytool.

After that you should have created a new keystore.

Now copy the keystore to the other appliances.

And that should be it.

Samir of vSAM.pro

Samir is the author of vSAM.Pro & a Life enthusiast who works as a consultant in the field of IT. With a great passion for Tech & Personal Development, he loves to help people with their problems, but also inspire them with a positive outlook on life.

Besides that, he is also a big Sport & Music junky that loves to spend a big chunk of his time on producing music or physically stretching himself.