VRO: no logs for workflow runs

Today I had another great issue that eventually got resolved. The problem was that on neither of my built-in vRO appliances I saw the logging of my workflows that ran. On both appliances I just saw “No logs found”. Not really handy while you’re in the middle of troubleshooting some workflows.

Phase 1 troubleshooting

First I thought it was related to my time-zone settings but that wasn’t the case.
Going further, I checked the state of my disks, which quickly revealed that on one of my appliances the /storage/log was 100% full. Unfortunately I went to dinner and I didn’t took the printscreens at that time and my putty connection timed out. But I highlighted the directory that was 100%.

Going to that directory I sorted the directories on size and in my case it was the /vrhb directory that was the biggest. (Again, due to dinner not a correct printscreen but the directory is highlighted).

There were a lot of *.hprof files that were quite big and filling up my disk. They are Java Memory Dump files that are created when Orchestrator runs into an OutOfMemory exception. So you can remove them without issues. When those files were removed, I was back at 16% used storage.

But unfortunately that didn’t solve my issue yet, so going to next phase now.

Phase 2 Troubleshooting

Going further in my quest I came across a VMware article that described a similar issue. Although I didn’t have the entries in the /var/log/vmware/vco/app-server/server.log they were referring to I did have multiple Lucene files. But when I wanted to stop the vRO service I got below error message.

I wanted to reboot this appliance. But before doing so, I first checked which appliance was my master and made sure that the master role wasn’t assigned to the appliance I was going to reboot. This can be done on the VAMI interface in vRA Settings => Database. Once that was done I could safely reboot my vRA appliance.

After the reboot I could see the correct status of my vRO service.