Beware the lotus.domino.UserID error in Domino 9.0.1 FP8/FP9 and other upgrade musings

Over the weekend we upgraded our production Domino servers to get to 9.0.1 FP9 HF139(64-bit) & HF137 (32-bit).  Well, we “tried” to upgrade the servers.

Then all hell broke loose.

Here’s what we did:

• Server1
  • Primary XPages server with NIFNSF enabled.
  • Upgrading from FP8 HF174 to FP9
  • Upgrade failed with error “Installer found the incorrect version…”.
• Server2
  • Backup XPages server.
  • Upgrading from FP6 to FP9.
  • Upgrade failed with error “Lotus Notes/Domino or a Notes/Domino related process is still running”.
  • Booted into Safe Mode and installed FP9 and HF62
  • Upgrade to FP9 HF137 failed with error “Installer found the incorrect version…”
  • Applied JVM Patch 1_8SR4FP10
• Server3
  • Emergency XPages server.
  • Upgrading from FP1 to FP7 (in the event there were issues with FP8/FP9 – this ended up being a saving grace)

After upgrading Server2 and attempting to upgrade Server1, both servers started throwing the following exception when attempting to open any Xpage with a document data source:

javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID

The only reference I could find to this specific issue was from this question on server fault.

In addition, Server1 started crashing any time we opened databases in the Notes client and started throwing long held lock dumps … over and over and over again:

Not sure if this was being caused by NIFNSF being enabled on the server?  Watching the view indexes on disk, the server was leaving 100s and then 1000s of temp .ndx files on disk as the server crashed over and over again.

Luckily, we intentionally upgraded Server3 to only FP7 in the event that we had issues with the other servers.  This saved our ass as we were able to point our users to this server with minimal interruption as we dealt with the issues above.

We were finally able to get Server1 & Server2 upgraded to FP9 HF60.

Here is what we learned in the process:

• If you are receiving the exception “javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID” when opening an XPage with a document data source, then the JVM is probably hosed.  Once we were able to successfully upgrade to FP9 HF60 then everything was fine, but it doesn’t explain why this issue started on a server where the upgrade process failed initially.
• If, when trying to install a fixpack/hotfix you get the message “Error installing Domino Fixpack: Installer found the incorrect version. . .” then you probably need to install the base version of Domino, in our case 9.0.1 and then the FP you are trying to get to plus the HF you are trying to get to.  This is detailed in this technote.
• If, when trying to install a fixpack/hotfix you get the message “Lotus Notes/Domino or a Notes/Domino related process is still running…” then you may need to disable the “Windows Management Instrumentation Service” in the Services list of the server.  We PAUSED the service and successfully upgraded the server.  See this technote.