Beware the lotus.domino.UserID error in Domino 9.0.1 FP8/FP9 and other upgrade musings
Posted: November 30, 2017 Filed under: Uncategorized 4 CommentsOver the weekend we upgraded our production Domino servers to get to 9.0.1 FP9 HF139(64-bit) & HF137 (32-bit). Well, we “tried” to upgrade the servers.
Then all hell broke loose.
Here’s what we did:
- Server1
- Primary XPages server with NIFNSF enabled.
- Upgrading from FP8 HF174 to FP9
- Upgrade failed with error “Installer found the incorrect version…”.
- Server2
- Backup XPages server.
- Upgrading from FP6 to FP9.
- Upgrade failed with error “Lotus Notes/Domino or a Notes/Domino related process is still running”.
- Booted into Safe Mode and installed FP9 and HF62
- Upgrade to FP9 HF137 failed with error “Installer found the incorrect version…”
- Applied JVM Patch 1_8SR4FP10
- Server3
- Emergency XPages server.
- Upgrading from FP1 to FP7 (in the event there were issues with FP8/FP9 – this ended up being a saving grace)
After upgrading Server2 and attempting to upgrade Server1, both servers started throwing the following exception when attempting to open any Xpage with a document data source:
javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID
The only reference I could find to this specific issue was from this question on server fault.
In addition, Server1 started crashing any time we opened databases in the Notes client and started throwing long held lock dumps … over and over and over again:
Not sure if this was being caused by NIFNSF being enabled on the server? Watching the view indexes on disk, the server was leaving 100s and then 1000s of temp .ndx files on disk as the server crashed over and over again.
Luckily, we intentionally upgraded Server3 to only FP7 in the event that we had issues with the other servers. This saved our ass as we were able to point our users to this server with minimal interruption as we dealt with the issues above.
We were finally able to get Server1 & Server2 upgraded to FP9 HF60.
Here is what we learned in the process:
- If you are receiving the exception “javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID” when opening an XPage with a document data source, then the JVM is probably hosed. Once we were able to successfully upgrade to FP9 HF60 then everything was fine, but it doesn’t explain why this issue started on a server where the upgrade process failed initially.
- If, when trying to install a fixpack/hotfix you get the message “Error installing Domino Fixpack: Installer found the incorrect version. . .” then you probably need to install the base version of Domino, in our case 9.0.1 and then the FP you are trying to get to plus the HF you are trying to get to. This is detailed in this technote.
- If, when trying to install a fixpack/hotfix you get the message “Lotus Notes/Domino or a Notes/Domino related process is still running…” then you may need to disable the “Windows Management Instrumentation Service” in the Services list of the server. We PAUSED the service and successfully upgraded the server. See this technote.
XPage.me 
In my experience, you can’t install a fix pack over a hotfix. first, you have to un-install the hotfix (by running the hotfix installer again). that would have taken you back to a vanilla FP8 install. then you should be able to install FP9.
LikeLike
Yes, IBM needs to adress the installer issue. See also: http://www.billmal.com/billmal/billmal.nsf/dx/notes_on_mac_osx_high_sierra.htm
The crossover from JVM 1.6 to 1.8 has introduced som additional complexity.
As noted by Stuart, it was already necessary to revert any hotfixes or Intermediate Fixes before istalling a FixPack or FeaturePack.
With the now much more frequent JVM fixes, I’ve hosed my JVM several times. Or had a FixPack or FeaturePack installer error out because it could not handle the updated JVM version.
So you also need to keep track of the JVM fix pack levels and versions, because those need to be at the state of when the previous FixPack was installed, or the FixPack or FeaturePack installer will hiccup on that.
LikeLike
When patching servers, I set IBM Domino and the IBM Diagnostic Service to start manually, and I set Windows Management Instrumentation Service to Disabled. If there is backup software installed, I set the one or two services associated with the backup software to start Manually. I then do an OS restart. I do a reinstall of 9.0.1, install the current Feature Pack and apply the current Interim Fix. I then apply the current JVM. I set the Windows services that I earlier set to Manual or Disabled to Automatic, and restart the OS.
LikeLike
I forgot to mention that the Windows Management Instrumentation service is by default configured to restart two minutes after it has been stopped. If you pause the service, it respawns while you are patching Domino.
LikeLike