Cheesy click-bait headline aside, I did recently have a need to analyze millions of records in a database and get a feel for what type of non-alphanumeric characters existed in one field in particular.
What I came up with was an incredibly simple view that gives me a count of how often every combination of English non-alphanumeric characters appears in this one field.
column1: Categorized with the following formula:
str := "abcdefghijklmnopqrstuvwxyz1234567890 "; @Explode(@Trim(@LowerCase(field_name));@NewLine+str)
column2: Totals column.
The results are… interesting, to say the least.
The “Not Categorized” category contains all the documents that do NOT have non-alphanumeric characters in field_name.
This type of analysis can be handy if you don’t have control over the input – for example, if your database contains a large amount of data from external sources or if users do a lot of copy and paste from external emails, Word docs, PDFs, and other files.
A short few weeks ago at CollabSphere I was fortunate to have the opportunity to present some functionality that I use everyday in the development of the XPage application that powers our entire business.
The session was titled “Real World XPages” and the presentation is now on Slideshare:
In the coming days and weeks I will be expanding on the topics in the presentation (and adding others) as well as making the code and demos available.
Over the weekend we upgraded our production Domino servers to get to 9.0.1 FP9 HF139(64-bit) & HF137 (32-bit). Well, we “tried” to upgrade the servers.
Then all hell broke loose.
Here’s what we did:
- Primary XPages server with NIFNSF enabled.
- Upgrading from FP8 HF174 to FP9
- Upgrade failed with error “Installer found the incorrect version…”.
- Backup XPages server.
- Upgrading from FP6 to FP9.
- Upgrade failed with error “Lotus Notes/Domino or a Notes/Domino related process is still running”.
- Booted into Safe Mode and installed FP9 and HF62
- Upgrade to FP9 HF137 failed with error “Installer found the incorrect version…”
- Applied JVM Patch 1_8SR4FP10
- Emergency XPages server.
- Upgrading from FP1 to FP7 (in the event there were issues with FP8/FP9 – this ended up being a saving grace)
After upgrading Server2 and attempting to upgrade Server1, both servers started throwing the following exception when attempting to open any Xpage with a document data source:
javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID
The only reference I could find to this specific issue was from this question on server fault.
In addition, Server1 started crashing any time we opened databases in the Notes client and started throwing long held lock dumps … over and over and over again:
Not sure if this was being caused by NIFNSF being enabled on the server? Watching the view indexes on disk, the server was leaving 100s and then 1000s of temp .ndx files on disk as the server crashed over and over again.
Luckily, we intentionally upgraded Server3 to only FP7 in the event that we had issues with the other servers. This saved our ass as we were able to point our users to this server with minimal interruption as we dealt with the issues above.
We were finally able to get Server1 & Server2 upgraded to FP9 HF60.
Here is what we learned in the process:
- If you are receiving the exception “javax.servlet.ServletException: java.lang.NoClassDefFoundError: lotus.domino.UserID” when opening an XPage with a document data source, then the JVM is probably hosed. Once we were able to successfully upgrade to FP9 HF60 then everything was fine, but it doesn’t explain why this issue started on a server where the upgrade process failed initially.
- If, when trying to install a fixpack/hotfix you get the message “Error installing Domino Fixpack: Installer found the incorrect version. . .” then you probably need to install the base version of Domino, in our case 9.0.1 and then the FP you are trying to get to plus the HF you are trying to get to. This is detailed in this technote.
- If, when trying to install a fixpack/hotfix you get the message “Lotus Notes/Domino or a Notes/Domino related process is still running…” then you may need to disable the “Windows Management Instrumentation Service” in the Services list of the server. We PAUSED the service and successfully upgraded the server. See this technote.
Last Tuesday I presented a Webinar on using DataTables with XPages which was hosted by TLCC and Teamstudio. There was a Q&A at the end of the session, but unfortunately due to time constraints we did not get to all of the questions and kind of rushed through the questions we did get to. So I wanted to dedicate a blog post to answering all of the questions that were submitted. Here we go!
Q1: (Submitted by Jena Chinmaya) – Can we use Data table of this format on a regular non xpage classic notes web database ?
A1: Absolutely! A lot of the concepts that were discussed would still apply in a classic Domino Web scenario. The big question that would have to be answered is, how is your data being delivered from the back end to the front end? If you are using a more recent version of Domino (say, 8.5.3 or greater) then you can create an XPage with a REST service control. If you are using an older version of Domino, then you still have a few different options:
- Using a $$ViewTemplateDefault form to display the view. This would be similar to using a xp:repeat in XPages since you would be applying DataTables to an existing html table of data. I created a quick demo so you can get an idea how this would function.
- Use an agent to deliver view data via REST
- Use ajax to get data using the ?readviewentries&outputformat=JSON option of a view url. This method would require some additional work in DataTables to read the data.
Q2: (Submitted by Chris Richards) Could you, using REST services – create one custom control, which you can drop anywhere, and just pass in a cusotm property, namely a view name so that it can be drag/dropped/re-used all over an application with ease?
A2: This can definitely be accomplished and is a great way to make adding view data as simple as drag and drop. I essentially do this now through an extremely configurable custom control. The key to success is making sure that the columns object gets properly defined if you are using JSON data. I hope to make my solution available in the very near future and I’m contemplating a “light” version that I can get out quicker.
Q3: (Submitted by Csaba Kiss) Do you see any performance difference using custom REST service vs standard JSON REST?
A3: While I have not done a direct performance comparison of these REST services, I think it’s reasonable to assume that the more “custom” the custom REST service is, the more it will slow down vs the standard REST service. For example, if while processing each row you are looping through column values to perform actions, you can certainly expect a performance hit. If the custom REST service is just traversing a view, then I would expect the performance to be similar to the standard REST service.
Q4: (Submitted by Jen Watkins) how does one format dates? The format that comes back by default from DataTables is crazy.
A4: I think the format you are referring to is the ISO-8601 format, which you can see in the screenshot below. Actually, this format is being delivered by Domino. You can see this at the bottom of the screenshot in the REST response.
There are a few ways to deal with this:
- When working strictly with dates, my personal preference is to just format the column as text in the view. DataTables recognizes the value as a date and column sorting will work appropriately.
- You can use a renderer to format the date to your preferred format. Here is a great post on the DataTables website demonstrating how to do so. This post references the moment.js library. I use this library as well and it makes working with dates incredibly easy, especially when doing date comparisons.
Q5: (Submitted by Jen Watkins) To filter data before it is returned, for example to return only the data that a particular user is allowed to see, do you have to use the CustomRestService as opposed to the ViewJsonService?
A5: The answer to this question depends on how you have security setup in your application. If your users are authenticating and they are calling REST services that are retrieving data from databases that require authentication, then the appropriate records will be returned based on Domino security.
On the other hand, if you are no so much concerned about security, but want to only show, for example, documents that are assigned to the current user, you can sort or categorize your view on the back end and submit the “keys” attribute with the REST service and the dataset returned will be filtered. You can actually get very creative with using categorized/sorted views to get away from the old Notes habit of having a different view for every different categorization/sort.
Q6: (Submitted by Kevin Gregg) What if you needed to combine data from multiple views into one DataTable – what is a good method of doing that?
A6: When answering this question during the webinar I may have misunderstood the intention of the question so I will answer it in two ways. This question can be interpreted in 2 ways:
- You want to add additional rows – I actually do this a lot. We have 2 databases that store almost identical information. In classic Notes land, we would have to show separate views and embedded views when displaying this information to the user.With DataTables it is very easy to combine data sets, as long as the configuration of the data sets match. What I do is load the initial data, then in the initComplete callback I make an ajax call to the 2nd set of data and then use the add rows functionality of DataTables to merge the data:
- you want to add additional columns – This is akin to having a xp:repeat control and having a column where you perform a @dblookup to another set of data based on information in the original set. For this scenario, the best course of action is probably to use a custom REST service or an XAgent to build the data set on the back end and deliver the completed set to DataTables.
However, you could also accomplish this in DataTables. In your original data set you could have one or more empty cells acting as placeholders. Then, similar as in #1 above, in your initComplete callback you can make an ajax call to another set of data and then using a shared key between the two, insert the retrieved data into the DataTable. I will try to work up an example of this once I can find a few minutes.
Q7: (Submitted by Miguel Angel) How I can lock a document from a datatable to prevent the save replication conflict? How I can show a online message from a Datatables, when a customer made a change at that moment?
A7: The lack of Document locking was certainly one of the early complaints of XPages. Ultimately, people worked around it or rolled their own document locking mechanism. Some of the solutions out there include:
- An OpenNTF project from Frederick Nordling
- This post from StackOverflow
- In our environment, we created our own document locking solution. We needed to do this because we had users that were updating documents in the Notes Client as well as our XPages application.
To relate this to DataTables, if you want to initialize a lock from DataTables, you can do this by adding an onclick or ondblclick event that submits information, such as docid, to a REST service. To release the lock, you would do the same.
To answer your 2nd question, I’m assuming you want to update the current user if data they are looking at was changed in the back end by someone else. Here’s what I think the simplest solution is:
- Add a setInterval to the initComplete callback that retrieves the REST data (or maybe just retrieves a column with date modified value).
- Loop through each record and compare the values to the original DataTable (or compare the date modified column).
- If there is a difference, notify the user. If you are using Bootstrap you can use the alert classes to display a message above the table, for example.
Join me next Tuesday, October 11th for a webinar sponsored by TLCC and Teamstudio – “XPages and jQuery DataTables – Simplifying View creation while maximizing functionality”
In this webinar I will show you how you can use DataTables to create rich, functional views, which is especially relevant if you are modernizing or rebuilding an existing Notes application. While the title references XPages, it is not a requirement. DataTables is compatible with any back end that can deliver data via REST.
You can register here:
I hope to see you there!
If you are modernizing a Notes/Domino application, chances are there are many underlying views that make use of column icons, which have always been one the great little features that users really loved about Notes. As Domino web apps became more prevalent, you could even create img tags that pointed to the icon source files on the server (oooohhhh!).
In the modern app world you are probably using icons that are more … modern, such as Font Awesome.
If you are using jQuery DataTables as well, then it is incredibly easy to add much more functional column icons to your DataTable using a Renderer.
Note: this article assumes the reader has an understanding of how a DataTable is configured using the columns object.
The example below takes the email column and turns it into an icon that, when clicked, will launch a new mail message addressed to the person listed on that line.
We want to turn this:
1.) Setup the view data source that feeds the DataTable.
This can be done by adding a new column to the datasource that contains a JSON string. This is personal preference – you can setup the column however you wish. The important thing is passing the data needed to render the icon to DataTables.
The JSON string in my new column has three properties:
- renderer – has the value “getIcon”.
- icon – this property contains the class(es) of the Font Awesome icon to be used.
- data – this is the raw email address. We want to include this for filtering and sorting.
2.) Now that the data source is setup, update the columns object of our DataTables initilization:
In previous examples our columns object looked like this:
Now, we are replacing the “email” entry with a new entry that will render the Font Awesome icon by using the “render” property of the columns object:
Our view that utilizes the renderer now shows an icon in the email column:
Clicking on one of the mail icons launches a new memo:
Even though our email column is now displaying an icon instead of an email address, we can still filter on email address due to DataTables’ handling of orthogonal data. By filtering on the domain spambob.com, we get only the records that have an email address with that domain.
The working demo for this example can be found here.
Even though this is a fairly basic example, it is incredibly easy to add icons to views created with DataTables – it was simply 2 additional steps. This is only the tip of the iceberg for both Renderers and View Icon Columns.
At MWLUG in August, I presented on using jQuery DataTables with XPages. I finally got the demo database(s) I used in the presentation tidied up and ready for download. Some additional demos have been added since MWLUG.
Better late than never, right?
Get the download here: mwlug-demos
Contents of the zip file:
|datatables.nsf||Contains all demo XPages, resource files, and scripts|
|rest.nsf||Contains REST services and XAgents used by DataTables|
|fakenamesdemo.nsf||Data source – this is the Fake Names database made available by David Leedy. Some modifications have been made for the purpose of these demos|
To get the demo database working:
- Note: I have only tested these demos on 9.0.1
- There are hard coded links to the path xpages\mwlug. You need to place the 3 .nsf files in that path or update the code to your desired path
- Sign the datatables.nsf file. If you are placing the files on a server, sign all 3 databases
- Launch the demo.xsp XPage. This page contains a table of contents of all the demos
I will have the demo database hosted online soon as well.
Online demos can be found here: http://demos.xpage.me/xpages/mwlug/datatables.nsf/demo.xsp
and I plan on adding additional demos to this database as I touch in different topics.
Last week at MWLUG in Austin I was fortunate to be given the opportunity to speak for the first time at a conference. I was very happy with the turnout for my session as well as the positive feedback from the content. I even got a thank you at the airport on the way out of town. (thanks, Tom!).
The slides are available here:
During the session there were some very good questions, which made me realize I probably need to clarify some things in my presentation and fill in some gaps. Over the next few weeks/months I plan on doing that, as well as expanding on some of the topics I covered.
From the conference as a whole, I took away some great information and hope to start utilizing it immediately. The tone of the conference was clear – CYA by looking at non-Domino options. I should note here that the concepts I covered in my session apply to any environment where data is being delivered via REST, not just Domino/XPages.
From a personal standpoint, I need to get better at meeting more people at conferences like these. It isn’t a strength, but needs to become one. I did have some great conversations and was able to do a little brainstorming with Red Pill about their graph implementation. I am really, really intrigued by the possibilities that provides.
Finally, thank you to Richard Moy and the team that made MWLUG a great success. Can’t wait until next year.
… XPages and DataTables (the jQuery plugin, not the XPages data source). This is my first opportunity to speak and present at a technical conference so I’m running the full gamut of emotions – from excitement to nervous and back again!
In my session, I’m hoping to share my experiences with using DataTables as the primary tool for displaying view data fed via REST services in my XPage applications. This experience led to a single, configurable custom control that, in conjunction with a Domino view administration tool, can be dropped into any XPage or custom control where view data is needed. My goal was to create less back-end views and provide more UI functionality while minimizing the need to make design changes when views need to be added or updated.
DataTables is incredibly powerful, and I’ve found that every time I thought I was going to have to customize something, inevitably, DataTables had that functionality built in, or at least had tools that made it easier to implement.
Hopefully this topic interests you as much as it does me and I hope to see you there!
tl;dr – If you have the “compute” option of a document datasource enabled, make sure:
- The underlying form does not have fields perform a DbLookup to a view that has @Today or @Now in a column/selection formula.
- If the form has embedded views, make sure the column/selection formulas of the view do not utilize @Today or @Now.
For the past year+ I’ve been in the midst of modernizing a very, very large Lotus Notes application that is comprised of many databases. Over the past few weeks, as I started building out the next phase, one particular XPage was loading extremely, EXTREMELY, slow, like in the neighborhood of 8 seconds to open. Not good.
Since we are modernizing an existing application (which is the lifeblood of the company), we are operating in a co-existence environment. This means dropping the XPages layer on top of the existing environment versus building and architecting an application from scratch. So in this case, the XPage in question has a document datasource that utilizes an existing Notes form.
After examining the XPage and determining that the performance issue was not there, I turned my attention to the underlying form. I do have the “compute” option turned on:
Now, keep in mind I did not build this form originally, so I am getting up to speed with it’s inner workings as I’m building out the XPages side.
My discovery process went something like this:
“There are 3 embedded views. Ok. No Problem. Lots of Notes forms have embdedded views.”
“This column of the first embedded uses @Today. That’s bad.”
“Wait … every embedded view has a column with @Today? WTF!! OK, we can deal with this.”
“This field is doing a lookup to a view with @Today … and this field … and this field.”
Using @Today and @Now in a view is bad enough … but six times on a form? OUCH!
This was a flat-out performance catastrophe. One of the first performance tips you learn in Notes Client development is do not use @Today or @Now in view column or selection formulas unless you absolutely have to. Even then, consider running an agent nightly to update the view with a hard coded date instead.
Luckily, the issue was discovered and handled, reducing the page load time from 8s to ~50ms.