EBMS Tickets

Issue Number 642
Summary EBMS Prod error
Created 2022-11-09 12:21:22
Issue Type Bug
Submitted By Boggess, Cynthia (NIH/NCI) [C]
Assigned To Kline, Bob (NIH/NCI) [C]
Status Closed
Resolved 2023-03-15 12:15:41
Resolution Cannot Reproduce
Path /home/bkline/backups/jira/oceebms/issue.332163
Description

Error received when trying to add a new topic to a citation. Issues started after the network issue at shady grove was resolved yesterday. When trying to filter queue the ebms would churn and then not fully display the citation queue. Seems intermittent. We did a troubleshoot yesterday and we were not seeing the same issue in chrome. Everything seemed fine but now today the same issues are happening again in both edge and chrome. Error message attached.

Comment entered 2022-11-09 12:24:10 by Osei-Poku, William (NIH/NCI) [C]

tagged

Comment entered 2022-11-09 12:34:57 by Kline, Bob (NIH/NCI) [C]
  1. Does the failure happen on non-production EBMS?

  2. Do you have a topic assignment I can use to reproduce the failure on production?

Comment entered 2022-11-09 12:38:39 by Kline, Bob (NIH/NCI) [C]

I see from a closer reading of the issue description that I should be able to reproduce the problem without having a specific topic assignment to make. However, I am unable to reproduce it either in production or on the stage server.

Comment entered 2022-11-09 12:44:57 by Kline, Bob (NIH/NCI) [C]

I have tried reproducing the failure both off the VPN and on it, with my own account and as using the Test Librarian account, and the review queue comes up right away, both for the initial load of the page, as well as in response to clicking the Filter button. Can you give me more specifics which might help me to reproduce exactly what you're doing? Can you reproduce the problem logged on with the Test Librarian account?

Comment entered 2022-11-09 12:55:03 by Kline, Bob (NIH/NCI) [C]

Is it possible that in spite of the wording in the issue description the failure is happening with the Submit (not the Filter) button, and that I will need a topic assignment in order to see the problem in action?

Comment entered 2022-11-09 12:56:17 by Boggess, Cynthia (NIH/NCI) [C]

The problem occurs after about 5-6 minutes of activity. At first everything seems fine, I log in, sort the queue, review a set or two of 10 citations and then upon a submit or refilter of the queue it starts churning endlessly and either does not display the page or only displays the top part not the list of citations. The error came after reviewing a page or two of citations when I tried to add a new topic.

Comment entered 2022-11-09 13:11:15 by Boggess, Cynthia (NIH/NCI) [C]

The endless churning was happening with submit as well as filter. When adding a topic the churning and then error occurred after just clicking on the pencil icon.

I have been trying to replicate the problem in DEV and have not been able to. DEV seems more responsive and quick than PROD is today.

Comment entered 2022-11-09 13:15:01 by Kline, Bob (NIH/NCI) [C]

Can you tell me the time when the last failure occurred?

Comment entered 2022-11-09 13:20:54 by Kline, Bob (NIH/NCI) [C]

Does it happen every time you attempt to assign a new topic from the queue page? Or is it more sporadic? Can you open up the console and take a screenshot of it next time it happens? On Chrome, press the F12 key, select Console from the menu above the window which opens up.

Comment entered 2022-11-09 13:22:44 by Kline, Bob (NIH/NCI) [C]

I don't see anything relevant in the logs.

Comment entered 2022-11-09 13:26:23 by Kline, Bob (NIH/NCI) [C]

Better still, if you can keep the Chrome console open at the bottom of your screen and keep an eye out for anything that looks like an error message, that would be helpful. You may get some warnings that are irrelevant (red herrings), and I'll let you know when you report any that are.

Comment entered 2022-11-09 13:32:36 by Boggess, Cynthia (NIH/NCI) [C]

The error occurred sometime between 10:30 - 11:05.

Comment entered 2022-11-09 13:34:41 by Boggess, Cynthia (NIH/NCI) [C]

It has only happened twice and I have added several topics without getting the error before and after these two occurrences.

Comment entered 2022-11-09 13:38:08 by Boggess, Cynthia (NIH/NCI) [C]

I primarily use edge. How do I see the console in edge?

Comment entered 2022-11-09 13:56:06 by Kline, Bob (NIH/NCI) [C]

F12 also works in Edge.

Comment entered 2022-11-09 14:05:58 by Boggess, Cynthia (NIH/NCI) [C]

Just got the same error again but this time when I was trying to change the number of citations per page. I was off the vpn in edge with console open. see attachment.

Earlier I was connected to the vpn and did not experience as much churning and did not get any errors. So definitely worse off the vpn.

Comment entered 2022-11-09 14:10:44 by Boggess, Cynthia (NIH/NCI) [C]

afterwards I clicked reset to reset the queue and an error appeared in the console. I'll attach it as well.

Comment entered 2022-11-09 14:16:49 by Boggess, Cynthia (NIH/NCI) [C]

Minutes later I tried to change the number displayed per page in the queue again and it worked fine. So this does seem to be a sporadic problem.

Comment entered 2022-11-09 14:21:50 by Kline, Bob (NIH/NCI) [C]

Do you know for sure that the problem has occurred at least once ON the VPN?

Comment entered 2022-11-09 14:34:21 by Boggess, Cynthia (NIH/NCI) [C]

I was able to review and submit several pages of citation decisions then when I submitted the most recent page of decisions at 2:28pm it churned until 2:31 and an error appeared in the console and nothing was displayed on the page. see attachment.

Comment entered 2022-11-09 14:37:58 by Boggess, Cynthia (NIH/NCI) [C]

I'll log back on to the vpn and test again while watching the console.

Comment entered 2022-11-09 14:57:41 by Kline, Bob (NIH/NCI) [C]

A ServiceNow ticket (NCI-INC0705652) has been filed with CBIIT reporting the failures.

Comment entered 2022-11-09 16:59:08 by Boggess, Cynthia (NIH/NCI) [C]

I was using the EBMS with vpn from about 2:30pm until about 4:30pm and did not get and errors or endless churns. It churned a bit here and there longer than usual but otherwise responded as expected.

Comment entered 2022-11-09 17:26:42 by Kline, Bob (NIH/NCI) [C]

That's encouraging, thanks. I still would like to push for an answer to the question above. Have the failures ONLY occurred when not on the VPN? Or has there been at least one confirmed failure when we know the user was ON the VPN? This could be an important piece of information as CBIIT tries to track down the cause.

Comment entered 2022-11-09 17:40:21 by Boggess, Cynthia (NIH/NCI) [C]

I'll work in the ebms with vpn ON again tomorrow and let you know how it goes.

Comment entered 2022-11-09 17:52:03 by Kline, Bob (NIH/NCI) [C]

👍

Comment entered 2022-11-10 15:26:25 by Boggess, Cynthia (NIH/NCI) [C]

I have been reviewing in the EBMS Prod with vpn ON for 3-4 hours today and did not experience any endless or even excessive churning and no errors. So it looks like these issues are not happening with the vpn ON 

Meanwhile in EBMS Prod with vpn OFF barely 5 minutes from logging in and I get endless churning and the same pop up and console error messages as posted yesterday while trying to change the number of citations to display per page from my queue.

Comment entered 2022-11-10 16:07:10 by Kline, Bob (NIH/NCI) [C]

Thanks. This is confirmed by Margaret's experience. I have passed this data point along to CBIIT. That makes it easier for them to track down the cause, but makes it almost impossible for me to reproduce the problem myself, because I don't want to do anything in production which would change any values in the database, and the non-production tiers are not accessible except when I'm on the VPN. I haven't been able to reproduce the failures on production with the restricted set of actions I can safely perform. I'm hoping the problem is very localized, by which I mean restricted to the review pages used by internal staff, because almost all of the board members are outside the firewall.

Comment entered 2022-11-10 17:40:54 by Boggess, Cynthia (NIH/NCI) [C]

Here is something you could try that would not impact production data but might allow you to recreate the problem.

Go to my queue and sort by board and submit.

Then add a summary topic to the sort and submit.

Then go to display options and choose to display 50 citations per page and submit.

Then hit Reset.

If you repeat these steps several times, I'm guessing you will probably be able to reproduce the problem. Today I was able to reproduce the error by doing this just once and did not have to record any decisions at all.

Comment entered 2022-11-10 17:53:53 by Kline, Bob (NIH/NCI) [C]

Excellent suggestions. I'll give it a shot. Thanks!

Comment entered 2023-03-15 12:15:41 by Osei-Poku, William (NIH/NCI) [C]

We decided that this is no longer an issue with the new EBMS. So, I am closing this ticket. Thanks!

Attachments
File Name Posted User
ebms error 11-9-22.docx 2022-11-09 12:21:12 Boggess, Cynthia (NIH/NCI) [C]
image-2022-11-09-14-02-53-569.png 2022-11-09 14:02:54 Boggess, Cynthia (NIH/NCI) [C]
image-2022-11-09-14-10-13-634.png 2022-11-09 14:10:14 Boggess, Cynthia (NIH/NCI) [C]
image-2022-11-09-14-33-54-037.png 2022-11-09 14:33:55 Boggess, Cynthia (NIH/NCI) [C]

Elapsed: 0:00:00.000799