EBMS Tickets

Issue Number 432
Summary Unable to Import Citations on PROD
Created 2017-04-19 11:14:02
Issue Type Bug
Submitted By Juthe, Robin (NIH/NCI) [E]
Assigned To Kline, Bob (NIH/NCI) [C]
Status Closed
Resolved 2017-04-19 13:55:54
Resolution Fixed
Path /home/bkline/backups/jira/oceebms/issue.206750
Description

We seem to be currently unable to import citations on PROD using the fast-track option. I'm not sure if this applies to batch imports that Cynthia and Minaxi do. I am attaching a screenshot of the error message ("Secure Connection Failed") that is displayed. Other work in the EBMS seems to be going OK.

Comment entered 2017-04-19 11:21:30 by Kline, Bob (NIH/NCI) [C]

What happened when she clicked "Learn more"? Is this repeatable? Can you reproduce it on any of the lower tiers? Can you provide the exact sequence of steps leading up to the failure?

Thanks.

Comment entered 2017-04-19 11:23:35 by Kline, Bob (NIH/NCI) [C]

Added Amy as a watcher, so she'll be aware that I'm looking into this. If fact, probably a good idea to keep her in the loop like this for any tasks which aren't part of my currently assigned project.

Thanks.

Comment entered 2017-04-19 11:26:33 by Kline, Bob (NIH/NCI) [C]

I just confirmed that the production EBMS server can communicate with PubMed's server and to write to its own database (by running the Journal Table Refresh command, which does both).

Comment entered 2017-04-19 11:35:13 by Juthe, Robin (NIH/NCI) [E]

This is the page that was displayed when she clicked "Learn More": https://support.mozilla.org/en-US/kb/what-does-your-connection-is-not-secure-mean

Bonnie reported that she received a "Page Cannot be Displayed" error in her browser - I'm guessing that was IE, but I will confirm. She has attempted to import several fast-track citations this morning and keeps getting the same error message (I have also tried and failed), so it is reproducible.

To reproduce the problem, here's an example.
1. Go to Import Citations page.
2. Select "Cancer Genetics" as the Board.
3. Select "Genetics of Breast and Ovarian Cancer" as the topic.
4. Enter the following PMID: 28398847
5. Check "Fast Track".
6. Enter the following placement level: Passed abstract review.

Comment entered 2017-04-19 11:58:08 by Kline, Bob (NIH/NCI) [C]

I was able to reproduce the failure on PROD, but the problem cannot be reproduced on the lower tiers. I will open a ticket with CBIIT to see if they have any log information which might be useful. It does not appear that any imports have been made at all today. Can you check with the librarians to see if they have tried and been unsuccessful? A couple more questions: do you know when you were last able to run a successful fast track import? And are you able to upload files with a substantial size (we'll want to determine whether the file system is full)?

Thanks.

Comment entered 2017-04-19 12:02:23 by Kline, Bob (NIH/NCI) [C]

Entered CBIIT ticket.

Comment entered 2017-04-19 12:06:32 by Juthe, Robin (NIH/NCI) [E]

I asked Minaxi about any imports and this was her response. I have not asked her to create a small file to test with, but we can if you think that would help.

"Hi Robin,

I have not imported any batch citations this morning and I will not have any batch that needs to be imported before last week of April.
If there is a need to experiment, then I will have to generate a small text file and import it. Please let me know if you want me to do that.

Thanks,
Minaxi"

As for your other questions, it appears that the last successful citation import happened yesterday. There were four batches of 1-8 citations. Journal articles were uploaded with each of those, but I don't know how big they were. Bonnie also uploaded a few summaries on Monday.

Comment entered 2017-04-19 12:15:18 by Kline, Bob (NIH/NCI) [C]

: I set the priority of the ticket for CBIIT to the highest available (not the same of the "Impact" field, which I interpret to be something like the number of users affected). Can you use your channels to see if we can get someone assigned to look at this?

Thanks!

Comment entered 2017-04-19 12:32:54 by Kline, Bob (NIH/NCI) [C]

I wouldn't want her to skew any numbers by importing things she wouldn't normally import, and I'm even reluctant to have her do anything which would alter the normal distribution of activity over time. Perhaps a path which would do what we want here without any of those undesirable side effects would be to try an import the article you're trying to bring in without checking the "Fast Track" option, and instead move it through the states as you normally would for an article imported by the librarians. Would that work? There would be two methods for doing this:

  1. enter the PMID

  2. perform a PubMed search which narrows the results to this one article (using the PMID in the search criteria) and posting the MEDLINE format for results page (I am attaching this reults page to this ticket)

Ideally, if one method failed we'd try the other one.

Looking at the MEDLINE results page for the article, I do see one odd thing. Under the block for each author the AD field (Affiliation) lists the affiliations for every author, so this information is repeated over and over again, significantly bloating the record. As weird as that is, it's hard to see how that could be related to this failure. I bet if we looked at other articles which were successfully imported, we'll see the same oddness.

Comment entered 2017-04-19 12:40:18 by Juthe, Robin (NIH/NCI) [E]

Just to clarify, there isn't anything special about the article I selected in the instructions to reproduce the problem. There have been several articles that we've tried (unsuccessfully) to import this morning. I just searched PubMed for a reasonable candidate for us to use for testing and that's where this one came from. It isn't the one that originally triggered the problem.

I'll try doing step 1 as you suggest and then ask the librarians to try step 2 (since I've never done that) if I'm unsuccessful.

Comment entered 2017-04-19 12:43:22 by Juthe, Robin (NIH/NCI) [E]

In fact, that was a really bad example since I just discovered it's already in there! However, the good news is that when I tried to import it (without checking fast track), I received the usual "statistics" information indicating that it was a duplicate. I didn't get this the last time. Let me try again with a new-to-the-system citation.

Comment entered 2017-04-19 12:44:28 by Dugan, Amy (NIH/NCI) [C]

and , I corresponded with Kiran on Jabber. He believes he fixed the issue and is asking us to check again. I'll follow up to try to find out why this happened/what fixed it. Please let me know whether everything is working as expected after checking again. Thanks!

Comment entered 2017-04-19 12:46:43 by Juthe, Robin (NIH/NCI) [E]

I just successfully imported PMID: 28301458 with the "fast track" tag. Maybe we're out of the woods?

https://ebms.nci.nih.gov/citations/full/495454

We'll try a few more before we claim "victory"...

Comment entered 2017-04-19 12:53:28 by Kline, Bob (NIH/NCI) [C]

If he fixed it that fast, my money's on disk space problems.

Comment entered 2017-04-19 13:13:05 by Dugan, Amy (NIH/NCI) [C]

Is it ok for me to tell Kiran that things look good now? He's standing by for me.

Comment entered 2017-04-19 13:16:11 by Kline, Bob (NIH/NCI) [C]

Sure as long as he tells us what he did to fix it. :-)

Comment entered 2017-04-19 13:19:44 by Dugan, Amy (NIH/NCI) [C]

FYI, from Kiran re: the issue and the fix

not disk space
1:16 PM

systems team copied a OS patch related to nss-sysinit-3.27.1 to the server to get ready for upcoming production patching on this Saturday; but it negatively impacted our apache; Apache restart fixed the issue;
1:18 PM

we are doing analysis on how we can prevent this in the future

Comment entered 2017-04-19 13:30:05 by Kline, Bob (NIH/NCI) [C]

Hmm. I wonder what "negatively impacted" means. It would be nice if they tested these patches on stage before applying them to production.

I assume "not disk space" is shorthand for "[the problem was] not [connected with the amount of available] disk space" (as opposed to "[there was] not [sufficient] disk space"). :-)

Comment entered 2017-04-19 13:46:28 by Dugan, Amy (NIH/NCI) [C]

I asked that question...and was told that this didn't happen on the lower tiers when the patch was added because the lower tiers were restarted soon after the patches were pushed. I was also told for the prod update: this is kind of preparation step to reduce the total maintenance window. I suggested that this would be a good lesson learned for the systems team.

Comment entered 2017-04-19 13:56:12 by Juthe, Robin (NIH/NCI) [E]

This is working well - we've imported several citations. Thank you!!

Comment entered 2017-04-19 14:07:20 by Kline, Bob (NIH/NCI) [C]

I suggested that this would be a good lesson learned for the systems team.

Good suggestion. I hope they take it seriously. :-)

Comment entered 2017-05-16 09:27:02 by Kline, Bob (NIH/NCI) [C]

You can close this ticket if the problems haven't recurred.

Attachments
File Name Posted User
EBMS Import Citation Error Message.docx 2017-04-19 11:14:57 Juthe, Robin (NIH/NCI) [E]

Elapsed: 0:00:00.000744