CDR Tickets

First question – do we want to drop some of the document types which are less interesting than they used to be from consideration, both for the URL check report, as well as for this global change? The current list of document types handled by the URL check report has:

• GlossaryTermConcept

• Summary

• InScopeProtocol

• CTGovProtocol

• DrugInformationSummary

• Person

• ClinicalTrialSearchString

• Citation

• MiscellaneousDocument

• Organization

Why (for example) would we care about dead links in InScopeProtocols?

Would I be right in assuming that we should use the same set of document types we adopted for OCECDR-4219 in performing this global change?

• Citation

• Clinical Trial Search String

• Drug Information Summary

• Glossary Term Concept

• Miscellaneous Doc

• Summary

Yes, same document types. Thanks, Bob.

I have attached a spreadsheet showing the old and new URLs for all of the redirects from "http" to "https" for external refs found in the CDR documents on PROD for the types we decided on. Please review the list and confirm that these are the mappings you want the global change to use.

I've spot checked several of these and they look reasonable. I think we can proceed. Thanks, Bob.

I also reviewed some of these and they appear to be fine. There are several links to the mobile site in patient summaries that we may have to cleanup manually since we stopped storing mobile links since NVCG. Examples:

http://m.cancer.gov/topics/treatment/bycancer/esophageal/Patient
http://m.cancer.gov/es/cancer/tratamiento/porcancer/esofago/Patient

~bklineIf it is not too much work, could you please include the CDR IDs of the documents in the spreadsheet?

You understand that that would be a significantly bigger spreadsheet, right? Because the one I gave you only showed each unique URL once. So, for example, that spreadsheet had one row for http://www.cancer.gov/about-cancer/treatment/clinical-trials, whereas the one you're asking for would have as many as 249 rows. Are you sure that will be as helpful as you think it will?

Does it mean you can't show me just one occurrence of each unique URL in the spreadsheet? How about if I Identify all the mobile URLS, can you provide me with the CDR IDs for just that?

How's this? I flattened out the rows for the CDR IDs to squash them all into a single cell per URL. New workbook attached. Note that sometimes the same URL could appear more than once in a given document.

Will this meet your needs?

Please remove all mobile URL elements. We no longer need to store these in the CDR so we don't need to include them in the global.

Implemented and run in test mode on DEV. Ready for user review.

https://cdr-dev.cancer.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2017-02-07_18-16-10

I checked several docs and this looks good to me. Mobile URLs are being removed and the URLs are being updated correctly. Could you please run the global in live mode on DEV?

Running now. Will let you know when it's done.

The live global job is finished. It seems most likely that the failure of the job yesterday had to do with transitory problems on the database server, as the same document was updated without any problems when I re-started the job. I have added more diagnostic support to the ModifyDocs module to assist in tracking down any similar failures in the future.

Please spot check the results of the global change.

I'm still reviewing, but I have found a couple of documents that appear to only have been partially updated:

CDR470863 (Merkel Cell patient summary - Spanish)
The Mobile URL element was removed and several URLs were updated but the Summary URL was not updated to https. It is on the spreadsheet (row 1685 in the version with CDR IDs).

CDR441548 (Merkel Cell patient summary - English)
The Mobile URL element was removed but the Summary URL and a few other URLs were not updated to https. (rows 616, 668, 2198, 2208 on the spreadsheet with CDR IDs)

I see the problem. I wasn't correctly handling documents in which the same URL was entered inconsistently (with regard to case) in the same document. I fixed the problem and ran a fresh job. There were a couple of locked documents, but most of the URLs that could be mapped were replaced. There are still 3,408 external refs using the HTTP scheme, but these are plain old dead links, which need to be dropped or replaced by hand (e.g., http://cancer.gov/cancerinfo/pdq/screening). They'll show up on the URL check report.

Before you do any further review, let me examine the documents myself. I'm still not convinced everything's working correctly.

OK, now I think we may really be ready for you to review the results of the live global change on DEV. I discovered that the web server was inappropriately closing the connection when we asked for the redirect of http://www.cancer.gov/cam instead of returning the 404 ("not found") HTTP code. This caused the script to record "www.cancer.gov" as a host which was off line. The script had been optimized to avoid trying to contact a server which had been determined to be unavailable, causing all subsequent URLs for that host to be skipped. I have removed the optimization and now the script is trying each unique URL, even those for which the host has misbehaved.

Please take a look at the results.

Is CDR517309 on the list of locked documents? I had that checked out until about 12:30pm today. If it is, then I'm comfortable saying that everything looks good!

Yes.

2017-02-17 11:30:52: Document 258011: Unable to check out CWD for CDR0000258011: Document CDR0000258011 already checked out to user oseipoku
2017-02-17 11:33:45: Document 258038: Unable to check out CWD for CDR0000258038: Document CDR0000258038 already checked out to user oseipoku
2017-02-17 11:39:03: Document 350260: Unable to check out CWD for CDR0000350260: Document CDR0000350260 already checked out to user volker
2017-02-17 11:47:25: Document 487564: Unable to check out CWD for CDR0000487564: Document CDR0000487564 already checked out to user volker
2017-02-17 11:49:31: Document 517309: Unable to check out CWD for CDR0000517309: Document CDR0000517309 already checked out to user juther
2017-02-17 11:50:50: Document 538354: Unable to check out CWD for CDR0000538354: Document CDR0000538354 already checked out to user lgrama
2017-02-17 12:07:03: Document 622340: Unable to check out CWD for CDR0000622340: Document CDR0000622340 already checked out to user volker
2017-02-17 12:10:39: Document 632567: Unable to check out CWD for CDR0000632567: Document CDR0000632567 already checked out to user oseipoku
2017-02-17 12:28:30: Document 747922: Unable to check out CWD for CDR0000747922: Document CDR0000747922 already checked out to user volker
2017-02-17 12:33:59: Document 760794: Unable to check out CWD for CDR0000760794: Document CDR0000760794 already checked out to user volker
2017-02-17 12:34:00: Document 760795: Unable to check out CWD for CDR0000760795: Document CDR0000760795 already checked out to user volker

Great, thanks! This is verified on DEV then.

What's the status of the test global run on QA?

All done.

https://cdr-qa.cancer.gov/cgi-bin/cdr/ShowGlobalChangeTestResults.py?dir=2017-02-23_12-51-24

The test results look good to me. Please do the live run. Thanks.

Hi Bob,

Just a quick reminder to please do the live run when you get a chance. Thanks! 🙂

I tried the run yesterday, but it broke (looks like the database wandered off for a bit), so I had to start it again. Still running.

Failed again, restarted again. The program doesn't have to start all over again, and the document at which the previous failure occurred succeeds on restart, which I take to mean that there isn't something in the code or the documents which is causing the failures. Will keep you posted.

Thanks for the update!

Finally got a clean run. Ready for you to take a look.

Verified on QA.

The live global change has completed on PROD. No documents were skipped because they were checkout by a user.