CDR Tickets

I noticed there exist terms without the NCIThesaurusConcept element although a C-code exists and is published.
Should these terms be handled as if the C-code does exist or not?

They should be treated as if the concept code does not exist.I assume at this point we are assuming the only true source of the code is the NCIThesaurusConcept. Meanwhile, is it possible to get a list of such terms?

Another question:
A comment in the filter code specifies to suppress the active clinical trials link if a DrugInfo Summary links to the current document.
Is this still the right thing to do after our switch to CTS?

I see some terms with only the active clinical trials link and some with only the NCI Thesaurus link. Other than the exception listed in the previous comment shouldn't both links get created for a C-code that's published?

I am not sure why this was specified. Margaret may be able help answer this question.

That is right. I don't see a reason why they shouldn't have both links.

Meanwhile, is it possible to get a list of such terms?

Yes, that's possible. However, I can not identify a clear pattern of how to pick up the existing C-code from the term document. I see the C-code is sometimes entered as part of the OtherName with a SouceTermType = PT (primary type, I suppose), but there are also term documents where I find the C-code listed as part of the definition block and a primary type does not exist.

This doesn't appear to be a simple SQL query and could turn into a project itself. Please create a separate ticket for this cleanup report.

I made the filter changes and ran a diff report on QA. Many (all?) of the clinical trials link errors will be resolved by dropping the sentence _Check for active clinical trials ... _ or replacing the c-code with a valid code.

However, I see many links to the NCI Thesaurus that are changing due to the existence of multiple C-codes for a single term.
~oseipokuw, how should the filter identify which C-code to use when creating the link?

See, for example, suramin (CDR40052) with the two C-codes: C853, C1848

Adding ~mbeckwit as a watcher to answer this question. I left the filter code as is for now, meaning the link to clinical trials will not be created for DIS terms.

The following filter has been updated to suppress creating the clinical trials link if the term has not been published yet.

• CDR134: Vendor Filter: Term [cts-hotfix 5c3fd00d]
  https://github.com/NCIOCPL/cdr-cts-hotfix/commit/5c3fd00d

I ran a diff report on QA and confirmed that I didn't break anything. There were two types of differences:

1. The broken links were removed as expected

2. Some of the links were created to a different C-code due to multiple existing C-codes for a single term (see comment above).

I published the changed filter to STAGE and PROD and ran a hot-fix for one of the reported drugs with a broken link:
https://www.cancer.gov/publications/dictionaries/cancer-drug?CdrID=791102

All terminology documents will be published as part of the nightly publishing job.

The drug terms have been updated as part of last night’s publishing job. Those broken links reported should not be displayed anymore. ~oseipokuw is still investigating the terms with multiple C-codes for the NCIThesaurusConcept elsewhere.

Please verify on PROD and close this ticket.

For my information.
The branch cts-hotfix has been merged into master of the cdr-server repository and has been deleted.
https://github.com/NCIOCPL/cdr-server/commit/3cb7610a

~oseipokuw, these filter changes are on PROD since October.
Is there anything left to do or can we close this ticket?

We haven't completed fixing the multiple codes problem. That is why I have it open. If you don't mind, I'd like to keep it open until we are done.

I didn't know we were waiting on the content fix of the multiple C-codes.
We can keep the ticket open if you prefer. Just close it once the cleanup is completed.

~oseipokuw, I was asked why so many drug terms are missing the C-code?

As we discussed in comments above, the C-code appears in many different places in the term document. We had fixed the links to the NCIThesaurus as part of this ticket by picking up the C-code from the NCIThesaurusConcept element. Unfortunately, there are now may terms with a published C-code in the EVS that don't have a link because the NCIThesaurusConcept isn't available in the CDR term document (i.e. nutraceutical TBL-12 (CDR696523)).

Another problem I see is that we have some documents that include the attribute NCIThesaurusConceptID as an attribute in the term document which is populated by the SourceInformation/VocabularySource ID which is currently missing for 2000+ term documents.
Shouldn't this attribute be populated by the NCIThesaurusConcept ID as well?

As we discussed in comments above, the C-code appears in many different places in the term document. We had fixed the links to the NCIThesaurus as part of this ticket by picking up the C-code from the NCIThesaurusConcept element. Unfortunately, there are now may terms with a published C-code in the EVS that don't have a link because the NCIThesaurusConcept isn't available in the CDR term document (i.e. nutraceutical TBL-12 (CDR696523)).

We used to do a global periodically to update the NCI Thesaurus IDs but we haven't done a global for several years now. Can we do what something similar to what we do for the glossary term links? That is, there is no C-code, then no link should be created.

Another problem I see is that we have some documents that include the attribute NCIThesaurusConceptID as an attribute in the term document which is populated by the SourceInformation/VocabularySource ID which is currently missing for 2000+ term documents.
Shouldn't this attribute be populated by the NCIThesaurusConcept ID as well?}

Are all the 2000+ terms drug terms? Can you please provide me with a few examples ?

Are all the 2000+ terms drug terms? Can you please provide me with a few examples ?

I believe those 2000+ are all drug terms. Here are a few of those but I can send you the entire list if needed.

37853|MVA-MUC1-IL2 vaccine
37880|enoxaparin sodium
37977|phenethyl isothiocyanate
38100|vincristine sulfate liposome
38202|DC-cholesterol liposome
38221|gavilimomab

Thanks! All these have the concept codes at the end of the documents which I believe is what is used to create the link. I also checked Cancer.gov for the first one MVA-MUC1-IL2 vaccine and it appears the links are working correctly. Can you please explain the problem with these ?

Yes, they have the correct concept code used for the link because the link get's created by using the element NCIThesaurusConcept but they don't include the corresponding Term attribute at all, so Cancer.gov doesn't know what the C-code is. The filter doesn't populate the attribute NCIThesaurusConceptID from the element NCIThesaurusConcept which, in my opinion, is wrong.

Please compare the output of these two filters and look at the Term root element:
https://cdr.cancer.gov/cgi-bin/cdr/Filter.py?DocId=37880&DocVer=lastp&newdtd=pdqCG.dtd&filter=set%3AVendor%20Term%20Set

https://cdr.cancer.gov/cgi-bin/cdr/Filter.py?DocId=789792&DocVer=lastp&newdtd=pdqCG.dtd&filter=set%3AVendor%20Term%20Set

I see the difference. Would you modify the filter to add the Term attribute on the NCIThesaurusConcept element ?

Yes, if we decide that the SourceInformation/VocabularySource/SourceTermId is not the proper source for the NCIThesaurusConcept information we will have to modify the filter.
However, a clean-up effort should probably go along with this to ensure all of the terms listing the concept ID in the SourceTermId field do include the /Term/NCIThesaurusConcept as well.

This ticket was left open until CIAT had fixed the multiple codes problem. According to ~oseipokuw, this is now completed and we can close this ticket.

Created a new ticket addressing the problem of the missing attribute with the C-code for the Term element.