It is likely that Drupal 7 EOL will be extended until November 2022. The date in the title of the announcement has been modified to 2022, although the body of the announcement (as of July 2020) still says support will end 2021. In either case, it has been decided that we will not delay upgrading the EBMS, taking advantage of the extension (assuming the extension is real). We have a number of major enhancements planned for the EBMS which would be less costly and risky to implement after the upgrade from Drupal 7, and we do not want to defer those enhancements for another year.

Drupal 9 was released on June 3.

Upgrade Options

~bryanp has suggested that we consider alternate solutions to Drupal as we migrate from the current Drupal 7 implementation of the EBMS. I am creating a list of some of the possible options we might consider. This list is just a start, and is not the result of thorough research or analysis. I am confident that Bryan (and others) will add other possibilities, and/or eliminate some of the candidates.

Whichever of these options we choose, I recommend that we take more advantage of the underlying framework's default approach for presenting data and forms, avoiding some of the heavy customizations required by the original style design (for example, making radio buttons and checkboxes visually indistinguishable from each other).

Pure Drupal 9

The initial assumption has been that we would upgrade the EBMS to Drupal 9 (bypassing Drupal 8, which will reach end-of-life at the same time as Drupal 7). One of the criticisms of the original implementation of the EBMS has been that we did not take full advantage of the services Drupal provides. In particular, the storage of the application's core data uses custom database tables instead of Drupal entities, which form the heart of Drupal data management. Part of the reason for this (beyond the constraints of the schedule) is the fact that when the EBMS was first designed and implemented these services were just emerging. When the first Drupal prototype of the EBMS was built, entities did not exist except in a beta version of Drupal. Now that these services have had some time to evolve and mature, it would seem to make more sense to use them. The arguments in favor of this approach are that we would benefit from work the Drupal core team has done to support creating, managing, finding, and displaying the application's primary data types (the most important of which, in the case of the EBMS, being PubMed articles). On the other side of the ledger, it would probably take more effort to convert the existing tables to entities, which introduce additional layers of complexity, and we know from experience that the latest version of Drupal still has some serious bugs.

Modified Drupal 9

We could upgrade to Drupal 9, but do it in a way that continues to use the custom tables for PubMed articles and their states, board information, reviews, etc. This would still be a major undertaking, but almost certainly less work than converting those to entities. This approach would do nothing to address the valid criticisms of the approach taken by the original implementation, and we'd still have to deal with Drupal bugs (just not as many of them).

Non-Drupal Solutions

The options listed here (at least the initial list) take less of "kitchen sink" approach than Drupal, which tries to provide canned solutions to every problem.

Symfony

Symfony is the PHP framework on which Drupal 8 and later have been built. This solution would combine that framework, which provides a rich set of components and the same templating system used by Drupal, with a client-side CSS framework such as Bootstrap, and take advantage of the data structures already used by the current EBMS implementation. We have plenty of developers familiar with PHP and Composer (the package manager for PHP).

Flask

Flask is a Python-based web micro framework, built on the Werkzeug WSGI web application library, using Jinja for templating. Bootstrap integration would provide client-side support for styling and scripting. We have fewer Python programmers on the team, but the staff primarily responsible for EBMS development are proficient and productive in that language.

NodeJS

NodeJS is perhaps the lowest-level solution in this list. I have no experience with this framework (not even enough to know whether calling it a "framework" is appropriate), but its popularity seems to have jumped in the last couple of years. I believe that some of the WCMS components are built with NodeJS, and the number of developers on the team with JavaScript skills is growing.

Feedback Solicited

Again, this is very much just a straw man (straw person?), intended to get the conversation started. Feel free to contribute suggestions, criticisms.

I'd like to revive this discussion about the migration of EBMS to Drupal 9 or another platform. We've been setting aside some larger enhancements to the system keeping in mind that we're going to be migrating soon, but it would be good to know if this is still in the rough timeline for 2021, even if Drupal 7/8 EOL has been extended to 2022. If it isn't likely that we'll do this in 2021, then we should talk about whether to make some of those enhancements in the meantime.

Also, ~bkline, I don't have much to contribute in terms of these various platforms from a technical side of things and I'm guessing that feedback request wasn't for me. 🙂  But if it was, and if there are business-related differences between then in terms of capabilities or pros/cons of the various frameworks, maybe we can discuss those. Thanks!

At this point I'm going on the assumption (partly based on the fact that Bryan didn't weigh in – which makes me think he was probably joking when he threw out the comment about upgrading to something other than Drupal – but mostly based on my own observations) that we'll be migrating to Drupal 9, and using more of the features Drupal has to offer, replacing our custom tables with Drupal entities. I don't think the decision about which technical approach we should use will be driven by business considerations, but if that changes I'll be sure to let you know.

Approach

After taking a closer look at the choices I have decided that our best option is do stick with Drupal, rewriting the application to take advantage of the features provided by Drupal 9. At the highest level, this will mean replacing most of the custom tables used in the existing implementation to store information about EBMS-specific data with Drupal entities.

Entities

Replace most of the current 80 custom EBMS database tables with entities.

Content entity types

These are the major entity types used to represent the top-level information in the EBMS.

Article

This is the core of the EBMS system. The current system implements an extremely complicated page (known as the "full citation page") which displays all the information about a given article, with lots of editing-in-place capabilities. Recreating that page will be a separate task, rather than use the stock Drupal entity editing scaffolding. In later years, when I was asked to add new elements (such as board manager comments or article relationships) I used the more traditional approach of bringing up a separate form, instead of shoehorning more popups and magically appearing inline forms on the existing page, and the users are fine with that. I'm inclined to apply that approach to any editing of the article in the rewrite. There will still be plenty of complexity in organizaing and display all the information for viewing. Investigate the use of tabs, collapsible sections, etc., especially if we have the support of something like the Foundation framework.

• title

• authors (custom field with last name, forename, initials, collective name)

• source (e.g., 'Pubmed')

• source ID

• source status

• source journal ID

• journal title (at the time the article was published)

• brief journal title

• brief citation

• abstract

• published date

• import date

• update date

• imported by

• data modified

• data checked

• status

• full text unavailable (custom field: datetime flagged, flagged by, comment)

• internal tags (custom field: tag, date added)

• internal comments (custom field: user, when posted, comment text)

• topics (ArticleTopic entity references)

• tags (ArticleTag entity references)

• legacy ID

ArticleRelationship

These get displayed on the "full citation" page referred to above. Need a separate editing form, but not a separate entity view.

• from article

• to article

• type

• creator

• created

• comment

• inactivated by

• inactivated

ArticleState

Entity representing the per-topic state of an article at a given point in time. Displayed on the "full citation" page and in reports. There is no single editing form, as transition to the various states has state-specific interfaces.

• state

• topic

• board (in case the topic's board changed since the state was entered)

• user

• date/time

• active

• current

• comments

• decisions (references to BoardDecision entities)

ArticleTag

Needs add/edit form but not separate display (shown on the "full citation" page).

• tag (reference to ArticleTagType configuration entity)

• topic

• user

• date/time

• status

• comments

ArticleTopic

Displayed on the "full citation" page and in reports. The bulk of these are established as part of the import process. Attaching additional topics can happen on the review queues or an editing form reached from the "full citation" page.

• topic

• cycle

• comments (see if it works to use Drupal core Comment entity references)

Board

Needs list display only, as well as add/edit form. The docs field is used to identify the supporting documents which will be listed at the bottom of the primary Summaries landing page for the board.

• name

• manager

• guidelines (reference to Document entity)

• auto imports (Boolean)

• docs (custom field: Document entity reference, archived, notes)

BoardDecision

Displayed on "full citation" page and reports. Needs separate add/edit form.

• decision (one of a list of valid values - maybe another config entity type?)

• meeting date (reference to Cycle configuration entity)

• discussed (Boolean)

• members (board members who participated in the decision)

Document

Needs list view and add/edit form.

• file

• posted

• description

• dropped

• tags

• boards

• topics

Event

Used for PDQ Board meetings. Listed through the calendar. Needs add/edit form.

• name

• status

• date

• category

• type

• boards

• subgroups

• ad-hoc groups

• individuals

• agenda (rich text field)

• agenda status

• meeting documents

• notes

• article states (references to ArticleState entities)

HotelRequest (currently webforms)

See notes on third-party modules.

• meeting (event entity reference)

• checkin date

• checkout date

• preferred hotel (valid values picklist)

• comment

ImportAction

An article can be represented more than once for a given import batch. For example, a record imported from NLM might already be in the database but not with the same topic as used in this import batch, and with a different cycle_id. It might also be a later record from NLM. In such a case there are two import disposition values - "Summary topic added" and "Replaced." Populated by the import software. Displayed in reports. No other UI needed.

• article

• disposition (reference to ImportDisposition entity)

• message

ImportBatch

Populated by the import software. Displayed in reports.

• topic

• source (typically Pubmed)

• import date/time

• cycle

• user

• journal suppression in effect (Boolean)

• input type (Regular, Fast track, Special search, Data refresh, Internal)

• article count (number of unique article IDs)

• comment

• messages (error messages produced by the import process)

• status (Success, Failure)

• actions (references to ImportAction entities)

Journal

Hold the current information for the journals. The journal fields in the Article entities capture information about the journal as given by the source (NLM) reflecting the time of publication of the article. So if a journal's title changes, the two locations might not necessarily match. Information for the journals is refreshed from NLM on demand. The set of "core" journals is so stable that I don't believe the interface implemented for adding a journal to or dropping a journal from the set has been used since the EBMS was first implemented.

• source (e.g., Pubmed)

• source ID

• title

• brief title

• core (Boolean)

JournalSuppression

Known affectionately to the users as the "Not List." Captures information about journals which, for a certain board, should have their articles rejected by default when an import request from Pubmed search results is submitted. The suppression can be overridden, but managers of the boards have learned over time which journals aren't usually worth reviewing. There is an interface for showing which journals are or are not on a board's automatic rejection list, as well as adding journals to or removing them from that list, but these entities don't have a proper editing form.

• journal

• board

• start date (when the decision to automatically reject took effect)

• user (who made the decision)

Packet

Gets a moderately tricky add/edit form, with cascading ajax calls to display and populate fields/picklists based on choices made earlier for other fields. For example, the user selects a topic, and checkboxes for the articles eligible for board member review appear. If the user then goes back and switches to another board then the articles and topics are replaced with topics associated with the newly selected board. By default the board members who are the default reviewers for a given topic appear on the form, but display of all of the board members can be toggled. Not all of the entity's fields appear on the add packet form. For example, "starred" is something which the board manager can set later on.

• topic

• created by

• created at

• name

• last seen

• status

• starred

• reviewers

• summaries (Document entity refs)

• articles (custom field: Article entity ref, dropped, archived)

• reviewer docs (reference to ReviewerDoc entity)

PrintJob

Captures the request parameters for a particular print job. Some requests will be to print packets for one particular board member, for example, or all of the packets created in a particular date range for a given board.

• reprint for (PrintJob entity reference)

• user

• job type

• packet start (only include packets created at or after this date/time)

• packet end (only include packets create before or at this date/time)

• printed (actual date/time packet was printed)

• board

• board member

• packet

• status (PrintStatus entity ref)

• mode (live, test, or report)

• comment

PrintedPackets

Used to keep track of which packets have already been printed for which board members. Maintained and used internally (no editing or display form).

• user

• packet

• print job

ReimbursementRequest (currently webforms)

See notes on third-party modules.

• meeting (event entity reference)

• arrival date

• departure date

• transportation expenses (custom field: date, type, amount)

• parking/toll expenses (custom field: date, type, amount)

• hotel payment (NCI paid, I paid, did not stay in a hotel)

• nights stayed (integer)

• hotel amount

• meals (per diem requested, per diem declined, not eligible for per diem)

• honorarium (requested, declined)

• reimbursement (work, home, other)

• total requested

• comment

• certify

• email address

Review

Need add (but not edit) form and display page.

• packet

• article

• reviewer

• posted

• loe_info

• dispositions

• rejection reasons

ReviewerDoc

Documents reviewers have posted to packets. Posted/displayed on the packet's page. Needs add form.

• document

• reviewer

• dropped (Boolean - has the posting of the document been retracted?)

• posted (date/time)

• title - display identification string for the document

• description - optional

SummaryPage

Needs list page, display page, add/edit form.

• board

• name

• archived

• topics

• links

• NCI docs (custom field: doc, archived, notes)

• reviewer docs (custom field: doc, archived, notes)

Topic

Needs list page and add/edit form (but not a separate display page).

• name

• board

• NCI reviewer

• status

• topic group (custom field, taxonomy, custom configuration entity?)

• default reviewers

Configuration entity types

These are lighter-weight entity types for things like valid values lists with supporting attributes. For at least some of these, taxonomy vocabularies might have been a tempting option, were it not for the fact that taxonomies add unnecessary overhead, such as versioning.

AdHocGroup

• name

• creator

• boards

ArticleTagType

• machine name (e.g., 'q_search')

• display name (e.g., 'Questionable search')

• description

• topic required (Boolean)

• status (Active/Inactive)

Cycle

Literature review is organized in review cycles. No form needed (these are created by the system).

• name

• start date

Disposition

• description

• position

• instructions

ImportDisposition

• machine name (e.g., topic_added)

• display name (e.g., Summary topic added)

• description

• status (Active, Inactive)

Internal tag

Used to mark imported articles not intended for review

• name (valid value for tag, e.g., 'SEO')

• active (Boolean)

PrintJobType

• machine name (e.g., 'packet')

• description (e.g., 'Printing a single packet for a user')

PrintStatus

• name (e.g., 'success')

• description (e.g., 'Job completed successfully')

Relationship type

• name (e.g., Article/Editorial)

• description

• active

Rejection reason

• name

• status (Active, Inactive)

• position

• instructions

State

• name

• display

• description

• sequence

• completed (Boolean - is this a terminal state?)

Subgroup

• name

• board

Tag

• name

• comment

Custom tables

These are tables which probably wouldn't benefit from the functionality provided by Drupal entities. They have no forms, they are not displayed to users, they're not searched, and they don't participate in processing on the site.

Import request

• params (JSON-encoded request parameters)

• report (JSON-encoded results of the import job)

Publish queue

• when created

• user

• filter (JSON-encoded selection criteria)

• article states (identifies articles, with topic and state)

Pubmed results

Captured Pubmed search results submitted by the librarians to the import process, used by the developers for troubleshooting problems with NLM.

• when - date/time the results were submitted for import

• results - file contents for the results

Report requests

• name

• user

• submitted

• parameters

Search

• when submitted

• user

• parameters

• type (db, cite-queue, fulltext-queue)

Custom fields for the User entity type

• boards (boards of which the user is a member or manager; usually one)

• topics (topics for which the users is a default reviewer)

• subgroups

• ad-hoc groups

• wants print (multiply-occurring custom field: start date, end date)

Third-party modules

The following non-core modules are used by the current EBMS software.

Automated Logout

Supported with a stable Drupal 8/9 release (8.x-1.3 released 18 April 2020).

Chain Menu Access API

I'm not aware of why this module is enabled for the EBMS. The description on the Modules page says "Helps client modules to chain their access callbacks into other modules' menu items." The first sentence of the project page's description says "Chain Menu Access API is has [sic] no functionality on its own — install it only if another module requests it." There are no modules listed as depending on this module, so it can probably be dropped.

Chaos tools

The bulk of the functionality provided by this module has been incorporated into Drupal core. We probably won't need to functionality which was not moved into core.

CKEditor Link

This is used to populate the link dropdown for matching documents as the user types into the field for a rich-text agenda. The module doesn't support Drupal 9, and it doesn't appear that it ever will. Investigate alternatives.

Custom Permissions

Required by NCI User Roles, which is not enabled, so we should be able to do without this extension.

Date/time

A set of modules providing calendar functionality. This has been one of the weak links in the EBMS systems because for a long time the project was completely abandoned (though without any acknowledgement from the maintainer that she was doing so). The Calendar module currently has only an alpha release available for Drupal 9, with no clear path to a stable release. Our best option for a calendar page in the EBMS may be to use a client-side library in combination with our own Event entity type.

Entity API and Entity token

Part of Drupal core now.

Features

This module was installed a few years ago by Yijun Zhang ("Edream"?) as I recall, to help manage settings for the EBMS. This is no longer the recommended tool for managing configuration in Drupal 8 and beyond.

Libraries

Most of the functionality for this extension is now incorporated in Drupal core. I'm hoping that will cover what we need, as this project only has an alpha release for Drupal 8.x, and no mention of Drupal 9. If it doesn't cover what we need, first step will be to find out if the new DP faced this problem and if so, what solution it adopted.

Message

Used for queueing notifications for things like recently added packets or reviews. Most of these appear on the user's home page. Still maintained, with a stable Drupal 8/9 release (8.x-1.1 released last month).

NCI SSO Module

Not completely appropriate to call this a "third-party" module, but it isn't specific to the EBMS. We'll need to rewrite it for Drupal 9, or find out if the team has already done so for another site.

Role delegation

Required by NCI User Roles, which is not enabled, so we should be able to do without this extension.

Token

Not sure we need it, but if we do, it's compatible with Drupal 9. Latest stable release is 8.x-1.7 (26 April 2020).

Video Filter

There is an alpha version available for Drupal 8, and nothing for Drupal 9. There haven't been any commits for this project for years, which is discouraging. There is some movement in the issues, though, including a ticket for Drupal 9 readiness. A quick search hasn't turned up alternate modules. Find out what cancer.gov does and possibly follow that example.

Views and Views UI

Included with Drupal core.

Webform

In the current system, this extension is used for travel reimbursement and hotel requests. Webform is the second most frequent source of compatibility problems encoutered in the EBMS over the years (behind the date/time modlues), so it might be good to drop the dependency on this module, as noted in the discussion of entities above. There are two branches for the Webform module, and they both use a completely new approach from the Drupal 7 versions, so we'll be rewriting the travel code anyway. Only one of the branches supports Drupal 9. The other requires the installation of the jQueryUI tabs, tooltip, and date picker modules.

Wysiwyg

Included with Drupal core.

WYSIWYG Filter

This has been included with Drupal core.

Theme

Drupal 8 completely replaced the theme engine used by earlier versions of Drupal, so the theme will need to be completely rewritten. We can use the existing layouts to guide what we create for Drupal 9, which makes the work easier than if we were building a new site from scratch. We will probably want to modify some of the visual design elements (assuming client approval) so that we're not straying so far from the system's defaults and standard practices. For example, I think everyone will agree that the original decision to make the site's radio buttons visually indistinguishable from checkboxes was a mistake.

I'm deferring the decision about whether to use a framework such as Foundation or Bootstrap in our theme. It would introduce yet another component with whose versions we would need to keep in sync, and such a framework might be more heavyweight than our needs would justify, but it might simplify the theming work we'll need to do. I believe Cancer.gov is using Foundation, so we might want to consult with the front-end developers to get their advice.

Tasks

Here's a list of the major tasks we can anticipate, with very rough estimates of how long each might take.

tl;dr

• The EBMS will be updated to use Drupal 9

• The EBMS has 80 custom tables, most of which will be replaced by Drupal 9 content and configuration entities.

• Many third-party modules have been moved into Drupal core; of the rest, we may need to abandon the Calendar and Webform modules, which do not have Drupal 9 releases and have a poor support track record.

• Consider whether the Foundation framework, which is used by cancer.gov, should be used as part of the theme

• A rough estimate is that the rewrite will take 30 person weeks.

The EBMS data in the rewritten system will be stored in entities documented in the project's GitHub wiki.

The EBMS has been rewritten in Drupal 9.