Demystifying drupal-core-require-dev and drupal-core-strict in the "Drupal Composer/Drupal Project" Composer template

If you build Drupal 8 sites using the Drupal Composer/Drupal Project Composer template (DCDP), then you've likely noticed the development dependency webflo/drupal-core-require-dev. If you're like me, you probably didn't give it much thought the first 20 or 30 times you used the template. 

After a while though, I started to dig deeper into the details of DCDP, wanting to be able to understand exactly how it worked and what customizations I may want to make. DCDP was really my first real exposure to Composer, and the more I learned, the more I wanted to learn (as is often the case). My curiosity led me to this drupal-core-require-dev rabbit hole.

Some background

First, let's level-set ourselves - when you run either "composer install" or "composer create-project" (which is actually calling "composer install" as well) without the "--no-dev" switch, Composer will install all dependencies listed in your composer.json file in both the "require" and "require-dev" sections (as well as dependencies of dependencies). If you take a look at DCDP, you'll notice that in the "require-dev" section, there is one entry: webflo/drupal-core-require-dev. 

So, as most folks who start Drupal 8 projects using the recommended DCDP command listed in the README (composer create-project drupal-composer/drupal-project:8.x-dev some-dir --no-interaction), Composer is installing everything in the "require" and "require-dev" sections - including webflo/drupal-core-require-dev.

What exactly is webflo/drupal-core-require-dev? Well, it is a "virtual" dependency - meaning it doesn't include any code, rather it just includes a composer.json file that specifies the specific versions of Drupal core development ("require-dev") dependencies that are used to run Drupal core tests. The interesting (and sometimes problematic bit) is that webflo/drupal-core-require-dev doesn't specify any versions for non-development ("require") dependencies. If you take a look at Drupal core's composer.json file, you'll see that for the most part, specific versions of dependencies aren't specified - rather a range is. 

This leads to the situation where a project built with webflo/drupal-core-require-dev could have different dependency versions (as long as they adhere to the version constraints is Drupal core's composer.json) than what comes with Drupal core if you had just downloaded it from drupal.org.

For example, if on the date version 8.7.0 of Drupal core was released one of the development dependencies was at version 1.3.1, then that is the version that is provided with Drupal core 8.7.0 downloaded from drupal.org regardless of when you download it. But, when using the DCDP as-is, if since the release of Drupal core 8.7.0 the development dependency was updated to 1.3.2, then when the project is installed using "composer create-project", your project will be using version 1.3.2 of the dependency. While this seems minor, it has led to some issues

Also - be aware that there are different versions of webflo/drupal-core-require-dev for every minor version of Drupal core. So, if you're updating your site from Drupal core 8.6.x to 8.7.x, then you must also update to webflo/drupal-core-require-dev to 8.7 as well. This is the reason the update command for DCDP includes webflo/drupal-core-require-dev:composer update drupal/core webflo/drupal-core-require-dev "symfony/*" --with-dependencies

After learning this, I had an obvious question: what's the advantage of having Composer install updated versions of Drupal core dependencies? The only thing I found was that if you're a core or contrib developer, then it would be useful to know if your code breaks using updated dependencies. I'm hard-pressed to think of another reason when this makes sense. For most Drupal 8 projects, I think it would be beneficial to use the exact dependencies that the particular version of Drupal core ships with. This way, we can be 100% certain that our project has the same dependency versions that the community's testing infrastructure has validated for the particular version. Luckily, that's what webflo/drupal-core-strict is for. 

It works almost the exact same way as webflo/drupal-core-require-dev except that it includes exact versions for all dependencies of Drupal core - for both development ("require-dev") and non-development ("require") packages. The exact versions are the ones that have been tested and are included in the "official" version of Drupal core (for each minor version) downloadable from drupal.org. Like webflo/drupal-core-require-dev, there is a minor version of webflo/drupal-core-strict for each minor version of Drupal core.

So, why does DCDP use webflo/drupal-core-require-dev? Well, there's some debate about if it should or not. 

As a side-note, if you host on Pantheon, and use their Pantheon-flavored version of DCDP, then you're probably already using webflo/drupal-core-strict.

Starting a project with DCDP using webflo/drupal-core-strict

First, the bad news - if you want to start a new project using webflo/drupal-core-strict, you can't use DCDP out-of-the-(virtual)-box. But, there's a couple of possibilities. At first glance, it seems that you could fork DCDP, make the relevant change to webflo/drupal-core-strict in the composer.json file, then use "composer create-project" on your fork. But, this would also require posting your fork on Packagist (which is discouraged), updating your fork's README (for the create-project and update commands) as well as keeping your fork up-to-date with any DCDP updates. I wouldn't recommend this method.

A better option is to use the "--no-install" option of Composer's "create-project" command:

1.  Use the recommended command on the DCDP page, but add a "--no-install" at the end of it:

composer create-project drupal-composer/drupal-project:8.x-dev some-dir --no-interaction --no-install

This will download DCDP to your local, but not install dependencies. 

2.  Edit the composer.json file with:

  • New project name
  • New project description
  • Remove "webflo/drupal-core-require-dev" from the "require-dev" section
  • Add "webflo/drupal-core-strict": "^8.7.0", to the "require" section (ensure the version matches drupal/core).
  • Change the version requirement for drupal/console to: "drupal/console": "^1.0", (to avoid version conflicts)
  • Change the version requirement for drush/drush to: "drush/drush": "^9.0", (to avoid version conflicts)
  • Remove "composer/installers" from the "require" section (it is already specified in webflo/drupal-core-strict). 

3.  Run "composer install". 

You'll need to remember that when you want to update Drupal core, you'll want to use the following command (instead of what is in the DCDP README):

composer update drupal/core webflo/drupal-core-strict "symfony/*" --with-dependencies

If you're not crazy about either of these two options, there is a third (future?) - leave a comment on this issue and ask for webflo/drupal-core-strict to be used in DCDP. 

Change an existing project from webflo/drupal-core-require-dev to webflo/drupal-core-strict

What if you already have a project based on DCDP and you want to change it from using webflo/drupal-core-require-dev to webflo/drupal-core-strict? Here's some possible ways of doing it:

As always, to be safe, please test things like this on a copy of your project.

Method one: manually downgrade dependencies

This is definitely a tedious process. It involves first removing webflo/drupal-core-require-dev using:

composer remove webflo/drupal-core-require-dev

Then, attempt to require drupal-core-strict:

composer require webflo/drupal-core-strict:^8.7.0

Depending on a number of factors you're likely to get a bunch of "Your requirements could not be resolved to an installable set of packages." messages. How many you get is mostly a result of the length of time since the previous minor release of Drupal core - the longer it has been, the more dependencies have probably been updated. For each dependency listed, you'll need to downgrade it using something like:

composer require symfony/yaml:3.4.26

What is happening is that webflo/drupal-core-require-dev allows dependencies to get upgraded outside of the Drupal core release timeline, while webflo/drupal-core-strict does not. So, you'll need to downgrade dependencies that have been updated. You'll have to do it one-at-a-time - try requiring webflo/drupal-core-strict, see the error message, downgrade the offending dependency, then repeat. In some cases, it isn't immediately obvious which dependency needs to be downgraded, or which version it needs to be downgraded to, so be prepared to use the "composer depends" command a few times. 

Eventually, requiring webflo/drupal-core-strict will succeed and you'll know that you're done.

There is one major downside to this method though - by requiring specific versions of each dependency, the versions are effectively pinned in the composer.json file. So, the next time you update Drupal core (and webflo/drupal-core-strict), these specific version constraints will conflict with the updated webflo/drupal-core-strict. One solution would be to remove all of these dependencies from the "require" section of your composer.json file. 

Method two: rebuilding your codebase

If Method one is tedious and precise, then this method is more of a (less tedious) big hammer. Depending on the complexity of your codebase, this might be a better option for simpler projects. In short, make a copy of your composer.json (for reference), then use "composer remove" to remove dependencies on drupal/core, webflo/drupal-core-require-dev, and anything that depends on them. Then, use "composer require" to add back drupal/core and webflo/drupal-core-strict: 

composer require webflo/drupal-core-strict:^8.7.0 drupal/core:^8.7.0

Then, add back (composer require) all the dependencies you had to remove. Be sure to add back the same versions of each dependency (this includes Drupal profiles, modules, and themes!) to end up where you were when you started. Once everything is back, then you'll probably want to "relax" the version constraints of your dependencies in your composer.json by adding a "^". For example, if you re-add a contrib module using:

composer require drupal/pathauto:8.1.3

Then in the "require section" of your composer.json you'll have:

"drupal/pathauto": "8.1.3",

This will prevent drupal/pathauto from being updated. So, you'll want to change this to:

"drupal/pathauto": "^8.1.3",

Method three: delete and update

While researching this topic, I posted an issue in the webflow/drupal-core-require-dev queue and Greg Anderson was kind enough to offer another method:

[One] solution is to modify your composer.json file, attach the same version limit to drupal/core and drupal-core-strict (e.g. ^8.7.3) to limit what [composer update] needs to look at, and then [delete] both your composer.lock and your vendor directory and run "composer update".

One caveat about this method is that it will update everything. Any outstanding dependency updates (including Drupal profiles, modules, and themes) will be applied (unless you constrain them in your composer.json). Here's what Greg suggests:

  • Pin your contrib modules that are not updated to an exact version in composer.json.
  • Remove vendor and composer.lock, add webflo/drupal-core-strict [to your composer.json], and generate a new lock file [with "composer update"].
  • Remove the pins of your contrib modules in your composer.json by adding ^ [similar to the example in the previous method.]
  • Run composer update --lock

Method four: ???

Is there an easier way to do this? If so, I'd love to hear about it. Let me know in a comment below.

Which to use?

So which one should you use? If all your contrib projects are up-to-date, then I'd go with Method 3. If not, then I'd recommend Method 2 or 3 depending on which you're more comfortable with.

The future

Of course, in the future, much of this may be moot (for new projects, at least), as there is an active effort to bring an official version of DCDP to Drupal, including a new scaffolding dependency (committed to drupal/core on July 10, 2019!) and something akin to drupal-core-require-dev and drupal-core-strict. To find out more, check out the Composer Support in Core Initiative

Thanks to Greg Anderson, one of the Composer in Core Initiative coordinators, for his input and review of this article.

Comments

I think it's important to mention that using drupal-core-strict or drupal-core-require-dev can be dangerous. Those packages are not maintained by the Drupal association, but instead one person. That means when the Drupal security team releases an update to Drupal core (perhaps a security update), you cannot update your project until drupal-core-strict/drupal-core-require-dev have also updated. It seems like it may be automated that those projects get updated when a new version of Drupal is released, but what if that process breaks and the owner of those projects is away on vacation? Now you cannot update your project without first ripping out the dependency on drupal-core-strict/drupal-core-require-dev

Submitted by Brian (not verified) on Mon, 07/15/2019 - 10:29

webflo/drupal-core-strict and webflo/drupal-core-require-dev would not exist if drupal/core would be highest-lowest tested by the maintainers. Related issue: https://www.drupal.org/project/drupal/issues/2947893

drupal/drupal is a composer project that could have and has composer.lock committed which specifies the expected version from the dependencies.
drupal/core is a composer library that should not have and do not have composer.lock committed. Consequently the version constraints inside its composer.json specifies a version range of the compatible dependencies. Unfortunately these version ranges are outdated/not validated with highest-lowest testing and therefore issues this: https://www.drupal.org/project/drupal/issues/2965887 or many other could happen when drupal/core is installed.

These webflo packages are only providing a workaround by "imitating" composer project behavior on a composer library.

Submitted by Dezső Biczó (not verified) on Wed, 07/17/2019 - 02:16

Add new comment