Podcast audio transcript

[0:00] Music.

[0:05] Hey now and welcome back to the drupal Easy Podcast. This is season 15, episode number six.
This is the final episode of season 15.
In today's episode, we'll be talking with Cameron Egans, maintainer of the composer patch's plug in.
We'll be talking about the past, present and future of this beloved composer plug in that is used by literally thousands and thousands of drupal developers every single day before we get to the interview.
Let me briefly remind you about drupal career online.
Drupal's 12 week drupal beginner training program.
This class is offered two half days a week plus office hours for a few hours a week.
And this course is now in its 12th year and provides best practice focused training for aspiring drupal professionals.
Class begins August 28th.
You can learn more at drupal dot com slash DC O.

[1:05] Welcome to the drupal podcast. Cameron. How are you? Nice to meet you.
I'm doing great. Thanks for having me on.
I don't think we've met in the past, have we? Uh I don't, I don't think so.
Yeah, I don't think so. So, for those of you who haven't met Cameron either, so Cameron Egans, you are the maintainer of the composer patches, composer plug in that I, I mean, I don't, you know, I'm not gonna say it's all drupal developers use, but the vast majority of drupal developers that I interact with utilize the composer patches plug-in.
So first off, let me just say thank you for that.
And thanks for that. I am I, I'm, I'm really happy that the uh that the plug in has been so widely used.
Do you feel pressure from having a plug in that so many drupal sites depend on?
Oh, for sure, sometimes. Yeah, it's, it's gotten some use outside the drupal community too.
It, I don't know. It, it, it seems like every commit, it's like, you know, am I gonna break somebody's workflow?
Is this gonna be a really bad day for somebody?
I was gonna ask you about that in a moment. But since you brought it up, let me ask now, like, do you have, or what type of indications do you have about how much it's used outside of the drupal community?

[2:20] I don't have a good indication of exactly how much, or, or how widely used it is um outside the drupal community.
But it, I know that I've gotten issues from like the type 03 community and Magento and.

[2:37] Yeah. And I mean, just kind of using like the, the github code search.
See it gets pulled into the composer on all over the place.
So I believe that when we were, when you and I were exchanging emails about this, I believe you told me that you're not, you're not actually a drupal developer anymore, at least. Is that accurate?

[2:58] Yeah, I, I have uh sort of changed technology entirely.
I, I haven't done drupal in, I don't know, 89 months now. How about PHP?
No, completely uh completely hands off.
I switched over to, to Go and devops and it's, it's been, it's been kind of a nice change of pace.
Yeah, I think you may have just made everyone who depends on your plug and a little bit nervous with that statement, but we'll get to that in a moment.
So let's give your company a little bit of a plug. You're currently a staff devops engineer at Swirls Lab. Am I saying that?
Right? Swirl Ds? So what does Swirls Labs do?
Uh So Swirls Labs works on the Hadera Cryptocurrency.
We have a contract with the Hadera Foundation to provide software engineering and and DEV ops services among other things.
So I, I primarily work on the, on the devops side of the world.

[4:00] So help to make sure the, the Hadera network stays up and running and, and healthy and, you know, kind of coordinate the releases to the various networks every month.
All right. So I'm gonna switch, obviously switch back to composer patches.
What is it like for someone who is new to composer patches has never used?
It has no idea what we're talking about yet.
How would you describe composer patches to kind of an entry level drupal developer? Sure.
So I, I guess if you, if you kind of think about the the best case scenario workflow with composer.

[4:39] You are, you're working on your project and you need to pull in some version of some library and you, you can just do that after you've done that.
Maybe you, maybe you find a bug or you, you need a new feature out of that library.
So you make those changes and you contribute in back upstream.
And then nothing the maintainer is mi a or, or on vacation or, I don't know, it just doesn't maintain the project anymore or whatever.
But I mean, at, at the end of the day, you still got a deadline to make and, and you need this thing to work.
So composer patches is a mechanism to take those changes and incorporate them into your project without necessarily needing to have those changes upstream, in your, in your dependencies.
Or, you know, some, some people sort of rely on making a fork of the upstream dependency and keeping their changes there.
And that's, that's just kind of a pain in the neck.
Yeah. And I think it's important to note that what composure patches does is something that could be done manually.
But composer patches just kind of automates the process of you specify the patch that you want to apply to the project or the up upstream dependencies or the project dependency and composer patches.
Just kind of does it any time you run and correct me if I'm wrong, a composer require composer update or composer remove command. Do I have that right?
You have that right for version one, it's, it's a little bit more explicit in version two.

[6:06] So there, there isn't quite so much, you know, a a assuming how, how you're using composer there, there's explicit commands to apply a patch or, or rea all, all the stuff in your project.
So for someone who uses composer patches in its simplest form, they, you know, they save some patches to a patches directory in the project root.
They add the extras configuration to their composure at Jason and they run composer install the first time to get that patch applied for someone who just does that sets it up.
Can they just go about their business and have confidence that that patch will be applied as they expect or is there something new that they're gonna have to do in the future based in, in two do based on what you just said um in, in two do if, if you've gone through the process of, of adding the patch to your project every time you do a composer install the, the patch will be applied it.
It's more like if you do a composer require a composer update or whatever, it's, it's not going to change the Patrick or do anything fancy like you're, it's not gonna break your composer workflows is, is kind of the goal. OK.

[7:19] I do want to talk about one statistic that I saw and I believe it was on your website the Cw Egans dot net where you have a blog about composer patches or I don't know if you call it a blog but there's some updates that, that you put there.
But one of the numbers I saw was that it's currently getting over 40,000 installs every day, which I, I, that, that's a huge number.
Yeah, it's, uh, it's just a mind boggling amount of use.
Um, I don't really have a question but I did want to say it out loud just so folks can appreciate like how many folks, uh you know, how many people are using, uh this plug in? Yeah.
You know, I honestly I'm, I'm really glad that package just exposes those statistics.
It's, it's kind of a good motivator, you know, it's, it's like 44 million cumulative installs over the course of six years or something.
So, yeah, I don't know, like looking at it is, is just sort of motivating.
Like if, if I just stopped maintaining the plug-in, like it would not be a good situation.

[8:24] So how did, how did you get started with this? Was, was, were you just scratching an itch or did you take over another project or like how, how did this all start for you? Yeah.
Well, I mean, you, you mentioned earlier that it, it could very well be a manual step and it, it was at the time it was a very painful manual step and we, we tried writing a little bash to automate it, but, you know, people would forget to run it or, or whatever and it, it was just not a pleasant experience.
So at the time I was working for NBC Universal and they had sort of an internal drupal seven based distribution for all of their various brands.
The, the project that I was brought on for was to essentially port that distribution to drupal Eight.
And this was in the really early days of triple eight.
We, we started at like drupal eight alpha two or something.

[9:20] Yeah. Yeah. So we, you know, naturally depended on, on a lot of modules that.
You know, if, if they were ported to drupal eight, they were only done there that, that was only done in a, in a patch in the issue queue.
And so we'd like pull in the drupal seven version of the module and then apply this patch to port it to drupal eight.
It was uh like that old, a pleasant experience. Oh, yeah. Yeah.
You know, and, and since it was more or less a Greenfield project, we were really committed to making sure that, you know, we're, we're gonna use composer and we're gonna make sure there's good test coverage and stuff and you just, you know, sort of enforce good engineering practices from the beginning.
So, you know, we, we were pulling in all the dependencies with composer and, and we needed to be able to, to apply the patches.
So it was kind of just uh you know, let, let's see if this can even work sort of sort of task that was assigned.
And uh it turns out it can, yeah, I was gonna ask that, is this something that you just took on yourself or was this a task that was assigned to you, in a very open ended way saying, figure out a way to make this better or was it specifically go write a composer plug in to make this better?

[10:43] It was, if I recall correctly, it was, it was sort of a research ticket.
It was like, how, how can our workflow not suck?
It was, was sort of the gist of it. But yeah, so I, I spent a little bit of time on it and found out that you could write composer plugins and, um.
Yeah, yeah, kind of away from there. So there's a 2.0 release coming up.
What can we expect you mentioned a minute ago, but kind of give us more of a big picture.

[11:12] So two do for me is, is a couple of big things.
So one is that dependency patch resolution is going away the the the process of of discovering patches that are defined in the dependencies that are pulled into your project.
And I, I think we'll, we'll talk about that in a, in a little bit.
But the, the thing that sort of goes hand in hand with that is that composer patches itself is now extensible.
So if you, you know, if, if you really need dependency patch resolution or, or some other way of discovering patches for your project, you can write a little composer plug-in that extends composer patches and it'll just kind of like plug into the, the the main process.

[12:00] And then the, the third thing was sort of maintainability and really thorough testing and, and things like that from, from my perspective, the, the one dot X branch was just really fragile, and it, it was really uh, I don't know, anxiety inducing to, to make big changes to it.
Um Or, or even just little changes that could potentially have weird side effects.
It seems like you have made the decision that you want to continue to, own and made to own, not own own but maintain composure patches, but you want to set it up for the long haul, and from what I've read and, and what you've, what you've just told us number one, that means getting rid of dependency patch resolution, which you've written a blog post about it, and how it has been the, the the root cause of many of the issues in the issue queue for composure patches.

[13:01] But also, you know, adding the testing and making it extensible.
So that as you said, if someone needs to do something outside of the, you know, 80% of all use cases, they can write code that, you know, as you said, will hook into composure patches to allow them to do that.
So it kind of takes the onus for all of those edge cases off of you and allows other folks to, you know, do what they need to do, interoperate with composure patches while not.
Adding code to compose your patches that could potentially make it buggy or unusable or, or, or unstable for other users.
Yeah. Yeah, exactly.

[13:44] Just, just sort of making it a really strong foundation that does, you know, a couple of the, the really straightforward cases and then has extension points for all of the special stuff that people want to do.
Are there other uh maintainers or other active contributors more than just like one off here and there?
Do you have folks that, you know, actively work with you on issues?

[14:08] Yeah. So Dane Powell is uh another drupal person.
Um He uh he now has commit access to the repository and so he's, he's able to, to make changes.
So I, I just added him earlier this year, I think.
So that that's been really helpful just knowing there's, there's somebody else sort of keeping an eye on things.
There's also sort of a core group of, you know, users of the plug in, I guess that make good bug reports or good suggestions about how to, how to approach a problem.
And I'm, I'm really grateful for those people taking the time to be involved.
Yeah. It must have been a huge relief and probably a little bit of anxiety in adding, you know, that first co committer with you.
Yeah. I, I've had a couple of requests over the years, you know, like, can, can I be a coal maintainer and, and, uh, it, it's always been kind of one of those. Well, ok.
Yeah. Work on some issues or, or open some support requests or whatever and we'll, we'll go from there but, you know, Dane.
Actually did that, which was really nice. He's a, he's a good guy.
Very good. And when can we expect 2.02 0.0 0.0.
Um Do you have a well previously, my answer was when it's ready.

[15:28] But I, um I, I guess the thing that I was sort of waiting for is for people to test it and, and try to use it in their projects.
Let me ask the most important question of that if someone wants to test it other than installing the new version, well, if there again, if there are in the 80% use case where they, you know, they've added some patches to their composer dot Jason file.

[15:58] Do they have to change anything other than getting the new version?
Should it just work as is it should just work it?
Uh two dot Is backwards compatible with the same, you know, patch definitions that, that were in your composer dot json or in your, in your patches dot On.
If you're using an external file, you know, it, it has some, some extra stuff if you wanted to find things in a, in a fancy way, but you shouldn't have to change anything other than the version of composer patches that's pulled into your project.
Yeah. Ok. Well, that's a, that's, um, probably a very nice thing for people to hear.
And I think from the other side of it, if people are testing it, you know, I'm gonna go out on a limb and say you would love for them to give you feedback in the issue queue.
You know, even if there's no problems because I think if you, if you, if you get a lot of feedback saying I'm using composer 2.0 there's no issues.
I didn't have to change anything. This is my, you know, this is a summary of my setup.
That's valuable data as well. You just don't want to hear about issues, you want to hear about successes.
So you can kind of gauge, do we have 90% successes and 10% issues?
Ok. Well, that, that tells us something. Yeah. Yeah, exactly.
I suspect that.
A number of projects won't be able to upgrade because they rely on that dependency patch resolution functionality.

[17:16] But it's, it's sort of a chicken and egg problem, right?
Like whoever wants to write the plug-in that provides that functionality is probably not going to do so until composer patches 2.0 is out.
But people won't be able to test 2.0 until that plug-in exists.

[17:32] Yeah, I had, I had heard about that, that you could do that, that you could put, you know, patch metadata in a dependencies composer by Jason file.
But for some reason that it never felt, never felt right.
It never felt like I would. That's something I wanted to do because to me, it felt like something you could get lost through the, you know, slip through the cracks or, you know, I don't even know the answer to this and I'm not looking for an answer.
But like in my head, I'm like, well, what would happen if I'm applying a patch or this, you know, dependency A is applying a patch to the same dependency that my core or my root composer at Jason is I'm sure I'm gonna get a patch will not apply, er, or something like that.
So, for me it was always just like I'm aware of that functionality but I don't want to add that complexity to my life but from what you're saying and from what I've read in your, in your articles, it seems like, it's probably not a significant number but there is a, a nonzero number of people who do use that, that dependency on patch resolution.

[18:32] Yeah, I mean, I, I would say that it's, it's probably a more significant number than, than you'd think.
It, it's, it's surprisingly widely used.
And if you go look at the, if you go look at the discussions and they get every there, there's a, a thread in there about like we, we use this for this reason and, and all that, but like AAA really common use case is, there's some drupal con control module that depends on a, a quarter patch.
You know, there's like core functionality being developed in tandem with, with some con module.
So in order for this thing to even work at all.

[19:12] Core has to be patched, right? Which, you know, that's, that's fine.
That, that's, that's how stuff gets fixed in court too.
I, I'm gonna play a devil's advocate with 40,000 installs per day.
I would still argue or I, I put money on that.
The number of folks using dependency patch resolution is the vast minority of those 40,000 installs per day.
Yeah, I'm granted that 40,000 stalls per day is not 40,000 people per day.
That's just how many times that packages is getting hit with it or something like that.
And I'm not trying to talk you out of what you've done by any stretch.
I, I actually think it's a great idea.
I, I like simpler modules that have, you know, the functionality if someone else wants to extend it and I believe modules should be small and focused and rock solid.
And I think that's kind of what you're going for with, with 2.0 by removing this feature slash functionality.

[20:08] Yeah, it's, this is something that I've gone back and forth on a lot.
The, the people that rely on the dependency patch resolution functionality is like, it's definitely the, the minority of the, the overall users of the plug in.
But it's, it's a big enough number that I, I went back and forth on whether or not to remove this functionality for a long time.
Yeah. It seemed like it from the, um, from the blog post you wrote and then I went into some of the issues and was poking around in there and it, it does not seem like it was an easy decision. No, no.
And I, I, you know, I, I, I'm, I'm still kind of trying to let go of like, I, I feel bad about breaking people's workflow, but ultimately, like it's, it's not something that I can commit to maintaining and.

[20:56] It if it stays in, like, I don't know how much maintenance composer patches is going to get it like that.
It was hugely demotivating to, to just trip on the same issues over and over and over again.
Well, I, you know, I, I think it's a, I think it's a great decision.
I mean, there, there is not that you're looking for approval for me by any imagination.

[21:21] But when I read, when I read it, I'm like, OK, that seems like a very like, uh like a mature decision right from, from your standpoint.
I could imagine that you want to continue maintaining it.
You've gotten a lot of joy and personal satisfaction out of maintaining it except for this one thing that's kind of dragging it down, and if this one thing can be removed and have someone else pick up on that and you provide the necessary hook so that they can write something that can hook into composer patches. I think everybody wins.
Granted, there's gonna have to be a small group of people who, you know, maybe come together and write the plug-in for dependency patch resolution and, you know, maybe they're going to need your help here or there, or maybe they're gonna ask you to, you know, I don't know, for something, but that seems a lot more maintainable and sustainable than, than the path that sounds like you were on before.
Yeah, definitely. You know, and I, I guess on, on that topic before we move away, I, I mentioned this in the blog post but, you know, if they're, if one or, or many people are, are interested in actually writing that plug in, you know, definitely reach out.
I'm, I'm happy to spend some time, you know, sort of discussing some of the edge cases and, and weird things that, that need to be handled, help answer questions about why it's tricky.

[22:43] It, it, it can be something that seems simple on its face and then gets a lot more complex when you get into it.
All right. Well, anybody wants to challenge contact Cameron.

[22:54] So let's finish up and this is something I've learned, I would say, maybe the past year and a half and I still ii I don't think this is as widely known as it should be.
And this has to do with one of the best practices of composer patches.
And I think I actually learned this from a blog post or a session from Tim Lennon from the, from the D A who mentioned that you should never linked to a patch directly from drupal dot org.
You should always download that patch, save it on your local as a static, you know, patch file and allow composure patches to utilize that.
I'm gonna assume that's something that you completely agree with.
Are there pros and cons for doing it that way?
And are there any other best practices that you would recommend?
But you don't necessarily folks using all the time as, as a recommendation, downloading your patches and applying them locally is not.
It's not like bad advice. And like if you just do that with, with all of your patches, it, it's fine.
The biggest reason to do that is if you.

[24:08] So get lab and github, if you have AAA merger request or a pull request respectively, you can tack on, you know dot patch to the end of the URL and, and get, you know, a a patch file of all the changes in that merger request or pull request.
People have used that as you know, the, the URL to the patch and the composer dot JSON.
And then when somebody pushes new changes to that merger request, suddenly their patch is different.
So, you know, somebody malicious could like push changes to the pull request, right? And then, and then it's your project is just host.
So it, it certainly if you're using merger request or pull request, URL S in your project, like, don't do that. Download the patches, apply them locally.

[24:49] Patches that were uploaded directly to drupal dot org are a little bit different.
Um Once those are there, they're there and, and it, there's a little bit more friction to, you know, changing the content of those files.
It would have to be somebody that has administrative access to drupal dot org. And, right.
And I think Tim's point was, you know, at least partially because to reduce the server load on, on drupal dot org, um, it, it's been a while since I've worked on drupal dot org.
But I think most of those files are served through the CD N if that's the recommendation from the D A like do that, if, if, if that's, if that makes a big impact to uh, you know, the das hosting costs or, or CD N costs or whatever, like, yeah, absolutely. Download your patches.

[25:39] All of that's happened. Composer patches one dot X was very simplistic in its mechanism for, you know, if, if it's a remote patch, we're gonna download it and then just apply it to the project.
Composer patches two is a little bit smarter about things.
So every patch that you apply to your project local or not, when you apply it, it the, the shot 2 56 of that patch is going to be saved into a patches dot Lock file.
So if the, if the content of your like pull request patch file changes, you'll know because composer patch is is gonna like, you know, ring the alarm bells and complain really loudly about it.
So the security aspect of, of that particular recommendation is maybe a little less severe in, in. Yeah, that's a nice little feature.
Yeah. I mean, it's, it's something that I wanted to do in one dot X but it, it's just too, it's just too fragile.
Are there any other best practices for using composure patches that you would, uh, like to see more people implement?

[26:47] You know, I honestly, I, I don't know that.
I, I don't know that I'm really the best person to prescribe those best practices.
So a lot of the best practices are sort of emergent, right?
Like you discover them from actually working on real projects that have composer patches installed and like, I, I don't anymore.
Well, but it could also be that when you wrote the plug in, you intended it to be used one way, but now it's being used a different way.
I don't think that's the case, but I figured since I had you here, I might, I might as well ask.
Yeah. No, I mean, it's, I mean, people use it in, in mostly predictable ways, I guess the.

[27:30] The big thing that has come up multiple times is like you, if you're, if you're using uh I don't know, say drupal 10 and you wanna pull in a module where the composer do Json requires drupal nine.
Like you can't, you can't patch that module's composer dot Json, right?
Yeah, it's just the way that composer works doesn't let you do that.
That's, that's really the only, like, big thing to keep in mind I think there is, I believe, isn't there a, there's a mechanism that somehow gets around that I can't think of what it is off the top of my head.
Yeah, there was some blog post about it.

[28:09] I think I might have it linked in the documentation for composer patches.
I'll find it, I'll find it and I'll put it in the show notes.
So don't worry about that right now.
Oh, yeah. OK. It's in the non patch targets. Page James William wrote an article about it.
All right. I think we're, I think that's everything I wanted to ask you.
So, look, I really appreciate you taking the time to join me today and thank you very much for myself and I'm sure a heck of all, probably 40,000 users a day.
Well, maybe not users a day, but a heck of a lot of PHP developers out there who depend on your plug.
And so thank you for your continued support and maintenance of the, of the project and just know you've got a lot of people out here who uh really appreciate what you do.
Yeah, I appreciate it. Thanks for having me on the show. All right. Thanks, Cameron.

[29:00] Thanks for listening to the drupal Easy Podcast. Don't forget to check out all of our long form drupal training courses at drupal Easy.

[29:08] Music.