Podcast audio transcript

Transcript

[0:00] Music.

[0:08] Hey, now welcome back to the drupal podcast. This is season 14, episode four,

and today we'll be talking with Ted Bowman from Aqua or as everyone knows them T bone today we'll be talking about how to get started with automatic updates and,

Ted is actually one of the leads on that initiative.

So before we get talking with our old friend Ted, let me tell you a little bit about our long form training courses.

We have our new course professional module development that is debuting to the public starting on January 31. It's 90 hours over 15 weeks.

There's also a light version where we drop a few of the topics, it's a few weeks shorter and it's less money as well.

The full version begins January 31 and you can learn more at drupal dot com slash P. M.

D. R. 12 year of drupal career online, that's our beginner focused class.

That begins february 13th, that's 12 weeks long, two times a week plus office hours and we do a free one hour taste of drupal webinar actually have a few of those coming up,

um where you can learn all about the class and get all of your questions answered about the drupal career online course.

You can learn more as always at drupal dot com slash D C. O.

[1:34] Ted Bowman. Hey welcome back to the podcast, how are you? Thank you. It's great to be back, I'm doing good.

It's been, it's been a moment since uh since we've had you on.

So you hear your voice Likewise. Likewise.

So automatic updates. This is like what you've been working on for what, seven years now?

Something like that. I think I started when I was 13. Yeah, well, it's been a it's been a great effort.

No. Seriously, how long has this been, like your primary focus?

Um It's probably been my primary focus for a couple of years, but there hasn't been like a larger team around it the whole time, and it hasn't been the only thing I've been doing the whole time.

Yeah, so there's been other stuff in between there, so it definitely hasn't been two years, only thing I've been doing, but two years probably when it's been Most of the time, it's been my major thing, whether that's like 51% or 99%.

Alright, cool, so automatic updates, it's one of the drupal strategic initiatives.

Yes, for the community, Yes, so tell us what, no, explain it, you know, to me like I've never heard of it and I don't know what it is.

Okay, um so the automatic updates initiative, the first thing we're trying to roll out is just the ability to update core through the user interface of drupal and.

[2:59] So this would be a composer enabled way to update core so that, you know, if after the update it wouldn't there be no sign that particularly that you had used automatic updates.

We have also a goal to do it through a form and also through Cron updates, which is not currently enabled.

And then down the road we want to, you know, allow you to do that, allow you to do extensions also.

Alright, so how far along is it at this point?

So right now there's a stable contrived module which has a um non experimental module which is just automatic updates.

That updates drupal core through the form there, it does minor and patch releases, not major updates, and I can talk a little bit about how minor updates work later.

And then there's an experimental module that does extensions, so it would,

update modules and themes and the only reason that's not stable, it's more probably we want to, you know, fix some things in the model module but mostly it's not stable because.

[4:09] I think non core updates a little more iffy depending on the modules.

So we want to sort of get that right or get documentation right about like, you know, what can go wrong in just module, just in general when you update them, whether that's through the Ui or through composer.

So I was wondering about that, about the word extensions because I did see that I was playing around with in the user interface.

[4:36] But that's because it's modules and themes that can be Yeah.

Yeah. And basically, the only reason it's its own module is because the first version that gets into drupal core, the goal will be only to update core.

So instead of adding that ability directly to the main control module and then having to rip it out when we get into core, the main control module, it's supposed to work.

And you know, we'll see if it works this way in practice, but we have a script that converts it to the merger quest for core.

So the extensions one will be added later. There's the extensions ability, but probably when it's in court won't be it's own separate module.

All right, so let's say someone who's heard this and they're like, oh my gosh, ted bowman, this sounds fantastic.

I need to get my hands on it. I want to play with it other than, you know, spinning up a local development environment and adding that the automatic updates uh, stable contributed module right now. Are there any other ways that folks can test it?

[5:41] Yeah, you can test it via get pod. Um, I think when I tested it, um Shawl has a repository that's like.

[5:52] De de get pot or something like that and you can use that, that will start you off with the generic drupal site.

The only got you for that, I forget how it starts your site but any time you want to test auto updates you have to start with a version of drupal core, that is at least one patch release behind,

otherwise there's nothing Yeah, I mean I guess actually do composer, you could downgrade before you actually install drupal,

um but yeah I have a link that I've used for beta testing, that's basically a snapshot of this, you know get pod with automatic updates already there and also with drupal core, one at least one version behind.

[6:35] Get pod works well I've a long time ago I tried to simply test on me and I think there were write permissions issues,

because automatic updates by the nature of it, it needs to update your code base so it's not going to work on all hosting and,

yeah, last time I tried simply test dot me it was a while ago so I can't say that definitely doesn't work but it didn't last time I tried,

so there's a lot of words there, so it sounds like it's the preferred way of testing it right now at least is on the local environment.

Yeah, definitely works pretty well. Okay, so I want to talk about some of the requirements because this is I mean this module, if you just think about like how does one manually do a core update?

There's a few steps there, so what are the requirements for this module?

Um so you have to have a writable filesystem wherever you're planning to run it.

So if you're hosting provides a rideable file system, you can do it there um a lot like acquia,

Pantheon, I think they both have protected file systems so you can't run it in production there locally. You can you can run it.

Usually obviously you have a writable filesystem there, you need to have a composer on your system, the composer execute Herbal at a path um that we can access um.

[7:57] It will try to find composer, but then if it can't find it then you can set something in setting stop PHP to say explicitly.

This is where composer is or say yeah, if you have two composers and you want to use one then you can tell us that way, it does not need rush and that is pretty much it for.

Well there's a dependent module, correct? There is a sub module called package manager.

Yeah, so what's that all about? Yeah, so there are some dependencies but I guess in the sense that if you installed it via composer itself you would get all the dependencies.

[8:37] There is a PHP library called Composer Stager that we use and it basically runs the composer commands in a copy of your site or it it really is not drupal aware.

So it runs for any PHP application, it would create a copy of this of that application.

Run the composer commands, you can check to see if everything was good and then it copies it back over.

Um so that's a dependency um that we've been developing and, you know, it is a is a direct composer dependency that so you get that automatically.

There's a sub module that's actually in the contract project called Package Manager, and that is a sort of generic drupal module that.

Right now. It basically calls Composer Stager but you can think of it as a Composer package manager module.

It does installs and um it's also what project browser is using to install modules and themes.

[9:40] So you've piqued my interest a little bit with this Composer Stager.

Yeah. So you kind of glossed over one point that I want to ask you about.

So you said that it will actually run the composer commands and a copy of your project and then check to make sure everything's okay.

No. Well so Composer Stages doesn't do that. It will stage the operation and you could check for things like you could manually. Yeah.

So so how does that work? How do we check to make sure everything's okay?

[10:11] Automatic updates and package manager will do the checks for you.

Composer Stager just provides a directory.

You know, that could be checked with the changes for the composer operation.

So the kind of things that we check for an automatic updates is are there say you're going to update core But for some reason when you update core, it updates the control module, we don't allow that during core updates.

So that's one of the things we would check.

[10:42] So another thing it would check is whether you have staged database updates in like you just ran a core update and it may or may not have database updates that you need to run.

We don't allow once Cron updates are enabled.

We won't allow and update to court to happen if it requires database updates because you always want to be there when you run those database updates, Cron updates.

You know, I mean most useful for security updates so you make sure you get them if you're not around those for the most part usually do not have um database updates because they try to make the.

[11:23] You know, if it's if it's a security fix, the update will only be that security fix.

So they're not going to roll other features in their patch updates. They generally try not to have database updates.

So you know, they could they potentially could but they usually don't.

So that's one of the things we check which we check some things to just inform you of things through the UI.

So for the automatic updates extension module for example, if you update say web you asked to update web form and I don't know if Web form uses C tools but say it does.

You see tools and during your Web form update it,

has to update C tools because the new version, Web form,

needs a newer version of sea tools, it would tell you like, hey, we updated Web form to whatever version you asked for and additionally we updated see tools and these other other modules because you can't avoid that with other.

[12:22] With updating modules as they require often dependency to be updated too. So that's the kind of stuff we would check by looking at the composer's, the stage composer directory.

And then of course, you know, if you're doing it through the form, then you have a choice to say, oh, I didn't realize I wanted to update one module and now 50 modules are getting updated.

So you can either think, oh, well that's fine. I'm developing the site or this.

You know, I'm doing this locally all update, 50 modules or maybe you think like, oh, I should look into this more. I'll, you know, finish this update later because we're in the middle.

You know, this is a busy time and I can't run the risk of having 50 modules update right now.

[13:04] So let's talk about the, you know, installation of the module. I noticed a couple of things when I was installing automatic updates.

Um, number one and I think this has happened to me two or three times now that I've downloaded a fresh version of automatic updates,

and that is, it doesn't, it conflicts with the latest version of brush so that I have to normally it's not a huge deal.

But you know, when that happens, I normally will uninstall or do a composer remove of brush and then composer require both automatic updates and brush at the same time.

Composer will do its thing and sort out the dependencies.

[13:46] Is that something that is just something we have to live with for now or um, I'm not sure about the drugs version conflict.

I'm assuming you're not talking about the sibling problem, which we're going to get to later. Yeah. I'm not talking about siblings. Yeah.

Um, yeah, that may be just a, I have a feeling what might fix that is we're trying to remove a couple of dependencies that we have in the module.

So it could be that one of them is symphony find or something. It could be that one of our Yeah, Yeah.

So for now that's just something to deal with. But once we remove the, I mean, I can't say for sure that would be solved afterwards because composer stager of course would have its own, um, composer dependencies.

And I don't write, not knowing what just dependencies are. I wouldn't know if they conflict, but that is, I just don't want folks to, you know, folks who are listening to podcast who go to install automatic updates and immediately see a composer um, conflict. You know,

don't walk away. Yeah. Alright, so let's talk about this.

Uh so you mentioned a second ago, so let's talk about this some link challenge, we'll call it.

So there's a new core dependency called drupal core vendor hardening.

[15:04] So tell us about that. Tell us about this sibling conflict and what Yeah, what that's all about.

Drupal core vendor hardening. I'm not sure how new it is, but one of the things that it's meant to do is you.

[15:19] We have a bunch of vendor dependencies, obviously in composer dependencies that are in a vendor folder when you install drupal when you do a composer install the drupal project and,

some of that code maybe stuff that you definitely don't need in production, like let's say testing or docs or something.

And basically, I'm not sure the stuff exactly that core removes. But the idea is like if you don't need code in production, it's better not to have it.

Especially some stuff might be a security concern.

So it basically will like when you do the composer install that says, oh, you have let's say, I don't know. Symphony finder maybe.

And symphony finder has a testing directory which has PHP files that we don't need for production, let's take them out.

So that may that's probably not a real example, but it basically can remove directories from vendor folders.

We right now don't support having some links in composer managed product packages.

Um you can have some links in parts of your.

[16:25] Drupal package, but just not things that are managed by composer that we're going to update and that's, you know, depending on how much of a blocker that is for people using it.

We may or may not try to support some links. Some links are very difficult and the fact that when you do an update and you do it in a staging environment, if those sim links point anywhere besides within your environment in your.

[16:50] Currently composer managed product packages, if they,

if basically we don't want the staged composer operation to affect anything out of the stage directory and having assembling makes that really tricky to figure out.

So are there are there dependencies that commonly have siblings that people should be watching out for?

Just kind of an edge case. I did some checking and I didn't find, it's definitely not unheard of, but I, I did find some, we found the drug,

one obviously because you know, often people have brush but the siblings within drug sh,

are in the docks and I think it's maybe just the docks folder.

So we added that to the vendor hardening.

We added some configuration to tell drupal cores, vendor hardening, say, hey, just remove the dress docks folder.

Um, actually maybe we just added documentation for how to do that.

Basically you'll get like a message and here's, here's the help page. If you have help unable to go figure that out.

Um, it's not super common. That's with drupal core, I think that was.

[18:04] Uh, there are none in with the dependencies that are just for drupal core, there's one direct, there's at least one sibling to wrestling directly in brush and then there's another one in a drug dependency.

[18:19] So that's, it seems, it seems a little bit scary to me that in order for,

well what seems scary to me is that there's this, you know, dependency, the core vendor hardening dependency that will actually just remove them,

without really knowing what the repercussions of removing that sibling is by removing that sibling.

It doesn't do I have it wrong? Yeah.

Drupal the core of inner harding doesn't look for siblings. You just tell it, hey, remove this where they are.

Yeah. Well it's not even really for siblings, it's basically for things that aren't needed to run your site that are basically non essential things.

Right? But see that's yeah, that's I and that's the part that kind of like, you have to have knowledge of your, all of your dependencies,

well, maybe not all of them, but let's say that you run into this issue with a sim link in some dependency that's in your vendor directory.

Yes, I would be very, very careful about removing the director you find the symbol again.

[19:33] Right, Because and maybe you don't even know how that dependency got there, you know, off the top of your head then you have to do some research and say, okay, what what depends on, you know, what does, why do I have this dependency and if I get rid of that, what's going to break?

And so that's just kind of an extra hurdle there.

But I'm glad to hear that you say that in your experience you haven't seen it too much.

Yeah, and I would definitely, maybe we should make an issue for people to report, you know how common this is right,

because you know obviously people have really varied drupal projects have varied composer dependencies.

So I'm sure some people will run into packages with some links.

Yeah. And if you have one that you know is stopping you from using automatic updates, let us know through the issue queue.

Because like you said, I mean it maybe in some cases like the dress case, it's pretty obvious that you can remove the docks folder.

Right? But in other things it's not gonna be like pretty much if it's not docks or tests, you probably can't remove it.

[20:43] Yeah. And so I spun this up down a pretty fresh drupal nine site and I had rush installed.

So I saw that issue when I tried to I don't know what happened when I was installing automatic updates or maybe after I installed it, but I couldn't run on automatic updates.

I saw the message so I had to add the core vendor hardening dependency and then add that little blurb to the extra section of my composure at Jason.

It was all really straightforward and once it was done, everything was working. But it kind of, you know, piqued my interest as far as well. How do I know you know what I need, what else I need to put in here, But I think I follow what you're saying.

Yeah, if you don't get alerted that we found some links and you don't need to put anything else in there.

[21:30] Right Right. Right. All right. So let's move on past So let's assume that we we jump over all those somewhat small hurdles. I know we've been talking about it for a few minutes but really they're not none of them are that bad.

Once the modules installed, is there any configuration we have to worry about or do we just install it and hit the button and watch an update?

[21:51] Well it's not going to update until you ask it to, fortunately you have to hit the button.

Yeah, no, there's no configuration for it to start to work right now Cron updates aren't enabled,

you know, we can talk about that but once if you had Cron updates there would be a configuration but right now there is one hidden configuration that is that enables minor updates.

Um, so there is documentation I think in the read me or hook help to say like this is how you enable probably hook help.

How you enable crown up minor updates to core without that it will just show your current miners, you know, your latest patch release in your current minor,

we have a current issue that I just opened up,

to have a form for that and have the default be for minors to be on,

minor updates beyond but with the ability to turn it off basically we didn't want to enable minor updates out of the box because one they're more disruptive.

So we wanted to make sure patch updates worked well first.

[23:02] And two we wanted to make sure there was some documentation to sort of warn you that patch updates are minor updates are more disruptive and it may be better to test this update locally versus on production.

[23:17] So just describe more destructive new features.

Go into minor updates. So anytime you're adding a new feature, there's a potential for something to go wrong.

I mean they're not more disruptive on purpose, but I guess historically drupal core minor updates have been more disruptive than than patch releases we move from.

We don't remove BC stuff and let me ask you this way, is there under the hood?

Yeah, with an automatic updates, you know, go does the patch release versus a minor release.

Is there any difference in code under the hood? No.

So I'm saying miners updates aren't more disruptive because of automatic updates. Just if you did them via composer directly, they would be more disruptive. Yeah.

[24:06] Right because they potentially have more changes and Okay, I got it, I got it.

Alright, so I think the first time we talked about automatic updates and this is probably a question that you get a lot,

is, you know, for most drupal developers when they're updating the site, get is part of that process Yeah, with automatic updates, how does get fit in?

So get is still something you would have to do manually?

Um it would change the files in the same way that a regular composer update would, I mean I guess depending on the actual the same type of changes, so you would have to commit those changes and push them up um or whatever your workflow is.

I think basically, you know, we decided that people do sort of manage stuff with get differently enough that it wasn't really the role of the module to decide how that would work.

Somebody is free to make a sub module that we have an event system or not a sub module.

Again, trip project we have an event system, you know, we would be pretty easy to write custom code to do the commits, you know, when something is updated but we won't do that for you.

[25:16] So it's really going to be dependent on hosting as well because,

you know, originally this this initiative was so that,

you know, make it easier for people to apply updates obviously,

but if you have a local environment up and running, I would argue, I mean it sure clicking a button is easier than running a couple of composer commands, so I think it's kind of a wash for for those of us who work in local environments,

but for folks who, you know, don't have a local environment and and and host somewhere where the file system is writable.

Are they necessarily using get? I don't know, I have, it's been a long time since I've seen a drupal project that that isn't using GIT.

So it seems like this is this is, you know, this is a weird like net niche, niche, niche.

Well, what's the proper way to say that word? Niche?

Well, once you have, once we have Cron updates enabled, I could definitely see people, even if they use get?

[26:18] Being okay with an update running in production that they know is not gonna you know be committed to get as a stopgap to be like okay it's gonna run but then I'm gonna either do it locally or I'm going to pull the changes down.

I'm gonna commit those changes you know in the server you know the next day when I wake up when the you know the automatic updates happened in the middle of the night,

and are you know it may be okay to not have that inversion control temporarily so that security updates get applied.

[26:51] Yeah I'm really I'm really curious to see how how this gets used over the next you know once it's released over the next few years because it's one of those features it's it's a cool freaking feature.

[27:04] But now we have to figure out how how to put it into our workflows.

Yeah and I think also it's like this is what the N. V.

P. Was supposed to be you know not handling get I mean I guess if everybody says like it's not useful without get.

[27:20] Integration either somebody will make a control module that takes care of it for you or you know we'll have to add add it to the actual module.

[27:29] It would yeah definitely would complicate things. It's not really like environmental where you know it's not it doesn't yeah so like at aqua we have these cloud I.

D. S where people can use you know you can spin one up I think other hosting companies have them and they are writable environments so you could potentially spend one up, do the update,

it has I think I think it's bs code like integration there where you can just say well commit the changes I just made without particularly going to the command line.

So you could use something like that where an environment, a separate environment where you can spin it up that is writable and then move those changes over to production,

and then looking in the long term I mean the combination of this and project browser.

[28:18] Potentially has us looking at a future where you know drupal code base is managed by composer but people are aren't running composer commands directly.

Yeah for a lot of like the common cases yeah you know it'll install modules for you and then automatic updates potentially can keep you keep them updated.

Yeah so that's kind of the dream that that would be a nice dream.

Yeah and then we'll to figure out like how get how it works in there.

Is it just is it okay that people have to go to the command line for that or? Yeah we'll see well but there's a lot of people who use get you know not from the you get from that.

[28:59] Alright. So there's a stable release for automatic updates in control that people can play with, let's say someone heard this podcast and I'm like, oh my gosh, I want to test this, I want to play with this, I want to get involved.

What's the, what's the number one thing they should do first try it out and let us know if it doesn't work, if they want to get involved.

There is our slack channel and the slack channel I think has a topic, it's just a pound auto updates.

One word and I think pinned or the topic channel has our meetings which are every other Tuesday, are they a synchronous slack meetings or asynchronous slack meetings?

Yeah, so I usually try to be around when they start and we sort of report on what we're doing.

You know, people can come by either in the meetings or just generally posting, they can either obviously post issues to the issue queue if they find problems or you know, post the idea in slack and.

[29:59] Yeah, to get help for like if you're having problems installing it or running it, you know, either file an issue or you know, you can ping me or just in that channel or just posting that channel.

Okay, I'm trying to use auto updates, but I'm hitting this hurdle or whatever, you know, let us know and we'll try to help you test it.

We're really also really interested in people testing it on hosting, especially lower price hosting that, you know, potentially is a larger market for this,

and that's to me that seems almost count, not counterintuitive, I don't know what the right word is, but lower price hosting generally doesn't have composer available.

[30:40] So it seems like there's gonna be a sweet spot in there somewhere. I was surprised.

Yeah, I tried host gator recently and you know, I think I had the baby plan or something that had composer available on the command line really. I was surprised.

Well maybe things have changed, things have changed since I've last played with.

Okay, so ted what is the goal for this module as far as core is concerned, are we looking at trying to get it into core as experimental and 10.1 10.2 or maybe stable, like Yeah,

we'd love to get it in 10.2 as beta would be great if it's an alpha it's great that it's in there.

But alpha modules get taken out of the releases and by the nature of this module, like a layout builder. If layout builder was ripped out of a release and you could get cloned and play around with it.

That's useful. But this we don't update from we don't allow you to update from dev versions because that's dangerous.

So if it's not in a release it's not as useful by the nature of it being a composer update. Er So so yeah, our goal is to have a beta in 10 1 and I think that's the project.

[31:53] Browsers folks goal too, but I'm not sure on that.

So yeah, we'd love to get an N. V. P would be only patch releases for drupal core with Cron updates enabled,

Cron updates, the D A as the drupal Association is working on some signing infrastructure on their side on the servers that will connect to.

So you know, that will make the updates even more secure than then say, a regular composer update is right now.

So if that gets in in time then we would have Krone, it's not getting in unless they're unless the crown update, the part is done.

We have it turned off with a feature flag right now so that the goal is patch updates through the form and through Cron for 10 1 and that's, you know, we're trying to hit that.

You'll see it looks like, you know, on the project page for automatic updates, there's over 600 ported usages of it.

So that's always, you know, that's always Well, no, that is a lot of drupal seven sites actually, to be honest, there's a drupal seven version of this module, which is not composer aware.

[33:07] No kidding.

So when this was before you came along, before you got involved right? Yeah, it was funded by.

[33:16] Do you in? Maybe I want to say. That's funny. Oh yeah.

Go to the usage page. Oh yeah. We need more people to test out this module.

We need a lot of people to test about 100 reported usages for well over 100 and 50 for the eight dot x dash 1.8 dot x dash two.

[33:37] Yeah. So it's yeah it's going up but we need we need more people to do it for sure.

[33:46] It is working for some people I have had reports that it's working so people so folks are using it um in not in production but for production sites let's say. I'm not sure what the best way to word that is.

[34:00] I mean they could be using it in production if they're if they're edible hosting. Yeah.

Yeah. Some people are I've talked to some people who are I think we've we've covered everything right.

Have we talked about everything there is to talk about when it comes to automatic updates.

Um Just try it out. That's the key. Try it out because it's it's pretty easy and it's pretty freaking cool.

Yeah. Yeah. Give it a try and let us know how it went and you know, let us know if it worked well for you. Just post something in slack.

I'm using it. You know, we're not looking for thanks. We're just looking for like,

to know people are having a good experience as opposed to ted is definitely not asking you to send him find gourmet coffee in exchange for his work in automatic updates. He definitely does not want that.

Yeah. No. But if you're using it and you, you know, you're using it on a particular hosting. Let us Yeah.

Any any info we can have about, like people who are using it. We have. I think there's I would actually have to check this on the we had a form for a while on the page. Actually. It's not there in a short form. Yeah.

[35:13] We do have a short form linked from the page that says like, hey, how are you using this?

What version did you use to use it locally? Did you use it on a survey?

So people using it and filling out that form would be great, but don't, don't think ted whatever you do know, you know, I think it's best that people act entitled around Ted like where the heck is that? Why isn't it done yet?

Yeah, that's probably that's the way that Ted is motivated.

Yeah, So let us, yeah, let us know final question. Ted you ready?

Yeah, I am. It's, let's say mid december.

It's almost mid december. It is. Do you live in beautiful Ithaca new york.

I do, are the waterfalls frozen? Yeah, no, and we have very little snow, so it'll be better when there's snow and frozen waterfalls. Is there no ice?

Yeah, there's got to be ice starting. No, you like making me feel bad about having to check the waterfall out later.

Well that's great because you've been working really hard on automatic updates so much so that you don't go outside anymore.

Yeah, I know actually I did go by one of the falls the other day and I don't remember I was driving by, it was a huge one that you can see from the road, so I didn't see any advice.

So very good. Get back to work.

[36:32] Yeah, it was 40° today, so there was definitely no ice today. Oh, that's a bummer.

Alright. Yeah. Alright, well thanks for your time today. Thank you.

[36:41] Music.