Should developers have access to production?

Joel Clermont (00:00):
Welcome to No Compromises, a peek into the mind of two old web devs who have seen some things. This is Joel.

Aaron Saray (00:08):
And this is Aaron.

Joel Clermont (00:16):
Our backgrounds are a little different when it comes to our development history. For example, I've tended to work either alone or on small teams, and I know, Aaron, in your past lives you've worked in larger organizations at different levels within the teams. I know one of the areas that sometimes comes up is when we talk about access to production. And maybe I have a little more liberal view of who should have access and how that access should be doled out than you do and I think some of that just has to do with our backgrounds and the different types of projects and teams we've worked in. Why don't we talk about this today? I'm sure we're not the only ones that have talked about or have thought about what is the right level of access in production for a developer. But let's give it a shot here today.

Aaron Saray (01:08):
Well, I think one of the first questions I would like to ask is, what's the reason for the access? I think it boils down to two things, troubleshooting or doing things. I know that doing things is kind of... Like, what I mean executing-

Joel Clermont (01:25):
Could you be more specific?

Aaron Saray (01:27):
Nope. Executing something, like I'm going to go and tinker and do a task. Or, I need to alter this file or I need to change configurations or something like that. For me, those are two separate reasons to kind of consider or understand. So I want to focus on the easier one first, I think would be troubleshooting. Should devs have access to production for troubleshooting? I think you're right, it is a difference between the size of the team and what the roles are. But what has your experience been for troubleshooting on the server? Have you had to do that a lot?

Joel Clermont (02:04):
Sure, yeah. I mean, I guess going back to the earliest days of deploying PHP applications to production. Generally where I would start, something goes wrong or something isn't working quite right, is to look at the server locks. Generally it depends on how the server is set up and it might even require some privileged access if those logs exist in a folder that my specific log in user doesn't have access to. But that's probably the most common scenario for troubleshooting is, I would start there, I just need to look at the logs. But beyond that, the other place where sometimes troubleshooting is useful is the old expression with computers. Is like, "Did you just turn it off and turn it back on?" That's a common way to troubleshoot things. So maybe it's restarting the web server, it's restarting the database, it's restarting the PHP process. But those are kind of the two big things that I historically have wanted access to production in the scope of troubleshooting.

Aaron Saray (03:05):
Well, I think that definitely solidifies the difference between our experience and size of teams then because when you said logs, right away in my head I was screaming, "Why would you be keeping logs on that server? Why aren't you offloading them with some sort of CIS log process? Or using Loggly or any of those tools? Then you set up the access to those logs specifically with who needs to look at those." I think that probably happens a lot more with larger teams, but I don't think it should be restricted to the size of your team. Meaning that if you're even working on something by yourself, consider, "Do I really want to be on here looking at these logs and stuff like that?" Or, if it's a two person team, think about the junior developer. "If I give them permission to access the logs, what other things am I giving them permission to do?" and things like that. Then the other thing for restarting servers is, again, I think it really has to do with, "Are you the only person that would notice that?" Or, are there other people on the team that would be like, "Why did our alerting system suddenly say engine X has stopped and restarted? That shouldn't happen."

Joel Clermont (04:05):
Okay, yeah.

Aaron Saray (04:06):
Suddenly there's a message that goes off to the ops team or something like that. I'll argue with you a little bit, turning on and off again is like a Band-Aid or sticking your head in the sand. Because I can think of almost every single time I've went that route and came back-

Joel Clermont (04:27):
Sure, yeah. Oh, yeah it's not a permanent solution.

Aaron Saray (04:28):
Yeah. So solve the problem, stop trying to push it off as... You'll thank me later, trust me. Just solve it now.

Joel Clermont (04:35):
Another dimension probably besides just team size is the size of the infrastructure. Because as you were talking about offloading logs or aggregating them somewhere else, you would have to do that if you had more than one web server. Because where would you look? You're like, are you going to tail logs if you have 10 web servers against all of them? No, you would automatically be aggregating them somewhere. So, yeah, these are processes that probably happen on smaller projects, smaller teams. But I also think they're pretty common, there's a lot of devs out there in the Laravel world and the PHP world in general that are working on those same sorts of projects that I had a lot of experience on early on.

Aaron Saray (05:15):
I think the other thing that should be thought about is when you're doing troubleshooting, you also should have some air monitoring tool and we're huge fans of that. But like, Bugs [inaudible 00:05:27] or Flair. So a lot of the information you're going to need is already going to be in that tool and you don't need to be poking around on a server for something like that.

Joel Clermont (05:33):
Yeah, absolutely.

Aaron Saray (05:34):
I think shifting gears a little bit, the other topic was... Or is it to do something? Pulling actions on a server or whatever. And that one I'm a little bit more staunch on. Doesn't matter the size of your team, you need to have a history. And you can definitely say you ran into this, where things have been done on a server and you had no idea what they were when you took over the project even as a single developer. I'm much more against either a dev having access to server just to do things, or if they do it all has to be scripted and version controlled timestamped in such a way that there's a clear trail of what has happened. Whether that's PHP scripts or Bash scripts, something like Ansible. Any tool like that, no running things inside of Tinker just because I have to do this thing real quick. Write a one-time script, run that script so that happens. So I guess you could say that's access to the server. But run that script and then delete the script and its history would be in version control, versus, "What did this person do that one time inside a Tinker and the production server?"

Joel Clermont (06:43):
You know, Aaron, the first time you told me about that I was really kind of annoyed. I thought, "This is silly. What does it matter if I take these 10 lines of code that I could run in Tinker?" Like, what is the benefit of packaging this up as a command, committing it to version control, pushing it out as a deployment and then running that command? But it has saved me, so you've won me over. Even that history aspect of it, because there's a project I was just touching where we did a whole bunch of data migrations and things over two years ago, and I had to look at it again and I did look through Git history. Because we deleted some of those commands even after they weren't needed anymore. But looking in Git history, it was a nice way to replay some of those one-off 'tasks' and get that context back without having to write documentation or detailed notes about it. So you've won me over, you've convinced me. You win.

Aaron Saray (07:40):
Yeah, I think though, I don't want to oversimplify. When someone's listening and saying, "I'm a single person shop, how can I not have access to the server when I'm the only one that does something?" I think it's a simplification, talk about whether developer should have access to the server. It's more these other things that we've went into, is like are you using the right tools for what you're trying to accomplish? Are you making a trackable history? Are you following any of the rules required in your infrastructure? Then one more I'll put on there is, are you following any of the rules that are required with your certifications? If you're PCI compliant, you can't necessarily deploy the code if you've written it. Different levels have different things. So I think it really matters there. It's a more complex question than, should developers have access to the server? If you just ask me that, I'd probably say "Probably not." But I would say, "But they should have visibility and they should have the ability to do things to the server, just not necessarily access," which is fine grained when you think about it.

Joel Clermont (08:48):
Yeah. As with so many things in the development world, it's nuanced. There's all sorts of pros and cons and context you have to take into account.

Aaron Saray (09:02):
I've been thinking about how tough it must be.... and I know many people will disagree but how tough it must be to be a dog. Now, hear me out. First of all, sometimes you leave them and they really love you. They notice what you do when you leave, and it's put on your shoes. So they go and destroy them so you can't leave and then they get yelled at. They're like, "Come on, I just want you to stay here." Or, they don't understand mistakes that happen or stuff like that, so maybe it's dark and dogs probably can see decently and you can't. So they're just lying there and then you trip over them and you start yelling. They're like, "I'm just laying here and you tripped on me, now you're yelling at me." Or, when you go to a place and you don't notice but there's a no dog sign and then suddenly everyone's really mad that they're there. They can just see everyone just staring to them like, "Ah." What other reason do you think it might be difficult to be a dog?

Joel Clermont (10:02):
I will admit I have given almost no consideration to what it's like to be a dog. One interaction I've observed because we have an older dog, a small dog, a little Shih Tzu, and we just got a new cat.

Aaron Saray (10:17):
A cat?

Joel Clermont (10:18):
Yes.

Aaron Saray (10:19):
Oh, so you have two cats?

Joel Clermont (10:21):
Okay. I see what you're doing but no. The cat has tons of energy, the dog is a little bit older and really never had much energy to begin with, so they are at different energy levels. One thing the dog will do when he gets nervous is he'll wag his tail, which the cat thinks is like a really cool thing to jump at. Which makes the dog more nervous and it's this infinite cycle. I imagine that's probably difficult for a dog to think about how another animal is viewing its own actions and it's actually making something worse for itself that it's trying to avoid.

Aaron Saray (10:57):
Or, I guess the differences in sizes of dogs. Like, two dogs are just hanging out and one can jump up on the couch and cuddle, and then the other one, you're like, "Get off the couch, you drooling animal." Because it's so big and so large. And, yeah, I would think that'd be tough. I guess I think probably the most difficult thing in my mind of being a dog is, you ask for them to protect the house and then when someone comes up to the house and they start barking. You bark at them and then they're supposed to stop. I think that's super confusing. Like, "Protect the house except for if I'm going to bark then you don't bark," right?

Joel Clermont (11:39):
Yeah. Or they're supposed to identify, the doorbell is not dangerous but somebody breaking in the middle of the night is. To a dog, those are both threatening, so why wouldn't you bark at both?

Aaron Saray (11:50):
Yeah. Or, you yell at them because they hear doorbell on TV. Like, they don't get what's going on. It's probably, "Why did you install a surround sound if you didn't want the doorbell to make me run and attack the door?"

Joel Clermont (12:01):
Well, and to be fair, I've picked up my phone sometimes when I heard like the iMessage sound on a TV show, so I'm probably not one to criticize a dog.

Aaron Saray (12:09):
So reasons why it's tough to be Joel.

Joel Clermont (12:12):
Exactly, on our next episode.

Aaron Saray (12:16):
Over the last couple of years, we've gathered together tons of tips and tricks and whittled those down to 17.

Joel Clermont (12:21):
We've assembled them together into a free eBook, which you can download by visiting our website nocompromises.io/tips.

No Compromises, LLC