Joel Clermont (00:00):
Welcome to No Compromises, a peek into the mind of two old web devs who have seen some things. This is Joel.

Aaron Saray (00:08):
And this is Aaron.

Joel Clermont (00:15):
Today I have the most exciting topic you might possibly imagine. Staging environments-

Aaron Saray (00:21):
Free money?

Joel Clermont (00:22):
No, staging environments. Aaron, that's even better than free money.

Joel Clermont (00:27):
Obviously, we'll start by kind of defining what I have in mind when I say staging environment. But the purpose or the audience for a staging environment, generally speaking, is the person you're doing the work for. So the client, the stakeholder. I kind of view it as a way to get things ready to show to them, sort of like a trial run before you move it into production and just kind of a safe environment to play around with the application and not worry about breaking things, corrupting data, et cetera. Would you agree with that definition?

Aaron Saray (01:06):
Yeah, I think that's accurate.

Joel Clermont (01:08):
Okay. I mean, you might even extend it to say, developers could use it too. In some cases though, you would have your own local dev environment you could play around with. Or, if you had some really big new version of something you might even have a separate dedicated server for that. But let's just focus kind on the primary use case here. The first thing you might wonder, and I've wrestled with this in the past, is where do you put the staging environment? We, generally when we deploy an app we don't deploy it with other apps. Like, if client A has an application we deploy it to their own server. Like, we don't share them between servers. I know some agency type models they'll pile as many sites onto a server as they can. But what about a staging site? Like, would you put that on the same server as production or what are your opinions on that?

Aaron Saray (01:59):
Well, I think it kind of depends on how you think you're going to use the staging environment. We kind of talked about, "This is a place to test out new features maybe," or whatnot. And I want to expand it, I would say it's a place to test out new features, it's a place to look at the code before it's going to go to production. But it's also a place to test existing functionality to make sure you know how it works without creating test accounts in production.

Joel Clermont (02:24):
Right, yup.

Aaron Saray (02:26):
And then the final thing I put on there is, it may be one of the environments that you use for load testing.

Joel Clermont (02:31):
Okay, sure.

Aaron Saray (02:32):
Because of that reason I would not put it on the same server as production because I just don't want either of them to affect each other. Staging is my last bastion of making sure this new code is going to work, or that I'm going to load test so there's all these different things and I don't want some broken code to be slowing down my production web servers. You know, things like that. And vice versa, I'm testing, staging and maybe I'm trying to see how fast it reacts to something. I only want it to have my traffic that I'm putting in it, not like if we're having a surge of Black Friday visitors or something on the production website and then we're like, "Why is our staging code so slow? We must have did something wrong." No it's not, there's just a lot of people using the site.

Joel Clermont (03:19):
Yeah, I think performance is a great reason especially, like you said, if you're doing load testing. Another reason I like to keep it separate is maybe the staging server is where you first trial, like, an upgrade of a PHP version or a new server OS. The PHP version, you could maybe do on the same server but I just like having it totally separate. I mean, yes it's an added cost, hosting costs, but they don't have to be long-lived either. You could make staging something you bring up as needed and maybe after you're done with the development effort, you bring it down. We typically leave them up because we're not running massive servers so it's not a... Even though it might be doubling the cost, the clients don't care about it. It's a drop in the bucket in terms of the overall project budget.

Aaron Saray (04:05):
Yeah. Our type of client usually would not freak out over having the staging environment running and I don't think I'd want to work with one like that anyway. Because, like I said, one of the big draws for me of having a staging environment is maybe even you're hiring a new person onto your team and you want to show them how the website works. They shouldn't be making test company in production, they should be doing that in the staging environment so that they can learn how the tool works.

Joel Clermont (04:31):
Yeah. The other thing I thought of too is, sometimes you get this weird overlap. Let's say you're running Redis on the web server, it's not a massive site. All of a sudden now are your staging jobs getting written to the same queue as your production job? Like, there's a potential there of things going wrong. Why not just spend the extra 20, 30 bucks a month to have a separate staging environment? Or whatever it costs. Another thing to talk about in a staging environment is, how is it different from production or is it different? Like, is it literally the same exact configuration? You might think that'd be great. I'll throw one thing out there that I definitely think should be different and I know we agree on this so this will be an easy one. Your staging server should have limits as to what sort of email it can send out, right?

Aaron Saray (05:23):
Mm-hmm (affirmation).

Joel Clermont (05:23):
Like, you're not sending accidentally live emails to users of your application. You know, maybe you whitelist your own email address or the developer email addresses or the domain of the company that you're working for so they can get emails. But you don't accidentally send them out to, like, 5,000 live users.

Aaron Saray (05:43):
Yeah, I agree with that. I think the way I handle the email is probably... I'm not sure if it works for every client and some of our larger ones just wouldn't, but I make sure no matter what it's always configured as SMTP and then locally I send to something like HELO, you know, local app by Beyond Code. When I'm in production, I'll use something like SendGrid to send SMTP that way. Then when I am staging, I will send to Mailtrap because I want people to see who's going to get the email, but I don't want it to go to that person. Part of the testing of email is, did the proper person receive it? And then they have to be able to verify that. So there are services out there where you can whitelist domains and stuff but I think then people end up always making the same account with Aaronnetcompany.com. Then does your mail server support the plus sign, Aaron+ second account? It probably won't.

Joel Clermont (06:41):
Sure.

Aaron Saray (06:41):
Sometimes they don't. So I think using something like Mailtrap in the middle is nice. On the flip side of course, when you have more people than just one or two testing that could be kind of tough.

Joel Clermont (06:54):
Yeah. It can even bring added cost, like the free version of Mailtrap only gives you so many users. But, yeah, it's a little more friction for the person using it. You know, you might even argue too, like it doesn't accurately... If you're looking at email design, seeing it in a browser versus people testing it with real mail clients but there's other tools for that too. But, yeah, that's a good point. One other thing too, bug reporting. We wire up bug tracking in production, would you wire that up in staging?

Aaron Saray (07:27):
Absolutely. Because the staging website is basically my production website. I mean it's hard to explain. When you go through your levels of development, and especially how you mature through how you use a project or you develop a project, you have to understand that staging really is the last step and basically it should be exactly configured like everything you would have in production. And if anything happens on staging, that's a huge bug. If I had a shared dev server, I wouldn't necessarily have it on there. I'd have these logs that go away after a day or two or whatever. And locally, I have it log into my local logs so I can take a look at that and whatnot with Xdebug, that too and everything. But, yeah, for your staging I would say it's a staging environment so if you're using something like BugSnag, you can say what stage this is and so those errors would come through properly.

Joel Clermont (08:27):
Yeah, especially if you're having users kick the tires there might be things thrown or things that happen and they'll even report it to you. That would be nice. But you don't have the context if you don't have something wired up to capture that. So, yeah, 100% agree I definitely have that wired up just as if it was production.

Aaron Saray (08:48):
I think one of the things that maybe we're glossing over a little bit and I want to double down on is, when users are using staging to test that out sometimes they're in a position where it's in their best interest to not necessarily report a bug either, because they might use staging-

Joel Clermont (09:05):
To see if you notice?

Aaron Saray (09:05):
No, because they might be a stakeholder in getting this project released.

Joel Clermont (09:09):
Oh, sure.

Aaron Saray (09:10):
So every bug they tell a developer is the slower it is, right?

Joel Clermont (09:13):
Yeah.

Aaron Saray (09:13):
So if I tell a developer about this, "Maybe that was just me, maybe I just messed something up," or whatever. If they don't tell a developer then the project will go out. I know for some people listening you're probably like, "I've never had a customer ignore bugs and please give me some of those customers." But I mean, it does happen depending on who your stakeholder is so knowing that ahead of time is cool. I'd also say, on the flip side one of the things I've ran into recently is, I had a customer testing and staging and they ran into an issue and I happened to be working at the same time and I saw the issue and I messaged them on our shared Slack. And I said, "Hey, I saw you just ran into an error. Just to let you know, I'll be fixing that. You can try it again in 10 minutes." Well, they were just blown away because they've never, ever, ever had a developer who knew about an error they experienced. I mean, that's a couple of quick wins I got with that client as well.

Joel Clermont (10:08):
Yeah. That's a good point, you can be really responsive to that. Maybe the last thing to touch on is, how do you secure the staging server? Like, you probably don't want random users stumbling across it. And maybe even if you're taking kind of a sanitized snapshot of production, you certainly don't want just an average user being able to log in and use it. One thing I've done and this is kind of where I'm leaning right now is just sort of a basic gate, like basic HTTP authorization gate. So that it can be up there, it can be accessible, you can give out a common set of credentials to a small set of users but it prevents Google from stumbling across it, it prevents a real user from finding it or maybe even somebody that's up to no good. Do you think that's enough, Aaron? Or what other techniques have you used to kind of protect that staging environment?

Aaron Saray (11:05):
I've actually ran into that a few times and that's been a problem for me too. Depending on the type of project you're making, I've had how do you make a staging API that has one form of auth that you would then put another one above?

Joel Clermont (11:19):
That's true.

Aaron Saray (11:20):
I don't know exactly how that's going to work. Also I've had customers that just... I mean, once it was saved in their browser that's great, but if I tell them to test on multiple browsers and then on their phone, they'll just get upset.

Joel Clermont (11:33):
It's a pain. It is a pain.

Aaron Saray (11:34):
Yeah. Another thing that I've done and I normally would say don't rely on security through obscurity. But I have come up with different phrases and things; the name, the staging server, URL, that isn't staging-

Joel Clermont (11:52):
Or test.

Aaron Saray (11:52):
Yeah, exactly. Staging, test, dev, any of those things. You know, it doesn't have that word in it and it has another phrase that maybe means something to the company internally or something like that. Or just, you know, fluffyballoon.com or whatever.

Joel Clermont (12:10):
Sure, yeah.

Aaron Saray (12:10):
That happens to be the staging environment. And I found that maybe that's not great but that friction has been a lot less than having people have to remember a password to type in, using the password on, you know... So I don't know if there's a really good answer for that. I've had people also to say, "Maybe you can restrict it by IP," and I've had people argue with me, Joel. That their home IP never changes. Joel.

Joel Clermont (12:39):
It only changed one time, Aaron.

Aaron Saray (12:41):
Then literally like days later it changes. I don't know if that's necessarily a great way of doing it either.

Joel Clermont (12:49):
Yeah, I agree. That one there's some tradeoffs, you kind of have to weigh it for yourself in the context of your own project and what really is at risk. Hopefully some things to think about. And maybe the thing you're thinking about right now is, "I don't have a staging environment," so we've given you a little head start on some things to consider when you go about setting that up.

Aaron Saray (13:15):
So we might have talked about this before but there's a lot of things that we maybe did back in the day or we thought were no longer going to be a thing but they sort of still are a thing but just differently. Maybe the flip version. I'll give you some examples and see if you can think of some.

Joel Clermont (13:31):
Please.

Aaron Saray (13:31):
We used to spend time finding old photographs, black and white photographs maybe colorizing them. But now if you think about it, especially in the future, people are going to be finding our photographs and removing filters from them. Like, just get it back to the normal colors or whatever. They can say, "What's all this filters?" Or, �Why does everyone from 2017 have a crown of stars on their head? I just don't understand."

Joel Clermont (14:02):
How can I remove these cat whiskers?

Aaron Saray (14:03):
Yeah, exactly. Or another thing that everyone kind of warned about is, I mean we email now we don't send letters so we're not going to need the post office. That's going to go away. But now more ever we need things like the post office because we're doing online shopping and we're sending more and more packages. Again we're like, "We're not going to need that."
In fact, we need that more. And then the other one I think about is people talk about maybe don't... we used to, "Oh, if there's a penny on the ground, pick up at the penny. And you're making money that's worthwhile." And I think people these days see a penny and they're like, "Wow, I'm not going to pick that up." But if you think about it mathematically, there's 3,600 seconds in an hour so that's really $36 an hour if you were to pick up a penny whenever you saw one. Because it takes about a second to pick one up. So really you can make more money now by picking up pennies than... Does this making any sense?

Joel Clermont (15:05):
Are you following behind somebody that's carrying a jug of pennies and keeps throwing them?

Aaron Saray (15:10):
(Inaudible 00:15:09) to try. I realize my math is saying that there's constantly a set of pennies to pick up. But the point is people think, "Oh, we don't need to do that." I'm like, "No, if you factor that out that's 36 bucks an hour. That's a lot of money still."

Joel Clermont (15:25):
What if every time you bent down there's a penny and a dime and you can only pick up one?

Aaron Saray (15:31):
Well, I'd pick up the penny because it's bigger.

Joel Clermont (15:34):
That's right, it is. Makes no sense. The only thing I thought of as you were saying this was in terms of a cycle as it comes to technology is when smartphones first came out, they loaded them up with apps and they clamored from our apps. And now we have so many apps I've seen this trend of people like, "I'm removing everything from my phone except the phone app and the music app." Like, "I don't want Slack and I don't want email and I don't want a browser." I'm like, "You kind of went back to the original iPhone." But I've seen that, it's like this digital minimalism trend. Like, "I don't want all the stuff on my phone with me all the time."

Aaron Saray (16:14):
Or, I mean same sort of idea with cellphones is, "Oh okay, I got a cellphone, let's make it smaller. Let's make it smaller."

Joel Clermont (16:20):
Too small.

Aaron Saray (16:20):
And they get really small and all of a sudden people are like, "Let's make them bigger." And pretty soon you see someone's walking around with a tablet as a phone. They're like it can't even fit in guys� pants pockets anymore.

Joel Clermont (16:34):
Nice.

Aaron Saray (16:37):
Do you want to be a hero to all your dev friends? Share the No Compromises podcast with them.

Joel Clermont (16:43):
Then send them to our podcast home on the web show.nocompromises.io.