Joel Clermont (00:01):
Welcome to No Compromises, a peek into the mind of two old web devs who have seen some things. This is Joel.

Aaron Saray (00:08):
And this is Aaron.

Joel Clermont (00:16):
We were recently starting up on a new, new to us, legacy project and it didn't have any sort of error tracking or comprehensive logging strategy or anything like that. So we were setting that up, right? And we used BugSnag and after, what? Like a day in production, there were so many errors being logged in production that it was actually throttling our BugSnag account. This was not an ideal scenario but thinking about it and considering our topic for today, I thought it was kind of a good way to kick off the discussion. Because even if you don't have this extreme situation, there's reasons why you want to pay attention to things like logs or error trackers and not let them get out of hand or let them get too noisy and just kind of have a method for dealing with it. So we'll talk about how we do that and kind of the benefits of taking that seriously.

Aaron Saray (01:21):
I think it's important that we talk about the reasons why. Because in the past, as a dev team leader, I've told my teams, "Clear out these logs, get us down to nothing except for fatal errors or whatever," and I never thought I needed to explain why. To me it was just 'obvious,' and I received a lot of pushback. Sometimes that's annoying, but other times it's just developers not understanding why they're being asked to do something that seems mundane. So I think it's good that we kind of talk about these reasons.

Joel Clermont (01:54):
Yeah. I mean, it could be viewed as busy work. Like, "Who cares?" right?

Aaron Saray (01:58):
Right.

Joel Clermont (01:58):
I'll kind of throw the first reason I think of and these are in no particular order. But one main reason is just the more noise you have in your logs or your Bug Tracker, the harder it is to actually know when something bad is happening. Like in this case, the default way... or I guess not the default, but the way we set up BugSnag is that it would even report things like notices and info level warnings, not just unhandled exceptions. But it was reporting quite a lot of things too.

Aaron Saray (02:32):
The reason we do that is obviously because, especially when it's a newer code base too, you want to know a little bit more about the code so those notices kind of tell you areas where the coding might be a little bit more lax or loose. In general, in my experience, those places have been where then more bugs can pop up. Where it's either programming bugs, errors or just things just happen to resolve in a way accidentally correct for the longest time, but no one actually knows why. So those notices and things like that also help give you the whole idea of what's going on in a project.

Joel Clermont (03:08):
In our case, we're extra sensitive to not breaking anything coming in new to a project. So just imagine there's 2000 errors in an eight-hour period normally. Well, now we deploy something and maybe we introduce a bug, well, now there's 2002. You're not going to notice that. That's where the noise comes into play. But even on a long running project that you've been in for a while, that noise to signal, or I guess signal to noise, ratio is the way I hear it expressed more often. It's good to have that as low as possible so that you can actually hear or detect when something is going on without having to pay really close attention to make sure nothing slips through that you would normally just ignore.

Aaron Saray (03:49):
Yeah. And I think when you start setting up yourself to see logs and think, "Oh, I need to ignore this because it's the standard log that we always see." It can be really easily, if you get like a hundred emails or something, than to just keep deleting those even when it says warning. I'll pick on Joel a little bit here. I've seen Joel see the word warning and be like, "Success, okay." I'm like, "No, it's a warning, dude." "Oh, whoops I didn't even see that." And I think we all do that.

Joel Clermont (04:19):
I don't know what you're talking about, I have never done that and my reading comprehension is perfect. Well, I mean, it'll get to a point too. I was on a different project with a team of developers and the guy who was supposed to be monitoring this, it was so noisy, he actually created a rule in his mail client to put them off in another folder. Days would go by sometimes before he'd look at it. I mean, it preserved his sanity when he looked at the inbox. But, man, that really didn't have the desired effect for why we're even reporting these things in the first case. That's kind of the first reason I think of, do you have a reason top of mind?

Aaron Saray (04:59):
Yeah, I think that kind of leads into a secondary reason. Which is the mental overhead of just seeing these errors and being able to process them and whatnot. If you're getting into a situation where you have to set up special rules to ignore certain types of errors and stuff, you're just making work for yourself or whatever, without understanding what's going on here. There's another thing too where... I know people will argue with me till the day they die. But I think that the difference between someone who has kind of inbox zero versus someone who has 10,000 messages, there's just a difference in how they feel about when they open their email too. When you open your log in system and you see, "Oh, it's been a day and there's 10,000 log entries for things that aren't necessary for me to log I guess." I think there's that weight on you where it gets you into almost a defeatist attitude too, where you're just like, "I don't even want to deal with this anymore."

Joel Clermont (05:56):
Yeah, it zaps your energy. I'm trying to remember which book this was, I think it was The Pragmatic Programmer. It used the analogy of a building that has five broken windows and after a while people are like, "Okay, a sixth one broke. Who cares?" It gets to the point where you're just so used to it, like, things get worse not better. And nobody wants to feel that way about their project.

Aaron Saray (06:20):
Yeah. I think another thing too to keep in mind with all these is that there is not only a mental load from looking at this but there is actually a performance load too. When you're writing out error messages or you're writing out logs that aren't necessary, it's not a lot, we shouldn't be like overly worried about it. But there is a difference between your app trying to have... having to send out 10,000 error messages in an hour versus not having to do that, right?

Joel Clermont (06:46):
Right.

Aaron Saray (06:48):
I think that there's something there, even if you're going to ignore our first two points. There's really kind of a performance thing here as well. I should point out now that we're not saying don't log things. That's not the point, it's not about not logging things. It's more about when there's errors or info�s on things that you don't necessarily need anymore. When we go full circle back to our project what we were talking about is, we scheduled in some time to start parsing down these errors into like, where are the errors that we can fix? And a lot of more notices. It was a lot of quick null coalesce operators we put in there and it's not great. And that's not how the project should be going forward but it is now accurate programming. It says, "If there's otherwise make it null," versus before just saying, "I don't know what this is so I'll default to no."
There's a little bit of a difference there and it's a little bit more accurate and it kind of explains them to the next programmer too when they see that. That it's possible in normal working conditions for this value not to be set. I think that's important to have said to them as when other programmers look at that as, "Hey, here is the area we know could be a little bit different or a little bit wonky. Sometimes these variables aren't there and that that's fine, everything still works." Then those help reduce our logs as well.

Joel Clermont (08:10):
Yeah. We've shared a few ideas as to why it's important to us and it's a balance. As Aaron said, we scheduled some time to do this. It wasn't like, "Okay, we're not doing any feature work until we fix these logs," But it's a balancing act. But in the long run your team will be happier and your project will be better for dealing with these things before they get out of hand.
Usually when I watch a program on TV it's recorded or it's a streaming show and so I don't have to watch commercials. But recently I was watching something live like an animal and I had to watch the commercials. And there was a commercial that had people, happy people, fit, energetic young people on a beach eating Taco Bell. And this was a Taco Bell commercial and I just had the thought, "Nobody eats Taco Bell on the beach." I know commercials are meant to kind of sell an idea but this was such a far-fetched idea. Nobody eats Taco Bell on the beach. First of all, do you agree with my statement that nobody eats Taco Bell on the beach?

Aaron Saray (09:25):
Yeah, I can't point to a specific time in my life where I've eaten Taco Bell and then been like, "That was a great decision." Or where I thought of Taco Bell because I was having a good day. Like, "Oh, it's such a good day. I should eat some Taco Bell." It's usually like, "Oh, I'm about ready to just pass out from hunger. What's something I can jam down my throat hole?" Or I've maybe had too many adult beverages and someone's driving and I'm going to yell across their lap at the drive-through and they're just going to hate me but I'm going to get a lot of fire sauces. Or just all of this stuff. No, I don't think I need Taco Bell at the beach and I don't think there's a lot of places that you actually see people eating Taco Bell too. It's a secret hidden sort of food.

Joel Clermont (10:17):
Dark private places.

Aaron Saray (10:19):
Yeah. Because I've seen people with like KFC buckets, random places, or the bags from McDonald's. But I don't think I've ever seen anyone in the daylight with Taco Bell bags.

Joel Clermont (10:31):
Yeah, this commercial is completely unrealistic then. I'm glad I wasn't alone.

Aaron Saray (10:38):
If you want to amp up your Laravel game, check out a new website we just launched masteringlaravel.io.

Joel Clermont (10:44):
Yeah, what he said.