Joel Clermont (00:00):
Welcome to No Compromises, a peek into the mind of two old web devs who have seen some things. This is Joel.

Aaron Saray (00:08):
And this is Aaron.

Joel Clermont (00:16):
Every now and then there'll be these online code puzzles and developers get, myself included, kind of excited about it. Like, "Ooh, this is fun, this is challenging." And generally, I'll do one of those and then Aaron will be like, "Joel, this other work isn't done? Why are you doing this?" And I'm like, "Shut up, Aaron."

Aaron Saray (00:34):
Yeah. I'll be like, "Why don't you do the task that we're three weeks behind on?"

Joel Clermont (00:38):
Yeah. Not three weeks behind, like a day behind.

Aaron Saray (00:41):
Four weeks.

Joel Clermont (00:42):
Anyways, the reason I bring that up is because yesterday I was working a bug that was really thorny and I thought, "Hey, this is actually like one of those code puzzles," and just kind of reframing it in my head that way made it more fun. So I thought for today's topic, we could kind of go through that bug and some of the things I did to try to understand it and ultimately solve it. And, as always, maybe try to come away with some takeaways, some principles. We all have bugs, right? Like, how do we triage them? How do we reproduce them? Where do we reproduce it? You know, how do we get to the root of the matter?

Aaron Saray (01:22):
So I think the first thing, we've mentioned before, is don't get too stuck or don't take too long on a bug.

Joel Clermont (01:28):
Oh, sure.

Aaron Saray (01:28):
Like, try to prioritize it and everything. I'll start out with the embarrassing part about this story. Which was, this was actually my bug and I ran into this issue and I went down kind of the troubleshooting rabbit hole and I said, "Well, I don't really need to fix this until later and I know Joel has some time coming up for this project." And we talked about it but I decided that I could continue the project because I'm the main developer on it and I have familiarity with all of it. And, you know, we don't need familiarity with the project per se to solve this bug.

Joel Clermont (02:03):
Right.

Aaron Saray (02:03):
It was easier for me to say, "Well, I've spent a couple hours, I'm going to give it to Joel." I think the first core principle I'm trying to make here is deciding, first of all, even when you have a bug, who is the best person to solve it or what makes the most sense? It's not saying, if you generate a bug, you should definitely not fix it, but it's after a while. Like, if you tried to make some progress and you haven't, what's the next step? Well, am I the best person to solve this?

Joel Clermont (02:31):
Yeah. And to cut you some slack, Aaron, this is an app that's not in production yet, right? These aren't users encountering these bugs and we're triaging it. But it's something we encountered in the course of development. I suppose I should say what it is, we've been kind of teasing this whole time. But the issue had to do with uploading a CSV file and using the Laravel file validation rules to say, "Hey, is this a CSV file?" You know, the mime_types rule. You had it working and then all of a sudden it wasn't working. And no matter what CSV file you uploaded, it would just say, "This is not a CSV file." I forget the specific text of the error, but it was that mime_types error failing. We're like, "What is happening? Why is this not working?" I think when you were working on it you had even thought, "Well, maybe this is just something about our Docker environment." You know, we use Alpine Linux as the main image and it's pretty stripped down. And maybe it's missing the Magic... What is it called? MIME Magic or?

Aaron Saray (03:31):
Yeah, the Magic MIMEs or whatever it is.

Joel Clermont (03:33):
Yeah. So that's kind of why we punted. And then it wasn't until later when we pushed it up to a staging environment and it happened again. It's like, "Oh no, this is actually a real bug we have to figure out." So the first thing I like to do is of course, to reproduce the issue. This is kind of where I hit my first head scratcher, because on the page where you can upload the file we provide a file template. Like, click this link, download a CSV file, put your data into it and re-upload it. Just to make it easy. If I downloaded the file and immediately re-uploaded it, it worked. It's like, "Great, thank you. All done." If I opened that file in, I'm on a Mac and so by default I think it opened in the Numbers app that you get from Apple. And if I re-saved it as a CSV, I made no changes to the file, and I uploaded that, it didn't work. And I'm like, "What is happening here? Come on." Then I thought, "Well, you know, maybe Numbers is weird. I'll use Excel." I have an Office 365 subscription or whatever, so I have Excel on my machine. I did the same exercise. I opened the downloaded template, I re-saved it from Excel to a CSV, no changes. Tried to upload it to our app, it's like, "No, that's not a CSV." So I am kind of pulling my hair out at that point, like, "What's going on?"

Aaron Saray (05:01):
So before you continue on here, I know we have some listeners that aren't programmers.

Joel Clermont (05:05):
Okay.

Aaron Saray (05:05):
If you're not a programmer and you work with programmers, yeah, there's things we can do to not work, but also there's other things that we're like, "No, we are trying to figure out how to do an upload and it took a whole day." This is an example of why. So far there's no clear reason why this is happening.

Joel Clermont (05:25):
No, exactly. It's like the file is the same. Then I got a little smart, I'm like, you know what, I can take... Because this is actually a perfect situation. You have one thing that works and you have another thing that doesn't work. I don't know what the difference is between them, but having a good test case and a bad test case is hugely helpful to debug an issue like this. It's much worse to like, "It just doesn't work at all and I can't figure out why."

Aaron Saray (05:52):
So really that's the second... the thing that you want to cover here is making sure that you can kind of duplicate the error, but also it really helps if you can find a scenario where it's not a problem as well.

Joel Clermont (06:05):
Absolutely. Yeah, for sure. Having that in hand, I have a tool I like to use for comparing files called Beyond Compare. You know, I like paying for stuff, Aaron? And it's a paid program, so.

Aaron Saray (06:17):
Yeah.

Aaron Saray (06:17):
But it has a mode. When I did the diff, it actually smart enough to know, "Oh, these are CSV files," so it kind of compares them in a tabular format. And it's like, "Nope, they're the same." I'm like, "No, they're not." Because I could see on disk they were one byte difference file size. So I overrode its helpful auto file detection. I said, "No, just give me like a hex dump and compare it." Like, I want to see the bytes by bytes. And, well, sure enough, when I did that I could see the file on the left that worked used Linux line endings. You know, \n, or backslash n. And the file on the right was using Windows for some reason. You know, \r\n combined. I said, "Oh, there we go. That's the difference." So I edited the file manually to \n, uploaded it. Still didn't work.

Aaron Saray (07:11):
Still didn't work.

Joel Clermont (07:12):
Still didn't work. I'm like, "Come on." And I think you and I were even pairing on this. And maybe you pointed out another difference is the file on the left, the good file, had a trailing line ending. So the CSV file was a header row and then two rows of data. But at the end of the second row of data, the last line in the file, it also had a line ending.

Aaron Saray (07:38):
The reason I noticed that is because right now we're using a package from PHP League called... you know, their CSV. But before that, back in the day, when you had to read this in with fgetcsv and stuff like that, the issue we'd run into a lot of times is there'd be that trailing line break. And so the last result you get in was just an empty row. So I was familiar with a lot of CSVs have a trailing line break and so that's kind of what made me notice. I'm like, "Well, yours doesn't."

Joel Clermont (08:07):
Yeah, so that was the difference. The file on the right, the file not working, didn't have one. Then what did I do? I added a trailing line break, uploaded it, and it worked. And I'm like, "Oh, well, clearly this somehow needs a trailing line break in order to detect the file type." Spoiler, that was not the was not the solution. But I felt like we were moving kind of closer to it. Then I'm thinking, "Well, how do we fix this?" I dug into the validation rule and I saw it was using the underlying Symfony UploadedFile, which was using the symfony MIME composer package, which when I dug all the way into that was using PHP's native fileinfo extension. Seeing that whole traversal of things, my instinct was then, "All right. Well, I'm going to take it out of this app." I think we've had an episode about this. And I went to PHP eval and I did the smallest reproducible test case I could, not using Laravel, not using the symfony MIME package, not using an uploaded CSV. But raw text, write it through a temp file and then have this fileinfo extension tell me what is it. And sure enough, I was able to get one to come back text CSV, one to come back as text plain. And by playing around with it, what I realized was happening is the trailing line ending was kind of a red herring. But also it was sort of the solution because it needs to have three lines of data in order to detect it as CSV. The trailing line, even though the next line- Yeah?

Aaron Saray (09:48):
What needs to have three lines?

Joel Clermont (09:50):
The CSV file we're uploading.

Aaron Saray (09:52):
No. I mean, what piece of code requires that?

Joel Clermont (09:57):
Okay, I didn't dig completely into it. I tried looking at the PHP source, and it was calling out to another C library called libmagic. And I'm like, "Yeah, I'm not going to figure that out." But that's ultimately what it's relying on. There's that Magic file database, which comes from libmagic. I was able to see PHP, like patches it a little bit to make it work inside of PHP. But ultimately, the definitions for, what is a jpeg? What is a pdf? What is a CSV? All come from that other libmagic library.

Aaron Saray (10:31):
Okay. Yeah, that's what I was trying to ask. Is like, was it the Laravel code? Was it the symfony packages? All those?

Joel Clermont (10:36):
No.

Aaron Saray (10:36):
Or was it the finfo module in PHP? Well, yeah, I mean, once you get to finfo, then you really can't do anything else.

Joel Clermont (10:44):
Yeah, exactly. I'm not opening a patch somewhere. And honestly, it's probably not a bug, it's just maybe unexpected behavior on my part. And in reality, you'd hardly ever run into this, right? Who's going to upload a bulk CSV file that has two rows? You can, but in reality this probably never would've hit a user. It was just the sample file we were using only had two rows and these other tools don't do a trailing line, so it didn't hit that three line threshold. And it's like, "Oh, it must be just a plain text file." Anyways, after having done that, I felt like a programming master but also I felt really dumb at the same time.

Aaron Saray (11:26):
Yeah, that's what programming should do to you.

Joel Clermont (11:29):
Yes. The last takeaway I have from this, maybe a principle to share, was when I was going through this at a certain point it was like I just wanted to know the answer. Like, at a few points in the process, we'd come up with workarounds. We could just do this, we could do that but I really wanted to know. So, I timeboxed it though. I said, "You know what, if this isn't ready, if I don't have an answer by the end of the hour, I'm just going to use this workaround we talked about." So that's another thing too, is like don't take it personal. Maybe bookmark it and come back into it later when the task is done. But don't get so obsessed with finding something weird. If you have a valid workaround sometimes that's an okay alternative.

Aaron Saray (12:12):
Yeah. Because the client wasn't going to pay us to continue their research into this, they just want the product done. So at some point we have to balance that with the amount of time we're spending. So, good point.

Aaron Saray (12:26):
Do you ever notice how there's an inverse relationship between how well you know someone and how much you clean your house before they come over?

Joel Clermont (12:35):
First of all, that is a very programmer way to say it. But I haven't thought about that, but I'm starting to now, and I think I see what you're saying.

Aaron Saray (12:44):
Yeah. Like the better you know someone you're like, "Oh yeah, just come over or whatever." But if it's a new friend or a brand new whatever, you tend to clean up stuff. But after you've known a person for a while, you're like, "Errr, they'll just accept me and all my garbage on the table."

Joel Clermont (12:57):
Yeah, Aaron, it's sort of like when we do video calls. Our first ones probably were very pristine backgrounds and now you can see like a heaping pile of Amazon boxes behind me and I just don't care anymore.

Aaron Saray (13:14):
Have you ever ran into this problem or do you even validate your MIME types when you upload your files?

Joel Clermont (13:20):
We have a whole book on the topic of validation, the different rules and how to use them. Head to masteringlaravel.io and click on validation book.