Are you testing your app or just the framework?

Welcome to No Compromises. A peek into the mind of two old web devs who have seen some things. This is Joel.

And this is Aaron.

I talk a lot about testing. It's kind of one of my pet topics. And, as with anything, like we do, I want to make sure the thing I'm doing has value.

Like, it's giving me what I think it's giving me, right? So, with testing, one of the things I've heard some pushback on, or some... no, debate's too strong a word. But like some questioning is, how do you know if you're writing a test that's actually testing your code, your logic?

Versus, maybe you're writing all these unnecessary tests that are testing what Livewire or Laravel, you know, the framework, is doing for you? Like, it should have its own test, right? So, have you heard this? Let me just throw that out there. Have you heard this kind of debate or discussion?

Yeah. There's a lot of discussion on where the boundary of where tests are. Like, am I testing just my logic? If I'm testing, let's just say I have a service class. Should I just test a service class?

Because I can assume that a controller I wrote works because it's Laravel? Well, maybe that's not the right place, maybe you have to test end-to-end. And then there's other things where people have tested things like the rules or custom messages.

Not running validation, but just testing the values. Like, is that testing that's technically testing your code, but if you're running it through the framework, isn't it testing that the framework interpreted that stuff, right? And should they have already been tested? So, yeah, there's a lot of questions on where the boundaries lie.

Yeah. And some of the ones you mentioned are great examples. Because I think they are a little fuzzy. Like, there are some of those tests that we write where I could take an objective step back and be like, "Yeah. I mean, this is testing framework behavior."

So, here's how I think about it. I want to share kind of my metric on, am I testing the framework? And what I mean by that is like, is this test valuable?

Okay.

Like, should it exist in my test suite? Because that's ultimately what we're trying to answer. It's not just a-

And I want to make sure we just define the word valuable because we're making an assumption that the Laravel code is fully tested, which we know it is.

Yes.

But when you say valuable, there's a question of like, "Well, if I'm testing the framework, the whole product is mine regardless if it has a framework inside of it or if it's my code."

Sure.

"So, should I be running the whole frameworks test suite all the time too?"

Oh, I definitely don't do that, but okay. Well, I'll let you make a case for that later if you want. So, here's my metric, and it's pretty simple. Which is, for my test, you know the test passes, everything is good.

Can I go into my application, right? So, outside the vendor folder in an app, config, whatever. Database, whatever. Can I change something that makes my test fail? And there's some limits to that.

Like I just said, database, like I can go delete all the migrations, and my test is going to fail. But, you know what I'm saying? Like, even the routes. I can delete a route, or I can change how it's bound, or I can remove a parameter from something. Like, can I do something that makes it fail?

And to me, if I can't, it's a pretty clear indication I am testing the framework, and it makes me question, does this test need to exist in my application?

That is a fuzzy concept, too. Because, again, the boundaries are confusing. So, let's imagine that you have a relationship on an Eloquent model and is maybe defined a custom key.

I get it that you could change that key in there, and then the relationship doesn't work, but then where do you test that even, too? Should you be testing each one of your relationships to make sure you never change the key?

Or, is it in the scope of an end-to-end test that you would discover that executing that relationship didn't work as you expected? And, let's just say that you wrote a relationship that returns... I can't remember the specifics, so this is horrible.

But like has many or has one sort of works with the other one, so you could have returned the wrong type or something. So, it's like that's a configuration, but it's also the code, and what is it? So, yeah, it's hard to define.

Well, and... I mean, this is maybe a secondary topic, but it does influence it. It does depend somewhat on the style of tests you write. Like, we tend to write more feature tests than unit tests, right? So, we're not unit testing relationship methods on Eloquent models.

At least I haven't seen you doing that, Aaron. Like, for the most part, maybe there is some weird case where we've done it. But, yeah, to that point.

But I think if I'm writing a feature test, I'm calling a controller that's writing attached records to some model, and then I make database assertions, I'm reading those records back. Like, if that works, then obviously the relationship had to work for those things to get there, right?

Yeah. I think I might interrupt you here and say I... Because you mentioned unit testing and all that kind of stuff. I have a little bit of a different assessment, but I think it sort of goes hand in hand with yours.

I look at my unit tests, my integration, and my feature tests. So, we have a unit which is just a bare piece of code, integration which is integrating with some other thing, like maybe database or another API or something, whatever. And feature end-to-end user input, user expected output, and byproducts.

I look at my unit test and say, "Is something I'm testing code I've written? Then it should be unit test." Like, code I've literally written and I know it's a code I've written because it's small enough to be in a unit test. So, that one's pretty easy.

If I wrote the code, it would be or should be unit tested. Then the next layer up, when I start to do that boundary, I start saying like, "Okay. I can integrate with the framework."

I'm not necessarily testing the framework, but all the code I've written may use framework code, but I'm still only testing the results of how I've configured or used, or implemented some of the framework code.

I'm not testing the actual interiors of that. So, it's kind of like what you said, but it's a little bit different. Like you say, "Can I delete something?" I'm looking at it more so from like, "Did I write that code?"

Okay.

If I wrote that code, I should be responsible for testing it.

When you say wrote that code, like it's a class that you came up with all on your own. It's not like a mailable?

Yeah.

Right, yeah.

Yeah.

Okay. Yeah, I agree with that. I do agree, like the places we've written the most percentage of unit tests are things like that. A service class or something that's like all your own logic.

It's not like building a piece of framework infrastructure or a scaffolding that sits on top of the framework directly.

Like you said, it depends on the type of tests you're writing, but it also depends on the type of project it is.

Yes.

So, for example, if we're working a lot with CRUD projects, the heavy feature set sort of code makes sense to those tests. But if we're working on something that's maybe working with data or ETLs and stuff.

We might not have that mentality, it might be a different level. So, I'm really nervous to be like, "Well, I can delete a route," and whatever, because that has nothing to do with routes, really. I know you were oversimplifying it.

Yes.

But that I kind of look from the other direction going up.

Yeah. No, it's good we balance each other out here. That's great. I do want to go back to one example you shared because I think this could be a little controversial. But I think you and I agree on it because we work together, we do this.

But it's testing validation logic. You know, because I think a lot of people look at that and like, well, you're, you're not... like an exists rule or something. Or, maybe even more basic, like a max:255.

Like, you're not writing the logic for that, but we will write a test that kind of makes that validation error trigger just to prove to ourself it's wired up correctly. So, what is your defense of that?

Like, why that's important, and why it's not just testing the framework? Or, do you want me to share my opinion on it?

I think in that particular case, it's those rules change functionally depending on how you input data into them.

Yes.

So, the test isn't testing the validation rule, it's testing the chain of validation rules and its placement. And therefore, that is code I wrote because I'm chaining together things that Laravel has.

Just so happens that they give me a nice interface where it looks like text, but it's actually classes, and I'm chaining them together.

That's a great point. Yeah, even those string-based rules, even if you're one of those people that jams them all into a pipe-delimited string, which we don't do, it is code. There's classes underneath that. And, the other argument I go to as well is like, sure.

Like, maybe writing required, or max:255, or whatever, feels kind of repetitive and boilerplate, but honestly, that's easy stuff to write.

Like, it doesn't take any time. It's all those other things, like required if, unless, like all of these things, where maybe even two fields are interacting or you have a custom rule. So, to me it's just like, "Well, just test your validation."

Yeah, there's some simple things to test, just like writing a assert okay in a feature test response is simple to do, and it's kind of boilerplate and in every test. Like, you may still do it because it still has some value.

Mm-hmm (affirmative), I agree. I think the important thing to know, too, is that if we're talking about what Laravel was testing it probably... I guess, I don't know.

But it probably tests those rules in isolation, saying these are all the incoming values, here's what that would result in. But I think about something like max:255, and the way we define the rule max:255 in our application actually depends on how it works.

So, because of that, that's why we write these tests. So, for example, if you put string in front of a max:255, it's going to be a length of 255. But if you put file, it's going to be 255 bytes or kilobytes, or whatever the default value is.

Sure.

So it's those bits of still tested still framework things that change based off how we've configured and put them together and stack them. That is what we're actually testing.

Yeah, that's a great point. I agree with that a hundred percent. And maybe the ending note is like, this is a good problem to have to think about. "Oh, am I writing too many tests?"

But I guess the takeaway is like, please write tests and get to the point where you are in a privilege where you can say, "I have too many tests, and maybe I'll write fewer of them."

I think I'm a pretty smart guy. I know how to do things around the house. I think you are probably the same, Joel. You can, I don't know, do things.

Pretty broad. But yes, I could do things.

Yeah, you're capable of stuff. So, then I have a really important question that everyone else seems to be able to handle, but I can't. How the (beep) do you use cling wrap? It always sticks to... I can never actually ever use it.

Like, whenever I pull it out of the thing, it's like, oh, it's an amazing clear thing, I'm like... But by the time I start to rip it, it's folding over on itself. Or, if you're just like, "Okay, I want to try to put this on something before ripping it."

Well, then you got this giant box, and maybe the place that you have your bowl that you want to put stuff on isn't big enough to have a giant box next to it. And you got to make sure you push that down. And then when you're like, "I want to seal it before I do the rip parts."

And then how do you do the rip part? And remember when there used to be metal on those, and now it's just like paper. I'm supposed to rip plastic with paper and not get it stuck on me. How does this happen? Help.

No, first of all, stop buying the cheap cling wrap. You got to get the nicer one that has the little like cutter. It's almost like a little paper cutter that goes along it, but that only helps so much, too. You're absolutely right.

I'll throw in there like grocery plastic bags, like in the produce section. Like I'm like, "Am I missing some ingredient in my fingers or something that's helping manipulate the surface?" Like, I can't open the bag up or cling wrap, it just all goes together. I'm right there with you. I don't have a magic solution for this, Aaron.

No, that's not even the same. Because with the bag, you could move it around a little bit, and when you move it around, they start to come apart.

You could try.

And then you can shake it. You can do all that kind of stuff. As soon as you move the plastic wrap, it's just (beep). If you want someone to tell you that you just wrote way too many tests or not enough tests, I can do that.

If you want to ask someone's opinion and maybe have a friendlier conversation, there's a place where you can do that too.

It's the Mastering Laravel community. If you'd like to check that out, head over to masteringlaravel.io/community.