At work, we're definitely turning away from traditional unit tests, which is not to say we're turning away from testing.
Instead, for complex or algorithmic code we're using generative testing, barraging the code with random inputs instead of a few cherry-picked test cases.
Then, we're structuring our systems and operations in such a way that we can spin up test instances of the entire system and run full, end-to-end integration tests at various levels of load.
We haven't figured it all out yet, but the results are promising compared to unit tests, especially tests which contain a lot of mocks and stubs.
This is the approach that I take with my personal projects; I firmly believe quickcheck style testing is a huge improvement over manually writing out individual test cases. Do you tend to find, as I do, that when you write you code to be testable using randomized testing that it tends to look more functional? (I'm referring to the paradigm rather than the opposite of dysfunctional)
I find that the same quickcheck style testing is the fastest way to get out quality code for a project that I am the sole devleoper of.
I usually code in a terminal running screen with many shells going, one of which is running ipython so I can interactively import and test snippets, classes, or small functions that I'm writing in a larger module. It works great.
However - those tests disappear! Once a recent project of mine was handed off to a team for maintenance and more features, I found that, in particular places, where the code was subtle, it was subject to breaking by others who were just trying to fix something unrelated. This was when I, personally, realized the value of unit testing.
Yes, its slower and takes more time up front, but if your code may someday be worked on by others, it protects the sections of your code that have test coverage against breaking - which over a longer time scale can save time.
Well, considering the code I'm talking about is mostly in Clojure, yeah, definitely functional. It's really cool to see how all the pieces fit together once you buy into the functional paradigm.
For my algorithmic code, the unit tests do the random input selection.
Actually, it's often a sort of guided random testing. For example, if the algorithm has a straightforward linear regime with an exact solution, and a nonlinear regime that has to be solved iteratively, the unit test will target, say, 2% of inputs at the linear regime, 1% at the breakpoint, and the rest at the nonlinear regime.
Mockups and stubs are another matter. My algorithmic code is numerical, so stubs are rarely an issue. If an algorithm needs, say, a matrix solver, you pretty much just have to use a real (unit-tested) matrix solver in there.
The issue with using random test data is the it doesn't expose your edge cases.
There was a post on HN a short while back that showed why - in essence your edge cases sit at the edge of the normal distribution ergo they will rarely be hit by the unit tests.
This leads to you doing more work, as you need to log what data caused the test to fail. It also gives you a false sense of security, since the tests aren't really doing anything 99% of the time.
One thing to consider is that most code that gets written is not built to be highly testable. This makes testing harder to the point that "it's not worth it". If you optimize your coding style for testability, writing tests is faster and easier.
The problem is, writing testable code requires more thought up front not just about individual objects and methods, but also the overall architecture of your project. If your code is highly tied to an ORM or framework, then it's going to suck to test.
Also, not every language is equally easy to test. For example, Ruby has an obsessive culture around testing, so they have a lot of fantastic testing tools and the language itself makes things like mocking pretty easy. PHP by comparison doesn't value testing as much as Ruby does. The testing tools are worse and there are fewer of them. Also, due to the lack of monkey patching, you have to write your PHP code with dependency injection and testability in mind from day one for it to be highly testable.
In short, testing is a useful thing, but most code isn't written to support it and many languages and frameworks and patterns encourage you to write unpleasant to test code. If code is hard to test, programmers won't write tests unless their job requires it.
In my experience, that clause is not required. Code that's been designed for easy testing is better code, period. Not just because it's been tested. Code designed for testing tends to have cleaner interfaces and is better modularized.
Unit testing can easily lead to arbitrary modularity or code fracturing, which makes code harder to use and reason about. Good code should be designed for solving problems, easy use and readability. If your code is designed primarily for passing code coverage checks, then it probably is not very good.
This is generally my goal with making code testable. If I can make it modular and re-usable, that makes it much easier for me to maintain and extend in the future.
"Does this thing roughly do what I think it does?" is a secondary goal for me.
Not always. The GP post is more insightful than it might appear. A lot of reasoning used for arguing in favor of adding more unit tests is circular. "Good code is unit testable, so unit testable code is good."
Very often, optimizing certain kinds of code for unit testing makes it significantly more complicated, with none of that complexity helping you in the actual program. This is especially true for "end user" code, such as controllers.
Actually, if it's tied to an ORM, then you have to make DB calls for each tests, which is to say running the full test suite could take minutes, hours, or even days? If it takes a long time to run your tests, how often will you run them?
I'm not trying to imply that integration tests that hit the DB are always a bad thing, just that slow tests are another reason people don't like or use tests.
Certainly a programmer should not be sitting idle waiting for tests to finish. If you have a test suite that takes hours, it should be run every night. If that's still not frequent enough, you want test runs during the day, and it would slow down your machine too much to do those while you are working, fire up an EC2 instance.
Google recommends a hierarchy of tests. Say, (1) a quick test to make sure no REST URL crashes that runs in seconds, (2) a full functionality test that runs in minutes, (3) a fuzzer test that runs for hours, and (4) a stress test that runs for days or weeks.
Wow, that's awesome. I love the idea of a test hierarchy. I'm sure as a system grows in complexity and scale, fuzzing and stressing would make a lot of sense.
I see unit testing (and especially TDD) as a barrier to agile development.
If I'm cranking out a prototype to show a customer then I don't want to spend precious time writing tests for code that stands a good chance of being thrown away at the start of the next iteration.
Tests often do need to be written, but the best time to do this seems like an unsolved problem to me.
Not only is the best time to write tests an unsolved problem, but also the method in which you write tests. I think TDD is a joke, but unit tests are still valuable.
In my experience, a prototype is never just a prototype. You crank out a prototype in an hour to demonstrate something. Then somebody says "add this and then we'll show it to the boss.". Your boss says "that's cool, add this and this and this and then we'll demo it to a potential customer". You do so, then play test the demo a few times, uncovering scenarios that need hardening. You demo to the customer, and they ask a few questions meaning that you add a few more features to the prototype for the next demo. Rinse and repeat.
After a few successful demos you're given an tight deadline to "finish things", so you have no choice but to polish up the prototype and ship it.
The moral: code that was written to be thrown away often isn't. You can still write code quickly, and take shortcuts (such as skipping tests), but recognize that you'll probably be working on that code for a very long time and may regret your decisions later.
This is what I was alluding to when I mentioned that it's not obvious when tests should be written. It simply isn't practical to write them as part of a quick prototype (or any other rapid iteration). But writing them at the end of a project may also be difficult due to the need to move on to something else. The same argument holds for things like documentation...
I definitely agree with the statement about prototyping and testing. Testing is another signal of quality, a guarantee if you will, to your stakeholders that your software provided is dependable and trustworthy. That doesn't matter if you have no customers to begin with!
I'd also like to throw it out there that testing can be especially helpful when writing code against known problems. Some benefits are:
- Since you already know the problem domain, your test writing will be extremely quick and effortless, so the benefit of having that test in the first place far outweighs the time/effort invested in writing them.
- In dynamic languages especially, typos are what kills you in this situation. Testing will quickly catch 'em as you program.
- Finally, testing will force you to think more about the edge cases, since implementation is a straight-forward process. Thus, you'll spend less time later doing QA and have an overall more polished feature.
Your argument can only lead to the logical conclusion that you consider agile development to be only for making prototypes, which I assume you don't mean. But if you don't, your objection becomes unsupported. Maybe the problem is in your seemingly equating unit testing with TDD.
Perhaps my choice of words could have been better... swap out "prototype" (a hyperbolic example) in favour of "iteration" in my second paragraph and see if it makes more sense.
The central point I am trying to make is that it seems wasteful to spend time writing tests for code that may soon be deleted.
OK then, but in an iteration, you don't throw away everything you have and start from scratch. You use unit tests to check that those parts that weren't changed, weren't affected by the parts that were. Unit testing pays off especially in those scenarios.
I understand this. But writing tests as you go means you can't introduce new functional code at the same pace. Although I think it's an unsolved problem, I lean heavily towards writing tests (and documentation) towards the end of a project.
I discovered for my self that the best time to add tests to a project is when you cant overlook everything anymore and you ran into problems when you add one feature on one side and on the other a whole house breaks down.
There is no need to write tests for fast moving objects and for this kind of work TDD is a waste of time in my opinion. But for core infrastructure this makes sense.
But driving blind all the time is not a good idea on the long run.
Maybe you shouldn't write tests for a "prototype" that you're going to "throw away". Tests are (in my opinion) necessary, but TDD is a style of coding and you don't have to use it.
I'm sitting here significantly refactoring a HTML-generating class hierarchy that I made a couple of foundational mistakes on a couple of years ago, causing excessively slow rendering. It has about 100 classes, and is used in over 1000 screens. Most of them trivial, but the non-trivial ones are definitely non-trivial. Fortunately, it happens that there is a very clean input and output that is, by design and by network structure, isolated, making it feasible to test. I've got decent automated test coverage on the existing set, and I'm in the process of adding an automated test that will verify that both the old and the new system emit byte-for-byte identical output. Without this test, this would be a hopeless exercise and I'd have to just live with the outcome forever.
I find myself wondering how many people complaining about unit tests work on leaf-node sorts of programming projects, because for me, many of the tasks I routinely take on in my environment where about a dozen projects are building on the code I am writing mean that the question isn't whether unit tests are "better" or "worse". It's a question of whether the task is possible or not.
Unit testing is a very very expensive operation. Consider the following: The developers add unit testing at beginning for specific results. Later on, after product is released a new design or feature change happens. Often times unit tests will have to be rewritten else they will fail.
This can be very expensive and take developers and QA engineers extra time when they try and figure out what's wrong... "Is the test failing because the test is out of date? Or is our program causing a bug?" I know for a fact HP deals with this issue constantly.
Not saying unit testing is bad (I think it's usually good) but I don't think it's always good.
> "Is the test failing because the test is out of date? Or is our program causing a bug?"
IMO, when you change code (extensively), you'd still have to invariably answer those questions. But in the absence of tests, it becomes much harder to find the answers.
Indeed. The tests tell you that you are breaking a contract that you once thought was important. It might not be any more, but good tests reflect good contracts. Removing the test requires you to think whether any other code expects this contract.
Unit tests take a little time to write, sure. But look what you get for your investment:
(for the sake of this discussion I'll avoid subjective issues such as improved design)
1) Faster feedback cycle - if you're developing new code you can run a single test and check inputs/outputs very quickly. If you're testing by refreshing a page, or shudder, restarting your java server, you're going to have to wait far longer to find out if your code changes work as expected.
2) Reduced risk when making changes to the code base - the risk increases the closer you get to your data store as the amount of dependent code increases. The only way to reduce risk without unit tests in place is to not change any code i.e. you have to keep building on top (see the open/closed principle). This inevitably means that your code is going to introduce duplication as you can't refactor it out where appropriate. The other option of course is that you just risk it and hope to god that your manual QA testing will catch any problems before your users do.
It's also worth noting that the cost of adding unit tests isn't fixed, it's dependent on
a) experience
b) the tooling available for your language
From personal experience I find adding unit tests in ruby takes very little time. This means that benefit 1 in particular kicks in far sooner. Currently I feel it's faster for me to write a piece of code with a unit test than without.
That's the major problem with system tests, but it shouldn't be a big problem for unit tests. In my experience system tests are so fragile that the cost of upkeep is larger than the benefit gained except in a few cases. Unit tests on the other hand are much more stable.
My experience is precisely the reverse. System tests are stable because they only need to change when the observable behavior of the entire system changes. Unit tests are intimately coupled to the internal structure, so if you use them, you have to change them every time you refactor so much as a single function.
Lowly PHP dev here, so take my words with a grain of sakt, but if you go with TDD, any future refactoring should also follow TDD - write the test first (or modify existing tests) to what the expected behavior should be, then modify the actual app code.
That debate exists because the people selling the idea of unit-tests were so goddamn gung-ho for unit testing everywhere and all the time.
I have friends in the medical industry, their tests must achieve 100% code coverage on multiple coverage metrics, and code coverage is only an afterthought to even more rigorous test requirements. Even they were pointing and laughing at the TDD ideologues.
Unit testing is beneficial, if you do the right amount of it and in the right places at the right time. You might have a totally sensible test plan. But like it or not, for the time being, plenty of people are unwilling to listen to you because some other assholes already burned them on the idea of unit testing.
Its going to take years and years to shake loose all the cobwebs in our industry caused by half-cocked advocacy. Then we can maybe get to the sensible discussion about how to analyze a project and develop a sensible test strategy that fits the project's needs.
1.
Unit Testing is useful ONLY for complex API method/function, which should not change too often and too frequently;
2.
Ideally, you should only test API not API's implementation but most of the mock framework will tie your test code directly against your API implementation. Once your API implementation changed, your test code fails, you have to rewrite test code, which is waste of time.
Someone may argue that it because you write bad test code;
Talking is easier than doing;
So, usually, I prefer to doing the functional integration test than the unit test, which is productive for me. (I know people will throw out thousands of reasons about the benefit of unit test and "correct" way of writing unit test.)
Anyway, I will avoid unit test as possible as I can and ONLY do my designed integration test, which works for me.
3.
And writing unit test for code coverage is stupid;
Some manager even brag about the unit test coverage, which is totally a false confidence about the quality of the code underline; This is even stupid;
4.
Writing unit test for those requirement-always-change area is waste of time;
5.
TDD is a BIGGEST JOKE, which weights too much on writing auto testable code over writing working functional productive code;
The biggest problem I have with unit tests is they tend to be
a) just testing frameworks (crud) and not much else
b) so heavily built on mocking that changing them is like redesigning a wool sweater. These tests tend to examine how the code was written, not necessarily what the expected output is (oh shoot, I need to mock up five more objects to make this test pass... hey look at that, this test doesn't even check what should be returned. Nice)
c) there is no context into or out of the unit test. The assertions are vague, it isn't clear how I got there, it isn't clear where I'm going (testXMLOutWithInProgressUsers ... what is the xml for, who are the in progress users... what is next in this chain of events)
d) the things that REALLY need unit tests don't have them because they are too complex to write (ah, that whole system that runs a nightly bash script, takes in a stack of csv files from the client and converts it in our database... yeah, that's not tested).
With all that said, tests are invaluable, but they need to actually, you know, TEST something.
I like nodejs in that respect. It takes me seconds to fire up the webserver and test the rest api. I don't mock objects, I pass values directly through the api and check the expected results. I like this because it shows the next user what the api actually looks like and I'm testing the full stack (In my experience controllers are notoriously undertested, mainly because you need the full stack running). Since the application is really just an api this means that I'm able to test everything equally, from batch processes to CRUD.
API testing has that wonderful property, its appearance is the whole deal, if it looks right then its working.
Anything but app code isn't so easily tested. When frameworks are written to support testing, they often are sub-useful - crippled versions of real programming languages etc.
Anecdote: I recall a QA guy asking me how to test our OS's file system. I said, make every filespec-based call with a good spec, a non-existant spec, an illegal spec, a null spec; log every result. After each release, diff the logs and explain every deviation.
He said "That's too hard, I'll just shotgun it." and left. Why ? Because his test-scripting framework wouldn't allow anything as sophisticated as lambdas etc.
So I spent half an hour writing a table and a loop, and sent it to him. He used it for the next 5 years to test every release.
My moral: do an exhaustive test when possible. The computer is fast, it doesn't feel any pain, and you will find more fencepost and boundary issues than you ever thought existed.
In practice, I find I write code faster and with more confidence after the tests (usually specs) have been written. So, my productivity increases. I also sleep better at night.
For me TDD is doing the exact opposite. It bores me to tears, which really brings down my edge.
Granted, this is one of the reasons why I'm recently thinking about whether I've got a slight case of ADHD, but I think a lot of it just stems from the fact that I like to brainstorm code and afterwards use something like tests or just good documentation to reason whether that was the most proper approach.
Never mind that things like cucumber make me cringe big time.
I agree that experimentation and brainstorming are not the time to be messing with unit tests. Specs maybe. I've been frustrated enough with failing tests to occasionally lose them.
But, when it comes to something I expect someone else to use, there is no question that I will start with a test.
Yes, this is an obvious downside to the type of TDD that you sometimes see advocated, where they write tests for every little thing.
I try to find a compromise solution. I pretty much only write one unit test for a given module to start with, the nominal case. This is important, because it forces you to think a little bit about how you actually want your code to work, which I find to be a rather healthy thing to do.
Once that is done, a little later I will no doubt find a bug. When I'm trying to debug, and I start wondering whether a block of code is doing what I think it is doing, I write a test to verify. This does one of two things - it shows up the bug, which I can then correct, and verify that I have corrected, or it allows me to remove the tested code as a possible cause for the bug, allowing me to concentrate on other hypotheses for the cause of the bug, speeding up the debug process.
By the time I get to the end of the project, I generally have a pretty comprehensive test suite, without ever once feeling like it was a chore.
I don't even mind declaring variables all that much. I think a lot of it comes down to locality, as I've got all the "testing" code in the same piece of code, instead of in a totally separate test suite. So type declarations, design-by-contract, debug statements would be my first choice, and I actually often start out with these (or approximations thereof) even in more dynamic languages, refactoring some of those as unit tests later on.
(In a similar vein, I don't usually start out with interfaces, abstract classes etc. - if possible not even with classes at all. Guess that makes me a spaghetti sorter.)
Granted, that's mostly for starting out. In an existing code base, especially a larger one, different rules apply. I don't believe in methodology panaceas.
Unit testing, like many other good and healthy coding practises is great. In theory.
In the big mean real world frequently you're shipping things in such narrow deadlines that if you say "Hey boss, I'm going to take some time to write some wicked cool unit tests, it'll make my life easier in future because we'll know this code works!" then Boss' head will pop as it's a non-tangible improvement.
Developer focused companies are great as they'll let you get away with this, but the large portion aren't.
I usually prefer "automatized system testing" where fewer "high level" tests involve a lot of code, preferably in a test environment that is close the actual production environment.
Of course, when a test fails, some more analysis is required to determine what went wrong. But it's usually related to recently modified code, hence easy to spot.
That way I get the benefits of automatized tests without the burden of tons of "micro" tests at the class/method level.
There's no debate about testing stateless functions. I think the debate is in whether the ledger reflects a cost benefit of spending all the time mocking up different functionalities in order to test stateful components or components with external dependencies. Since these are hard to calculate ahead of time, it really does become a question of engineering culture and time to market.
why do i get the feeling that you probably haven't spent much time "debugging" unit-tests...
unit testing give you the false confidence that your stuff is working, it has little to no bearing on stuff actually working when pushed to production (unless you have 100% coverage and actually thought of and defended againt every possible use scenario before hand)
Yes, unit tests are code and all code can have bugs, but you're missing the point. Automated tests are a reflection on the code just like code reviews or documentation or formal specifications. If they are well written (ie. orthogonally structured to the code they are testing and with good readability) then they provide a strong window into the intent of the original developer, and this intent can be continually tested for almost no marginal cost for the lifetime of the code.
Yes there is a maintenance cost, but I find most people's objection to testing is based on poor testing skills. Once you get good at testing with a decent framework, you find that it's often about the same time to write formal tests as it is to do the manual testing which we all know is necessary to have any certainty that code is working.
The reason is that automated tests costs 2-3x as much time upfront. The assumption is then you will recoup that investment in the future. The problem is that automated testing doesn't cover everything so you have to augment it with human testers. Which then begs the question are the human testers being effectively utilised.
And for those of that do develop have a think about all the time spent on buggy web UI testing, screenshot comparison testing, tests being flakey due to software choices etc.
Automated _unit_ tests cost two or three times as much time, sure. That doesn't mean don't use automated tests, it means don't use unit tests. Use integration tests instead. Not only do they provide much better coverage (you're testing all the code from top to bottom, each module in the context in which it is actually used), but they are also far cheaper.
Agreed, I would focus on hiring shit hot testers and automating end to end testing if possible. Unit testing is useful for a small number of examples like a method that does complicated calculations,but I see TDD zealots writing them to test a method that simply forwards data to another layer, all designed around dependency injection.
You know, actual luminaries in our field are also against them. I've seen UNIX programming gods among men dismiss them. It's not like it's a clear cut issue.
For some things like libraries and such, there are absolutely needed, but unit testing everything and especially TDD are stupid.
To be frank, they mostly sound like some Java EE-like monstrosity that for some bizarro reason caught on with the hip crowd too. I'd put them out there with FactoryOfFactory and XML documentation...
When I think about UNIX programming gods, I think about people who deal with tools that are never modified, deal with very small problems. They're trivial to test functionally, but are next to impossible to test internally (have you ever seen source for tools like tar?)
It's definitely some solution for testing. But when you're writing a webservice, you should think twice before assuming that anything a "UNIX programming god" says is a good advice for you.
Instead, for complex or algorithmic code we're using generative testing, barraging the code with random inputs instead of a few cherry-picked test cases.
Then, we're structuring our systems and operations in such a way that we can spin up test instances of the entire system and run full, end-to-end integration tests at various levels of load.
We haven't figured it all out yet, but the results are promising compared to unit tests, especially tests which contain a lot of mocks and stubs.