Tested Reality
I learned a valuable lesson today. I wouldn't say that I started learning it today...that started about 9 or so years ago, when I started really programming. But today the full force of this essential truth hit me:
The entire reality of the software universe is defined by tests.
I don't mean to say that every scrap of code out there is tested; quite the contrary. What I mean to say - and this is very Buddhist, so hang on - is that the only software that really exists in the world is tested. If you define software as a standing wave in the continuous looping of the SDLC (software development lifecycle), then the only way that wave can propagate is through tests. For software, tests are transcendental; they really are the way, the truth, and the light. Maybe I'm waxing philosophical here, so let me see if I can ground this discussion a bit more.
A business defines its software as machine automation that applies the rules of one or more business processes. This means that it encodes the business rules. But this isn't enough. Since software lives in an ever-changing environment, one criterion for success is its ability to adapt to new permutations of the environment - faster processors, new hardware architectures, different network topologies. As the business itself evolves, so must the software in order to continue to serve its purpose as an automator of the business' processes. This is all really just a fancy way of saying that the SDLC is the context in which software lives...the SDLC is its reality.
What I'm really driving at here is the fact that the SDLC - a cycle of repeated revisions to the encoded and automated rules for a business - is the basis for true software. If software fails to adapt to change, it is pruned out of reality. Therefore, software's continued propagation into new environments hinges on its ability to adapt.
So, what is needed to guarantee adaptability? When you cast off a former incarnation, and reinvent your software, how can you guarantee that this iteration fits the same needs that the old one fulfilled? Testing is the answer. The test suite for an application is the development team's articulation of the reality they perceive, the same reality that the application assumes. Remember, we're talking about software over time here. Regardless of what the development team understands at the time of writing, anything not encoded in a test will fade and disappear with time.
Where does that leave development teams whose applications have no tests? Simply put, they don't really understand anything about the reality in which the application exists. But this is rarely the case. Most teams do perform some level of testing, and do posess some level of knowledge about the system for which they are responsible. Manual testing by domain experts is a very common approach to testing, for example. Domain experts represent one way of encoding application reality, since they can verify that an application continues to serve its function from iteration to iteration. However, domain experts aren't very good at storing this information.
If you rely on this form of testing, please understand that I'm not calling your domain experts incompetent or making any assertion about the quality of their memories. However, the simple fact is that domain experts degrade over time; they have a shelf life. If you catch them really early, you might get as much as 40-45 years out of them...after that, they have an annoying tendency to retire and go fishing. Also remember, they're not called experts for nothing; they have to accumulate a certain amount of business knowledge before they can verify much of anything.
Another problem with domain-expert-driven testing is that it doesn't automate or parallellize very well. If you have one application with five new patches to apply, and you want to isolate any test failures by patch, you can't simply apply one patch to each of five copies of the original application and test. You're limited by the number of hours available to your domain experts. If you want your tests to run every night at two in the morning while you sleep, you might not want to be in the room when you tell your domain experts. Combine this with the near-universal acceptance that the cost of a software bug increases exponentially the later it is detected, and you've got a process that will bleed you dry and leave you hopelessly stuck on the wrong side of an industrial revolution.
You can't refactor your code if you don't have tests. No business will ever be able to find the political will to successfully refactor its code if it relies on domain experts and manual testing. Manual testing puts an upper limit on the size, complexity, and flexibility of your software - in short, the number of changes which can be processed at once. So, ironically, in analyzing applications (business rules) and their reality (the process by which these rules are implemented), I've come to see application testing as another iteration in the principle that continues to fuel the value proposition of software in general:
Automate or die. If you don't automate, you cannot scale.
The entire reality of the software universe is defined by tests.
I don't mean to say that every scrap of code out there is tested; quite the contrary. What I mean to say - and this is very Buddhist, so hang on - is that the only software that really exists in the world is tested. If you define software as a standing wave in the continuous looping of the SDLC (software development lifecycle), then the only way that wave can propagate is through tests. For software, tests are transcendental; they really are the way, the truth, and the light. Maybe I'm waxing philosophical here, so let me see if I can ground this discussion a bit more.
A business defines its software as machine automation that applies the rules of one or more business processes. This means that it encodes the business rules. But this isn't enough. Since software lives in an ever-changing environment, one criterion for success is its ability to adapt to new permutations of the environment - faster processors, new hardware architectures, different network topologies. As the business itself evolves, so must the software in order to continue to serve its purpose as an automator of the business' processes. This is all really just a fancy way of saying that the SDLC is the context in which software lives...the SDLC is its reality.
What I'm really driving at here is the fact that the SDLC - a cycle of repeated revisions to the encoded and automated rules for a business - is the basis for true software. If software fails to adapt to change, it is pruned out of reality. Therefore, software's continued propagation into new environments hinges on its ability to adapt.
So, what is needed to guarantee adaptability? When you cast off a former incarnation, and reinvent your software, how can you guarantee that this iteration fits the same needs that the old one fulfilled? Testing is the answer. The test suite for an application is the development team's articulation of the reality they perceive, the same reality that the application assumes. Remember, we're talking about software over time here. Regardless of what the development team understands at the time of writing, anything not encoded in a test will fade and disappear with time.
Where does that leave development teams whose applications have no tests? Simply put, they don't really understand anything about the reality in which the application exists. But this is rarely the case. Most teams do perform some level of testing, and do posess some level of knowledge about the system for which they are responsible. Manual testing by domain experts is a very common approach to testing, for example. Domain experts represent one way of encoding application reality, since they can verify that an application continues to serve its function from iteration to iteration. However, domain experts aren't very good at storing this information.
If you rely on this form of testing, please understand that I'm not calling your domain experts incompetent or making any assertion about the quality of their memories. However, the simple fact is that domain experts degrade over time; they have a shelf life. If you catch them really early, you might get as much as 40-45 years out of them...after that, they have an annoying tendency to retire and go fishing. Also remember, they're not called experts for nothing; they have to accumulate a certain amount of business knowledge before they can verify much of anything.
Another problem with domain-expert-driven testing is that it doesn't automate or parallellize very well. If you have one application with five new patches to apply, and you want to isolate any test failures by patch, you can't simply apply one patch to each of five copies of the original application and test. You're limited by the number of hours available to your domain experts. If you want your tests to run every night at two in the morning while you sleep, you might not want to be in the room when you tell your domain experts. Combine this with the near-universal acceptance that the cost of a software bug increases exponentially the later it is detected, and you've got a process that will bleed you dry and leave you hopelessly stuck on the wrong side of an industrial revolution.
You can't refactor your code if you don't have tests. No business will ever be able to find the political will to successfully refactor its code if it relies on domain experts and manual testing. Manual testing puts an upper limit on the size, complexity, and flexibility of your software - in short, the number of changes which can be processed at once. So, ironically, in analyzing applications (business rules) and their reality (the process by which these rules are implemented), I've come to see application testing as another iteration in the principle that continues to fuel the value proposition of software in general:
Automate or die. If you don't automate, you cannot scale.
Technorati Tags: code, process, smells, testing
Posted on June 20, 2006 12:12:40 AM EDT
#