Test management approach
From Wikipedia, the free encyclopedia
In software engineering, the Test Management Approach' (TMap) is an approach to the structured testing of software products, both low-level and high-level. TMap a popular standard for software testing in Germany and the Benelux, being used by more than three hundred organisations, including banks, insurance companies, government departments[citation needed].
Contents |
[edit] An insight in testing
Before venturing in the domain of the TMap method, it is wise to have a look at the scope of the method and create a feeling about what the purpose is of the method. To fully grasp the testing process some definitions need to be given to explain the context of given terms. Of course everyone knows what testing means, but how can it be used in such a way that its results can trigger important decisions?
[edit] What is testing?
The main activity within a testing process is making a comparison. To make this comparison, an object to be tested and a point of reference or a so called test base are needed. The testing process is then used to give information about the object and the differences between its current and required state. An ISO definition about testing is as follows:
“Activities performed to identify one or more characteristics of a product, process or service using a prescribed procedure” [ISO/IEC Guide 2, 1991]
So testing is about identifying the difference between the actual and required state. This of course has to do with the quality of the test object. Though quality can be perceived as “meeting the demands”, the testing process is about assigning value to the quality of the test object. Low quality meaning that the required state has not yet been achieved, high quality meaning close proximity to the required state.
When assigning a value to the quality factor of a test object, the management in an organisation can make decisions based on this value. It can give insight in the risks that are taken when accepting a product with less quality than required. This is what the testing process is all about. It is a tool for decisions on a higher level to give insight in the level of quality of a product that the organisation wishes to put on the market. Is the software product good enough to distribute? Is our system as fault tolerant as we planned it to be? These are questions that are asked on a higher level that can be answered with correct testing of the test object. So in a sense, testing is a tool on a high level, but a method on a lower level. On a high level a question is put in the black box of testing, and the answer comes out. On a low level a question comes in, and a set of prescribed actions, techniques, activities and steps help to provide an answer to the question.
Testing must not be mistaken for checking. Checking is about testing intermediate products and development processes. In the context of software development, it answers the question: “Is the building process properly executed?”. Testing on the other hand has a larger scope. It is about inspecting finished products or finished intermediate products. It answers the question: “Does the product meet the specified demands?”. The difference between these two concepts is thin, but it has mostly to do with scope. In the formal sense, testing is about finding errors, but not as a goal but as a means. The goal is to assign a value to quality. The lack of quality is measured by the fault sensitivity of the test object. This leads to another, more suitable definition of testing:
“Testing is a process of planning, preparing, executing and evaluating used to identify the characteristics of an information system and pointing out the difference between actual and required state” [Testen volgens TMap 2e druk, 2000]
[edit] Why test?
A question that can be asked is “why test?”. It is nice that you can assign a value to the quality of a software product, but what is the use of this value, what can we do with it? The use of IT is widespread in the organisational world. It is often used to support various business processes within companies. The market poses high demands on these companies (e.g. localisation and globalisation). A high level of flexibility is therefore desired in combination with shorter planning periods to release products quicker.
To cope with these demands, the management is becoming more and more dependent on the quality of the information systems they sell. As ISO formulates: Is it certain that the product possesses the characteristics to meet the given or self-evident demand? Especially the self-evident demand is important. It means that the product should have certain characteristics that are logical, i.e. have characteristics that would seem normal to have under the given conditions. Have all the errors been dealt with, and has this not yielded new errors? Does the system provide the reliable solution for the situation for which it was designed? Can it be launched within the borders of the present risk factors? For not having to answer these questions in the final stages of the development process a structured testing process is required. It can answer these questions in an earlier stage so fitting solutions can be effectively implemented in time.
Another question that can be asked is: “Where is the process of testing, positioned in the scope of organisations nowadays?”. Unfortunately, the answer is that in many organisations the process of testing is underdeveloped. They tend to postpone the introduction of a testing structure to structurally improve quality of their product. Lack of organisational know-how to set up such a branch is often the cause of this. The scope and result of testing is also perceived in a wrong way. The result of testing is thought of as perfect. The common thought is that it is able to identify all errors within a product. This is unfortunately not the way this works in practice.
[edit] Use of testing in practice
The development and maintenance of information systems demands special attention for quality. In this case, quality is directly linked to the demands of the customer or intended user. In many cases a software product is tailor-made, or is a specialised product serving a specific process in the customers business. To obtain the demanded level of quality more and more activities are planned. Some claim that activities to prevent errors from happening at all suffice. This would mean that with the right preparation no errors would occur at any time. Due to the fact that errors are dependent on numerous incontrollable factors this is a perfect situation that does not match with the situation in practice.
The goal is to prevent errors from occurring by identifying the cause of the present errors. In fact it is trying to cure a disease rather than trying to fight the symptoms caused by it. This does not mean that all errors can be prevented as stated earlier. It is a combination of prevention and trying to optimize the process of finding errors. The testing process becomes more efficient when the time between the creation and detection of an error is decreased. In an ideal situation, developers of software are more motivated to produce less faulty code when they are supported by a test team that indicates in what area they are most likely to make mistakes. The test process as a whole would then yield not only a direct, but also an indirect positive result on the quality of a product.
As stated in multiple related articles, quality of a product should be built, not tested. The improvement of quality simply by the process of testing is extremely cost intensive. Using this principle, a test indicating a poor system should not be tested over and over again, but be completely redesigned to prevent the occurrence of similar defects in the future. Software companies rarely come to the conclusion that it is better to completely redesign a product. Maybe the thought is there, but other options tend to be more plausible than a complete redesign though it would be the sensible thing to do when the scope is extended over the overall project. Redesign in an early stage prevents the production of a severely quality lacking product that turns out to be an expensive fiasco.
The testing process is not just about executing a test. It should be looked at in a broader scope. There are more activities involved in the process than just the actual test. The planning and preparation are equally if not more important than the actual test. Testing as a concept is often seen as just performing a test but actually in most cases the actual execution of a test contributes to just forty percent of the total time spent on the process. The remaining sixty percent is comprised of planning, preparation, specification and completion of the test.
Organisations tend to give less attention to these concepts whereas they are the most important aspect of the process. If, for example, the execution of a test is not properly prepared, how can the results of the test be reliable enough to base decisions on them? The trend is that the balance between total test process time and test execution time is shifting. Less hours of the total amount of time available are spent on the actual execution, and more time is assigned to the supporting processes.
Testing is not about acceptance. It is about supporting the decision making process. It gives advice about the quality of a product. In the scope of a development process, for example within the DSDM development method, it is not a separate phase. It is a series of prescribed activities that have to be performed from the start of the overall development. So from the view of just the testing, it is a separate process, but it cannot be detached from a development process. From the moment the functional demands or requirements are documented, the test process can be planned. Though the test and building process tend to be performed simultaneously, both have to be given equal attention to. The building cannot continue without proper testing, and the testing cannot be performed if nothing is built. Testing is an expensive undertaking. It can form up to fifty percent of total development costs. This may seem much, but research has shown that costs increase exponentially when the time between an error and its detection increases. So the relation between time and cost is exponentially [Boehm, 1981].
[edit] What is quality and how does it relate to testing?
The concept of quality is a vague concept. What is quality? How can quality be measured? How can we assign a value to the concept of quality? This remains to this date an important issue in the IT world. Because testing is only a part of the puzzle in trying to achieve a certain level of quality, other aspects have to be involved. Testing in itself cannot guarantee quality. It can only detect if a certain level of quality is reached. The ISO gives the following definition of quality:
“The totality of characteristics of a product or service that bear on its ability to satisfy stated or implied needs” [ISO 8402, 1994]
This definition is still vague. Especially the part that states that implied needs have to be incorporated in the quality. What are the implied needs? In an ideal situation there are no implied needs because all of them have been recorded in the stated needs. When dealing with the concept of quality it is evident that it is difficult to use and be specific at the same time. The concept should be specified further for it to be used in a sense that decisions can be based on it. The way to do this is to break up the concept of quality into sub characteristics. Per sub characteristic, some form of measurement should be specified so a value can be assigned to it. In 1977, McCall proposed to divide the concept of quality into so called quality attributes. ISO defined a standard for these quality attributes [ISO 9126-1, 1999].
Six characteristics have been defined, each with corresponding sub-characteristics. The idea is that all these characteristics together tackle the entire field of quality present in the field of software development. The attributes are as follows:
- Functionality
- Suitability
- Accuracy
- Interoperability
- Security
- Reliability
- Maturity
- Fault tolerance
- Recoverability
- Usability
- Understandability
- Learnability
- Operability
- Attractiveness
- Efficiency
- Time behaviour
- Resource utilization
- Maintainability
- Analysability
- Changeability
- Stability
- Testability
- Portability
- Adaptability
- Installability
- Co-existence
- Replaceability
In testing, and in the TMap method in particular, the quality attributes are divided into two groups or categories. First you have the dynamic quality attributes, and secondly the static quality attributes. The dynamic quality attributes are related to the use of the software product. Does the system do what it is supposed to do, what it is designed to do? Is input processed correctly? Is the system stable and are errors easily dealt with? Is the system available when needed? The static quality attributes are related to maintenance and management of the software product. Is it easy to modify the system and how fast are these changes? Is the code reusable? Is the infrastructure behind the software suited for its use?
Summarizing the different aspects of testing discussed it can be said that testing is more difficult than it would seem at first glance. It is not just about detecting errors, it is about assigning value to the quality of a system so decisions can be based upon it. It is an underestimated field in the software industry that has only recently gained attention with the development of structured testing methods such as TMap. It provides management with tools to assess its main product and enables it to make decisions based on rational results rather than “gut feelings”. To do this the concept of quality is broken down in measurable quality attributes on which tests can be specified. How the TMap method deals with these attributes will be explained in the next part.
[edit] Tmap fundamentals
The Tmap method approaches the testing process in a structural way. In order to define what basic demands should be met in order to successfully implement a structured testing process in an organisation several characteristics or bottlenecks in businesses or projects should be reviewed. The process of testing is not simply conducting an investigation indicating that there are errors in a system; it is a process that should be professionally dealt with and supported in all layers of an organisation. There are several key factors in a project or organisation that can negatively influence the way testing is done or is to be planned.
[edit] Bottlenecks
[edit] Time pressure
Time is a key factor in every project. Insufficient personnel, lack of decent planning and postponed preparation can cause serious damage to the quality of the tests performed.
Phasing of test activities: In projects a structured approach for testing is often omitted. Basic questions like “what test activities should be performed and by whom?” are not always answered. The lack of planning and phasing of test activities causes the test process to be unmanageable and therefore unreliable.
[edit] Techniques and tools
Though often present the available tools and techniques for testing are not used properly. A common mistake is that the acquisition of a test tool solves the testing problems. It is not the presence of a tool, but the correct use of it that makes a test reliable and usable. But not only in tools this is the case, also the incorrect use of techniques is a problem. When important steps in prescribed processes are deliberately changed or even worse, left out, by testers with no impression of the overall testing structure, the result of the process is useless.
[edit] Requirement specification
The customer demands regarding software products are often captured in requirements. These requirements form the basis upon which testing activities are conducted. If these requirements are not worked out properly, structured testing is impossible.
[edit] Coordination of test activities
In projects and organisations the quality of the documentation of the available test methods and techniques is often poor. Employees seem to do what they find suitable. In many cases, the available techniques and tools overlap in certain areas, which means that they are competing techniques and tools for the same cause. Lack of proper coordination of the use of these techniques and tools causes a divided development team. Small groups or even individuals don’t know the overall testing structure and aren’t aware of the activities performed by their colleagues.
[edit] Recommendations
The factors and examples given shed a light on the difficulty and importance of a structured approach of testing a software product. There are so many factors involved that it is easy to negatively influence the quality of the whole testing process. It is easy to see that these factors have to be controlled in such a way that the quality of the performed tests is as high as possible. To do this, a testing method is required to structure the way tests are prepared and performed while keeping the scope of the overall development process in mind. Some recommendations to such a method are given.
[edit] Reduction of time pressure
We saw earlier that time pressure can be a real killer in a project. There are several ways to decrease the time pressure in a project. The first thing is a flexible planning beginning with the early creation of a master test plan. This deliverable will be reviewed later. The evaluation of the progress is also a key issue. Knowing if the execution of activities is going according to the planning can give an early indication that the process is being delayed.
The availability and adequacy of personnel is also important. If the personnel is properly trained, tasks and activities are faster and more thoroughly performed. Another important issue is making agreements prior to starting the testing process. With skilled personnel, well worked out planning, solid agreements and progress monitoring time pressure can be heavily reduced.
[edit] Project based
An approach to optimise the quality of the tests could be to separate testers (test personnel) from the daily project activities. In this way there is no interference between different goals. In this way, test activities can be handled project based and separately based on the planned progress of the whole software product.
[edit] Phasing of test activities parallel to the software development process
As stated above, it is wise to plan the test activities parallel to the building activities. When these two are synchronised, no one is waiting for the other and no delays will arise. Per phase details should be recorded to specify what has to be done and when. Concepts such as goals, techniques and deliverables have to be specified. The result of these can then be used in the overall project to determine if the quality of the product is sufficient to proceed to the next building phase.
[edit] Structured and suited use of tools
An array of tools is available to support the different aspects of the testing process, direct and indirect. Direct tools are tools that support the managing of the testing process. Examples are tools that evaluate test results or record and playback test activities. Indirect tools are the tools that support the testing process as a whole. Examples are tools for budgeting or progress managing. It is important to keep in mind that the use of tools is not a goal in itself, but a means to goal. The goal is to guarantee a certain level of quality for a software product.
[edit] Teambuilding and reduction of conflicting interests
By creating a master test plan in an early stage of the overall project, agreements can be documented in which the involved personnel can decide who does what. By doing so, the situation and activities are clear to everyone. This prevents possible disturbances or irritations in a future phase. It is wise to create a structure of cooperation in which every employee is working on the same goal whilst retaining a certain level of responsibility.
[edit] Setup of a test helpdesk
Testers need methods, techniques, tools, education and support. It could be beneficial for these testers to profit from the experience of other testers. This could differ from advice to a plug and play solution for a given situation. The facilities for coordinating this supporting function should be implemented on an organisational level for the process to be fully supported. Examples of functions this supporting branch could provide are:
- Prescriptions for methods, techniques and tools
- Maintenance of these prescriptions
- overall planning of concurring test activities
- creation and maintenance of the test infrastructure
- Education and coaching of test personnel
[edit] Method structure
TMap is an abbreviation of Test Management Approach. It is a structured method for testing. Structure in this context means that the method needs certain external conditions that have to met in order for the method to be executed successfully. The method stands or falls in relation to the structure it is used in. The method defines four main concepts which have to be implemented correctly in order to successfully use the method. These concepts are:
- Phasing of test activities, related to the development cycle of the software product (P)
- Embedding in organisation (O)
- Matching infrastructure (I)
- Usable Techniques for executing the activities (T)
[edit] Phasing
The activities that are performed in the TMap method can be structured in a phasing model. This phasing model has to be implemented parallel to the development phasing of the software product. As stated earlier, a consequence of this is the streamlining of the development process as a whole. Some tests require a testobject in time. This testobject is provided by the software development process. When planning such a test it would be wise to plan it close to the production of the corresponding testobject which is often a subpart of the system.
In the TMap method three main activities can be distinguished: planning, preparation and execution. These activities are divided into five phases: Planning and Control, Preparation, Specification, Execution and Completion. The completion phase was added to finish the test process properly and to carry the responsibility to hand over the results to the management. The TMap phasing model is a generic model which means that it is applicable to a wide selection of tests. For each phase different activities can be distinguished. It is up to the manager to select which of the available activities in the TMap method are assigned to which phase. This selection has to be made because it is often not required to execute all the available tests. It all depends on which quality attributes matter the most for the product or subproduct.
To accurately describe the phases and provide an overall view of the TMap method, a model showing the different phases is given.
The model illustrates how the different phases relate to one another. Though it may seem logical, there is something awkward about it. Though the Planning and Control phase passes on to the Preparation phase, it still continues in itself. This is because it contains activities that have to be performed throughout the process which cannot be put in other phases. This has to do with the control aspect of the phase which will be elaborated on later. The remaining phases seem to have a logical order. To view these phases in relation to a software development process, let’s have look at a metamodel to describe this.
[edit] Planning and Control
Now the scope and the phasing as a whole have been looked at it is time to take a closer look to the individual phases and explain what phase is intended for what purpose. Also some activities and techniques will be elaborated on per phase. The first phase is the Planning and Control phase. It starts in the specification stage of a software development project. It is the most important phase. In this early phase difficult aspects of the testing process have to be anticipated such as the assigning of personnel and resources and the planning of certain tests in relation to milestones of the software project.
One of the key aspects in this phase is the creation of a so called strategy matrix. It gives details about the different quality attributes important to the system. Only in a perfect situation a system can be tested on all quality attributes with a hundred percent attention. Together with the client it has to be decided what quality attributes are important for what subpart of the system and how much attention each attribute should have. This could result in the following example strategymatrix and its meta datamodel:
This is a meta datamodel of this strategy matrix because it is important for the future phases that this is done correctly. Within the matrix there can be an arbitrary number of subsystems per row, but the quality attribute and a relative importance have to be present. Also important is that the percentages of the importance column add up to a hundred percent. This is logical, because you cannot give more than a hundred percent attention to the attributes. This strategy matrix is used in later phases to base tests on and calculations for manhours.
Also during the Planning and Control phase the design for the testinfrastructure is made. The infrastructure is important because it reflects the situation the product is designed for. It makes sense to test the product in an environment that it was designed for, for the results of the tests to be reliable. The first part of the Planning and Control phase can be captured in the development of a master testplan. This mastertestplan contains all the activities needed to set up a test process and creating the supporting concepts.
The remaining activities have to do with the control aspect of the Planning and Control phase. These activities are designed for monitoring the progress of the tests and their relationship with the use of time and resources. During the creation of the mastertestplan a planning was set up to determine the amount of time that could be spent on what test. The monitoring aspect of the phase can indicate if the testprocess is running smoothly or not, whether correcting actions are necessary or not. The most important outcome of the testprocess is the advice it gives about the quality of the product. This can also be the quality of the subproducts. So during the phases the monitoring can provide an indication about where the system stands and what its quality is in relation to the specified strategy matrix. The process should be able to answer questions the decisionmaking asks of it. So not just a simple “no, the system is not ready”, but an indication of what quality attribute is lacking in what subsystem and how much time is necessary to fix it. Decisions can then be made to go into production after all, or to postpone the release until the quality constraint is met.
[edit] Preparation
After the mastertestplan has been created the Preparation phase is initiated. Input for the Preparation phase is the testbase. This testbase is the object on which the tests will be executed and the mastertesplan created in the earlier phase. In conjunction with the builders of the product the testbase will be divided into subparts. In most cases these subparts of the testbase correspond with the subparts of the overall product as specified in the strategy matrix. For each of the subparts testtechniques will be assigned and a planning will be made for the tests to be executed.
[edit] Specification
In the Specification phase the testcases are specified and the corresponding testinfrastructure defined in an earlier phase is realised. The testcases are created in two steps, the logical and physical testcase. From the moment the testbase is available it is possible to define testcases for testing. For a given testcase this could be specifying the starting situation, the process within and predicting the outcome of the result. This is a logical representation of the testcase. When the testbase is actually realised more technical aspects of the testcases can be specified such as testscripts. To facilitate these testcases the corresponding infrastructure is created. As stated earlier, this infrastructure has to match the environmental factors of the situation in practice.
[edit] Execution
In the Execution phase the actual testing takes place. It starts when the first testable components of a system are available. The component forms the testbase on which the tests are performed. The part is first checked for completeness and then installed in the corresponding test infrastructure. The first test is if the component actually works in its test environment. If this is the case the testdata is prepared for the execution of the test. This is an important process that is often executed with actual system functions. If all these demands are met, the so called pre-tests are executed first to check the main functions of the testobject. They indicate whether the testobject has a sufficient level of quality for it to be tested thouroughly. If the pre-tests indicate that all is clear, the actual testscripts can be run on the testobject and the test executed. During the testing intermediate results can be handed over to management for decisionmaking, for example tot stop the test if necessary. This has to do with the Planning and Control phase which monitors the overall testing process.
[edit] Completion
After all the tests have been completed there are some remaining activities to be executed. These activities tend to be forgotten because of the time pressure involved in the actual testing. This would be a waste of resources because the testresults are the reason the tests were conducted and they have to be documented properly for further and future use. A selection is made for certain testcases and testdate to be preserved for future cases. In a later case this data does not have to be recreated and the testcases not reinvented. Also an evaluation takes place. Have the tests been conducted properly? Is the result what was exptected from it? Should some of the processes be changed or planned diffently in the future to provide better outcome? These are all evaluating questions that seem redundant but can provide significant gain of time in future test processes. Another aspect is the setup of a gain / loss evaluation. It can be calculated how much time and money the testing process cost, but also what it gained. If there was any doubt within the organisation to spend a lot of time on testing, a nice gain/loss ratio will surely fix that!
[edit] Organisation
Every test process which is not fully supported by a structured organisation will fail. There are several aspects involved in managing an organisation to accommodate the testprocess. Conflict of interest, unpredictability, complicated managingtasks, lack of experience and timepressure all make the organisation a dynamic force. Also the fact that there are many parties involved in the test organisation makes it difficult to manage. These parties are for example system designers, functional management, task forces, accountancy, clients and project management. As [thierry, 1973] describes it: “Test organisation is the creation of opportune proportions between testfunctions, testfacilities and testactivities with the common purpose of providing good quality advice the time it is needed”.
The organisation of a structured testing process focusses on five main aspects:
- The operational testprocess
- The structural testorganisation
- Testmanagement
- Personnel and training
- Structuring the testprocess
[edit] Infrastructure
The test infrastructure is comprised of all the means and facilities required to execute the tests properly. A distinction is made between facilities directly necessary to execute the test, the test environment, and the facilities supporting the testactivities, the testtools. The infrastructure is not an option but a necessity to correctly execute testactivities.
Three different test environments can be distinguished. A laboratory environment to perform early whitebox tests such as an algorithm test. A bit more controllable is the systemtest environment which is primarily for blackbox testing. The most advanced and situationally accurate environment is the production environment in which the acceptancy tests are conducted.
For testtools there is also a distinction. They are distinguished into the activities they support. The activities are themselves categorised into the Tmap phases. For the execution phase for example, there are the following activities:
- Testdata generator: tools for supporting the construction of physical testdata
- Debugger: tools for finding hard-to-find errors
- Simulators: tools mimicing a realisting environment for the tests to be executed in
- Stubs and drivers: Replacements of non-functional components of a system
[edit] Techniques
The Tmap method is supported by a large number of techniques. These techniques offer the testers universal ways of performing activities and offer management the opportunity to evaluate the testprocess as a whole. A definition is “A testtechnique is a collection of actions to produce, in a universal way, a testproduct” [Testen volgens TMap]. Further on the TPA will be elaborated on. This Test Point Analysis technique is a way to calculate the manhours required to perform all the planned testing. To give examples of testtechniques, the algorithm test will also be elaborated on, but it is placed in a separate entry.
[edit] Algorithm Test
The algorithm test is elaborated on in a separate wikipedia entry.
[edit] Test Point Analysis (TPA)
Another important technique in the Tmap arsenal is the Test Point Analysis or TPA. This is one of the most, if not the most important technique available because it deals with the calculation of the manhours needed for the overall test process. This gives information about what the testprocess costs and in what timespan it can be achieved which reflects in the planning.
The TPA procedure is illustrated below. The number of test points necessary for testing the dynamic measurable quality characteristics is calculated for each function on the basis of the number of function points assigned to the function, the function-dependent factors (complexity, interfacing, uniformity, user-importance and usage-intensity) and the quality requirements regarding or test strategy for the dynamic quality characteristics. The sum of the test points assigned to the individual functions is the number of “dynamic test points”. The number of test points necessary for testing the static measurable quality characteristics is calculated on the basis of the total number of function points for the information system and the quality or test strategy for the static quality characteristics. This gives the number of static test points.
The total number of test point is the sum of the dynamic and static test points. The number of primary test hours can then be calculated by multiplying the total number of test points by the calculated environmental factor and the applicable productivity factor. The primary test hours represents the volume of work involved in the primary testing activities, i.e. the time required for completion of the TMap phases Preparation, Specification, Execution and Completion.
Finally, the total number of test hours is obtained by adding an allowance for secondary test activities (Planning and Control) to the primary number of test hours. The size of this allowance, which represents the volume of work involved in the management activities, depends on the size of the test team and the availability of management tools. The total number of test hours is an estimate of the time required for all test activities, excluding formulation of the test plan.
Note: derived from: “Test Point Analysis, a method for test estimation” by Erik van Veenendaal
[edit] References
- Martin Pol, Ruud Teunissen, Erik van Veenendaal Testen volgens TMap®, 2e druk, 2000, ISBN 90-72194-58-6
- Erik van Veenendaal The testing Practitioner, 2002, ISBN 90-72194-65-9
- Tim Koomen, Rob Baarda TMap® Test Topics, 2004, ISBN 90-72194-70-5
- Ton Dekkers, John Kammelar “A Functional Sizing Meta-Model”,12p
- Erik van Veenendaal, J. Trienekens, 1997, published in “Reliability, Quality and Safety of Software-intensive systems” “Testing Based on Users’ Quality Needs”, 14p
