Assuring Quality in the Crime Laboratory

Assuring Quality

in the Crime Laboratory

William J. Tilstone,

h

1. G

R

At the time of writing this chapter, forensic science is under considerable pressure from scientific peers, the legal community, and the public for what they regard as poor standards of practice. The reality is much different.

Laboratories process hundreds of thousands of cases each year in a timely manner and without errors of fact or interpretation. This is achieved in laboratories because of the application of resources and personal commitment to quality assurance. The aim of this chapter is to give the nonforensic scientist an insight into what is involved in assuring quality in the crime laboratory.

1.1. Quality Assurance Standards

Quality assurance (QA) standards have never been higher than they are currently. There are two main reasons why: (1) the resources that laboratories are devoting to QA and (2) the recognition that although analysts all have a personal responsibility for the quality of their work, the best assurance comes from implementation of effective systems.

QA can add as much as 25% to the direct costs of running a laboratory.

Even today some laboratory administrations and analysts balk at the cost and try to minimize the resources given to the area. Here is an example of the argument against QA costs that usually ensues: the laboratory employs capable and experienced staff who know what they are doing; their job is to apply their

From:The Forensic Laboratory Handbook: Procedures and Practice Edited by: A. Mozayani and C. Noziglia © Humana Press Inc., Totowa, NJ

211

judgment and practical skills to each case, and everything will be fine. This is the personal standards approach to quality. Unfortunately, experience shows that the personal expert approach is not effective. Equally unfortunate are the errors that are made. The expectations that the public and the justice system place on foren- sic science are so high that these have been headline grabbers. Most of them share a common thread: errors result from misjudgment made by an individual analyst working independently of a quality system. In other words, the personal expert approach does not provide a sufficient assurance of quality. Not only that, but the cost of the resulting quality failure is greater than the cost of the alter- native quality systems standards approach. Judicial reviews, resources diverted to review of errors, compensation payments to and erroneous incarceration of wrongly convicted individuals, and the disruption of response to the need for improvement all are very expensive in dollars. Effective QA in forensic science is not only a necessity, but also is a worthwhile investment.

2. H

G

I

R

The goal of every forensic scientist is to get the right answer every time. The expectation of every user is that all testing is error free. In the context of testing,

“right” means the correct identification in qualitative analyses and the correct amount in quantitative analyses. This is reasonably straightforward, but incom- plete. “Right” in forensic science means making a correct interpretation of results in the context of the case. The circumstances of the case also become factors when deciding what is the “right” test or range of tests to apply to which sample or samples. Nobody knows what really happened in the case. The forensic scientist is expected to seek the truth in the middle of an extremely partisan contest. One side, the prosecution, believes it has found the perpetrator and is trying to wring out every piece of evidence to support its case. Meanwhile, the other side, the defense, has a responsibility to represent the interest of the defendant and seeks to minimize the impact of the laboratory testing by any means it can. “Right” can become contextually dependent when the case circumstances are to be considered.

The approach of this chapter is to concentrate on those objective areas that are within the control of the analyst. The more subjective case-dependent issues are legitimate and must be addressed by the laboratory through effective case management, but that is another topic.

2.1. Variation in Testing

One of the first things that an analyst learns is that there is intrinsic variability in all testing, especially quantitative analyses. How then can we add “every time”

to “right?” The answer is that if we understand, identify, and measure the sources

of variation and their consequences, we can make meaningful decisions about the reliability of our testing and show that it is producing results that are sufficiently reliable to meet the “right every time” requirement.

Therefore, the objectives of QA are to:

• Identify the sources of variation.

• Measure them.

• Minimize and control them.

The process of quality management is identifying, measuring, minimizing, and controlling variation in order to produce reliable results fit for the intended purpose. I will now discuss the sources and control of variation in testing, and identify how the QA system in the laboratory can minimize variation to an acceptable level.

2.2. Sources of Variation in Testing

The first area to be considered is how variation can arise. There are three main sources of variation in testing: the physical environment, variation in sampling, and factors resulting from the testing itself.

2.2.1. Physical Environment

The physical environment contributes variation in many ways. At a micro- environmental level, temperature control can affect the stability of materials.

Accreditation standards generally require that the laboratory provide storage facilities that will protect against degradation. These standards are typically aimed at temperature control through cold rooms and local refrigerators or freezers. Humidity control is also important. Poor climate control at the room temperature level has been shown to introduce variability in DNA testing using capillary electrophoresis units. Poor lighting—low levels or the wrong color temperature is unacceptable in many areas of forensic science testing, such as searching of evidence, comparison of colors, and document examination.

Physical accommodation has a substantial indirect influence on the

reliability of testing. A forensic science laboratory needs to have a design that

prevents contamination. The relative layout of work areas must be designed

to ensure separation of processing of known and unknown, as well as separation

of storage of evidence from victim and accused, and separation of testing of

trace drug levels in toxicology from bulk levels in controlled substance

examination. The area of space available for work also contributes at least

the potential for contamination. There are various guidelines but in practice the

minimum realistic target should be in the range of 500–750 square feet per

analyst, with an absolute minimum acceptable level of 400.

2.2.2. Sampling

Sampling is generally accepted as the main source of variation in testing.

Some authorities estimate that collection can account for two-thirds of total variation in testing, and preparation of samples for analysis can contribute a further 20%. Sampling in forensic science repeatedly arises as a significant factor. In toxicology, the concentration of drug in a postmortem blood sample can vary depending on the site from which it was taken. On the other hand, multiple-site sampling is recommended for alcohol analysis where postmortem ethanol production may be a factor. In controlled substance cases, sampling from a bulk seizure of powder or taking a representative sample of plants from a cannabis plantation present sampling challenges to the analyst. In biology, there is always a sampling issue whenever there are multiple blood stains at a scene or on clothing.

In all cases, the rule is to take all reasonable steps to ensure that the results obtained from the testing give a sufficiently accurate and reliable account of the true value. Tools, such as coning and quartering for powders, sample sizes of the square root of the number of original items, and statistically based random sampling, can be used. One thing to avoid is basing estimates of variation on replicate testing of a composite, as valuable data regarding variation between samples will be lost.

2.2.3. Test Procedures

Test procedures are the third source of variation. The method used, the correct calibration and functioning of the test equipment, and the competency of the analyst all contribute to the reliability of the actual testing.

There are many sources of methods. In general there is a hierarchy of reliability. Official and consensus-based standard methods, such as those publi- shed by American Society of Testing Materials (ASTM), are best. Use of meth- ods developed in house is acceptable provided that they have been thoroughly validated. The typical matrix encountered, and the expected range of analyte in it, influence the selection and performance of the method. The laboratory should be able to demonstrate that the methods it uses satisfy the user require- ments for variation. It can do this by simple verification of performance for offi- cial published methods or by more extensive performance checks for in house ones.

Validations should include the use of certified reference materials (if they

exist) and some form of interlaboratory check, such as external proficiency

samples. Analyst competency goes hand in hand with validation. The laboratory

should have records that demonstrate that the analysts using the technique are

trained and have successfully completed some form of competency test.

2.3. Measuring Variation

The performance of each quantitative test can be described in terms of its accuracy and precision. Accuracy is how close the test answer is to the true con- centration of analyte in the sample. Precision is a measure of the dispersion of repeat analyses. Accuracy and precision come together in situations, such as:

• Quantitation of controlled substance in a seized sample in a jurisdiction where sentencing depends on the amount of drug,

• Measurement of serum concentrations of drug in toxicology and reference to published data on therapeutic, toxic, and lethal levels, and

• In blood or breath alcohol testing where the offense is defined by the concentra- tion of ethanol in the biological sample.

In the first and third examples, the test must be capable of producing an accuracy and precision such that the legal decisions based on it are dependable beyond reasonable doubt. In the case of the toxicology assay, both the legal demands and the actual capability of the test are less stringent.

Accuracy is typically measured by spiking blank matrix and analyzing the product. It is also addressed thorough comparison of results in interlaboratory collaborative trials and comparison of results to those obtained by a reference method of known performance. Precision is measured by comparing the spread of repeat analyses. The repetitions can cover different analysts working on dif- ferent equipment sets on different days.

Accuracy is usually expressed in terms of the mean of several test results compared to the known or target concentration in the samples tested. Precision is measured by the variance of the set of results about that mean. Variance has a valuable property in QA, in that the total variance of testing is the sum of the variances contributed by each step. In practice, the spread of results is often expressed as the standard deviation, which is the square root of the variance.

Sometimes the standard deviation is reported as a percentage of the mean, which is called the coefficient of variation. These various statistical parameters can be used to answer questions of reliability, for example approximately two- thirds of all results will lie within the mean plus or minus 1 standard deviation.

The reliability of the estimate of the mean can be described from its standard error and tables of the t-statistic. The advantage of this approach is that it permits an estimate of the effect of repeats on the validity of the mean.

Qualitative testing is more straightforward. The basic principles of stan-

dard methods, validation, use of reference materials, effect of matrix, analyst

competency, and participation in collaborative trials are the same. However, the

only variable to be considered is that of incidence of false identifications, either

false-positive or false-negative.

2.4. Uncertainty of Measurement

V ariation in testing is a complex and specialized discipline in itself.

Practitioners make use of the term “uncertainty of measurement” (UM) to describe the intrinsic and inevitable variation that exists in any test procedure.

The tools described previously, such as direct measurement of the standard deviation from repeat tests, are used to quantify UM. The goal is that the UM of a test is known and can be compared to the variation that is acceptable to the user in the application of the test results. Thus, the UM for a blood alcohol test, where the result not only can determine guilt or innocence, but also can affect someone’s livelihood, has to be sufficiently small in order that the court be confident that a reported level above the statutory limit is indeed valid. In contrast, the UM for a low blood level of a sedative drug found at an autopsy of someone who has died of a heroin overdose can (and will) be considerable. The law has tried to address the same issue, for example the Supreme Court deci- sion in Daubert v. Merrell Dow Pharmaceuticals, Inc., (Daubert v. Merrell Dow 509 US 579) requires something it calls “error rate” to be considered as a fac- tor in determining the admissibility of scientific evidence in federal courts. The factor that the court intended to be considered is UM. The use of the expression

“error rate” is extremely misleading, implying that the variability arises from mistakes and failing to appreciate that the variability is intrinsic.

3. C

V

Now that the context of QA in forensic science has been set, we can con- sider how to make QA happen. A description of the essential elements of a QA system is discussed in the following sections.

3.1. The Quality System

A quality system is the organizational structure, procedures, processes, and resources needed to implement quality management. There is a hierarchy within the quality system. The highest level consists of the quality manual, which describes the quality system. Next, are procedures, which are the activities needed to implement the elements detailed in the quality manual. Last, the laboratory will have documented instructions, which are detailed work documents.

Quality assurance is all the planned and systematic actions necessary to pro-

vide adequate confidence that a product or service will satisfy given requirements

for quality. The planned and systematic actions are specified in the laboratory

quality manual and standard operating procedures. The quality manual is the

formal document that describes the organization’s quality system. The manual

documents all policies, systems, programs, procedures, and instructions to the extent necessary to assure the quality of the test results.

Quality control is the operational techniques and activities that are used to fulfill requirements for quality.

3.2. The Quality Manual

The purpose of the quality manual is straightforward. However, many laboratories embarking on documentation of their quality system for the first time find it extremely difficult to actually write the manual. There are some easy-to-follow guidelines that will ensure success.

First, the stimulus is often preparation for accreditation. The accreditation program should detail what it requires to be in the manual. It is a good idea to use the specified areas as section titles. Even if you are not preparing for accreditation, the requirements from a program can still be used as a template.

Using the clauses in International Organization for Standardization 17025 in this way would result in a manual with 25 main sections, as shown in Table 1.

This then would be the skeleton of the manual. How do we put flesh on its bones? There are three simple rules for a quality system and its documentation:

• If you do it, write it.

• If you write it, do it.

• If it isn’t written, it didn’t happen.

Every laboratory will have a way of doing things that work, more or less. The main problem is that the way is often enshrined in oral or informal traditions that depend on personnel knowing, retaining, and implementing the traditions. The best way to begin preparing a quality manual is to capture what actually happens; take the time to record the actual daily activities.

Then they can be edited into a suitable format and reviewed for compliance with the requirements of the agency or accreditation program. This is a much more effective way than borrowing the quality manual from another agency, writing in your own name in place of the original, and expecting effective implementation.

Once you have a manual, you must follow it. If you do not, you will not achieve accreditation because you will be out of compliance. You will have wasted the time it took to prepare the manual and will have denied yourself a valuable tool. It is vital to understand at the outset that the manual is a living document. If it turns out that there is a better way to do something than detailed in the manual, change the manual.

The only way to collect objective information about the performance of the

quality system is to record events. If an assay depends on the temperature when

a critical step is conducted, that should be specified in the relevant part of the quality system documentation. Having identified the critical factor, it makes sense to measure it and show that the assay conditions are within the acceptable limits when the testing was conducted. If you consistently find that conditions are not being met, then you know to proceed to improve the reliability of the control. This cannot be done without records.

The format of each section should address these factors:

• Policy or reference to policy, which defines the governing requirement.

• Purpose and scope, i.e., why, what for, area covered, and exclusions.

• Responsibility.

Table 1

Quality Manual Structure

Chapter Title

1

Organizational Structure Roles and Responsibilities

2 Policies and Procedures to Establish, Implement, and Maintain the Quality System

3 Document Control

4 Review of Requests, Tenders, and Contracts

5 Subcontracting

6 Purchasing Services and Supplies

7 Service to Client

8 Complaints

9 Control of Nonconforming Work

10 Corrective Action

11 Preventative Action

12 Control of Records

13 Internal Audits

14 Management Reviews

15 Competency of Personnel

16 Accommodation and Environmental Conditions

17 Method Validation

18 Uncertainty of Measurement

19 Control of Data

20 Equipment

21 Measurement Traceability

22 Sampling

23 Handling of Test Items

24 Assuring Quality

25 Reporting the Results

•

Actions and methods in the form of a step-by-step list of what needs to be done.

• Documentation and references, including a listing of any documents or forms required.

• Identification of what records are generated using the procedure and description of their retention.

3.3. Accreditation

As mentioned previously, accreditation should be used as guidance for preparing the quality manual. Accreditation is a powerful QA tool because it is the step that ensures that standards are in fact put into practice.

First, accreditation requires defined standards of performance, compe- tence, and professionalism. These may be consensus practitioner standards, e.g., those developed for DNA and controlled substance testing through the Scientific Working Group on DNA Analysis Methods and the Scientific Working Group for forensic drug testing methods, respectively. They may be more broad-based peer programs, such as the American Society of Crime Laboratory Directors/Laboratory accreditation program, or they may be formal international consensus standards, such as International Organization for Standardization/IEC 17025 “General requirements for the competence of testing and calibration laboratories.”

Second, accreditation requires that the laboratory demonstrates compli- ance with the standards through a third party review of its performance—self assessment is not acceptable. The value of the external review is strengthened by the requirement that it is conducted by an impartial and competent authority.

3.4. Objective Tests

The International Laboratory Accreditation Cooperation Guide (ILAC) 19 introduces a powerful concept in regard to the reliability of testing in forensic science, namely the objective test. An objective test has been documented and validated and is under control so that it can be demonstrated that all appropriately trained staff will obtain the same results within defined limits. Objective tests are controlled by:

• Documentation of the test.

• Validation of the test.

• Training and authorization of staff.

• Maintenance of equipment.

• Calibration of equipment.

• Use of appropriate reference materials.

• Provision of guidance for interpretation.

• Checking of results.

• Testing of staff proficiency.

• Recording of equipment/test performance.

This list will closely resemble any chapter on QA in forensic science. It provides a road map for ensuring quality and it is one that can be applied to all areas of the discipline: from toxicology to document examination and from the crime laboratory to the environmental testing one.

G

Accreditation:

The formal assessment and recognition by an impartial com- petent authority that a laboratory is capable of meeting and maintaining defined standards of performance, competence, and professionalism.

Accuracy: The closeness of a test measurement to the true value.

Sometimes represented by the Greek character

Calibrate: To standardize by determining the deviation from standard, especially so as to ascertain the proper correction factors.

Competency: The quality or state of being functionally adequate or of hav- ing sufficient knowledge, judgment, skill, or strength.

Competency test: A test to establish the sufficiency of knowledge, judgment, or skill of an analyst in a specified field.

Contamination: The action by something external to an object which, by entering into or coming in contact with the object, destroys its purity.

Error rates: The term used (erroneously) by the Supreme Court in Daubert for uncertainty of measurement.

IEC: International Electrotechnical Commission. See ISO.

ILAC: The International Laboratory Accreditation Cooperation. A voluntary organization representing stakeholders with an interest in accreditation. Among other things, ILAC pre- pares consensus guides assisting implementation of International Standards in accreditation for testing and cal- ibration laboratories. ILAC Guide 19 refers to forensic testing.

ISO: The International Organization for Standardization. ISO publishes Standards and Guides covering a wide range of topics, including the accreditation of testing and calibration laboratories.

Nonstandard: A method that has not been adopted as a consensus standard method through evaluation and publication by a reputable standards body, such as the ASTM.

Objective test: A test which, having been documented and validated, is

under control so that it can be demonstrated in order for all

appropriately trained staff to obtain the same results within

defined limits.

Precision:

The agreement or repeatability of a set of replicate results among themselves or the agreement among repeated obser- vations made under the same conditions.

Proficiency test: The use of interlaboratory comparisons of results from a number of laboratories to determine laboratory testing performance.

Quality assurance (QA): All the planned and systematic actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality.

Quality control (QC): The operational techniques and activities that are used to fulfill requirements for quality.

Quality manual: A written document or documents that identify the policy, organization, objectives, functional activities, and specific quality assurance activities designed to achieve the quality goals set for the operation of the laboratory.

Quality system: The organizational structure, procedures, processes, and resources needed to implement quality management.

Reference: A material or substance, one or more properties of which are Material (RM): sufficiently well established for it to be used for the calibration of an apparatus, the assessment of a mea- suring method, or for assigning values to materials. A Certified Reference Material (CRM) is a reference material of the highest metrological quality available that has values certified by a technically valid procedure, accompanied by or traceable to a certificate issued by the certifying body.

Sample: A sample is a set of data obtained from a population.

Standard deviation (SD): The square root of the sample variance.

Standard methods: Standard consensus methods are those developed by recog- nized organizations using collaborative studies involving a number of laboratories and then published.

Technical records: Accumulation of data and information which result from the performing of tests and/or calibrations and which indicate whether specified quality or process parameters are achieved.

Test report: An accurate and unambiguous presentation of test results and all relevant information in a format agreed upon by the laboratory and its clients.

Uncertainty of: A parameter associated with the result of a measurement that measurement (UM): characterizes the dispersion of val- ues that could reasonably be attributed to the measurand.

Validation: The determination of the statistical parameters of a method to demonstrate that it is fit for a specified purpose.

Variance: The sum of the squares of the differences between individual

values of a set and the arithmetic mean of the set, divided by

one less than the number of values.

S

R

Wenclawiak BW, Koch M, Hadjicostas E, Hadjicostas E, eds. Quality Assurance in

Analytical Chemistry: Training and Teaching. Berlin, Heidelberg: Springer Verlag, 2004.

Dux JP. Handbook of Quality Assurance for the Analytical Chemistry Laboratory, 2nd Edition. New York, NY: Van Nostrand Reinhold, 1998.

Garfield FM, Klesta E, and Hirsch J. Quality Assurance Principles for Analytical

Laboratories, 3rd Edition. Gaithersburg, MD, AOAC International, 2000.