Testing Biometric Systems

 

 

 

There are several aspects of biometric system testing that must be addressed:

·         The acceptance of the biometric feature capture method by the population using the system.  For example, does the majority of the user population accept or reject having their fingerprints captured on a glass surface?  Or, what is the acceptance of retinal scan that involves projecting a laser light through the pupil of the eye to illuminate the retina? 

In some applications, population acceptance may not be an issue; for instance, the possibility that persons under arrest may object to having their finger placed on a glass surface may be of no consequence.

·         The repeatability of the biometric scan.  If only slight variations in the nearby environment significantly change subsequent scans, the biometric could be useless for comparison against an existing “target” database of previous scans.  The aspect of biometric scan repeatability must also be measured in terms of repeatability over time.  Does the biometric scan of a person match the same individual one, two, or more years later?

·         The discrimination of biometric data between any two individuals.  If the biometric being measured does not differ in any measurable way between different individuals, the biometric is useless for most purposes.  The concept of “discrimination” can be expanded to include the notion of the amount of data that is obtained by a biometric scan.  If the range of data content for a biometric measurement is very low, there will be a limited number of unique data “states,” or possibilities; in this case, the “target” database of previous scans is limited to a low number and, once this number is reached, the next individual added to the database will effectively duplicate an already-enrolled individual.

·         The practicality of the biometric capture method.  If obtaining a biometric measurement is overly complicated or cumbersome, the system will not find favor with the users and will be quickly dropped from use.

·         The throughput capability of the system must meet the needs of the user.  If a biometric is perfect in every other aspect, but the system can only perform one biometric-based match per hour whereas you need to do 100 per hour, then the biometric is essentially useless for your purposes.

·         Can the result of the biometric comparison be subsequently validated by a person (or another – near perfect -- algorithm) with very little or no chance for error?  If there is no way for the user to know if the system is consistently producing correct answers or to know how often and under what circumstances the system faults, then operation of the system will be risky. 

However, there may be applications where a consistent error in one direction (either false acceptance or false rejection) is permissible and even preferred.  For instance, entry into a very high security area may require a biometric system that, when it faults, consistently faults toward false rejection.  On the other hand, a door lock for a health club may not want its members to be rejected on scans that just miss a “certain hit” threshold; in this case, perhaps the system will be set to fail on the side of higher false acceptance.

·         The response time of the biometric comparison (search) – measured from the time the biometric is scanned or “captured” to the time the comparison result is returned (including a validation step, if required) – is an important consideration when designing or selecting a biometric system.  If the response time of the system too high, the system may be of no use.

·         Cost effectiveness of the biometric system is always an important consideration.  Can the design goal of a biometric system be met for less money by installing a low technology solution?  For instance, is the cost of paying for armed guards at high-security entry points less than the cost of a fully installed system that is just as effective as the armed personnel?

·         Finally, the accuracy of the biometric system is an important – and highly controversial – aspect of selecting a “correct” biometric system.  From our experience, it would seem that system accuracy is sometimes the only metric by which a biometric system is selected.  To compound the problem, accuracy tests are usually not completely thought out (designed) properly and the results are not interpreted correctly – or fairly.  For instance, given a test consisting of 25 “tests” or “samples” with vendor A matching 21 of 25 and vendor B matching 17 of 25, is vendor A significantly more accurate than vendor B?  The answer is that both vendors are essentially equal in accuracy.  For an explanation of how this is so (i.e., both vendors have essentially equal accuracy), click on the following links:

 

How Accurate is the Biometric? 

Testing is expensive in terms of money, time, and resources.  On the other hand, the test must be rigorous enough to yield a very close approximation of the inherent matching capabilities of the biometric system in question

Sample Test Plan

This test plan was proposed for a former client.  The objective of the test was twofold; determine the inherent accuracy of the system, and, develop a set of “samples” – a “test set” – that could be used in future accuracy tests of the system as the systems database(s) grew in size.