Testing
Biometric Systems
There
are several aspects of biometric system testing that must be addressed:
·
The
acceptance of the biometric feature capture
method by the population using the system.
For example, does the majority of the user population accept or reject
having their fingerprints captured on a glass surface? Or, what is the acceptance of retinal scan
that involves projecting a laser light through the pupil of the eye to
illuminate the retina?
In some applications, population
acceptance may not be an issue; for instance, the possibility that persons
under arrest may object to having their finger placed on a glass surface may be
of no consequence.
·
The
repeatability of the biometric scan.
If only slight variations in the nearby environment significantly change
subsequent scans, the biometric could be useless for comparison against an
existing “target” database of previous scans.
The aspect of biometric scan repeatability must also be measured in
terms of repeatability over time.
Does the biometric scan of a person match the same individual one, two,
or more years later?
·
The
discrimination of biometric data between any two individuals. If the biometric being measured does not
differ in any measurable way between different individuals, the biometric is
useless for most purposes. The concept
of “discrimination” can be expanded to include the notion of the amount of data
that is obtained by a biometric scan. If
the range of data content for a biometric measurement is very low, there will
be a limited number of unique data “states,” or possibilities; in this case,
the “target” database of previous scans is limited to a low number and, once
this number is reached, the next individual added to the database will
effectively duplicate an already-enrolled individual.
·
The
practicality of the biometric capture method. If obtaining a biometric measurement is
overly complicated or cumbersome, the system will not find favor with the users
and will be quickly dropped from use.
·
The
throughput capability of the system must meet the needs of the
user. If a biometric is perfect in every
other aspect, but the system can only perform one biometric-based match per
hour whereas you need to do 100 per hour, then the biometric is essentially
useless for your purposes.
·
Can
the result of the biometric comparison be subsequently validated by a
person (or another – near perfect -- algorithm) with very little or no chance
for error? If there is no way for the
user to know if the system is consistently producing correct answers or to know
how often and under what circumstances the system faults, then operation of the
system will be risky.
However, there may be applications
where a consistent error in one direction (either false acceptance or false
rejection) is permissible and even preferred.
For instance, entry into a very high security area may require a
biometric system that, when it faults, consistently faults toward false
rejection. On the other hand, a door lock
for a health club may not want its members to be rejected on scans that just
miss a “certain hit” threshold; in this case, perhaps the system will be set to
fail on the side of higher false acceptance.
·
The
response time of the biometric comparison (search) – measured from the
time the biometric is scanned or “captured” to the time the comparison result
is returned (including a validation step, if required) – is an important
consideration when designing or selecting a biometric system. If the response time of the system too high,
the system may be of no use.
·
Cost effectiveness of the biometric system is always an important consideration. Can the design goal of a biometric system be
met for less money by installing a low technology solution? For instance, is the cost of paying for armed
guards at high-security entry points less than the cost of a fully installed
system that is just as effective as the armed personnel?
·
Finally,
the accuracy of the biometric system is an important – and highly
controversial – aspect of selecting a “correct” biometric system. From our experience, it would seem that
system accuracy is sometimes the only metric by which a biometric system
is selected. To compound the problem,
accuracy tests are usually not completely thought out (designed) properly and
the results are not interpreted correctly – or fairly. For instance, given a test consisting of 25
“tests” or “samples” with vendor A matching 21 of 25 and vendor B matching 17
of 25, is vendor A significantly more accurate than vendor B? The answer is that both vendors are
essentially equal in accuracy. For an
explanation of how this is so (i.e., both vendors have essentially equal
accuracy), click on the following links:
|
Testing is expensive in terms of money, time, and
resources. On the other hand, the test must be rigorous enough to yield
a very close approximation of the inherent matching capabilities of the
biometric system in question |
|
|
This
test plan was proposed for a former client.
The objective of the test was twofold; determine the inherent accuracy
of the system, and, develop a set of “samples” – a “test set” – that could be
used in future accuracy tests of the system as the systems database(s) grew
in size. |