Accuracy Test Scenario

 

Description of Accuracy Tests

AFIS match accuracy will be measured several times before Final Acceptance of the system. The first system matching accuracy measurement will be made during the Benchmark Accuracy Test. The Benchmark Accuracy Test will be conducted within two (2) months of final Contract signature by the State.

1. Benchmark Accuracy Test

a. [Mandatory] The Bidder must conduct a system accuracy test against the converted Background dB database (see Section I, Key Action Dates, for schedule). The test will be based on a minimum of 500 two-finger records searched against a corresponding set of 500 two-finger records. The Bidder is responsible for supplying the test system and for development of the 500 sets of test images (target and search images). The Bidder must also load 100,000 records (Background dB) onto the background database. The minimum acceptable search accuracy rate achieved during the Benchmark Accuracy Test is 96% (two-finger OPEN searches against corresponding two-finger records). All searches conducted during the Benchmark Accuracy Test will be run against the entire database of Background dB records used as the background database plus the target records loaded by the Bidder; i.e., every search record will be matched against every record in the test database (background records and target records). The Bidder must demonstrate that the Background dB records used as the background records for the test cannot be distinguished from the target records added by the Bidder.

Note: The accuracy rate achieved during the Benchmark Accuracy Test will only be used to assure the State that the systemís matching subsystem is functional prior to initial implementation of the system. Passing the Benchmark Accuracy Test accuracy tests does not mean that the Bidder has fulfilled the accuracy requirements required for the installed system.

b. [Mandatory] The Bidder is responsible for converting the Background dB database and loading the AFIS database with fingerprint images that will be used in the Benchmark Accuracy Test.

2. Factory Acceptance Test

a. [Mandatory] During the Factory Acceptance Test the Bidder must conduct system throughput and accuracy tests against the converted Background dB database of two-finger records. The test will be based on a minimum of 500 two-finger records searched against a corresponding set of 500 two-finger records. The Bidder is responsible for supplying the test system and for development of the 500 sets of test images (target and search images). The minimum acceptable search accuracy rate achieved during the Factory Acceptance Test is 96% (two-finger OPEN searches against corresponding two-finger records). All searches conducted during the Factory Acceptance Test will be run against the entire database of Background dB records used as the background database plus the target records loaded by the Bidder; i.e., every search record will be matched against every record in the test database (background records and target records). The Bidder must demonstrate that the Background dB records used as the background records for the test cannot be distinguished from the target records added by the Bidder.

Note:. The accuracy rate achieved during the Factory Acceptance Test will only be used to assure the State that the systemís matching subsystem is functional prior to initial implementation of the system. Passing the Factory Acceptance Test accuracy tests does not mean that the Bidder has fulfilled the accuracy requirements required for the installed system.

b. [Mandatory] The Bidder is responsible for converting and loading the AFIS database with fingerprint images that will be used in the Factory Acceptance Test.

3. Description of System Accuracy Tests

A System Accuracy Test (SAT) will be performed periodically on the system to determine if the system meets the minimum accuracy levels required of the operational AFIS. The first SAT will be conducted immediately after all "Existing Caseload" records have been input to the AFIS (see Section IV, Table IV.E.1.).

The System Accuracy Test will be conducted with a test set of six thousand (6,000) records. Using a sample size of 6,000 records, a test having a 95% Level of Confidence will have the following Confidence Intervals;

      0.5% Confidence Interval for a system that has a predicted accuracy of 96% (Two Finger OPEN SEARCH);

      0.55% Confidence Interval for a system that has a predicted accuracy of 95% (Single Finger OPEN SEARCH);

      0.08% Confidence Interval for a system that has a predicted accuracy of 99.9% (Two Finger CLOSED SEARCH); and,

      0.25% Confidence Interval for a system that has a predicted accuracy of 99% (Single Finger CLOSED SEARCH).

 

System Accuracy Tests will be done in four parts (para. a -> d) as follows:

a. System Accuracy Test, Part 1

System Accuracy Test, Part 1 will be comprised of 6,000 two-finger OPEN searches against corresponding two-finger database records.

The first time a SAT is performed, 6,000 welfare recipients will be asked to return to their respective welfare offices to be re-fingerprinted and re-photographed (as per a "normal" search input transaction). Each of the welfare recipients asked to return for a fingerprint check will be individuals who have already been entered into the AFIS database and whose records have never been subsequently purged from the AFIS database.

After each of the 6,000 search records are saved locally at the Input Workstation for subsequent transmission to the Central Site system, the Input Workstation will create a copy of the search request transaction onto a high-density magnetic tape media. After all 6,000 search records are recorded onto magnetic media, all media will be collected by the State. The State will hold this data for use in future System Accuracy Tests.

When the search transaction is initiated at the Remote Input Workstation, each of the 6,000 search records will be given a special, unique CIN that has not been recorded in either the SAWS TA database or AFIS database; when the record is submitted to the Central Site for processing, the fact that a CIN "unknown" to the AFIS database has been assigned as the primary identifier for the search record will force the AFIS to perform an OPEN SEARCH on each of the 6,000 records (see Diagram IV.B.1., Proposed AFIS Workflow).

In addition to the special CIN that is assigned to the search record as the primary search identification number, the actual CIN of the welfare client (i.e., the CIN for the client that is currently in the AFIS database) will also be recorded within the search record in a secondary index number field. The actual CIN number (now recorded as a secondary number within the search record) will be used in the System Accuracy Test, Parts 3 & 4 (described below).

All 6,000 searches will be processed by the AFIS. At the conclusion of the search process for each of the 6,000 search requests (i.e., after verification of the OPEN SEARCH candidate list for each search or after the AFIS match response returns a "NO CANDIDATES FOUND" message), the AFIS will purge the transaction from the Central Site transaction queues. The search record must not be recorded in the AFIS database. In addition, fingerprint images from the search record must not be substituted for AFIS database images if the search record image is of higher quality than the AFIS database record (in the case of SAT search records, requirement VI.B.1.25. [Replacement of Fingerprint Images with Images of Higher Quality] must not be part of the transaction process).

After all 6,000 search requests have been processed, search results for all searches will be compiled and a Measured Accuracy rate will be determined by dividing the sum of the total number of searches performed minus the number of searches that resulted in a "No-Hit" (or "Miss" or "NO CANDIDATES FOUND") by the total number of searches performed and multiplying the result by 100 to arrive at a "Measured Accuracy."

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

The search accuracy rate for the System Accuracy Test, Part 1, must be within the range

(TFOS - Confidence Interval) £ Measured Accuracy £ (TFOS + Confidence Interval)

 

Where

TFOS =

Two Finger OPEN SEARCH accuracy rate specified in the Bidderís Bidder Qualification Response (see Section V.B.)

 

Confidence Interval =

0.5% (corresponding to a test with a 95% Level Of Confidence, a test size of 6,000 records, and a predicted accuracy rate of [at least] 96%)

 

Measured Accuracy =

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

Since the upper limit of the range is meaningless in this case, the acceptable range for the Two-Finger OPEN SEARCH accuracy rate is

TFOS - 0.5% £ Measured Accuracy £ 100%

Part 1 of the System Accuracy Test will be considered as PASSED if the Measured Accuracy of the Two-Finger OPEN SEARCH equals or exceeds the Two-Finger OPEN SEARCH accuracy defined in the Contractorís Bidder Qualification Response minus 0.5%.

For example, if, in their Bidder Qualification Response, the Bidder proposes a 97% Two-Finger OPEN SEARCH accuracy rate, then, in order to pass Part 1 of the System Accuracy Test, the Two-Finger OPEN SEARCH accuracy rate must be at least 96.5% (97.0% - 0.5%).

b. System Accuracy Test, Part 2

System Accuracy Test, Part 2 will be comprised of 6,000 single-finger OPEN searches against corresponding AFIS database records.

The test set recorded on magnetic tape during the first time that the System Accuracy Test, Part 1 is run will be used in the System Accuracy Test, Part 2. The Contractor will copy the data onto a second set of magnetic media; the Contractor will modify this second test set copy so that all search requests have one fingerprint image per search record. The images will be randomly selected (right hand or left hand) from the original set when constructing each single-finger test set record.

The CIN that was used for the System Accuracy Test, Part 1 test will also be used in the System Accuracy Test, Part 2; this will force the AFIS to perform an OPEN SEARCH on each of the 6,000 records (see Diagram IV.B.1., Proposed AFIS Workflow).

All 6,000 searches will be processed by the AFIS. At the conclusion of the search process for each of the 6,000 search requests (i.e., after verification of the OPEN SEARCH candidate list for each search or after the AFIS match response returns a "NO CANDIDATES FOUND" message), the AFIS will purge the transaction from the Central Site transaction queues. The search record must not be recorded in the AFIS database. In addition, fingerprint images from the search record must not be substituted for AFIS database images if the search record image is of higher quality than the AFIS database record (in the case of SAT search records, requirement VI.B.1.25. [Replacement of Fingerprint Images with Images of Higher Quality] must not be part of the transaction process).

After all 6,000 search requests have been processed, search results for all searches will be compiled and a Measured Accuracy rate will be determined by dividing the sum of the total number of searches performed minus the number of searches that resulted in a "No-Hit" (or "Miss" or "NO CANDIDATES FOUND") by the total number of searches performed and multiplying the result by 100 to arrive at a "Measured Accuracy."

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

The search accuracy rate for the System Accuracy Test, Part 2, must be within the range

(95% - Confidence Interval) £ Measured Accuracy £ (95% + Confidence Interval)

 

Where

Confidence Interval =

0.55% (corresponding to a test with a 95% Level Of Confidence, a test size of 6,000 records, and a predicted accuracy rate of [at least] 95%)

 

Measured Accuracy =

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

Since the upper limit of the range is meaningless in this case, the acceptable range for the Single-Finger OPEN SEARCH accuracy rate is

94.45% £ Measured Accuracy £ 100%

Part 2 of the System Accuracy Test will be considered as PASSED if the Measured Accuracy of the Single-Finger OPEN SEARCH equals or exceeds 94.45%.

c. System Accuracy Test, Part 3

System Accuracy Test, Part 3 will be comprised of 6,000 two-finger CLOSED searches against corresponding two-finger database records.

The test set recorded on magnetic tape during the first time that the System Accuracy Test, Part 1 is run will be used in the System Accuracy Test, Part 3. The Contractor will copy the data onto a third set of magnetic media. The Contractor will modify the third test set copy so that the CIN that was recorded within the (Part 1) search record as a secondary index number (i.e., the actual CIN of the welfare client that is currently in the AFIS database) will be recorded within the search record as the primary number for the search, thus replacing the "special" CIN used in Part 1.

When the each of the Part 3 test records are submitted to the Central Site for processing, the fact that the search record CIN is "known" to the AFIS database will force the AFIS to perform a CLOSED SEARCH on each of the 6,000 records (see Diagram IV.B.1., Proposed AFIS Workflow).

All 6,000 searches will be processed by the AFIS. At the conclusion of the search process for each of the 6,000 search requests (i.e., after verification of the CLOSED SEARCH), the AFIS will purge the transaction from the Central Site transaction queues. The search record must not be recorded in the AFIS database. In addition, fingerprint images from the search record must not be substituted for AFIS database images if the search record image is of higher quality than the AFIS database record (in the case of SAT search records, requirement VI.B.1.25. [Replacement of Fingerprint Images with Images of Higher Quality] must not be part of the transaction process).

After all 6,000 search requests have been processed, search results for all searches will be compiled and a Measured Accuracy rate will be determined by dividing the sum of the total number of searches performed minus the number of searches that resulted in a "No-Hit" (or "Miss" or "NO CANDIDATES FOUND") by the total number of searches performed and multiplying the result by 100 to arrive at a "Measured Accuracy."

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

The search accuracy rate for the System Accuracy Test, Part 3, must be within the range

(99.9% - Confidence Interval) £ Measured Accuracy £ (99.9% + Confidence Interval)

 

Where

Confidence Interval =

0.08% (corresponding to a test with a 95% Level Of Confidence, a test size of 6,000 records, and a predicted accuracy rate of [at least] 99.9%)

 

Measured Accuracy =

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

Since the upper limit of the range is meaningless in this case, the acceptable range for the Two-Finger CLOSED SEARCH accuracy rate is

99.82% £ Measured Accuracy £ 100%

Part 3 of the System Accuracy Test will be considered as PASSED if the Measured Accuracy of the Two-Finger CLOSED SEARCH equals or exceeds 99.82%.

 

d. System Accuracy Test, Part 4

System Accuracy Test, Part 4 will be comprised of 6,000 single-finger CLOSED searches against corresponding AFIS database records.

The test set recorded on magnetic tape for the System Accuracy Test, Part 3 will be used in the System Accuracy Test, Part 4. The Contractor will copy the data onto a fourth set of magnetic media; the Contractor will modify this fourth test set copy so that all search requests have one fingerprint image per search record; the images will be randomly selected (right hand or left hand) from the original set when constructing each single-finger test set record.

When the each of the Part 3 test records are submitted to the Central Site for processing, the fact that the search record CIN is "known" to the AFIS database will force the AFIS to perform a CLOSED SEARCH on each of the 6,000 records (see Diagram IV.B.1., Proposed AFIS Workflow).

All 6,000 searches will be processed by the AFIS. At the conclusion of the search process for each of the 6,000 search requests (i.e., after verification of the CLOSED SEARCH), the AFIS will purge the transaction from the Central Site transaction queues. The search record must not be recorded in the AFIS database. In addition, fingerprint images from the search record must not be substituted for AFIS database images if the search record image is of higher quality than the AFIS database record (in the case of SAT search records, requirement VI.B.1.25. (Replacement of Fingerprint Images with Images of Higher Quality) must not be part of the transaction process).

After all 6,000 search requests have been processed, search results for all searches will be compiled and a Measured Accuracy rate will be determined by dividing the sum of the total number of searches performed minus the number of searches that resulted in a "No-Hit" (or "Miss" or "NO CANDIDATES FOUND") by the total number of searches performed and multiplying the result by 100 to arrive at a "Measured Accuracy."

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

The search accuracy rate for the System Accuracy Test, Part 4, must be within the range

(99.0% - Confidence Interval) £ Measured Accuracy £ (99.0% + Confidence Interval)

 

Where

Confidence Interval =

0.25% (corresponding to a test with a 95% Level Of Confidence, a test size of 6,000 records, and a predicted accuracy rate of [at least] 99.0%)

 

Measured Accuracy =

(total number of searches - total number of "No-Hits") / (total number of searches) x 100 = Measured Accuracy (expressed as a percentage)

Since the upper limit of the range is meaningless in this case, the acceptable range for the Two-Finger CLOSED SEARCH accuracy rate is

98.75% £ Measured Accuracy £ 100%

Part 4 of the System Accuracy Test will be considered as PASSED if the Measured Accuracy of the Single-Finger CLOSED SEARCH equals or exceeds 98.75%.

4. System Accuracy Test Requirements

a. [Mandatory] The minimum acceptable search accuracy rate achieved during a System Accuracy Test Part 1 (two-finger OPEN searches against corresponding AFIS two-finger database records) is as specified in requirements VI.G.2.3.a. and VI.G.1.1.

b. [Mandatory] The minimum acceptable search accuracy rate achieved during a System Accuracy Test Part 2 (single-finger OPEN searches against corresponding AFIS database records) is as specified in requirements VI.G.2.3.b. and VI.G.1.2.

c. [Mandatory] The minimum acceptable search accuracy rate achieved during a System Accuracy Test Part 3 (two-finger CLOSED searches against corresponding AFIS two-finger database records) is as specified in requirements VI.G.2.3.c. and VI.G.1.3.

d. [Mandatory] The minimum acceptable search accuracy rate achieved during a System Accuracy Test Part 4 (single-finger CLOSED searches against corresponding AFIS database records) is as specified in requirements VI.G.2.3.d. and VI.G.1.4.

e. [Mandatory] A complete System Accuracy Test must be conducted during the Final Acceptance Test, Part 2, after all existing client records have been "converted." In addition, State reserves the right to conduct additional System Accuracy Tests at any time during the life of the contract; in this event, the State will give notice of itís intent to conduct the test at least fifteen (15) working days prior to the start of the test.

f. [Mandatory] Test data used in each System Accuracy Test will be image data collected at certain AFIS workstations (designated by the State). The search test set used in the System Accuracy Test must be no smaller than 6,000 sets of two (2) finger data records.

g. After the System Accuracy Test has been run, all "No-Hit" results will be examined. For each test "No-Hit," the State may elect to throw the result out of consideration if, in the sole opinion of the State, that particular test result is not a fair test of the system. Search sets that are thrown out as a result of this consideration will not be counted in the overall totals for the SAT; however, in each instance, at the sole discretion of the State another set of data may be substituted for the invalidated set of data. RFP requirement VI.C.1.13. shall apply to the determination of system accuracy scores after all test results have been examined.

 h. [Mandatory] The Contractor must save all search test set images on to a high-capacity storage medium at the Remote Input Workstation. All images must be stored as "raw" (i.e., full resolution, uncompressed) images; in addition, the Contractor may elect to store the compressed images and corresponding minutiae sets for each record. The State will store the test set images for subsequent testing; this will obviate the need to recall existing health and welfare clients into the County Welfare Office each time a System Accuracy Test must be run.

i. [Mandatory] After the first System Accuracy Test has been run, the State intends to keep the high-capacity storage medium containing test records in itís possession. The Bidder must propose a method to enable a batch launch of test searches from the high-capacity storage medium (containing the test data captured during the first System Acceptance Test) during subsequent System Accuracy Tests.

j. [Mandatory] The minimum set of data returned from each System Accuracy Test search is

                                                                                                                                             i.            Search and Target Record Numbers,

                                                                                                                                           ii.            "Match," "No-Match" indication for each search,

                                                                                                                                          iii.            demographics used to limit search extent (list demographics of search record as well as demographics of candidate record);

                                                                                                                                          iv.            "quality" scores of the search and target fingers (if the Bidderís system produces an image quality indication), and

                                                                                                                                           v.            match score.

The data listed above will be returned for all System Accuracy Test searches; data returned from each System Accuracy Test search will include at least fifty (50) non-Background dB target records. For example; if 1., a particular System Accuracy Test search results in a candidate list; and 2., the 50th highest-scoring non-Background dB candidate is listed 60th in the candidate list (i.e., ten Background dB data records had match scores higher than the 50th highest-scoring non-Background dB candidate); then data must be returned for the top 60 candidates.

k. After the first System Accuracy Test, subsequent System Accuracy Tests will be conducted during non-operational hours (e.g., between 7PM and 7AM weekdays and during any hour of weekends).