GAO REPORT (GAO-10-119, OCTOBER 2009) TO CONGRESS: ARMY BODY ARMOR TESTING:

COMPARISON OF GAO CONCLUSIONS & RECOMMENDATIONS,

DOD COMMENTS, AND GAO RESPONSES TO DOD COMMENTS

GAO Conclusions/Recommendations DOD Comment/Position GAO Response to DOD Comment

1. Questionable if Army met its FAT objectives 1. GAO report points out some weaknesses 1. Several deviations to testing protocols call
for armor designs fully meeting Army and discrepancies in testing, but findings have testing results into question. Until independent
requirements before armor purchased no significant impact on test results and expert evaluates impact of deviations on testing,
subsequent contracting actions. Do not concur DOD can’t be assured plates that passed testing
-- Observed several instances in which actual PDM and FAT may not have achieved testing can defeat the threat.
test practices deviated from established test objectives
protocols -- DOT&E led the integrated product team for
--PDM: Army followed “deliberative internal testing and approved test plans, but when GAO
-- Four of five designs that passed FAT would process,” consulted with DOT&E, and was at ATC for PDM and FAT we did not observe
have failed either during PDM or FAT if modified contract solicitation for PDM to use DOT&E monitoring testing beyond incidental
established protocols has been fully followed point of aim for measuring back-face visits during VIP events and other demonstrations
deformation. Did not bias test results. Design
-- It’s possible deviations had no significant models passed in accordance with modified --PDM: Disagree with DOD/Army statement that
net effect or may have resulted in armor being solicitation PDM testing achieved objectives: (1) Test
tested to more rigorous standard Protocols called for measurement of back-face
--FAT: Achieved objective of verifying deformation at deepest point of depression in clay
-- Also possible some deviations may have contracted vendors could produce plates that backing. (2) DOD-Army claim solutions passed
resulted in armor being tested to less stringent passed PDM in full-rate capacity because protocols were modified, but this
requirement than required overlooks Army had to modify solicitation because
--Multi-Phase Test concept included PDM, it deviated from testing protocols laid in out
-- Unable to determine full effect of FAT, and extended ballistic testing. Army purchase descriptions. Army Research Lab and
deviations—can only be made by thorough refined procedures lessons learned in PDM for NIJ-certified labs use benchmark process of
assessment of test data by independent FAT measuring back-face deformation at deepest point,
ballistics testing experts not point of aim.

-- IVO uncertainty and critical need for --FAT: Restates FAT may not have achieved its
confidence, Army would take unacceptable objectives, because of questions regarding scoring
risk to field designs without additional steps to methods. Two solutions that passed FAT would
gain needed confidence have failed if back-plate deformation had not been
rounded and had been scored as it was in PDM.
Third would have failed if a complete penetration
had been scored as stated in purchase description.
Recommendation 1: SECDEF should direct
Army to provide for independent evaluation of
FAT results before armor is fielded under
solicitation. GAO cited issues/deviations
associated with:
-- Rounding of back-face deformation
measurements: (Army was inconsistent: was
not spelled out in testing protocols.
Measurements were not rounded down to
nearest whole number in PDM, but were in
FAT). Rounding down lowered requirements
and favored vendors.
-- Scoring of complete penetration of a
plate: Tear in Kevlar fibers in rear of plate
appear to show evidence of perforation and
ATC found particles in soft backing material
behind plate. Test protocols define complete
penetration as occurring when projectile,
fragment, or fragment of armor material is
embedded or passes into soft undergarment
behind protective plates—not when threads are
broken.
-- Laser Scanner Certification: Was not
properly performed—was certified to standard
that did not meet requirement of test protocols.
Army testers said software modifications were
designed to change measurements reported by
laser. Was certified to wrong standard and
certification study was not performed in actual
test environment using actual shots. DOD can’t
validate its assertions without side-by-side
comparison of laser with mechanical calipers
Non-concur. DOD maintains FAT objectives
were achieved. DOD satisfied FAT was
properly scored despite process discrepancies
and documentation issues noted by GAO.
DOD will review test processes, with “external
assistance” in selected areas. DOD does not
believe NIJ-certified labs should be considered
external experts to critique DOD policies and
procedures. DOD will continue to scrutinize
procedures and pursue more open
collaboration …primarily with PEO Soldier,
ATEC, and DOT&E.
-- Non-concur. Procedure used by NIJ-
certified labs since 1999. IPT agreed to use
common rounding-down method cited in
ASTM E-29 for FAT and subsequent LAT.
Policy not prohibited by any DOD or NIJ
standard.
-- Non-concur. First layer behind plate serves
as a “witness plate.” If not penetrated—as
determined by breaking of threads on first
layer of soft armor—test is not scored as a
complete penetration iaw PEO scoring criteria.
Breaking of a thread in deeper layers not a
penetration since force of impact can cause
deformation. Scorers can examine with
microscope. Three independent SMEs agreed
and scored as partial penetration. ATC SME
found only dust particles and discoloration.
-- Non-concur. Laser scanner provides
superior tool. Army’s TMDE Activity
conducted study leading to certification.
Certification testing performed in both lab and
on actual range. Software upgrades did not
affect measurement in laser scanner—they
improved post-processing of data and
enhanced graphical interface of system.
Upgrades had no effect on measurement
process.
Disagree PDM and FAT achieved objectives.
Found numerous deviations from protocols that
allowed solutions to pass that otherwise would
have failed. Majority seem to make testing easier
to pass and favor vendors. NIJ-
certified labs consistently proven to be capable of
following Army test protocols. NIJ labs conducted
previous Army FAT and LAT. NIJ certification
includes independent peer review of controls, mgt
practices, and lab practices. NIJ certification of lab
and NIJ certification of body armor are two
different things.
-- DOD cited ASTM E-29 as basis for rounding
down, but this is not referenced in protocols.
Detailed test plans state solutions will incur a
penalty for deformation greater than 43mm. Army
did not round down in PDM, but did in FAT. This
favored vendors.
-- DOD position not consistent with testing
protocols. No witness plate required to determine
if penetration occurred. Do not state that
“breaking of threads” is the criterion for
determining penetration. Testing protocols, not
undocumented criteria, should be used in scoring
to determine penetration/results. Multiple
observations and thorough inspection in soft armor
in question revealed particles had penetrated at
least three Kevlar layers. Shot should have been
ruled a complete penetration.
-- Once properly certified, tested, and evaluated
laser may eliminate human error. Maintains Army
used scanner as a new method to measure back-
plate deformation without adequate certification.
Do not agree software upgrades did not affect
measurement. Was told that upgrades made after
FAT corrected software laser malfunction during
subsequent LAT. Vendors say studies show laser
overestimates back-plate deformation by about 2
millimeters.
-- Exposure of clay to cold, heavy rain: May
have affected test results during one day of
testing. Army, NIJ-certified labs, and clay
manufacturer said water may contaminate clay,
change its chemical bonding, and cause rapid
and uneven cooling.
-- Improper clay conditioning and use of
additional clay calibration drops: Army
followed orally agreed procedures that
deviated from test protocols. Used clay that
had failed initial clay calibration test. Several
clay calibration drops were not within
specifications. Protocols only allow for one
series of drops per clay box, but additional
series of drops were used when first series
were outside specifications.
Recommendation 2: Army should document
all key decisions made to clarify or change
testing protocols.
Recommendation 3: Based on results of
GAO-recommended external review of test
results, Army should determine if practices
that deviated from protocols will be continued
and change the protocols to reflect revised
procedures
-- Non-concur. Do not agree with GAO
statistical analyses. ATEC analyses show poor
performance in question was caused by
marginal performance against most formidable
threat round, not exposure of clay to cold,
heavy rain. Although this had no effect on test
results, ATC completed planned installations
of new clay conditioning chambers inside
ranges to preclude environmental interaction.
-- Partially Concur. Concurs using written
standard when first series of drops doesn’t pass
specifications; non-concurs that failed clay
blocks were used in testing.
All clay used passed calibration standard in
effect at that time. NIS does not address
specifically the issue of repeated test, but
would recommend only one series of drops for
clay calibration—but this is not a NIJ standard.
DOD will partner with NIST to conduct
experiments to improve understanding of clay
performance.
-- Concur. DOD recognizes need for
documentation and proper approvals to support
any significant change to testing protocols.
DOD will publish series of standard test
protocols for PPE, beginning with soft and
hard body armor. Will include detailed
documentation requirements and remediate
process discrepancies noted by the GAO.
-- Partially concur. Inconsistencies in
procedures to implement test protocols in FAT
did not alter results. DOD recognizes need to
update protocols necessitated by new
technology and improved test procedures.
DOD will use DOT&E to promulgate standard
test protocols across DOD.
-- Exposure to cold, heavy rain may have had an
impact on test results. Supported by statistical
analyses of test results and conversations with
industry experts, including clay manufacturer.
Witnessed unusually high number of clay
calibration failures during testing of plates from
multiple vendors—not just example cited by
Army. In addition to cold water’s effect on clay
bonding and temperature, Army lowered
temperatures of clay conditioning trailers, saying
ovens and clay were too hot. Failure rate of plates
were very high this day—of both complete
penetration and back-face deformation.
-- On Nov 13, there were seven calibration drops
not within specifications. Some boxes were
discarded iaw testing protocols, but others were
repaired, redropped, and used if they passed
second drop series. Calibration called for “a series
of drops” of one series of three drops, not multiple
series of three drops. An external entity needs to
evaluate the impact of Army practice on FAT
results.
No comment
--Do not agree that inconsistencies in procedures
did not alter test results. Observations clearly show
that if measurement of deepest point of
deformation was used in PDM testing, two designs
that passed would have failed. If Army had not
rounded deformation measurements down in FAT,
two designs that passed would have failed. Four of
five designs that passed PDM and FAT would have
failed if testing protocols had been followed.
Recommendation 4: Based on the results of
GAO-recommended external expert review of
test results, Army should reevaluate and
recertify laser scanner.
-- Partially concur. DOD does not concur -- GAO does not agree that an adequate,
with GAO conclusion regarding independent certification of the laser measurement
inconsistencies and need to recertify the laser system and process was conducted. GAO continues
measurement system. DOD does concur with to assert that the software changes added after
concept of an independent certification of the certification did affect the measurement system in
laser measurement system and processes. the laser.
States this was done with Army TM&DE
Agency calibration and CG ATEC certification
for use during Army testing. Software changes
reported by GAO did not affect measurement
system in the laser scanner.

Recommendation 5: Army should provide an -- Partially concur. DOD will conduct -- GAO agrees that a review conducted by panel of
external independent peer review of Aberdeen independent evaluation of ATC’s test external experts that also includes DOD members
Test Center’s body armor test protocols, protocols, facilities, and instrumentation by could satisfy its recommendation, however, to
facilities, and instrumentation. subject matter experts. DOD in discussion with maintain independence of the panel the DOD
NIST to form team of SMEs to review DOD members should not be composed of those
testing procedures. Review will be broad and personnel from organizations involved in the body
include measurement processes, clay armor testing (such as the DOT&E, the Army Test
conditioning, and other areas. DOD will Center, or PEO Soldier).
include experts from within the DOD as part of
this team.
Matter for Congressional Consideration:
DOD asserted issues GAO identified do not
alter the effects of testing. GAO maintains that
its analyses and findings provide sufficient
evidence to raise questions as to whether issues
it identified had an impact on testing results.
GAO continues to believe it is necessary to
have an independent external review of test
results and the overall effect of the testing
deviations on those results before armor is
fielded. Without independent review, FAT
results remain questionable, undermining
confidence of the public and those who rely on
armor for protection.

Congress should consider directing OSD either

to require independent external review of body
armor test results or that DOD officially amend
its testing protocols to reflect revised test
procedures and repeat FAT to ensure only
properly tested designs are fielded.