APPLIEDDATAVALIDATIONPROCEDURES FOR INDICATOR5.

1OFGLOBALFUNDROUND9
( A R T S i t e s O n l y )

November,2011. NayPyiTaw

AppliedDataValidation ProceduresforARTMonthlyReports

Contents
1. 2.

Overview ExistingDocuments
2.1. ARTEnrollmentRegister 2.2. ARTWhiteCard 2.3. DrugDispensingRegister 2.4. ARTMonthlyReport 2.5. DataFlowChart

3 4

3.

Methods
3.1. Aggregation/Recounting 3.1.1.ReaggregationusingARTEnrollmentRegister 3.1.2.ReaggregationusingARTCard 3.2. Crosschecks 3.2.1.ARTMonthlyReportVsARTMonthlyReport(CentralVsField) 3.2.2.ARTMonthlyReportVsNAPARTsoftwaredata 3.2.3.ARTMonthlyReportVsDrugDispensingRegister 3.3. Spotchecks 3.3.1.ARTEnrollmentRegister/ARTCard/DrugDispensingRegisters 3.3.2.OPDorPatientRecordsBooks 3.3.3.InterviewwithPatientsonART

6 6

4.

FeedbacksofDQA
4.1. Experiencesfrom3DF 4.2. ExperiencesfromGFATMR9DQA

10

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

1. Overview
Theavailabilityofdatathatvalid,reliable,comprehensiveandtimelyisoneoftheprinciplesofa successful M&E system. Data quality assurance system in place will permit management level and decision makers to substantiate the reported data quality and provide ongoing information on the underlyingdatamanagementandreportingsystemsforataminimumoutputlevelindicators. Monitoring data quality periodically is essential to address any obstacles hampering the production of high quality HIV programme data. Data quality can be assessed through following elements:
1) Accuracy/validity:Doesthedatameasurewhatitisintendedtomeasure?Areerrors(recording,

transcription,sampling,biases,etc.)minimized?
2) Reliability:Isdatameasuredandcollectedconsistently?Areprotocolsandproceduresinplace? 3) Precision:Doesthedataprovidesufficientdetail(forexample,disaggregatingbysex)? 4) Completeness:Doesdataincludethecompletelistofparticipants,eligiblepersons,etc.? 5) Timeliness:Isdataavailableontime? 6) Integrity:Isthedataprotectedfrombiasesorpurposefulmanipulation? 7) Confidentiality:Isdatastoredwithanappropriatelevelofsecurity?Istheidentityofparticipants

sufficientlyprotected?1 Alsodatasafetyisimportantwiththedramaticallyincreasingworkloadofdatahandling. AsapartofDataQualityAssuranceprocedure,thisproceduralhighlighthasbeendevelopedto help ART sites to do data validation processes. Again this is not an operational manual or

guideline covering the whole dimensions of Data Quality, but just a tool to do supportive
supervisiontoARTsitesregardingoneofsixdimensionsofDataQuality. The principles of DQA will be included in upcoming National Monitoring and Evaluation Plan, which will provide guideline for monitoring and evaluation of the Operational Plan by providing meanstoverifydataquality.ItwasclearlystatedthatNationalAIDSProgrammewilldevelopaDQA protocol, tools and checklist based on existing in country documents and international guidelines (suchasRDQAtoolbyGlobalFund) toallowsubnationalandnationallevelstafftodotheregular DQAprocesshandinhandwithprogrammemonitoring2. As the Global Funds central aim is to increase coverage3, this procedural highlight rigorously concernsGFR9outputlevelindicator5.1:NumberofadultsandchildrenwithadvancedHIVinfection currently receiving antiretroviral therapy. It is also included in Indicators for M&E of National ResponsetoHIVandAIDSofdraftNationalM&EPlan.UnfortunatelyDQAguidelineforGFR9isstill underprocessbyPRUNOPS. Incurrentsituation,ARTmonthlyreportsaresentdirectlytocentralNAPatNayPyiTawwithout intermediateaggregation.Frequencyismonthlyandtimingisduringfirstweekofsucceedingmonth. However,clearinstructionandguidelinesneededtobeprovidedtoARTsites.Atthesametimedata responsibilityshouldbestrengthened.
1 2

DataQualityAssuranceGuidelines,TheThreeDiseasesFund,Draftversion,March2010 NationalM&EPlan,NAP,Draftversion,October2011 3 MonitoringandEvaluationToolkit,TheGlobalFund,June2004

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports The highlighted parts of the following figure4 shows specific indicator for this procedural highlight.

2. ExistingDocuments5
Thefollowingdocumentsareusedinthevalidationprocesscoveringallstagesofdataflowfrom primarysourcetointermediateandcentralaggregationlevels. 2.1 ARTEnrollmentRegister(RedBook) 2.2 ARTCard(WhiteCard) 2.3 DrugDispensingRegister(GreenBook) 2.4 ARTMonthlyReport 2.5 DataFlowChart(refertopage5.)

4 5

MonitoringandEvaluationToolkit,TheGlobalFund,June2004 InformationfromFocalPerson,ARTProgram,NAP

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports 2.5DataFlowChart6

CentralNAP StateorRegion RegionalOfficer /TeamLeader

MedicalSuperintendent StateorRegion HealthOfficer Physician/Pediatrician

CentralLevel

ARTMedicalOfficer

IntermediateLevel ServiceDeliveryPoint Reporting Feedback

DataEntryFocal

DataFlowChart,DevelopedbyNAPARTFocalon04/11/2011

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

3. Methods
AsindraftNationalM&EPlan,followingmethodshasbeenusedindataverificationprocess.

3.1 Aggregation/Recounting
Thesemethodsareidenticalwiththemethodsusedwhenthereporteddatawasproduced.

Checkinginformationintallysheets,clientorlabregistersagainstsummaryreportdatafor differencesinnumbers; Checkingpictorialtallysheetsagainstmonthlyreportsordataincomputerdatabase.7 ReaggregationusingARTEnrollmentRegister Refertothefigurebesides,toseehowto aggregatedata. First step: By seeing OT/ ST/ MIS/ LFU/etc.., in last updated column of each individual (dash line), the current status of ART foreachpatientwillbegot. Secondstep:Bycounting them(dotline), totalnumbertobereportedat block8and9of ART Monthly Report can be produced. Except numberstobereportedat9.6.1,9.6.2and9.6.3. Explanation: Assume the reporting period to be November2011. If a patient started enrollment at September 2010 then mo.14 here will be the status for November 2011. In the same page of this ART enrollment register, if another patient started enrollment at November 2010 then mo.12 here will be the status for November 2011. It is advisable to check whether this registerhasbeenupdatedforreportingmonth,by looking at Date of start of ART. See the figure below.

3.1.1

DataQualityAssuranceGuidelines,TheThreeDiseasesFund,Draftversion,March2010

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

Forthereportednumbersatindicator 9.6.1,9.6.2,9.6.3and12,recountingcouldbedoneby
seeingcolumnsasdescribedinbelowfigure.Thisprocedureshouldbedoneonlyafterrecounting indicator9.6.ARTstatuswith1stLineregimen,Substituted1stLineregimenand2ndLineregimencan beseen,includingdates.

3.1.2 ReaggregationusingARTCard
SomeARTsitespreferARTcardsratherthanARTEnrollmentRegistertoproducemonthlyreport. As described earlier, identical validation procedure could be used the same as the steps used in producingmonthlyreport.

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

3.2 Crosschecks
Thebasicprincipleofthesemethodsistocheckthesamedataacrossdifferentrecords/registers, whicharesupposedtobefilledupinthesametimeordifferenttime.Forexample:

Checkingthedrugregisterforevidenceofstockoutscomparedwithstockoutsreportedin monthlyoraggregatedreports Checkingprocurementrecordsagainstpatientrecordsfortreatment.8 ARTMonthlyReportVsARTMonthlyReport(CentralVsField) ToproduceProgressReport(AnnexE.),compilationofARTMonthlyReports(Indicator9.6)is doneatcentrallevel.Duringthisprocess,latestreceivedARTMonthlyReportsfromARTsitesare used. Sometimes, there would be discrepancies between reported ART Monthly Reports (at CentralNAP)andofficecopiesofARTMonthlyReports(atARTSites). To minimize such discrepancies, it is advisable to inform/report Central NAP whenever correctionsormodificationsaredoneatARTsites.

3.2.1

3.2.2

ARTMonthlyReportVsNAPARTsoftwaredata

SinceNAPARTsoftwareisnotinoperationalstatus,thiscrosscheckmethodcannotbeused. 3.2.3 ARTMonthlyReportVsDrugDispensingRegister Inmostofthesettings,DrugDispensingRegisterservesasthemostreliableprimarysource tocoverARTstatusofpatients.Patientswhotooktheirdrugsduringthereportingperiodcould serveasasourcefordataverification. It is advisable to pay attention to those patients, who took drugs for more than 1 month period(i.e.,intervalbetweentwovisitsismorethan30days).

DataQualityAssuranceGuidelines,TheThreeDiseasesFund,Draftversion,March2010

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

3.3 Spotchecks
Thebasicprincipleistovalidatethedata,usingindepthassessmentswithinendbeneficiaries. Forexample: Checkingthedrugregistertocomparetodaysinventorywithphysicallyverifiedpresenceof drugstoday; Askingsamplebeneficiarieswithinthecommunityiftheyreceivedinsecticidetreatednets,etc.; Askingstaffontraininglistiftheyreceivedtraining.9 3.3.1 ARTEnrollmentRegister/ARTCard/DrugDispensingRegisters

TheseseriesofdocumentsareusedasstandardizedformatsinARTsites.Eachindividualpatient isrecordedindifferenttypesofrecordsforthesameday.Thistime,eachpatientlistedinART Monthlyreportcouldbetrackedusingthedescribeddocuments. 3.3.2 OPDorPatientRecordBooks

In some ART sites, OPD or Patient Record books are used to monitor clinical progress and prescriptions.Suchbooksmayserveasanothersourcetovalidateindividualdata. Itisadvisabletouse minimum necessarystandardizedformats, sothatfindingsindatavalidity maybeimproved. 3.3.3 InterviewwithpatientsonART Thisrigorousproceduremaybecarriedoutforindividualcases.

DataQualityAssuranceGuidelines,TheThreeDiseasesFund,Draftversion,March2010

NAPGFR9|CentralProjectCoordinator(ART)

AppliedDataValidation ProceduresforARTMonthlyReports

10

4. FeedbacksofDQA
4.1 Experiencesfrom3DF
As DQA guideline from PRUNOPS is still under process, grading can be studied from 3DF DQA guideline. All errors may not be serious enough to be concerned about. If there is an occasional adding error of a few numbers, this may not be important enough for any action except perhaps verbal feedback at the time the error is discovered on the need for additional caution in preparing data. Although the data may ultimately be found to be correct, if it is sufficiently beyond a threshold a moreindepthevaluationwillbeneeded.Atthetimeathresholdisreached,the3DFFMwilllaunch a specific investigation to identify the problem and will complete a written report outlining the problemidentifiedandthestepstakentoinvestigateandresolveanyproblemsidentified. 3DFpolicyisthata10%differenceinindividualitemsforquantitativedatachecksisthelevelat whichformalstepsaretobeinitiatedformoreindepthcheckingandmakingcorrections.3DFwill followthispolicyforthefieldvisitsandforthecrosschecks.AllIPswillreceivefeedbackontheIP DQAFeedbackFormwhichwillprovideanoverallgrade(A,B1,B2,andC)fortheDQAcheckalong with the same grading for each individual indicator checked. An A= 10% or less margin of error; B1=1020%errormargin;B2=20%orgreatererrormargin;C=noDQAsysteminplace.10
11

4.2 ExperiencesfromGFATMR9DQA
Assessmentofdatamanagementandreportingsystem Regarding ART indicator at service delivery points, the monthly reports were manually consolidatedatARTsitebasedonthreedifferentregistrationbookswhichare: (1)PreARTScreeningRegister(Black) (2)TreatmentEnrollmentRegister(Red) (3)DrugDispensingRegister(Green)whichissignedbypatientsthemselves. Regardingcentralleveldatarecordingsystem,themonthlyreportssubmittedfrom25ARTsites wereelectronicallycompiledandsavedinMicrosoftExcelsheetdatamanagementsystem. Keyfindings ServiceDeliveryPoints At ART site (Pyay Hospital), Consultant Physician takes overall responsibility for clinical management and data entry at the same time. Data are manually recorded in register books and compiledinNAPmonthlyreportsummarysheet. IntermediateAggregationLevels ThereisnointermediateaggregationlevelforARTindicatorsincemonthlyreportfromARTsiteis directlysubmittedtoNAPcentral. CentralM&EUnit ARTdataarekeptinMicrosoftExcel. Grading Indicator5.1:
DataQualityAssuranceGuidelines,TheThreeDiseasesFund,Draftversion,March2010 DQAreport,PRUNOPSM&ETeam,July2011

10 11

NAPGFR9|CentralProjectCoordinator(ART)