Session #65

Draft Results of Quality

Assurance Program Data from
Award Year 2006-07

David Rhodes and Anne Tuccillo

Goals
• Share draft results of program-
wide analysis of 2006-07 data

• Illustrate additional ways to

analyze ISIR data

2
Background
• Quality Assurance (QA) Program

• Participants granted regulatory

flexibility to decide which ISIR
data they verify

• ISIR Analysis Tool

3
The 2006-07 Analysis
• 146 Quality Assurance Program
institutions
• Each school drew a random
sample of at least 350 applicants
• Each school verified the ISIR
information for the entire
random sample of applicants
• We analyzed 68,077 records

4
Previous data collections

2005-06 2004-05

ISIR Records
Random
subject to
Population sample of all
school
applicants
verification
Number of
140 133
schools

5
See a pattern?
• Alternating focus
–Random sample of all applicants
–Institutionally verified applicants
• This year (2007-08)
–All QA Program institutions
–ISIRS subject to institutional
verification

6
Is analysis over time
appropriate?
• For individual schools, yes with
care
• For program-wide analyses the
answer “should” be no, but…..

7
When Looking Across Years Keep
in Mind:

• The two different populations

• Changes to institutional verification
criteria between years
• Other differences between award
years
• Percents and averages not counts

8
Why we “shouldn’t”
• QA schools supplying data differ
slightly from year to year (146
≠140 ≠133)
• Unmeasured differences in
institutional verification across
years

9
Key areas of analysis
• Description of population
• Critical ISIR fields
• How changes affect aid eligibility
• Improper payments in the Pell
Grant program

10
Characteristics of Aid Applicants
at QA Program Schools 2006-07

Eligible for Pell 58%

Selected for CPS Verification 40%

Selected for School Verification 44%

Dependent 62%

0% 25% 50% 75% 100%

11
Dependent and independent
record counts over time
150,000
125,000
42,297
100,000
75,000
50,000 25,622
94,866
14,785
25,000 42,455
24,310
0
2006-07 2005-06 2004-05
Dependent Independent

12
Dependent and independent
percentages over time
100%

38% 31% 38%

75%

50%

62% 69% 62%

25%

0%
2006-07 2005-06 2004-05
Dependent Independent

13
Percentage of dependent applicants with
changes to the most commonly changed
ISIR Fields: 2006-07
Parent AGI 29%

Parent Total from

29%
Worksheet B

Mother Income from Work 26%

Father Income from Work 23%

Parent Tax Filing Status 22%

0% 25% 50% 75% 100%

14
Percentage of independent applicants with
changes to the most commonly changed
ISIR Fields: 2006-07

Student Income from Work 19%

Student AGI 18%

Student Total from Worksheet B 14%

Student Tax Filing Status 12%

Student Total from Worksheet C 11%

0% 25% 50% 75% 100%

15
Changes to critical ISIR fields among
dependent students over time
100%

75%

50%
29% 33% 30% 29% 27% 32%
25%

0%
Parent AGI Parent Total from
Worksheet B
2006-07 2005-06 2004-05

16
Changes to critical fields among
independent students over time
100%

75%

50%
25%
25% 18% 19% 14% 15% 14%

0%
Student AGI Student Total from
Worksheet B
2006-07 2005-06 2004-05

17
Dependent records: percent of ISIR fields
experiencing an EFC change - 2006-07

Parent AGI 89%

Parent Total from Worksheet B 90%

Mother Income from Work 88%

Father Income from Work 89%

Parent Tax Filing Status 87%

0% 25% 50% 75% 100%

18
Independent records: Percent of ISIR fields
experiencing an EFC change - 2006-07

Student Income from Work 66%

Student AGI 68%

Student Total from

68%
Worksheet B

Student Tax Filing Status 62%

Student Total from

71%
Worksheet C

0% 25% 50% 75% 100%

19
Comparison of changes to EFC among
dependent students with a change to the
indicated ISIR field
100% 89% 88% 91% 90% 90% 93%

75%

50%

25%

0%
Parent AGI Parent Total from
Worksheet B
2006-07 2005-06 2004-05

20
Comparison of changes to EFC among
independent students with a change to the
indicated ISIR field

100%

75% 66% 67% 71% 68% 66% 72%

50%

25%

0%
Student AGI Student Total from
Worksheet B
2006-07 2005-06 2004-05

21
Percentage of dependent records with a
change to the indicated ISIR field and a
change to a Pell Grant: 2006-07
Parent AGI 49%

Parent Total from Worksheet B 38%

Mother Income from Work 46%

Father Income from Work 42%

Parent Tax Filing Status 43%

0% 25% 50% 75% 100%

22
Percentage of independent records with a
change to the indicated ISIR field and a
change to a Pell Grant: 2006-07
Student Income from Work 36%

Student AGI 38%

Student Total from

34%
Worksheet B

Student Tax Filing Status 31%

Student Total from

35%
Worksheet C

0% 25% 50% 75% 100%

23
Comparison of change to a Pell Grant among
dependent students with a change to the
indicated field
100%

75%
58% 54%
49%
50% 42% 38% 35%
25%

0%
Parent AGI Parent Total from
Worksheet B
2006-07 2005-06 2004-05

24
Comparison of change to Pell Grants among
independent students with a change to the
indicated field

100%

75%
48% 45%
50% 38% 37% 34% 34%
25%

0%
Student AGI Student Total from
Worksheet B
2006-07 2005-06 2004-05

25
Average EFC change among dependent
records with a change to the indicated
ISIR field: 2006-07
Parent AGI 648

Parent Total from Worksheet B 819

Mother Income from Work 645

Father Income from Work 913

Parent Tax Filing Status 1,039

0 400 800 1,200

26
Average EFC change among independent
records with a change to the indicated
ISIR field: 2006-07
Student Income from Work 100

Student AGI 186

Student Total from Worksheet B 387

Student Tax Filing Status 490

Student Total from Worksheet C 470

0 500 1,000 1,500

27
Average EFC change among dependent
students with a change to the indicated
field over time
1,500
1,214
980
1,000 819
648
500

0
Parent AGI Parent Total from
Worksheet B
2006-07 2005-06

28
Average EFC change among independent
students with a change to the indicated
field over time
1,500

1,000
591
500 371 387
103
0
Student AGI Student Total from
Worksheet B
2006-07 2005-06

29
Average Pell change for dependent
records with a change to the indicated
ISIR field: 2006-07
-$270 Parent AGI

-$314 Parent Total from Worksheet B

-$257 Mother Income from Work

-$305 Father Income from Work

-$344 Parent Tax Filing Status

-$600 -$400 -$200 $0

30
Average Pell change for independent
records with a change to the indicated
ISIR field: 2006-07
-$102 Student Income from Work

-$122 Student AGI

-$257 Student Total from Worksheet B

-$347 Student Tax Filing Status

-$368 Student Total from Worksheet C

-$600 -$400 -$200 $0

31
Average Pell Grant change for dependent
records with a change to the indicated field
over time
Parent Total from
Parent AGI Worksheet B
$0

-$200

-$270
-$314
-$400

-$469
-$504
-$600
2006-07 2005-06

32
Average Pell Grant change for independent
records with a change to the indicated field
over time
Student Total from
Student AGI Worksheet B
$0

-$200 -$122

-$257
-$400 -$344
-$459
-$600
2006-07 2005-06

33
Potential improper payments in Pell
Grants prior to verification at QA Schools:
2006-07
9.1% potential over- 6.7% potential
payments under-payments

84.2% Pell Grant

dollars not at risk
prior to verification

34
Improper Payments in Pell Grants before
and after school verification
10.0%
6.7%
Percent of Pell Dollars

5.0% 2.9%
Potenial over- Net potenial
awards over-awards
0.0%
Potential -0.7%
under-awards -2.4%
-5.0% -3.6%

-10.0% -9.1%
Before verification After school verification

35
Improper Payments in Pell Grants before
and after CPS verification
10.0%
6.7%
Percent of Pell Dollars

Net potenial
4.1% under-awards
5.0%
Potenial over- 2.4%
awards
0.0%
Potential
-1.7%
under-awards -2.4%
-5.0%

-10.0% -9.1%
Before verification After CPS verification

36
Lessons Learned
• While verification is crucial in some
cases, many ISIR records do not
experience a meaningful change
from their initial transaction value
• QA school verification procedures
target larger EFC and Pell changes
• QA school verification reduces
improper payment risks in Pell

37
What Do the Findings Mean for
Schools and FSA?
Given that relatively few ISIR
records experience a change that
affects eligibility for need-based
aid, verification efforts should
strive to focus on the records that
matter.

38
Two Ways to Focus
• Look for the records that matter
• Look for the records that don’t

39
Suggested Additional Analysis
Look for ways to reduce the
number of school verified records
that experience no or only trivial
changes to aid eligibility after
verification

40
Additional Analysis
• ISIR Analysis Tool reports
• Exporting data from the Tool

41
ISIR Analysis Tool
• Key Filters
–EFC change < 400 & EFC
change > -400
–EFC change > 400
–EFC change < -400
–Institutional verification criteria

42
ISIR Analysis Tool
• Key Reports
–Sample summary
–Field Increment (EFC or AGI)
• Drill down to refine
understanding

43
New Analytic “Recipe”

See handout

44
Exporting Data

See handout

45
Institutional Profiles

Comparison of Program-
wide and school specific
data
2006-07 Institutional Profile

Example of a
2006-2007
Institutional
Profile

47
Profile Differs from Year to Year
• 2005-06 provided data of
improper payments PREVENTED
by institutional verification
• 2006-07 provides data on
estimated levels of POTENTIAL
improper payments in a schools
applicant population

48
Contact Information
We appreciate your feedback and
comments. We can be reached
at:

Email: david.rhodes@ed.gov
anne.tuccillo@ed.gov

49