ITEMAN and Lertap 5

Larry Nelson
Curtin University of Technology
Document date: 17 February 2009
website: www.lertap.curtin.edu.au

ITEMAN is a true software veteran with an origin dating back to the 1970s. For
many years it was the system to use for classical item and test analysis. It has
had decades to cement a reputation as a solid, reliable workhorse. Some see its
interface as dated, but its use remains widespread.

At a basic level, item and test statistics, ITEMAN and Lertap 5 produce similar re-
sults. However, Lertap has several features not found in ITEMAN – among these,
graphics stand out as a major difference. Lertap has options which produce
graphs, charts, and colourful tables of results; its quintile plots reflect aspects of
item quality in a visual manner, sometimes making it easier to detect poorly
functioning items.

But Lertap’s interface is vastly different to ITEMAN’s. Experienced ITEMAN users

might be reluctant to go through the steps needed in order to master Lertap.
This document is for them. It is now quite an easy matter to go from ITEMAN to
Lertap. The discussion below also is applicable to Assessment Systems’ RASCAL
and XCALIBRE software for Rasch and 2- and 3-parameter IRT analysis, respec-
tively, since they use the same input file format as ITEMAN. Thus, any input file
used with RASCAL or XCALIBRE can be analysed by Lertap 5, as described below.

The Ins and Outs of ITEMAN

I want to bedazzle you with Lertap, but the beginning of the dazzle will feature
ITEMAN.

Figure 1

No doubt you will recognise the contents of Figure 1 as an ITEMAN DAT file. In
this case the file’s name is SAMPLE1.DAT.
The first four lines of each and every ITEMAN DAT file tell ITEMAN about the data
in the file. I call these “control lines”.

The first line in Figure 1 says there are 30 item responses, and zero ID characters
(these are the first and last of the four entries on the line). The second line has
30 characters with the item scoring keys, the correct answers to each item.

The third line, the line of 4s, says that each item used four alternatives.

The final line, in this case, says “Y”es, each item is to be included in the analysis;
there are 30 Ys on this line, one for each item. If an item is to be excluded from
the analysis, an N is used instead of a Y. No Ns in this example – all items are to
be included.

Figure 2

Figure 3

ITEMAN and Lertap 5, page 2.

Right. I start up ITEMAN and get it to produce its output. Because I don’t really
use ITEMAN, I am rather unfamiliar with its options. I take the easy way out by
deciding to run with the default options.

Figure 2 shows part of ITEMAN’s standard output. Figure 3 shows a bit more.

The Ins of Lertap

ITEMAN is a text-file-based system. Lertap is an Excel-based system. Excel is

the spreadsheet program found in a Microsoft software suite called “Office”. In
order to run Lertap you must presently have either a Windows or a Macintosh
version of Excel 1 .

Importing an ITEMAN DAT file to Lertap is very simple, trust me. Actually, you
don’t need to trust me as later on I go through all the steps required, just to
show you. For the moment, though, trust me.

The routine in Lertap which does the importing is a macro 2 called “Iteman1”.

Iteman1 reads the DAT file and creates a new Excel workbook with three
worksheets. One of the worksheets contains a complete copy of the original DAT
data. This worksheet has the remarkably original name of “Original”.

Another worksheet, “Data”, contains the re-formatted data. A snapshot of part of

a Data worksheet made by Iteman1 is on display in Figure 4 3 .

Figure 4

1
Within a year or so, Lertap may also work with “Calc”, the OpenOffice equivalent
to Excel. Calc is already an excellent product, but its present support for macros
falls a bit short.
2
A macro is a small computer program designed to meet a special need.
3
Note that some of the screen shots in this paper are from Excel 2007, while
others are from Excel 2003.

ITEMAN and Lertap 5, page 3.

I imagine you’ll be able to understand Figure 4? The item responses have been
spread across columns, starting from column 2 and going out to column 31 (not
visible). Each item has been given a label, starting with Q1.

Now, much as ITEMAN requires four lines with “control” information, Lertap asks
for a set of “control cards”. The Iteman1 macro puts these in a worksheet called
“CCs”. Behold Figure 5, if you will.

Figure 5

The first CCs line tells Excel that item responses are found in columns 2 through
31 of the Data worksheet. The second line gives the test data a short label, or
“Title”, and tells Excel to regard 1, 2, 3, and 4 as valid response codes.

The third line in the CCs worksheet corresponds to the second line in an ITEMAN
DAT file: the item scoring keys, the correct answers, one for each item. Note
that it’s possible, in Lertap, to multiply-key an item, that is, have more than one
right answer for a cognitive item. This is done by including *mws lines in the CCs
worksheet.

It is also possible to exclude an item from the analysis. ITEMAN does this by
using “N” on its fourth control line. Lertap accomplishes the same thing in a
variety of ways. One way is to use a *exc line in the CCs worksheet; another is
to replace an item’s entry in the *alt line with an asterisk.

From One Inn to Another

Just a while ago I asked for your trust. Thank you. Now let me make good on
my word. I shall show you how tremendously uncomplex it is to convert an
ITEMAN DAT input file to the three worksheets which form the “In” part of a
Lertap workbook.

I will start by opening the Excel 2003 version of Lertap. See Figure 6, please.

ITEMAN and Lertap 5, page 4.

 Figure 6

I then click on the Macs drop-down menu, and hooroo!, there’s that Iteman1
macro name of which so much has been said already. Figure 7, maestro.

Figure 7

ITEMAN and Lertap 5, page 5.

Okay then. Surely everyone knows what to do next: click on Iteman1. In theory,
this should allow me to invite you to proceed to take in Figure 8. However, for
users with Excel 2003, or an even earlier version of Excel, such as Excel 97, an
unwelcome surprise may intervene at this point. This surprise may take a form
similar to that seen here:

Should this happen to you, please do one of two things: either bring down the
Macs menu again (Figure 7) and click on “Lelp”, the last entry in the menu list, or
jump to the appendix found at the back of this document. Lelp and/or the
appendix will provide the steps required to fix this temporary impasse. Please
follow those steps now. When you’ve finished, come back here and continue
reading.

Once clicked on, Iteman1 starts its job by displaying a file browser, a tool which
allows you to pick out that ITEMAN DAT file you want to use. Figure 8, mate.

Figure 8

After I’ve cruised to the file, SAMPLE1.DAT in this example, I click on Open and
fat city here I am – before I can recall the names of all my nephews and nieces,
my new friend, Iteman1, tailors a new Excel workbook for me, and kits it out with
those worksheets seen above in Figures 4 and 5 4 .

If you’re using Excel 2007, the Macs drop-down menu is not located on the front
control panel. Access to it is via the Move+ menu drop-down found in the “Other
menus” area of the ribbon. Take a shine to Figure 9.

4
Lessen your trust level a bit at this point. The import process can take longer
than ten seconds if there are lots of data records and a fair number of items.

ITEMAN and Lertap 5, page 6.

 Figure 9

The Common Outs of Lertap

So much for the ins. Getting outs from Lertap can be as simple as one click5 , but
generally involves a bit more; all fun, of course. I will assume you are not a one-
clicker; like myself, you prefer to have some control over the goings-on, no? Yes.

I’ve had Iteman1 make those nifty worksheets. I let my mouse drift up to the
Run menu. I click on Interpret.

Lertap checks on the lines in the CCs worksheet to make sure that they are
interpretable. If they’ve been written by Iteman1 chances are they’re pretty
good, that is, have no syntax errors.

Then Lertap asks you to peruse its “Freqs” report, a simple little summary of item
response frequencies. Simple, but often extremely useful. Figure 10.

5
Lertap has a “production mode” option, useful when data have been prepared
by a scanner, and when users always want the same output reports, time after
time.

ITEMAN and Lertap 5, page 7.

 Figure 10

I have supreme confidence in your ability to interpret Figure 10 with no hand

holding from me. It’s just a report, for each item, of the number of times people
selected an item’s alternatives. It comes into its own when the little tables
report, for example, that someone answered alternative 5 on Q1 when there were
only four alternatives. Such events are not uncommon at all. They’re usually
data processing errors.

For the moment let’s say that the aftermath of using the Interpret option is a non
event. Lertap was able to interpret the CCs lines without fuss, and your scan of
the little Freqs tables left you feeling warm and cosy (no data processing errors).

Next: to Elmillon, the option which goads Lertap into making a considerable
number and variety of reports, each in a separate worksheet with a unique name.

So. Elmillon. Click. Excel engages in a remarkable amount of activity. The

screen flashes, the lights dim, the curtains draw back and lo: I see that I now
have worksheets with the enchanting names of Scores, Stats1f, Stats1b, csem,
and Stats1ul.

The Scores worksheet reports on each person’s test score, as you might well
expect, and then, at the end, we get the little summary seen in Figure 11.

ITEMAN and Lertap 5, page 8.

 Figure 11

Being an experienced ITEMAN user, you will recall that ITEMAN will process sev-
eral scales at once. Lertap too. Each ITEMAN scale becomes a Lertap subtest.
When there are multiple subtests, the Scores report will have one column for
each subtest, and, like ITEMAN, Lertap will report on the correlations among the
subtests.

After the Scores worksheet, Lertap outputs two of its three main item analysis
worksheets: Stats1f and Stats1b. The latter, the “b” worksheet, is a brief
summary of the core item statistics found in “f”, the full-statistics report.

The information found in the upper parts of the Stats1f and Stats1b reports is
similar but not at all identical to that found in ITEMAN’s item analysis section.

Figure 12 displays the upper part of Stats1f; Figure 13 does likewise for Stats1b.

ITEMAN and Lertap 5, page 9.

 Figure 12

Figure 13

The ?-mark column in the Stats1b report is used to flag distractors which may
have performed poorly. How this is determined is covered in the manual.

It would be the lower portions of the Stats1f and Stats1b reports which provide
information in a format not found in ITEMAN.

ITEMAN and Lertap 5, page 10.

 Figure 14

Figure 15

ITEMAN and Lertap 5, page 11.

 Figure 16

Figures 14, 15, and 16 have all been taken from the lower reaches of a Stats1f
report.

Figure 14 will no doubt be reasonably self-explanatory. Figure 15 is a summary

of item difficulty and discrimination figures grouped by “bands”. The hardest item
in the test, the one with the lowest classical difficulty index, was Q20. This item’s
difficulty was greater than .40 but less than .50 (its exact value may be found
higher up in the Stats1f report). Many of this test’s items were on the easy side,
having difficulty figures above .90. Four of the test’s items had very low discrimi-
nation figures (less than .10).

Figure 16 indicates how subtest alpha would change if items were omitted from
the subtest. For this test, alpha with all 30 items included was .7895. Were Q1
to be omitted, alpha would decrease to 0.786.

The scatterplot of item difficulty by discrimination seen in Figure 17 is found at

the bottom of the Stats1b report. This figure presents another chance to confirm
that our little 30-item test was on the easy side; most of the items were correctly
answered by more than 80% of the students. When this plot is first displayed,
each blip in the plot has the corresponding item label attached to it, such as Q1,
Q2, and so on. These are easy to delete and I usually do so as I find it easier to
use the plot without the clutter caused by item labels.

The Stats1f and Stats1b worksheets are followed by the “csem” worksheet as
exemplified in Figure 18. I have moved the plot over the table to save space in
this document. The conditional standard errors of measurement are based on the
binomial and compound binomial error models. A discussion of Lertap’s CSEMs is
found in this paper:

http://www.lertap.curtin.edu.au/Documentation/JERM2007d.pdf

ITEMAN and Lertap 5, page 12.

 Figure 17

Figure 18

After the CSEMs comes the third, and last, report of item performance, the
Stats1ul worksheet. Take in Figure 19 at this point, if you would.

ITEMAN and Lertap 5, page 13.

 Figure 19

The Stats1ul report is an upper-lower one. Its statistics are based on the time-
honoured method of separating test results into groups, such as High and Low (or
Upper and Lower), and then comparing the endorsement rates of the keyed-
correct answer and the distractors among the groups.

Lertap allows for up to five groups. In fact, its default for the number of groups
is five – this may be adjusted to four, three, or two as desired. The keyed-
correct answer is underlined; for Q1 the keyed-correct answer was 4; for Q2 it
was 2.

Figure 20

Figure 20 displays the little table which always appears at the bottom of a
Stats1ul report.

Lertap has support for mastery testing and the use of cut-scores. The mastery
test option is set by using a mastery= assignment on the *sub line of the CCs
worksheet. Figure 21 indicates how the Stats1ul table is altered in the case of a
mastery test analysis.

ITEMAN and Lertap 5, page 14.

 Figure 21

The statistics calculated in the mastery testing case are described in the manual,
and further detailed in:

http://www.lertap.curtin.edu.au/Documentation/JERM2007d.pdf

The Extra Outs of Lertap

You’ve just been reading about Lertap’s common output. All of the stuff above
was generated by 2 + 2 mouse clicks. One click on the Interpret option on the
Run menu. Then one click on an OK button to confirm receipt of Interpret’s
output. Then one click on Elmillon, followed by a click on another OK button.

Your wrist will be tired after four mouse clicks, I know. After you’ve had a break,
a bit of intense table tennis perhaps, you might be truly tickled to hear that your
wrist and mouse combination can get you even more results from Lertap.

For example, let’s suppose you happened to be gazing at a Stats1ul report

(Figure 19), and thought to yourself: “Hmmm, I bet Excel would convert one of
Lertap’s Stats1ul tables to a chart … now, where would I find out how to do it?”

Here (!). Right here. No need for the Excel documents. Push your mouse up to
the “Graphics trio” on the Lertap ribbon tab, and click on “Res. Charts”.

Faster than you can name the capitals of all the countries in the European Union,
Lertap makes its quintile plots 6 . Figure 22, just for you.

6
Truth to tell time. Excel 2003 will generate quintile plots very, very rapidly. Not
so Excel 2007 (maybe a coming release of Excel 2007 will be faster).

ITEMAN and Lertap 5, page 15.

 Figure 22

These quintile plots can be fun, extremely useful, and something to impress the
School Board with. I have found them to be a very good way to discuss concepts
of discrimination, difficulty, and distractor quality to people who prefer pictures to
numbers.

Read more in this yet-to-win-an-award-but-deserving-of-one paper:

www.lertap.curtin.edu.au/Documentation/UsingLertapQuintilePlots.pdf

Now then, let’s suppose you’re looking down a Scores worksheet, and long for a
histogram. Long no longer.

Go back to the Graphics trio. Click on Histograms. Figure 23.

ITEMAN and Lertap 5, page 16.

 Figure 23

If your wrist is up to it, back to the Graphics trio, then the little down arrow to
the right of the Histogram option. A “Histogram E” is pictured in Figure 24.

Figure 24

Thus far we have been looking at results from a single test. When a data set
involves more than one scale, or subtest, ITEMAN and Lertap both output a
matrix of correlations among the subtests. Another of the options on the
Graphics trio, “Scatterplot”, may be used to produce a graph of the scores from
any pair of subtests. Figure 25 is an example.

ITEMAN and Lertap 5, page 17.

 Figure 25

Figure 26 reveals what’s uncovered by clicking on the More option on the Run
menu.

Figure 26

The “Item scores and correlations” option will produce a matrix of interitem
correlations. This may have SMCs on its diagonal 7 . The eigenvalues of the
matrix may be extracted and plotted in a traditional scree test: Figure 27.

7
Squared multiple correlations.

ITEMAN and Lertap 5, page 18.

 Figure 27

Figure 28 shows what can happen when test scores have been broken out by
groups, with group differences plotted in an almost-classic box and whiskers
display. In this case, three groups of teacher trainees, early childhood, primary,
and secondary, responded to a survey on ability to self-regulate their own study
time.

Figure 28

The “RSA (response similarity analysis)” option is used to look into the possibility
of exam cheating. The following paper has more details on this option, should
you have an interest:

http://www.lertap.curtin.edu.au/Documentation/JERM2006mod1.pdf

ITEMAN and Lertap 5, page 19.

And the winner is …

Well, there you have a rather rushed introduction to many of Lertap’s capabilities.
There’s much more documentation, samples and examples, on the main Lertap
website:
www.lertap.curtin.edu.au

Is Lertap better than ITEMAN? Maybe, maybe not; depends on what you’re after.
You want classic item analysis results, and have used ITEMAN with total satisfac-
tion for yonks 8 ? ITEMAN is for you. It’s a tight, speedy little package which does
not require a relatively large system such as Excel to run in. Compared to Lertap,
ITEMAN might be said to be parsimonious – it gets the job done without an
excess of verbiage. It’s backed by a good manual, and readily interfaces to
several other products at www.assess.com. (Lertap will also interface to other
systems; in addition to RASCAL and XCALIBRE, Lertap can interface with Bilog
MG, the FastTEST Test Development System, and the FastTEST professional
Testing System.)

You’ve seen that Lertap outputs no less than three item analysis reports.
ITEMAN’s item analysis section is more condensed, and for some users this may
be preferable. On the other hand, the graphs which Lertap will output serve, I
believe, to enhance data analysis. I say this with one type of graph particularly in
mind: quintile plots, exemplified back in Figure 22. These plots are nifty, and
without too much practice can be put to good value by a real variety of test
developers and users – they don’t require any knowledge of statistics to speak of.
Even if one is full bottle 9 on statistics, the quintile plots have a way of quickly
capturing and presenting a lot of information about an item’s performance.

If this paper has served to arouse your interest in Lertap, why not take it for a
test spin? Download a 30-day trial from www.assess.com at no cost, install it,
feed it some of your ITEMAN, RASCAL, or XCALIBRE DAT files, and see what you
think.

Addendum: technical note

In this paper, I ran with the default ITEMAN configuration settings seen in
Figure 29.

Figure 29

8
Yonks: Australian slang meaning a long time (if not forever).
9
Full bottle means a person knows everything there is to know about a topic
(slang, probably Australian).

ITEMAN and Lertap 5, page 20.

 Appendix

You have come to read this appendix for one of two reasons: you love this
document, and don’t want it to end, or, more likely, you’re an Excel 2003 user
and have been presented with the “surprise” mentioned just after Figure 7 in the
text above.

The reason the surprise message appears is because Lertap has tried to activate
the Iteman1 macro, but Excel doesn’t know where this macro is. You need to tell
Excel that Iteman1 is found in a file named Lertap5MacroSetA.xla.

How?

Go up to the Excel menu bar, click on Tools, and do whatever is then necessary in
order to get the “Add-Ins” option to show, as pictured below:

Figure 30

Clicking on Add-Ins should result in something which bears a strong resemblance

to what’s seen in Figure 31.

ITEMAN and Lertap 5, page 21.

 Figure 31

Now make use of the Browse… button so that you can point Excel to the
Lertap5MacroSetA.xla macro file.

Where is this file likely to be? Try looking in the C:\ASC\Lertap56 folder – this
became the default Lertap folder for systems installed after November 2008. If
you installed Lertap prior to this date, you should pay a visit to www.assess.com
and download the latest “30 Day Trial Copy for use with Windows Excel 2002 and
2003”. Unzip the downloaded file, and then run the Lertap5.6.3.Setup.exe
program. This will install the latest Lertap as well as the latest version of the
macro file, Lertap5MacroSetA.xla.

Once you’ve browsed out to the Lertap5MacroSetA.xla file, Excel’s Add-Ins

available list ought to look something like Figure 32.

If it does, and if the box next to Lertap 5 Macro Set A is ticked as in Figure 32,
then you should be set. Click on the OK button (as in Figure 32) and Excel will
return to its main screen. At this point, return to the text, to Figure 7, and see
how you go. There’s a 50% chance that things will work okay, that is, Figure 8
will come into play as it should. If it doesn’t, exit Excel and then start up Lertap
again. Use the Excel menu bar to get to Tools / Add-Ins – does the Add-Ins
available list have Lertap 5 Macro Set A? If not, take a break. Then go back to
the start of this appendix and start again. Another option would be to click on
the Lelp option, after MacroMia9, as seen in Figure 7. Lelp’s explanation of how
to set up an Add-In is a bit more extensive.

If this still doesn’t do the trick, ask support@lertap.com for assistance.

ITEMAN and Lertap 5, page 22.

 Figure 32

ITEMAN and Lertap 5, page 23.