Code Coverage On IOS

Code Coverage on iOS
Richard Buckle
Sailmaker Systems Ltd
Tuesday, 2 October 12
What well cover

What is the idea here?
What are the benefits?
Good and bad ways to use it
Taming Xcodes Code Coverage tools
Integration with Jenkins
What is the idea here?

You exercise your app in some way
- Unit tests
- Automated tests like Cucumber
- Manually
Then you see what code was executed by
that exercise, and what wasnt
Sounds a lot like profiling

Yes, both use the same instrumentation
Profiling focusses on performance
- based on release configs
Coverage focusses on test completeness
- based on debug configs
What are the benefits?

Identifying something you forgot to test
Identifying code that was added without
proper tests
Retro-fitting tests to existing code

Identifying dead wood in the codebase
- But be careful there!
Dead code?
Like all metrics, Code

Coverage is a good
servant and a bad
master
How to make your life

miserable with it
Obsess about Coverage
Put it on the Jenkins dashboard
Expose it to management
Use it as a stick to beat yourself with
- or worse, your team
How to make your life

miserable with it
Require an arbitrary %age of Coverage
Write meaningless tests just to get the
Coverage %age up
and then stop adding tests as soon as

you reach that %age
Test coverage is a useful tool for finding

untested parts of a codebase. Test coverage is
of little use as a numeric statement of how
good your tests are.
Martin Fowler
http://martinfowler.com/bliki/TestCoverage.html
10
If a part of your test suite is weak in a way

that coverage can detect, its likely also weak
in a way coverage cant detect.
Brian Marick
http://www.exampler.com/testing-com/writings/
coverage.pdf
11
How to make Coverage

work for you
Write good tests! Ignore coverage to start
with!
Use the coverage results to point out areas

you overlooked
Use your experience and judgement to

decide what else needs to be covered
Never slavishly write a test merely to get the

Coverage %age up
12
How to make Coverage

work for you
You may decide that a critical module
requires 100% coverage, in which case go

for it
You may decide that some 3rd-party code
isnt worth trying to cover at all: thats fine

too
The coverage tools can exclude code by

subdirectory on disk
13
So how do I actually do
this?
14
The basic workflow

Instrument a target in Xcode
Building emits .gcno files
Running emits .gcda files
- if the app exits cleanly
Tools process these into human-readable
output
and maybe Jenkins-readable metrics
15
For minimum pain, use

Xcode 4.5
Ill also give a workaround for Xcode 4.4.x
16
Finding the output

Xcode buries this in the intermediate
objects directory under

~/Library/.../DerivedData/...
Fortunately it sets an env var
$OBJECT_FILE_DIR_normal for this
open ${OBJECT_FILE_DIR_normal}
Ill come back to taming Xcode later
17
Viewing the output

CLI: gcov
GUI: CoverStory
Shell and Python scripts using gcov
18
gcov
Fine if you like working at such a low level
man gcov
Can do pretty much anything you want
19
CoverStory
Apache Licence 2.0
http://code.google.com/p/coverstory/
Scriptable, can emit customisable HTML
Probably the easiest for most people
Best option for interactive use
Its icon is different
20
Non-feasible code
CoverStory supports marking of nonfeasible code via comments, such as:
Sanity checks
Branching based on runtime platform
http://code.google.com/p/coverstory/wiki/
NonFeasibleCode
Dont abuse this just to get the metrics up!

21
Shell script
Add a Run Script build phase

OK if you like that kind of thing
# Run gcov on the framework getting tested

if [ "${CONFIGURATION}" = 'Coverage' ]; then
OBJ_DIR=${OBJECT_FILE_DIR_normal}/${CURRENT_ARCH}
mkdir -p Coverage
pushd Coverage
find "${OBJ_DIR}" -name *.gcda -exec gcov -o "${OBJ_DIR}" {} \;
2>/tmp/gconv-stderr | egrep "^File|^Lines" | sed -E "s@File
'$SRCROOT/@@;s@(\.[a-zA-Z])'@\1: @;s@Lines executed:([0-9.%]+)
of ([0-9]+)@\1 (\2)@" | paste -d" " - - | egrep -v "^File '" |
sed -E "s@^([^:]+):([^(]*)($[^)]+$)@\2:\1\3@" | sort -n | sed
-E "s@^([^:]+):([^(]*)($[^)]+$)@\2:\1\3@"; cat /tmp/gconvstderr | grep -v "version.*, prefer.*"; rm /tmp/gconv-stderr
popd
fi
22
Python
gcovr from https://software.sandia.gov/
trac/fast/downloader
Can emit Cobertura XML for Jenkins

Best option for Jenkins integration
Bit of a heavyweight installation
23
Taming Xcodes
Coverage tools
Yes, it can be done, on both simulator and
devices!
24
Unit test coverage

In your unit test target, set Generate Test
Coverage Files and Instrument Program
Flow
Same for your main target but only for debug
Thats all!
25
Running Manually
Duplicate your Debug config and call it Coverage
Set Generate Test Coverage Files and Instrument

Program Flow only for the Coverage config
Duplicate your scheme and call it Coverage too

Edit the Coverage scheme to use the Coverage config
everywhere
26
A small problem
Coverage data is only emitted when the
app exits cleanly
As of iOS 4+ this never normally happens!

Quitting the simulator used to work but
not any more
So whats a poor developer to do?

27
Two solutions
Have the coverage config opt out of
backgrounding
Doesnt let you measure background code
Have a coverage-only backdoor GUI that

calls exit(0)
Make sure you dont ship with that! Its

grounds for rejection
28
Opting out of backgrounding
Turn on Info.plist preprocessing and define,

say, GCOV for the coverage config only
<key>UIApplicationExitsOnSuspend</key>
#ifdef GCOV
<true/>
#else
<false/>
#endif
29
Backdoor exit UI
Define a preprocessor macro for the
Coverage target only and use it to govern

your exit code
#if GCOV
exit(0);
#endif
30
Running on the device
On the device, the app is running in a sandbox

so cant write .gcda files to the global location
We can use the environment variables

GCOV_PREFIX and GCOV_PREFIX_STRIP to
fix this
<http://gcc.gnu.org/onlinedocs/gcc/Cross_002dprofiling.html>
The correct path can only be determined at

run-time
31
Setting env vars at runtime

int main(int argc, char *argv[])
{
#if GCOV_IOS // Device only
@autoreleasepool {
const char *prefix = "GCOV_PREFIX";
const char *prefixValue = [[NSHomeDirectory()
stringByAppendingPathComponent: @"tmp/gcov"]
cStringUsingEncoding: NSASCIIStringEncoding];
const char *prefixStrip = "GCOV_PREFIX_STRIP";
const char *prefixStripValue = "5";
setenv(prefix, prefixValue, 1);
setenv(prefixStrip, prefixStripValue, 1);
! }
#endif
!
!
!
}
@autoreleasepool {
! int retVal = UIApplicationMain(argc, argv, nil, nil);
! return retVal;
}
32
Getting the files off the

device
Now we can retrieve

the apps sandbox from
Xcodes Organiser
Difficult to automate
unfortunately
The .gcda files must

be copied into the
corresponding path
under the real
DerivedData folder
33
Workaround for Xcode 4.4.x
The instrumentation in Xcode 4.4.x
erroneously tries to use fopen$UNIX2003

and fwrite$UNIX2003, which dont exist on
iOS
Simply implement them yourself:

#if GCOV
FILE *fopen$UNIX2003(const char *filename, const char *mode) {
return fopen(filename, mode);
}
size_t fwrite$UNIX2003(const void *ptr, size_t size,

size_t nitems, FILE *stream) {
return fwrite(ptr, size, nitems, stream);
}
#endif
34
Jenkins integration
Build the project in the usual way, making sure the tests
run:
xcodebuild -sdk iphonesimulator -project
MyProject.xcodeproj -target MyProjectTests
-configuration Debug clean build
TEST_AFTER_BUILD=YES
Pass it to gcovr to generate a Cobertura report:

ROOT=`pwd`
/path/to/gcovr -r $ROOT --exclude
'.*Developer.*' --exclude
'.*MyProjectTests.*' --xml > MyProject/
coverage.xml
35
General References
Martin Fowler
http://martinfowler.com/bliki/TestCoverage.html
Brian Marick
http://www.exampler.com/testing-com/writings/
coverage.pdf
How much testing do you need?
http://www.developertesting.com/archives/
month200705/20070504-000425.html
36
Xcode references
Be aware that most of the online articles are now

out of date. That said:
Xcode 4.3 and Jenkins Code Test Coverage Updated
How to setup quality metrics on your Jenkins job?
How to get your code coverage right?
http://jonboydell.blogspot.ch/2012/06/xcode-43-and-jenkinscode-test-coverage.html
http://blog.octo.com/en/jenkins-quality-dashboard-iosdevelopment/
http://blog.octo.com/en/ios-development-right-code-coverage/
37
Q&A
richardb@sailmaker.co.uk
Twitter @RichardBuckle
app.net @RichardBuckle
38

Code Coverage On IOS

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Code Coverage On IOS

Hochgeladen von

Copyright:

Verfügbare Formate

Code Coverage on iOS

What well cover

What is the idea here?

Sounds a lot like profiling

What are the benefits?

Retro-fitting tests to existing code

Like all metrics, Code

How to make your life

How to make your life

and then stop adding tests as soon as

Test coverage is a useful tool for finding

If a part of your test suite is weak in a way

How to make Coverage

Use the coverage results to point out areas

Use your experience and judgement to

Never slavishly write a test merely to get the

How to make Coverage

requires 100% coverage, in which case go

You may decide that some 3rd-party code

isnt worth trying to cover at all: thats fine

The coverage tools can exclude code by

The basic workflow

and maybe Jenkins-readable metrics

For minimum pain, use

Finding the output

objects directory under

Fortunately it sets an env var

$OBJECT_FILE_DIR_normal for this

Viewing the output

Dont abuse this just to get the metrics up!

Add a Run Script build phase

# Run gcov on the framework getting tested

Can emit Cobertura XML for Jenkins

Unit test coverage

Same for your main target but only for debug

Duplicate your Debug config and call it Coverage

Set Generate Test Coverage Files and Instrument

Duplicate your scheme and call it Coverage too

As of iOS 4+ this never normally happens!

So whats a poor developer to do?

Doesnt let you measure background code

Have a coverage-only backdoor GUI that

Make sure you dont ship with that! Its

Opting out of backgrounding

Turn on Info.plist preprocessing and define,

Coverage target only and use it to govern

Running on the device

On the device, the app is running in a sandbox

We can use the environment variables

The correct path can only be determined at

Setting env vars at runtime

Getting the files off the

Now we can retrieve

The .gcda files must

Workaround for Xcode 4.4.x

The instrumentation in Xcode 4.4.x

erroneously tries to use fopen$UNIX2003

Simply implement them yourself:

size_t fwrite$UNIX2003(const void *ptr, size_t size,

Pass it to gcovr to generate a Cobertura report:

How much testing do you need?

Be aware that most of the online articles are now