Plugging into the Java

Compiler
amonn McManus <emcmanus@google.com>
Christian Gruber <cgruber@google.com>

Speaker introduction: amonn McManus

At Google since 2011

Gmail servers initially

App Engine SDK now
20% time on Java Core Libraries (Guava)

Formerly at Sun then Oracle, on the JDK team

Author of two annotation processors

AutoValue (based on an idea by Kevin Bourrillion)

JavaFX Builder generator

JavaOne 2014

Speaker introduction: Christian Gruber

At Google since 2009

Test Mercenary
Mobile testing and development infrastructure
Java Core Librarian

focusing on Dependency Injection (guice, dagger), and testing

Formerly at Sun/JavaSoft (via Lighthouse) and Oracle Consulting

Co-author of
Dagger 1 and 2, dependency injection using annotation processors

auto-common, tools for easing annotation processor development

Created (with other Googlers) the Truth assertion/testing library

Open-source bigot
JavaOne 2014

Overview

What is an annotation processor?

Example: AutoValue
Example: Dagger
Write your own annotation processor
Q&A

JavaOne 2014

Reminder: Annotations
@SuppressWarnings(unchecked)
public class Foo {
@SafeVarargs
public void bar(Optional<String>... args) {...}
public void baz(@Nullable String s) {...}
}

JavaOne 2014

What is an annotation processor?

A way to extend the Java compiler
Standardized by JSR 269 in Java 6

javax.annotation.processing, javax.lang.model
supported by javac (JDK) and ecj (Eclipse)

Analyze Java code being compiled

Maybe introduce new errors and warnings
Maybe generate new Java code

JavaOne 2014

JavaOne 2014

Au am
to pl
Va e:
lu
e

Ex

Value Types
A value type is a class where:

properties never change (immutable)

instances with the same properties are interchangeable

Immutability is good!

easy to reason about

thread-safe

Many uses in Java, for example:

returning more than one value from a method

combining values for use in a map key or value

Map<Tuple3<String, Integer, Country>, Tuple2<Long, Long>> ?

JavaOne 2014

Value types (ideal simplicity)

public class Address {
public final String streetAddress;
public final int postCode;
public Address(
String streetAddress, int postCode) {
this.streetAddress = streetAddress;
this.postCode = postCode;
}
}
JavaOne 2014

Accessors and validation

public class Address {
private final String streetAddress;
private final int postCode;
public Address(String streetAddress, int postCode) {
this.streetAddress = Preconditions.checkNotNull(streetAddress);
this.postCode = postCode;
}
public String streetAddress() {
return streetAddress;
}
public int postCode() {
return postCode;
}
}

JavaOne 2014

equals, hashCode, toString

public class Address {
private final String streetAddress;
private final int postCode;
public Address(String streetAddress, int postCode) {
this.streetAddress = Preconditions.checkNotNull(streetAddress);
this.postCode = postCode;
}
public String streetAddress() {
return streetAddress;
}
public int postCode() {
return postCode;
}
@Override public boolean equals(Object o) {
if (o instanceof Address) {
Address that = (Address) o;
return this.streetAddress.equals(that.streetAddress)
&& this.postCode == that.postCode;
} else {
return false;
}
}
@Override public int hashCode() {
return Objects.hash(streetAddress, postCode);
}
@Override public String toString() {
return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}";
}
}

JavaOne 2014

equals, hashCode, toString

public class Address {
private final String streetAddress;
private final int postCode;
public Address(String streetAddress, int postCode) {
this.streetAddress = Preconditions.checkNotNull(streetAddress);
this.postCode = postCode;
}
public String streetAddress() {
return streetAddress;
}
public int postCode() {
return postCode;
}

postCode

@Override public boolean equals(Object o) {

if (o instanceof Address) {
Address that = (Address) o;
return this.streetAddress.equals(that.streetAddress)
&& this.postCode == that.postCode;
} else {
return false;
}
}
@Override public int hashCode() {
return Objects.hash(streetAddress, postCode);
}
@Override public String toString() {
return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}";
}
}

JavaOne 2014

AutoValue to the rescue

@AutoValue public abstract class Address {
public abstract String streetAddress();
public abstract int postCode();
public static Address create(
String streetAddress, int postCode) {
return new AutoValue_Address(
streetAddress, postCode);
}
}
JavaOne 2014

AutoValue generates subclass

class AutoValue_Address extends Address {
private final String streetAddress;
private final int postCode;
AutoValue_Address(String streetAddress, int postCode) {
...check streetAddress not null...
...assign fields...
}
@Override public String streetAddress() {...}
@Override public int postCode() {...}
@Override public boolean equals(Object o) {...}
@Override public int hashCode() {...}
@Override public String toString() {...}
}
JavaOne 2014

Annotation processing (1)

Foo.java

Bar.java

compile

@AutoValue

generate

AutoValue_Address.java

AutoValueProcessor

@AutoValue

Address.java

JavaOne 2014

Annotation processing (2)

AutoValue_Address.java

compile

No further
annotations

JavaOne 2014

Annotation processing (3)

Foo.java

Foo.class

Bar.java

Bar.class
generate
code

Address.java

Address.class

AutoValue_Address.java

AutoValue_Address.class

JavaOne 2014

o
D
em
JavaOne 2014

JavaOne 2014

Ex
D am
ag p
ge le
r :

Dagger
Dependency-injection framework using JSR 330
Directed Acyclic Graph... of classes and their dependencies.
Dagger 1.x

Created by Googlers, and ex-Googlers at Square

Christian Gruber, Jesse Wilson, Jake Wharton, and others...

Open source with contributions from Square, Google and others

Source code generation and compile-time analysis

Dagger 2.x

100% compile-time, via annotation processing and generated sources

originated at Google, with design oversight by Dagger 1 contributors.

Concept by Greg Kick and Christian Gruber, impl mainly by Greg Kick
JavaOne 2014

Dependency Injection
Having a class know how to obtain its collaborators is fragile

Hard to change implementation

Hard to unit-test

Pattern described by Martin Fowler:

http://www.martinfowler.com/articles/injection.html

Early examples:

Spring, PicoContainer, Apache HiveMind/Tapestry

Later examples:

Guice, CDI, Dagger

JavaOne 2014

Static dependencies crystallized in constructors

public class MailServer {
// ...
public MailServer() {
this.messageStore = new MessageStoreImpl();
this.userService = new UserServiceImpl();
// ...
}
}

Cannot test MailServer without invoking MessageStoreImpl, etc.

Cannot swap in alternate messaging, auth, etc.
Shared collaborators (singletons) end up requiring global statics
JavaOne 2014

Instead, declare the dependencies, and pass them in...

public class MailServer {
// ...

public MailServer(
MessageStore messageStore,
UserService userService) {
this.messageStore = messageStore;
this.userService = userService;
// ...
}
}
JavaOne 2014

Automation from annotation signals - no writing the wiring

public class MailServer {
// ...
@Inject
public MailServer(
MessageStore messageStore,
UserService userService) {
this.messageStore = messageStore;
this.userService = userService;
// ...
}
}
JavaOne 2014

Explicit configuration is defined using annotations...

@Module public class MailServerModule {
// Binding an implementation to an interface
@Provides MessageStore store(MessageStoreImpl impl) {
// MessageStoreImpl itself has @Inject signals.
return impl;
}
// Adapting non-DI-friendly code
@Provides UserService userService(DBConnection conn) {
// Type is not in our control and no @Inject signals.
return new UserServiceImpl.create(conn);
}
}
JavaOne 2014

Access to the graph defined by @Component

@Component public interface Services {
MailServer mailServer();
NotificationServer notificationServer();
}

Annotations provide the signals

Graph analysis begins at annotated interfaces
All the wiring or glue code for the graph is generated
@Component interfaces implementation generated with a builder
for configuration
Plain old java code
JavaOne 2014

Explicit configuration is defined using annotations...

@Module public class MailServerModule {
// Binding an implementation to an interface
@Provides MessageStore store(MessageStoreImpl impl) {
// MessageStoreImpl itself has @Inject signals.
return impl;
}
// Adapting non-DI-friendly code
@Provides UserService userService(DBConnection conn) {
// Type is not in our control and no @Inject signals.
return new UserServiceImpl.create(conn);
}
}
JavaOne 2014

Why Annotation Processors for D-I?

Guice, Spring, etc. work well, so why processors and code-gen?
Performance

Reflection is expensive in some environments, such as Android

Graph Validation is work - can affect startup times

Developer productivity:

Errors at compile-time vs. load time or even later improves velocity

Generated code less "magical" and can be seen and reasoned about

Design

Knowing the structure of the code lets us build cleaner approaches

than the "magic map" of Injector/Container frameworks.
JavaOne 2014

o
D
em
Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues
JavaOne 2014

JavaOne 2014

Yo rite
O ur
w
n

What processors can and cannot see

Processors can see the structure of your code:

Class names, inheritance, generics

Method names, parameter types, return types, generics
Field names, types, compile-time constant values

Processors cannot see the contents of the code

Static initialization blocks

Method bodies
Initializer expressions

JavaOne 2014

What processors can and cannot do

Processors can do quite a few things, such as

generate new Java source code to be compiled

generate other files (XML, META-INF/services, arbitrary text)
perform analysis and emit warnings and errors
associate the errors with specific source elements

Processors cannot, however

modify the code of existing classes

including source they generate once written

introduce new fields or methods into existing classes

introduce new nested classes
JavaOne 2014

"Annotation" processors
Annotation processors are usually associated with annotations,
obviously
But, you can also write a processor that analyzes all input classes,
whether annotated or not

Return Collections.singleton("*") from getSupportedAnnotationTypes()

JavaOne 2014

Defining an annotation
import java.lang.annotation.*;
@Retention(RetentionPolicy.SOURCE)
@Target(ElementType.TYPE)
// @Documented
public @interface MyAnnotation {
String value() default "";
}
// @MyAnnotation class Foo {...}
// @MyAnnotation("bar") interface Baz {...}
// @MyAnnotation(value = "buh") enum Wibble {...}

JavaOne 2014

Outline of a processor (1)

import javax.annotation.processing.*;
import javax.lang.model.*;
@AutoService(Processor. class)
public class MyProcessor extends AbstractProcessor {
@Override public Set<String> getSupportedAnnotationTypes() {
return ImmutableSet.of(MyAnnotation. class.getName());
}
@Override public SourceVersion getSupportedSourceVersion() {
return SourceVersion.latestSupported();
}
...
}

JavaOne 2014

Outline of a processor (2)

import javax.annotation.processing.*;
import javax.lang.model.*;
@AutoService(Processor. class)
public class MyProcessor extends AbstractProcessor {
...
@Override public boolean process(
Set<? extends TypeElement> annotations,
RoundEnvironment roundEnv) {
Collection<? extends Element> annotatedElements =
roundEnv.getElementsAnnotatedWith(MyAnnotation. class);
handle(annotatedElements);
return false ;
}
}
JavaOne 2014

Claiming annotations
An annotation processor can say that it "supports" one or more
annotations (getSupportedAnnotationTypes)
Its process method will be called only for program elements
where those annotations appear
If it returns true, it has "claimed" the annotations and a later
processor that also supports them will not be called
We recommend never claiming an annotation

You don't know what that later processor is or whether it should run

JavaOne 2014

API tips: Types and Elements

A lot of useful functionality is contained in javax.lang.model.util.
{Types,Elements}
If you can't figure out how to do something, check these interfaces
to see if they hold the solution
Get an instance of either in your process method or any method it
calls, via the inherited processingEnv field:
@Override public boolean process(...) {
Types typeUtils =
processingEnv.getTypeUtils();
Elements elementUtils =
processingEnv.getElementUtils();
JavaOne 2014

API Tips: ElementFilter

@Override public boolean process(
Set<? extends TypeElement> annotations,
RoundEnvironment roundEnv) {
Collection<? extends Element> annotatedElements =
roundEnv.getElementsAnnotatedWith(MyAnnotation. class);
List<ExecutableElement> annotatedMethods =
ElementFilter.methodsIn(annotatedElements);
handle(annotatedMethods);
return false ;
}

JavaOne 2014

API tips: TypeMirror and TypeElement

The distinction between TypeElement and TypeMirror is subtle
In practice, use whichever one the API gives you, or convert to the
other if the method you need is there
TypeMirror TypeElement:
Types typeUtils = processingEnv.getTypeUtils();
TypeElement typeElement =
(TypeElement) typeUtils.asElement(typeMirror);

TypeElement TypeMirror:
TypeMirror typeMirror = typeElement.asType();

or
DeclaredType typeMirror = typeUtils.getDeclaredType(typeElement);

JavaOne 2014

API tips: Upstream errors can cause processor failure

Processed code may be missing literal elements, be missing imports,
be in a broken state, etc.

Very strange results including core javac types in ClassCastExceptions or

NullPointerExceptions deep within the compiler.
Upstream compile errors are masked by these exceptions.

Auto-common provides SuperficialValidation.validateElements(...)

Simple sanity check for each Element and contents

If validateElements() returns false, two options:

skip processing supplied elements (if done in a loop)

return from the processor immediately

JavaOne 2014

API tips: Gotchas

TypeMirror.equals() is not a reliable comparison. Should use:

Types.isSameType(mirror1, mirror2) (if available)

MoreTypes.equal(mirror1, mirror2) (if in a static context)

TypeMirror instanceof checks are generally incorrect

MoreTypes.asDeclared(typeMirror) converts correctly or throws IAE

MoreTypes.asArray(...)... etc. - converters for all TypeMirror kinds.

JavaOne 2014

API tips: Googles auto-common utilities

MoreTypes and MoreElements

Wrappers for equivalence

Static methods for conversion and comparison

SuperficialValidation

For simple sanity checking of elements before processing

AnnotationMirrors and AnnotationValues

convenience methods and Equivalence wrappers

static utilities to handle default values

Types/Elements have useful methods, but are instances. Auto

common utilities provides many static utility method equivalents.
Note: AnnotationValues and AnnotationMirrors pending extraction from Dagger 2.
JavaOne 2014

Generating code: JavaWriter

Build up your class programmatically.
Refer to elements without using stringly-typed references.
Write to any Appendable.
JavaWriter javaWriter = JavaWriter.inPackage("some.package");
ClassWriter klass = javaWriter.addClass("SomeType");
VariableWriter field = klass.addField(String.class, "foo");
field.addModifiers(PRIVATE, FINAL);
ConstructorWriter constructor = klass.addConstructor();
VariableWriter p0 = constructor.addParameter(Key.class, "key");
constructor.body()
.addSnippet("this.%s = %s.getFormat();", field.name(), p0.name());
javaWriter.write(appendable);

JavaOne 2014

Generating code: JavaWriter

Get clean, organized and readable code out.
package some.package;
import java.security.Key;
class SomeType {
private final String foo;
SomeType(Key key) {
this.foo = key.getFormat();
}
}
Note: JavaWriter 3 is pending review with Square and extraction from Dagger2. JavaWriter 2 is available.
JavaOne 2014

Generating code: JavaWriter

Pros:

Programmatic creation of code

Reduce errors with cross referencing
Automatic handling of imports and type shortening
Very useful in loops where each loop touches many parts of the code
Some built-in validation based on code structure

some erroneous things are simply impossible to write

Cons:

Code that generates code can look quite different in shape than output

JavaOne 2014

Generating code: templates

#foreach ($p in $props)
@Override ${p.access}${p.type} ${p}() {
#if ($p.kind == "ARRAY")
#if ($p.nullable)
return $p == null ? null : ${p}.clone();
#else
return ${p}.clone();
#end
#else
return $p; @Override public int[] postCodes() {
return postCodes == null ? null : postCodes.clone();
#end
}
}
#end
JavaOne 2014

Generating code: templates

Apache Velocity is a good choice of template engine

Supported in all major IDEs (even Emacs)

Directives clearly distinguishable from Java code snippets

In AutoValue we do a post-processing step to remove superfluous

spaces and blank lines, just so the generated code looks nicer

JavaOne 2014

Testing Processors - Unit testing environment

Unit testing still requires javac environment, for Types/Elements
Compile-testing has a CompilationRule suitable for JUnit4
import com.google.testing.compile.CompilationRule;
@RunWith(JUnit4.class)
public class SomeProcessorTest {
@Rule public final CompilationRule compilationRule = new CompilationRule();
private Elements elements;
private Types types;
@Before public void setUp() {
this.elements = compilationRule.getElements();
this.types = compilationRule.getTypes();
}
JavaOne 2014

Testing Processors - Failing integration tests

Need to test failing compiles without failing the outer build
Compile-testing has assertions based on the Truth library
These assertions can variously:

assert about javac runs success or failure

be configured to run arbitrary Processor instances
run against source from files or strings
execute and store results in-memory
assert about errors with specific message contents and locations
assert against contents of generated files
assert against contents of generated java source, comparing AST

JavaOne 2014

Testing Processors - Failing integration test assertions

JavaFileObject file0 = JavaFileObjects.forSourceLines("test.Foo",
"package test",
"class Foo {";
...
);
JavaFileObject file1 =
JavaFileObjects.forResource("expected/SomeFile.java");
assert_().about(JavaSourcesSubject.javaSources())
.that(ImmutableSet.of(file0, file1))
.processedWith(new BlahProcessor())
.failsToCompile()
.withErrorContaining"Invalid use of Annotation @Blah")
.in(javaFileObject).onLine(7);
JavaOne 2014

Testing Processors - Succeeding integration tests

For successful compilation, two approaches:

Use compile-testing to compare against a golden file, or

Functionally test that the generated code behaves as it should

Golden file tests:

Brittle - small changes can require a lot of change in test files/code

Useful to demonstrate expected output

Functional tests:

Exercise more code paths and use-cases with less verbosity

JavaOne 2014

Testing Processors - Overall strategy

Use Functional tests to cover the bulk of use-cases
Use CompilationRule to unit-test the internals of your Processor
Use compile-testing assertions to test expected errors
Use compile-testing assertions to test a small number of golden
expectation files, for illustration of processor output

JavaOne 2014

y
m
ar
Su
m

JavaOne 2014

Summary
Annotation processors are a powerful way to plug in to javac
An ecosystem is evolving to ease writing custom processors

JavaWriter, auto-common, compile-testing, @AutoService

Several useful annotation processors exist presently

AutoValue, Dagger

Lots of advantages to annotation processors

Early error-checking
Performance improvements
Viewable generated source code

JavaOne 2014

Annotation Processor Resources

Check out existing annotation processors:

Google Auto: https://github.com/google/auto

Dagger 1: https://github.com/square/dagger
Dagger 2: https://github.com/google/dagger

Projects that may help:

JavaWriter: https://github.com/square/javawriter
CompileTesting: https://github.com/google/compile-testing
Google Auto common utilities: https://github.
com/google/auto/tree/master/common

JavaOne 2014

&A
Q

JavaOne 2014

Manual IDE configuration

Make a jar file with your processor and all of its dependencies
Ensure it has META-INF/services/javax.annotation.processing.
Processor

@AutoService(Processor.class) is the easiest way

NetBeans:

Project Properties > Libraries > Processor > Add JAR/Folder

Eclipse:

Properties > Java Compiler > Annotation Processing

Enable Annotation Processing + Factory Path

JavaOne 2014