Sie sind auf Seite 1von 58
Plugging into the Java Compiler Éamonn McManus < emcmanus @g oo g le.com > Christian

Plugging into the Java Compiler

Plugging into the Java Compiler Éamonn McManus < emcmanus @g oo g le.com > Christian Gruber

Éamonn McManus <emcmanus@google.com> Christian Gruber <cgruber@google.com>

Speaker introduction: Éamonn McManus ● At Google since 2011 ○ Gmail servers initially ○ App

Speaker introduction: Éamonn McManus

● At Google since 2011

○ Gmail servers initially

○ App Engine SDK now

○ 20% time on Java Core Libraries (Guava)

● Formerly at Sun then Oracle, on the JDK team

● Author of two annotation processors

○ AutoValue (based on an idea by Kevin Bourrillion)

○ JavaFX Builder generator

JavaOne 2014

Speaker introduction: Christian Gruber ● At Google since 2009 ○ “Test Mercenary” ○ Mobile testing

Speaker introduction: Christian Gruber

● At Google since 2009

○ “Test Mercenary”

○ Mobile testing and development infrastructure

○ Java Core Librarian

■ focusing on Dependency Injection (guice, dagger), and testing

● Formerly at Sun/JavaSoft (via Lighthouse) and Oracle Consulting

● Co-author of

Dagger 1 and 2, dependency injection using annotation processors

○ auto-common, tools for easing annotation processor development

● Created (with other Googlers) the Truth assertion/testing library

● Open-source bigot

JavaOne 2014

Overview ● What is an annotation processor? ● Example: AutoValue ● Example: Dagger ● Write

Overview

● What is an annotation processor?

● Example: AutoValue

● Example: Dagger

● Write your own annotation processor

● Q&A

JavaOne 2014

Reminder: Annotations @SuppressWarnings(“unchecked”) public class Foo { @SafeVarargs public void bar

Reminder: Annotations

@SuppressWarnings(“unchecked”) public class Foo { @SafeVarargs public void bar(Optional<String>

args) {

public void baz(@Nullable String s) {

}

}

}

JavaOne 2014

What is an annotation processor? ● A way to extend the Java compiler ● Standardized

What is an annotation processor?

● A way to extend the Java compiler

● Standardized by JSR 269 in Java 6

○ javax.annotation.processing, javax.lang.model

○ supported by javac (JDK) and ecj (Eclipse)

● Analyze Java code being compiled

● Maybe introduce new errors and warnings

● Maybe generate new Java code

JavaOne 2014

JavaOne 2014
JavaOne 2014

JavaOne 2014

Value Types ● A value type is a class where: ○ properties never change (immutable)

Value Types

● A value type is a class where:

○ properties never change (immutable)

○ instances with the same properties are interchangeable

● Immutability is good!

○ easy to reason about

○ thread-safe

● Many uses in Java, for example:

○ returning more than one value from a method

○ combining values for use in a map key or value

■ Map<Tuple3<String, Integer, Country>, Tuple2<Long, Long>> ?

JavaOne 2014

Value types (ideal simplicity) public class Address { public final String streetAddress ; public final

Value types (ideal simplicity)

public class Address { public final String streetAddress; public final int postCode;

public Address( String streetAddress, int postCode) { this.streetAddress = streetAddress; this.postCode = postCode;

}

}

JavaOne 2014

Accessors and validation public class Address { private final String streetAddress ; private final int

Accessors and validation

public class Address { private final String streetAddress; private final int postCode;

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode;

}

public String streetAddress() { return streetAddress;

}

public int postCode() { return postCode;

}

}

JavaOne 2014

equals, hashCode, toString public class Address {   private final String streetAddress ; private final

equals, hashCode, toString

public class Address {

 

private final String streetAddress; private final int postCode;

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode;

}

public String streetAddress() { return streetAddress;

}

public int postCode() { return postCode;

}

@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false;

}

}

@Override public int hashCode() { return Objects.hash(streetAddress, postCode);

}

@Override public String toString() { return "Address{streetAddress="+ streetAddress+ ", postCode=" + postCode + "}";

}

}

JavaOne 2014

equals, hashCode, toString public class Address { private final String streetAddress ; private final int

equals, hashCode, toString

public class Address {

private final String streetAddress; private final int postCode;

String streetAddress ; private final int postCode ; public Address (String streetAddress, int postCode) {

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode;

}

public String streetAddress() { return streetAddress;

}

public int postCode() { return postCode;

}

@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false;

}

}

@Override public int hashCode() { return Objects.hash(streetAddress, postCode);

}

@Override public String toString() { return "Address{streetAddress="+ streetAddress+ ", postCode=" + postCode + "}";

}

}

postCode

+ streetAddress + ", postCode=" + postCode + "}" ; } } postCode JavaOne 2014

JavaOne 2014

AutoValue to the rescue @AutoValue public abstract class Address { public abstract String streetAddress ();

AutoValue to the rescue

@AutoValue public abstract class Address { public abstract String streetAddress(); public abstract int postCode();

public static Address create( String streetAddress, int postCode) { return new AutoValue_Address( streetAddress, postCode);

}

}

JavaOne 2014

AutoValue generates subclass class AutoValue_Address extends Address { private final String streetAddress ; private

AutoValue generates subclass

class AutoValue_Address extends Address { private final String streetAddress; private final int postCode;

AutoValue_Address(String streetAddress, int postCode) {

check

streetAddress not null

assign

fields

}

@Override public String streetAddress() { } @Override public int postCode() { } @Override public boolean equals(Object o) { } @Override public int hashCode() { } @Override public String toString() { }

}

JavaOne 2014

Annotation processing (1) Foo.java Bar.java @AutoValue Address.java @AutoValue AutoValue_Address.java ⇒ generate

Annotation processing (1)

Foo.java Bar.java @AutoValue Address.java
Foo.java
Bar.java
@AutoValue
Address.java
@AutoValue AutoValue_Address.java ⇒ generate compile AutoValueProcessor JavaOne 2014
@AutoValue
AutoValue_Address.java
⇒ generate
compile
AutoValueProcessor
JavaOne 2014
Annotation processing (2) compile No further annotations AutoValue_Address.java JavaOne 2014

Annotation processing (2)

Annotation processing (2) compile No further annotations AutoValue_Address.java JavaOne 2014
compile No further annotations
compile
No further
annotations
AutoValue_Address.java

AutoValue_Address.java

JavaOne 2014

Annotation processing (3) Foo.java Foo.class Bar.java Bar.class generate code Address.java Address.class

Annotation processing (3)

Foo.java Foo.class Bar.java Bar.class generate code Address.java Address.class AutoValue_Address.java
Foo.java
Foo.class
Bar.java
Bar.class
generate
code
Address.java
Address.class
AutoValue_Address.java
AutoValue_Address.class

JavaOne 2014

JavaOne 2014
JavaOne 2014

JavaOne 2014

JavaOne 2014
JavaOne 2014

JavaOne 2014

Dagger ● Dependency-injection framework using JSR 330 ● Directed Acyclic Graph ● Dagger 1.x of

Dagger

● Dependency-injection framework using JSR 330

● Directed Acyclic Graph

● Dagger 1.x

of classes and their dependencies.

○ Created by Googlers, and ex-Googlers at Square

■ Christian Gruber, Jesse Wilson, Jake Wharton, and others

○ Open source with contributions from Square, Google and others

○ Source code generation and compile-time analysis

Dagger 2.x

○ 100% compile-time, via annotation processing and generated sources

○ originated at Google, with design oversight by Dagger 1 contributors.

■ Concept by Greg Kick and Christian Gruber, impl mainly by Greg Kick

JavaOne 2014

Dependency Injection ● Having a class know how to obtain its collaborators is fragile ○

Dependency Injection

● Having a class know how to obtain its collaborators is fragile

○ Hard to change implementation

○ Hard to unit-test

● Pattern described by Martin Fowler:

● Early examples:

○ Spring, PicoContainer, Apache HiveMind/Tapestry

● Later examples:

○ Guice, CDI, Dagger

JavaOne 2014

Static dependencies crystallized in constructors public class MailServer { // public MailServer () { this

Static dependencies crystallized in constructors

public class MailServer { // public MailServer() { this.messageStore = new MessageStoreImpl(); this.userService = new UserServiceImpl(); //

}

}

● Cannot test MailServer without invoking MessageStoreImpl, etc.

● Cannot swap in alternate messaging, auth, etc.

● Shared collaborators (singletons) end up requiring global statics

JavaOne 2014

Instead, declare the dependencies, and pass them in public class MailServer { // public MailServer(

Instead, declare the dependencies, and pass them in

public class MailServer { //

public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; //

}

}

JavaOne 2014

Automation from annotation signals - no writing the wiring public class MailServer { // @Inject

Automation from annotation signals - no writing the wiring

public class MailServer { //

@Inject public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; //

}

}

JavaOne 2014

Explicit configuration is defined using annotations @Module public class MailServerModule { // Binding an

Explicit configuration is defined using annotations

@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl;

}

// Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn);

}

}

JavaOne 2014

Access to the graph defined by @Component @Component public interface Services { MailServer mailServer();

Access to the graph defined by @Component

@Component public interface Services { MailServer mailServer(); NotificationServer notificationServer();

}

● Annotations provide the signals

● Graph analysis begins at annotated interfaces

● All the “wiring” or “glue” code for the graph is generated

● @Component interfaces’ implementation generated with a builder for configuration

● Plain old java code

JavaOne 2014

Explicit configuration is defined using annotations @Module public class MailServerModule { // Binding an

Explicit configuration is defined using annotations

@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl;

}

// Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn);

}

}

JavaOne 2014

Why Annotation Processors for D-I? ● Guice, Spring, etc. work well, so why processors and

Why Annotation Processors for D-I?

● Guice, Spring, etc. work well, so why processors and code-gen?

● Performance

○ Reflection is expensive in some environments, such as Android

○ Graph Validation is work - can affect startup times

● Developer productivity:

○ Errors at compile-time vs. load time or even later improves velocity

○ Generated code less "magical" and can be seen and reasoned about

● Design

○ Knowing the structure of the code lets us build cleaner approaches than the "magic map" of Injector/Container frameworks.

JavaOne 2014

Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues JavaOne 2014
Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues
Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues
Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues

Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues

Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues

JavaOne 2014

JavaOne 2014
JavaOne 2014

JavaOne 2014

What processors can and cannot see ● Processors can see the structure of your code:

What processors can and cannot see

● Processors can see the structure of your code:

○ Class names, inheritance, generics

○ Method names, parameter types, return types, generics

○ Field names, types, compile-time constant values

● Processors cannot see the contents of the code

○ Static initialization blocks

○ Method bodies

○ Initializer expressions

JavaOne 2014

What processors can and cannot do ● Processors can do quite a few things, such

What processors can and cannot do

● Processors can do quite a few things, such as

○ generate new Java source code to be compiled

○ generate other files (XML, META-INF/services, arbitrary text)

○ perform analysis and emit warnings and errors

○ associate the errors with specific source elements

● Processors cannot, however

○ modify the code of existing classes

■ including source they generate once written

○ introduce new fields or methods into existing classes

○ introduce new nested classes

JavaOne 2014

"Annotation" processors ● Annotation processors are usually associated with annotations, obviously ● But,

"Annotation" processors

● Annotation processors are usually associated with annotations, obviously

● But, you can also write a processor that analyzes all input classes, whether annotated or not

○ Return Collections.singleton("*") from getSupportedAnnotationTypes()

JavaOne 2014

Defining an annotation import java.lang.annotation.*; @Retention(RetentionPolicy.SOURCE) @Target(ElementType.TYPE) //

Defining an annotation

import java.lang.annotation.*;

@Retention(RetentionPolicy.SOURCE) @Target(ElementType.TYPE) // @Documented public @interface MyAnnotation { String value() default "";

}

// @MyAnnotation class Foo {

// @MyAnnotation("bar") interface Baz {

// @MyAnnotation(value = "buh") enum Wibble {

}

}

}

JavaOne 2014

Outline of a processor (1) import javax.annotation.processing.*; import javax.lang.model.*; @AutoService(Processor.

Outline of a processor (1)

import javax.annotation.processing.*; import javax.lang.model.*;

@AutoService(Processor. class) public class MyProcessor extends AbstractProcessor { @Override public Set<String> getSupportedAnnotationTypes() { return ImmutableSet.of(MyAnnotation. class.getName());

}

@Override public SourceVersion getSupportedSourceVersion() { return SourceVersion.latestSupported();

}

}

JavaOne 2014

Outline of a processor (2) import javax.annotation.processing.*; import javax.lang.model.*; @AutoService(Processor.

Outline of a processor (2)

import javax.annotation.processing.*; import javax.lang.model.*;

@AutoService(Processor. class) public class MyProcessor extends AbstractProcessor {

@Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class); handle(annotatedElements); return false ;

}

}

JavaOne 2014

Claiming annotations ● An annotation processor can say that it "supports" one or more annotations

Claiming annotations

● An annotation processor can say that it "supports" one or more annotations (getSupportedAnnotationTypes)

● Its process method will be called only for program elements where those annotations appear

● If it returns true, it has "claimed" the annotations and a later processor that also supports them will not be called

● We recommend never claiming an annotation

○ You don't know what that later processor is or whether it should run

JavaOne 2014

API tips: Types and Elements ● A lot of useful functionality is contained in javax.lang.model.util.

API tips: Types and Elements

● A lot of useful functionality is contained in javax.lang.model.util. {Types,Elements}

● If you can't figure out how to do something, check these interfaces to see if they hold the solution

● Get an instance of either in your process method or any method it calls, via the inherited processingEnv field:

@Override public boolean process( Types typeUtils = processingEnv.getTypeUtils(); Elements elementUtils = processingEnv.getElementUtils();

)

{

JavaOne 2014

API Tips: ElementFilter @Override public boolean process ( Set<? extends TypeElement> annotations,

API Tips: ElementFilter

@Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class); List<ExecutableElement> annotatedMethods = ElementFilter.methodsIn(annotatedElements); handle(annotatedMethods); return false ;

}

JavaOne 2014

API tips: TypeMirror and TypeElement ● The distinction between TypeElement and TypeMirror is subtle ●

API tips: TypeMirror and TypeElement

● The distinction between TypeElement and TypeMirror is subtle

● In practice, use whichever one the API gives you, or convert to the other if the method you need is there

● TypeMirror → TypeElement:

Types typeUtils = processingEnv.getTypeUtils(); TypeElement typeElement = (TypeElement) typeUtils.asElement(typeMirror);

● TypeElement → TypeMirror:

TypeMirror typeMirror = typeElement.asType();

or

DeclaredType typeMirror = typeUtils.getDeclaredType(typeElement);

JavaOne 2014

API tips: Upstream errors can cause processor failure ● Processed code may be missing literal

API tips: Upstream errors can cause processor failure

● Processed code may be missing literal elements, be missing imports, be in a broken state, etc.

○ Very strange results including core javac types in ClassCastExceptions or NullPointerExceptions deep within the compiler.

○ Upstream compile errors are masked by these exceptions.

● Auto-common provides SuperficialValidation.validateElements(

)

○ Simple sanity check for each Element and contents

○ If validateElements() returns false, two options:

■ skip processing supplied elements (if done in a loop)

■ return from the processor immediately

JavaOne 2014

API tips: Gotchas ● TypeMirror.equals() is not a reliable comparison. Should use: ○ Types.isSameType(mirror1,

API tips: Gotchas

● TypeMirror.equals() is not a reliable comparison. Should use:

○ Types.isSameType(mirror1, mirror2) (if available)

○ MoreTypes.equal(mirror1, mirror2) (if in a static context)

● TypeMirror instanceof checks are generally incorrect

○ MoreTypes.asDeclared(typeMirror) converts correctly or throws IAE

○ MoreTypes.asArray(

)

etc. - converters for all TypeMirror kinds.

JavaOne 2014

API tips: Google’s auto-common utilities ● MoreTypes and MoreElements ○ Wrappers for equivalence ○ Static

API tips: Google’s auto-common utilities

● MoreTypes and MoreElements

○ Wrappers for equivalence

○ Static methods for conversion and comparison

● SuperficialValidation

○ For simple sanity checking of elements before processing

● AnnotationMirrors and AnnotationValues

○ convenience methods and Equivalence wrappers

○ static utilities to handle default values

● Types/Elements have useful methods, but are instances. Auto common utilities provides many static utility method equivalents.

Note: AnnotationValues and AnnotationMirrors pending extraction from Dagger 2.

JavaOne 2014

Generating code: JavaWriter ● Build up your class programmatically. ● Refer to elements without using

Generating code: JavaWriter

● Build up your class programmatically.

● Refer to elements without using “stringly-typed” references.

● Write to any Appendable.

JavaWriter javaWriter = JavaWriter.inPackage("some.package"); ClassWriter klass = javaWriter.addClass("SomeType"); VariableWriter field = klass.addField(String.class, "foo"); field.addModifiers(PRIVATE, FINAL); ConstructorWriter constructor = klass.addConstructor(); VariableWriter p0 = constructor.addParameter(Key.class, "key"); constructor.body() .addSnippet("this.%s = %s.getFormat();", field.name(), p0.name()); javaWriter.write(appendable);

JavaOne 2014

Generating code: JavaWriter ● Get clean, organized and readable code out. package some.package; import

Generating code: JavaWriter

● Get clean, organized and readable code out.

package some.package;

import java.security.Key;

class SomeType { private final String foo;

SomeType(Key key) { this.foo = key.getFormat();

}

}

Note: JavaWriter 3 is pending review with Square and extraction from Dagger2. JavaWriter 2 is available.

JavaOne 2014

  Generating code: JavaWriter ● Pros:     ○ Programmatic creation of code ○ Reduce
 

Generating code: JavaWriter

● Pros:

 
 

○ Programmatic creation of code

○ Reduce errors with cross referencing

○ Automatic handling of imports and type shortening

○ Very useful in loops where each loop touches many parts of the code

○ Some built-in validation based on code structure

 

some erroneous things are simply impossible to write

● Cons:

 

Code that generates code can look quite different in shape than output

JavaOne 2014

Generating code: templates #foreach ( $p in $props ) @Override ${ p . access }${

Generating code: templates

#foreach ($p in $props) @Override ${p.access}${p.type} ${p}() { #if ($p.kind == "ARRAY") #if ($p.nullable)

return $p == null ? null : ${p}.clone(); #else return ${p}.clone(); #end

#else return $p; #end

}

#end

@Override public int[] postCodes() { return postCodes == null ? null : postCodes.clone();

}

JavaOne 2014

Generating code: templates ● Apache Velocity is a good choice of template engine ○ Supported

Generating code: templates

● Apache Velocity is a good choice of template engine

○ Supported in all major IDEs (even Emacs)

○ Directives clearly distinguishable from Java code snippets

● In AutoValue we do a post-processing step to remove superfluous spaces and blank lines, just so the generated code looks nicer

JavaOne 2014

Testing Processors - Unit testing environment ● Unit testing still requires javac environment, for Types/Elements

Testing Processors - Unit testing environment

● Unit testing still requires javac environment, for Types/Elements

● Compile-testing has a CompilationRule suitable for JUnit4

@RunWith(JUnit4.class)

}

JavaOne 2014

Testing Processors - Failing integration tests ● Need to test failing compiles without failing the

Testing Processors - Failing integration tests

● Need to test failing compiles without failing the outer build

● Compile-testing has assertions based on the Truth library

● These assertions can variously:

○ assert about javac runs success or failure

○ be configured to run arbitrary Processor instances

○ run against source from files or strings

○ execute and store results in-memory

○ assert about errors with specific message contents and locations

○ assert against contents of generated files

○ assert against contents of generated java source, comparing AST

JavaOne 2014

Testing Processors - Failing integration test assertions JavaFileObject file0 =

Testing Processors - Failing integration test assertions

JavaFileObject file0 = JavaFileObjects.forSourceLines("test.Foo", "package test", "class Foo {";

);

JavaFileObject file1 = JavaFileObjects.forResource("expected/SomeFile.java");

assert_().about(JavaSourcesSubject.javaSources()) .that(ImmutableSet.of(file0, file1)) .processedWith(new BlahProcessor()) .failsToCompile() .withErrorContaining"Invalid use of Annotation @Blah")

.in(javaFileObject).onLine(7);

JavaOne 2014

Testing Processors - Succeeding integration tests ● For successful compilation, two approaches: ○ Use

Testing Processors - Succeeding integration tests

● For successful compilation, two approaches:

○ Use compile-testing to compare against a golden file, or

○ Functionally test that the generated code behaves as it should

● Golden file tests:

○ Brittle - small changes can require a lot of change in test files/code

○ Useful to demonstrate expected output

● Functional tests:

○ Exercise more code paths and use-cases with less verbosity

JavaOne 2014

Testing Processors - Overall strategy ● Use Functional tests to cover the bulk of use-cases

Testing Processors - Overall strategy

● Use Functional tests to cover the bulk of use-cases

● Use CompilationRule to unit-test the internals of your Processor

● Use compile-testing assertions to test expected errors

● Use compile-testing assertions to test a small number of golden expectation files, for illustration of processor output

JavaOne 2014

JavaOne 2014
JavaOne 2014

JavaOne 2014

Summary ● Annotation processors are a powerful way to plug in to javac ● An

Summary

● Annotation processors are a powerful way to plug in to javac

● An ecosystem is evolving to ease writing custom processors

○ JavaWriter, auto-common, compile-testing, @AutoService

● Several useful annotation processors exist presently

○ AutoValue, Dagger

● Lots of advantages to annotation processors

○ Early error-checking

○ Performance improvements

○ Viewable generated source code

JavaOne 2014

Annotation Processor Resources ● Check out existing annotation processors: ○ Google Auto:

Annotation Processor Resources

● Check out existing annotation processors:

● Projects that may help:

Google Auto common utilities: https://github. com/google/auto/tree/master/common

JavaOne 2014

JavaOne 2014

JavaOne 2014

Manual IDE configuration ● Make a jar file with your processor and all of its

Manual IDE configuration

● Make a jar file with your processor and all of its dependencies

● Ensure it has META-INF/services/javax.annotation.processing. Processor

○ @AutoService(Processor.class) is the easiest way

● NetBeans:

○ Project Properties > Libraries > Processor > Add JAR/Folder

● Eclipse:

○ Properties > Java Compiler > Annotation Processing

■ Enable Annotation Processing + Factory Path

JavaOne 2014