The upcoming Jack & Jill compilers

guardsquare.com /en/blog/the_upcoming_jack_and_jill_compilers_in_android

12/27/2014

Update: On the 14th of March 2017, Google announced that it is deprecating the Jack toolchain. You can
continue to rely on DexGuard to shrink, optimize and obfuscate your Android applications.

First sightings

In the shadow of the release of Android 5.0 "Lollipop", the Android tools are seeing updates as well. This is good
news for Android developers, who have to manage all these shiny new features. Useful improvements like
resource shrinking and multidex have trickled out over the past few weeks. There are some signs of more
goodness to come:

The Android SDK 21.1 contains two new build tools: jack.jar and jill.jar.
The Gradle plugin contains a new, undocumented flag: useJack.
The Android source repository contains a new toolchain named jack.

It seems like these additions haven't been announced or even noticed thus far, but they beg to be explored. I had
a look, as an external observer. In short: Jack and Jill form a new compiler toolchain to transform java code into
dalvik bytecode. They introduce a new intermediate representation called Jayce.

History

The Android platform has always had close ties with the Java platform. This approach leverages the existing
expertise of developers and the wealth of supporting libraries, frameworks, IDEs, and other tools. This probably
won't change anytime soon, but it does entail a dependency on the Java platform. For instance, java has evolved
to Java 8, with extensive changes to support closures. The Android toolchain still doesn't support Java 8.

Based on its logs, the new compiler project was started many months ago by the Google development team
from FlexyCore, a company that Google acquired in 2013. At the time, they offered a prototype for an ahead-of-
time compiler for Android, based on the gcc toolchain. This concept made it into Android in the form of the
Android RunTime, ART, now based on the LLVM toolchain. ART compiles code on the user's device, from Dalvik
bytecode to native code, offering better and smoother performance.

The development team has now quietly been working on the compilation of java to dalvik bytecode. Developers
go through this process many times a day, so they surely appreciate its importance.

The build process

The current Android build process transforms java code into dalvik bytecode:

A regular java compiler converts the application's source code into java bytecode. This is typically a compiler
1/4
from the OpenJDK project, from Oracle, or from Eclipse.

The dex compiler takes the combined java bytecode and converts it to Android-specific dalvik bytecode. The
build process may cache dexed versions of libraries, to speed up the conversion. The dalvik bytecode goes into
the application.

If ProGuard is enabled, it combines the compiled bytecode and the library bytecode and optimizes it, before
passing it to the dex compiler:

ProGuard is of course our own open-source optimizer and obfuscator for java bytecode. It produces code that is
more compact (in a shrinking step), more efficient (in an optimization step), and more difficult to reverse engineer
(in an obfuscation step). ProGuard reads a configuration file to steer its processing. It writes out a mapping file
that allows to de-obfuscate any stack traces that the processed application might produce.

Conceptually, the jack compiler consolidates the functionality of javac, ProGuard, and dex in a single conversion
step:

In actual builds, the jill compiler adds a new preprocessing and caching step:

2/4
Jill shields jack from java bytecode by converting it to intermediate jayce bytecode. Jack converts the java
source code and the intermediate bytecode to dalvik bytecode. It includes ProGuard's functionality, reading
ProGuard configurations, shrinking and obfuscating the code, and writing out a ProGuard mapping file.

In practice

You can already try the new compilers by enabling them in a Gradle build with the Android plugin 0.14.+:

defaultConfig {
useJack true
}

You can see which commands are run with Gradle's "-i" option, and you can similarly run the compilers from the
command-line. The new compilers are only activated when building application projects. Library projects are still
built with javac and ProGuard.

The compilers already seem quite functional. Jack doesn't support some of the advanced features of ProGuard
(yet), for instance to remove logging code:

Proguard flag is not supported: -assumenosideeffects

...

In general, jack doesn't seem to perform any whole-program optimizations, which are a great performance-
improving feature of ProGuard. I also encountered a compiler error in one of my quick experiments:

Internal compiler error (version 1.0.001 (UJX30A

41a1...3ce9)).
Warning: This may have produced partial or corrupted output.

The compilers will no doubt enter a long public testing phase when they are officially announced.

Internals

3/4
The jack compiler is built around the Eclipse java compiler. The back-end of course generates dalvik bytecode
and jayce bytecode instead of java bytecode. It is an optimizing compiler, with optional shrinking and
obfuscation. For this purpose, it reads ProGuard configuration files and it writes ProGuard mapping files.

Both compilers use the intermediate jayce format. Each java source file or class file can be compiled to a jayce
file, which is then packaged and stored in a jar file. Intermediate representations are very common in optimizing
compilers to speed up builds. Conceptually, the format could also be used to distribute pre-compiled libraries.

Java 8?

So does the compiler support Java 8? Unfortunately, not yet. It's even aware of its own limitations, for instance
when you try to compile source code with closures:

Lambda expressions are allowed only at source level 1.8 or

above

The compiler supports source levels up to 1.7, but not 1.8...

In the Java world, the implementations of Java 8 include pervasive changes to the language, the bytecode
format, the runtime libraries, and the virtual machine. These changes were by choice; support for closures could
have been implemented at a compiler level. In the Android world, changing the dalvik bytecode and runtime
presents more serious compatibility problems. Mobile devices are updated at a much slower pace, if at all.
Having complete control over the compiler offers a possible way out.

Implications for external libraries and tools

Since the toolchain still supports both java source code and java bytecode, existing libraries can be used without
problems. Even tools that generate or process bytecode during the build process, such as Dagger, should be
fine, with the proper integration.

We will of course continue to make sure that DexGuard can be integrated nicely. Its advanced optimization and
hardening features go far beyond the basic features, so it's definitely worth it!

Conclusion

The Jack and Jill compilers will streamline the Android build process and improve build performance. The
toolchain strives for backward compatibility and opens up possibilities for future extension. We're looking forward
to see it mature and grow.

(all opinions, predictions, and inaccuracies are my own)

Eric Lafortune

4/4