Reply

APT: Compile-Time Annotation Processing with Java

At 1:32 AM on Mar 21, 2005, R.J. Lorimer wrote:

Oops

I'm afraid I have a confession to make. I unintentionally lied to all of you regarding the state of annotations in Java 5. As can be seen by this tip , I honestly believed that annotations could only be processed at runtime. I was *very* mistaken. There is an extensive API written around the new compiler tool that ships with Java 5 - APT . APT stands for A nnotation P rocessing T ool - not to be confused with the Debian A dvanced P ackaging T ool. Let it never be said that I haven't eaten my own share of humble pie. Incidentally, I also need to apologize for my brief hiatus last week from tip posting. Combined with celebrating my birthday and spending time trying to understand APT, I didn't get as many tips out as I hoped. Anyway - enough groveling. Before you try reading this tip, I highly recommend you either a.) read my Javalobby Annotation Tip Series (see in references), or b.) find some other resource to learn about Java annotations. I don't plan to go back and rehash already-covered subjects too much in this tip series. I plan on covering over the next few days all of the ins-and-outs of using the APT tool with your source. Please see the references section for a detailed reference (imagine that) of the APT tool as it exists in the Sun Java 5 JDK.

What is APT?

APT, as I said above, is an annotation processing tool for Java. More specificially, APT allows you to plug code in to handle annotations in a source file as the code compilation is occuring - and in that process, you can emit notes, warnings, and errors. Let me clarify that APT is not a replacement for Javac, however - instead it is a wrapper for Javac that allows developers to glue annotation processing into their code build process for compile time. Yes, that's right. When I said in my previous post that there was no useful compile-time annotation tool, I was unequivocally, 100% incorrect. APT is a combination of two things - 1.) the APT tool itself, and 2.) an API for plugging into the APT tool. The tool itself works much the same as Javac (and for good reason since it is a wrapper to the compilation process). The API for developers is commonly referred to as the 'Mirror API'. Why is it called the mirror API? Well, the primary focus of the classes in the APT API is to mirror the features of the Java language specification (e.g. class declarations, method declarations, yadayadayada) - and as such, mirrors the language features as they exist in the source file (prior to compilation). Let it be known, the Mirror API is not guaranteed to stay unchanged . APT is a Sun JDK specific tool, and in addition to that, is very fragile to language changes. Therefore, expect that if your platform changes, your APT processing code may have to be updated as well.

Setting up for APT

APT (the tool itself) is easy to use. The executable/script/whatever-for-your-platform can be found in your Java 5 [JDK HOME]/bin folder. The execution process is very similar to Javac, (with the exception of the process for plugging in your annotation processing code). Integrating your code into APT can be done (as with many Java tasks) in a multitude of ways. I don't plan on covering each of them in this tip, as it will detract from the main point - which is how you actually process annotations. I will be focusing on how to target a particular factory for annotation processors. The main hook for all external developers using the APT tool is the com.sun.mirror.apt.AnnotationProcessorFactory interface. I will get into how you write these factories in a minute, but for now, go along with me as I show you how we tell APT to use a particular implementation of this factory API. Given a JAR (called javalobby-tnt-annotations.jar) containing an implementation of this API (say we call it com.javalobby.tnt.apt.NoteAnnotationProcessorFactory ), you could tell APT to process your classes by using a command line similar to this:

apt -classpath /path/to/jar/javalobby-tnt-annotations.jar -factory com.javalobby.tnt.apt.NoteAnnotationProcessorFactory /path/to/source/*.java

As I said before, there are many other ways to glue one or more annotation processor factories into APT - please see Sun's documentation (linked to in the references section below).

Finally, in last week's tip - Eclipse: Per-Project JRE Libraries I made a reference to the fact that I would use that technique for showing off some Java 5 features - APT is that feature. APT isn't included as part of the rt.jar (the runtime jar), but rather, part of the tools.jar of the JDK. What I like to do is refer to tools.jar in a custom Eclipse JRE reference so that I can write code to work with APT as well as the core libraries -here is how:

1.) Create a new JRE as seen by this screenshot:

2.) Add tools.jar to your JRE definition by turning of default classpath, and adding it as seen here:

3.) Refer to this new JRE definition in your APT tool project as seen by the screenshot series under 'Targeting The Right Runtime Libraries' in the Eclipse tip.

... and that is it. You can now refer to the com.sun.apt.mirror APIs in your project.

Annotation Processor Factories

Annotation processor factories are the first step developers take when interacting with APT. The entire job of the annotation processor factory is to hand out annotation processors. If an annotation processor factory is the heart of annotation processing, annotation processors are the brain. without one, the other is pretty much useless. Factories are a relatively simple API, with three methods you must implement - Collection<String> supportedAnnotationTypes() , Collection<String> supportedOptions() , and AnnotationProcessor getProcessorFor(Set<AnnotationTypeDeclaration>, AnnotationProcessorEnvironment) . Let's talk about each one.

• Collection<String> supportedAnnotationTypes() - The entire purpose of this method is to report what annotation types this factory can produce processors for. Processors are intended to be able to work with as coarse or granular annotation declaration as necessary. While it may be useful for a processor to handle only one annotation, it may also be useful for a processor to handle many or even all annotation types. This method allows you to define what annotations your factory can build processors to... process. The return result is a collection of strings. Each string should be either a.) a fully qualified annotation interface name (e.g. 'java.lang.annotation.Retention'; 'com.javalobby.tnt.annotation.Note') or b.) a prefix followed by the '*' wildcard (e.g. 'java.lang.annotation.*' meaning all annotations in the java.lang.annotation package, or even '*', meaning all annotations).
• Collection<String> supportedOptions() - The supportedOptions() method also returns a collection of strings. The APT tool currently allows the user of the APT tool to pass in various options to the annotation processor factory(-ies) being invoked by using the -A prefix on the options. The APT tool reference details exactly where in the command line these options would be included. This method on the factory defines what options should be made available to your annotation processors at their construction time. Annotation options come in two forms - keys with a value, and those without. You will always simply return the accepted keys from this method. The example provided by the documentation is: For example, if this factory recognizes options such as -Adebug -Aloglevel=3, it will return the strings "-Adebug" and "-Aloglevel". .
• AnnotationProcessor getProcessorFor(Set<AnnotationTypeDeclaration>, AnnotationProcessorEnvironment) - This method is the culmination of the results from the two methods above. Callers to this method are asking for an annotation processor implementation (covering this next) for the set of annotation declarations passed in, and the environment provided. So what does that mean? Well, the annotation declarations are just that - declarations of annotations. The sets provided here will be directly related to the results you returned from the supportedAnnotationTypes() method. The declarations passed in may not match what your factory supports (in particular, it may ask for annotation processors for an empty set), in those cases it is recommended you return com.sun.mirror.apt.AnnotationProcessors.NO_OP - which is, as it implies, a no-operation annotation processor implementation. The AnnotationProcessorEnvironment is the context that the annotation processor is being asked to run in - among other things this environment contains the options passed by the user, utility classes, the Filer and Messager (to be covered later), and various other helpful methods. In most cases you can simply pass this object to your new annotation processors in their constructor (as I will show shortly).

As is usually my way, I'm going to reuse an existing example. In my annotation tip series (see references), I discussed evolving an annotation (@Note) that encompassed the idea of developer documentation that was programmatically accessible. For those who don't want to look back, here is what the annotation looks like:

package com.javalobby.tnt.annotation;
 
import java.lang.annotation.*;
 
@Documented
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD, ElementType.CONSTRUCTOR, ElementType.PACKAGE, ElementType.FIELD})
public @interface Note {
    String value();
    Priority priority() default Priority.MEDIUM;
}

... and here is what the class utilizing this may look like:

package com.javalobby.tnt.annotation;
 
@Note(value = "This class isn't finished", priority = Priority.HIGH)
public class MyClass {
 
 @Note("This field isn't finished")
 private String fieldA;
 
 @Note("Constructor isn't finished")
 public MyClass() {
 
 }
 
 @Note("methodA isn't finished")
 public void methodA() {
 
 }
}

Here is what an annotation processor factory might look like to handle these annotations:

package com.javalobby.tnt.apt;
 
import java.util.*;
import com.sun.mirror.apt.*;
import com.sun.mirror.declaration.*;
 
public class NoteAnnotationProcessorFactory implements
		AnnotationProcessorFactory {
 
	/**
	 * Returns a note annotation processor.
	 * 
	 * @return An annotation processor for note annotations if requested, 
	 * otherwise, returns the NO_OP annotation processor.
	 */
	public AnnotationProcessor getProcessorFor(
			Set<AnnotationTypeDeclaration> declarations,
			AnnotationProcessorEnvironment env) {
		AnnotationProcessor result;
		if(declarations.isEmpty()) {
			result = AnnotationProcessors.NO_OP;
		}
		else {
			// Next Step - implement this class:
			result = new NoteAnnotationProcessor(env);
		}
		return result;
 
	}
 
	/**
	 * This factory only builds processors for the 
	 * {@link com.javalobby.tnt.annotation.Note} annotation.
	 * @return a collection containing only the note annotation name.
	 */
	public Collection<String> supportedAnnotationTypes() {
		return Collections.singletonList("com.javalobby.tnt.annotation.Note");
	}
 
	/**
	 * No options are supported by this annotation processor.
	 * @return an empty list.
	 */
	public Collection<String> supportedOptions() {
		return Collections.emptyList();
	}
}

Annotation Processors

The next logical step for building an annotation processor toolset (and incidentally, the last I plan to tackle today) is building the annotation processor itself. Annotation processors are deceptively simple from an API perspective - they simply have a single, no-argument, no-return-type, no-throws-declaration method called process() . The variation in implementation (and in 99% of the cases, the effective usage of the AnnotationProcessorEnvironment object) is where all of the power is with annotation processors. The environment object gives you access to the entire mirror API - everything you can do as an annotation processor starts there. One thing that I found deceptive about this API that I thought I should clarify here, is that the process() method isn't always expecting you to respond to a particular annotation in the code anywhere. Rather, it may expect you to hook into the mirror API using the comprehensive com.sun.mirror.apt.DeclarationVisitor API so that as APT is processing, it can perform callbacks to your code to process (although this isn't strictly necessary in many cases). It would be foolish of me to brush over the visitor pattern usage in this context, as it is where all of the power (and all of the complexity) of this API exists. However, I don't want this tip to drag into a full reference manual - so keep from getting too long-winded in this tip, I've managed to implement our example without using the visitor pattern. Let's just jump right in and implement the compile-time counter-part to our result from our final tip in the the annotation series - Finding Annotations at Runtime (see references). The result in this tip was a block of code that could be run to print out all @Note annotations on all various declarations. Useful, even if not particularly complex. In the (very) near future, I will post a follow-up tip that continues to delve into the complexities of this visitor API. Here is an AnnotationProcessor that duplicates our runtime example we already have:

package com.javalobby.tnt.apt;
 
import java.util.*;
 
import com.sun.mirror.apt.*;
import com.sun.mirror.declaration.*;
import com.sun.mirror.util.*;
 
public class NoteAnnotationProcessor implements AnnotationProcessor {
 
	private AnnotationProcessorEnvironment environment;
 
	private AnnotationTypeDeclaration noteDeclaration;
 
	public NoteAnnotationProcessor(AnnotationProcessorEnvironment env) {
		environment = env;
		// get the annotation type declaration for our 'Note' annotation.
		// Note, this is also passed in to our annotation factory - this 
		// is just an alternate way to do it.
		noteDeclaration = (AnnotationTypeDeclaration) environment
				.getTypeDeclaration("com.javalobby.tnt.annotation.Note");
	}
 
	public void process() {
 
		// Get all declarations that use the note annotation.
		Collection<Declaration> declarations = environment
				.getDeclarationsAnnotatedWith(noteDeclaration);
		for (Declaration declaration : declarations) {
			processNoteAnnotations(declaration);
		}
	}
 
	private void processNoteAnnotations(Declaration declaration) {
		// Get all of the annotation usage for this declaration.
		// the annotation mirror is a reflection of what is in the source.
		Collection<AnnotationMirror> annotations = declaration
				.getAnnotationMirrors();
		// iterate over the mirrors.
		for (AnnotationMirror mirror : annotations) {
			// if the mirror in this iteration is for our note declaration...
			if(mirror.getAnnotationType().getDeclaration().equals(
					noteDeclaration)) {
				
				// print out the goodies.
				SourcePosition position = mirror.getPosition();
				Map<AnnotationTypeElementDeclaration, AnnotationValue> values = mirror
						.getElementValues();
 
				System.out.println("Declaration: " + declaration.toString());
				System.out.println("Position: " + position);
				System.out.println("Values:");
				for (Map.Entry<AnnotationTypeElementDeclaration, AnnotationValue> entry : values
						.entrySet()) {
					AnnotationTypeElementDeclaration elemDecl = entry.getKey();
					AnnotationValue value = entry.getValue();
					System.out.println("    " + elemDecl + "=" + value);
				}
			}
		}
	}
 
}

... and here is the output...

C:\temp2\javalobby\com\javalobby\tnt\annotation>c:/Progra~1/Java/jdk1.5.0/bin/a
t -cp c:\temp2\javalobby\test-apt.jar -factory com.javalobby.tnt.apt.NoteAnnota
ionProcessorFactory *.java
warning: Annotation types without processors: [java.lang.annotation.Documented,
java.lang.annotation.Retention, java.lang.annotation.Target]
Declaration: methodA()
Position: MyClass.java:15
Values:
    value()="methodA isn\'t finished"
Declaration: MyClass()
Position: MyClass.java:10
Values:
    value()="Constructor isn\'t finished"
Declaration: fieldA
Position: MyClass.java:7
Values:
    value()="This field isn\'t finished"
Declaration: com.javalobby.tnt.annotation.MyClass
Position: MyClass.java:4
Values:
    value()="This class isn\'t finished"
    priority()=com.javalobby.tnt.annotation.Priority.HIGH
1 warning

We have only scratched the surface of APT today. Aside from reporting the existence of annotations, APT can print notes, warning and errors into the compilation process. APT can create text files and binary files, and can even create intermediate source files that will then be run BACK through APT on a subsequent pass where they can be compiled (EJB descriptors anyone????). I hope that I will be able to show you how to delve into these features in a little while. See below for the references.

References

Sun Java 5 APT Documentation
http://java.sun.com/j2se/1.5.0/docs/guide/apt/

My Javalobby Annotation Tip Series: -

Eclipse: Per-Project JRE Libraries:
http://www.javalobby.org/java/forums/t17843

Until next time,

R.J. Lorimer
rj -at- javalobby.org
http://www.coffee-bytes.com