Gradle 4.1 Task Dependencies

While migrating to Gradle 4.1 and Android Gradle Plugin 3.0 we discovered an issue that wasn’t documented. I would love to share this issue with you and explain how we fixed the problem.

The Problem

My team develops libraries that get integrated into the apps we have at Under Armour and, because of that, most of the time we are working on AARs that are consumed by those apps. In our deploy script we run a series of tasks which, up until Gradle 4.1, all ran sequentially (even though you could enable parallel tasks prior to 4). So here is a rough idea of what our CI server runs when we merge code in:

./gradlew clean assemble artifactoryPublish

The problem we found after upgrading to 4.1 is that our tasks were now being run in parallel and our artifactoryPublish task would run before the end of the assemble task. This as you can imagine adds some complication, unless you can get the finished AARs from an alternative reality where the build is already done.

This is now parallelized due to the Worker API which was introduced in Gradle 4.1. I won't delve too deeply, but this new API was made with Android in mind and allows work from a task and build level to be parallelized with minimal effort (from the app developers side). The effort is mostly on the plugin developer, which I am not so I am going to refrain from saying more. Although I will note there is a great talk given by Paul Merlin and Gary Hale on this topic I would suggest watching.

After looking a bit more into the Worker API, I found that not only will this allow in-task parallel work, but it will also allow tasks with no relationship or dependencies to run in parallel as well:

(From the Writing Custom Tasks Guide): Of course, any tasks that are dependent on this task (and any subsequent task actions of this task) will not begin executing until all of the asynchronous work completes. However, other independent tasks that have no relationship to this task can begin executing immediately.

I was unaware of this initially, so after about two hours of experimentation we found that even though there is no way to turn the feature off there are two options for getting around the issue.

The Solution

Solution 1: Run tasks one at a time

Instead of running:

./gradlew clean assemble artifactoryPublish

We could do the following:

./gradlew clean 
./gradlew assemble 
./gradlew artifactoryPublish

Though I think this looks ugly, it gets the job done. Many people will scoff at this, there are times when the quick-and-dirty helps us get through a gauntlet unscathed. Most of the time I disregard these types of solutions, but in a time of need it is still valid.

Now, the reason that I say Solution 1 is viable is that I suspect many people in the Android community are not Gradle experts. I have spent 10-15% of my work time over the last year learning more about Gradle and I feel confident in my knowledge.. I suspect however, that I have still probably only touched 10 to 20% of Gradle.

Solution 2: Create task dependencies in your build gradle files

If you get into a situation where tasks are running out of order due to the parallelization of tasks in 4.1., think about adding task specific dependencies. In order to enforce the dependency between the assemble task and the artifactoryPublish task you can put the following line in your build.grade:

artifactoryPublish.dependsOn assemble

Note: If you have multiple modules in your gradle project you can cascade this to each of them using the subprojects configuration block in your project build.gradle.

The dependsOn method is part of the Task API and allows us to tell our artifactoryPublish that we depend on the assemble task to finish before it can run. Creating this dependency will now stop the assemble task from running before it should and causing build issues.

There have been many things that have changed with the Android plugin and the team at Google have done a great job documenting almost all of these changes. However, there are always things that are overlooked or may be under documented. I hope if you ran into task parallelization issues that this helped you get through the issue and I look forward to hearing other people's experiences.

Dex Redux

In October of 2014 a friend and past colleague of mine, Mustafa Ali, wrote a great Medium article to help solve a problem that many Android developers have encountered at one point in their career. The Dex method issue in short is a limit to the amount of executable methods allowed in Android's Dalvik Executable file (the one used to execute Android code). Shortly after writing his guide Google did the unthinkable... they released a support library. The library creates multiple dex files to help apps that are large get around the single dex file limit issue. Though this is a great solution for large apps Google suggests to use Proguard to strip your app of unused code or to reduce your dependency on libraries as a whole. The former is much easier than the latter.

I have to be totally transparent and say that I am not as familiar with the depth of the dexing process. I had initially planned on learning more about this and then writing up a large post however due to a family emergency I have not really had the time I wanted to devote to this. I am writing this in hopes to bring more exposure to the problem and maybe get some people who have a great understanding to help contribute to this post.

Finally The problem

So the other day while integrating two feature branches into our development branch. The following error occurred on our build server:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':application:transformClassesWithDexForNormalDebug'.
com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.dex.DexException: Too many classes in --main-dex-list, main dex capacity exceeded

As I am not 100% familiar with what was going on I googled to see if this was something that others have been seeing. To my surprise there was only one result and that was a Google Issue Ticket. There isn't a ton of info in there, some people are suggesting that Google has broken a filter they have that creates the main-dex-list, which is a file that seems to be used to know which classes that should be included in the dex (I could be totally wrong about this).

Google acknowledged the issue, suggested a work around (which I have tried but haven't seen work) and opened another ticket to account for the actual solution. Because I couldn't get their solution to work I decided to test out a solution suggested by Alex Lipov, who was the one who created the Google Issue. Alex has a blog that talks through what he has found while working on this issue. Oddly enough he wrote this blog December of 2014 but didn't create the Google Tracker ticket till April 7th of 2016.

His updated solution is to access the CreateManifestKeepList class which is responsible for what gets kept for the dex. Then he modifies the accessibility of the class so it won't throw any exceptions while being modified. He basically makes the ImmutableMap mutable. It is a drastic approach but it works.

This is where the real fun starts. After utilizing his fix, I removed it so I could prove to myself that the fix is in fact real. When I removed his code from my Gradle build file...it still compiled without any issues. So I am interested in answering the following:

1. What is the purpose of the main-dex-list?
2. Why after removing the workaround am I not seeing the bug anymore?

Once I get back to real life I will pick back up and continue understanding the issue but I figured I would put this out and see if the community had any ideas what was going on. I would love to hear if you have had the same issue and how you solved it or if you have any extra info!