A guide to Maven Repositories

We at Appwrite recently published our Android SDK to Maven Central and we learnt a whole lot about the Gradle and Maven ecosystem in the process.

In this series of articles, we'll aim to teach you how you can build and publish your own Android Libraries using Gradle. Before we dive into building and publishing artifacts with Gradle, we need to get familiar with Maven because after all, Gradle is a build tool for Java based projects ( among several others ).

If you've worked on a Java project, it's highly likely that you would've heard of Maven, as it is the most popular build tool for Java projects. Most of us associate Maven with a pom.xml file. However, there's much more to Maven than meets the eye and you can very quickly get lost in a sea of knowledge. This article will aim to walk you through the important concepts in Maven and introduce you to various terminologies in the Maven ecosystem.

🗃️ The POM file

The POM file is the cornerstone of any Maven project. Think of it as an entry point to any Maven project. The POM file has all the information required to build the project; a recipe book of sorts. If you're familiar with the package.json file in node projects, composer.json file in php projects, requirements.txt in python projects and build.gradle in other Java/Kotlin projects, the POM file should seem familiar.

Though we will be using Gradle for building and publishing artifacts, it's a good idea to learn about the structure of the POM file as these concepts can be extended to Gradle as well.

Let's take a sample POM file to understand its structure. You can use the following command to create a simple Maven project if you want to test it out yourself.

mvn -B archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4

Once you have executed this command, you will notice a few things have happened. First, you will notice that a directory named my-app has been created for the new project, and this directory contains a file named pom.xml that should look like this:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mycompany.app</groupId>
  <artifactId>my-app</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>my-app</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
       ... lots of helpful plugins
    </pluginManagement>
  </build>
</project>

pom.xml contains the Project Object Model (POM) for this project. The POM is the most basic unit of work in Maven. This is important to remember because Maven is inherently project-centric in that everything revolves around the notion of a project. In short, the POM contains every important piece of information about your project and is the go-to place for finding anything related to your project.

This is a very simple POM but still displays the key elements every POM contains, so let's walk through each of them to familiarize you with the POM essentials:

project This is the top-level element in all Maven pom.xml files.
modelVersion This element indicates what version of the object model this POM is using. The version of the model itself changes very infrequently but it is mandatory in order to ensure stability of use if and when the Maven developers deem it necessary to change the model.
groupId This element indicates the unique identifier of the organization or group that created the project. The groupId is one of the key identifiers of a project and is typically based on the fully qualified domain name of your organization. For example org.apache.maven.plugins is the designated groupId for all Maven plugins.
artifactId This element indicates the unique base name of the primary artifact being generated by this project. The primary artifact for a project is typically a JAR file. Secondary artifacts like source bundles also use the artifactId as part of their final name. A typical artifact produced by Maven would have the form -. (for example, myapp-1.0.jar).
version This element indicates the version of the artifact generated by the project. Maven goes a long way to help you with version management and you will often see a SNAPSHOT designator in a version, which indicates that a project is in a state of development. We will discuss the use of snapshots and how they work further on in this guide.
name - This element indicates the display name used for the project. This is often used in Maven's generated documentation.
url This element indicates where the project's site can be found. This is often used in Maven's generated documentation.
properties This element contains value placeholders accessible anywhere within a POM.
dependencies This element's children list the dependencies. The cornerstone of the POM.
build This element handles things like declaring your project's directory structure and managing plugins.

For a complete documentation of all the POM attributes, you can refer to the official docs.

🧩 Artifacts

An artifact is something that is either produced or used by a project. Examples of artifacts produced by Maven for a project include: JARs ( Java Archives ), source and binary distributions, WARs, AARs ( Android Archives ) etc. Each artifact is identified by a group id, an artifact ID, a version, an extension and a classifier.

Take a look at Appwrite's Android SDK. You will find the following artifacts ( and more ).

sdk-for-android-0.0.1-javadoc.jar                  2021-06-29 15:13       261            
sdk-for-android-0.0.1-sources.jar                  2021-06-29 15:13     18663      
sdk-for-android-0.0.1.aar                          2021-06-29 15:13     46437      
sdk-for-android-0.0.1.pom                          2021-06-29 15:13      3277

You will find a lot of .md5, .sha1, .sha256, .sha512 and .asc files which are all used by Maven Central and other consumers for verifying the signature and authenticity of your artifacts.

You will notice that, these artifacts follow a naming convention as mentioned below.

<artifactId>-<version>-<classifier>.<extension>

Note : classifier is an optional field and is absent in the case of Primary Artifacts which in the case of our Android project are the aar and pom files.

In our case the following artifacts are the primary artifacts

sdk-for-android-0.0.1.aar
sdk-for-android-0.0.1.pom

Our aar file can contain Android resources and a Manifest file, which allows you to bundle in shared resources like layouts and drawables in addition to all the Java classes and methods. Our pom file contains information about our library including all the dependencies we use internally.

And the following are the secondary artifacts ( which are supplementary to the primary artifacts and are not necessarily required to build your project. )

sdk-for-android-0.0.1-javadoc.jar
sdk-for-android-0.0.1-sources.jar

When we talk about publishing our project in the next article, we will be referring to all these artifacts.

🗺️ Maven Coordinates

We've talked about the pom file and primary and secondary artifacts. In this section we will dive into the structure of a Maven repository and understand how Maven resolves dependencies when you build your project!

What's a Maven dependency you ask ? A Maven Dependency is any artifact that your project depends on. A typical dependency declaration in a Maven project looks like this

<dependencies>
  <dependency>
    <groupId>io.appwrite</groupId>
    <artifactId>sdk-for-android</artifactId>
    <version>0.0.1</version>
  </dependency>
</dependencies>

Together, the groupID, artifactID, and version make up the Maven Coordinates. Every Maven artifact is identified by these coordinates.

The next question arises. Given the Maven coordinates, how does maven locate an artifact? Maven uses the coordinates values for a given dependency to construct a URL according to the maven repository layout.

Let's take the example of Appwrite's Android SDK with the following coordinates

<dependency>
  <groupId>io.appwrite</groupId>
  <artifactId>sdk-for-android</artifactId>
  <version>0.0.1</version>
</dependency>

According to the official specification, the URL for primary artifacts looks like:

/$groupId[0]/../${groupId[n]/$artifactId/$version/$artifactId-$version.$extension

The groupId array in the URL is formed from the <groupId> element in the pom file as follows

$groupId is a array of strings made by splitting the groupId’s on “.” into directories.

So for the groupId value of org.example.subdomain, our $groupId array would be [org, example, subdomain], which when translated into directories, becomes org/example/subdomain.

In our case, the groupId io.appwrite is translated to the following path io/appwrite/.

Our next step is to build the URL for the primary and secondary artifacts.

🔧 Primary Artifacts

One of the core features of Maven is its ability to handle Transitive Dependencies. That is, to find and download the dependencies of your dependencies, and their dependencies and so on until all of dependencies are satisfied.

Just how your own Maven project has a pom.xml file listing its main dependencies, those dependencies also have a remote pom file serving a similar purpose. Maven uses this file to figure out what other dependencies to download. When a coordinate does not contain a <classifier>, it is considered a primary artifact and is expected to have a pom available.

In our case, the primary artifacts are the .pom file and the .aar file.

pom - If we follow the URL convention above, the pom file for Appwrite's Android SDK can be located at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.pom

aar - If we follow the URL convention above, the aar file for Appwrite's Android SDK can be located at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.aar

⛏️ Secondary Artifacts

Secondary artifacts, or “attached artifacts” on the other hand are dependencies that you want Maven to download that are ancillary to your project. Most often they are used to download the javadocs and/or sources for a particular dependency. However, unlike a primary artifact, a secondary artifact is not expected to have a remote pom and thus never has any dependencies.

They can be specified in the <dependencies> section just like primary artifacts:

<dependency>
  <groupId>io.appwrite</groupId>
  <artifactId>sdk-for-android</artifactId>
  <version>0.0.1</version>
  <classifier>sources</classifier>
</dependency>

The URL for secondary artifacts is similar to that of the primary artifacts except that they have an optional $classifier variable.

/$groupId[0]/../$groupId[n]/$artifactId/$version/$artifactId-$version-$classifier.$extension

In our case, the secondary artifacts are the javadocs and the sources.

javadoc - If we follow the URL convention above, the javadoc file for Appwrite's Android SDK can be located at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1-javadoc.jar

sources - If we follow the URL convention above, the sources file for Appwrite's Android SDK can be located at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1-sources.jar

🏁 Checksums

To verify the downloaded artifacts Maven computes the md5, sha1, sha256 and sha512 checksum for that artifact and compares it to the values found in the checksum files located at $ARTIFACT_URL.md5, or $ARTIFACT_URL.sha1, respectively.

This is strictly meant as a way to quickly verify downloads, and it is NOT meant to be used for authentication or security purposes. This is also NOT a substitute for using HTTPS, as checksums can be trivially intercepted and modified along with the modified artifacts.

sha1 - the sha1 checksum for our aar artifacts for example, would be present at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.aar.sha1

md5 - the md5 checksum for our aar artifacts for example, would be present at

io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.aar.md5

Similarly for all other artifacts (javadoc.jar.md5, pom.md5, sources.jar.md5 and so on).

🔐 Signatures

The Maven Central repository makes in mandatory to include a signature for all the uploaded artifacts and checksums. This ensures absolute confidence in the authenticity of the downloaded artifacts. Every artifact and checksum has a corresponding .asc file that contains the signature of the author of the artifact.

$artifacts.asc

For our .aar file for example, the corresponding .asc file would be located at
```
io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.aar.asc
```
$checksum.asc

For our .md5 ( sha1, sha256 etc.) files, the corresponding .asc file would be located at
```
io/appwrite/sdk-for-android/0.0.1/sdk-for-android-0.0.1.aar.md5.asc
```

📦 The Maven Repository

A Repository is Maven holds artifacts and dependencies of all kinds. There are exactly two kinds of repositories

Local
The local repository is a directory on the computer where Maven runs. It caches remote downloads and contains temporary build artifacts that you have not yet released. This is usually located in $HOME/.m2/repository
Remote
Remote repositories refer to any other type of repository, accessed by a variety of protocols such as file:// and https://. These repositories might be a truly remote repository set up by a third party to provide their artifacts for downloading (for example, repo.maven.apache.org). Other remote repositories may be internal repositories set up on a file or HTTP server within your company, used to share private artifacts between development teams and for releases.

Local and remote repositories are structured the same way so that scripts can run on either side, or they can be synced for offline use. The layout of the repositories is completely transparent to the Maven user, however.

The last topic that we'll discuss is the concept of a Release repository and a SNAPSHOT repository.

🏆 Release repository

A release repository is where all the published artifacts are stored in Maven. Because caching plays a really important role in maven repositories, it is highly recommended to make the Release repositories immutable. i.e. artifacts once uploaded cannot be modified. Let's say you published 0.0.1 of your library, you must not modify the existing version, instead publish the next version let's say 0.0.2. This is enforced by default in Maven Central.

There is also the concept of a Staging repository in Maven which serves as a staging area for you to verify all your artifacts before promoting it to the release repository.

🧪 SNAPSHOT repository

A SNAPSHOT variant of a library refers to the latest code along a development branch, and provides no guarantee the code is stable or unchanging and these variants are published to the SNAPSHOT repository. Conversely, the code in a Release variant (any version value without the suffix SNAPSHOT) is unchanging.

In other words, a SNAPSHOT version is the development version before the final Release version. The SNAPSHOT is older than its release.

During the release process, a version of x.y.z-SNAPSHOT changes to x.y.z. The release process also increments the development version to x.y.(z+1)-SNAPSHOT. For example, version 1.0.0-SNAPSHOT is released as version 1.0.0, and the new development version is version 1.0.1-SNAPSHOT.

We already discussed that a Release repository is immutable. So the idea behind a SNAPSHOT repository is to allow mutability and easy development. You can continuously push your latest changes to 1.0.0-SNAPSHOT and anyone depending on it will get the latest changes every time they build their project. Then, after a few iterations, and everyone is happy the latest state of 1.0.0-SNAPSHOT, it can be permanently released as 1.0.0, and rapid development can continue on 1.0.1-SNAPSHOT.

Sonatype's Open Source Software Repository Hosting (OSSRH) service allows open source projects and individuals to create a Maven Central repository and publish artifacts for free! In the next article, we'll go through all the steps involved in publishing an Android library to Maven Central. So stay tuned 😉