Skip to content

Commit

Permalink
RATIS-49. Add a clean profile to remove shaded source. Contributed by…
Browse files Browse the repository at this point in the history 
… Tsz Wo Nicholas Sze.
  • Loading branch information
@Jing9
Jing9 committed Mar 29, 2017
1 parent 768d7ca commit 930f4c0
Show file tree
Hide file tree
Showing 4 changed files with 99 additions and 54 deletions.
55 changes: 35 additions & 20 deletions BUILDING.md
View file
Original file line number Diff line number Diff line change
@@ -1,38 +1,54 @@
# Building
Ratis uses Apache Maven for the builds. A 3.0+ version of Maven is required as well as
Apache Ratis uses Apache Maven for the builds. A 3.2+ version of Maven is required as well as
at least Java-1.8.

When Ratis is build for the first time, shaded protobuf files needs to be generated first
which happens only in the mvn package phase. That is why you should run:
When building Ratis the first time, shaded files need to be generated by the following command:
```
$ mvn package -DskipTests
```
the first time the sources are checked out first. After doing that mvn compile or mvn test
can be used as normal.
After that, `mvn compile` or `mvn test` can be used as normal.
For example, we may run the basic tests by
```
$ mvn test -Dtest=TestRaftWith\*
```

# Raft Proto Shaded
# Shading

We shade protos, protobuf and other libraries such as Netty, gRPC and Hadoop so that
applications using Raft may use protobuf and other libraries with versions different
from the versions used here.
We shade protos, protobuf and other libraries such as Netty, gRPC, Guava and Hadoop
so that applications using Ratis may use protobuf and other libraries with versions
different from the versions used here.

The library requires the shaded sources for compilation. The generated sources are stored in
`ratis-proto-shaded/src/main/java/`. They are not checked-in git though.
Ratis requires the shaded sources for compilation.
The generated sources are stored in the following directories.
```
ratis-proto-shaded/src/main/java/
ratis-hadoop-shaded/src/main/java/
```
They are not checked-in to git though.

If you want to force-compile the proto files (for example after changing them), you should
run with
If you want to force compiling the proto files (for example after changing them),
you should run
```
$ mvn package -DskipTests -Dcompile-protobuf
```

For removing the shaded sources, run
```
$ mvn clean -Pclean-shade
```

Note that `mvn clean` alone does not remove the shaded sources.

## What are shaded?

| Original packages | Shaded packages |
| ----------------------------------|----------------------------------------------------------|
| `com.google.protobuf` | `org.apache.ratis.shaded.com.google.protobuf` |
| `io.grpc` | `org.apache.ratis.shaded.io.grpc` |
| `io.netty.handler.codec.protobuf` | `org.apache.ratis.shaded.io.netty.handler.codec.protobuf` |
| `org.apache.hadoop.ipc.protobuf` | `org.apache.ratis.shaded.org.apache.hadoop.ipc.protobuf` |
| Original packages | Shaded packages |
| ------------------------------------|--------------------------------------------------------------|
| `com.google.common` | `org.apache.ratis.shaded.com.google.common` |
| `com.google.protobuf` | `org.apache.ratis.shaded.com.google.protobuf` |
| `com.google.thirdparty.publicsuffix`| `org.apache.ratis.shaded.com.google.thirdparty.publicsuffix` |
| `io.grpc` | `org.apache.ratis.shaded.io.grpc` |
| `io.netty` | `org.apache.ratis.shaded.io.netty` |
| `org.apache.hadoop.ipc.protobuf` | `org.apache.ratis.shaded.org.apache.hadoop.ipc.protobuf` |

The protos defined in this project are stored in the `org.apache.ratis.shaded.proto` package.

Expand All @@ -53,7 +69,6 @@ To publish, use the following settings.xml file ( placed in ~/.m2/settings.xml )
<username>ratis_committer</username>
<password>********</password>
</server>
</servers>
</settings>
```
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@
<maven-assembly-plugin.version>2.4</maven-assembly-plugin.version>
<maven-bundle-plugin.version>2.5.3</maven-bundle-plugin.version>
<maven-checkstyle-plugin.version>2.15</maven-checkstyle-plugin.version>
<maven-clean-plugin.version>2.5</maven-clean-plugin.version>
<maven-clean-plugin.version>3.0.0</maven-clean-plugin.version>
<maven-clover2-plugin.version>3.3.0</maven-clover2-plugin.version>
<maven-compiler-plugin.version>3.2</maven-compiler-plugin.version>
<maven-dependency-plugin.version>2.10</maven-dependency-plugin.version>
Expand Down
42 changes: 33 additions & 9 deletions ratis-hadoop-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
<!--The Default target dir-->
<classes.dir>${project.build.directory}/classes</classes.dir>
<!--The Default location for sources-->
<sources.dir>src/main/java</sources.dir>
<shaded.sources.dir>src/main/java</shaded.sources.dir>

<!--Version of Hadoop to be shaded -->
<shaded.hadoop.version>${hadoop.version}</shaded.hadoop.version>
Expand All @@ -41,7 +41,7 @@
<build>
<!--I want to override these in profile so define them
with variables up here-->
<sourceDirectory>${sources.dir}</sourceDirectory>
<sourceDirectory>${shaded.sources.dir}</sourceDirectory>
<outputDirectory>${classes.dir}</outputDirectory>
<plugins>
<!-- Make a jar and put the sources in the jar -->
Expand Down Expand Up @@ -72,17 +72,20 @@
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${shaded.protobuf.version}</version>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.google.protobuf.nano</groupId>
<artifactId>protobuf-javanano</artifactId>
<version>${shaded.protobuf.version}</version>
<optional>true</optional>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${shaded.hadoop.version}</version>
<optional>true</optional>
</dependency>
</dependencies>

Expand All @@ -101,27 +104,49 @@
</profile>

<profile>
<id>skipShade</id>
<id>skipCompileProto</id>
<activation>
<file>
<exists>${sources.dir}</exists>
<exists>${shaded.sources.dir}</exists>
</file>
<property>
<name>!shade-hadoop</name>
<name>!compile-protobuf</name>
</property>
</activation>
</profile>

<profile>
<id>shade-hadoop</id>
<id>clean-shade</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<configuration>
<filesets>
<fileset>
<directory>${shaded.sources.dir}</directory>
<followSymlinks>false</followSymlinks>
</fileset>
</filesets>
</configuration>
</plugin>
</plugins>
</build>
</profile>

<profile>
<id>compile-protobuf</id>
<!--
Shade and drop the generated java files under src/main/java.
-->
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<profile.id>shade-hadoop</profile.id>
<profile.id>compile-protobuf</profile.id>
<!--When the compile for this profile runs, make sure it makes jars that
can be related back to this shading profile. Give them a shading prefix.
-->
Expand Down Expand Up @@ -273,8 +298,7 @@
<classifier>sources</classifier>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>${basedir}/src/main/java
</outputDirectory>
<outputDirectory>${basedir}/${shaded.sources.dir}</outputDirectory>
<includes>**/*.java</includes>
</artifactItem>
</artifactItems>
Expand Down
54 changes: 30 additions & 24 deletions ratis-proto-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,7 @@
<!--The Default target dir-->
<classes.dir>${project.build.directory}/classes</classes.dir>
<!--The Default location for sources-->
<sources.dir>src/main/java</sources.dir>

<shaded.sources.dir>src/main/java</shaded.sources.dir>

<!--Version of grpc to be shaded -->
<shaded.grpc.version>1.0.1</shaded.grpc.version>
Expand All @@ -44,7 +43,7 @@
<build>
<!--I want to override these in profile so define them
with variables up here-->
<sourceDirectory>${sources.dir}</sourceDirectory>
<sourceDirectory>${shaded.sources.dir}</sourceDirectory>
<outputDirectory>${classes.dir}</outputDirectory>
<plugins>
<!-- Make a jar and put the sources in the jar -->
Expand Down Expand Up @@ -372,10 +371,10 @@
<profiles>
<!-- Skip the tests in this module -->
<profile>
<id>skip-proto-shaded-tests</id>
<id>skip-shaded-tests</id>
<activation>
<property>
<name>skip-proto-shaded-tests</name>
<name>skip-shaded-tests</name>
</property>
</activation>
<properties>
Expand All @@ -387,34 +386,42 @@
<id>skipCompileProto</id>
<activation>
<file>
<exists>${sources.dir}</exists>
<exists>${shaded.sources.dir}</exists>
</file>
<property>
<name>!compile-protobuf</name>
</property>
</activation>
</profile>

<profile>
<id>clean-shade</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<build>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<configuration>
<filesets>
<fileset>
<directory>${shaded.sources.dir}</directory>
<followSymlinks>false</followSymlinks>
</fileset>
</filesets>
</configuration>
</plugin>
</plugins>
</build>
</profile>

<profile>
<id>compile-protobuf</id>
<!--
Generate and shade proto files. Drops generated java files
under src/main/java. Check in the generated files so available
at build time. Run this profile/step everytime you change proto
Shade and drop the generated java files under src/main/java.
Run this profile/step everytime you change proto
files or update the protobuf version.
The below does a bunch of ugly stuff. It purges current content
of the generated and shaded com.google.protobuf java files first.
It does this because later we apply patches later and patches
fail they've already been applied. We remove too because we
overlay the shaded protobuf and if files have been removed or
added, it'll be more plain if we have first done this delete.
Next up we generate proto, build a jar, shade it (which
includes the referenced protobuf), undo it over the src/main/java
directory, and then apply patches.
The result needs to be checked in.
-->
<activation>
<activeByDefault>true</activeByDefault>
Expand Down Expand Up @@ -594,8 +601,7 @@
<classifier>sources</classifier>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>${basedir}/src/main/java
</outputDirectory>
<outputDirectory>${basedir}/${shaded.sources.dir}</outputDirectory>
<includes>**/*.java</includes>
</artifactItem>
</artifactItems>
Expand Down

0 comments on commit 930f4c0

Please sign in to comment.