Add support to include dependencies from p2 repository

[amitjoy]

It would be nice if one can include dependencies from p2 repository for the integration test. tycho-surefire-plugin has the provision to include dependencies from p2 repository to run integration tests.

[balazs-zsoldos]

The new version of eosgi-maven-plugin will be released pretty soon (currently an Eclipse M2E support is under hard development to be able to start and refresh environments from Eclipse).

Part of this release is that Eclipse Aether is used to resolve dependencies instead of the Maven 2.x API. I will have a quick look if Eclipse Aether can resolve dependencies from P2 repositories. If it can, I will add this to the next version. If not, I will have to have a deeper look, how P2 repositories and dependency resolution work.

[balazs-zsoldos]

@amitjoy How do you imagined to specify dependencies from P2 repositories? Specifying them in the configuration of the plugin, in maven dependencies or a plugin.xml file in the project?

May I ask what kind of project you develop that need dependencies from P2 projects? Server-side Java application, Eclipse plugin, IOT, ....

[amitjoy]

@balazs-zsoldos In tycho-surefire-plugin, this is achieved using the following way:

<dependencies>
    <dependency>
       <type>p2-installable-unit</type>
       <artifactId>org.eclipse.equinox.ds</artifactId>
    </dependency>

    <dependency>
       <type>p2-installable-unit</type>
       <artifactId>org.eclipse.core.jobs</artifactId>
    </dependency>
</dependencies>

This is how it is configured inside the configuration xml element. So, I believe, it could be achieved the same way. Just as a suggestion, it could be done as follows:

<bundleSettings>
        <bundle>
            <symbolicName>com.google.guava</symbolicName>
            <version>19.0.0</version>
            <startLevel>3</startLevel>
            <type>p2</type>
        </bundle>
</bundleSettings>

It should have an option of using type to specify where to get the dependency from or else it would by default use the maven dependencies as the source of the dependency. What do you think?

[amitjoy]

@balazs-zsoldos As you asked about the use of p2 in my project, most of the Eclipse Open Source Projects use p2 to manage its dependencies. As eosgi-maven-plugin is a full-fledged plugin to test the bundles in OSGi environment, it would be of great help if you can include managing dependencies from p2 repository. Apart from it, tycho-surefire-plugin only looks for dependencies from p2 repo and that's why, I believe eosgi-maven-plugin would be a great contribution to test bundles in OSGi environment.

[balazs-zsoldos]

Thanks for the info. I will have a look how tycho does this in the source code.

[balazs-zsoldos]

I think I will have a look how p2-maven-plugin works. It inspired the configuration of the 4.0.0 plugin anyway, where additional dependencies can be configured for the plugin (to be able to have two versions of something)

An example is here: https://github.com/reficio/p2-maven-plugin/blob/master/examples/p2/pom.xml

@amitjoy Do you think this can be a way? How about transitivity? At the moment, there is not transitive dependency resolution for those artifacts that are configured directly for the plugin. Would it be a big issue for you if p2 dependencies would not pull their dependencies (you should list them by hand)?

[amitjoy]

@balazs-zsoldos The p2-maven-plugin is an extended plugin from the actual maven tycho plugin from Eclipse. But you are right, the idea is to incorporate other dependencies while running integration test using eosgi-maven-plugin.

<plugin>
                <groupId>org.eclipse.tycho</groupId>
                <artifactId>tycho-surefire-plugin</artifactId>
                <version>${tycho-version}</version>
                <configuration>
                    <providerHint>junit4</providerHint>
                    <redirectTestOutputToFile>true</redirectTestOutputToFile>
                    <testSuite>dummy.test</testSuite>
                    <testClass>dummy.AllCoreTests</testClass>
                    <useUnlimitedThreads>true</useUnlimitedThreads>
            <bundleStartLevel>
            <bundle>
                <id>org.eclipse.equinox.ds</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.console</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.felix.gogo.shell</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.felix.gogo.command</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.felix.gogo.runtime</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.core.contenttype</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.core.jobs</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.core.runtime</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.io</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.app</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.cm</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.common</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.event</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.metatype</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.preferences</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.registry</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.util</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.osgi.services</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.osgi.util</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.hamcrest.core</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>osgi.cmpn</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.equinox.launcher</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.eclipse.soda.dk.comm</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>slf4j.api</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>com.google.protobuf</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>mqtt-client</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.hsqldb.hsqldb</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.junit</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.commons.io</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>javax.servlet</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.felix.dependencymanager</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            <bundle>
                <id>org.apache.felix.deploymentadmin</id>
                <level>4</level>
                <autoStart>true</autoStart>
            </bundle>
            </bundleStartLevel>
                    <dependencies>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.ds</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.console</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.felix.gogo.shell</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.felix.gogo.command</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.felix.gogo.runtime</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.core.contenttype</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.core.jobs</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.core.runtime</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.io</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.app</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.cm</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.common</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.event</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.metatype</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.preferences</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.registry</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.util</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.osgi</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.osgi.services</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.osgi.util</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.hamcrest.core</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>osgi.cmpn</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.equinox.launcher</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.eclipse.soda.dk.comm</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>slf4j.api</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>com.google.protobuf</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.hsqldb.hsqldb</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.junit</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.commons.io</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>javax.servlet</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>log4j</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>log4j.apache-log4j-extras</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.felix.dependencymanager</artifactId>
                        </dependency>
                        <dependency>
                            <type>p2-installable-unit</type>
                            <artifactId>org.apache.felix.deploymentadmin</artifactId>
                        </dependency>
                    </dependencies>
                </configuration>
            </plugin>

This is a sample configuration for tycho-surefire-plugin. So, in eosgi-maven-plugin, it would be configured in similar fashion where developers include other dependencies.

[balazs-zsoldos]

@amitjoy I have been thinking a lot about this and also I checked how p2-maven-plugin resolves the dependencies from P2 repositories. I have an idea that is a bit more generic than supporting P2 repositories.

In eosgi-maven-plugin 4.0.0 the configuration of additional dependencies will look like this:

<configuration>
  <artifacts>
    <artifact><id>groupId:artifactId:version</id></artifact>
    <artifact><id>groupId:artifactId:classifier:version</id></artifact>
  </artifacts>
</configuration>

The id is the coordinates of the maven dependency in the format that Aether uses. As with Aether, the user will be able to configure additional properties. Like the following:

<configuration>
  <artifacts>
    <artifact>
      <id>groupId:artifactId:version</id>
      <properties>
        <bundle.startLevel>4</bundle.startLevel>
        <bundle.autoStart>true</bundle.autoStart>
        <myCustomPropertyThatDistPackageUnderstands>xxx</myCustomPropertyThatDistPackageUnderstands>
      </properties>
    </artifact>
  </artifacts>
</configuration>

This works well with maven dependencies. As much as I see, plugins like tycho or p2-maven-plugin download p2 based dependencies into the target folder. That is problematic as if there is a clean install, the artifacts have to be downloaded again. I was thinking the availability of specifying a download url (any kind) like the following:

<configuration>
  <artifacts>
    <artifact>
      <id>myproject.p2dependencies:mydependency:0.1.1</id>
      <downloadURL>http://myp2repo/plugins/mydependency_0.1.1.jar</downloadURL>
    </artifact>
  </artifacts>
</configuration>

When download URL is present, the plugin will

• Check if maven can resolve the dependency first (probably from local repository)
• If not, it downloads the artifact from downloadURL and installs the dependency to the local maven repository with the groupId, artifactId and version that is specified in the id element.

What do you think? By having this solution, you can not only download artifacts from p2 repositories, but from anywhere for your project.

Direct P2 support could come later, but this generic solution could help until them (and it can be part of the next release as it is not that hard to implement).

[amitjoy]

@balazs-zsoldos Your solution sounds pretty good. But sometimes, I have seen that people don't want to expose the actual URL of every bundle from a p2 repository and that's why people want to set specific repositories in the maven configuration in the following way:

<repositories>
        <repository>
            <id>p2-repo-common</id>
            <layout>p2</layout>
            <url>file:///${project.basedir}/target-definition/common/repository/</url>
        </repository>
        <repository>
            <id>project_addons</id>
            <name>Project Addons Maven Repository</name>
            <url>https://raw.github.com/company/project_addons/mvn-repo/</url>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>always</updatePolicy>
            </snapshots>
        </repository>
    </repositories>

What if you provide an option whether to download the file from a specific URL (HTTP or FILE or anything else) or P2 which resolves the bundle from the p2 repositories. In such case, you have to provide an option to add p2 repositories in the Maven POM as shown above.

[balazs-zsoldos]

Let's make a deal :-)

There will be the downloadURL support first. Than the downloadURL will support the p2 protocol. Like this:

<configuration>
  <artifacts>
    <artifact>
      <id>myproject.p2dependencies:mydependency</id>
      <downloadURL>p2://artifactId:version</downloadURL>
    </artifact>
  </artifacts>
</configuration>

The original id should still be a real Aether coordinate as in this case the plugin will know where to save the downloaded artifact. In this case, it will search the artifact in the p2 repositories and download them to the local maven repository with myproject.p2dependencies groupId and myDependency artifactId.

[amitjoy]

Perfect proposal. I believe it would then be very easy for any OSGi developer to test OSGi bundles using eosgi testrunner.

Hope to see it soon.

[amitjoy]

Any probable release date?

[balazs-zsoldos]

I would like to release these as soon as possible. However, I would like to release this together with the eclipse plugin and the new version of TestRunner to be sure that no incompatible changes will be necessary in the future.

There is 1-2 days work with eosgi-maven-plugin, 1-2 days work with the testrunner and 1-2 weeks with the Eclipse plugin.

The TestRunner will be similar except that instead of the EOSGi-TestNum header an OSGi Capability has to be defined. Luckily it can be defined via an annotation. In practice: One has to add the @OSGiTest annotation to the class and no MANIFEST headers have to be edited manually in the future.

[balazs-zsoldos]

@amitjoy One more question. eosgi-maven-plugin is responsible to create a test environment and download artifacts for that. And here the question comes. Do you need those dependencies from those P2 sites to compile your code or only to test them?

If they are necessary during compilation, probably a separate plugin would solve the issue that runs before collecting the project dependencies.