Discussion:
Preferred way to offer zipped, standalone update sites?
(too old to reply)
Andreas Sewe
2015-03-26 14:37:40 UTC
Permalink
Hi,

some of our users are working behind corporate firewalls which block or
otherwise disfigure .jar or .pack.gz files. To accommodate those users,
I am searching for a solution that packs (.zip and/or .tar.gz) a
(composite) update site such that it can be used standalone. However,
standalone should not mean completely self-contained. No need to ship
the entire Eclipse IDE, just to deliver our plugins...

Here's the solutions I have explored so far:

- Use tycho-p2-extras-plugin:mirror [1] + assembly:single

The mirror goal gives me pretty good control over why update sites are
included. Unfortunately, the goal seems to download all IUs anew on each
execution [2]. :-( Also, the maven-assembly-plugin is a bit heavy-weight
for just packing a directory.

- Use the tycho-p2-repository-plugin's includeAllDependencies option

AFAIK, includeAllDependencies takes its contents from the target
platform, which may or may not be the same content that the users of our
composite update site see. :-(

Also, I would have to create a new, nearly identical eclipse-repsitory
project (the only difference to the existing one being the value of
includeAllDependencies); while I can run multiple executions of
tycho-p2-repository:assemble-repository and :assembly-repository in a
single eclipse-repository project, I cannot get one of the executions to
use an output directory different from
${project.build.directory}/repository so I have to have another project.

Are there any other options I have missed?

Any pointers are greatly appreciated.

Andreas

[1]
<http://eclipse.org/tycho/sitedocs-extras/tycho-p2-extras-plugin/mirror-mojo.html>
[2] <https://bugs.eclipse.org/bugs/show_bug.cgi?id=463203>
[3]
<http://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-repository-plugin/assemble-repository-mojo.html#includeAllDependencies>
--
Codetrails GmbH
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Phone: +49-6151-276-7092
Mobile: +49-170-811-3791
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940
Jens Reimann
2015-03-26 15:22:14 UTC
Permalink
Hi,

first of all I have to say that I am the author of what I am suggest
next ;-)

You could use Package Drone [1], which allows you to host e.g. P2
repositories. You can upload a full P2 repository zip (manually or using
mvn deploy) and consume it as P2 repository. You can upload multiple P2
Zips into one "channel". Each channel will be rendered as a P2
repository, so you can combine multiple P2 repository. Although this is
not a composite repository, from a technical point of view. So the
amount of data you upload is under you control.

For the Maven Tycho build, or Eclipse PDE, you can then reference the
repository over HTTP.

If you need a bit of help, just let me know.

Jens

[1] http://packagedrone.org/
[2] https://github.com/ctron/package-drone
Post by Andreas Sewe
Hi,
some of our users are working behind corporate firewalls which block or
otherwise disfigure .jar or .pack.gz files. To accommodate those users,
I am searching for a solution that packs (.zip and/or .tar.gz) a
(composite) update site such that it can be used standalone. However,
standalone should not mean completely self-contained. No need to ship
the entire Eclipse IDE, just to deliver our plugins...
- Use tycho-p2-extras-plugin:mirror [1] + assembly:single
The mirror goal gives me pretty good control over why update sites are
included. Unfortunately, the goal seems to download all IUs anew on each
execution [2]. :-( Also, the maven-assembly-plugin is a bit heavy-weight
for just packing a directory.
- Use the tycho-p2-repository-plugin's includeAllDependencies option
AFAIK, includeAllDependencies takes its contents from the target
platform, which may or may not be the same content that the users of our
composite update site see. :-(
Also, I would have to create a new, nearly identical eclipse-repsitory
project (the only difference to the existing one being the value of
includeAllDependencies); while I can run multiple executions of
tycho-p2-repository:assemble-repository and :assembly-repository in a
single eclipse-repository project, I cannot get one of the executions to
use an output directory different from
${project.build.directory}/repository so I have to have another project.
Are there any other options I have missed?
Any pointers are greatly appreciated.
Andreas
[1]
<http://eclipse.org/tycho/sitedocs-extras/tycho-p2-extras-plugin/mirror-mojo.html>
[2] <https://bugs.eclipse.org/bugs/show_bug.cgi?id=463203>
[3]
<http://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-repository-plugin/assemble-repository-mojo.html#includeAllDependencies>
Andreas Sewe
2015-03-26 16:00:05 UTC
Permalink
Hi Jens,
Post by Jens Reimann
first of all I have to say that I am the author of what I am suggest
next ;-)
You could use Package Drone [1], which allows you to host e.g. P2
repositories. You can upload a full P2 repository zip (manually or using
mvn deploy) and consume it as P2 repository. You can upload multiple P2
Zips into one "channel". Each channel will be rendered as a P2
repository, so you can combine multiple P2 repository. Although this is
not a composite repository, from a technical point of view. So the
amount of data you upload is under you control.
that sounds very useful (a bit like the Nexus Unzip plugin) if you have
a zipped update site to host. However, my original question was more
like "How can I create such a zipped update site?"

My inputs are

- An eclipse-repository produced in the current build
- A composite update site pointing to several external p2 repositories

My output should be

- A packed p2 repository containing all my inputs

Hope that clarifies things.

Best wishes,

Andreas
--
Codetrails GmbH
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Phone: +49-6151-276-7092
Mobile: +49-170-811-3791
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940
Jens Reimann
2015-03-26 16:06:15 UTC
Permalink
Hi,

I see .. so every time you push a new eclipse repository from the
current build, you would like to refresh the composite repository to
contain all required (not all) dependencies?

Or should the composite update site only be mirrored once?

Jens
Post by Andreas Sewe
Hi Jens,
Post by Jens Reimann
first of all I have to say that I am the author of what I am suggest
next ;-)
You could use Package Drone [1], which allows you to host e.g. P2
repositories. You can upload a full P2 repository zip (manually or using
mvn deploy) and consume it as P2 repository. You can upload multiple P2
Zips into one "channel". Each channel will be rendered as a P2
repository, so you can combine multiple P2 repository. Although this is
not a composite repository, from a technical point of view. So the
amount of data you upload is under you control.
that sounds very useful (a bit like the Nexus Unzip plugin) if you have
a zipped update site to host. However, my original question was more
like "How can I create such a zipped update site?"
My inputs are
- An eclipse-repository produced in the current build
- A composite update site pointing to several external p2 repositories
My output should be
- A packed p2 repository containing all my inputs
Hope that clarifies things.
Best wishes,
Andreas
Andreas Sewe
2015-03-26 16:28:56 UTC
Permalink
Hi,
Post by Jens Reimann
I see .. so every time you push a new eclipse repository from the
current build, you would like to refresh the composite repository to
contain all required (not all) dependencies?
Or should the composite update site only be mirrored once?
our Tycho build builds a couple of eclipse-repository projects (aka
update sites) each time it runs. Different update sites (head,
milestones, stable, ...) are promoted (copied to a webserver) at
different times, but all update sites are assembled during *every*
build. This doesn't slow down the build very much (as the update sites
contents are either cached or produced by the current reactor) and
prevents nasty surprises; we know that all update sites can be assembled
at any time.

Now, in addition to plain old eclipse-repository update sites, I would
also like to build an archive (.zip, .tar.gz, or similar) containing the
contents of one or more eclipse-repositories from the reactor *plus* the
contents of some external update sites. Again, I would like to
reassemble this archive during every build.

Now, these external update sites should of course not be downloaded in
their entirety upon each and every build, but upstream changes should be
picked up automatically; thus, it is fine if every build downloads a
remote content.jar, but it is not if downloads all bundles and features
(sort of like how Tycho deals with the target platform).

Hope that explains things a bit better.

Andreas
--
Codetrails GmbH
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Phone: +49-6151-276-7092
Mobile: +49-170-811-3791
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940
Jens Reimann
2015-03-26 16:34:53 UTC
Permalink
Thanks for the explanation.

So actually what you want is some kind of "P2 proxy". Which picks up
changes in the meta data, but does not re-download the unchanged JARs?

So I would suggest to use some simple HTTP proxy, like Squid. And
configure it as transparent proxy. Tune it in a way that all content is
cached for a longer period of time, with the exception of files called
"(compositeContent|compositeArtifacts|context|artifacts)\.(jar|xml)".

Would that do the trick?

Jens
Post by Andreas Sewe
Hi,
Post by Jens Reimann
I see .. so every time you push a new eclipse repository from the
current build, you would like to refresh the composite repository to
contain all required (not all) dependencies?
Or should the composite update site only be mirrored once?
our Tycho build builds a couple of eclipse-repository projects (aka
update sites) each time it runs. Different update sites (head,
milestones, stable, ...) are promoted (copied to a webserver) at
different times, but all update sites are assembled during *every*
build. This doesn't slow down the build very much (as the update sites
contents are either cached or produced by the current reactor) and
prevents nasty surprises; we know that all update sites can be assembled
at any time.
Now, in addition to plain old eclipse-repository update sites, I would
also like to build an archive (.zip, .tar.gz, or similar) containing the
contents of one or more eclipse-repositories from the reactor *plus* the
contents of some external update sites. Again, I would like to
reassemble this archive during every build.
Now, these external update sites should of course not be downloaded in
their entirety upon each and every build, but upstream changes should be
picked up automatically; thus, it is fine if every build downloads a
remote content.jar, but it is not if downloads all bundles and features
(sort of like how Tycho deals with the target platform).
Hope that explains things a bit better.
Andreas
--
IBH SYSTEMS GmbH
D-85235 Pfaffenhofen an der Glonn
Läutenring 43
Geschäftsführer / CEO: Dr. Thomas Heitzig

Amtsgericht München
Handelsregister Nummer HRB 197959
USt ID: DE267945175

Office Munich
D 80992 München
Agnes-Pockels-Bogen 1
T +49 89 18 9 17 49 0

The information transmitted is intended only for the person or entity
to which it is addressed and may contain confidential and/or pivileged
material. Any review, retransmission, dissemination or other use of,
or taking of any action in reliance upon, this information by persons
or entities other than the intended recipient is prohibited. If you
received this in error, please contact the sender and delete the
material from any computer.
Andreas Sewe
2015-03-28 14:40:42 UTC
Permalink
Hi Jens,
Post by Jens Reimann
Thanks for the explanation.
So actually what you want is some kind of "P2 proxy". Which picks up
changes in the meta data, but does not re-download the unchanged JARs?
So I would suggest to use some simple HTTP proxy, like Squid. And
configure it as transparent proxy. Tune it in a way that all content is
cached for a longer period of time, with the exception of files called
"(compositeContent|compositeArtifacts|context|artifacts)\.(jar|xml)".
Would that do the trick?
Could be, but you would introduce another moving part with external
configuration (not in pom.xml or Hudson config) into the build, which I
would like to avoid.

Since there already exits a cache of target platform contents (in
~/.m2/repository/p2) maintained by Tycho, I had hoped for that there
exists a goal that can mirror repositories while re-using the existing
cache.

(FWIW, just tested target-platform-utils:mirror-target-to-repo from
JBoss Tools Maven plugins [1] and it doesn't cache artifacts either.
Just like tycho-p2-extras-plugin:mirror, the goal uses Tycho's
MirrorApplicationService, which apparently doesn't cache.)

Best wishes,

Andreas

[1] <https://github.com/jbosstools/jbosstools-maven-plugins>
--
Codetrails GmbH
The knowledge transfer company

Robert-Bosch-Str. 7, 64293 Darmstadt
Phone: +49-6151-276-7092
Mobile: +49-170-811-3791
http://www.codetrails.com/

Managing Director: Dr. Marcel Bruch
Handelsregister: Darmstadt HRB 91940
Continue reading on narkive:
Loading...