Bintray support for Eclipse p2 repositories - eclipse

Question

Probably this is a recurring issue, but I could not find a reliable way to publish Eclipse p2 repositories on Bintray.
Manually creating a repo/product/version and populating with files was partly ok, but, for production environments, a reliable scriptable solution is needed.
Purpose
Deploy Eclipse p2 repositories to Bintray.
What are Eclipse p2 repositories?
(Sorry Eclipse folks, but for Bintry support people we better define what we are talking about).
An Eclipse p2 repository is a folder which must be published at a single URL that is stable and does not change, even if multiple versions are published in time.
An Eclipse p2 repository folder, generated with the latest versions of the Tycho Maven plugins, has the following structure:
5 files in root (p2.index, artifacts.jar, artifacts.xml.xz, content.jar, content.xml.xz)
2 sub-folders, plugins and features, each with multiple .jar files, with version specific names, like ilg.gnuarmeclipse.core_3.3.1.201702251311.jar
For example:
artifacts.jar
artifacts.xml.xz
content.jar
content.xml.xz
features
ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar
...
p2.index
plugins
ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar
...
The exact p2 repository folders that I want to deploy are:
https://sourceforge.net/projects/gnuarmeclipse/files/Eclipse/updates/
https://sourceforge.net/projects/gnuarmeclipse/files/Eclipse/updates-test/
(both part of the GNU ARM Eclipse project).
The actual URLs that must be configured in Eclipse to access these two p2 repositories are:
http://gnuarmeclipse.sourceforge.net/updates
http://gnuarmeclipse.sourceforge.net/updates-test
Access to these p2 repositories is actually a sequence of accesses to files directly beneath these URLs, like:
$ curl -L http://gnuarmeclipse.sourceforge.net/updates/p2.index
#Sat Feb 25 15:11:37 EET 2017 version=1
metadata.repository.factory.order=content.xml.xz,content.xml,\!
artifact.repository.factory.order=artifacts.xml.xz,artifacts.xml,\!
$
Use of version specific sub-folders
Eclipse p2 repositories to not have version specific sub-folders, both subfolders used (plugins and features) have the same name in each version; it is not possible to access version specific sub-folders.
As a consequence, deploying multiple versions should not create version specific subfolders, since their content will be ignored.
Use of version specific URLs
Eclipse plug-ins have configured inside them a single URL that can be used to automatically get new updates. This is the URL of a p2 repository, and it cannot be changed to point to version specific URLs, so, for updates to work, the p2 repository should have a unique URL.
Eclipse p2 repository lifecycle
The lifecycle of an Eclipse p2 repository should allow new versions to completely replace the previous version, i.e. the top 5 files and the two-subfolders should all be part of a single version; if, for any reason, publishing fails, the previous version should continue to be visible, unchanged
once a version is released, the files associated with it will never change, so it is not necessary to allow a given file to be replaced with a file with the same name, but different content
however, top files and folders have the same names for all releases, and the server should allow uploading them without complaining that the name was already uploaded by a previous version
the moments in time when new versions are released are not known in advance, there might be releases every month, but there might also be releases at more than 180 days apart
Publish to the product/version URL
The first attempt was to upload all files to the product/version URL, using the following bash function:
curl \
--request PUT \
--upload-file "${file_absolute_path}" \
--user ${BINTRAY_USER}:${BINTRAY_API_KEY} \
"${API}/content/${BINTRAY_OWNER}/${repo}/${package}/${version}/${file_relative_path}?publish=1?override=1?explode=0"
The upload was successful:
Processing artifacts.jar file...
{"message":"success"}
Processing artifacts.xml.xz file...
{"message":"success"}
Processing content.jar file...
{"message":"success"}
Processing content.xml.xz file...
{"message":"success"}
Processing p2.index file...
{"message":"success"}
Processing feature: features/ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar file...
{"message":"success"}
Processing plugin: plugins/ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar file...
{"message":"success"}
but, although all files were uploaded identically, some of the files were stored in the repo folder, not in the product/version folder, as expected:
artifacts.xml.xz
content.xml.xz
features
ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar
pack3
3.2.1-201701141320
artifacts.jar
content.jar
p2.index
plugins
ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar
Please note that although I did not explicitly set the list_in_downloads property to any of the files, some of the files, uploaded to product/version were moved to the parent repo folder.
As it can be seen, the *.xz files and the features and plugins folders were promoted to the repo folder, while the *.jar files and the p2.index file were ignored.
A repository created with this procedure is:
https://dl.bintray.com/ilg-ul/repo3/
Publish to the product/version URL with different POST methods
As documented, there are 3 methods of passing parameters to curl. The previous test used one; in two more tests, I tried the next two, with the following upload code:
curl \
--request PUT \
--upload-file "${file_absolute_path}" \
--user ${BINTRAY_USER}:${BINTRAY_API_KEY} \
--header "X-Bintray-Package: ${package}" \
--header "X-Bintray-Version: ${version}" \
--header "X-Bintray-Publish: 1" \
--header "X-Bintray-Override: 1" \
--header "X-Bintray-Explode: 0" \
"${API}/content/${BINTRAY_OWNER}/${repo}/${file_relative_path}"
and separately with
curl \
--request PUT \
--upload-file "${file_absolute_path}" \
--user ${BINTRAY_USER}:${BINTRAY_API_KEY} \
"${API}/content/${BINTRAY_OWNER}/${repo}/${file_relative_path};bt_package=${package};bt_version=${version};publish=1;override=1;explode=0"
Both behaved better than the previous test, the upload for the first version was successful and the folder structure was preserved:
artifacts.jar
artifacts.xml.xz
content.jar
content.xml.xz
features
ilg.gnuarmeclipse.codered_1.1.1.201701141320.jar
p2.index
plugins
ilg.gnuarmeclipse.codered_1.1.1.201701141320.jar
but when uploading the second version, most files were ok, except that uploading artifacts.xml.xz and content.xml.xz failed:
Upload 'artifacts.jar' to '/repo6/pack6/3.3.1-201702251311/'
{"message":"success"}
Upload 'artifacts.xml.xz' to '/repo6/pack6/3.3.1-201702251311/'
{"message":"Unable to upload files: An artifact with the path 'artifacts.xml.xz' already exists under another version"}
Upload 'content.jar' to '/repo6/pack6/3.3.1-201702251311/'
{"message":"success"}
Upload 'content.xml.xz' to '/repo6/pack6/3.3.1-201702251311/'
{"message":"Unable to upload files: An artifact with the path 'content.xml.xz' already exists under another version"}
Upload 'p2.index' to '/repo6/pack6/3.3.1-201702251311/'
{"message":"success"}
...
Please note that, as far as I can tell, there is nothing special with these files.
A repository created using this procedure is
https://dl.bintray.com/ilg-ul/repo6/
Although it looks like a valid p2 repository, it isn't, since most files are from the second version, but artifacts.xml.xz and content.xml.xz are from the first version, so the repository is not consistent.
Publish to the repo URL
Although not officially mentioned in the Bintray documentation, some suggested to try to upload to a shorter path, corresponding to the root, or repo URL.
I did, using the following code:
curl \
--request PUT \
--upload-file "${file_absolute_path}" \
--user ${BINTRAY_USER}:${BINTRAY_API_KEY} \
"${API}/content/${BINTRAY_OWNER}/${repo}/${file_relative_path}?publish=1?override=1"
but in this case I got errors for most of the files:
Processing artifacts.jar file...
{"message":"success"}
Processing artifacts.xml.xz file...
{"message":"Invalid file path and name"}
Processing content.jar file...
{"message":"success"}
Processing content.xml.xz file...
{"message":"Invalid file path and name"}
Processing p2.index file...
{"message":"success"}
Processing feature: features/ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar file...
{"message":"Invalid file path and name"}
Processing plugin: plugins/ilg.gnuarmeclipse.codered_1.1.1.201702231729.jar file...
{"message":"Invalid file path and name"}
It looks like the upload mechanism is picky, and accepts to upload some files (like artifacts.jar, content.jar, and p2.index), to the repo URL, but for all other files it fails.
A repository created with this procedure is:
https://dl.bintray.com/ilg-ul/repo1/
Publish both to repo and product/version URLs
I also tried to selectively upload some files to the repo and some files to the product/version (artifacts.xml.xz, content.xml.xz and the features/plugins folders); this created a correct p2, but when I tried to repeat the process for another version, I got errors:
Processing artifacts.jar file...
{"message":"success"}
Processing artifacts.xml.xz file...
{"message":"Unable to upload files: An artifact with the path 'artifacts.xml.xz' already exists under another version"}
Processing content.jar file...
{"message":"success"}
Processing content.xml.xz file...
{"message":"Unable to upload files: An artifact with the path 'content.xml.xz' already exists under another version"}
Processing p2.index file...
{"message":"success"}
The override flag
Please note that the override flag was set on all tests.
The publish flag
Please note that the publish flag was set on all tests, although this is not the expected behaviour.
To keep repositories consistent, the expected behaviour is to try to upload all files without the publish flag, and to do the publish at the end, only if all files were correctly uploaded; if an error occur, without the publish command issued, it is expected that the files published for the previous version will remain accessible.
The complete test script
The complete bash script used for these tests (and a few more) is available from GitHub gists:
https://gist.github.com/ilg-ul/568a6806d5e97fcc1384d7acda4ffe36
To download this script, use the following
mkdir -p "${HOME}/Downloads"
curl -L https://gist.github.com/ilg-ul/568a6806d5e97fcc1384d7acda4ffe36/raw/2df98f4899862f1d7e65f1601ccdbd320dce9021/bintray-test.sh -o "${HOME}/Downloads/bintray-test.sh"
This script expects the following variables to be set in the environment:
export BINTRAY_USER=<user>
export BINTRAY_API_KEY=<auth>
export BINTRAY_OWNER=${BINTRAY_USER}
To run the script, enter:
bash "${HOME}/Downloads/bintray-test.sh"
Problems identified
Refusal of the server to upload the artifacts.xml.xz and content.xml.xz
Considering that publishing to the product/version URL with different POST methods (repo6) was the most advanced test, the only problem identified was the refusal of the server to upload the artifacts.xml.xz and content.xml.xz.
Creating intermediate folders and storing content
Passing the package and version as part o the URL (repo3) produced the most bizarre results, with additional folders:
pack3
3.2.1-201701141320
artifacts.jar
content.jar
p2.index
All other files were uploaded correctly, but these three files were processed in a special (and I would say erroneous) manner.
Attempts to publish to the repo URL fail for most files
If this is not a legal way of publishing to Bintray, please ignore section, but attempts to publish to the repo URL were successful only for the following 3 files artifacts.jar, content.jar, and p2.index) and failed for all other.
Conclusion
As a conclusion, based on the existing documentation, I could not find a reliable method to publish usual Eclipse p2 repositories to Bintray.
I saw several proposals with curious solutions to post composite p2 repositories, but this is not my case, I have two common repositories, which do not need any versioning (http://gnuarmeclipse.sourceforge.net/updates and http://gnuarmeclipse.sourceforge.net/updates-test), and I would like to publish them on Bintray.
Suggestions for Bintray
Remove special processing for some files in Generic repositories
As proved by these tests, Bintray generic repositories are not that generic, since they do not process all files equally, as expected; it looks like attempts to support Eclipse p2 repositories were made, and the server upload code was patched to process some Eclipse files differently, but the result is not fully functional, and very confusing.
Add explicit support for Eclipse p2 repositories
Instead of making unfortunate patches to the Generic repo, it would be great if Bintray would support a new repository type "Eclipse p2", where there are no products nor versions, and each publish will be allowed to remove all existing files and add the new ones.
This is equivalent of being allowed to publish in the repo folder, and being allowed to remove and upload again files later, at any time.
If getting rid of the versioning mechanism is not possible, it would be acceptable to publish to the version folder, but automagically the files from the latest version to be also visible in the product folder, as in repo6, but to be sure all files are accepted, including artifacts.xml.xz and content.xml.xz.
2017-03-31 Update
After countless messages exchanged with Bintray support, they finally understood the problem and provided a fix.
Running the script now is functional for tests 4, 5 and 6, which, basically, are identical, except small variations in how the information is passed to Bintray.
The results of the test are:
Upload to the repo root in not functional
Direct upload, specifying package and version as a path prefix to the file's target path in not functional
Uploading while specifying package and version using HTTP headers is functional
Uploading while specifying package and version using HTTP matrix parameters is functional
In conclusion, do not try to upload to the root url and use HTTP headers or HTTP matrix parameters.
2017-07-31 Update
I'm hosting the update site on Bintray for a few months already, and things seem ok: https://bintray.com/gnu-mcu-eclipse.
The actual script used to publish is: https://github.com/gnu-mcu-eclipse/eclipse-plugins/blob/develop/scripts/publish-updates.sh
The public URLs used for the update site look like: https://dl.bintray.com/gnu-mcu-eclipse/updates.
Actually there are multiple Bintray repositories, for different 'stages' of the project (https://bintray.com/gnu-mcu-eclipse); below them there is a single Bintray package (I called it p2) and below this are multiple Bintray versions (https://bintray.com/gnu-mcu-eclipse/v4-neon-updates-experimental/p2).

Answer 1

Last year I was struggling with exactly the same problem
when trying to upload a simple Eclipse p2 repository via curl to Bintray.
Inspired by an article of Lorenzo Bettini
I found a solution.
The key is to use path and matrix parameters in the URL, for example like this:
curl -X PUT -T $F -u $BINTRAY_USER:$BINTRAY_API_KEY "https://api.bintray.com/content/$BT_OWNER/$BT_REPO/$BT_PACKAGE/$BT_VERSION/$F;bt_package=$BT_PACKAGE;bt_version=$BT_VERSION;publish=1"
Feel free to look at my shell script deployToBintray.sh.

Answer 2

I am hosting my eclipse project on bintray and had also some problems but was able to fix:
Preparation
created a generic repository - e.g. "myproject"
created one package inside - e.g. "update-site"
created one version - e.g. "current"
So at my project I wanted to use only one URL for the update site which will not change and simply put all new content at same bintray-version ("current"). site.xml, content.jar etc. will be overriden.
HowTo upload with override
The "new" REST API has the ability to override files.
PUT /content/:subject/:repo/:file_path;bt_package=:package;bt_version=:version[;publish=0/1][;override=0/1][;explode=0/1]
I am using an adopted version of https://github.com/vogellacompany/bintray-publish-p2-updatesite/blob/master/pushToBintray.sh to upload my files.
180 day override problem and how to fix
When you look at https://bintray.com/docs/api/index.html#_content_uploading_publishing
there exists a passus:
Optionally publishing the uploaded artifact(s) as part of uploading
(off by default). Additional content can be uploaded to a published
version within 180 days from its publish date.
And I did run into the problem after 180 days I was not able to upload any longer...
But there is an easy way to solve:
Edit your version
https://bintray.com/$userName/myProject/update-site/current/edit?tab=general
and just klick on "Update Version" button.
After doing this I was able to upload again, so it seems this resets the 180 days. Currently I hosting my upate-site nearly one year now and have no problems with updating etc.