How do you upload files/folders to Pydio Cells using the Pydio Cells API - rest

Question

So far the API calls that appear to help me in getting to my end goal of eventually uploading or viewing files and folders via the API are as follows:
POST https://demo.pydio.com/a/tree/admin/list
POST https://demo.pydio.com/a/workspace
GET https://demo.pydio.com/a/config/datasource
GET https://demo.pydio.com/a/config/virtualnodes/
Pydio Cells API Documentation
https://pydio.com/en/docs/developer-guide/cells-api

Answer 1

Cells provides S3 api to interact with data. The action upload/download with curl is divided into steps:
1. Get jwt
2. Upload/Download
You can use following bash file:
./cells-download.sh CELL_IP:PORT USER PASSWORD CLIENT_SECRET FILENAME WORKSPACE_SLUG/PATH NEW_NAME_AFTTER_DOWNLOAD
./cells-upload.sh CELL_IP:PORT USER PASSWORD CLIENT_SECRET ABS_PATH_FILE NEW_NAME WORKSPACE_SLUG/PATH
CLIENT_SECRET is found in /home/pydio/.config/pydio/cells/pydio.json >> dex >> staticClients >> Secret:
cells-download.sh
=============================
#!/bin/bash
HOST=$1
CELLS_FRONT="cells-front"
CELLS_FRONT_PWD=$4
ADMIN_NAME=$2
ADMIN_PWD=$3
FILE=$5
DEST=$6
NEW_NAME=$7
AUTH_STRING=$(echo cells-front:$CELLS_FRONT_PWD | base64)
AUTH_STRING=${AUTH_STRING::-4}
JWT=$(curl -s --request POST \
--url http://$HOST/auth/dex/token \
--header "Authorization: Basic $AUTH_STRING" \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data "grant_type=password&username=$ADMIN_NAME&password=$ADMIN_PWD&scope=email%20profile%20pydio%20offline&nonce=123abcsfsdfdd" | jq '.id_token')
JWT=$(echo $JWT | sed "s/\"//g")
#!/bin/bash -e
#
# Copyright 2014 Tony Burns
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Upload a file to AWS S3.
file="${5}"
bucket="io"
prefix="io/$DEST"
region="us-east-1"
timestamp=$(date -u "+%Y-%m-%d %H:%M:%S")
content_type="application/octet-stream"
#signed_headers="date;host;x-amz-acl;x-amz-content-sha256;x-amz-date"
signed_headers="host;x-amz-content-sha256;x-amz-date"
if [[ $(uname) == "Darwin" ]]; then
iso_timestamp=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%Y%m%dT%H%M%SZ")
date_scope=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%Y%m%d")
date_header=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%a, %d %h %Y %T %Z")
else
iso_timestamp=$(date -ud "${timestamp}" "+%Y%m%dT%H%M%SZ")
date_scope=$(date -ud "${timestamp}" "+%Y%m%d")
date_header=$(date -ud "${timestamp}" "+%a, %d %h %Y %T %Z")
fi
payload_hash() {
# empty string
echo "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
canonical_request() {
echo "GET"
echo "/${prefix}/${file}"
echo ""
echo "host:$HOST"
echo "x-amz-content-sha256:$(payload_hash)"
echo "x-amz-date:${iso_timestamp}"
echo ""
echo "${signed_headers}"
printf "$(payload_hash)"
}
canonical_request_hash() {
local output=$(canonical_request | shasum -a 256)
echo "${output%% *}"
}
string_to_sign() {
echo "AWS4-HMAC-SHA256"
echo "${iso_timestamp}"
echo "${date_scope}/${region}/s3/aws4_request"
printf "$(canonical_request_hash)"
}
AWS_SECRET_ACCESS_KEY="gatewaysecret"
signature_key() {
local secret=$(printf "AWS4${AWS_SECRET_ACCESS_KEY}" | hex_key)
local date_key=$(printf ${date_scope} | hmac_sha256 "${secret}" | hex_key)
local region_key=$(printf ${region} | hmac_sha256 "${date_key}" | hex_key)
local service_key=$(printf "s3" | hmac_sha256 "${region_key}" | hex_key)
printf "aws4_request" | hmac_sha256 "${service_key}" | hex_key
}
hex_key() {
xxd -p -c 256
}
hmac_sha256() {
local hexkey=$1
openssl dgst -binary -sha256 -mac HMAC -macopt hexkey:${hexkey}
}
signature() {
string_to_sign | hmac_sha256 $(signature_key) | hex_key | sed "s/^.* //"
}
curl \
-H "Authorization: AWS4-HMAC-SHA256 Credential=${JWT}/${date_scope}/${region}/s3/aws4_request,SignedHeaders=${signed_headers},Signature=$(signature)" \
-H "Host: $HOST" \
-H "Date: ${date_header}" \
-H "x-amz-acl: public-read" \
-H 'Content-Type: application/octet-stream' \
-H "x-amz-content-sha256: $(payload_hash)" \
-H "x-amz-date: ${iso_timestamp}" \
"http://$HOST/${prefix}/${file}" --output $NEW_NAME
=============================
cells-upload.sh
=============================
#!/bin/bash
HOST=$1
CELLS_FRONT="cells-front"
CELLS_FRONT_PWD=$4
ADMIN_NAME=$2
ADMIN_PWD=$3
FILE=$5
NEW_NAME=$6
DEST=$7
AUTH_STRING=$(echo cells-front:$CELLS_FRONT_PWD | base64)
AUTH_STRING=${AUTH_STRING::-4}
JWT=$(curl -s --request POST \
--url http://$HOST/auth/dex/token \
--header "Authorization: Basic $AUTH_STRING" \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data "grant_type=password&username=$ADMIN_NAME&password=$ADMIN_PWD&scope=email%20profile%20pydio%20offline&nonce=123abcsfsdfdd" | jq '.id_token')
JWT=$(echo $JWT | sed "s/\"//g")
#!/bin/bash -e
#
# Copyright 2014 Tony Burns
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Upload a file to AWS S3.
file="${5}"
bucket="io"
prefix="io/$DEST"
region="us-east-1"
timestamp=$(date -u "+%Y-%m-%d %H:%M:%S")
content_type="application/octet-stream"
#signed_headers="date;host;x-amz-acl;x-amz-content-sha256;x-amz-date"
signed_headers="content-type;host;x-amz-acl;x-amz-content-sha256;x-amz-date"
if [[ $(uname) == "Darwin" ]]; then
iso_timestamp=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%Y%m%dT%H%M%SZ")
date_scope=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%Y%m%d")
date_header=$(date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%a, %d %h %Y %T %Z")
else
iso_timestamp=$(date -ud "${timestamp}" "+%Y%m%dT%H%M%SZ")
date_scope=$(date -ud "${timestamp}" "+%Y%m%d")
date_header=$(date -ud "${timestamp}" "+%a, %d %h %Y %T %Z")
fi
payload_hash() {
local output=$(shasum -ba 256 "$file")
echo "${output%% *}"
}
canonical_request() {
echo "PUT"
echo "/${prefix}/${NEW_NAME}"
echo ""
echo "content-type:${content_type}"
echo "host:$HOST"
echo "x-amz-acl:public-read"
echo "x-amz-content-sha256:$(payload_hash)"
echo "x-amz-date:${iso_timestamp}"
echo ""
echo "${signed_headers}"
printf "$(payload_hash)"
}
canonical_request_hash() {
local output=$(canonical_request | shasum -a 256)
echo "${output%% *}"
}
string_to_sign() {
echo "AWS4-HMAC-SHA256"
echo "${iso_timestamp}"
echo "${date_scope}/${region}/s3/aws4_request"
printf "$(canonical_request_hash)"
}
AWS_SECRET_ACCESS_KEY="gatewaysecret"
signature_key() {
local secret=$(printf "AWS4${AWS_SECRET_ACCESS_KEY}" | hex_key)
local date_key=$(printf ${date_scope} | hmac_sha256 "${secret}" | hex_key)
local region_key=$(printf ${region} | hmac_sha256 "${date_key}" | hex_key)
local service_key=$(printf "s3" | hmac_sha256 "${region_key}" | hex_key)
printf "aws4_request" | hmac_sha256 "${service_key}" | hex_key
}
hex_key() {
xxd -p -c 256
}
hmac_sha256() {
local hexkey=$1
openssl dgst -binary -sha256 -mac HMAC -macopt hexkey:${hexkey}
}
signature() {
string_to_sign | hmac_sha256 $(signature_key) | hex_key | sed "s/^.* //"
}
curl \
-T "${file}" \
-H "Authorization: AWS4-HMAC-SHA256 Credential=${JWT}/${date_scope}/${region}/s3/aws4_request,SignedHeaders=${signed_headers},Signature=$(signature)" \
-H "Host: $HOST" \
-H "Date: ${date_header}" \
-H "x-amz-acl: public-read" \
-H 'Content-Type: application/octet-stream' \
-H "x-amz-content-sha256: $(payload_hash)" \
-H "x-amz-date: ${iso_timestamp}" \
"http://$HOST/${prefix}/${NEW_NAME}"

Answer 2

Turns out my original thoughts regarding the Pydio Cells s3 buckets requiring an AWS account were wrong. Pydio Cells uses the same code or syntax (not sure 100%) that is used when working with AWS Buckets. The file system can be accessed using s3 buckets when working with the Pydio Endpoint https://demo.pydio.com/io. io is the s3 Bucket.
To test I am using Postman to first place a file named 'Query.sql' with content into the 'Personal Files' Workspace.
Authorization: AWS Signature
AccessKey: Token returned when using OpenID Connect. The "id_token" contained in the body.
SecretKey: The demo uses the key: 'gatewaysecret'
Advanced Options:
AWS Region: Default is 'us-east-1'. I didn't have to enter anything here but it still worked when I set it to 'us-west-1'.
Service Name: 's3' - I found that this is Required
Session Token: I left this blank.
Create files using PUT. Download files using GET.
PUT https://demo.pydio.com/io/personal-files/Query.sql
The below example shows how to first create a file and then pull it's content/download the file.
In my GET example I manually place a file named Query.sql onto the demo.pydio.com server in the Personal Files workspace. This example shows how to access the data and/or download the Query.sql file I manually placed into the Personal Files workspace.
GET https://demo.pydio.com/io/personal-files/Query.sql