How to release a build artifact asset on GitHub with a script? - github

I am trying to figure out a one-command process for generating a build on GitHub.
What I anticipate doing is running some sort of command- make release, say, and the make release script builds up the release artifact and then uploads it to GitHub in some fashion.
However, I'm fairly confused about how to actually get a release artifact on GitHub. Source code is awesome, but not everyone wants to do their own builds. :-)

Update 2022: The official GitHub CLI comes with gh release upload
Upload asset files to a GitHub Release.
You can create the release first with gh release create
Upload all tarballs in a directory as release assets
$ gh release create v1.2.3 ./dist/*.tgz
Upload a release asset with a display label
$ gh release create v1.2.3 '/path/to/asset.zip#My display label'
Update September 2013, you can automate a release (API in preview mode)
Update January 2014, there's an unofficial command-line app, called github-release by Nicolas Hillegeer (aktau), for creating releases and uploading (binary) artifacts.
It uses the new github releases API mentioned above. Look at the Makefile of the project to see how to automate it more still.
Example:
# create a formal release
$ github-release release \
--user aktau \
--repo gofinance \
--tag v0.1.0 \
--name "the wolf of source street" \
--description "Not a movie, contrary to popular opinion. Still, my first release!" \
--pre-release
This API is a little different due to the binary assets. We use the Accept header for content negotation when requesting a release asset.
Pass a standard API media type to get the API representation:
$ curl -i -H "Authorization: token TOKEN" \
-H "Accept: application/vnd.github.manifold-preview" \
"https://uploads.github.com/repos/hubot/singularity/releases/assets/123"
HTTP/1.1 200 OK
{
"id": 123,
...
}
Pass “application/octet-stream” to download the binary content.
$ curl -i -H "Authorization: token TOKEN" \
-H "Accept: application/octet-stream" \
"https://uploads.github.com/repos/hubot/singularity/releases/assets/123"
HTTP/1.1 302 Found
Uploads are handled by a single request to a companion “uploads.github.com” service.
$ curl -H "Authorization: token TOKEN" \
-H "Accept: application/vnd.github.manifold-preview" \
-H "Content-Type: application/zip" \
--data-binary #build/mac/package.zip \
"https://uploads.github.com/repos/hubot/singularity/releases/123/assets?name=1.0.0-mac.zip"
Update 2d July 2013, you now can define a release.
Releases are accompanied by release notes and links to download the software or source code.
Following the conventions of many Git projects, releases are tied to Git tags. You can use an existing tag, or let releases create the tag when it's published.
You can also attach binary assets (such as compiled executables, minified scripts, documentation) to a release. Once published, the release details and assets are available to anyone that can view the repository.
This is what replaces the old binary upload service, which was removed in December 2012!
the make release script builds up the release artifact and then uploads it to github in some fashion.
That would mean adding it ("it" being the delivery made of one or several files, generally including binaries) to a regular local repo, and then pushing that repo to its matching GitHub repo.
That being said, the reason GitHub isn't mention in any "release" task is because Git is a source control management system, and is ill-suited for binaries.
It can have those files (binaries) of course, but isn't made to have them regularly, because of the bloated size of the repo after a while: each cloning would take longer and longer.
See What are the Git limits, and also "git - should source files and repository be on the same machine ?".

Preparation:
1) Download github-releases and put its executable in your PATH.
2) Create a token at https://github.com/settings/applications#personal-access-tokens let's say abc123
Uploading an artifact:
1) Let's say you have just compiled what you decide to call version 3.1, and want to upload it.
2) Make sure you have committed everything.
3) Run these five commands:
git tag v3.1
git push
git push --tags
github-release release --security-token abc123 --user <you> --repo <yourrepo> \
--tag v3.1
github-release upload --security-token abc123 --user <you> --repo <yourrepo> \
--tag v3.1 --name <thefile> --file <thefile>
You can upload several files, for instance for different operating systems.
(Based on VonC's answer, which unfortunately does not detail how to upload an artifact)

hub official Go-based GitHub CLI tool
https://github.com/github/hub
First install Go. On Ubuntu: https://askubuntu.com/questions/959932/installation-instructions-for-golang-1-9-into-ubuntu-16-04/1075726#1075726
Then install hub:
go get github.com/github/hub
There is no Ubuntu package: https://github.com/github/hub/issues/718
Then from inside your repo:
hub release create -a prebuilt.zip -m 'release title' tag-name
This:
prompts for your password the first time, and then automatically creates and stores an API token locally
creates a non annotated tag on the remote called tag-name
creates a release associated to that tag
uploads prebuilt.zip as an attachment
You can also provide your existing API token with the GITHUB_TOKEN environment variable.
For other release operations, see:
hub release --help
Tested on hub de684cb613c47572cc9ec90d4fd73eef80aef09c.
Python APIv3 upload example without any external dependencies
Usage:
GITHUB_TOKEN=<token> ./create-release username/reponame <tag-name> <path-to-upload>
Script:
#!/usr/bin/env python3
import json
import os
import sys
from urllib.parse import urlencode
from urllib.request import Request, urlopen
repo = sys.argv[1]
tag = sys.argv[2]
upload_file = sys.argv[3]
token = os.environ['GITHUB_TOKEN']
url_template = 'https://{}.github.com/repos/' + repo + '/releases'
# Create.
_json = json.loads(urlopen(Request(
url_template.format('api'),
json.dumps({
'tag_name': tag,
'name': tag,
'prerelease': True,
}).encode(),
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
},
)).read().decode())
# This is not the tag, but rather some database integer identifier.
release_id = _json['id']
# Upload.
with open(upload_file, 'br') as myfile:
content = myfile.read()
_json = json.loads(urlopen(Request(
url_template.format('uploads') + '/' + str(release_id) + '/assets?' \
+ urlencode({'name': os.path.split(upload_file)[1]}),
content,
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
'Content-Type': 'application/zip',
},
)).read().decode())
Both release and asset creation will fail with 422 if they already exist. Work around that by first deleting the release or asset. Here is an example.

If you use Maven, you can add GitHub's Downloads Maven Plugin ( https://github.com/github/maven-plugins/#downloads-plugin ) and simply do:
$ mvn clean install ghDownloads:upload

Github has an API to access their own file download system.
Repo downloads allow you to provide binaries for users - although there may be a limit to the size and number. The API allows access from automated agents.
Take a look at:
http://developer.github.com/v3/repos/downloads/ for usage info.
The feature isn't in use much, but definitely works. You can go to any github repo, click the "Downloads" tab to see them.
For an example of downloadable files:
http://github.com/dannystaple/emacs_cheat_sheets/downloads - the HTML file offered there is actually a built artefact, and not in the source. I am trying to rustle up a better (binary) example - but there is no reason that executables, zips/tarballs and other filetypes couldn't be offered.
These downloads are NOT the same as source tarballs for a repo or its tags. Any arbitrary file can be uploaded this way.

I had the same problem, hacked up a little python to do it for me. I must say it was a pain, s3 is a total freakshow.
https://raw.github.com/reklis/utilityscripts/master/github-upload
#!/opt/local/bin/python2.7
import json
import requests
import sys
import argparse
import os
import mimetypes
import pycurl
import cStringIO
from xml.dom import minidom
github_api_root = "https://api.github.com/"
def parse_args():
parser = argparse.ArgumentParser(description='post a file to github as a download')
parser.add_argument('--user', dest='user', help='github username', required=True)
parser.add_argument('--pass', dest='password', help='github password', required=True)
parser.add_argument('--repo', dest='repo', help='the name of the github repo', required=True)
parser.add_argument('--file', dest='filepath', help='path of the local file to upload', required=True)
parser.add_argument('--desc', dest='description', help='descriptive text about this file', required=True)
parser.add_argument('--owner', dest='owner', help='owner of the github repository', required=True)
args = parser.parse_args()
# print args
return args
def make_dl_post_url(owner, repo):
url = "%srepos/%s/%s/downloads" % (str(github_api_root), str(owner), str(repo))
# print url
return url
def make_dl_delete_url(owner, repo, dlid):
url = "%srepos/%s/%s/downloads/%s" % (str(github_api_root), str(owner), str(repo), str(dlid))
# print url
return url
def add_github_reference(args):
dl_post_url = make_dl_post_url(args.owner, args.repo)
fp = args.filepath
filename = os.path.basename(fp)
filesize = os.path.getsize(fp)
mtype, mdetails = mimetypes.guess_type(fp)
file_description = {
'name': filename,
'size': filesize,
'description': args.description,
'content_type': mtype
}
# print json.dumps(file_description, indent=2)
github = requests.post(dl_post_url, auth=(args.user, args.password), data=json.dumps(file_description))
resp = github.json
# print json.dumps(resp, indent=2)
return resp
def remove_github_reference(args, dlid):
dl_delete_url = make_dl_delete_url(args.owner, args.repo, dlid)
github = requests.delete(dl_delete_url, auth=(args.user, args.password))
delete_ok = (204 == github.status_code)
return delete_ok
def post_file_to_s3(file_path, gh):
# s3 is very particular with field ordering
# curl \
# -F "key=downloads/octocat/Hello-World/new_file.jpg" \
# -F "acl=public-read" \
# -F "success_action_status=201" \
# -F "Filename=new_file.jpg" \
# -F "AWSAccessKeyId=1ABCDEF..." \
# -F "Policy=ewogIC..." \
# -F "Signature=mwnF..." \
# -F "Content-Type=image/jpeg" \
# -F "file=#new_file.jpg" \
# https://github.s3.amazonaws.com/
s3_ok = 201
xml_buffer = cStringIO.StringIO()
try:
post_fields = [
('key', str(gh['path'])),
('acl', str(gh['acl'])),
('success_action_status', str(s3_ok)),
('Filename', str(gh['name'])),
('AWSAccessKeyId', str(gh['accesskeyid'])),
('Policy', str(gh['policy'])),
('Signature', str(gh['signature'])),
('Content-Type', str(gh['mime_type'])),
('file', (pycurl.FORM_FILE, file_path))
]
# print post_fields
s3 = pycurl.Curl()
s3.setopt(pycurl.SSL_VERIFYPEER, 0)
s3.setopt(pycurl.SSL_VERIFYHOST, 0)
s3.setopt(pycurl.POST, 1)
s3.setopt(pycurl.URL, str(gh['s3_url']))
s3.setopt(pycurl.HTTPPOST, post_fields)
# s3.setopt(pycurl.VERBOSE, 1)
# accumulate string response
s3.setopt(pycurl.WRITEFUNCTION, xml_buffer.write)
s3.perform()
file_upload_success = (s3_ok == s3.getinfo(pycurl.HTTP_CODE))
xml_payload = minidom.parseString(xml_buffer.getvalue())
if (file_upload_success):
location_element = xml_payload.getElementsByTagName('Location')
print location_element[0].firstChild.nodeValue
else:
print xml_payload.toprettyxml()
except Exception, e:
print e
file_upload_success = False
finally:
s3.close()
return file_upload_success
def main():
mimetypes.init()
args = parse_args()
# step 1: tell github about the file
gh = add_github_reference(args)
# step 2: upload file to s3
if ('errors' in gh):
print json.dumps(gh, indent=2)
else:
file_upload_success = post_file_to_s3(args.filepath, gh)
# cleanup if upload failed
if (False == file_upload_success):
removed_ok = remove_github_reference(args, gh['id'])
if (removed_ok):
print "removed github reference"
else:
print "failed to remove github reference"
if __name__ == '__main__':
main()

Update 2021: You can create a GitHub Actions automation to create a release from a tag, then use the runners to create release assets and upload them to the release. See here for an example.

For those using gradle, the plugin gradle-github-plugin also allows to create releases and attach files to them.
Add the plugin to the gradle.build:
plugins {
id "co.riiid.gradle" version "X.Y.Z"
}
Configure the upload. Example:
github {
owner = 'riiid'
repo = 'gradle-github-plugin'
token = 'XXXXXXXXXXXXXXXXXXXXX'
tagName = '0.1.0'
targetCommitish = 'master'
name = 'v0.1.0'
body = """# Project Name
Write `release note` here.
"""
assets = [
'app/build/outputs/apk/app-release.apk',
'app/build/outputs/mapping/release/mapping.txt',
'app/build/outputs',
...
]
}

Related

yocto-bitbake mirror keep branch

for my yocto project I'm mirroring to my repo to a private one, it's working fine for basic SRC_URI, but for url that have a branch defined it will erase it and bring the default one (master), making it fail since the branch referenced in the original uri was a tag.
DEBUG: Fetcher failure: Unable to find revision xxx_git_commit_hash_xxx in branch master even from upstream
In my .bb file:
SRC_URI = "git://github.com/google/protobuf.git;branch=3.11.x \
and in my local.conf (in the build folder) for the mirroring i have:
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
PREMIRRORS_prepend ?= "\
git://.*/.* git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh\n \
"
(Yes i want to keep HOST and PATH as path to get the repo)
I can see in the logs that my repo is correctly fetch, but the "new" url is without branch and I can't find a way to keep the .bb one
I tried to understand and look at the documentation, but i don't see a way to keep a specific parameter (like the HOST and PATH variable)
So i tried to look at the init.py which define the rules, but i couldn't figure out if it's possible.
I think the right notation would be specifying the branch as well into the premirror URL.
SOURCE_MIRROR_URL = "git://${SOURCE_TOMYMIRROR};protocol=ssh;branch=3.11.x"
PREMIRRORS_prepend = " git://.*/.* ${SOURCE_MIRROR_URL} \
"
Yes it did work,i didn't need premirror_prepend, here's mine :
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
SOURCE_MIRROR_URL="git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh"
INHERIT += "own-mirrors"
And if i want to add new url mirror i can use Mirror
MIRRORS += "\
git://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
https?$://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
"

How can I add in-code annotations in PR reviews usign Github's `gh` tool?

On the Github web UI, I can click on a line and say something like:
Good architecture, but please pass the std::vector<std::uint8_t> hugedata as const &, to avoid a copy.
and bundle such comments as one review with a final verdict.
So far, I've only found gh pr review, which only allows me to generally approve/comment/reject a PR that I'm reviewing.
Is there a way to do detailed in-code reviews using the gh CLI?
if not, how can I use the github api to do that myself?
gh doesn't seem to have built in support for this, but you can still use gh api to call the API:
Note the repository owner, repository name, and pull request ID
Get a diff of the pull request so you can get the right files and positions
gh api \
-H "Accept: application/vnd.github.v3.diff" \
/repos/OWNER/REPO/pulls/ID
Note any files you want to comment on after +++
Note any positions you want to comment on after ## (by number of lines after that line)
Create a pull request review with your comments (using the file as path, the line offset from the start of the hunk as position, and your commend as body)
gh api \
-X POST \
/repos/OWNER/REPO/pulls/ID/reviews \
-d '{ "comments": [{"path": ...,"position": ...,"body": ...}, ...] }'
Submit the pull request review on GitHub (alternatively if you want to automate this, add the body and event properties to your review's body)

How to extract the list of all repositories in Stash or Bitbucket?

I need to extract the list of all repos under all projects in Bitbucket. Is there a REST API for the same? I couldn't find one.
I have both on-premise and cloud Bitbucket.
Clone ALL Projects & Repositories for a given stash url
#!/usr/bin/python
#
# #author Jason LeMonier
#
# Clone ALL Projects & Repositories for a given stash url
#
# Loop through all projects: [P1, P2, ...]
# P1 > for each project make a directory with the key "P1"
# Then clone every repository inside of directory P1
# Backup a directory, create P2, ...
#
# Added ACTION_FLAG bit so the same logic can run fetch --all on every repository and/or clone.
import sys
import os
import stashy
ACTION_FLAG = 1 # Bit: +1=Clone, +2=fetch --all
url = os.environ["STASH_URL"] # "https://mystash.com/stash"
user = os.environ["STASH_USER"] # joedoe"
pwd = os.environ["STASH_PWD"] # Yay123
stash = stashy.connect(url, user, pwd)
def mkdir(xdir):
if not os.path.exists(xdir):
os.makedirs(xdir)
def run_cmd(cmd):
print ("Directory cwd: %s "%(os.getcwd() ))
print ("Running Command: \n %s " %(cmd))
os.system(cmd)
start_dir = os.getcwd()
for project in stash.projects:
pk = project_key = project["key"]
mkdir(pk)
os.chdir(pk)
for repo in stash.projects[project_key].repos.list():
for url in repo["links"]["clone"]:
href = url["href"]
repo_dir = href.split("/")[-1].split(".")[0]
if (url["name"] == "http"):
print (" url.href: %s"% href) # https://joedoe#mystash.com/stash/scm/app/ae.git
print ("Directory cwd: %s Project: %s"%(os.getcwd(), pk))
if ACTION_FLAG & 1 > 0:
if not os.path.exists(repo_dir):
run_cmd("git clone %s" % url["href"])
else:
print ("Directory: %s/%s exists already. Skipping clone. "%(os.getcwd(), repo_dir))
if ACTION_FLAG & 2 > 0:
# chdir into directory "ae" based on url of this repo, fetch, chdir back
cur_dir = os.getcwd()
os.chdir(repo_dir)
run_cmd("git fetch --all ")
os.chdir(cur_dir)
break
os.chdir(start_dir) # avoiding ".." in case of incorrect git directories
Once logged in: on the top right, click on your profile pic and then 'View profile'
Take note of your user (in the example below 'YourEmail#domain.com', but keep in mind it's case sensitive)
Click on profile pic > Manage account > Personal access token > Create a token (choosing 'Read' access type is enough for this functionality)
For all repos in all projects:
Open a CLI and use the command below (remember to fill in your server domain!):
curl -u "YourEmail#domain.com" -X GET https://<my_server_domain>/rest/api/1.0/projects/?limit=1000
It will ask you for your personal access token, you comply and you get a JSON file with all repos requested
For all repos in a given project:
Pick the project you want to get repos from. In my case, the project URL is: <your_server_domain>/projects/TECH/ and therefore my {projectKey} is 'TECH', which you'll need for the command below.
Open a CLI and use this command (remember to fill in your server domain and projectKey!):
curl -u "YourEmail#domain.com" -X GET https://<my_server_domain>/rest/api/1.0/projects/{projectKey}/repos?limit=50
Final touches
(optional) If you want just the titles of the repos requested and you have jq installed (for Windows, downloading the exe and adding it to PATH should be enough, but you need to restart your CLI for that new addition to be detected), you can use the command below:
curl -u $BBUSER -X GET <my_server_domain>/rest/api/1.0/projects/TECH/repos?limit=50 | jq '.values|.[]|.name'
(tested with Data Center/Atlassian Bitbucket v7.9.0 and powershell CLI)
For Bitbucket Cloud
You can use their REST API to access and perform queries on your server.
Specifically, you can use this documentation page, provided by Atlassian, to learn how to list you're repositories.
For Bitbucket Server
Edit: As of receiving this tweet from Dan Bennett, I've learnt there is an API/plugin system for Bitbucket Server that could possibly cater for your needs. For docs: See here.
Edit2: Found this reference to listing personal repositories that may serve as a solution.
AFAIK there isn't a solution for you unless you built a little API for yourself that interacted with your Bitbucket Server instance.
Atlassian Documentation does indicate that to list all currently configured repositories you can do git remote -v. However I'm dubious of this as this isn't normally how git remote -v is used; I think it's more likely that Atlassian's documentation is being unclear rather than Atlassian building in this functionality to Bitbucket Server.
I ended up having to do this myself with an on-prem install of Bitbucket which didn't seem to have the REST APIs discussed above accessible, so I came up with a short script to scrape it out of the web page. This workaround has the advantage that there's nothing you need to install, and you don't need to worry about dependencies, certs or logins other than just logging into your Bitbucket server. You can also set this up as a bookmark if you urlencode the script and prefix it with javascript:.
To use this:
Open your bitbucket server project page, where you should see a list of repos.
Open your browser's devtools console. This is usually F12 or ctrl-shift-i.
Paste the following into the command prompt there.
JSON.stringify(Array.from(document.querySelectorAll('[data-repository-id]')).map(aTag => {
const href = aTag.getAttribute('href');
let projName = href.match(/\/projects\/(.+)\/repos/)[1].toLowerCase();
let repoName = href.match(/\/repos\/(.+)\/browse/)[1];
repoName = repoName.replace(' ', '-');
const templ = `https://${location.host}/scm/${projName}/${repoName}.git`;
return {
href,
name: aTag.innerText,
clone: templ
}
}));
The result is a JSON string containing an array with the repo's URL, name, and clone URL.
[{
"href": "/projects/FOO/repos/some-repo-here/browse",
"name": "some-repo-here",
"clone": "https://mybitbucket.company.com/scm/foo/some-repo-here.git"
}]
This ruby script isn't the greatest code, which makes sense, because I'm not the greatest coder. But it is clear, tested, and it works.
The script filters the output of a Bitbucket API call to create a complete report of all repos on a Bitbucket server. Report is arranged by project, and includes totals and subtotals, a link to each repo, and whether the repos are public or personal. I could have simplified it for general use, but it's pretty useful as it is.
There are no command line arguments. Just run it.
#!/usr/bin/ruby
#
# #author Bill Cernansky
#
# List and count all repos on a Bitbucket server, arranged by project, to STDOUT.
#
require 'json'
bbserver = 'http(s)://server.domain.com'
bbuser = 'username'
bbpassword = 'password'
bbmaxrepos = 2000 # Increase if you have more than 2000 repos
reposRaw = JSON.parse(`curl -s -u '#{bbuser}':'#{bbpassword}' -X GET #{bbserver}/rest/api/1.0/repos?limit=#{bbmaxrepos}`)
projects = {}
repoCount = reposRaw['values'].count
reposRaw['values'].each do |r|
projID = r['project']['key']
if projects[projID].nil?
projects[projID] = {}
projects[projID]['name'] = r['project']['name']
projects[projID]['repos'] = {}
end
repoName = r['name']
projects[projID]['repos'][repoName] = r['links']['clone'][0]['href']
end
privateProjCount = projects.keys.grep(/^\~/).count
publicProjCount = projects.keys.count - privateProjCount
reportText = ''
privateRepoCount = 0
projects.keys.sort.each do |p|
# Personal project slugs always start with tilde
isPrivate = p[0] == '~'
projRepoCount = projects[p]['repos'].keys.count
privateRepoCount += projRepoCount if isPrivate
reportText += "\nProject: #{p} : #{projects[p]['name']}\n #{projRepoCount} #{isPrivate ? 'PERSONAL' : 'Public'} repositories\n"
projects[p]['repos'].keys.each do |r|
reportText += sprintf(" %-30s : %s\n", r, projects[p]['repos'][r])
end
end
puts "BITBUCKET REPO REPORT\n\n"
puts sprintf(" Total Projects: %5d Public: %5d Personal: %5d", projects.keys.count, publicProjCount, privateProjCount)
puts sprintf(" Total Repos: %5d Public: %5d Personal: %5d", repoCount, repoCount - privateRepoCount, privateRepoCount)
puts reportText
The way I solved this issue, was get the html page and give it a ridiculous limit like this. thats in python :
cmd = "curl -s -k --user " + username + " https://URL/projects/<KEY_PROJECT_NAME>/?limit\=10000"
then I parsed it with BeautifulSoup
make_list = str((subprocess.check_output(cmd, shell=True)).rstrip().decode("utf-8"))
html = make_list
parsed_html = BeautifulSoup(html,'html.parser')
list1 = []
for a in parsed_html.find_all("a", href=re.compile("/<projects>/<KEY_PROJECT_NAME>/repos/")):
list1.append(a.string)
print(list1)
to use this make sure you change and , this should be the bitbucket project you are targeting. All , I am doing is parsing an html file.
Here's how I pulled the list of repos from Bitbucket Cloud.
Setup OAauth Consumer
Go to your workspace settings and setup an OAuth consumer, you should be able to go here directly using this link: https://bitbucket.org/{your_workspace}/workspace/settings/api
The only setting that matters is the callback URL which can be anything but I chose http://localhost
Once setup, this will display a key and secret pair for your OAuth consumer, I will refer to these as {oauth_key} and {oauth_secret} below
Authenticate with the API
Go to https://bitbucket.org/site/oauth2/authorize?client_id={oauth_key}&response_type=code ensuring you replace {oauth_key}
This will redirect you to something like http://localhost/?code=xxxxxxxxxxxxxxxxxx, make a note of that code, I'll refer to that as {oauth_code} below
In your terminal go to curl -X POST -u "{oauth_key}:{oauth_secret}" https://bitbucket.org/site/oauth2/access_token -d grant_type=authorization_code -d code={oauth_code} replacing the placeholders.
This should return json including the access_token, I’ll refer to that access token as {oauth_token}
Get the list of repos
You can now run the following to get the list of repos. Bear in mind that your {oauth_token} lasts 2hrs by default.
curl --request GET \
--url 'https://api.bitbucket.org/2.0/repositories/pageant?page=1' \
--header 'Authorization: Bearer {oauth_token}' \
--header 'Accept: application/json'
This response is paginated so you'll need to page through the responses, 10 repositories at a time.

How can I move a Git repository with all branches from Bitbucket to GitHub?

What is the best way to move a Git repository with all branches and full history from Bitbucket to GitHub?
Is there a script or a list of commands I have to use?
It's very simple.
Create a new empty repository in GitHub (without a README or license, you can add them later) and the following screen will show.
In the import code option, paste your Bitbucket repository's URL and voilà!!
You can refer to the GitHub page "Duplicating a repository"
It uses:
git clone --mirror: to clone every references (commits, tags, branches)
git push --mirror: to push everything
That would give:
git clone --mirror https://bitbucket.org/exampleuser/repository-to-mirror.git
# Make a bare mirrored clone of the repository
cd repository-to-mirror.git
git remote set-url --push origin https://github.com/exampleuser/mirrored
# Set the push location to your mirror
git push --mirror
As Noted in the comments by L S:
it is easier to use the Import Code feature from GitHub described by MarMass.
See https://github.com/new/import
Unless... your repo includes a large file: the problem is, the import tool will fail without a clear error message. Only GitHub Support would be able to diagnose what happened.
In case you couldn't find the "Import code" button on GitHub, you can:
Directly open GitHub Importer and enter the URL. It will look like:
Give it a name (or it will import the name automatically)
Select Public or Private repository
Click Begin Import
In May 2016, GitHub announced the ability to "Import repositories with large files".
Se Moving Repository from Bitbucket to GitHub.
This helped me move from one Git provider to another. At the end of it, all the commits were in the destination Git repository. Simple and straightforward.
git remote rename origin bitbucket
git remote add origin https://github.com/edwardaux/Pipelines.git
git push origin master
Once I was happy that the push had been successful to GitHub, I could
delete the old remote by issuing:
git remote rm bitbucket
I had the reverse use case of importing an existing repository from GitHub to Bitbucket.
Bitbucket offers an Import tool as well. The only necessary step is to add URL to repository.
It looks like:
There is the Importing a repository with GitHub Importer
If you have a project hosted on another version control system as Mercurial, you can automatically import it to GitHub using the GitHub Importer tool.
In the upper-right corner of any page, click, and then click Import repository.
Under "Your old repository's clone URL", type the URL of the project you want to import.
Choose your user account or an organization to own the repository, then type a name for the repository on GitHub.
Specify whether the new repository should be public or private.
Public repositories are visible to any user on GitHub, so you can benefit from GitHub's collaborative community.
Public or private repository radio buttonsPrivate repositories are only available to the repository owner, as well as any collaborators you choose to share with.
Review the information you entered, then click Begin import.
You'll receive an email when the repository has been completely imported.
Importing your projects to GitHub
Importing a repository with GitHub Importer
I found this question several months ago when I was trying to do the same thing, and was underwhelmed by the answers given. They all seemed to deal with importing from Bitbucket to GitHub one repository at a time, either via commands issued à la carte, or via the GitHub importer.
I grabulated the code from a GitHub project called gitter and modified it to suite my needs.
You can fork the gist, or take the code from here:
#!/usr/bin/env ruby
require 'fileutils'
# Originally -- Dave Deriso -- deriso#gmail.com
# Contributor -- G. Richard Bellamy -- rbellamy#terradatum.com
# If you contribute, put your name here!
# To get your team ID:
# 1. Go to your GitHub profile, select 'Personal Access Tokens', and create an Access token
# 2. curl -H "Authorization: token <very-long-access-token>" https://api.github.com/orgs/<org-name>/teams
# 3. Find the team name, and grabulate the Team ID
# 4. PROFIT!
#----------------------------------------------------------------------
#your particulars
#access_token = ''
#team_id = ''
#org = ''
#----------------------------------------------------------------------
#the version of this app
#version = "0.2"
#----------------------------------------------------------------------
#some global parameters
#create = false
#add = false
#migrate = false
#debug = false
#done = false
#error = false
#----------------------------------------------------------------------
#fancy schmancy color scheme
class String; def c(cc); "\e[#{cc}m#{self}\e[0m" end end
#200.to_i.times{ |i| print i.to_s.c(i) + " " }; puts
#sep = "-".c(90)*95
#sep_pref = ".".c(90)*95
#sep_thick = "+".c(90)*95
#----------------------------------------------------------------------
# greetings
def hello
puts #sep
puts "BitBucket to GitHub migrator -- v.#{#version}".c(95)
#puts #sep_thick
end
def goodbye
puts #sep
puts "done!".c(95)
puts #sep
exit
end
def puts_title(text)
puts #sep, "#{text}".c(36), #sep
end
#----------------------------------------------------------------------
# helper methods
def get_options
require 'optparse'
n_options = 0
show_options = false
OptionParser.new do |opts|
opts.banner = #sep +"\nUsage: gitter [options]\n".c(36)
opts.version = #version
opts.on('-n', '--name [name]', String, 'Set the name of the new repo') { |value| #repo_name = value; n_options+=1 }
opts.on('-c', '--create', String, 'Create new repo') { #create = true; n_options+=1 }
opts.on('-m', '--migrate', String, 'Migrate the repo') { #migrate = true; n_options+=1 }
opts.on('-a', '--add', String, 'Add repo to team') { #add = true; n_options+=1 }
opts.on('-l', '--language [language]', String, 'Set language of the new repo') { |value| #language = value.strip.downcase; n_options+=1 }
opts.on('-d', '--debug', 'Print commands for inspection, doesn\'t actually run them') { #debug = true; n_options+=1 }
opts.on_tail('-h', '--help', 'Prints this little guide') { show_options = true; n_options+=1 }
#opts = opts
end.parse!
if show_options || n_options == 0
puts #opts
puts "\nExamples:".c(36)
puts 'create new repo: ' + "\t\tgitter -c -l javascript -n node_app".c(93)
puts 'migrate existing to GitHub: ' + "\tgitter -m -n node_app".c(93)
puts 'create repo and migrate to it: ' + "\tgitter -c -m -l javascript -n node_app".c(93)
puts 'create repo, migrate to it, and add it to a team: ' + "\tgitter -c -m -a -l javascript -n node_app".c(93)
puts "\nNotes:".c(36)
puts "Access Token for repo is #{#access_token} - change this on line 13"
puts "Team ID for repo is #{#team_id} - change this on line 14"
puts "Organization for repo is #{#org} - change this on line 15"
puts 'The assumption is that the person running the script has SSH access to Bitbucket,'
puts 'and GitHub, and that if the current directory contains a directory with the same'
puts 'name as the repo to migrated, it will deleted and recreated, or created if it'
puts 'doesn\'t exist - the repo to migrate is mirrored locally, and then created on'
puts 'GitHub and pushed from that local clone.'
puts 'New repos are private by default'
puts "Doesn\'t like symbols for language (ex. use \'c\' instead of \'c++\')"
puts #sep
exit
end
end
#----------------------------------------------------------------------
# git helper methods
def gitter_create(repo)
if #language
%q[curl https://api.github.com/orgs/] + #org + %q[/repos -H "Authorization: token ] + #access_token + %q[" -d '{"name":"] + repo + %q[","private":true,"language":"] + #language + %q["}']
else
%q[curl https://api.github.com/orgs/] + #org + %q[/repos -H "Authorization: token ] + #access_token + %q[" -d '{"name":"] + repo + %q[","private":true}']
end
end
def gitter_add(repo)
if #language
%q[curl https://api.github.com/teams/] + #team_id + %q[/repos/] + #org + %q[/] + repo + %q[ -H "Accept: application/vnd.github.v3+json" -H "Authorization: token ] + #access_token + %q[" -d '{"permission":"pull","language":"] + #language + %q["}']
else
%q[curl https://api.github.com/teams/] + #team_id + %q[/repos/] + #org + %q[/] + repo + %q[ -H "Accept: application/vnd.github.v3+json" -H "Authorization: token ] + #access_token + %q[" -d '{"permission":"pull"}']
end
end
def git_clone_mirror(bitbucket_origin, path)
"git clone --mirror #{bitbucket_origin}"
end
def git_push_mirror(github_origin, path)
"(cd './#{path}' && git push --mirror #{github_origin} && cd ..)"
end
def show_pwd
if #debug
Dir.getwd()
end
end
def git_list_origin(path)
"(cd './#{path}' && git config remote.origin.url && cd ..)"
end
# error checks
def has_repo
File.exist?('.git')
end
def has_repo_or_error(show_error)
#repo_exists = has_repo
if !#repo_exists
puts 'Error: no .git folder in current directory'.c(91) if show_error
#error = true
end
"has repo: #{#repo_exists}"
end
def has_repo_name_or_error(show_error)
#repo_name_exists = !(defined?(#repo_name)).nil?
if !#repo_name_exists
puts 'Error: repo name missing (-n your_name_here)'.c(91) if show_error
#error = true
end
end
#----------------------------------------------------------------------
# main methods
def run(commands)
if #debug
commands.each { |x| puts(x) }
else
commands.each { |x| system(x) }
end
end
def set_globals
puts_title 'Parameters'
#git_bitbucket_origin = "git#bitbucket.org:#{#org}/#{#repo_name}.git"
#git_github_origin = "git#github.com:#{#org}/#{#repo_name}.git"
puts 'debug: ' + #debug.to_s.c(93)
puts 'working in: ' + Dir.pwd.c(93)
puts 'create: ' + #create.to_s.c(93)
puts 'migrate: ' + #migrate.to_s.c(93)
puts 'add: ' + #add.to_s.c(93)
puts 'language: ' + #language.to_s.c(93)
puts 'repo name: '+ #repo_name.to_s.c(93)
puts 'bitbucket: ' + #git_bitbucket_origin.to_s.c(93)
puts 'github: ' + #git_github_origin.to_s.c(93)
puts 'team_id: ' + #team_id.to_s.c(93)
puts 'org: ' + #org.to_s.c(93)
end
def create_repo
puts_title 'Creating'
#error checks
has_repo_name_or_error(true)
goodbye if #error
puts #sep
commands = [
gitter_create(#repo_name)
]
run commands
end
def add_repo
puts_title 'Adding repo to team'
#error checks
has_repo_name_or_error(true)
goodbye if #error
puts #sep
commands = [
gitter_add(#repo_name)
]
run commands
end
def migrate_repo
puts_title "Migrating Repo to #{#repo_provider}"
#error checks
has_repo_name_or_error(true)
goodbye if #error
if Dir.exists?("#{#repo_name}.git")
puts "#{#repo_name} already exists... recursively deleting."
FileUtils.rm_r("#{#repo_name}.git")
end
path = "#{#repo_name}.git"
commands = [
git_clone_mirror(#git_bitbucket_origin, path),
git_list_origin(path),
git_push_mirror(#git_github_origin, path)
]
run commands
end
#----------------------------------------------------------------------
#sequence control
hello
get_options
#do stuff
set_globals
create_repo if #create
migrate_repo if #migrate
add_repo if #add
#peace out
goodbye
Then, to use the script:
# create a list of repos
foo
bar
baz
# execute the script, iterating over your list
while read p; do ./bitbucket-to-github.rb -a -n $p; done<repos
# good enough
In case you want to move your local Git repository to another upstream, you can also do this:
To get the current remote URL:
git remote get-url origin
will show something like:
https://bitbucket.com/git/myrepo
To set new remote repository:
git remote set-url origin git#github.com:folder/myrepo.git
now push contents of current (develop) branch:
git push --set-upstream origin develop
You now have a full copy of the branch in the new remote.
Optionally, return to original git-remote for this local folder:
git remote set-url origin https://bitbucket.com/git/myrepo
It gives the benefit you can now get your new Git repository from GitHub in another folder so that you have two local folders both pointing to the different remotes, the previous (Bitbucket) and the new one both available.
I made the following Bash script in order to clone all of my Bitbucket (user) repositories to GitHub as private repositories.
Requirements:
jq (command-line JSON processor) | macOS: brew install jq
Steps:
Go to Personal access tokens and create an access token. We only need the "repo" scope.
Save the move_me.sh script in a working folder and edit the file as needed.
Don't forget to chmod 755
Run! ./move_me.sh
Enjoy the time you have saved.
Notes:
It will clone the Bitbucket repositories inside the directory the script resides (your working directory).
This script does not delete your Bitbucket repositories.
Need to move to public repositories on GitHub?
Find and change the "private": true to "private": false below.
Moving an organization's repositories?
Check out the developer guide. It's a couple of edits away.
Happy moving.
#!/bin/bash
BB_USERNAME=your_bitbucket_username
BB_PASSWORD=your_bitbucket_password
GH_USERNAME=your_github_username
GH_ACCESS_TOKEN=your_github_access_token
###########################
pagelen=$(curl -s -u $BB_USERNAME:$BB_PASSWORD https://api.bitbucket.org/2.0/repositories/$BB_USERNAME | jq -r '.pagelen')
echo "Total number of pages: $pagelen"
hr () {
printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' -
}
i=1
while [ $i -le $pagelen ]
do
echo
echo "* Processing Page: $i..."
hr
pageval=$(curl -s -u $BB_USERNAME:$BB_PASSWORD https://api.bitbucket.org/2.0/repositories/$BB_USERNAME?page=$i)
next=$(echo $pageval | jq -r '.next')
slugs=($(echo $pageval | jq -r '.values[] | .slug'))
repos=($(echo $pageval | jq -r '.values[] | .links.clone[1].href'))
j=0
for repo in ${repos[#]}
do
echo "$(($j + 1)) = ${repos[$j]}"
slug=${slugs[$j]}
git clone --bare $repo
cd "$slug.git"
echo
echo "* $repo cloned, now creating $slug on GitHub..."
echo
read -r -d '' PAYLOAD <<EOP
{
"name": "$slug",
"description": "$slug - moved from Bitbucket",
"homepage": "https://github.com/$slug",
"private": true
}
EOP
curl -H "Authorization: token $GH_ACCESS_TOKEN" --data "$PAYLOAD" \
https://api.github.com/user/repos
echo
echo "* mirroring $repo to GitHub..."
echo
git push --mirror "git#github.com:$GH_USERNAME/$slug.git"
j=$(( $j + 1 ))
hr
cd ..
done
i=$(( $i + 1 ))
done
Here are the steps to move a private Git repository:
Step 1: Create a GitHub repository
First, create a new private repository on GitHub. It’s important to keep the repository empty, e.g., don’t check option Initialize this repository with a README when creating the repository.
Step 2: Move existing content
Next, we need to fill the GitHub repository with the content from our Bitbucket repository:
Check out the existing repository from Bitbucket:
git clone https://USER#bitbucket.org/USER/PROJECT.git
Add the new GitHub repository as upstream remote of the repository checked out from Bitbucket:
cd PROJECT
git remote add upstream https://github.com:USER/PROJECT.git
Push all branches (below: just master) and tags to the GitHub
repository:
git push upstream master
git push --tags upstream
Step 3: Clean up the old repository
Finally, we need to ensure that developers don’t get confused by having two repositories for the same project. Here is how to delete the Bitbucket repository:
Double-check that the GitHub repository has all content
Go to the web interface of the old Bitbucket repository
Select menu option Setting → Delete repository
Add the URL of the new GitHub repository as the redirect URL
With that, the repository completely settled into its new home at GitHub. Let all the developers know!
Following on from #MarMass' answer, if the GitHub importer is constantly redirecting you to the authentication screen, you'll need to create an App Password in BitBucket in order to import your private repository:
Go to Bitbucket > Personal Settings > App Passwords.
Create an app password with repository read access.
When prompted for your username/password in the GitHub importer, enter your BitBucket username, and the token created above as your password.
After managing to sort out the authentication issue, my imports also errored out with the following message: "There was an error pushing commits to GitHub.".
The issue here, at least for me, was that my GitHub account was set to "Block command line pushes that expose my email", and the repository I was attempting to import from Bitbucket contained commits from my personal email address. After temporarily disabling this setting (GitHub > Settings > Email) I was good to go.
The simplest way of doing it:
git remote rename origin repo_bitbucket
git remote add origin https://github.com/abc/repo.git
git push origin master
Once the push to GitHub is successful, delete the old remote by running:
git remote rm repo_bitbucket

Download latest GitHub release

I'd like to have "Download Latest Version" button on my website which would represent the link to the latest release (stored at GitHub Releases). I tried to create release tag named "latest", but it became complicated when I tried to load new release (confusion with tag creation date, tag interchanging, etc.). Updating download links on my website manually is also a time-consuming and scrupulous task. I see the only way - redirect all download buttons to some html, which in turn will redirect to the actual latest release.
Note that my website is hosted at GitHub Pages (static hosting), so I simply can't use server-side scripting to generate links. Any ideas?
You don't need any scripting to generate a download link for the latest release. Simply use this format:
https://github.com/:owner/:repo/zipball/:branch
Examples:
https://github.com/webix-hub/tracker/zipball/master
https://github.com/iDoRecall/selection-menu/zipball/gh-pages
If for some reason you want to obtain a link to the latest release download, including its version number, you can obtain that from the get latest release API:
GET /repos/:owner/:repo/releases/latest
Example:
$.get('https://api.github.com/repos/idorecall/selection-menu/releases/latest', function (data) {
$('#result').attr('href', data.zipball_url);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<a id="result">Download latest release (.ZIP)</a>
Github now provides a "Latest release" button on the release page of a project, after you have created your first release.
In the example you gave, this button links to https://github.com/reactiveui/ReactiveUI/releases/latest
You can use the following where:
${Organization} as the GitHub user or organization
${Repository} is the repository name
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball > ${Repository}.tar.gz
The top level directory in the .tar.gz file has the sha hash of the commit in the directory name which can be a problem if you need an automated way to change into the resulting directory and do something.
The method below will strip this out, and leave the files in a folder with a predictable name.
mkdir ${Repository}
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball | tar -zxv -C ${Repository} --strip-components=1
Since February 18th, 2015, the GitHUb V3 release API has a get latest release API.
GET /repos/:owner/:repo/releases/latest
See also "Linking to releases".
Still, the name of the asset can be tricky.
Git-for-Windows, for instance, requires a command like:
curl -IkLs -o NUL -w %{url_effective} \
https://github.com/git-for-windows/git/releases/latest|\
grep -o "[^/]*$"| sed "s/v//g"|\
xargs -I T echo \
https://github.com/git-for-windows/git/releases/download/vT/PortableGit-T-64-bit.7z.exe \
-o PortableGit-T-64-bit.7z.exe| \
sed "s/.windows.1-64/-64/g"|sed "s/.windows.\(.\)-64/.\1-64/g"|\
xargs curl -kL
The first 3 lines extract the latest version 2.35.1.windows.2
The rest will build the right URL
https://github.com/git-for-windows/git/releases/download/
v2.35.1.windows.2/PortableGit-2.35.1.2-64-bit.7z.exe
^^^^^^^^^^^^^^^^^ ^^^^^^^^^
Maybe could you use some client-side scripting and dynamically generate the target of the link by invoking the GitHub api, through some JQuery magic?
The Releases API exposes a way to retrieve the list of all the releases from a repository. For instance, this link return a Json formatted list of all the releases of the ReactiveUI project.
Extracting the first one would return the latest release.
Within this payload:
The html_url attribute will hold the first part of the url to build (ie. https://github.com/{owner}/{repository}/releases/{version}).
The assets array will list of the downloadable archives. Each asset will bear a name attribute
Building the target download url is only a few string operations away.
Insert the download/ keyword between the releases/ segment from the html_url and the version number
Append the name of the asset to download
Resulting url will be of the following format: https://github.com/{owner}/{repository}/releases/download/{version}/name_of_asset
For instance, regarding the Json payload from the link ReactiveUI link above, we've got html_url: "https://github.com/reactiveui/ReactiveUI/releases/5.99.0" and one asset with name: "ReactiveUI.6.0.Preview.1.zip".
As such, the download url is https://github.com/reactiveui/ReactiveUI/releases/download/5.99.0/ReactiveUI.6.0.Preview.1.zip
If you using PHP try follow code:
function getLatestTagUrl($repository, $default = 'master') {
$file = #json_decode(#file_get_contents("https://api.github.com/repos/$repository/tags", false,
stream_context_create(['http' => ['header' => "User-Agent: Vestibulum\r\n"]])
));
return sprintf("https://github.com/$repository/archive/%s.zip", $file ? reset($file)->name : $default);
}
Function usage example
echo 'Download';
As I didn't see the answer here, but it was quite helpful for me while running continuous integration tests, this one-liner that only requires you to have curl will allow to search the Github repo's releases to download the latest version
https://gist.github.com/steinwaywhw/a4cd19cda655b8249d908261a62687f8
I use it to run PHPSTan on our repository using the following script
https://gist.github.com/rvanlaak/7491f2c4f0c456a93f90e31774300b62
If you are trying to download form any linux — even old or tiny versions — or are trying to download from a bash script then the failproof way is using this command:
wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}' | xargs wget
do not forget to replace $OWNER and $REPO with the right owner and repository names. The command downloads a json page with the data of the latest release. then awk gets the value from the browser_download_url key.
If you are in a really old linux or a tiny embedded system with a small wget, the download name can be a problem. In such case you can always use the ultra-reliable:
URL=$(wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}'); wget $URL -O $(basename "$URL")
As noted by #Dan Dascalescu in a comment to accepted answer, there are some projects (roughly 30%) which do not bother to file formal releases, so neither "Latest release" button nor /releases/latest API call would return useful data.
To reliably fetch the latest release for a GitHub project, you can use lastversion.