Sometimes, We meet a situation that remote source code fetched by a recipe need to be modified so that suit a specific machine.
How do we create a patch for remote source code locally? After that everytime we build the recipe (even clean it all) we can patch the remote source code automatically.
For example, I have a special machine with architecture A which is not common, so the remote source code need to be modified so that support architecture A.
Suppose there was a file called utils.h (which is code that we fetched by example.bb from remote git repository)
#if defined(__x86_64__) || \
defined(__mips__) || \
defined(__powerpc__) || defined(__ppc__) || defined(__ppc64__) \
#define SOME_FUNCTIONALITY 1
Apparently I need to add archtecture A support in the file.
#if defined(__x86_64__) || \
defined(__mips__) || \
defined(__powerpc__) || defined(__ppc__) || defined(__ppc64__) || \
defined(__A__) \
#define SOME_FUNCTIONALITY 1
But if we just modified like that, next time we execute
bitbake -c cleanall example
bitbake example
then we get a unchanged copies again(which means we have to modify it again).
How do we create a Add-architecture-A-support.patch locally so that we can patch the remote source code automatically?
This is a simple one from answers.
(Note: If there was no git in the source code directory, before modifying the source code, you need to create a git repository and commit all in the top directory of the source code.)
git init # create a git repository
git add .
git commit -m "First commit" # first commit
After change the utils.h as above, we can check the git status. It usually looks like that.
$ git status
HEAD detached from 87b933c420
Changes not staged for commit:
(use "git add <file>..." to update what will be comitted)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: ../../utils.h
...
no changes added to commit (use "git add" and/or "git commit -a")
Then we add and commit the change locally (usually we don't have the permission to push to upper stream).
$ git add utils.h
$ git commit -m "Patch test"
After that we can use git to create a patch for the recent commit.
$ git show >Add-architecture-A-support.patch
It will creat a patch in the current directory with contents looks like that
commit a79e523...
Author: 杨...
Date: ...
Patch test
diff --git a/somedir/utils.h b/somedir/utils.h
index 20bfd36c84..
--- a/somedir/utils.h
+++ b/somedir/utils.h
...
+ defined(__A__) \
...
Then we can move the patch to the local layer where the recipe stayed.
recipe-example
|-- example
| |-- Add-architecture-A-support.patch
|-- example.bb
And add the patch in example.bb with this.
SRC_URI += "\
file://Add-architecture-A-support.patch \
"
Work finished. (Also, if want to undo the local commit after creating the patch, you can use git reset HEAD^ utils.h. emmm, I think so, maybe there are some faults, just google it)
I am working on adding my own module to the build of buildroot using $BR2_External. The make file of my package is as follows,
##############################################################
#
# GPIO
#
##############################################################
GPIO_VERSION = '2851a05c9b613c1736f79faa185a11118b229852'
GPIO_SITE = '<URL of git repo>'
GPIO_SITE_METHOD = git
GPIO_GIT_SUBMODULES = YES
GPIO_MODULE_SUBDIRS = GPIO_driver/
# GPIO_MODULE_SUBDIRS += GPIO_driver/
# define LDD_BUILD_CMDS
# $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(#D)/misc-modules
# $(MAKE) $(TARGET_CONFIGURE_OPTS) -C $(#D)/scull
# endef
#
# # TODO add your writer, finder and finder-test utilities/scripts to the installation steps below
define GPIO_INSTALL_TARGET_CMDS
#module
# $(INSTALL) -m 0755 $(#D)/01_simple_LKM/* $(TARGET_DIR)/usr/bin
$(INSTALL) -m 0755 $(#D)/GPIO_driver/* $(TARGET_DIR)/usr/bin
endef
$(eval $(kernel-module))
$(eval $(generic-package))
This make file always pulls only a specific commit (mentioned in GPIO_VERSION variable) from gitHub. This is getting a little frustrating as, everytime I push new code to git I have to update the make file with the new commit number as well. So, is there any way to write the make file such that the most recent commit is pulled.
I have these three commands in my package.json
{
"build-major": "npm version major --no-git-tag-version && node ./replace.build.js && node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng build --prod --base-href /QMobile/",
"build-minor": "npm version minor --no-git-tag-version && node ./replace.build.js && node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng build --prod --base-href /QMobile/",
"build-patch": "npm version patch --no-git-tag-version && node ./replace.build.js && node --max_old_space_size=8192 node_modules/#angular/cli/bin/ng build --prod --base-href /QMobile/",
}
My replace.build.js Contains following Code:
var replace = require('replace-in-file');
var package = require("./package.json");
var buildVersion = package.version;
const options = {
files: 'src/environments/environment.prod.ts',
from: /version: '(.*)'/g,
to: "version: '"+ buildVersion + "'",
allowEmptyPaths: false,
};
const htmlOptions={
files:'src/index.html',
from: /<meta name="version" content="(.*)">/g,
to: '<meta name="version" content="'+buildVersion+'">',
allowEmptyPaths: false
}
try {
let changedFiles = replace.sync(options);
if (changedFiles == 0) {
throw "Please make sure that file '" + options.files + "' has \"version: ''\"";
}
let indexFileChanged = replace.sync(htmlOptions);
if (indexFileChanged == 0) {
throw "Index.html Version Change Failed";
}
console.log('Build version set: ' + buildVersion);
}
catch (error) {
console.error('Error occurred:', error);
throw error
}
So every time i take my Angular Build, i get 3 files changes?
index.html, environment.prod.ts and package.json
I am looking at a solution to make these changes get pushed to the current branch too while build is in Process or a Standard way to maintain the Build Number in sync with all Branches.
I am looking at a solution to make these changes get pushed to the
current branch too while build is in Process or a Standard way to
maintain the Build Number in sync with all Branches.
Just put a git-related task at the end of your pipeline to push the changed files to remote branch. You can use CMD task, Powershell task or third-party git tasks here to push the changed files.
For my pipeline that calls Replace token task to modify one file:
1.The cmd task at the start of the pipeline:
git checkout $(Build.SourceBranchName)
git config --global user.email "xxx#xxx.com"
git config --global user.name "xxx"
For build source directory, git repo is in HEAD detached by default, so we need to switch to the build branch by git checkout first.
2.The cmd task at the end of the pipeline:
git add .
git commit -m "Update files in Build $(Build.BuildNumber)."
git push https://{MyPat}#dev.azure.com/{OrganizationName}/{ProjectName}/_git/{RepoName} $(Build.SourceBranchName)
Check this similar issue for more details about git push command. About PAT see here. You can also define the PAT as an secret variable and reference it in command.
You can put your real tasks between the first CMD and the last CMD, then every time when the pipeline is finished, the changes will be pushed to current branch. (With BuildNumber in commit message.)
I am developing a library on github that has travis checks attached to it. I'd like to have a WIP pull request open to discuss ideas easily. There is a lot of tests set up for the project on travis, so I'd like to not trigger the tests every time I push a commit (to prevent server for being overloaded), as the code is not expected to pass anyway.
Is there a way I could do this on github without having access to travis configuration?
To skip builds on a per-commit basis you can add [ci skip] to the commit message, as described in the Docs, for example:
Before: Add blerb.
After:
Add blerb
[ci skip]
To skip all non-PR builds, you can early-exit if the TRAVIS_PULL_REQUEST environnment variable is set to "false" fron your .travis.yml:
before_install: # Earliest build step
- if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then echo "Not a PR, skipping" && exit; fi
My approach is to use an alias that I defined that appends the [ci skip] to each commit message of the current branch.
wip = "!f() { git filter-branch -f --msg-filter 'sed -e "\"s/$/ \\[ci skip\\]/g\""' ${1-master}..HEAD ; }; f"
unwip = "!f() { git filter-branch -f --msg-filter 'sed -e "\"s/ \\[ci skip\\]//g\""' ${1-master}..HEAD ; }; f"
wpush = "!f() { git wip $1 && git fpush && git unwip $1 ; }; f"
So basically what I do is simply git wpush.
What is the best way to move a Git repository with all branches and full history from Bitbucket to GitHub?
Is there a script or a list of commands I have to use?
It's very simple.
Create a new empty repository in GitHub (without a README or license, you can add them later) and the following screen will show.
In the import code option, paste your Bitbucket repository's URL and voilà!!
You can refer to the GitHub page "Duplicating a repository"
It uses:
git clone --mirror: to clone every references (commits, tags, branches)
git push --mirror: to push everything
That would give:
git clone --mirror https://bitbucket.org/exampleuser/repository-to-mirror.git
# Make a bare mirrored clone of the repository
cd repository-to-mirror.git
git remote set-url --push origin https://github.com/exampleuser/mirrored
# Set the push location to your mirror
git push --mirror
As Noted in the comments by L S:
it is easier to use the Import Code feature from GitHub described by MarMass.
See https://github.com/new/import
Unless... your repo includes a large file: the problem is, the import tool will fail without a clear error message. Only GitHub Support would be able to diagnose what happened.
In case you couldn't find the "Import code" button on GitHub, you can:
Directly open GitHub Importer and enter the URL. It will look like:
Give it a name (or it will import the name automatically)
Select Public or Private repository
Click Begin Import
In May 2016, GitHub announced the ability to "Import repositories with large files".
Se Moving Repository from Bitbucket to GitHub.
This helped me move from one Git provider to another. At the end of it, all the commits were in the destination Git repository. Simple and straightforward.
git remote rename origin bitbucket
git remote add origin https://github.com/edwardaux/Pipelines.git
git push origin master
Once I was happy that the push had been successful to GitHub, I could
delete the old remote by issuing:
git remote rm bitbucket
I had the reverse use case of importing an existing repository from GitHub to Bitbucket.
Bitbucket offers an Import tool as well. The only necessary step is to add URL to repository.
It looks like:
There is the Importing a repository with GitHub Importer
If you have a project hosted on another version control system as Mercurial, you can automatically import it to GitHub using the GitHub Importer tool.
In the upper-right corner of any page, click, and then click Import repository.
Under "Your old repository's clone URL", type the URL of the project you want to import.
Choose your user account or an organization to own the repository, then type a name for the repository on GitHub.
Specify whether the new repository should be public or private.
Public repositories are visible to any user on GitHub, so you can benefit from GitHub's collaborative community.
Public or private repository radio buttonsPrivate repositories are only available to the repository owner, as well as any collaborators you choose to share with.
Review the information you entered, then click Begin import.
You'll receive an email when the repository has been completely imported.
Importing your projects to GitHub
Importing a repository with GitHub Importer
I found this question several months ago when I was trying to do the same thing, and was underwhelmed by the answers given. They all seemed to deal with importing from Bitbucket to GitHub one repository at a time, either via commands issued à la carte, or via the GitHub importer.
I grabulated the code from a GitHub project called gitter and modified it to suite my needs.
You can fork the gist, or take the code from here:
#!/usr/bin/env ruby
require 'fileutils'
# Originally -- Dave Deriso -- deriso#gmail.com
# Contributor -- G. Richard Bellamy -- rbellamy#terradatum.com
# If you contribute, put your name here!
# To get your team ID:
# 1. Go to your GitHub profile, select 'Personal Access Tokens', and create an Access token
# 2. curl -H "Authorization: token <very-long-access-token>" https://api.github.com/orgs/<org-name>/teams
# 3. Find the team name, and grabulate the Team ID
# 4. PROFIT!
#----------------------------------------------------------------------
#your particulars
#access_token = ''
#team_id = ''
#org = ''
#----------------------------------------------------------------------
#the version of this app
#version = "0.2"
#----------------------------------------------------------------------
#some global parameters
#create = false
#add = false
#migrate = false
#debug = false
#done = false
#error = false
#----------------------------------------------------------------------
#fancy schmancy color scheme
class String; def c(cc); "\e[#{cc}m#{self}\e[0m" end end
#200.to_i.times{ |i| print i.to_s.c(i) + " " }; puts
#sep = "-".c(90)*95
#sep_pref = ".".c(90)*95
#sep_thick = "+".c(90)*95
#----------------------------------------------------------------------
# greetings
def hello
puts #sep
puts "BitBucket to GitHub migrator -- v.#{#version}".c(95)
#puts #sep_thick
end
def goodbye
puts #sep
puts "done!".c(95)
puts #sep
exit
end
def puts_title(text)
puts #sep, "#{text}".c(36), #sep
end
#----------------------------------------------------------------------
# helper methods
def get_options
require 'optparse'
n_options = 0
show_options = false
OptionParser.new do |opts|
opts.banner = #sep +"\nUsage: gitter [options]\n".c(36)
opts.version = #version
opts.on('-n', '--name [name]', String, 'Set the name of the new repo') { |value| #repo_name = value; n_options+=1 }
opts.on('-c', '--create', String, 'Create new repo') { #create = true; n_options+=1 }
opts.on('-m', '--migrate', String, 'Migrate the repo') { #migrate = true; n_options+=1 }
opts.on('-a', '--add', String, 'Add repo to team') { #add = true; n_options+=1 }
opts.on('-l', '--language [language]', String, 'Set language of the new repo') { |value| #language = value.strip.downcase; n_options+=1 }
opts.on('-d', '--debug', 'Print commands for inspection, doesn\'t actually run them') { #debug = true; n_options+=1 }
opts.on_tail('-h', '--help', 'Prints this little guide') { show_options = true; n_options+=1 }
#opts = opts
end.parse!
if show_options || n_options == 0
puts #opts
puts "\nExamples:".c(36)
puts 'create new repo: ' + "\t\tgitter -c -l javascript -n node_app".c(93)
puts 'migrate existing to GitHub: ' + "\tgitter -m -n node_app".c(93)
puts 'create repo and migrate to it: ' + "\tgitter -c -m -l javascript -n node_app".c(93)
puts 'create repo, migrate to it, and add it to a team: ' + "\tgitter -c -m -a -l javascript -n node_app".c(93)
puts "\nNotes:".c(36)
puts "Access Token for repo is #{#access_token} - change this on line 13"
puts "Team ID for repo is #{#team_id} - change this on line 14"
puts "Organization for repo is #{#org} - change this on line 15"
puts 'The assumption is that the person running the script has SSH access to Bitbucket,'
puts 'and GitHub, and that if the current directory contains a directory with the same'
puts 'name as the repo to migrated, it will deleted and recreated, or created if it'
puts 'doesn\'t exist - the repo to migrate is mirrored locally, and then created on'
puts 'GitHub and pushed from that local clone.'
puts 'New repos are private by default'
puts "Doesn\'t like symbols for language (ex. use \'c\' instead of \'c++\')"
puts #sep
exit
end
end
#----------------------------------------------------------------------
# git helper methods
def gitter_create(repo)
if #language
%q[curl https://api.github.com/orgs/] + #org + %q[/repos -H "Authorization: token ] + #access_token + %q[" -d '{"name":"] + repo + %q[","private":true,"language":"] + #language + %q["}']
else
%q[curl https://api.github.com/orgs/] + #org + %q[/repos -H "Authorization: token ] + #access_token + %q[" -d '{"name":"] + repo + %q[","private":true}']
end
end
def gitter_add(repo)
if #language
%q[curl https://api.github.com/teams/] + #team_id + %q[/repos/] + #org + %q[/] + repo + %q[ -H "Accept: application/vnd.github.v3+json" -H "Authorization: token ] + #access_token + %q[" -d '{"permission":"pull","language":"] + #language + %q["}']
else
%q[curl https://api.github.com/teams/] + #team_id + %q[/repos/] + #org + %q[/] + repo + %q[ -H "Accept: application/vnd.github.v3+json" -H "Authorization: token ] + #access_token + %q[" -d '{"permission":"pull"}']
end
end
def git_clone_mirror(bitbucket_origin, path)
"git clone --mirror #{bitbucket_origin}"
end
def git_push_mirror(github_origin, path)
"(cd './#{path}' && git push --mirror #{github_origin} && cd ..)"
end
def show_pwd
if #debug
Dir.getwd()
end
end
def git_list_origin(path)
"(cd './#{path}' && git config remote.origin.url && cd ..)"
end
# error checks
def has_repo
File.exist?('.git')
end
def has_repo_or_error(show_error)
#repo_exists = has_repo
if !#repo_exists
puts 'Error: no .git folder in current directory'.c(91) if show_error
#error = true
end
"has repo: #{#repo_exists}"
end
def has_repo_name_or_error(show_error)
#repo_name_exists = !(defined?(#repo_name)).nil?
if !#repo_name_exists
puts 'Error: repo name missing (-n your_name_here)'.c(91) if show_error
#error = true
end
end
#----------------------------------------------------------------------
# main methods
def run(commands)
if #debug
commands.each { |x| puts(x) }
else
commands.each { |x| system(x) }
end
end
def set_globals
puts_title 'Parameters'
#git_bitbucket_origin = "git#bitbucket.org:#{#org}/#{#repo_name}.git"
#git_github_origin = "git#github.com:#{#org}/#{#repo_name}.git"
puts 'debug: ' + #debug.to_s.c(93)
puts 'working in: ' + Dir.pwd.c(93)
puts 'create: ' + #create.to_s.c(93)
puts 'migrate: ' + #migrate.to_s.c(93)
puts 'add: ' + #add.to_s.c(93)
puts 'language: ' + #language.to_s.c(93)
puts 'repo name: '+ #repo_name.to_s.c(93)
puts 'bitbucket: ' + #git_bitbucket_origin.to_s.c(93)
puts 'github: ' + #git_github_origin.to_s.c(93)
puts 'team_id: ' + #team_id.to_s.c(93)
puts 'org: ' + #org.to_s.c(93)
end
def create_repo
puts_title 'Creating'
#error checks
has_repo_name_or_error(true)
goodbye if #error
puts #sep
commands = [
gitter_create(#repo_name)
]
run commands
end
def add_repo
puts_title 'Adding repo to team'
#error checks
has_repo_name_or_error(true)
goodbye if #error
puts #sep
commands = [
gitter_add(#repo_name)
]
run commands
end
def migrate_repo
puts_title "Migrating Repo to #{#repo_provider}"
#error checks
has_repo_name_or_error(true)
goodbye if #error
if Dir.exists?("#{#repo_name}.git")
puts "#{#repo_name} already exists... recursively deleting."
FileUtils.rm_r("#{#repo_name}.git")
end
path = "#{#repo_name}.git"
commands = [
git_clone_mirror(#git_bitbucket_origin, path),
git_list_origin(path),
git_push_mirror(#git_github_origin, path)
]
run commands
end
#----------------------------------------------------------------------
#sequence control
hello
get_options
#do stuff
set_globals
create_repo if #create
migrate_repo if #migrate
add_repo if #add
#peace out
goodbye
Then, to use the script:
# create a list of repos
foo
bar
baz
# execute the script, iterating over your list
while read p; do ./bitbucket-to-github.rb -a -n $p; done<repos
# good enough
In case you want to move your local Git repository to another upstream, you can also do this:
To get the current remote URL:
git remote get-url origin
will show something like:
https://bitbucket.com/git/myrepo
To set new remote repository:
git remote set-url origin git#github.com:folder/myrepo.git
now push contents of current (develop) branch:
git push --set-upstream origin develop
You now have a full copy of the branch in the new remote.
Optionally, return to original git-remote for this local folder:
git remote set-url origin https://bitbucket.com/git/myrepo
It gives the benefit you can now get your new Git repository from GitHub in another folder so that you have two local folders both pointing to the different remotes, the previous (Bitbucket) and the new one both available.
I made the following Bash script in order to clone all of my Bitbucket (user) repositories to GitHub as private repositories.
Requirements:
jq (command-line JSON processor) | macOS: brew install jq
Steps:
Go to Personal access tokens and create an access token. We only need the "repo" scope.
Save the move_me.sh script in a working folder and edit the file as needed.
Don't forget to chmod 755
Run! ./move_me.sh
Enjoy the time you have saved.
Notes:
It will clone the Bitbucket repositories inside the directory the script resides (your working directory).
This script does not delete your Bitbucket repositories.
Need to move to public repositories on GitHub?
Find and change the "private": true to "private": false below.
Moving an organization's repositories?
Check out the developer guide. It's a couple of edits away.
Happy moving.
#!/bin/bash
BB_USERNAME=your_bitbucket_username
BB_PASSWORD=your_bitbucket_password
GH_USERNAME=your_github_username
GH_ACCESS_TOKEN=your_github_access_token
###########################
pagelen=$(curl -s -u $BB_USERNAME:$BB_PASSWORD https://api.bitbucket.org/2.0/repositories/$BB_USERNAME | jq -r '.pagelen')
echo "Total number of pages: $pagelen"
hr () {
printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' -
}
i=1
while [ $i -le $pagelen ]
do
echo
echo "* Processing Page: $i..."
hr
pageval=$(curl -s -u $BB_USERNAME:$BB_PASSWORD https://api.bitbucket.org/2.0/repositories/$BB_USERNAME?page=$i)
next=$(echo $pageval | jq -r '.next')
slugs=($(echo $pageval | jq -r '.values[] | .slug'))
repos=($(echo $pageval | jq -r '.values[] | .links.clone[1].href'))
j=0
for repo in ${repos[#]}
do
echo "$(($j + 1)) = ${repos[$j]}"
slug=${slugs[$j]}
git clone --bare $repo
cd "$slug.git"
echo
echo "* $repo cloned, now creating $slug on GitHub..."
echo
read -r -d '' PAYLOAD <<EOP
{
"name": "$slug",
"description": "$slug - moved from Bitbucket",
"homepage": "https://github.com/$slug",
"private": true
}
EOP
curl -H "Authorization: token $GH_ACCESS_TOKEN" --data "$PAYLOAD" \
https://api.github.com/user/repos
echo
echo "* mirroring $repo to GitHub..."
echo
git push --mirror "git#github.com:$GH_USERNAME/$slug.git"
j=$(( $j + 1 ))
hr
cd ..
done
i=$(( $i + 1 ))
done
Here are the steps to move a private Git repository:
Step 1: Create a GitHub repository
First, create a new private repository on GitHub. It’s important to keep the repository empty, e.g., don’t check option Initialize this repository with a README when creating the repository.
Step 2: Move existing content
Next, we need to fill the GitHub repository with the content from our Bitbucket repository:
Check out the existing repository from Bitbucket:
git clone https://USER#bitbucket.org/USER/PROJECT.git
Add the new GitHub repository as upstream remote of the repository checked out from Bitbucket:
cd PROJECT
git remote add upstream https://github.com:USER/PROJECT.git
Push all branches (below: just master) and tags to the GitHub
repository:
git push upstream master
git push --tags upstream
Step 3: Clean up the old repository
Finally, we need to ensure that developers don’t get confused by having two repositories for the same project. Here is how to delete the Bitbucket repository:
Double-check that the GitHub repository has all content
Go to the web interface of the old Bitbucket repository
Select menu option Setting → Delete repository
Add the URL of the new GitHub repository as the redirect URL
With that, the repository completely settled into its new home at GitHub. Let all the developers know!
Following on from #MarMass' answer, if the GitHub importer is constantly redirecting you to the authentication screen, you'll need to create an App Password in BitBucket in order to import your private repository:
Go to Bitbucket > Personal Settings > App Passwords.
Create an app password with repository read access.
When prompted for your username/password in the GitHub importer, enter your BitBucket username, and the token created above as your password.
After managing to sort out the authentication issue, my imports also errored out with the following message: "There was an error pushing commits to GitHub.".
The issue here, at least for me, was that my GitHub account was set to "Block command line pushes that expose my email", and the repository I was attempting to import from Bitbucket contained commits from my personal email address. After temporarily disabling this setting (GitHub > Settings > Email) I was good to go.
The simplest way of doing it:
git remote rename origin repo_bitbucket
git remote add origin https://github.com/abc/repo.git
git push origin master
Once the push to GitHub is successful, delete the old remote by running:
git remote rm repo_bitbucket