How to use github cli to auto pull all newly created or updated repos to local pc?
I think I need to listen for new repo creation/updation and pull the repos. How to do it with cli?
If can't listen, I need to pull latest 100 repos to local machiene. How to do it?
I tried https://api.github.com/users/xxxx/repos?per_page=100. It gives in alphabetical order.
I use following code
#!/bin/sh
cat repolist.txt | while read line
do
REPOSRC=$line
LOCALREPO=$line
# We do it this way so that we can abstract if from just git later on
LOCALREPO_VC_DIR=$LOCALREPO/.git
if [ ! -d $LOCALREPO_VC_DIR ]
then
cd ~/xxxx
gh repo clone $REPOSRC
cd ~
else
cd ~
gh repo sync $REPOSRC -s $REPOSRC
fi
done
# End
The sort key
you're looking for seems to be sort=pushed.
Try curl -s 'https://api.github.com/users/xxxx/repos?sort=pushed&per_page=100' | jq '.[].name' to verify.
Related
First off, I am not using command line git at all. I am only using SourceTree's gui interface. I would prefer to solve my problem this way; if possible.
Somehow, my submodule has become corrupted. Attempting to fetch or pull gives me the following error:
I haven't found any answers for this particular problem. I am fortunate in that my remote master is ok, 100% up to date, and I have no local changes. So, I think the easiest way will be to just fully reset my local submodule.
However, I can't figure out how to do so.
I considered trying to remove my submodule and then re add it. However, I have had problems with that in the past, and so am gun shy.
I found handfuls of posts about resetting to a specific commit. However, the SourceTree gui is failing to populate my history because of this error.
Any help would be appreciated
Thank you
You can try this
Right click at your conflicted branch at BRANCHES and delete it.
Double click at your remote branch at REMOTES to re-download and switch to that branch.
Easy, good luck.
I finally did get this fixed. I had to give up and use the command line.
I found this page (git fatal: failed to read object xxx: Invalid argument). That pointed me to using the "git fsck --full" command.
That pointed me to a very specific folder in the .git hierarchy that was corrupted.
I needed to delete this folder, but doing so wasn't easy. Windows would let me delete it. Not in safe mode. Not with cmd del or rmdir. I had to run a scan disk from windows on my entire drive. That ended up detecting the folder and removing it.
Finally, from there, I was able to fetch and pull master again.
From all the solutions I was able to find to perform an actually hard reset with proper cleanup and reinitialization of submodules nothing worked. So I ended up creating a custom script. Following will perform a full cleanup of your submodules and will recreate them as if you were cloning the parent repo.
#!/bin/bash
echo "Backing up current .gitmodules"
cp -f .gitmodules .gitmodules.bkp
has_submodules=$(echo "$(git submodule)")
if [ "$has_submodules" != "" ]; then
git submodule deinit --force .
rm -rf .git/modules
git submodule | cut -c43- | while read -r line; do (git rm -f "$line"); done
fi
cp -f .gitmodules.bkp .gitmodules
PARAMS=("path" "url" "branch")
CHUNKS=${#PARAMS[#]}
PARAMS_STR=$(IFS='|'; echo "${PARAMS[*]}")
readarray -t SUBMODULES_INFO_ARRAY < <(git config -f .gitmodules --get-regexp '^submodule\..*\.('"$PARAMS_STR"')$')
SUBMODULES_INFO_ARRAY_STR=$(IFS='|'; echo "${SUBMODULES_INFO_ARRAY[*]}")
function process_submodules_parsed_array()
{
name=${SUBMODULES_PARSED_ARRAY[0]}
path=${SUBMODULES_PARSED_ARRAY[1]}
repo_url=${SUBMODULES_PARSED_ARRAY[2]}
branch=${SUBMODULES_PARSED_ARRAY[3]}
echo "Running: git submodule add --force -b "$branch" --depth 1 --name "$name" -- "$repo_url" "$path""
git submodule add --force -b "$branch" --depth 1 --name "$name" -- "$repo_url" "$path"
}
echo "Parsing current .gitmodules"
for((chunk=0; chunk<${#SUBMODULES_INFO_ARRAY[#]}; chunk+=CHUNKS))
do
section_name="$(sed -nE 's/^submodule\.(.*?)\.'${PARAMS[0]}'\s(.*?)$/\1/p' <<< "${SUBMODULES_INFO_ARRAY[chunk]}")"
SUBMODULES_PARSED_ARRAY=("$section_name")
for ((param_i=0; param_i<CHUNKS; param_i+=1))
do
param_unparsed="${SUBMODULES_INFO_ARRAY[chunk+param_i]}"
param_parsed="$(sed -nE 's/^submodule.*?\.'${PARAMS[param_i]}'\s(.*?)$/\1/p' <<< "$param_unparsed")"
SUBMODULES_PARSED_ARRAY+=($param_parsed)
done
process_submodules_parsed_array
done
echo "Restoring .gitmodules"
mv -f .gitmodules.bkp .gitmodules
echo "Initializing submodules"
git submodule init
git submodule update
How do I batch archive my repositories? I'd preferably want to be able to sort through them and figure out a way to not archive my active repositories.
I have hundreds of old GitHub repositories in my account since before the GitHub notifications feature, and now I get vulnerability notifications for all of them. Here's what my notifications look, for projects that were last used maybe 6 years ago:
You can use the GitHub API along with two tools to achieve this. I'll be using:
Hub, but you can make direct API calls
jq, but you can use any JSON parser
Here's how:
Fetch a list of all the GitHub repositories in our account, and saving them in a file:
hub api --paginate users/amingilani/repos | jq -r '.[]."full_name"' > repos_names.txt
Go through that file manually, remove any repositories you don't want to archive
Archive all the repositories in the file:
cat repos_names.txt | xargs -I {} -n 1 hub api -X PATCH -F archived=true /repos/{}
Note: since 2020:
gh repo list has been released (with gh 1.7.0 in commit 00cb921, Q1 2021): it does take pagination in account, as it is similar to an alias like:
set -e
repos() {
local owner="${1?}"
shift 1
gh api graphql --paginate -f owner="$owner" "$#" -f query='
query($owner: String!, $per_page: Int = 100, $endCursor: String) {
repositoryOwner(login: $owner) {
repositories(first: $per_page, after: $endCursor, ownerAffiliations: OWNER) {
nodes {
nameWithOwner
description
primaryLanguage { name }
isFork
pushedAt
}
pageInfo {
hasNextPage
endCursor
}
}
}
}
' | jq -r '.data.repositoryOwner.repositories.nodes[] | [.nameWithOwner,.pushedAt,.description,.primaryLanguage.name,.isFork] | #tsv' | sort
}
repos "$#"
gh repo list --no-archived can limit the list to your not-yet-archived repositories
gh repo archive can then, for each element of that list, archive the GitHub repository.
wolfram77 also proposes in the comments:
gh repo list <org> | awk '{NF=1}1' | \
while read in; do gh repo archive -y "$in"; done
Using only gh.
gh repo list --no-archived --limit 144 --visibility public --source --json nameWithOwner --jq ".[].nameWithOwner" > repos_names.txt
Set --limit to the number of repositories you have.
Use vim to delete line which you don't want to archive by pressing dd on the line:
vim repos_names.txt
Run the the following command to arrive them
cat repos_names.txt | while read in; do gh repo archive -y "$in"; done
Clear after:
rm repos_names.txt
From the answer to a related question I know it's possible to batch clone repositories based on a GitHub search result:
# cheating knowing we currently have 9 pages
for i in {1..9}
do
curl "https://api.github.com/search/repositories?q=blazor+language:C%23&per_page=100&page=$i" \
| jq -r '.items[].ssh_url' >> urls.txt
done
cat urls.txt | xargs -P8 -L1 git clone
I also know that the Hub client allows me to make API calls.
hub api [-it] [-X METHOD] [-H HEADER] [--cache TTL] ENDPOINT [-F FIELD|--input FILE]
I guess the last step is, how do I archive a repository with Hub?
You can update a repository using the Update a Repository API call.
I put all my repositories in a TMP variable in the following way, and ran the following:
echo $TMP | xargs -P8 -L1 hub api -X PATCH -F archived=true
Here is a sample of what the $TMP variable looked like:
echo $TMP
/repos/amingilani/9bot
/repos/amingilani/advent-of-code-2019
/repos/amingilani/alan
/repos/amingilani/annotate_models
Looking for particular command or python script to download all repositories or sub branches of the particular organization from Github at once
This gist (or this one) allows to list and clone all repos from an organization
curl -s https://api.github.com/orgs/twitter/repos?per_page=200 | ruby -rubygems -e 'require "json"; JSON.load(STDIN.read).each { |repo| %x[git clone #{repo["ssh_url"]} ]}'
You have the same in python with the project muhasturk/gitim
It isn't hard to curl the zip archive of a repo instead (instead of cloning the repo):
curl -u '<git username>' -L -o master.zip https://github.com/<organization>/<reponame>/zipball/master
I am looking to backup all my Starred repositories and I'm searching for a simple a way to do it.
This should do it. Note you will need
jq
curl https://api.github.com/users/<user>/starred |
jq -r '.[].html_url' |
xargs -l git clone
If you don't want to use jq you can replace with this ugly awk line
awk '/^ {4}"html_url"/&&$0=$4' FS='"' |
Yes, here is a simple one-liner (change foo to your username):
USER=foo; curl "https://api.github.com/users/$USER/starred?per_page=1000" | grep -o 'git#[^"]*' | xargs -L1 git clone
Add -P parameter for xargs to increase speed by setting number of parallel processes (e.g. -P4 = 4 processes).
To raise the GitHub limits, you can authenticate by specifying your API key.
If you don't have any issues with ruby and can install a few gems, you can give this a shot.
gem install octokit git parallel
Then this should do it.
ruby -e "require 'octokit'; require 'git'; require 'parallel'; Parallel.each(Octokit.starred('__username__'), :in_processes=>4){|s| Git.clone(s[:html_url], s[:name])}"
For readability:
require 'octokit'
require 'git'
require 'parallel'
Parallel.each(Octokit.starred('__username__'), :in_processes=>4){|s| Git.clone(s[:html_url], s[:name])}
This does seem like overkill though.
You could also fetch the list as suggested, and create a new Repo for thoose projects that you are really interested on backuping. This method will help you update your backups easily.
Create a new repository, public/private, with submodules and clone/pull all toggether. You should see something like this.
git clone https://github.com/<user>/<repo>
cd <repo>
git submodule add https://github.com/<someone>/<repo1>
git submodule add https://github.com/<someother>/<repo2>
git submodule update --init --recursive
git commit -m "submodules added"
git push
Then do
git clone --recurse-submodules https://github.com/<user>/<repo>
# OR
git pull --recurse-submodules