According to a note in Cloud Build documentation titled Accessing private GitHub repositories:
When you run builds using Cloud Build triggers, you can automatically connect to any private repository you own without storing your credentials in Secret Manager.
Based on this, I have tried to git clone my private GitHub repo (without piping ssh keys from Secret Manager to ssh files which the doc states is unnecessary using a build trigger) to no avail. Using ssh below in my cloudbuild.yaml file:
steps:
- name: google/cloud-sdk:alpine
id: Clone repo
entrypoint: git
args: ['clone', 'git#github.com:my-org/my-repo.git']
results in error:
Step #0: Host key verification failed.
Step #0: fatal: Could not read from remote repository.
And using https
args: ['clone', 'https://github.com/my-org/my-repo.git']
I get:
Step #0 - "Clone repo": fatal: could not read Username for 'https://github.com': No such device or address
Is there any way to clone a private GitHub repo within cloudbuild.yaml without tediously piping ssh keys from Secret Manager to volumes before the clone? Any tips would be much appreciated.
As mentioned in the note shared, You need to configure your Cloud Build Trigger, if you want to avoid Secret Manager.
The Build Trigger setup step involves authenticating to your source repository with your username and password.
So when you fire this Cloud Build Trigger, it will not ask for your credentials in Secret Manager, as the authentication is already provided in an earlier step (Trigger Setup).
I found a similar case that has been created as an issue in github which can help you to resolve your errors while using ssh.
For https approach, I would recommend you to remove https://github.com from the url.
And I found another issue that has been created in github which can help you to resolve your issue while using https approach.
Related
Since Github changed their authentication methods to only accept Personal Access Tokens I've had some trouble with getting my GCP Build Trigger to run when I push to the main branch of my repo.
Does anyone know how I can re-authenticate, or change the password that's being used to connect GCP to Github?
On GCP I have tried reconnecting to the repo, 'forgetting' the repo and then reconnecting, I'm not incredibly clued up on this platform, I've only been using it for a few weeks.
"token" would be used for HTTPS URL.
The official GCP documentation uses SSH URLs, which does not need tokens (but SSH keys): that would be one alternative.
I am using GitHub Actions to deploy to Azure. In this project I am using our own private repository's which we host on GitHub. These repository's will be installed during build and their links are stored in requirements.txt, for example:
git+ssh://git#github.com/org-name/package-name.git
Locally, there is no problem installing the requirements, since I have access to these private repository's with SSH. But how would I access these during build in GitHub actions.
I get the error:
Collecting git+ssh://****#github.com/org-name/package-name.git (from -r requirements.txt (line 1))
Cloning ssh://****#github.com/org-nam/package-name.git to /tmp/pip-req-build-9nud9608
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****#github.com/org-name/package-name.git' /tmp/pip-req-build-9nud9608 Check the logs for full command output.
Error: Process completed with exit code 1.
Which makes sense, since it is a private repository.
You might try and include in your GitHub Action workflow the webfactory/ssh-agent action:
When running a GitHub Action workflow to stage your project, run tests or build images, you might need to fetch additional libraries or vendors from private repositories.
GitHub Actions only have access to the repository they run for.
So, in order to access additional private repositories:
create an SSH key with sufficient access privileges.
Then, use this action to make the key available with ssh-agent on the Action worker node.
Once this has been set up, git clone commands using ssh URLs will just work. Also, running ssh commands to connect to other servers will be able to use the key.
That would give a workflow like:
# .github/workflows/my-workflow.yml
jobs:
my_job:
...
steps:
- actions/checkout#v1
# Make sure the #v0.4.1 matches the current version of the
# action
- uses: webfactory/ssh-agent#v0.4.1
with:
ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
- ... other steps
For those wondering, another solution which I found and is easier to apply, is to use an acces token:
- name: Install requirements
run: |
git config --global url."https://${{ secrets.ACCESS_TOKEN }}#github".insteadOf https://github
pip install -r requirements.txt
Don't forget to create a personal access token and set it as ACCESS_TOKEN in your repository secrets.
I'm trying to create an automated build on Docker Hub from a Github private repository.
I linked my Github account with my Docker Hub account.
I verified that the deploy keys provided by Docker Hub are created in my Github repository
But I still have an error. I'm not sure if it's related to SSH keys or gitmodules.
Here is the log I get (in Build Details tab in my Docker Hub repo) :
Starting to clone
Cloning into 'bsntx4mjvrqncsdvzcn7tqp'...
Warning: Permanently added the RSA host key for IP address 'XXX.XX.XX.XX' to the list of known hosts.
No submodule mapping found in .gitmodules for path 'vendor/bundle/ruby/2.1.0/bundler/gems/logan-3722sdf9a9db5'
please ensure the correct public key is added to the list of trusted keys for this repository and the remote branch exists. (1)
Does anyone has the same error ?
My solution : I have solved my problem by removing the file vendor/bundle/ruby/2.1.0/bundler/gems/logan-3722sdf9a9db5 in my github repo. So the error about the public keys has nothing to do with it.
The issue is not with the key.
I don't know the exact issue. You can use the following command to sort the issue out.
git submodule update --init
The output of the command will be passed to
git rm --cached {{output}}
Push the commit.
I can't comment, so I'm posting this as an "answer" instead (sorry!)
I have the same issue with Bitbucket, it fails to fetch from my submodule which is included in my main repo. I haven't been able to fix this issue yet. I even bough a paid subscription and added an automated build in hub.docker, hoping that would fix it, but nope..
Starting to clone
Cloning into 'bokyrk3hkhspby5smhf6kby'...
Warning: Permanently added the RSA host key for IP address 'xxx.xxx.xxx' to the list of known hosts.
Submodule 'lib/shared' (https://xxx#bitbucket.org/xxx/xxx_sharedlibs.git) registered for path 'lib/shared'
Cloning into 'lib/shared'...
fatal: could not read Password for 'https://xxx#bitbucket.org': No such device or address
Clone of 'https://xxx#bitbucket.org/xxx/xxx_sharedlibs.git' into submodule path 'lib/shared' failed
please ensure the correct public key is added to the list of trusted keys for this repository and the remote branch exists. (1)
Following procedure followed by me made it work:
Added my private key to local ssh through git bash.
Added my public SSH key to github.
This ssh key was already available in my dockerhub.
So, I linked another public dockerhub userid (where automated build was successful) as a collaborator to this private automated build.
For work with private repo, you should add ssh key from Docker Hub (showing on add auto build) to your github ssh keys.
When the automated build fails the orange alert on the top of the build details can be a bit deluding:
Build failed: please ensure the correct public key is added to the
list of trusted keys for this repository and the remote branch exists.
(128)
Even when you have set up you ssh keys this message always displays. You need to look at your Logs at the bottom of the build details to see more information if you know that you have set up your ssh keys for a private repo already.
For example your logs might say:
fatal: no submodule mapping found in .gitmodules for path
'my-app/my-blog'
or maybe it could be
Dockerfile not found at ./my-app/Dockerfile
in your case you had a problem related to having the file vendor/bundle/ruby/2.1.0/bundler/gems/logan-3722sdf9a9db5 in your solution which needed to be removed.
I would like to create a webhook within Gitlab to automatically update a mirror repository on Github, whenever a push event happens. I've checked this page, but I didn't understand how it is done.
My Gitlab version is 6.5. Here is the configuration page:
What should I put in URL? Where do I need to place the script to update the repository?
You don't need a webhook for that. A regular post-receive hook will work very well.
To create and use such a hook you just have to login on the server where your gitlab is installed and create an ssh key for git user.
sudo -u git ssh-keygen -f /home/git/.ssh/reponame_key
(do not type any passphrase when prompted)
Go to your github account and add the public key (it's been created as /home/git/ssh/reponame_key.pub) to your project as a deploy key.
have a look at https://help.github.com/articles/managing-deploy-keys if you need help with that.
Once that is done, you just have to configure the connection between your git server and github's:
add an alias to git user's ssh configuration (add following lines to /home/git/.ssh/config - create it if it's not present)
Host reponame
IdentityFile /home/git/.ssh/reponame_key
HostName github.com
User git
Now add the new remote (using the alias you just created) to your repository:
cd /home/git/repositories/namespace/reponame.git
git remote add --mirror github reponame:youruser/reponame.git
Now that everything is in place you'll have to create the actual hook:
cd /home/git/repositories/namespace/reponame.git/hooks
echo "exec git push --quiet github &" >> post-receive
chmod 755 post-receive
The lastcommand is very important because git will check if a hook is executable before running it.
That's it!
(Replace reponame, namespace and youruser according to your real accounts and enjoy).
Last note: if you want your name andavatar near commits on github, make sure that the email address you are using on gitlab is one of the addresses inked to your github account as well. You'll see your gitlab username otherwise.
If you aren't hosting your own GitLab, GitLab.com has introduced this feature directly, without any workarounds.
From within a project use the gear icon to select Mirror Repository
Scroll down to Push to a remote repository
Checkmark Remote mirror repository: Automatically update the remote mirror's branches, tags, and commits from this repository every hour.
Enter the repository you want to update; for GitHub you can include your username and password in the URL, like so: https://yourgithubusername:yourgithubpassword#github.com/agaric/guts_discuss_resource.git —as noted in the comments, it is much better securitywise to use your GitHub access token here instead of login credentials; will update the answer when i've tested.
For WebHooks processing I'm using sinatra web server.
require 'sinatra'
post '/pew' do
puts JSON.parse request.body.read
# here can be placed signal code to run commit processing script
end
register webhook for push events(or other) to http://localhost:4567/pew within GitLab
and since this moment on each commit gitlab will be sending commit info to url.
I have attempted to add a new repository to github per their usual instructions with an odd rejection...
Owner#Owner-HP ~/Bescrewed
$ git push -u origin master
ERROR: Permission to TangibleDream/Bescrewed.git denied to TangibleDream/demo_app.
fatal: The remote end hung up unexpectedly
Owner#Owner-HP ~/Bescrewed
The problem is, demo_app was my last repository. The one I'm doing presently is bescrewed and should be in no way connected to demo_app.
Has anyone had this issue before?
It might be tied to an ssh issue as described in GitHub help page:
Permission to user/repo2 denied to user/repo1
This error occurs when you attach your key as a deploy key on repo1.
You can push and pull from that repo without issue, but you won’t have access to any other repo with your key.
To solve this, remove the key from repo1’s deploy keys and attach it on your account page instead. This key will now have access to all repos your account has access to.
See Deploy Keys
What are deploy keys?
Deploy keys are ssh keys just like the ones you attach to your account to allow you to push to and pull from your repos.
The only difference is that deploy keys are designed to allow access to a single private repo.
This will allow your staging or production server to pull in from your repo
The other possibility is that somehow your 'origin' remote is incorrect (check what git remote -v returns)