We have a Flutter app which uses a private repository as a dependency.
The SSH key has been added to CircleCI, and the remote access to the repository works just fine locally using this same key.
The config has been added to the .circleci/config.yml:
- add_ssh_keys:
fingerprints:
- "84:1a:so:me:ke:y:14:31:0f"
But CircleCI keeps failing to access the private dependency repo , giving the following error:
" Running "flutter pub get" in project... Git error. Command: git clone --mirror git#bitbucket.org:our_account/priv_repo.git /home/circleci/development/flutter/.pub-cache/git/cache/priv_repo-3456accd54b38ec5b3820944f77e90ce2ddc9887
stdout: stderr: Cloning into bare repository
'/home/circleci/development/flutter/.pub-cache/git/cache/priv_repo-3456accd54b38ec5b3820944f77e90ce2ddc9887'...
Warning: Permanently added the RSA host key for IP address
'18.205.93.1' to the list of known hosts. Unauthorized fatal: Could
not read from remote repository.
Please make sure you have the correct access rights and the repository
exists. exit code: 128 "
Has anyone successfully created a CircleCI deployment which includes a private repo dependency?
What might possibly be missing to cause this issue?
OK - there were a couple of things I had wrong,
The main one was that I had the add_ssh_keys line in the wrong place.
It really needs to be the first step, or at least be before the flutter/install_sdk_and_pub step.
eg. This works (but if the add_ssh_keys step was at the bottom of the list of 4 steps here then it fails):
steps:
- add_ssh_keys:
fingerprints:
- "84:1a:so:me:ke:y:14:31:0f"
- checkout
- aws-cli/setup:
profile-name: example
- flutter/install_sdk_and_pub:
flutter_version: 2.5.3
In addition to that it is worth noting that I added my SSH key as an "Additional SSH Keys" type key (adding a "User Key" broke the deploy) (that is under Project Settings > SSH Keys),
and I set the Hostname for the key to "bitbucket.org".
So CircleCI is now successfully pulling in my private repo dependency.
It is failing on versioning mismatch stuff, but that is another issue, and shall be solved at another time.
I want to clarify some things first.
The main one was that I had the add_ssh_keys line in the wrong place.
It really needs to be the first step, or at least be before the flutter/install_sdk_and_pub step.
Yes, it's because each step runs in a different session, so they don't share the configs from each other. Your ssh-key only works because the environment variables defined on the Project Settings, are used across all the steps.
eg. This works (but if the add_ssh_keys step was at the bottom of the list of 4 steps here then it fails):
Yes, and if you want to use this fingerprint in another step, you need to paste it there too.
There are some things that you can do to fix your problem, you can leave the main ssh-key on the Project Settings and you can create an environment variable with the public key content (from another ssh-key).
After that, before using that second key, you need to run some commands like:
- run:
name: Step I'm using the second key
command: |
echo -e $MY_ENV_VAR | base64 -d > key
chmod 400 key
eval $(ssh-agent -s)
ssh-add key
git clone...
docker build...
Doing that, you will be able to clone the second repo.
Two things that worked for me:
Adding checkout step before updating the submodules
Adding the User public key(from CircleCI) to Bitbucket both repositories,the submodule repository and the parent repository that uses the submodule.
Checkout
As mentioned here https://circleci.com/docs/configuration-reference#checkout:
Note: CircleCI does not check out submodules. If your project requires submodules, add run steps with appropriate commands as shown in the following example:
- checkout
- run: git submodule sync
- run: git submodule update --init
User Key
The "User Key" public key is not added automatically to Bitbucket. Here is an explanation on how to add it https://circleci.com/docs/bitbucket-integration#create-a-bitbucket-user-key.
After you copy the key from the "Developer Tools" you need to add it to "Access keys" in Bitbucket.
Related
While trying to build a project in CircleCI I got the below error:
Either git or ssh (required by git to clone through SSH) is not installed in the image. Falling back to CircleCI's native git client but the behavior may be different from official git. If this is an issue, please use an image that has official git and ssh installed.
Cloning git repository
error cloning repository: repository not found
And this causes the build to fail.
Here's how I solved it:
The issue was that the engineer who set up the project set a Deploy Key and User Key specific to him left the company, so his access to the company GitHub repo projects was revoked, which made the existing Deploy Key and User Key for the project in Circle CI obsolete.
All I had to do was to:
Go to Project Settings
Then SSH Keys
Remove the existing Deploy Key and User Key (if any)
Add a new Deploy Key
(Optional - Not needed) Add a new User key (by authorizing with GitHub)
After this, I was able to build the project successfully in Circle CI
How would i include a private repo thats on bitbucket in my pubspec.yaml?
Locally with the code checked out we can add this to pubspec.yaml
sp_utilities:
path: ../../packages/sp-utilities
But this would fail if you build from a build server.
We tried adding it as follow to no avail
sp_utilities:
git:
url: https://xxx#bitbucket.org/xxx/xxx-utilities.git
From the Dart Documentation:
Even if the package repo is private, if you can connect to the repo using SSH, then you can depend on the package by using the repo’s SSH URL:
So the lines in the Pubspec.yaml should be as follow:
dependencies:
kittens:
git: https://xxx#bitbucket.org/xxx/xxx-utilities.git
This however requires setting up a SSH Key for the authentication.
These steps can vary a little according to your system, and specific instructions can be found on the link above, but the tasks roughly are the same:
Creating a pair o SSH key files using the command ssh-keygen.
Adding this files to the SSH-Agent so you don't have to type your password everytime. The command related to this is ssh-add.
Log in to your Bitbucket account and add the public key file contents there. That option should be inside Personal settings > Account Settings > SSH Keys.
Confirm the operation on your e-mail.
You can confirm you have made all the steps above correctly by trying to SSH directly to it:
ssh -T git#bitbucket.org
I tried to follow the instructions in this article:
http://developer.cloudbees.com/bin/view/DEV/GitHub+Commit+Hooks+HOWTO
But I have two problems:
It doesn't work. When I push to github nothing happens on the jenkins side. On Jenkins the GitHub Hook log says 'Polling has not run yet.'
The possible cause for this is that I can't specify the git repository as described in this image:
There is no textfield for a public key, nor the other fields. Instead I have the fields
Repository URL, Name, Refspec
Also I can specify a http url (and everything but the triggering works). But if I specify the git/ssh url as in the image I get the following error:
Failed to connect to repository : Command "git ls-remote -h
git#github.com:schauder/degraph.git HEAD" returned status code 128:
stdout: stderr: Warning: Permanently added
'github.com,207.97.227.239' (RSA) to the list of known hosts.
Permission denied (publickey). fatal: Could not read from remote
repository.
Please make sure you have the correct access rights and the repository
exists.
I copied the url from github so a typo is somewhat unlikely.
I verified that I have the github plugin installed (upgraded to version 1.5)
Any hints how to fix or at least debug the problem?
Update answering questions in the comments:
Yes I checked the checkbox 'Build when a change is pushed to GitHub' in the job configuration.
And Yes I setup the webhook in GitHub. When I click the test button there, nothing happens. When I put the url from the webhook in my browser I get a http 500 saying
Not intended to be browsed interactively (must specify payload parameter)
Plus some lengthy stacktrace.
Another Update
I checked the GIT related plugins installed on the Jenkins side. Everything showing their with GIT in the text is installed and at the most current version
Based on Git error message above, I assume you can't be authenticated. Did you add the CloudBees public SSH key on GitHub as described here?
Clarification by OP
The image referenced in the question seemed to suggest that one puts a public key from GitHub into CloudBees. But actually it is the other way round, and also all the labels seem to be changed, so it is kind of difficult to match that screenshot to what you see in CloudBees.
So what I did is this:
in the Job configuration at Cloudbees there is a Public Key (generated by CloudBee). I it in the CloudBees DEV#cloud Authorization section and labeled CloudBees Public Key. Copy that Key.
Hop over to GitHub and go to Settings > Deploy Keys. Add the key there.
go to Settings > Service Hooks > WebHook URLs. Click on the test button. You are not going to see much on the GitHub side.
Hop back to CloudBees Jenkins. From the main page of the job go to the ´GitHub Hook Log´. There you should see something like:
Started on Mar 15, 2013 8:55:14 AM
Using strategy: Default
[poll] Last Built Revision: Revision 581acde89a62317fd8dabaa3f4c6025d1c9dd413 (origin/master)
Fetching changes from the remote Git repositories
Polling for changes in
Done. Took 0.34 sec
Changes found
And your job should start.
If the last line says No Changes this means the hook worked, but you already build the current version so the build is not triggered again.
If you still have problems, have look at https://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin#GitHubPlugin-TriggerabuildwhenachangeispushedtoGitHub under 'trouble shooting' it tells you how to set up logging on the jenkins side. With this I get all kind of stuff in the Jenkins SystemLog.
I've just published a tutorial on continuous integration for Play applications hosted on GitHub using CloudBees. This is intended to provide a comprehensive guide:
http://ics-software-engineering.github.io/play-example-continuous-integration/
I've been trying to start a project using EGit with Bitbucket but I can't push the project to the repository in the server.
These are the steps I've followed (I'll try to be very specific so other people can use it as a tutorial mode; even though it's not working for me it might work for others):
1) I installed Git on Ubuntu 12.10 using this tutorial
2) Followed this tutorial for GitHub.
Start a new C++ project in Eclipse
Under Window > Preferences > General > Network Connection > SSH2 > Key Management tab, I generated a new RSA key and added it to the list of deployment keys of my repository on Bitbucket.
Right click on the eclipse project: Team > Share project ; Team > Add to index ; Team > Commit
Window > Show View > Other > Git > Git Repositories
Right click on Remotes > Create Remote > Checked "Configure push" and typed my project's name
Add Push URL: ssh://git#bitbucket.org/myuser/myrepository.git, Protocol: ssh; port: blank; user: git; password: blank
Advanced button under Ref mappings and selected the "master" version. If this step works, it means that the RSA key is configured in Bitbucket and EGit properly. Otherwise it will show a connection error.
Everything is done. However, when I press "Push" I get the following error:
ssh://git#bitbucket.org/myuser/myrepository.git: push not permitted
I've tried every configuration possible: switched to a new work space, updated Eclipse, created new repositories, re-installed Git... Any ideas?
Alright, I've finally found a way to work with it.
It's just changing the authentication mode. If the HTTPS link is used, instead of using the SSH link, everything works perfectly. It seems that SSH works just in a read-only mode.
Of course the authentication method is different: you have to type the password and the SSH keys stored in Bitbucket are no longer necessary.
For those who are having problems with EGit, I recommend following eugener's way to automatically configure EGit. Then you just have to select HTTPS as authentication mode to push the stored commits.
There is much simpler way of configuring remote Git repos in Eclipse:
Create an empty repo on BitBicket
Copy repo's URL
Clone repo to your machine using "Git Repositories" view.
Share your project into newly cloned repo.
Since a repo was cloned - remote settings were configured automatically. This means you will be able to push immediately
It might be a user name misconfiguration in .gitconfig.
Probably related, I fixed "auth fail" problems by following the instructions on:
http://wangpidong.blogspot.com.es/2012/05/how-to-use-bitbucket-with-egit-in.html
Got it from:
Configuring egit with BitBucket
In this specific case I agree with resolution from Olivier J.'s comment under original question.
But I have had similar problem and it was caused by something else. I resolved it and here is what happened:
I have been using some private repository at Bitbucket (A) with some user account and some public key (X) to access it. I have had key X loaded in my pageant Putty key chain.
Then I created a new public repository (B), also at Bitbucket with a different user account and another public key (Y). I added this key to pageant too.
When I tried to access repository B apparently public key X and not Y was used, because ..for SSH client both connections are the same - same user (git) at the same host (bitbucket.org)! And key X was used first because ..it was loaded earier? Or maybe because its name was alphabetically first? I am not sure but the result was that...
Read-only access for B using X was granted because the repository was public and any valid Bitbucket user key was enough to gain this kind of access!
..but writing (pushing) required key Y, which was not used by SSH.
So the general tip is: to resolve strange read-only git via SSH access errors check carefully which private key is actually used by SSH.
I resolved this problem by loading only one key - X or Y - in pageant at a time but I would love to do it a smarter way - please let me know in comments if you know how!
Using Bitbucket I do it this way:
create a repository in bitbucket
clone the empty repository using the https url
connect the project in eclipse to the repository using the Team -> Share option and select the location where you cloned the repo
index (stage) your project files
commit and push
What I have done so far:
Deployed a tomcat6.0 server to c:\www
Downloaded and deployed the Hudson.war in the c:\www\webapps folder
Installed the Github plugin
Created a private repository on Github
On the server, with hudson installed, generated ssh keys.
Environment variable %HOME% is set to c:\Documents and settings[username] (there is a .ssh directory in there with the keys)
Environment variable %HUDSON_HOME% set to c:\www\webapps\hudson
In hudson I have the following configurations:
Github Project: https://github.com/[my organization]/[project name]
Source Code Management: Git
URL of Repository: git#github.com:[my organization]/[project name].git
Branch specifier: **
Repository Browser: (Auto)
When I run a build, and click the console output link I see this --
Started by user anonymous
Checkout:workspace / C:\www\webapps\hudson\jobs\[project name] (git)\workspace - hudson.remoting.LocalChannel#2e8f6d20
Using strategy: Default
Checkout:workspace / C:\www\webapps\hudson\jobs\[project name] (git)\workspace - hudson.remoting.LocalChannel#2e8f6d20
Fetching changes from the remote Git repository
Fetching upstream changes from git#github.com:[organization name]/[project name].git
... at which point it hangs. When I cancel the build, the following are added --
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
ERROR: (Underlying report) : Error performing command: git.exe fetch -t git#github.com:[organization name]/[project name].git +refs/heads/*:refs/remotes/origin/*
null
ERROR: Could not fetch from any repository
FATAL: Could not fetch from any repository
hudson.plugins.git.GitException: Could not fetch from any repository
at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:796)
at hudson.plugins.git.GitSCM$3.invoke(GitSCM.java:754)
at hudson.FilePath.act(FilePath.java:756)
at hudson.FilePath.act(FilePath.java:738)
at hudson.plugins.git.GitSCM.gerRevisionToBuild(GitSCM.java:754)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1180)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:506)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:422)
at hudson.model.Run.run(Run.java:1362)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
First off, since you are starting off with a new CI instance, I would highly recommend that you install the Jenkins fork instead (since it is actively maintained by most of the original developers of Hudson).
Secondly, install the DumpInfo Wrapper plugin and re-run the build again. This plugin prints the system properties and environmental variables in effect during the build and allows you to verify them.
Update:
That plugin should log the system properties and environmental variables, I am surprised you do not see them. As for the passphrase, I suggest you generate a separate private key (from your existing pair) that is not passphrase-protected instead, otherwise you may need to consider the solution offered for: Why git can't remember my passphrase under Windows. I have verified that it works (when I set up my own CI on windows) but I don't feel it is worth the trouble (there are other nuances including installing and running the tomcat instance as the currently logged in user, and NOT as local service, so that pageant will work with it properly), hence I would recommend the first option.
In my case it turned out to be the problem with Git client: I was using v1.6.0 which was triggering
fatal: https://github.com/dmak/jaxb-xew-plugin.git/info/refs download error - The requested URL returned error: 403
error message in Hudson. First it looked like this issue, but strace analysis of git run shown that is was Nginx WebServer (which GitHub is running on) returning 403, not the proxy.
When I've updated to v1.7.3 the problem was gone. So general advise: don't use old clients with GitHub.
P.S. I have tested the cloning both with "Branch Specifier (blank for default): origin/master" in Hudson job configuration and also with blank (default): in both cases Git was able to discover the master branch (origin/master) correctly and used it for cloning.