capistrano and ec2 and github - unable to deploy - github

I am trying to deploy my first webapp to EC2 using capistrano and the repo is on github. But I am facing issues with "cap deploy:cold". The deploy.rb is pasted below. Following are the steps I followed.
1) Logged into ec2 instance using ssh from my local machine. Generated keys using ssh-keygen without any passphrase.
2) Took the contents of id_rsa.pub and copied to the github repo settings deploy keys.
3) Then from my local machine, ran "cap deploy:cold". I get the following error
user1#laptop:~/MyExample$ cap deploy:cold
* 2013-03-01 19:08:06 executing deploy:cold'
* 2013-03-01 19:08:06 executingdeploy:update'
** transaction: start
* 2013-03-01 19:08:06 executing `deploy:update_code'
updating the cached checkout on all servers
executing locally: "git ls-remote git#github.com:user1/MyExample.git HEAD"
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
* [deploy:update_code] rolling back
* executing "rm -rf /var/www/MyExample.com/releases/20130301133835; true"
servers: ["181.73.124.219"]
[181.73.124.219] executing command
command finished in 1186ms
set :application, "MyExample.com"
set :scm, "git"
set :repository, "git#github.com:thisuser/example.git"
default_run_options[:pty] = true
set :user, 'ubuntu'
set :use_sudo, true
set :deploy_to, "/var/www/#{application}"
set :deploy_via, :remote_cache
role :web, "181.73.124.219"
role :app, "181.73.124.219"
role :db, "181.73.124.219", :primary => true
after "deploy", "deploy:bundle_gems"
after "deploy:bundle_gems", "deploy:restart"
namespace :deploy do
task :bundle_gems do
run "cd #{deploy_to}/current && bundle install vandor/gems"
end
task :start do ; end
task :stop do ; end
task :restart, :roles => :app, :except => { :no_release => true } do
run "#{try_sudo} touch #{File.join(current_path,'tmp','restart.txt')}"
end
end
What am I missing here? Also any pointers to a URL/blog that can provide detailed steps would help.
Thanks.

I managed to get it working. As seen in the log,
executing locally: "git ls-remote git#github.com:user1/MyExample.git HEAD"
the capistrano is trying to run the above command local machine and not on the server. Running the above command on the terminal also returned the same error - Permission denied (Public Key).
So I had to copy the generated id_rsa and id_rsa.pub from the deploy server to the local machine. And after copying, add it to the ssk keys being used by running "ssh-add /path/to/keys".
After adding those keys, I was able to move ahead.

Related

jenkins -github build issue

I have been facing issues while trying to build a project in jenkins which fetches the files from github .
Below is the error log i got :
p.s: for security reason i have removed the ssh url of git .I can able to clone from my locally but when im trying it from jenkins it fails . I have generated ssh key in linux server and its able to connect to github .Can anyone please help me on this
Building in workspace /var/lib/jenkins/jobs/tr-data-events-testing/workspace
> /usr/local/bin/git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> /usr/local/bin/git config remote.origin.url git#github.url---------git # timeout=10
Fetching upstream changes from git#github.url-----------s.git
> /usr/local/bin/git --version # timeout=10
using GIT_SSH to set credentials jenkins dxc key
> /usr/local/bin/git fetch --tags --progress git#----url--------------.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git#---url-------.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1282)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:610)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:532)
at hudson.model.Run.execute(Run.java:1741)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:529)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:381)
Caused by: hudson.plugins.git.GitException: Command "/usr/local/bin/git fetch --tags --progress git#github.-------events.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: error: The requested URL returned error: 503 while accessing https://--url-----------/info/refs
fatal: HTTP request failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1441)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764)
... 11 more
ERROR: null
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Finished: FAILURE
Help us localize this page
i resolved this issue by un setting the proxy in .gitconfig file . thank you for your answers
First, make sure Jenkins is running with the same user account you have created your ssh keys with.
It it does not, it would search for ~/.ssh/id_rsa in its own $HOME.
Second, to debug, you can launch Jenkins with this environment variable set first:
export GIT_SSH_COMMAND='ssh -Tv'
That way, any time it will try a git command involving ssh, you will see exactly where Jenkins/git tries to look for ssh keys, and what is the error.
Make sure you Git is recent enough to support this (Git 2.3+, Q4 2015).
But your logs show:
The requested URL returned error: 503 while accessing https://
So this has nothing to do with ssh. As mentioned, a proxy can interfere.

How configure job execution for sudo users with NOPASSWD option?

I try run command from another user remotely from rundeck jobs.
Rundeck provide sudo login\passwod mechanism for escalate privilege.
On my remote server in sudoers file NOPASSWD option. How i can configure rundeck jobs in this case?
my node configurations:
<project>
<node name="testServer"
type="Node"
description="testNode"
hostname="IP_ADDRESS"
username="${option.Login}"
ssh-authentication="password"
sudo-command-enabled="true"
ssh-password-option="option.Password"
/>
</project>
Example command in job for become another user:
sudo /bin/su suuser -
Error's after trying executions:
Remote command failed with exit status -1
08:57:13 Sudo execution password response failed: Failed waiting for input prompt: Expected input was not seen in 5000 milliseconds
08:57:13 Failed: NonZeroResultCode: Remote command failed with exit status -1
I added the following three attributes to my node and it appears to work:
sudo-command-enabled="true"
sudo-prompt-pattern=""
sudo-command-pattern="^sudo.*"
You have to set the below attributes in Edit Project Configuration File
project.ssh-authentication=password
project.ssh-keypath=/home/rundeck/.ssh/id_rsa
project.ssh-password-storage-path=keys/xyz.password
project.sudo-command-enabled=true
project.sudo-password-option=option.jobPassword
project.sudo-password-storage-path=keys/xyz.password
project.sudo-prompt-pattern=^Password\:.*

How to upgrade HAProxy on OpsWorks

We are using an AWS OpsWorks stack with Chef Version 11.10. with the default HAProxy Layer. We would like to upgrade the HAProxy to the latest version 1.6-stable (from the default 1.4-stable).
There seems to be a dedicated PPA for our Ubuntu version.
But where can we make OpsWorks use this PPA to install HAProxy?
In the default cookbook there is a default attributes file that has the following lines:
default[:haproxy][:version] = '1.4.22'
default[:haproxy][:patchlevel] = '1'
default[:haproxy][:rpm] = "haproxy-#{node[:haproxy][:version]}-#{node[:haproxy][:patchlevel]}.#{rhel_arch}.rpm"
default[:haproxy][:rpm_url] = "#{node[:opsworks_commons][:assets_url]}/packages/#{node[:platform]}/#{node[:platform_version]}/#{node[:haproxy][:rpm]}"
Overwriting the file in our cookbooks and naïvely changing the version number here does not have the desired effect.
We ended up overwriting the recipe haproxy/recipes/default.rb like so:
#Install software-properties-common if not installed
package 'software-properties-common' do
action :install
end
#Add PPA for haproxy 1.6 and update repo
execute "add-ppa-update" do
command "add-apt-repository ppa:vbernat/haproxy-1.6 && apt-get update -y"
action :run
end
package "haproxy" do
retries 3
retry_delay 5
version '1.6.4-3ppa1~trusty'
action :install
end
if platform?('debian','ubuntu')
template '/etc/default/haproxy' do
source 'haproxy-default.erb'
owner 'root'
group 'root'
mode 0644
end
end
include_recipe 'haproxy::service'
template '/etc/haproxy/haproxy.cfg' do
source 'haproxy.cfg.erb'
owner 'root'
group 'root'
mode 0644
notifies :restart, "service[haproxy]"
end
template "/etc/haproxy/server.pem" do
source "server.pem.erb"
owner 'root'
group 'root'
mode 0600
notifies :restart, "service[haproxy]"
end
service 'haproxy' do
action [:enable, :start]
end
Additionally we needed to update the haproxy.conf to work with the new version.
Everything runs beautifully now.

Chef : Opsworks : run rake task

My objective is to execute a rake task on my apps running in Opsworks.
It appears to me that my opsworks cookbook is not running rake from the correct directory.
How can I tell the cookbook to run in the app home dir (so it can pick up the Gemfile)?
Do I need to specify an RAILS_ENV?
My cookbooks default.rb:
Chef::Log.info("****** Audit Photo URLS : Running Rake Task ******")
execute "rake audit:audi_image_urls" do
command "bundle exec rake audit:audi_image_urls"
end
Errors from Opsworks log:
[2014-11-28T18:36:33+00:00] INFO: Running queued delayed notifications before re-raising exception
[2014-11-28T18:36:33+00:00] ERROR: Running exception handlers
[2014-11-28T18:36:33+00:00] ERROR: Exception handlers complete
[2014-11-28T18:36:33+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache.stage2/chef-stacktrace.out
[2014-11-28T18:36:33+00:00] ERROR: execute[rake audit:audi_image_urls] (auditphoto::default line 3) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '10'
---- Begin output of bundle exec rake audit:audi_image_urls ----
STDOUT: Could not locate Gemfile
STDERR:
---- End output of bundle exec rake audit:audi_image_urls ----
Ran bundle exec rake audit:audi_image_urls returned 10
[2014-11-28T18:36:33+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The execute resource can take a cwd attribute for the working directory from which the command is run.
execute "rake audit:audi_image_urls" do
command "bundle exec rake audit:audi_image_urls"
cwd '/over/there'
environment 'RAILS_ENV' => 'production'
end
OpsWorks Deploy events and Execute Recipes commands
Chef 11
OpsWorks populates node[:deploy]['appshortname'] for Deploy events and Execute Recipes stack commands to house each applications configuration. With this data you could use:
execute "rake audit:audi_image_urls" do
command "bundle exec rake audit:audi_image_urls"
cwd node[:deploy]['appshortname'][:deploy_to]
user node[:deploy]['appshortname'][:user]
group node[:deploy]['appshortname'][:group]
environment( { 'RAILS_ENV' => node[:deploy]['appshortname'][:rails_env] } )
end
You may want to source :environment_variables for the environment if you have anything related configured there.
Chef 12
From the AWS stack settings docco
In Chef 12 Linux, stack settings are available as Chef data bags and are accessed only through Chef search. Data bags are stored on AWS OpsWorks Stacks instances in a set of JSON files in the /var/chef/runs/run-ID/data_bags directory, where run-ID is a unique ID that AWS OpsWorks Stacks assigns to each Chef run on an instance. Stack settings are no longer available as Chef attributes, so stack settings can no longer be accessed through the Chef node object. For more information, see the AWS OpsWorks Stacks Data Bag Reference.
app = search("aws_opsworks_app").first
execute "rake audit:audi_image_urls" do
command "bundle exec rake audit:audi_image_urls"
cwd app['app_source']['deploy_to']
user app['app_source']['user']
group app['app_source']['group']
environment( { 'RAILS_ENV' => app['app_source']['rails_env'] } )
end
Other events and commands
It looks like OpsWorks runs a little differently to a normal Chef server and supplies it's own JSON blob to a local chef instance for each run which means (as you mentioned) the :deploy attributes will be missing for other events/commands Amazon chooses not to supply JSON for.
It might be possible, but very hacky and prone to breakage, to populate the :deploy attributes from the last JSON file that contains deploy state: {"deploy": { "app_name": { "application": "app_name" } in /var/lib/aws/opsworks/chef
You would also need to source the deploy::default attributes after that JSON load to fill in any defaults.

cap deploy - getting permission denied

I am trying to deploy my rails project into sub-domain, but I am getting following error while running cap deploy
[Deprecation Warning] This API has changed, please hook `deploy:create_symlink` instead of `deploy:symlink`.
* 2013-09-27 21:14:44 executing `deploy'
* 2013-09-27 21:14:44 executing `deploy:update'
** transaction: start
* 2013-09-27 21:14:44 executing `deploy:update_code'
executing locally: "git ls-remote git#github.com:anveshnarepo/techzoo.git master"
Permission denied (publickey).
fatal: The remote end hung up unexpectedly
*** [deploy:update_code] rolling back
* executing "rm -rf /var/www/anveshnatech.com/public_html/edupdu/releases/20130927154447; true"
servers: ["http://anveshnatech.com/edupdu/www"]
** [deploy:update_code] exception while rolling back: Capistrano::ConnectionError, connection failed for: http://anveshnatech.com/edupdu/www (SocketError: getaddrinfo: Name or service not known)
Command git ls-remote git#github.com:anveshnarepo/techzoo.git master returned status code pid 12392 exit 128
If i am trying to run cap deploy:setup it is displaying something like:
[Deprecation Warning] This API has changed, please hook `deploy:create_symlink` instead of `deploy:symlink`.
* 2013-09-27 21:23:57 executing `deploy:setup'
* executing "mkdir -p /var/www/anveshnatech.com/public_html/edupdu /var/www/anveshnatech.com/public_html/edupdu/releases /var/www/anveshnatech.com/public_html/edupdu/shared /var/www/anveshnatech.com/public_html/edupdu/shared/system /var/www/anveshnatech.com/public_html/edupdu/shared/log /var/www/anveshnatech.com/public_html/edupdu/shared/pids"
servers: ["http://anveshnatech.com/edupdu/www"]
connection failed for: http://anveshnatech.com/edupdu/www (SocketError: getaddrinfo: Name or service not known)
How to fix this problem?
One error is that you defined a URI where host should go. I suspect you have
role :app, "http://anveshnatech.com/edupdu/www"
in your capfile, when you need something like:
role :app, "anveshnatech.com"
or (example)
role :app, "10.1.1.1"
These are probably not be the only errors, you have some permission problems with your git key and some strangeness with deprecation warnings.