Linking to multiple subdirectories using :repo_tree - capistrano

My repository is set up similar to the following:
repo_base
- artwork
- app
- designsystem
- api
Since each of the other folders in the repo (e.g. app, api, designsystem) depend on artwork, I have symlinks in place when running locally. This is working fine, as the path for images in the designsystem subdirectory is something like ../../artwork. When you check out the repository, the entire tree is checked out, so the symlinks are pointing to the correct directory.
However, when I deploy with capistrano, I use :repo_tree to only deploy a portion of the overall monorepo. For example, the deploy.rb script for the designsystem folder looks like:
# config valid for current version and patch releases of Capistrano
lock "~> 3.11.0"
set :application, "designsystem"
set :repo_url, "git#gitlab.com:myuser/mymonorepo"
set :deploy_to, "/var/www/someplace.net/designsystem.someplace.net"
set :deploy_via, "remote_cache_with_project_root"
set :repo_tree, 'designsystem'
set :log_level, :error
before 'deploy:set_current_revision', 'deploy:buildMonolith'
The problem, of course, is that this only ends up deploying the designsystem subdirectory. Thus, the symlinks aren't valid, and are actually skipped in the building (buildMonolith step).
I'm wondering how I might go about having capistrano check out another subdirectory, artwork, and placing it somewhere in the repository source tree.

I was able to solve this by adding a capistrano task called assets.rb:
require 'pathname'
##
# Import assets from a top level monorepo directory into the current working
# directory.
#
# When you use :repo_tree to deploy a specific directory of a monorepo, but your
# asset repository is in a different directory, you need to check out this
# top-level directory and add it to the deployment path. For example, if your
# monorepo directory structure looks something like:
#
# - /app
# - src/
# - assets -> symlink to ../../assets
# - /assets
# - /api
#
# And you want to deploy /app, the symlink to the upper directory won't exist if
# capistrano is configured to use :repo_tree "app". In order to overcome this,
# this task checks out a specified piece of the larger monorepo (in this case,
# the assets directory), and places it in the deployment directory at a
# specified location.
#
# Configuration:
# In your deploy/<stage>.rb file, you will need to specify two variables:
# - :asset_path - The location within the deployment directory where the
# assets should be placed. Relative to the deployment working
# directory.
# - :asset_source - The location of the top-level asset folder in the
# monorepo. Relative to the top level of the monorepo (i.e.
# the directory that would be used as a deployment if
# :repo_tree was not specified).
#
# In the above example, you would specify:
#
# set :asset_path, "src/assets"
# set :asset_source, "assets"
#
namespace :deploy do
desc "Import assets from a top-level monorepo directory"
task :import_assets do
on roles(:all) do |host|
within repo_path do
final_asset_location = "#{release_path}/#{fetch(:asset_path)}"
asset_stat_result = capture "stat", "-t", "#{final_asset_location}"
asset_stat_result = asset_stat_result.split(" ")
if asset_stat_result[0] == "#{final_asset_location}"
info "Removing existing asset directory #{final_asset_location}..."
execute "rm", "-rf", "#{final_asset_location}"
end
source_dir = Pathname.new(final_asset_location).parent.to_s
info "Importing assets to #{source_dir}/#{fetch(:asset_source)}"
execute "GIT_WORK_TREE=#{source_dir}", :git, "checkout", "#{fetch(:branch)}", "--", "#{fetch(:asset_source)}"
info "Moving asset directory #{source_dir}/#{fetch(:asset_source)} to #{final_asset_location}..."
execute :mv, "#{source_dir}/#{fetch(:asset_source)}", "#{final_asset_location}"
end
end
end
end
It would be nice if I could somehow link into the git scm plugin, rather than calling git from the command line directly.

Related

How to display new Yocto image option after source poky/oe-init-env

let say i have a new yocto image call stargazer-cmd
what file should i edit so that every time i source poky/oe-init-env
it display as a build option to the user?
kj#kj-Aspire-V3-471G:~/stm32Yoctominimal$ source poky/oe-init-build-env build-mp1/
### Shell environment set up for builds. ###
You can now run 'bitbake <target>'
Common targets are:
core-image-minimal
core-image-sato
meta-toolchain
meta-ide-support
I wish to add stargazer-cmd on top of core-image-minimal, i am not sure what to google and what is the file i need to change.
Let me explain how to add a custom configuration to the OpenEmbedded build process.
First of all, here is the process that is done when running:
source poky/oe-init-build-env
The oe-init-build-env script initializes OEROOT variable to point to the location of the script itself.
The oe-init-build-env script sources another file $OEROOT/scripts/oe-buildenv-internal which will:
Check if OEROOT is set
Set BUILDDIR to your custom build directory name $1, or just build if you do not provide one
Set BBPATH to the poky/bitbake folder
Adds $BBPATH/bin and OEROOT/scripts to PATH (This will enable commands like bitbake and bitbake-layers ...)
Export BUILDDIR and PATH to the next file
The oe-init-build-env script continues by running the final build script with:
TEMPLATECONF="$TEMPLATECONF" $OEROOT/scripts/oe-setup-builddir
The oe-setup-builddir script will:
Check if BUILDDIR is set
Create the conf directory under $BUILDDIR
Sources a template file that will check if there is a TEMPLATECONF variable is set:
. $OEROOT/.templateconf
That file contains:
# Template settings
TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
it means that if TEMPLATECONF variable is not set, set it to meta-poky/conf, and that is where the default local.conf and bblayers.conf are coming from.
Copy $TEMPLATECONF to $BUILDDIR/conf/templateconf.cfg
Set some variables pointing to custom local.conf and bblayers.conf:
OECORELAYERCONF="$TEMPLATECONF/bblayers.conf.sample"
OECORELOCALCONF="$TEMPLATECONF/local.conf.sample"
OECORENOTESCONF="$TEMPLATECONF/conf-notes.txt"
In the oe-setup-builddir there is a comment saying that TEMPLATECONF can point to a directory:
#
# $TEMPLATECONF can point to a directory for the template local.conf & bblayers.conf
#
Copy local.conf.sample and bblayers.conf.sample from TEMPLATECONF directory into BUIDDIR/conf:
cp -f $OECORELOCALCONF "$BUILDDIR/conf/local.conf"
sed -e "s|##OEROOT##|$OEROOT|g" \
-e "s|##COREBASE##|$OEROOT|g" \
$OECORELAYERCONF > "$BUILDDIR/conf/bblayers.conf"
Finally it will print what is inside OECORENOTESCONF which points to TEMPLATECONF/conf-notes.txt:
[ ! -r "$OECORENOTESCONF" ] || cat $OECORENOTESCONF
and by default that is located under meta-poky/conf/conf-notes.txt:
### Shell environment set up for builds. ###
You can now run 'bitbake <target>'
Common targets are:
core-image-minimal
core-image-sato
meta-toolchain
meta-ide-support
You can also run generated qemu images with a command like 'runqemu qemux86'
Other commonly useful commands are:
- 'devtool' and 'recipetool' handle common recipe tasks
- 'bitbake-layers' handles common layer tasks
- 'oe-pkgdata-util' handles common target package tasks
So, now, after understanding all of that, here is what you can do:
Create a custom template directory for your project, containing:
local.conf.sample
bblayers.conf.sample
conf-notes.txt
Do not forget to set the path to poky in bblayers.conf to ##OEROOT## as it will be set automatically by the build script.
Set your custom message in conf-notes.txt
Before any new build, just set TEMPLATECONF:
TEMPLATECONF="<path/to/template-directory>" source poky/oe-init-build-env <build_name>
Then, you will find a build with your custom local.conf and bblayers.conf with additional file conf/templateconf.cfg containing the path of TEMPLATECONF
conf/conf-notes.txt in your layer.
OECORENOTESCONF should point to the file.

how to regenerate meson for newly added yaml files

I have added yaml files to add new dbus objects and I added PHOSPHOR_MAPPER_SERVICE_append = " com/newCoName"
(newCoName is the name of my company)
But when I run bitbake, do_configure for phosphor_mapper bails when it passes the option -Ddata_com_newCoName to meson. The following readme says I need to run ./regenerate_meson from the gen directory when I add new YAML files. But how do I do that from a recipe file?
https://github.com/openbmc/phosphor-dbus-interfaces
One option is to generate these files outside ot yocto environment (i.e. not involving bitbake). Thus
clone that git repo
place your yaml file where you cloned repo
do what readme tells, i.e. go to gen directory and execute meson-regenerate script
collect changes that are done by script and create patch
add patch to your layer and reference it in .bbappend file (meta-/recipes-phosphor/dbus/phosphor-dbus-interfaces_git.bbappend)
Another option would be to add to .bbappend file additional task that runs before do_configure - and call that script from there:
do_configure_prepend() {
cd ${S}/gen && ./meson-regenerate
}
Along this .bbappend you should add your yaml so that it lands inside gen folder in patch or directly in your layer (check FILESEXTRAPATHS).
In both cases you'll need to patch meson_options.txt: add option
option('data_com_newCoName', type: 'boolean', value: true)

bitbake recipe - doing a simple copy of the image

I am attempting to write a recipe that would simple copy two files (MyfileA , MyfileB) to a specific directory when the overall image is built. This is what my directory structure looks like:
MyDir/MyRecipe.bb
MyDir/files/MyfileA
MyDir/files/MyfileB
I would like the two files to be copied to a folder in home (which would not exist initially hence the directories should be created)The folder lets say is called "Testfolder"
This is what my bitbake file looks like
DESCRIPTION = "Testing Bitbake file"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
do_install() {
install -d MyfileA ~/TestFolder/
}
Kindly let me know if I am doing something wrong here?
When i run bitbake on this I get the following
The BBPATH variable is not set and bitbake did not find a conf/bblayers.conf file in the expected location.
Maybe you accidentally invoked bitbake from the wrong directory?
DEBUG: Removed the following variables from the environment: LANG, LS_COLORS, LESSCLOSE, XDG_RUNTIME_DIR, SHLVL, SSH_TTY, OLDPWD, LESSOPEN, SSH_CLIENT, MAIL, SSH_CONNECTION, XDG_SESSION_ID, _, BUILDDIR
Any help in this regard would be appreciated.
First of all, to create your own meta-layer, you should run command yocto-layer create MyRecipe in your Yocto Environment. This is to make sure that you have all the necessary element in your meta layer. Make sure to put the new meta-layer into conf/bblayers.conf
Creating HelloWorld Recipe Video can be found here
Second, to copy a file from one to another directories.
DESCRIPTION = "Testing Bitbake file"
SECTION = "TESTING"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
#specify where to get the files
S = "${WORKDIR}"
inherit allarch
#create the folder in target machine
#${D} is the directory of the target machine
#move the file from working directory to the target machine
do_install() {
install -d ${D}/TestFolder
install -m ${WORKDIR}/MyfileA ${D}/TestFolder
}
To get more in details, this is my understanding of how the files move around in Yocto.
You have a directory that stored metadata in /sourced/meta-mylayer/recipes-myRecipe/. In that directory, there would be a folder with the same name as the recipe. I.E. myRecipe/ myRecipe_001.bb.
You would store the files that are related to myRecipe.bb (usually it is a patch) in myRecipe/ so that SRC_URI will get into that myRecipe/ directory to grab files. I.E. myFileA, myFileB
Then, you specify the S. This is the location in the Build Directory where unpacked recipe source code resides. By that mean, myFileA and myFileB are moved/copied to there when myRecipe builds.
Usually, S is equal to ${WORKDIR}, this is equivalent to ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
The actual directory depends on several things:
TMPDIR: The top-level build output directory
MULTIMACH_TARGET_SYS: The target system identifier
PN: The recipe name
EXTENDPE: The epoch - (if PE is not specified, which is usually the case for most recipes, then EXTENDPE is blank)
PV: The recipe version
PR: The recipe revision
After that, we inherit allarch. This class is used for architecture independent recipes/data files (usually scripts).
Then, the last thing we have to do is copy the files.
${D} is the location in the Build Directory where components are installed by do_install task. This location defaults to ${WORKDIR}/image
${WORKDIR}/image can also be described as the / directory in the target system.
Go to ${D} directory and create a folder call TestFolder
Then, copy myFileA from ${WORKDIR} to the ${D}/TestFolder
P.S. Please add comment to fix. There might be mistaken information here, cause I learned all this by myself.

CocoaPods podspec generated from GitHub not matching any source_files

I am developing a Swift 2 project which includes many CocoaPods that work and am struggling to create a podspec file for this OrderedDictionary class from GitHub since the author didn't create a Podfile. I ran:
pod spec create "OrderedDictionary|https://github.com/lukaskubanek/OrderedDictionary"
which created OrderedDictionary.podspec in the root of my project directory:
#
# Be sure to run `pod spec lint OrderedDictionary.podspec' to ensure this is a
# valid spec and to remove all comments including this before submitting the spec.
#
# To learn more about Podspec attributes see http://docs.cocoapods.org/specification.html
# To see working Podspecs in the CocoaPods repo see https://github.com/CocoaPods/Specs/
#
Pod::Spec.new do |s|
# ――― Spec Metadata ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# These will help people to find your library, and whilst it
# can feel like a chore to fill in it's definitely to your advantage. The
# summary should be tweet-length, and the description more in depth.
#
s.name = "OrderedDictionary"
s.version = "0.5"
s.summary = "An implementation of OrderedDictionary in Swift"
# This description is used to generate tags and improve search results.
# * Think: What does it do? Why did you write it? What is the focus?
# * Try to keep it short, snappy and to the point.
# * Write the description between the DESC delimiters below.
# * Finally, don't worry about the indent, CocoaPods strips it!
s.description = "This is a lightweight implementation of an ordered dictionary data structure in Swift packed into a µframework."
s.homepage = "https://github.com/lukaskubanek/OrderedDictionary"
# s.screenshots = "www.example.com/screenshots_1.gif", "www.example.com/screenshots_2.gif"
# ――― Spec License ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# Licensing your code is important. See http://choosealicense.com for more info.
# CocoaPods will detect a license file if there is a named LICENSE*
# Popular ones are 'MIT', 'BSD' and 'Apache License, Version 2.0'.
#
s.license = "MIT"
# s.license = { :type => "MIT", :file => "LICENSE.md" }
# ――― Author Metadata ――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# Specify the authors of the library, with email addresses. Email addresses
# of the authors are extracted from the SCM log. E.g. $ git log. CocoaPods also
# accepts just a name if you'd rather not provide an email address.
#
# Specify a social_media_url where others can refer to, for example a twitter
# profile URL.
#
s.author = { "Lukas Kubanek" => "lukas.kubanek#me.com" }
# Or just: s.author = "Lukas Kubanek"
# s.authors = { "Lukas Kubanek" => "lukas.kubanek#me.com" }
# s.social_media_url = "http://twitter.com/Lukas Kubanek"
# ――― Platform Specifics ――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# If this Pod runs only on iOS or OS X, then specify the platform and
# the deployment target. You can optionally include the target after the platform.
#
# s.platform = :ios
s.platform = :ios, "8.0"
# When using multiple platforms
s.ios.deployment_target = "8.0"
# s.osx.deployment_target = "10.7"
# s.watchos.deployment_target = "2.0"
# s.tvos.deployment_target = "9.0"
# ――― Source Location ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# Specify the location from where the source should be retrieved.
# Supports git, hg, bzr, svn and HTTP.
#
s.source = { :git => "https://github.com/lukaskubanek/OrderedDictionary.git", :tag => "v0.5" }
# ――― Source Code ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# CocoaPods is smart about how it includes source code. For source files
# giving a folder will include any swift, h, m, mm, c & cpp files.
# For header files it will include any header in the folder.
# Not including the public_header_files will make all headers public.
#
s.source_files = "Sources"
#s.exclude_files = "Classes/Exclude"
# s.public_header_files = "Classes/**/*.h"
# ――― Resources ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# A list of resources included with the Pod. These are copied into the
# target bundle with a build phase script. Anything else will be cleaned.
# You can preserve files from being cleaned, please don't preserve
# non-essential files like tests, examples and documentation.
#
# s.resource = "icon.png"
# s.resources = "Resources/*.png"
# s.preserve_paths = "FilesToSave", "MoreFilesToSave"
# ――― Project Linking ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# Link your library with frameworks, or libraries. Libraries do not include
# the lib prefix of their name.
#
# s.framework = "SomeFramework"
# s.frameworks = "SomeFramework", "AnotherFramework"
# s.library = "iconv"
# s.libraries = "iconv", "xml2"
# ――― Project Settings ――――――――――――――――――――――――――――――――――――――――――――――――――――――――― #
#
# If your library depends on compiler flags you can set them in the xcconfig hash
# where they will only apply to your library. If you depend on other Podspecs
# you can include multiple dependencies to ensure it works.
# s.requires_arc = true
# s.xcconfig = { "HEADER_SEARCH_PATHS" => "$(SDKROOT)/usr/include/libxml2" }
# s.dependency "JSONKit", "~> 1.4"
end
next to my Podfile:
source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '8.0'
use_frameworks!
pod 'OrderedDictionary', :podspec => "OrderedDictionary.podspec"
I ran:
pod cache clean OrderedDictionary && pod install
which created a Pods/OrderedDictionary group under the Pods project.
Unfortunately the group is empty, so the project fails to build at:
import OrderedDictionary
error: No such module 'OrderedDictionary'
Linting passes:
pod spec lint OrderedDictionary.podspec --verbose
For the life of me I can't get OrderedDictionary.swift and OrderedDictionary.h to copy into the pod's group in the project.
The weird thing is that I did see OrderedDictionary.h appear in the group once but couldn't get it to happen again. I'm concerned that CocoaPods might not be idempotent, but could be overlooking something obvious. Thanks for any help you can provide.
Xcode 7.2.1 (7C1002), OS X 10.10.5 (14F27), CocoaPods 0.39.0
Answering my own question based on IvanRublev's comment. I tried my podspec in this IceCreamShop CocoaPods swift example and then after it worked there, it worked in my original project.
Reproduction steps:
After I wrote this question last night, I bit the bullet and downloaded the zip of the OrderedDictionary GitHub project, expanded it to the root level of my project directory, moved my podspec inside it, and changed my Podfile to say:
pod 'OrderedDictionary', :path => "OrderedDictionary"
That allowed me to use it as a development pod, and I saw OrderedDictionary.swift and OrderedDictionary.h copied into the pod's group.
Today after I got IceCreamShop to work, I removed the OrderedDictionary directory from my project directory, moved my podspec back to the root level of my project directory, and changed the Podfile back to:
pod 'OrderedDictionary', :podspec => "OrderedDictionary.podspec"
The pod now worked, even though I didn't change anything.
I believe this may be a caching issue in CocoaPods, causing it to be inconsistent for one of the following reasons:
Being nontransparent, where issues like the subproject failing to build or a process being terminated leave the cache in an inconsistent state that allow us to detect the presence of the cache by its malfunction.
Not being invalidated correctly, so once it found files in my project dir, it somehow remembered them and was able to include them after they were removed.
Loads and stores behaving differently depending on where they are performed, allowing side effects to affect future loads and stores.
Having temporal issues where loads and stores are nondeterministic because timeouts take priority over the source of truth.
A cosmic ray from a distant galaxy flipping a bit in the cache causing my wave state to collapse into the reality where I spent hours chasing a bug instead of the one where I never knew the bug existed.
I like CocoaPods and realize this is all a bit tongue in cheek, but caching issues often have a detrimental impact on productivity and I would recommend that all projects have an option to disable their cache as well as have it run an internal consistency check every time it's used.
Here are some commands that might help someone experiencing similar problems:
sudo rm -rf ~/Library/Caches/CocoaPods/
sudo rm -rf ~/.cocoapods/repos/master/
sudo rm -rf Pods/
pod install

Rails 3.1 - how can I tell if assets are precompiling on production?

Trying to get the hang of deploying a rails 3.1 App ...
Based on what I've read, I've put the following code in my deploy.rb:
before "deploy:symlink", "assets:precompile"
namespace :assets do
desc "Compile assets"
task :precompile, :roles => :app do
run "cd #{release_path} && rake RAILS_ENV=#{rails_env} assets:precompile"
end
end
But to tell you the truth, I can't notice any difference with or without it. Is there something I'm missing here?
EDIT* found the answer:
http://spreecommerce.com/blog
To pre-compile assets for production you would normally execute the following rake task (on the production server).
$ bundle exec rake assets:precompile
This would write all the assets to the public/assets directory while including an MD5 fingerprint in the filename for added caching benefits.
NOTE: In production all references to assets from views using image_tag, asset_path, javascript_include_tag, etc. will automatically include this fingerprint in the file name so the correct version will be served.
There is configuration to do, but it should be correctly set by default. Get in your config/application.rb and see if you find this:
if defined?(Bundler)
# If you precompile assets before deploying to production, use this line
Bundler.require(*Rails.groups(:assets => %w(development test)))
# If you want your assets lazily compiled in production, use this line
# Bundler.require(:default, :assets, Rails.env)
end
...
config.assets.enabled = true
You should also have those in your production.rb file:
# Compress JavaScripts and CSS
config.assets.compress = true
# Don't fallback to assets pipeline if a precompiled asset is missed
config.assets.compile = false
This should be set that way. Is it?