copy task in Cakefile - copy

I am trying to copy all the files in a list of directories and paste them into an output directory. The problem is whenever I use an *, the output says there is no file or directory by that name exists. Here is the specific error output:
cp: cannot stat `tagbox/images/*': No such file or directory
cp: cannot stat `votebox/images/*': No such file or directory
If I just put the name of a specific file instead of *, it works.
here is my Cakefile:
fs = require 'fs'
util = require 'util'
{spawn} = require 'child_process'
outputImageFolder = 'static'
imageSrcFolders = [
'tagbox/images/*'
'votebox/images/*'
]
task 'cpimgs', 'Copy all images from the respective images folders in tagbox, votebox, and omnipost into static folder', ->
for imgSrcFolder in imageSrcFolders
cp = spawn 'cp', [imgSrcFolder, outputImageFolder]
cp.stderr.on 'data', (data) ->
process.stderr.write data.toString()
cp.stdout.on 'data', (data) ->
util.log data.toString()

You are using the * character, probably because that works for you in your shell. Using * and other wildcard characters that expand to match multiple paths is called "globbing" and while your shell does it automatically, most other programs including node/javascript/coffeescript will not do it by default. Also the cp binary itself doesn't do globbing, as you are discovering. The shell does the globbing and then passes a list of matching files/directories as arguments to cp. Look into the node module node-glob to do the globbing and give you back a list of matching files/directories, which you can then pass to cp as arguments if you like. Note that you could also use a filesystem module that would have this type of functionality built in. Note however that putting async code directly into a Cakefile can be problematic as documented here.

Related

How to copy whole directories containing subdirectories to /boot (i.e bootfs) in Yocto while inheriting core-image class?

I have a directory which again contains subdirectories, which are built has part of other recipe and moved to DEPLOY_DIR_IMAGE using deploy bb class. So now I want to copy it to main image boot partition.
If it was a single file then appending required filename to IMAGE_EFI_BOOT_FILES variable, then yocto copies it to /boot. But same doesn't work for directories containing subdirectories please provide style to include even the subdirectories. Thank you
PS: I have tried appending IMAGE_EFI_BOOT_FILES += "parent_dir/*" didnt work.
It is obvious that IMAGE_EFI_BOOT_FILES is acting like the well known IMAGE_BOOT_FILES and other variables that are responsible for having the files necessary to be shipped in the boot partition. And that needs files and not directories.
So, if you do not need to specify all the files by hand, but instead you want to pass the directory, I suggest you use a python method to collect the files for you and append them to the variable.
See the following example I developed and tested:
def get_files(d, dir):
import os
dir_path = dir
if not os.path.exists(os.path.dirname(dir)):
dir_path = d.getVar(dir)
return ' '.join(f for f in os.listdir(d.getVar(dir)) if os.path.isfile(f))
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
The method will test if the argument is a real path then it will directly check for files, if not it will assume that it is a bitbake variable and it will get its content, so if DEPLOY_DIR_IMAGE is, for example, /home/user/dir, passing DEPLOY_DIR_IMAGE or /home/usr/dir will give the same result.
IMPORTANT
It is obvious also that IMAGE_EFI_BOOT_FILES is used in a .conf file such as local.conf or a custom machine configuration file. So adding that python function in .conf file will not work. I suggest creating a class for it and inherit it globally in your .conf file:
meta-custom/classes/utils.bbclass
local.conf:
INHERIT += "utils"
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
Try this and let me know in the comments.
EDIT
I have just realized that bitbake already imports os within python expressions expansions, so you can do it in one line without any need for a separate python function:
PATH = "/path/to/directory/" or
PATH = "A variable containing the path"
IMAGE_EFI_BOOT_FILES += "${#' '.join('%s' % f for f in os.listdir('${PATH}') if os.path.isfile(f))}"
Note: I am looking for Yocto built-in which can achieve solution for above mentioned , would like to share other way to resolve the functionality for community's benefit.
Add following in bb file if you are using one or refer to talel-belhadjsalem answer to use utils.bbclass.
def add_directory_bootfs(d, dirname, bootfs_dir):
file_list = list()
boot_files_list = list()
deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
for (dirpath, dirnames, filenames) in os.walk(os.path.join(deploy_dir_image, dirname)):
file_list += [os.path.join(dirpath, file) for file in filenames]
for file in file_list:
file_rel_path = os.path.relpath(file, os.path.join(deploy_dir_image, dirname))
boot_file_entry = os.path.join(dirname, file_rel_path) + ';' + os.path.join(bootfs_dir, dirname, file_rel_path)
boot_files_list.append(boot_file_entry)
return ' '.join(boot_files_list)
IMAGE_EFI_BOOT_FILES += "${#add_directory_bootfs(d, 'relative_path_to_dir_in_deploy_dir_image', 'EFI/BOOT')}"

How to copy specific headers in Soong

How can I extract specific files in Soong to use as headers?
I was recently writing a blueprint (Android.bp) file for paho-mqtt-c. My consumers require MQTTClient.h which paho-mqtt-c stores in src/ - what I would consider a "private" location. Reading their CMakeLists.txt file, it actually installs this and some other headers to include/.
As far as I can tell, Soong doesn't have this concept of installing so it seems like I could export_include_dirs the src directory - which seems wrong, or use a cc_genrule to copy these headers elsewhere.
But that's where I hit another issue: I can't seem to figure out how to create a cc_genrule that takes n inputs and writes n outputs (n-to-n). i.e.
cc_genrule {
name: "paho_public_headers",
cmd: "cp $(in) $(out)",
srcs: [ "src/MQTTAsync.h", "src/MQTTClient.h", "src/MQTTClientPersistence.h", "src/MQTTLogLevels.h" ]
out: [ "public/MQTTAsync.h", "public/MQTTClient.h", "public/MQTTClientPersistence.h", "public/MQTTLogLevels.h" ],
}
results in the failed command cp <all-inputs> <all-outputs>, rather than what I wanted which would be closer to iterating the command over each input/output pairs.
My solution was simply to write four cc_genrules, but that doesn't seem great either.
Is there a better way? (ideally without writing a custom tool)
The solution was to use gensrcs with a shard_size of 1.
For example:
gensrcs {
name: "paho_public_headers",
cmd: "mkdir -p $(genDir) && cat $(in) > $(out)",
srcs: [":paho_mqtt_c_header_files"],
output_extension: "h",
shard_size: 1,
export_include_dirs: ["src"],
}
The export_include_dirs is important, as with gensrcs there is little control over the output filename other than the extension, so it's easiest to keep the directory structure $(in) on the $(out) files.

how to get ${THISDIR} inside do_unpack_append in .bbappend file

I'm attempting to replace a file from another layer with a .bbappend file. My goal is to overwrite a specific configuration file with a customized one during the unpack stage.
In my .bbappend I'm attempting to append the do_unpack to copy a file from the same directory as the .bbappend file into the working directory ${WORKDIR} The problem is: When inside do_unpack_append, ${THISDIR} is returning the directory of the original .bb recipe, rather than the directory of .bbappend
Here's an example:
The original recipe resides in: meta-origLayer/recipe.bb
My *.bbappend resides in: meta-newLayer/recipe.bbappend
recipe.bbappend:
`FILESEXTRAPATHS_prepend := "${THISDIR}:"`
do_unpack_append(){
bb.build.exec_func('replace_file', d)
}
replace_file(){
cp -f ${THISDIR}/fileToBeReplaced ${WORKDIR}/fileToBeReplaced
echo ${THISDIR} > ${WORKDIR}/shouldContain_meta-newLayer
}
There are two issues with recipe.bbappend:
I would expect the file shouldContain_meta-newLayer to contain meta-newLayer, but instead it contains meta-origLayer.
I'd primarily like to understand why ${THISDIR} behaves differently when placed inside do_unpack_append() from when it is used for prepending FILESEXTRAPATHS
When running bitbake, the recipe fails, producing the following error:
cp: cannot stat '/fileToBeReplaced': No such file or directory
This error occurs because fileToBeReplaced resides in a subdirectory of meta-origLayer (i.e. meta-origLayer/machine1/fileToBeReplaced) and the .bbappend expects to find the file in /fileToBeReplaced
My Question. . .
I have assumed ${THISDIR} would behave consistently within the same .bbappend, but it doesn't appear to. What is the best way to reference meta-newLayer/fileToBeReplaced from within do_unpack_append()?
This *.bbappend correctly overwrites fileToBeReplaced in the working directory during the unpack task:
FILESEXTRAPATHS_prepend := "${THISDIR}:"
SRC_URI_append += " file://fileToBeReplaced "
SAVED_DIR := "${THISDIR}"
do_unpack_append(){
bb.build.exec_func('replace_file', d)
}
replace_file(){
cp -f ${SAVED_DIR}/fileToBeReplaced ${WORKDIR}/fileToBeReplaced
}
Thanks for the explanation between bbappend parsing and execution johannes-schaub-ltb

Copy all files with given extension to output directory using CMake

I've seen that I can use this command in order to copy a directory using cmake:
file(COPY "myDir" DESTINATION "myDestination")
(from this post)
My problem is that I don't want to copy all of myDir, but only the .h files that are in there. I've tried with
file(COPY "myDir/*.h" DESTINATION "myDestination")
but I obtain the following error:
CMake Error at CMakeLists.txt:23 (file):
file COPY cannot find
"/full/path/to/myDIR/*.h".
How can I filter the files that I want to copy to a destination folder?
I've found the solution by myself:
file(GLOB MY_PUBLIC_HEADERS
"myDir/*.h"
)
file(COPY ${MY_PUBLIC_HEADERS} DESTINATION myDestination)
this also works for me:
install(DIRECTORY "myDir/"
DESTINATION "myDestination"
FILES_MATCHING PATTERN "*.h" )
The alternative approach provided by jepessen does not take into account the fact that sometimes the number of files to be copied is too high. I encountered the issue when doing such thing (more than 110 files)
Due to a limitation on Windows on the number of characters (2047 or 8191) in a single command line, this approach may randomly fail depending on the number of headers that are in the folder. More info here https://support.microsoft.com/en-gb/help/830473/command-prompt-cmd-exe-command-line-string-limitation
Here is my solution:
file(GLOB MY_HEADERS myDir/*.h)
foreach(CurrentHeaderFile IN LISTS MY_HEADERS)
add_custom_command(
TARGET MyTarget PRE_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CurrentHeaderFile} ${myDestination}
COMMENT "Copying header: ${CurrentHeaderFile}")
endforeach()
This works like a charm on MacOS. However, if you have another target that depends on MyTarget and needs to use these headers, you may have some compile errors due to not found includes on Windows. Therefore you may want to prefer the following option that defines an intermediate target.
function (CopyFile ORIGINAL_TARGET FILE_PATH COPY_OUTPUT_DIRECTORY)
# Copy to the disk at build time so that when the header file changes, it is detected by the build system.
set(input ${FILE_PATH})
get_filename_component(file_name ${FILE_PATH} NAME)
set(output ${COPY_OUTPUT_DIRECTORY}/${file_name})
set(copyTarget ${ORIGINAL_TARGET}-${file_name})
add_custom_target(${copyTarget} DEPENDS ${output})
add_dependencies(${ORIGINAL_TARGET} ${copyTarget})
add_custom_command(
DEPENDS ${input}
OUTPUT ${output}
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${input} ${output}
COMMENT "Copying file to ${output}."
)
endfunction ()
foreach(HeaderFile IN LISTS MY_HEADERS)
CopyFile(MyTarget ${HeaderFile} ${myDestination})
endforeach()
The downside indeed is that you end up with multiple target (one per copied file) but they should all end up together (alphabetically) since they start with the same prefix ORIGINAL_TARGET -> "MyTarget"

Protobufs import from another directory

While trying to compile a proto file named UserOptions.proto which has an import named Account.proto using the below command
protoc --proto_path=/home/project_new1/account --java_out=/home/project_new1/source /home/project_new1/settings/Useroptions.proto
I get the following error :
/home/project_new1/settings/UserOpti‌​ons.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file.
PS: UserOptions.proto present in the directory /home/project_new1/settings
imports Account.proto present in the directory
/home/project_new1/account
Proto descriptor files:
UserOptions.proto
package settings;
import "Account.proto";
option java_outer_classname = "UserOptionsVOProto";
Account.proto
package account;
option java_outer_classname = "AccountVOProto";
message Object
{
optional string userId = 1;
optional string service = 2;
}
As the error message states, the file you pass on the command line needs to be in one of the --proto_paths. In your case, you have only specified one --proto_path of:
/home/project_new1/
But the file you're passing is:
/home/project_new1/settings/UserOpti‌ons.proto
Notice that the file is not in the account subdirectory; it's in settings instead.
You have two options:
(Not recommended) Pass a second --proto_path argument to add .../settings to the path.
(Recommended) Use the root of your source tree as the proto path. E.g.:
protoc --proto_path=/home/project_new1/ --java_out=/home/project_new1 /home/project_new1/settings/UserOpti‌ons.proto
In this case, to import Account.proto, you'll need to write:
import "acco‌​unt/Account.proto";
For those of us who want this really spelled out, here is an example where I have installed the protoc beta for gRPC using NuGet Packages Google.Protobuf, Grpc.Core and Grpc.Tools. My solution packages are one level above my Grpc directory (i.e. at BruTrader\packages). My .proto files are at BruTrader\Grpc\protos.
1. My .proto file:
syntax = "proto3";
import "timestamp.proto";
import "enums.proto";
package BruTrader.Grpc;
message DividendMessage {
double amount = 1;
google.protobuf.Timestamp dateUnix = 2;
}
2. my GenerateProto.bat file:
..\packages\Google.Protobuf.3.0.0-beta2\tools\protoc.exe -I..\Grpc\protos -I..\packages\Google.Protobuf.3.0.0-beta2\tools\google\protobuf --csharp_out=..\Grpc\Generated --grpc_out=..\Grpc\Generated --plugin=protoc-gen-grpc=..\packages\Grpc.Tools.0.13.0\tools\grpc_csharp_plugin.exe %1
3. my BuildProtos.bat
call GenerateProto ..\Grpc\protos\masterinstrument.proto
call GenerateProto .\protos\instrument.proto
etc.
4. BuildProtos.bat is executed as a Pre-build event on my Grpc project like this:
CD $(ProjectDir)
CALL "$(ProjectDir)BuildProtos.bat"
For my environment, Windows 10 Pro operating system and C++ programming languaje, I used the protoc-3.12.2-win64.zip that you can downloat it from here. You should open a Windows PowerShell inside the protoc-3.12.2-win64\bin path and then you must execute one of the next commands:
.\protoc.exe -I=C:\Users\UserName\Desktop\SRC --cpp_out=C:\Users\UserName\Desktop\DST C:\Users\UserName\Desktop\SRC\addressbook.proto
Or
.\protoc.exe --proto_path=C:\Users\UserName\Desktop\SRC --cpp_out=C:\Users\UserName\Desktop\DST C:\Users\UserName\Desktop\SRC\addressbook.proto
Note:
1- My source folder is in: C:\Users\UserName\Desktop\SRC
2- My destination folder is in: C:\Users\UserName\Desktop\DST
3- My .proto file is in: C:\Users\UserName\Desktop\SRC\addressbook.proto