scons and ctags - tags

I am trying to get ctags to be automatically run when I do a scons. I have a custom builder based on this answer that allows me to run ctags from within my SConscript file. This works in as far as I get a tags file in the appropriate directory. However, since the builder runs within the root directory and not the subdirectory, I get the wrong paths (aka from the root, not the subdirectory) for all the files contains within the tags file. Apart from using sed to strip paths in the tags file, anyone can offer a suggestion as to how to make scons generate the right paths?
In effects, I would like to run the builder in a specific directory (aka the one where the SConscript is located, maybe passed as an option?) and not from the project root.

This is the builder code I ended up using. Notice that the ctags command has a --tag-relative=yes option to it.
# -*- coding: utf-8 -*-
import SCons.Builder
import SCons.Action
def complain_ctags(target, source, env):
print 'INFORMATION: ctags binary was not found (see above). Tags have not been built.'
def generate(env):
env['CTAGS'] = find_ctags(env)
if env['CTAGS'] != None:
#env['CTAGSCOM'] = 'cd $TARGET.dir; ctags -R .'
env['CTAGSCOM'] = '$CTAGS --tag-relative=yes -f $TARGET $SOURCES'
env['BUILDERS']['ctags'] = SCons.Builder.Builder(action=env['CTAGSCOM'])
else:
env['BUILDERS']['ctags'] = SCons.Builder.Builder(action=env.Action(complain_ctags))
def find_ctags(env):
b=env.WhereIs('ctags')
if b == None:
print 'Searching for ctags: not found. Tags will not be built'
else:
print 'Searching for ctags: ', b
return b
def exists(env):
if find_ctags(env) == None:
return 0
return 1
In my Sconscript file(s), I can now do:
Alias('tags', env.ctags(source=all_python_files, target='tags'))
The tags file now contains tags with path relative to the location of the tags file and not from the scons root directory.

Related

How to copy whole directories containing subdirectories to /boot (i.e bootfs) in Yocto while inheriting core-image class?

I have a directory which again contains subdirectories, which are built has part of other recipe and moved to DEPLOY_DIR_IMAGE using deploy bb class. So now I want to copy it to main image boot partition.
If it was a single file then appending required filename to IMAGE_EFI_BOOT_FILES variable, then yocto copies it to /boot. But same doesn't work for directories containing subdirectories please provide style to include even the subdirectories. Thank you
PS: I have tried appending IMAGE_EFI_BOOT_FILES += "parent_dir/*" didnt work.
It is obvious that IMAGE_EFI_BOOT_FILES is acting like the well known IMAGE_BOOT_FILES and other variables that are responsible for having the files necessary to be shipped in the boot partition. And that needs files and not directories.
So, if you do not need to specify all the files by hand, but instead you want to pass the directory, I suggest you use a python method to collect the files for you and append them to the variable.
See the following example I developed and tested:
def get_files(d, dir):
import os
dir_path = dir
if not os.path.exists(os.path.dirname(dir)):
dir_path = d.getVar(dir)
return ' '.join(f for f in os.listdir(d.getVar(dir)) if os.path.isfile(f))
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
The method will test if the argument is a real path then it will directly check for files, if not it will assume that it is a bitbake variable and it will get its content, so if DEPLOY_DIR_IMAGE is, for example, /home/user/dir, passing DEPLOY_DIR_IMAGE or /home/usr/dir will give the same result.
IMPORTANT
It is obvious also that IMAGE_EFI_BOOT_FILES is used in a .conf file such as local.conf or a custom machine configuration file. So adding that python function in .conf file will not work. I suggest creating a class for it and inherit it globally in your .conf file:
meta-custom/classes/utils.bbclass
local.conf:
INHERIT += "utils"
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
Try this and let me know in the comments.
EDIT
I have just realized that bitbake already imports os within python expressions expansions, so you can do it in one line without any need for a separate python function:
PATH = "/path/to/directory/" or
PATH = "A variable containing the path"
IMAGE_EFI_BOOT_FILES += "${#' '.join('%s' % f for f in os.listdir('${PATH}') if os.path.isfile(f))}"
Note: I am looking for Yocto built-in which can achieve solution for above mentioned , would like to share other way to resolve the functionality for community's benefit.
Add following in bb file if you are using one or refer to talel-belhadjsalem answer to use utils.bbclass.
def add_directory_bootfs(d, dirname, bootfs_dir):
file_list = list()
boot_files_list = list()
deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
for (dirpath, dirnames, filenames) in os.walk(os.path.join(deploy_dir_image, dirname)):
file_list += [os.path.join(dirpath, file) for file in filenames]
for file in file_list:
file_rel_path = os.path.relpath(file, os.path.join(deploy_dir_image, dirname))
boot_file_entry = os.path.join(dirname, file_rel_path) + ';' + os.path.join(bootfs_dir, dirname, file_rel_path)
boot_files_list.append(boot_file_entry)
return ' '.join(boot_files_list)
IMAGE_EFI_BOOT_FILES += "${#add_directory_bootfs(d, 'relative_path_to_dir_in_deploy_dir_image', 'EFI/BOOT')}"

Docopt and Classes

Ive gotten quite comfortable over the last year coding in python on an off, but I have stayed away from Classes (as in structuring my code in them) because I have not understood them.
I am now trying to get my head around what I need to change in my coding practices to take advantage of using Classes in all their glory.
I have been trying to use an example script I wrote and pipe that to a Class based version. Safe to say I am sucking bad and cant get my simple script to work. Im sure there are a myriad of this Im most likely doing incorrectly. I would really appreciate someone pointing them out to me.
I dont mind finger points and belly laughs too ^_^
Coder After (not working)
"""
Description:
This script is used to walk a directory and print out each filename and directory including the full path.
Author: Name
Usage:
DirLister.py (-d <directory>)
DirLister.py -h | --help
DirLister.py --version
Options:
-d <directory> The top level directory you want to list files and directories from.
-h --help Show this screen.
--version Show version.
"""
import os
from docopt import docopt
class walking:
def __init__(self, directory):
self.directory = arguments['-d']
def walk(self, directory):
for root, dirs, files in os.walk(self.directory):
for filename in files:
print os.path.join(root, filename)
if __name__ == '__main__':
arguments = docopt(__doc__, version= '1.0.0')
print arguments
if arguments['-d'] is None:
print __doc__
exit(0)
else:
walking.walk(directory)
Original Non-Class Based Code (working)
"""
Description:
This script is used to walk a directory and print out each filename and directory including the full path.
Author: Name
Usage:
DirLister.py (-d <directory>)
DirLister.py -h | --help
DirLister.py --version
Options:
-d <directory> The top level directory you want to list files and directories from.
-h --help Show this screen.
--version Show version.
"""
import os
from docopt import docopt
arguments = docopt(__doc__, version= '1.0.0')
def walk(dir):
for root, dirs, files in os.walk(dir):
for filename in files:
print os.path.join(root, filename)
if __name__ == '__main__':
if arguments['-d'] is None:
print __doc__
exit(0)
else:
walk(arguments['-d'])
You've forgotten to post the error you get (since you say it's not working).
But indeed there are several issues. First, I'd call the class Walking.
Then in your __init__ function, you try to access arguments which is neither a global variable nor an argument; you wanted to write:
def __init__(self, directory):
self.directory = directory
But you also need to create an instance of your class in you main:
walking = Walking(arguments['-d'])
That assumes that the name of the class is Walking instead of walking. I advise you to look at PEP8 for the naming conventions.
The general idea is that the class is the type of an object, but not the object itself*, so the class Walking: block is basically defining a new kind of objects. And then you can create objects that are instances of this class. It's the same when you create a list: mylist = list() (but there are also other ways for lists like mylist = [1, 2]).
*It happens that most things in Python are objects, including classes, but they have obviously other methods and they have another base class.

How to set the default encoding in a buildout script, or during virtualenv creation?

I have a Plone project which is created by a buildout script and needs a default encoding of utf-8. This is usually done in the sitecustomize.py file of the Python installation. Since there is a virtualenv, I'd like to generate this file automatically, to contain something like:
import sys
sys.setdefaultencoding('utf-8')
After generation I have two empty sitecustomize.py files - one in parts/instance/, and one in parts/buildout; but none of these seems to be used (I couldn't find them in sys.path).
I tried zopepy:
>>> from os.path import join, isfile
>>> from pprint import pprint
>>> import sys
>>> pprint([p for p in sys.path
... if isfile(join(p, 'sitecustomize.py'))
... ])
and found another one in my local lib/python2.7/site-packages/ directory which looks good; but it doesn't work:
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
This directory sits near the end of the sys.path, because I needed to add it by an extra-paths entry (to get another historical package).
Any pointers? Thank you!
System information: CentOS 7, Python 2.7.5
Edit:
I deleted them two empty sitecustomize.py files; now I have a default encoding of utf-8 in the zopepy session but still ascii in Plone; this surprises me, because I have in my buildout script:
[zopepy]
recipe=zc.recipe.egg
eggs = ${instance:eggs}
extra-paths = ${instance:extra-paths}
interpreter = zopepy
scripts = zopepy
To debug this, I created a little function which I added to my code, and which displays a little information about relevant modules in the sys.path:
import sys
from os.path import join, isdir, isfile
def sitecustomize_info():
plen = len(sys.path)
print '-' * 79
print 'sys.path has %(plen)d entries' % locals()
for tup in zip(range(1, plen+1), sys.path):
nr, dname = tup
if isdir(dname):
for fname in ('site.py', 'sitecustomize.py'):
if isfile(join(dname, fname)):
print '%(nr)4d. %(dname)s/%(fname)s' % locals()
spname = join(dname, 'site-packages', 'sitecustomize.py')
if isfile(spname):
print '%(nr)4d. %(spname)s' % locals()
else:
print '? %(dname)s is not a directory' % locals()
print '-' * 79
Output:
sys.path has 303 entries
8. /usr/lib64/python2.7/site-packages/sitecustomize.py
295. /opt/zope/instances/wnzkb/lib/python2.7/site-packages/sitecustomize.py
? /usr/lib64/python27.zip is not a directory
298. /usr/lib64/python2.7/site.py
? /usr/lib64/python2.7/lib-tk is not a directory
? /usr/lib64/python2.7/lib-old is not a directory
303. /usr/lib/python2.7/site-packages/sitecustomize.py
All sitecustomize.py files look the same (switching to utf-8), and I didn't tweak site.py (for now; if everything else fails, I might need to.)
If you really want/need to use the sitecustomize.py trick, you could include this part in your buildout:
[fixencode]
recipe = plone.recipe.command
stop-on-error = yes
update-command = ${fixencode:command}
command =
SITE_PACKAGES=$(${buildout:executable} -c \
'from distutils.sysconfig import get_python_lib;print(get_python_lib())')
cat > $SITE_PACKAGES/../sitecustomize.py << EOF
#!${buildout:executable} -S
import sys
sys.setdefaultencoding('utf-8')
EOF
It will be added into the site-packages folder from your virtualenv.
It looks like sitecustomize.py is not found anymore unless placed in the global lib directories (Discussion "deleting setdefaultencoding in site.py is evil" (2009), Tracker ticket "sitecustomize.py not found") - and this was made on purpose (!).
This is to prevent users from overriding the default encoding which might have been adjusted by some library. Libraries shouldn't do that, however.
Thus, whoever needs to set the default encoding is seduced to do it globally, which looks like a very silly idea to me. I'd consider it much more reasonable to set this in my virtual environment.
Unfortunately the sitecustomize.py modules in a virtualenv seem to be silently ignored; but it is possible to edit the local site.py. Here is a little sed script:
# vv-------1------vv vv---2---vv vv--------3------vv vv-----------4------------vv
sed --in-place -e 's,^\( encoding =\) \("ascii"\) \(# Default value\) \(set by _PyUnicode_Init()\),\1 "utf-8" \3 \2 \4,' lib/python2.7/site.py

Protobufs import from another directory

While trying to compile a proto file named UserOptions.proto which has an import named Account.proto using the below command
protoc --proto_path=/home/project_new1/account --java_out=/home/project_new1/source /home/project_new1/settings/Useroptions.proto
I get the following error :
/home/project_new1/settings/UserOpti‌​ons.proto: File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file.
PS: UserOptions.proto present in the directory /home/project_new1/settings
imports Account.proto present in the directory
/home/project_new1/account
Proto descriptor files:
UserOptions.proto
package settings;
import "Account.proto";
option java_outer_classname = "UserOptionsVOProto";
Account.proto
package account;
option java_outer_classname = "AccountVOProto";
message Object
{
optional string userId = 1;
optional string service = 2;
}
As the error message states, the file you pass on the command line needs to be in one of the --proto_paths. In your case, you have only specified one --proto_path of:
/home/project_new1/
But the file you're passing is:
/home/project_new1/settings/UserOpti‌ons.proto
Notice that the file is not in the account subdirectory; it's in settings instead.
You have two options:
(Not recommended) Pass a second --proto_path argument to add .../settings to the path.
(Recommended) Use the root of your source tree as the proto path. E.g.:
protoc --proto_path=/home/project_new1/ --java_out=/home/project_new1 /home/project_new1/settings/UserOpti‌ons.proto
In this case, to import Account.proto, you'll need to write:
import "acco‌​unt/Account.proto";
For those of us who want this really spelled out, here is an example where I have installed the protoc beta for gRPC using NuGet Packages Google.Protobuf, Grpc.Core and Grpc.Tools. My solution packages are one level above my Grpc directory (i.e. at BruTrader\packages). My .proto files are at BruTrader\Grpc\protos.
1. My .proto file:
syntax = "proto3";
import "timestamp.proto";
import "enums.proto";
package BruTrader.Grpc;
message DividendMessage {
double amount = 1;
google.protobuf.Timestamp dateUnix = 2;
}
2. my GenerateProto.bat file:
..\packages\Google.Protobuf.3.0.0-beta2\tools\protoc.exe -I..\Grpc\protos -I..\packages\Google.Protobuf.3.0.0-beta2\tools\google\protobuf --csharp_out=..\Grpc\Generated --grpc_out=..\Grpc\Generated --plugin=protoc-gen-grpc=..\packages\Grpc.Tools.0.13.0\tools\grpc_csharp_plugin.exe %1
3. my BuildProtos.bat
call GenerateProto ..\Grpc\protos\masterinstrument.proto
call GenerateProto .\protos\instrument.proto
etc.
4. BuildProtos.bat is executed as a Pre-build event on my Grpc project like this:
CD $(ProjectDir)
CALL "$(ProjectDir)BuildProtos.bat"
For my environment, Windows 10 Pro operating system and C++ programming languaje, I used the protoc-3.12.2-win64.zip that you can downloat it from here. You should open a Windows PowerShell inside the protoc-3.12.2-win64\bin path and then you must execute one of the next commands:
.\protoc.exe -I=C:\Users\UserName\Desktop\SRC --cpp_out=C:\Users\UserName\Desktop\DST C:\Users\UserName\Desktop\SRC\addressbook.proto
Or
.\protoc.exe --proto_path=C:\Users\UserName\Desktop\SRC --cpp_out=C:\Users\UserName\Desktop\DST C:\Users\UserName\Desktop\SRC\addressbook.proto
Note:
1- My source folder is in: C:\Users\UserName\Desktop\SRC
2- My destination folder is in: C:\Users\UserName\Desktop\DST
3- My .proto file is in: C:\Users\UserName\Desktop\SRC\addressbook.proto

copy task in Cakefile

I am trying to copy all the files in a list of directories and paste them into an output directory. The problem is whenever I use an *, the output says there is no file or directory by that name exists. Here is the specific error output:
cp: cannot stat `tagbox/images/*': No such file or directory
cp: cannot stat `votebox/images/*': No such file or directory
If I just put the name of a specific file instead of *, it works.
here is my Cakefile:
fs = require 'fs'
util = require 'util'
{spawn} = require 'child_process'
outputImageFolder = 'static'
imageSrcFolders = [
'tagbox/images/*'
'votebox/images/*'
]
task 'cpimgs', 'Copy all images from the respective images folders in tagbox, votebox, and omnipost into static folder', ->
for imgSrcFolder in imageSrcFolders
cp = spawn 'cp', [imgSrcFolder, outputImageFolder]
cp.stderr.on 'data', (data) ->
process.stderr.write data.toString()
cp.stdout.on 'data', (data) ->
util.log data.toString()
You are using the * character, probably because that works for you in your shell. Using * and other wildcard characters that expand to match multiple paths is called "globbing" and while your shell does it automatically, most other programs including node/javascript/coffeescript will not do it by default. Also the cp binary itself doesn't do globbing, as you are discovering. The shell does the globbing and then passes a list of matching files/directories as arguments to cp. Look into the node module node-glob to do the globbing and give you back a list of matching files/directories, which you can then pass to cp as arguments if you like. Note that you could also use a filesystem module that would have this type of functionality built in. Note however that putting async code directly into a Cakefile can be problematic as documented here.