Yocto - git revision in the image name - yocto

By default Yocto adds build timestamp to the output image file name, but I would like to replace it by the revision of my integration Git repository (which references all my layers and configuration files). To achieve this, I put the following code to my image recipe:
def get_image_version(d):
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
return subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
except:
return d.getVar("MACHINE", True) + "-" + d.getVar("DATETIME", True)
IMAGE_VERSION = "${#get_image_version(d)}"
IMAGE_NAME = "${IMAGE_BASENAME}-${IMAGE_VERSION}"
IMAGE_NAME[vardepsexclude] = "IMAGE_VERSION"
This code works properly until I change Git revision (e.g. by adding a new commit). Then I receive the following error:
ERROR: When reparsing /home/ubuntu/yocto/poky/../mylayer/recipes-custom/images/core-image-minimal.bb.do_image_tar, the basehash value changed from 63e1e69797d2813a4c36297517478a28 to 9788d4bf2950a23d0f758e4508b0a894. The metadata is not deterministic and this needs to be fixed.
I understand this happens because the image recipe has already been parsed with older Git revision, but why constant changes of the build timestamp do not cause the same error? How can I fix my code to overcome this problem?

The timestamp does not have this effect since its added to vardepsexclude:
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#variable-flags
[vardepsexclude]: Specifies a space-separated list of variables that should be excluded from a variable’s dependencies for the purposes of calculating its signature.
You may need to add this in a couple of places, e.g.:
https://git.yoctoproject.org/poky/tree/meta/classes/image-artifact-names.bbclass#n7
IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"

After some research it turned out the problem was in this line
IMAGE_VERSION = "${#get_image_version(d)}"
because the function get_image_version() was called during parsing. I took inspiration from the source file in aehs29's post and moved the code to the anonymous Python function which is called after parsing.
I also had to add vardepsexclude attribute to the IMAGE_NAME variable. I tried to add vardepvalue flag to IMAGE_VERSION variable as well and in this particular case it did the same job as vardepsexclude. Mentioned Bitbake class uses both flags, but I think in my case using only one of them is enough.
The final code is below:
IMAGE_VERSION ?= "${MACHINE}-${DATETIME}"
IMAGE_NAME = "${IMAGE_BASENAME}-${IMAGE_VERSION}"
IMAGE_NAME[vardepsexclude] += "IMAGE_VERSION"
python () {
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
version = subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
d.setVar("IMAGE_VERSION", version)
except:
bb.warning("Could not get Git revision, image will have default name.")
}
EDIT:
After some research I realized it's better to define a global variable in layer.conf file of the layer containing the recipes referencing the variable. The variable is set by a python script and is immediately expanded to prevent deterministic build warning:
layer.conf:
require conf/image-version.py.inc
IMAGE_VERSION := "${#get_image_version(d)}"
image-version.py.inc:
def get_image_version(d):
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
return subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
except:
bb.warn("Could not determine image version. Default naming schema will be used.")
return d.getVar("MACHINE", True) + "-" + d.getVar("DATETIME", True)
I think this is cleaner approach which fits BitBake build system better.

Related

Which property does the visio.Masters.MatchByName method refer to?

The Master of vsoDoc can integrate Masters with the same name at drop time by using the MatchByName method.
However, it may not work well in some cases, and we are investigating the cause.
Which property does MatchByName refer to to integrate Master? Is it Master.Name? Or is it Master.NameU?
If you divide the case
The copy destination is Master (vsoDoc.Masters) of the visio file (A.vsdx).
There can be two cases for the copy source.
One is to copy the Shapes of the visio file (B.vsdx)
Or drop it from the vssx file (X.vssx) to A.vsdx.
When copying Shapes, do you refer to the Name or NameU of each Master?
Since the ReplaceShame method changes the name of Master,
I am dealing with the new Master and the old Master by swapping the names with the following procedure.
First, in preparation for ReplaceShape, change the MatchByName of all vsoMasters to False.
Then "Replace Shape" all vsoShape
For Each vsoMaster in vsoMasters
vsoMaster.MatchByName = False
Next
Remove the replaced Master from Masters
For Each vsoPage In vsoDoc.Pages
For Each vsoShape in vsoPage.Shapes
If Not (vsoShape.Master is Nothing) then
vsoShape.Master.MatchByName = False
Set vsoShape = vsoShape.ReplaceShape(vssxMasters.ItemU(vsoShape.Master.NameU), 1)
end if
Next
Next
Remove the replaced Master from Masters
For Each vsoMaster in vsoMasters
If InStr(vsoMaster.Name, ".") == 0 then
vsoMasters.ItemU(vsoMaster.NameU).Delete
end if
Next
Change MatchByName of all vsoMasters to True
For Each vsoMaster In vsoMasters
vsoMaster.MatchByName = True
Next
For Each vsoPage In vsoDoc.Pages
For Each vsoShape in vsoPage.Shapes
vsoShape.Master.MatchByName = True
Next
Next
vsoDoc.SaveAs NewName
############################################################
I examined the phenomenon related to MatchByName in three cases.
MatchByName = True in all of the following cases.
Case A-1
When using Master (NameU: "test") of the same stencil file (M.vssx)
Procedure
Place the "test" stencil in the A.vsdx file.
Copy the A.vsdx file and create the A-1_copy.vsdx file
Copy the "test" stencil in which A_copy.vsdx, and pasting it into the A.vsdx file.
Result
As expected, MatchByName works fine and the document stencil Masters doesn't increase.
Case B
Create a Master (NameU: "test") in the stencil file (M-1.vssx)
Create another Master (NameU: "test") in another stencil file (M-2.vssx)
In this case B, the Master Name U is same, but the Base ID and Unique ID are different. I think this is an important point.
Case B-1
Procedure
Place the "test" stencil of the M-1.vssx in the B.vsdx file.
Place the "test" stencil of the M-2.vssx in the C.vsdx file.
Copy the "test" stencil from the C.vsdx file and paste it into the B.vsdx file.
MatchByName doesn't work. "test.10" is added to the document stencil.
I suspect that this is because the stencil's BaseID and UniqueID are different.
Case B-2
Procedure
Place the M-1.vssx "test" stencil in the B.vsdx file.
Launch another VISIO application and drop the "test" stencil of M-2.vssx into the B.vsdx file.
Result
Unlike the B-1 result, MatchByName WORKS.
 I can not understand the cause.
The following code is the code to rewrite the XML file in the vsdx file. By rewriting the Unique ID of the Masters of the vsdx file and the vssx file, MatchByName can be operated as expected.
import zipfile
from pathlib import Path
import xml.etree.ElementTree as ET
import sys
import shutil
import os
# https://yuukou-exp.plus/handle-xlsx-with-python-intro/
# https://livingdead0812.hatenablog.com/entry/2019/07/18/183322
# https://pg-chain.com/python-xml-read-write
# https://note.com/you_memolog/n/ne7d9e8d8a0d3
# https://stackoverflow.com/questions/16721010/write-a-xssfworkbook-to-a-zip-file
#visiopath = Path(sys.argv[1])
visiopath = Path(r"Flowchart_v1.0.17.vssx")
folderpath = visiopath.parent
xml_path = None
with zipfile.ZipFile(visiopath) as zf:
for info in zf.infolist():
if "masters.xml" in info.filename:
xml_path = info.filename
archive = zipfile.ZipFile(visiopath)
#archive = zipfile.ZipFile(visiopath, "w")
print(f"xml_path {xml_path}")
NameU_UniqueID_dict = {'OPERATOR' : '{00FBF49C-0014-0000-8E40-00608CF305B2}',
'TERMINATOR' : '{00634DFF-0015-0000-8E40-00608CF305B2}',
'BRANCH' : '{07639B38-0034-0000-8E40-00608CF305B2}',
'SQUARE' : '{07622477-0012-0000-8E40-00608CF305B2}',
'STARTER' : '{00489A6D-0032-0000-8E40-00608CF305B2}',
'CONNECTOR' : '{006BE21A-0004-0000-8E40-00608CF305B2}',
}
if xml_path is not None:
tree = ET.parse(archive.open(xml_path))
root = tree.getroot()
for child in root:
if child.attrib['NameU'] in NameU_UniqueID_dict.keys():
UniqueID = NameU_UniqueID_dict[child.attrib['NameU']]
child.attrib['UniqueID'] = UniqueID
tree.write('masters.xml')
print('finished writing.')
with zipfile.ZipFile(visiopath, 'a') as zf:
zf.write('masters.xml', arcname='visio\masters\masters.xml')
del archive
zippath = visiopath.with_suffix('.zip')
print(zippath)
print(visiopath)
shutil.move(visiopath, zippath)
shutil.unpack_archive(zippath, 'dir_out')
os.remove(zippath)
shutil.make_archive(zippath.parent / zippath.stem, format='zip', root_dir='dir_out')
#shutil.make_archive(zippath, format='zip', root_dir='dir_out')
shutil.move(zippath, visiopath)
When copying Shapes, do you refer to the Name or NameU of each Master?
NameU
When working with Visio objects, developers should always use universal names to reference their objects. This advice applies to more than names. A number of properties and methods in the Visio API have universal syntax equivalents. If you find a property and notice the same property exists but ending in a ‘U’, use the universal property. A classic example is to use the FormulaU property to get and set formulas instead of Formula.
Read more about Name and NameU
when you use ReplaceShape method in Document stecnil created new master, this new master have different Name and NameU anyway !

subprocess.run managing optional stdin and stdout

In python >= 3.5 we can give optional stdout, stdin, stderr to subprocess.run()
per the docs:
Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer),
an existing file object, and None. PIPE indicates that a new pipe to the child
should be created
I want to support passing through (at least) None or existing file objects whilst managing resources pythonically.
How should I manage the optional file resources in something like:
import subprocess
def wraps_subprocess(args=['ls', '-l'], stdin=None, stdout=None):
# ... do important stuff
subprocess.run(args=args, stdin=stdin, stdout=stdout)
A custom contextmanager (idea taken from this answer) seems to work:
import contextlib
#contextlib.contextmanager
def awesome_open(path_or_file_or_none, mode='rb'):
if isinstance(path_or_file_or_none, str):
file_ = needs_closed = open(path_or_file_or_none, mode)
else:
file_ = path_or_file_or_none
needs_closed = None
try:
yield file_
finally:
if needs_closed:
needs_closed.close()
which would be used used like
import subprocess
def wraps_subprocess(args=['ls', '-l'], stdin=None, stdout=None):
# ... do important stuff
with awesome_open(stdin, mode='rb') as fin, awesome_open(stdout, mode='wb') as fout:
subprocess.run(args=args, stdin=fin, stdout=fout)
But I think there is probably a better way.

Sed and awk application

I've read a little about sed and awk, and understand that both are text manipulators.
I plan to use one of these to edit groups of files (code in some programming language, js, python etc.) to make similar changes to large sets of files.
Primarily editing function definitions (parameters passed) and variable names for now, but the more I can do the better.
I'd like to know if someone's attempted something similar, and those who have, are there any obvious pitfalls that one should look out for? And which of sed and awk would be preferable/more suitable for such an application. (Or maybe something entirely else? )
Input
function(paramOne){
//Some code here
var variableOne = new ObjectType;
array[1] = "Some String";
instanceObj = new Something.something;
}
Output
function(ParamterOne){
//Some code here
var PartOfSomething.variableOne = new ObjectType;
sArray[1] = "Some String";
var instanceObj = new Something.something
}
Here's a GNU awk (for "gensub()" function) script that will transform your sample input file into your desired output file:
$ cat tst.awk
BEGIN{ sym = "[[:alnum:]_]+" }
{
$0 = gensub("^(" sym ")[(](" sym ")[)](.*)","\\1(ParameterOne)\\3","")
$0 = gensub("^(var )(" sym ")(.*)","\\1PartOfSomething.\\2\\3","")
$0 = gensub("^a(rray.*)","sA\\1","")
$0 = gensub("^(" sym " =.*)","var \\1","")
print
}
$ cat file
function(paramOne){
//Some code here
var variableOne = new ObjectType;
array[1] = "Some String";
instanceObj = new Something.something;
}
$ gawk -f tst.awk file
function(ParameterOne){
//Some code here
var PartOfSomething.variableOne = new ObjectType;
sArray[1] = "Some String";
var instanceObj = new Something.something;
}
BUT think about how your real input could vary from that - you could have more/less/different spacing between symbols. You could have assignments starting on one line and finishing on the next. You could have comments that contain similar-looking lines to the code that you don't want changed. You could have multiple statements on one line. etc., etc.
You can address every issue one at a time but it could take you a lot longer than just updating your files and chances are you still will not be able to get it completely right.
If your code is EXCEEDINGLY well structured and RIGOROUSLY follows a specific, highly restrictive coding format then you might be able to do what you want with a scripting language but your best bets are either:
change the files by hand if there's less than, say, 10,000 of them or
get a hold of a parser (e.g. the compiler) for the language your files are written in and modify that to spit out your updated code.
As soon as it starts to get slightly more complicated you will switch to a script language anyway. So why not start with python in the first place?
Walking directories:
walking along and processing files in directory in python
Replacing text in a file:
replacing text in a file with Python
Python regex howto:
http://docs.python.org/dev/howto/regex.html
I also recommend to install Eclipse + PyDev as this will make debugging a lot easier.
Here is an example of a simple automatic replacer
import os;
import sys;
import re;
import itertools;
folder = r"C:\Workspaces\Test\";
skip_extensions = ['.gif', '.png', '.jpg', '.mp4', ''];
substitutions = [("Test.Alpha.", "test.alpha."),
("Test.Beta.", "test.beta."),
("Test.Gamma.", "test.gamma.")];
for root, dirs, files in os.walk(folder):
for name in files:
(base, ext) = os.path.splitext(name);
file_path = os.path.join(root, name);
if ext in skip_extensions:
print "skipping", file_path;
else:
print "processing", file_path;
with open(file_path) as f:
s = f.read();
before = [[s[found.start()-5:found.end()+5] for found in re.finditer(old, s)] for old, new in substitutions];
for old, new in substitutions:
s = s.replace(old, new);
after = [[s[found.start()-5:found.end()+5] for found in re.finditer(new, s)] for old, new in substitutions];
for b, a in zip(itertools.chain(*before), itertools.chain(*after)):
print b, "-->", a;
with open(file_path, "w") as f:
f.write(s);

Resource Not Found Exception in pyglet

I'm using Python 2.6.6 and pyglet 1.1.4. In my "Erosion" folder, I have "Erosion.py" and a folder named "Images." Inside images, there are .png images. One image is named "Guard.png."
In "Erosion.py" there is a segment that goes like so:
pyglet.resource.path = ['Images']
pyglet.resource.reindex()
self.image = pyglet.resource.image('%s%s' % (character, '.png'))
When I run this, I am given
File "C:\Python26\lib\site-packages\pyglet\resource.py", line 394, in file raise ResourceNotFoundException(name)
ResourceNotFoundException: Resource "Guard.png" was not found on the path. Ensure that the filename has the correct captialisation.
I have tried changing the path to ['./Images'] and ['../Images']. I've also tried removing the path and the reindex call and putting Erosion.py and Guard.png in the same folder.
This is what i do to be able to load resources:
pyglet.resource.path = ['C:\\Users\\myname\\Downloads\\temp']
pyglet.resource.reindex()
pic = pyglet.resource.image('1.jpg')
texture = pic.get_texture()
gl.glTexParameteri(gl.GL_TEXTURE_2D, gl.GL_TEXTURE_MAG_FILTER, gl.GL_NEAREST)
texture.width = 960
texture.height = 720
texture.blit(0, 0, 0)
try this
pyglet.image.load('path/to/image.png')
If the relative path is not working you can try with the absolute path using the os module.
import pyglet
import os
working_dir = os.path.dirname(os.path.realpath(__file__))
pyglet.resource.path = [os.path.join(working_dir,'Images')]
pyglet.resource.reindex()
image = pyglet.resource.image('character.png'))
It's better to use the os.path.join method instead of writing it as a string for a better cross-platform support.
I get a problem like this using pyglet and pyscripter.(the text editor) In order for the file to be found I have to restart the editor before running the program.
This might be a problem with pyscripter however.
It is worth looking into: http://pyglet.readthedocs.io/en/pyglet-1.3-maintenance/programming_guide/resources.html
You should include all the directories into:
pyglet.resource.path = [...]
Then pass the ONLY the file name when you call:
pyglet.resource.image("ball_color.jpeg")
Try:
import os
import pyglet
working_dir = os.path.dirname(os.path.realpath(__file__))
image_dir = os.path.join(working_dir, 'images')
sound_dir = os.path.join(working_dir, 'sound')
print working_dir
print "Images are in: " + image_dir
print "Sounds are in: " + sound_dir
pyglet.resource.path = [working_dir, image_dir, sound_dir]
pyglet.resource.reindex()
try:
window = pyglet.window.Window()
image = pyglet.resource.image("ball_color.jpeg")
#window.event
def on_draw():
window.clear()
image.blit(0, 0)
pyglet.app.run()
finally:
window.close()
The use of try and finally are not necessary, but they are recommended. You might end up with a lot of open windows and pyglet back processes if you don't close the window when there are errors.
pyglet 1.4.4 format: HUD_img = pyglet.resource.image('ui\ui_gray_block_filled.png')
pyglet 1.4.8 format: HUD_img = pyglet.resource.image('ui/ui_gray_block_filled.png')
I was using pyglet 1.4.4 and was able to use this the first format.
After upgrading to pyglet 1.4.8 I had the error: pyglet.resource.ResourceNotFoundException: Resource "your_resource_path" was not found on the path.
Switching to the 2nd format with forward slashes solved this issue for me. Not sure why this change was made...I'm sure there was a good reason for it.

Jython Build has no Output

Hey, I've been playing around with Jython a bit and I wrote the following test program:
from javax.swing import *
from java.awt import *
from java.awt.event import ActionListener
class JythonTest(JFrame):
_windowTitle = ""
def __init__(self):
self.initVars()
self.initLookAndFeel()
self.initComponents()
self.initGui()
def initVars(self):
self._windowTitle = "Jython Test"
JFrame.__init__(self, self._windowTitle)
def initLookAndFeel(self):
UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName())
def initComponents(self):
label = JLabel("Hello World!", JLabel.CENTER)
label.setFont(Font("Arial", Font.BOLD, 30))
tabs = JTabbedPane()
tabs.addTab("Test", label)
tabs.addTab("Calculator", self.CalculatorPane())
self.add(tabs)
def initGui(self):
self.setSize(400,200)
self.setDefaultCloseOperation(self.EXIT_ON_CLOSE)
self.setVisible(1)
class CalculatorPane(JPanel, ActionListener):
_txt1 = 0
_txt2 = 0
_box = 0
def __init__(self):
self.initVars()
self.initComponents()
def initVars(self):
pass
def initComponents(self):
self._txt1 = JTextField(5)
self._box = JComboBox(["+", "-", "*", "/"])
self._txt2 = JTextField(5)
btn = JButton("Go")
btn.addActionListener(self)
self.add(self._txt1)
self.add(self._box)
self.add(self._txt2)
self.add(btn)
def actionPerformed(self, ev):
val1 = self._txt1.getText()
val2 = self._txt2.getText()
operation = self._box.getSelectedItem()
val1 = int(val1)
val2 = int(val2)
if operation == "+":
answer = val1+val2
elif operation == "-":
answer = val1-val2
elif operation == "*":
answer = val1*val2
elif operation == "/":
answer = val1/val2
JOptionPane.showMessageDialog(self, "The answer is: " + str(answer))
if __name__ == "__main__":
win = JythonTest()
Here's my system info:
Operating System: Ubuntun 10.10
Netbeans Version: 6.9
My problem is that I can't compile the above code. It runs just fine when I click the run button, however, when I hit build or clean & build then I don't get any results. The build process runs in the bottom right corner for about half a second and then finishes. The output box opens up but it's entirely empty, even after the process ends. When I look at my project folder, nothing changes. Only two folders exist, nbproject and src. There probably should be a dist folder with a jar inside of it. Here's what's in the file structure:
user#computer: ~/NetBeansProjects/pythontest$ ls
nbproject src
user#computer: ~/NetBeansProjects/pythontest$ ls nbproject
private project.properties project.xml
user#computer: ~/NetBeansProjects/pythontest$ ls nbproject/private
private.xml
user#computer: ~/NetBeansProjects/pythontest$ ls src
pythontest.py setup.py
All I did to set up was install netbeans from the debian package (quite a while ago) and set up python/jython through the NetBeans python plugin. Any idea what's wrong?
The short answer is that it doesn't really work like that; I'm not aware of any IDE or tool support for packaging jython programs.
Usually what I do is just make a shell script that says:
java -cp "the/classpath/;" org.python.util.jython myscript.py
I've found that is the most foolproof way to run a jython program, and has saved me many headaches from not-working .jar files during development.
That said, there are methods of packaging jython programs in standalone .jar files, if that's what you want.
The best resource that I've found is the Distributing Jython Scripts page in the Jython FAQ, which describes a few different techniques for distributing jython scripts.
I usually only use the methods described there when "publishing" a program.