I have a recipe:
BB_STRICT_CHECKSUM = "1"
SRC_URI += "file://foo1.zip;md5sum=1234;unpack=0"
SRC_URI += "file://foo2.tar.gz;md5sum=5678;unpack=0"
Both checksums are wrong, but they can still pass bitbake.
You can try to do file check by hand with something like:
ZOO1_MD5 = "1234"
python do_package_prepend(){
input = oe.path.join(d.getVar('B'), 'foo1.zip')
cks = bb.utils.md5_file(input)
xpct = d.getVar('ZOO1_MD5')
if cks != xpct:
raise bb.fetch2.FetchError("MD5 fails for ...")
}
Another solution could be to put zoo1.zip in separate git repository and rely on git fetcher checks.
Related
By default Yocto adds build timestamp to the output image file name, but I would like to replace it by the revision of my integration Git repository (which references all my layers and configuration files). To achieve this, I put the following code to my image recipe:
def get_image_version(d):
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
return subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
except:
return d.getVar("MACHINE", True) + "-" + d.getVar("DATETIME", True)
IMAGE_VERSION = "${#get_image_version(d)}"
IMAGE_NAME = "${IMAGE_BASENAME}-${IMAGE_VERSION}"
IMAGE_NAME[vardepsexclude] = "IMAGE_VERSION"
This code works properly until I change Git revision (e.g. by adding a new commit). Then I receive the following error:
ERROR: When reparsing /home/ubuntu/yocto/poky/../mylayer/recipes-custom/images/core-image-minimal.bb.do_image_tar, the basehash value changed from 63e1e69797d2813a4c36297517478a28 to 9788d4bf2950a23d0f758e4508b0a894. The metadata is not deterministic and this needs to be fixed.
I understand this happens because the image recipe has already been parsed with older Git revision, but why constant changes of the build timestamp do not cause the same error? How can I fix my code to overcome this problem?
The timestamp does not have this effect since its added to vardepsexclude:
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#variable-flags
[vardepsexclude]: Specifies a space-separated list of variables that should be excluded from a variable’s dependencies for the purposes of calculating its signature.
You may need to add this in a couple of places, e.g.:
https://git.yoctoproject.org/poky/tree/meta/classes/image-artifact-names.bbclass#n7
IMAGE_VERSION_SUFFIX ?= "-${DATETIME}"
IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME SOURCE_DATE_EPOCH"
IMAGE_NAME ?= "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
After some research it turned out the problem was in this line
IMAGE_VERSION = "${#get_image_version(d)}"
because the function get_image_version() was called during parsing. I took inspiration from the source file in aehs29's post and moved the code to the anonymous Python function which is called after parsing.
I also had to add vardepsexclude attribute to the IMAGE_NAME variable. I tried to add vardepvalue flag to IMAGE_VERSION variable as well and in this particular case it did the same job as vardepsexclude. Mentioned Bitbake class uses both flags, but I think in my case using only one of them is enough.
The final code is below:
IMAGE_VERSION ?= "${MACHINE}-${DATETIME}"
IMAGE_NAME = "${IMAGE_BASENAME}-${IMAGE_VERSION}"
IMAGE_NAME[vardepsexclude] += "IMAGE_VERSION"
python () {
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
version = subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
d.setVar("IMAGE_VERSION", version)
except:
bb.warning("Could not get Git revision, image will have default name.")
}
EDIT:
After some research I realized it's better to define a global variable in layer.conf file of the layer containing the recipes referencing the variable. The variable is set by a python script and is immediately expanded to prevent deterministic build warning:
layer.conf:
require conf/image-version.py.inc
IMAGE_VERSION := "${#get_image_version(d)}"
image-version.py.inc:
def get_image_version(d):
import subprocess
import os.path
try:
parentRepo = os.path.dirname(d.getVar("COREBASE", True))
return subprocess.check_output(["git", "describe", "--tags", "--long", "--dirty"], cwd = parentRepo, stderr = subprocess.DEVNULL).strip().decode('UTF-8')
except:
bb.warn("Could not determine image version. Default naming schema will be used.")
return d.getVar("MACHINE", True) + "-" + d.getVar("DATETIME", True)
I think this is cleaner approach which fits BitBake build system better.
I'm using multiprocess.Pool to run several git clone process, but when there is a git repo much larger than others, the git clone process hangs.
The Pool I used as follows
def ProcessHelper(runfunc,incl_infos,project_root,skip_dirs):
p = Pool(processNum)
multi_res = [p.apply_async(runfunc, args=(incl_info,project_root,skip_dirs,)) for incl_info in incl_infos]
p.close()
p.join()
resMesList = []
for res in multi_res:
resMes = str(res.get())
if (resMes.find("Failed") != -1):
resMesList.append(resMes)
Then thread function use to check whether it's a git repo and clone the data if it is.
def runfunc(incl_info,project_root,skip_dirs):
if incl_info == 0:
git_url = project_root["git_url"]
dir_path = project_root["dir_path"]
GitCloneInShell(git_url,dir_path)
def GitCloneInShell(git_url,dir_path):
cmd = ("git clone %s %s " % (git_url, dir_path))
LogInfo("git clone cmd : %s" % cmd)
status = subprocess.call(cmd, shell=True)
return status
It usually works well. But when deal with a large repo with some other smaller repos, it will hangs on the large repo.
Can anyone give me some suggestions please ? Thanks a lot.
There is an existing project that uses Scalatra (2.6) and Swagger:
scalaMajorVersion = '2.12'
scalaVersion = "${scalaMajorVersion}.8"
scalatraVersion = "${scalaMajorVersion}:2.6.4"
compile "org.scalatra:scalatra-swagger_${scalatraVersion}"
I easily could add a new end point like:
get ("/upload", op[String]( // op finally invokes apiOperation
name = "Test method",
params = List(
query[Long]("id" -> "ID"),
query[String]("loginName" -> "login name")
),
authorizations = List(Permission.xxxxx.name)
)) {
...
}
but I cannot upload a file.
I expect to see a file selector button, but instead I see a single-line edit field.
(There are numerous things I'm uncertain about: form or file, [String] or [FileItem], which trait(s), what kind of initialization, etc.)
In the existing code I found a comment that someone could not get swagger to handle file upload. At the same time, I read that Scalatra and Swagger can do that, not all versions of them, but it looks like the version used in the project should be able to do that.
I could find code examples with yml/json interface definitions, but in the project there is no yml, only the apiOperation-based stuff.
Is there a working example using Scalatra 2.6, Swagger, and apiOperation?
I managed to get the file chooser (file selector, "Browse") button; there was no predefined constant (like DataType.String) for that. After I used DataType("File"), everything else just worked.
https://docs.swagger.io/spec.html says:
4.3.5 File
The File (case sensitive) is a special type used to denote file
upload. Note that declaring a model with the name File may lead to
various conflicts with third party tools and SHOULD be avoided.
When using File, the consumes field MUST be "multipart/form-data", and
the paramType MUST be "form".
post ("/uploadfile", op[String]( // op finally invokes apiOperation
name = "Upload File",
params = List(
new Parameter(
`name` = "kindaName",
`description` = Some("kindaDescription2"),
`type` = DataType("File"), // <===== !!!!!!
`notes` = Some("kindaNotes"),
`paramType` = ParamType.Form, // <===== !!
`defaultValue` = None,
`allowableValues` = AllowableValues.AnyValue,
`required` = true
)
),
consumes = List("multipart/form-data"), // <===== !!
...
)) {
val file: FileItem = fileParams("kindaName") // exception if missing
println("===========")
println("file: " + file)
println("name: " + file.getName + " size:"+file.getSize+" fieldName:"+file.getFieldName+ " ContentType:"+file.getContentType+" Charset:"+file.getCharset)
println("====vvv====")
io.copy(file.getInputStream, System.out)
println("====^^^====")
val file2: Option[FileItem] = fileParams.get("file123") // None if missing, and it is missing
println("file2: " + file2)
PS the apiOperation stuff is called "annotations".
Context - I am trying out Postgres' Geographic Information System extension PostGis that enables stories latitude and longitudes as Point and operations on it.
If I understand correctly then I need to add a custom converter that can convert the point between JOOQ and PostGis and add it to the gradle file.
Problem - When I generate the jooq-code, few files are generated incorrectly and have the fields defined twice which fail compilation. These are:
<configured-generation-dir>/tables/StValuecount.java
<configured-generation-dir>/tables/records/StValuecountRecord.java
<configured-generation-dir>/tables/records/StValuepercentRecord.java
<configured-generation-dir>/tables/_StValuecount.java
<configured-generation-dir>/tables/records/_StValuecountRecord.java
<configured-generation-dir>/tables/_StHistogram.java
<configured-generation-dir>/tables/records/_StHistogramRecord.java
<configured-generation-dir>/tables/_StQuantile.java
Gradle config =>
jooq{
myAwesomeApp(sourceSets.main){
logging = 'WARN'
jdbc {
driver = 'org.postgresql.Driver'
url = db_url
user = db_user
password = db_password
}
generator {
name = 'org.jooq.codegen.DefaultGenerator'
strategy {
name = 'org.jooq.codegen.DefaultGeneratorStrategy'
}
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
inputSchema = 'public'
forcedTypes {
forcedType {
userType = 'org.postgis.Point'
converter = 'com.example.JooqBreaksWithPostGis.jooq.converters.PostgresPointJooqConverter'
expression = '.*\\.point'
types = '.*'
}
}
}
generate {
routines = false
relations = true
deprecated = false
records = true
immutablePojos = false
fluentSetters = true
}
target {
packageName = 'jooq.fancy.app'
directory = 'src/main/java/generated'
}
}
}
}
What am I doing wrong?
I have also created a minimal project where I have reproduced the problem in case someone wants to quickly try it.
Steps to reproduce
Checkout project
git clone git#github.com:raj-saxena/JooqBreaksWithPostGis.git
Go to the project directory and start postgis docker container with
docker-compose up
Similarly, to remove postgis docker container run
docker-compose down
Run migrations that add a simple City table containing Point type with
./gradlew flywayMigrate
I have added few rows in a second migration to verify if the DB structure was working. Details to connect to Postgres instance in the build.gradle file.
Generate jooq files with
./gradlew generateMyAwesomeAppJooqSchemaSource
Verify that the files are generated in the configured src/main/java/generated directory.
Verify that the files mentioned above fail to compile.
Taking Lukas' advice, I added the exclude configuration to the jooq config as below:
database {
name = 'org.jooq.meta.postgres.PostgresDatabase'
...
excludes = '.*ST_ValueCount' +
'|.*St_Valuepercent' +
'|.*St_Histogram' +
'|.*St_Quantile' +
'|.*St_Approxhistogram' +
'|.*St_PixelOfValue' +
'|.*St_Approxquantile' +
'|.*ST_Tile'
}
This allowed the code to compile.
This sounds a lot like https://github.com/jOOQ/jOOQ/issues/4055. jOOQ 3.11 currently cannot handle overloaded table valued functions in any RDBMS that supports table valued functions. Your best option here is to exclude all the affected functions from the code generation, using <excludes>:
https://www.jooq.org/doc/latest/manual/code-generation/codegen-advanced/codegen-config-database/codegen-database-includes-excludes/
Hi under my bitbake file I want to stop the execution of certain tasks and want compile function to be executed every time. For this, I have done the following changes.
do_compile[nostamp] = "1"
do_clean[noexec] = "1"
do_cleanall[noexec] = "1"
do_cleansstate[noexec] = "1"
do_fetch[noexec] = "1"
do_patch[noexec] = "1"
do_unpack[noexec] = "1"
And it worked perfectly fine. I was able to stop the execution of tasks like clean, cleanall, cleansstate, fetch, patch and unpack. Also, I was able to make sure that the compile task runs every time.
However, I want to put some restrictions on the same. I want to make sure that noexec and nostamp on relevant task only applies when DEVMODE variable is set to 1. Psuedo code as follows.
if DEVMODE == 1 then
do_compile[nostamp] = "1"
do_clean[noexec] = "1"
do_cleanall[noexec] = "1"
do_cleansstate[noexec] = "1"
do_fetch[noexec] = "1"
do_patch[noexec] = "1"
do_unpack[noexec] = "1"
endif
How to achieve the same in a bitbake file? I have tried this and this links but I am not able to craft a working if condition.
NOTE: Am ok using BB_ENV_EXTRAWHITE, but am not able to code a working if condition for the bitbake file.
Use python anonymous function could work for you.
python () {
#add "export DEVMODE=1" under conf/setenv
#add DEVMODE under BB_ENV_EXTRAWHITE variable under conf/setenv
if d.getVar("DEVMODE", True) == "1":
d.setVarFlag("do_compile", 'nostamp', "1")
}
Or set directly:
do_compile[nostamp] = "${#'1' if d.getVar('DEVMODE') == '1' else '0'}"