bitbake do_image dependency not cached - yocto

I have a task do_image_custom that has a dependency on task do_image_ext4.
That task (do_image_ext4) generates an image file containing DATETIME.
The first time I build my image, no errors.
dependency_DATETIME.rootfs.ext4 is generated and used by its dependents.
If I make a change to the consuming task of the ext4 file, because I need to stipulate the dependency on DATETIME.rootfs.ext4.
After I build a second time (without cleaning), I get the error that do_image_custom cannot find newer_datetime.rootfs.ext4
I check the IMGDEPLOYDIR and sure enough, that file doesn't exist and the do_image_ext4 task still has the first timestamp.
My question is, what am I doing wrong here in do_image_custom such that it re-evaluates DATETIME every time it is run without checking with (perhaps) the sstate cache?

The problem was that my custom task (do_image_custom) depended on the output of a prior task. That task output generates an ext4 image with a timestamp in the name.
do_image_custom re-evaluated the DATETIME, even though the dependency (the ext4 file with an earlier DATETIME did not, and therefore was not rebuilt. Hence when do_image_custom executed, it referenced a file that did not exist (the error) because it was not generated (correctly so, because the basehash for the dependency task was unchanged).
The solution was (in front of me all along) to modify my custom task (do_image_custom) to refer to a symlink (also generated in the same step as the ext4) which does not have a DATETIME in the symlink name, hence making do_image_custom invariant to any or no changes to it's dependent step.

Related

BitBake - what defines the order of tasks for a recipe?

Is there a way to list the order of tasks that shall be executed when a recipe is built with BitBake? I know I can build the recipe and then inspect log.task_order, but that's not what I'm after - I want to know where is the order of tasks defined for given recipe, without actually building it. I also know that there's bitbake <recipe_name> -c listtasks, but AFAIK, that lists all the available tasks, regardless if they are actually executed during the build.
Update:
The recipe I'm interested in is the kernel recipe, this is how its log.task_order looks like after a full clean build:
do_fetch
do_unpack
do_prepare_recipe_sysroot
do_kernel_checkout
do_symlink_kernsrc
do_validate_branches
do_kernel_metadata
do_patch
do_kernel_version_sanity_check
do_populate_lic
do_kernel_configme
do_configure
do_kernel_configcheck
do_compile
do_shared_workdir
do_kernel_link_images
do_compile_kernelmodules
do_strip
do_sizecheck
do_install
do_populate_sysroot
do_package
do_packagedata
do_package_qa
do_package_write_ipk
do_bundle_initramfs
do_deploy
I would expect that this sequence is defined somewhere in the recipe metadata, but I haven't found it.

Azure DevOps deployment fails with the error message "The operation was canceled."

My Azure DevOps pipeline tasks successfully complete without issues except for the final deployment step:
Job Issues - 1 Error
The job running on agent XXXX ran longer than the maximum time of 00:05:00 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
The build logs state the operation was canceled:
021-03-02T20:50:00.4223027Z Folders: 695
2021-03-02T20:50:00.4223319Z Files: 10645
2021-03-02T20:50:00.4223589Z Size: 672611102
2021-03-02T20:50:00.4223851Z Compressed: 249144045
2021-03-02T20:50:03.6023001Z ##[warning]Unable to apply transformation for the given package. Verify the following.
2021-03-02T20:50:03.6032907Z ##[warning]1. Whether the Transformation is already applied for the MSBuild generated package during build. If yes, remove the <DependentUpon> tag for each config in the csproj file and rebuild.
2021-03-02T20:50:03.6034584Z ##[warning]2. Ensure that the config file and transformation files are present in the same folder inside the package.
2021-03-02T20:50:04.5268038Z Initiated variable substitution in config file : C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\Areas\Admin\sitemap.config
2021-03-02T20:50:04.5552027Z Skipped Updating file: C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\Areas\Admin\sitemap.config
2021-03-02T20:50:04.5553082Z Initiated variable substitution in config file : C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\web.config
2021-03-02T20:50:04.5642868Z Skipped Updating file: C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\web.config
2021-03-02T20:50:04.5643366Z XML variable substitution applied successfully.
2021-03-02T20:51:00.8934630Z ##[error]The operation was canceled.
2021-03-02T20:51:00.8938641Z ##[section]Finishing: Deploy IIS Website/App:
When I examine the deployment states, I notice one of my tasks takes quite a while for what should be a fairly simple operation:
The file transform portion takes over half of the allotted 5 minutes? Could this be the issue?
steps:
- task: FileTransform#1
displayName: 'File Transform: '
inputs:
folderPath: '$(System.DefaultWorkingDirectory)/_site.com/drop/Release/Nop.Web.zip'
fileType: json
targetFiles: '**/dataSettings.json'
It may be inefficient but FileTransform log shows a significant amount of time spent after the variable has been substituted. Not sure what's causing the long delay, but the logs don't account for the time after the variable has been successfully substituted:
2021-03-02T23:04:44.3796910Z Folders: 695
2021-03-02T23:04:44.3797285Z Files: 10645
2021-03-02T23:04:44.3797619Z Size: 672611002
2021-03-02T23:04:44.3797916Z Compressed: 249143976
2021-03-02T23:04:44.3970596Z Applying JSON variable substitution for **/App_Data/dataSettings.json
2021-03-02T23:04:45.2396016Z Applying JSON variable substitution for C:\azagent\A2\_work\_temp\temp_web_package_0182869515217865\App_Data\dataSettings.json
2021-03-02T23:04:45.2399264Z Substituting value on key DataConnectionString with (string) value: ***
**2021-03-02T23:04:45.2446986Z JSON variable substitution applied successfully.
2021-03-02T23:07:25.4881687Z ##[section]Finishing: File Transform:**
The job running on agent XXXX ran longer than the maximum time of 00:05:00 minutes.
Based on the error message, the cause of this issue is that the running time of the Agent Job has reached the maximum value set.
If you are using the release pipeline, you could set the timeout in Agent Job -> Execution plan ->timeout.
If you are using the build pipeline, you could set the timeout in Agent Job -> Agent Job -> Execution plan -> timeout (For An Agent Job) and Options -> Build job -> Build job timeout in minutes (For whole Build pipeline).
The file transform portion takes over half of the allotted 5 minutes
From the task log, the zip package contains many files and folders. Therefore, transform task will take more time to traverse the file to find the target file.

Yocto: ROOTFS_POSTPROCESS_COMMAND and do_rootfs

I have a task which has a dependency set as below:
bb.build.addtask('do_function2', 'do_build', 'do_rootfs', d)
There is another task do_function1 that modifies some files in the rootfs:
ROOTFS_POSTPROCESS_COMMAND_append = " do_function1;"
I require do_function2 to be triggered only after rootfs for the image has been populated.
Query is whether the tasks that come under ROOTFS_POSTPROCESS_COMMANDS_append satisfy the do_rootfs dependency that I have set up in the addtask() call?
I want do_function2 to run only after do_function1. Is that the case even if do_function1 is specified under ROOTFS_POSTPROCESS_COMMAND_append?

Spring batch, handle failed files again, skip successfully handled

How to ideologically correct organize file handling?
I have a folder for new files (NEW), folder for old files (OLD), a folder for failed files (FAIL). New file puts in NEW, then if the handling was correct, the file goes to OLD, if the handling was failed, the file goes to ERR. Then we take this file again and correcting it and put in NEW if all ok file goes to OLD if failed goes to ERR. And repeat again and again.
I have job with constant name "fileHandlingJob", in job i have some steps: "extract", "handling", "utilize", and i have job parameters: "filePath", "fileName".
Thanks!
If you state that uniqueness criteria of file - it's file's name, then you are on right way.
If job was in state FAILED (ERR folder) then you can retrigger it with same set of parameters. If job was COMPLETED - you can't run it again. Spring batch will complain.
You can ensure this behaviour by having unique file name as Job's parameter. So no other job could be triggered with same file name. Spring batch will simply prevent this.
Second parameter filePath can be additional non-unique parameter.
JobParametersBuilder jobParametersBuilder = new JobParametersBuilder()
.addString("fileName", "myfile.xml", true)
.addDate("filePath", "C:\new\myfile.xml", false);
true/false here means whether parameter is unique or not.

Model Error: It appears that build process was unable to locate some Utility(e.g. make,compiler,linker, etc.)

I am new to S-Functions and Real TIme WorkShop. The S-Function has been created using an Embedded Matlab Function. The S-Function Works fine, but when i try to build the same, i get the following error report:
It appears that the build process was unable to locate some utility (e.g. make, compiler, linker, etc.). Please verify your path and tool environment variables are correct. You should be able to execute the make command: .\Radius_S_func2.bat at an MS DOS Command Prompt in the directory: C:\Users\skaushik\Desktop\Matlab Models\WIP\TESTs\Sfunc_2\Radius_S_func2_Source Currently, this generates the following error:
C:\Users\skaushik\Desktop\Matlab Models\WIP\TESTs\Sfunc_2\Radius_S_func2_Source>set MATLAB=C:\MATLAB\R2010b
C:\Users\skaushik\Desktop\Matlab Models\WIP\TESTs\Sfunc_2\Radius_S_func2_Source>.......\u_Utils\Build\make -f Radius_S_func2.mk GENERATE_REPORT=0 INCLUDE_MDL_TERMINATE_FCN=0 COMBINE_OUTPUT_UPDATE_FCNS=1 MAT_FILE=0 MULTI_INSTANCE_CODE=0 INTEGER_CODE=0 PORTABLE_WORDSIZES=0 GENERATE_ERT_S_FUNCTION=0 GENERATE_ASAP2=0 EXT_MODE=0 EXTMODE_STATIC_ALLOC=0 EXTMODE_STATIC_ALLOC_SIZE=1000000 EXTMODE_TRANSPORT=0 TMW_EXTMODE_TESTING=0 MODELLIB=Radius_S_func2lib.lib RELATIVE_PATH_TO_ANCHOR=.. MODELREF_TARGET_TYPE=NONE OPTS="-DRT -DUSE_RTMODEL -DERT" The system cannot find the path specified.
PLease guide me how to understand the error, so i can take care of it myself.
Thank you!!
Solution: The build target was not specified therefore matlab gave the above mentioned error. To resolve this one should specify the build environment for a specific target.