On Yocto Project FAQ:
The Yocto Project and OpenEmbedded share a core collection of metadata
called openembedded-core. However, the two organizations remain
separate, each with its own focus. OpenEmbedded provides a
comprehensive set of metadata for a wide variety of architectures,
features, and applications. The Yocto Project focuses on providing
powerful, easy-to-use, interoperable, well-tested tools, metadata, and
board support packages (BSPs) for a core set of architectures and
specific boards.
I still not getting nothing clear. The two frameworks are meant to build Linux distributions. But I would like to know on what they are distinct specifically. Not only techically but also objectivly, so I can argue why to choose one or another.
Moreover, why Yocto has so much prominence? altough OE being the first build framework.
ps: I have worked with Yocto Project, but not with OE.
The key point is that the Yocto Project is a community/organisation, and not something you can buy/download/install.
Some of the things that the Yocto Project works on includes bitbake (the build tool), OpenEmbedded Core (the essential recipes to build systems, such as glibc/gcc/systemd), some BSPs, and tooling/services (error reporting service, autobuilder, etc).
The OpenEmbedded community predates Yocto and at the time had a different focus, but now we both contribute to the same projects so there's no real difference.
Related
In our company we have switched to Linux embedded systems (2 different platforms). Our build is based on yocto.
Unfortunately we are all quite unexperienced with it. Especially I have some questions about collaboration within yocto and with reusing its deployment (SDK).
In our company there are multiple teams involved in development:
Team1: Maintaining yocto and the Linux base system
Team2: Developing company-own libraries which will become part of the base system (own git repository, which is added in yocto recipe)
Team3: Developing software (based on Linux base system + company-own libraries)
The problem: Our current development workflow is quite slow.
For example:
There is some bug in a small company-own library LibA.
Team2 commits some bugfix for LibA (small project, a standalone build would take ~2min).
In a next step there has to be a Pull-Request updating the CommitID within LibA's yocto recipe (~15min including a small CI build + merge checks).
As soon as the Pull-Request is merged, a release-build can be triggered (CI). The result of this build is a SDK and a MfgTool (~120min for all platforms/variants).
Now Team3 has to download the new SDK and update their build to use the new SDK (~5min).
Then they trigger a release build of their software (~10min) which results in a flashable image.
So all in all a small change of LibA takes about 2 hours until it can be integrated by Team3, and another 15min for the complete software to be available to our test team.
What is the recommended standard way for this (I think we are not the only company with this issue)?
How can we improve the yocto build workflow?
Is there any way to prevent that the yocto build must be used if LibA changes?
Thank you very much for your help.
I would takle your main issue, the build time of 120m seems very long.
I would recommend to create use Source Mirror , sstate mirror. Do one buildjob in the night which creates those contents. Take a build machine with at least 8 cores and 16-32GB ram.
Suppose I have a class library which I want to target netstandard1.3, but also use BigInteger. Here's a trivial example - the sole source file is Adder.cs:
using System;
using System.Numerics;
namespace Calculator
{
public class Adder
{
public static BigInteger Add(int x, int y)
=> new BigInteger(x) + new BigInteger(y);
}
}
Back in the world of project.json, I would target netstandard1.3 in the frameworks section, and have an explicit dependency on System.Runtime.Numerics, e.g. version 4.0.1. The nuget package I create will list just that dependency.
In the brave new world of csproj-based dotnet tooling (I'm using v1.0.1 of the command-line tools) there's an implicit metapackage package reference to NETStandard.Library 1.6.1 when targeting netstandard1.3. This means that my project file is really small, because it doesn't need the explicit dependency:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard1.3</TargetFramework>
</PropertyGroup>
</Project>
... but the nuget package produced has a dependency on NETStandard.Library, which suggests that in order to use my small library, you need everything there.
It turns out I can disable that functionality using DisableImplicitFrameworkReferences, then add in the dependency manually again:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netstandard1.3</TargetFramework>
<DisableImplicitFrameworkReferences>true</DisableImplicitFrameworkReferences>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="System.Runtime.Numerics" Version="4.0.1" />
</ItemGroup>
</Project>
Now my NuGet package says exactly what it depends on. Intuitively, this feels like a "leaner" package.
So what's the exact difference for a consumer of my library? If someone tries to use it in a UWP application, does the second, "trimmed" form of dependencies mean that the resulting application will be smaller?
By not documenting DisableImplicitFrameworkReferences clearly (as far as I've seen; I read about it in an issue) and by making the implicit dependency the default when creating a project, Microsoft are encouraging users to just depend on the metapackage - but how can I be sure that doesn't have disadvantages when I'm producing a class library package?
In the past, we've given developers the recommendation to not reference the meta
package (NETStandard.Library) from NuGet packages but instead reference
individual packages, like System.Runtime and System.Collections. The
rationale was that we thought of the meta package as a shorthand for a bunch of
packages that were the actual atomic building blocks of the .NET platform. The
assumption was: we might end up creating another .NET platform that only
supports some of these atomic blocks but not all of them. Hence, the fewer packages you reference, the more portable you'd be. There were also concerns regarding how our tooling deals with large package graphs.
Moving forward, we'll simplify this:
.NET Standard is an atomic building block. In other words, new platforms
aren't allowed to subset .NET Standard -- they have to implement all of it.
We're moving away from using packages to describe our platforms,
including .NET Standard.
This means, you'll not have to reference any NuGet packages for .NET Standard
anymore. You expressed your dependency with the lib folder, which is exactly how
it has worked for all other .NET platforms, in particular .NET Framework.
However, right now our tooling will still burn in the reference to
NETStandard.Library. There is no harm in that either, it will just become
redundant moving forward.
I'll update the FAQ on the .NET Standard repo to include this question.
Update: This question is now part of the FAQ.
The team used to recommend figuring out what the slimmest package set was. They no longer do this, and recommend people just bring in NETStandard.Library instead (in the case of an SDK-style project, this will be done automatically for you).
I've never gotten a totally straight forward answer as to why that was, so allow me to make some educated guesses.
The primary reason is likely to be that it allows them to hide the differences in versions of the dependent libraries that you would otherwise be required to track yourself when changing target frameworks. It's also a much more user friendly system with the SDK-based project files, because you frankly don't need any references to get a decent chunk of the platform (just like you used to with the default references in Desktop-land, especially mscorlib).
By pushing the meta-definition of what it means to be a netstandard library, or a netcoreapp application into the appropriate NuGet package, they don't have to build any special knowledge into the definition of those things as Visual Studio (or dotnet new) sees them.
Static analysis could be used during publishing to limit the shipped DLLs, which is something they do today when doing native compilation for UWP (albeit with some caveats). They don't do that today for .NET Core, but I presume it's an optimization they've considered (as well as supporting native code).
There's nothing stopping you from being very selective, if you so choose. I believe you'll find that you're nearly the only one doing it, which also defeats the purpose (since it'll be assumed everybody is bringing in NETStandard.Library or Microsoft.NETCore.App).
You shouldn't need to disable the implicit reference. All platforms that the library will be able to run on will already have the assemblies that the NETStandard.Library dependency would require.
The .NET Standard Library is a specification, a set of reference assemblies that you compile against that provides a set of APIs that are guaranteed to exist on a know set of platforms and versions of platforms, such as .NET Core or the .NET Framework. It is not an implementation of these assemblies, just enough of the API shape to allow the compiler to successfully build your code.
The implementation for these APIs are provided by a target platform, such as .NET Core, Mono or .NET Framework. They ship with the platform, because they are an essential part of the platform. So there is no need to specify a smaller dependency set - everything's already there, you won't change that.
The NETStandard.Library package provides these reference assemblies. One point of confusion is the version number - the package is version 1.6.1, but this does not mean ".NET Standard 1.6". It's just the version of the package.
The version of the .NET Standard you're targeting comes from the target framework you specify in your project.
If you're creating a library and want it to run on .NET Standard 1.3, you'd reference the NETStandard.Library package, currently at version 1.6.1. But more importantly, your project file would target netstandard1.3.
The NETStandard.Library package will give you a different set of reference assemblies depending on your target framework moniker (I'm simplifying for brevity, but think lib\netstandard1.0, lib\netstandard1.1 and dependency groups). So if your project targets netstandard1.3, you'll get the 1.3 reference assemblies. If you target netstandard1.6, you'll get the 1.6 reference assemblies.
If you're creating an application, you can't target the .NET Standard. It doesn't make sense - you can't run on a specification. Instead, you target concrete platforms, such as net452 or netcoreapp1.1. NuGet knows the mapping between these platforms and the netstandard target framework monikers, so knows which lib\netstandardX.X folders are compatible with your target platform. It also knows that the dependencies of NETStandard.Library are satisfied by the target platform, so won't pull in any other assemblies.
Similarly, when creating a standalone .NET Core app, the .NET Standard implementation assemblies are copied with your app. The reference to NETStandard.Library does not bring in any other new apps.
Note that dotnet publish will create a standalone application, but it won't doesn't currently do trimming, and will publish all assemblies. This will be handled automatically by tooling, so again, trimming dependencies in your library won't help here.
The only place I can imagine where it might help to remove the NETStandard.Library reference is if you are targeting a platform that doesn't support the .NET Standard, and you can find a package from the .NET Standard where all of the transitive dependencies can run on your target platform. I suspect there aren't many packages that would fit that bill.
In order to achieve Continuous Integration, in the book Continuous Integration: Improving Software Quality and Reducing Risks, one can read that one of the principles behind it is centralizing software assets on a version control system, may these be compiled artifacts, libraries, third party components and so on.
I can understand such statement in case one is only using a SCM and nothing else. But what about if one is to be using a tool such as Nexus? Would you say that Nexus or the alike makes such statement obsolete when referring to already built artifacts?
The short answer is: "yes"
The longer answer is that using Nexus to centralize software assets is perfectly legitimate, as it is also a versioned store. It is more appropriate for technologies that need to compile binaries, but even for scripted technologies I favour creating tarballs or better still RPM packages. Finally a deployment process that incorporates an SCM system marries you to that software.... More worryingly every person who needs to install your software must also have access (credentials) to the tool.
I'm working with an embedded system software project right now, and we're facing some problems dealing with some precompiled binaries living inside our repository.
We have several repositories for different parts of our project: One for the application itself, one for the OS, one for the bootloader and several libraries. All of them, except the one for our application, are shared with other teams, for other projects. We are using git (and changing is not an option right now), but I think we'd have the same problem with any VCS.
Right now, we have a precompiled binary for each of those components living inside our application repository. The idea was to speed up the build time, since the OS alone takes about 20 minutes to build from scratch and most guys work only with the application.
Problem is, there are several bugs/features in those binaries (and related application code) to be integrated at any time and, as you know, diffing and merging binaries won't work.
So, how do you guys do when you have to work with those external dependencies?
Thanks a lot =)
One viable solution is to use an external binary repository like Nexus.
It is not linked to a VCS, meaning you can easily clean up old versions of said binaries you don't need anymore
It is light-weight (simple HTTP client-server protocol, no need to clone all the repo with all the versioned binaries like you would with a DVCS -- git or mercurial --)
What are the recommendations for including your compiler, libraries, and other tools in your source control system itself?
In the past, I've run into issues where, although we had all the source code, building an old version of the product was an exercise in scurrying around trying to get the exact correct configuration of Visual Studio, InstallShield and other tools (including the correct patch version) used to build the product. On my next project, I'd like to avoid this by checking these build tools into source control, and then build using them. This would also simplify things in terms of setting up a new build machine -- 1) install our source control tool, 2) point at the right branch, and 3) build -- that's it.
Options I've considered include:
Copying the install CD ISO to source control - although this provides the backup we need if we have to go back to an older version, it isn't a good option for "live" use (each build would need to start with an install step, which could easily turn a 1 hour build into 3 hours).
Installing the software to source control. ClearCase maps your branch to a drive letter; we could install the software under this drive. This doesn't take into account non-file part of installing your tools, like registry settings.
Installing all the software and setting up the build process inside a virtual machine, storing the virtual machine in source control, and figuring out how to get the VM to do a build on boot. While we capture the state of the "build machine" with ease, we get the overhead of a VM, and it doesn't help with the "make the same tools available to developers issue."
It seems such a basic idea of configuration management, but I've been unable to track down any resources for how to do this. What are the suggestions?
I think the VM is your best solution. We always used dedicated build machines to get consistency. In the old COM DLL Hell days, there were dependencies on (COMCAT.DLL, anyone) on non-development software installed (Office). Your first two options don't solve anything that has shared COM components. If you don't have any shared components issue, maybe they will work.
There is no reason the developers couldn't take a copy of the same VM to be able to debug in a clean environment. Your issues would be more complex if there are a lot of physical layers in your architecture, like mail server, database server, etc.
This is something that is very specific to your environment. That's why you won't see a guide to handle all situations. All the different shops I've worked for have handled this differently. I can only give you my opinion on what I think has worked best for me.
Put everything needed to build the
application on a new workstation
under source control.
Keep large
applications out of source control,
stuff like IDEs, SDKs, and database
engines. Keep these in a directory as ISO files.
Maintain a text document, with the source code, that has a list of the ISO files that will be needed to build the app.
I would definitely consider the legal/licensing issues surrounding the idea. Would it be permissible according to the various licenses of your toolchain?
Have you considered ghosting a fresh development machine that is able to build the release, if you don't like the idea of a VM image? Of course, keeping that ghosted image running as hardware changes might be more trouble than it's worth...
Just a note on the versionning of libraries in your version control system:
it is a good solution but it implies packaging (i.e. reducing the number of files of that library to a minimum)
it does not solves the 'configuration aspect' (that is "what specific set of libraries does my '3.2' projects need ?").
Do not forget that set will evolves with each new version of your project. UCM and its 'composite baseline' might give the beginning of an answer for that.
The packaging aspect (minimum number of files) is important because:
you do not want to access your libraries through the network (like though dynamic view), because the compilation times are much longer than when you use local accessed library files.
you do want to get those library on your disk, meaning snapshot view, meaning downloading those files... and this is where you might appreciate the packaging of your libraries: the less files you have to download, the better you are ;)
My organisation has a "read-only" filesystem, where everything is put into releases and versions. Releaselinks (essentially symlinks) point to the version being used by your project. When a new version comes along it is just added to the filesystem and you can swing your symlink to it. There is full audit history of the symlinks, and you can create new symlinks for different versions.
This approach works great on Linux, but it doesn't work so well for Windows apps that tend to like to use things local to the machine such as the registry to store things like configuration.
Are you using a continuous integration (CI) tool like NAnt to do your builds?
As a .Net example, you can specify specific frameworks for each build.
Perhaps the popular CI tool for whatever you're developing in has options that will allow you to avoid storing several IDEs in your version control system.
In many cases, you can force your build to use compilers and libraries checked into your source control rather than relying on global machine settings that won't be repeatable in the future. For example, with the C# compiler, you can use the /nostdlib switch and manually /reference all libraries to point to versions checked in to source control. And of course check the compilers themselves into source control as well.
Following up on my own question, I came across this posting referenced in the answer to another question. Although more of a discussion of the issue than an aswer, it does mention the VM idea.
As for "figuring out how to build on boot": I've developed using a build farm system custom-created very quickly by one sysadmin and one developer. Build slaves query a taskmaster for suitable queued build requests. It's pretty nice.
A request is 'suitable' for a slave if its toolchain requirements match the toolchain versions on the slave - including what OS, since the product is multi-platform and a build can include automated tests. Normally this is "the current state of the art", but doesn't have to be.
When a slave is ready to build, it just starts polling the taskmaster, telling it what it's got installed. It doesn't have to know in advance what it's expected to build. It fetches a build request, which tells it to check certain tags out of SVN, then run a script from one of those tags to take it from there. Developers don't have to know how many build slaves are available, what they're called, or whether they're busy, just how to add a request to the build queue. The build queue itself is a fairly simple web app. All very modular.
Slaves needn't be VMs, but usually are. The number of slaves (and the physical machines they're running on) can be scaled to satisfy demand. Slaves can obviously be added to the system any time, or nuked if the toolchain crashes. That'ss actually the main point of this scheme, rather than your problem with archiving the state of the toolchain, but I think it's applicable.
Depending how often you need an old toolchain, you might want the build queue to be capable of starting VMs as needed, since otherwise someone who wants to recreate an old build has to also arrange for a suitable slave to appear. Not that this is necessarily difficult - it might just be a question of starting the right VM on a machine of their choosing.