I have a large project that is rather heterogenous - different languages and compilers are involved, altogether producing a build with the help of GNU make.
The project folder structure includes:
project
src
haxe --Haxe source code
graphics --Embeddable graphic resources
locale --Locale-specific resources
chinese --Chinese language resources
english --Generic English resources
build
china --Chinese market
debug --Debugging/Testing for developer(s)
release --Release
europe --European market
debug -- ...
release -- ...
All builds are produced by setting up and running 'make'. What I can't decide on is whether these makefiles should be put in 'src' directory as well? I generally consider original material I write by hand to be source code (since by my line of thought it originated from me and not produced by any program from some other input.) and I DO write my makefiles by hand. Another reason I consider this is because ONLY 'src' directory is a Git repository - I don't really need to version track anything else. Do I put all Makefiles into 'src'?
What's currently in your build directory structure? Is that the compiled output?
Intuitively (and I may be coming from a very different development world than you, so "intuitively" is a relative term) when I see a build folder adjoining a src folder, I expect the program's source code to be in the latter and the scripts/tools/etc. needed to build it to be in the former. The scripts (makefiles in this case, though potentially also to include other things) are themselves source code, as you state, but aren't the program's source. The distinction is that one is "what is being built" and the other is "how to build it."
If I understand you correctly, src is what's bound to your source control and build isn't? Under that circumstance, I would probably create a build (or builder or building or something of that nature) under src to house the scripts. It may be slightly unintuitive that it has to climb another folder before producing its output, but it should sit nicely alongside the resource folders you have there already.
Standard practice is to create a Makefile outside your src/ directory, which builds your project and within that src/ directory, another Makefile builds individual modules. That said, I think in your case it is sufficient to keep your Makefile solely in src/ though. I'm not sure if this applies, but you might want to look at the GNU autoconf package, it's used for exactly this type of thing.
Related
I am interested in understanding whether bazel can handle "two stage builds", where dependencies are discovered based on the file contents and dependencies must be compiled before the code that depends on them (unlike C/C++ where dependencies are mostly header files that are not separately compiled). Concretely, I am building the Coq language which is like Ocaml.
My intuition for creating a build plan would use an (existing) tool (called coqdep) that reads a .v file and returns a list of all of its direct dependencies. Here's the algorithm that I have in mind:
invoke coqdep on the target file and (transitively) on each of its dependent files,
once transitive dependencies for a target are computed, add a rule to build the .vo from the .v that includes transitive dependencies.
Ideally, the calls to coqdep (in step 1) would be cached between builds and so only need to be re-computed when the file changes. And the transitive closure of the dependency information would also be cached.
Is it possible to implement this in bazel? Are there any pointers to setting up builds for languages like this? Naively, it seems to be a two-stage build and I'm not sure how this fits into bazel's compilation model. When I looked at the rules for Ocaml, it seemed like it was relying on ocamlbuild to satisfy the build order and dependency requirements rather than doing it "natively" in bazel.
Thanks for any pointers or insights.
(don't have enough rep to comment yet, so this is an answer)
#2 of Toraxis' answer is probably the most canonical.
gazelle is an example of this for Golang, which is in the same boat: dependencies for Golang files are determined outside a Bazel context by reading the import statements of source files. gazelle is a tool that writes/rewrites Golang rules in BUILD files according to the imports in source files of the Bazel workspace. Similar tools could be created for other languages that follow this pattern.
but the generated BUILD file will be in the output folder, not in the source folder. So you also have to provide an executable that copies the files back into the source folder.
Note that binaries run via bazel run have the environment variable BUILD_WORKSPACE_DIRECTORY set to the root of the Bazel workspace (see the docs) so if your tool uses this environment variable, it could edit the BUILD files in-place rather than generating and copying back.
(In fact, the generating-and-copying-back strategy would likely not be feasible, because purely-generated files would contain only Coq rules, and not any other types of rules. To generate a BUILD file with Coq rules from one with other types of rules, one would have to add the BUILD files themselves as dependencies - which would create quite the mess!)
I'm looking into similar questions because I want to build ReasonML with Bazel.
Bazel computes the dependencies between Bazel targets based on the BUILD files in your repository without accessing your source files. The only interaction you can do with the file system during this analysis phase is to list directory contents by using glob in your rule invocations.
Currently, I see four options for getting fine-grained incremental builds with Bazel:
Spell out the fine-grained dependencies in hand-written BUILD files.
Use a tool for generating the BUILD files. You cannot directly wrap that tool in a Bazel rule to have it run during bazel build because the generated BUILD file would be in the output folder, not in the source folder. But you can run rules that call coqdep during the build, and provide an executable that edits the BUILD file in the source folder based on the (cacheable) result of the coqdep calls. Since you can read both the source and the output folder during the build, you could even print a message to the user if they have to run the executable again. Anyway, the full build process would be bazel run //tools/update-coq-build-files && bazel build to reach a fixed point.
Have coarse-grained dependencies in the BUILD files but persistent workers to incrementally rebuild individual targets.
Have coare-grained dependencies in the BUILD files but generate a separate action for each target file and use the unused_inputs_list argument of ctx.actions.run to communicate to Bazel which dependencies where actually unused.
I'm not really sure whether 3 and 4 would actually work or how much effort would be involved, though.
When I create a Fortran project by Eclipse,
it includes bin and src directories.
The automatically created Makefile has a rule that compiles test.f90 in src and its runnable is written in bin.
I am a pretty new to Fortran.
What if test.f90 is supposed to be just an object which is required to compile an executable file, where should the object file, say test.o, be put? and when test.f90 has a module inside, where should the module test.mod be output? mod at the same level as src and bin. or share, lib?
In that case, when I should separate all intermediate file in several directories, are there any template Makefile?
It depends on what kind of Fortran project you created. If you created a Makefile project, you have full control/responsibility for where files are located. It appears to create a src and bin directory but beyond that, it looks like you can put other files like *.mod and *.o wherever you want. I'm not sure what the convention is for Fortran Makefile projects.
I prefer to avoid using Fortran Makefile projects because I have zero desire to manually sort out dependencies and targets with Makefiles, but that's a personal preference; if you're familiar with make and need the fine control it gives you, this is a reasonable choice. Just starting out with a Fortran Makefile project seems really painful unless you already know make.
A simpler, though less obvious to choose "File | New | Fortran Project | Others" when creating a Fortran project, then select the compiler and what you want to build. An example is "Executable (Gnu Fortran on Linux/*nix)" for building an executable with gfortran on a unix/linux-like system. Build targets include executables, static libraries, and shared libraries, each using gfortran, ifort, and a few other compilers, on Windows, OSX, or linux/Unix.
The src and bin directories are not automatically created but Makefiles are managed by Eclipse. Eclipse will automatically create a directory like Debug or Debug_Intel64 and place executables, *.o, and *.mod files within it. If you make your own src directory, object files will be placed under Debug/src but the executables and module files will be under Debug. This directory name changes if you create a Release build configuration (Debug is the default).
By default, Eclipse puts all new source files in the project root but you can create Fortran source folders and organize your project however you want. The downside of automatic build management is that Eclipse is really only set up to build one thing, so if your project contains multiple executables (e.g. separate unit test, utilities, etc.) you need to manually build with make, CMake, etc.
Eclipse takes some getting used to, but integration with Mercurial is pretty good, and the Photran plugin (part of the Parallel Tools Project) helps with source templates, code analysis & Fortran refactoring tools, and a reasonably decent UI for configuring the compiler (at least for ifort on Linux). Plus it's cross-platform and works with a lot of other languages so it's far from the worst choice, especially if you're using Photran.
If you're not sold on Eclipse, I'd look at some less heavyweight options. Eclipse can be a pain some times and you might find a better IDE for what you're doing. OTOH, Eclipse works fine and the Photran plugin has some really nice features.
For fortran beginners the best IDE is codeblocks
download the binaries with Mingw fortran compiler.
As your requirement it puts all your .mod and .o files to obj folder and executables in bin folder and source files are just used from the directory where they are.
Since I first saw a dist/ directory in many open source projects, usually on GitHub, I've been wondering what it means.
With dist, vendor, lib, src, and many other folder names that we see quite often, I sometimes wonder how I should name my own folders.
Correct me if I'm wrong!
src: Contains the sources. Sometimes only the pure sources, sometimes with the minified version, depends on the project.
vendor: Contains other dependencies, like other open source projects.
lib: Good question, it's really close to vendor actually, depending on the project we can see one or another or both...
dist: From what I saw, it contains the "production" files, the one we should use if we want to use the library.
Why is open source so confusing? Isn't it possible to do things clearer? At least per language because some languages use specific names.
To answer your question:
/dist means "distributable", the compiled code/library.
Folder structure varies by build system and programming language. Here are some standard conventions:
src/: "source" files to build and develop the project. This is where the original source files are located, before being compiled into fewer files to dist/, public/ or build/.
dist/: "distribution", the compiled code/library, also named public/ or build/. The files meant for production or public use are usually located here.
There may be a slight difference between these three:
build/: is a compiled version of your src/ but not a production-ready.
dist/: is a production-ready compiled version of your code.
public/: usually used as the files runs on the browser. which it may be the server-side JS and also include some HTML and CSS.
assets/: static content like images, video, audio, fonts etc.
lib/: external dependencies (when included directly).
test/: the project's tests scripts, mocks, etc.
node_modules/: includes libraries and dependencies for JS packages, used by Npm.
vendor/: includes libraries and dependencies for PHP packages, used by Composer.
bin/: files that get added to your PATH when installed.
Markdown/Text Files:
README.md: A help file which addresses setup, tutorials, and documents the project. README.txt is also used.
LICENSE.md: any rights given to you regarding the project. LICENSE or LICENSE.txt are variations of the license file name, having the same contents.
CONTRIBUTING.md: how to help out with the project. Sometimes this is addressed in the README.md file.
Specific (these could go on forever):
package.json: defines libraries and dependencies for JS packages, used by Npm.
package-lock.json: specific version lock for dependencies installed from package.json, used by Npm.
composer.json: defines libraries and dependencies for PHP packages, used by Composer.
composer.lock: specific version lock for dependencies installed from composer.json, used by Composer.
gulpfile.js: used to define functions and tasks to be run with Gulp.
.travis.yml: config file for the Travis CI environment.
.gitignore: Specification of the files meant to be ignored by Git.
To answer your original question about the meaning of the dist folder:
The shortform dist stands for distributable and refers to a directory where files will be stored that can be directly used by others without the need to compile or minify the source code that is being reused.
Example: If I want to use the source code of a Java library that someone wrote, then you need to compile the sources first to make use of it. But if a library author puts the already compiled version into the repository, then you can just go ahead. Such an already compiled version is saved into the dist directory.
Something similar applies to JavaScript modules. Usually JavaScript code is minified and obfuscated for use in production. Therefore, if you want to distribute a JavaScript library, it's advisable to put the plain (not minified) source code into an src (source) directory and the minified and obfuscated version into the dist (distributable) directoy, so others can grab the minified version right away without having to minify it themselves.
Note: Some developers use names like target, build or dest (destination) instead of dist. But the purpose of these folders is identical.
Summary of the folders:
bin: binaries
contrib: contributions to the project
dist: -- see 1. and 2.
doc/s: documentation
include: headers (C/C++)
lib: libraries (C/C++)
man: short for man/manual pages (Unix/Linux), c.f. man(1)
src: source
"/dist means "distributable", the compiled code/library." ref.
"The shortform dist stands for distributable and refers to a directory where files will be stored that can be directly used by others without the need to compile or minify the source code that is being reused." ref.
Actually! "dist folder" is the result you get after modifying a source code with "npm run build" or "ng build" or "ng build --prod" for production.
Meanwhile! After getting "dist folder" there might still be few things that you still need to do depending on your project type ✌️
Trying to understand how EDE works by using it to generate Makefiles for
a project directory that contains several targets under a specific
hierarchy. I'm not having any luck, and the info pages don't seem to
answer my question.
My directory structure looks something like:
(The asterix (*) marks files containing main() functions)
research/
flow/
flow.cpp
flow.hpp
program.cpp *
samples/
sample1.yuv
utils/
yuvreader.cpp
yuvreader.hpp
tests/
yuvreader_test.cpp *
I want to create EDE project(s) with one or more subprojects; or maybe I
just want one or more targets...?
flow's program.cpp requires flow/ and utils/ sources, but yuvreader_test
only requires utils/ sources.
I did ede-new in the root directory, and all subdirectories. I also did ede-new-target in the root directory, but when adding source files in subdirectories, it does not recognize the target I created.
I would appreciate it if someone could point me to some more complicated
Project.ede files for something like I'm trying to do. You can guess
that I have more subdirectories containing class code files, some of
which have standalone programs that use that code; also I have more test
code under tests/. Any example files/command workflows would be
appreciated.
The EDE feature that will generate Makefile or Automakefiles has a few more constraints than either Make or Automake. For example, the files belonging to a target must be in the same directory as the project containing the target. In your example, you would probably have no projects in your root directory.
To bring multiple sources together into a single program, a simple mechanism is to have each subdirectory create an archive (.a) or a shared lib (.so) that is linked together in your program.
If that is too constraining, you can also write your own automake files, and EDE will read those directly, so you can have a more complex build procedure as needed.
The CEDET distribution uses EDE as its build process, so you could look at that as a complex example. It doesn't build C++ files though, so it may not be as helpful as you would like.
I want my Ant build to take all Java sources from src/main/*, compile them, and place them inside bin/main/. I also want it to compile src/test/* sources to bin/test/. I wan this behavior because I want to package the binaries into different JARs, and if they all just go to a single bin/ directory it will be impossible* (extremely difficult!) to know which class files belong where.
When I go to configure my build path and then click the Source tab I see an area toward the bottom where it reads Default output folder: and then allows you to browser for its location.
I'm wondering how to create bin/main and bin/test in an existing project without "breaking" Eclipse (it happens). I'm also wondering that if I just have my Ant build make and delete those directories during the clean-n-build process, that Eclipse might not care what the default output is set to. But I can't find any documentation either way.
Thanks in advance for any help here.
In Eclipse, you can only have one output folder per project for your compiled Java files. So you cannot get Eclipse to do the split you want (i.e. compile src/main to bin/main and src/test to bin/test).
You can, if you want, create two Eclipse projects, one main project and one test project, where the test project depends on (and tests) the main project. However, in that case, each project should be in its own directory structure, which is not what you are asking for. But this is a common approach.
Another way, which I would recommend, would be to not mix Ant compilation and Eclipse's compilation. Make the Ant script the way you describe (i. e. compile the main and test directories separately and create two separate jar files). Change the Eclipse compile directory to something different, for instance bin/eclipse. Use the Ant script when making official builds or building for release. Use Eclipse's building only for development/debugging. This way, your two build systems will not get in each other's way and confuse each other.
Hope this answers your question and I understood it correctly. Good luck!