Bazel build rules for BLAS & LAPACK - lapack

I have not found any project/repo providing Bazel build rules for Blas or Lapack.
This is quite unfortunate as these tools are often the primary libraries one must use for project oriented towards numerical computations.
Does such thing already exist somewhere?

No BUILD files yet...
However, a quick fix if you want to use already installed Blas or Lapack libs is to add a line like this one (according to the installed libs on your computer):
build --linkopt="-llapacke -llapack -lblas"
in your bazel.rc file (in the tools/ directory):
YourBazelProject/.
├── ...
├── WORKSPACE
└── tools
└── bazel.rc

I am not aware of BUILD files for these libraries either.
If you do create them, sharing them publicly or trying to submit them upstream (in the library code) will be a great way for others to benefit from your efforts.

Related

Is there an idiomatic way to structure a Lua project?

I'm creating a Neovim plugin in Lua and am looking for a reasonable way to structure my projects for testing, loading modules from Luarocks for development, and distributing this to users.
For example, in a Javascript project it's common to see something like the following:
README.md
node_modules/
dist/
src/
test/
After reviewing a number of Github repos for Lua projects, I'm not seeing a common pattern, so suggestions or boilerplates are welcome. Many thanks.

Fine-grained builds with dynamic dependencies?

I am interested in understanding whether bazel can handle "two stage builds", where dependencies are discovered based on the file contents and dependencies must be compiled before the code that depends on them (unlike C/C++ where dependencies are mostly header files that are not separately compiled). Concretely, I am building the Coq language which is like Ocaml.
My intuition for creating a build plan would use an (existing) tool (called coqdep) that reads a .v file and returns a list of all of its direct dependencies. Here's the algorithm that I have in mind:
invoke coqdep on the target file and (transitively) on each of its dependent files,
once transitive dependencies for a target are computed, add a rule to build the .vo from the .v that includes transitive dependencies.
Ideally, the calls to coqdep (in step 1) would be cached between builds and so only need to be re-computed when the file changes. And the transitive closure of the dependency information would also be cached.
Is it possible to implement this in bazel? Are there any pointers to setting up builds for languages like this? Naively, it seems to be a two-stage build and I'm not sure how this fits into bazel's compilation model. When I looked at the rules for Ocaml, it seemed like it was relying on ocamlbuild to satisfy the build order and dependency requirements rather than doing it "natively" in bazel.
Thanks for any pointers or insights.
(don't have enough rep to comment yet, so this is an answer)
#2 of Toraxis' answer is probably the most canonical.
gazelle is an example of this for Golang, which is in the same boat: dependencies for Golang files are determined outside a Bazel context by reading the import statements of source files. gazelle is a tool that writes/rewrites Golang rules in BUILD files according to the imports in source files of the Bazel workspace. Similar tools could be created for other languages that follow this pattern.
but the generated BUILD file will be in the output folder, not in the source folder. So you also have to provide an executable that copies the files back into the source folder.
Note that binaries run via bazel run have the environment variable BUILD_WORKSPACE_DIRECTORY set to the root of the Bazel workspace (see the docs) so if your tool uses this environment variable, it could edit the BUILD files in-place rather than generating and copying back.
(In fact, the generating-and-copying-back strategy would likely not be feasible, because purely-generated files would contain only Coq rules, and not any other types of rules. To generate a BUILD file with Coq rules from one with other types of rules, one would have to add the BUILD files themselves as dependencies - which would create quite the mess!)
I'm looking into similar questions because I want to build ReasonML with Bazel.
Bazel computes the dependencies between Bazel targets based on the BUILD files in your repository without accessing your source files. The only interaction you can do with the file system during this analysis phase is to list directory contents by using glob in your rule invocations.
Currently, I see four options for getting fine-grained incremental builds with Bazel:
Spell out the fine-grained dependencies in hand-written BUILD files.
Use a tool for generating the BUILD files. You cannot directly wrap that tool in a Bazel rule to have it run during bazel build because the generated BUILD file would be in the output folder, not in the source folder. But you can run rules that call coqdep during the build, and provide an executable that edits the BUILD file in the source folder based on the (cacheable) result of the coqdep calls. Since you can read both the source and the output folder during the build, you could even print a message to the user if they have to run the executable again. Anyway, the full build process would be bazel run //tools/update-coq-build-files && bazel build to reach a fixed point.
Have coarse-grained dependencies in the BUILD files but persistent workers to incrementally rebuild individual targets.
Have coare-grained dependencies in the BUILD files but generate a separate action for each target file and use the unused_inputs_list argument of ctx.actions.run to communicate to Bazel which dependencies where actually unused.
I'm not really sure whether 3 and 4 would actually work or how much effort would be involved, though.

Eclipse Fortran project directory structure

When I create a Fortran project by Eclipse,
it includes bin and src directories.
The automatically created Makefile has a rule that compiles test.f90 in src and its runnable is written in bin.
I am a pretty new to Fortran.
What if test.f90 is supposed to be just an object which is required to compile an executable file, where should the object file, say test.o, be put? and when test.f90 has a module inside, where should the module test.mod be output? mod at the same level as src and bin. or share, lib?
In that case, when I should separate all intermediate file in several directories, are there any template Makefile?
It depends on what kind of Fortran project you created. If you created a Makefile project, you have full control/responsibility for where files are located. It appears to create a src and bin directory but beyond that, it looks like you can put other files like *.mod and *.o wherever you want. I'm not sure what the convention is for Fortran Makefile projects.
I prefer to avoid using Fortran Makefile projects because I have zero desire to manually sort out dependencies and targets with Makefiles, but that's a personal preference; if you're familiar with make and need the fine control it gives you, this is a reasonable choice. Just starting out with a Fortran Makefile project seems really painful unless you already know make.
A simpler, though less obvious to choose "File | New | Fortran Project | Others" when creating a Fortran project, then select the compiler and what you want to build. An example is "Executable (Gnu Fortran on Linux/*nix)" for building an executable with gfortran on a unix/linux-like system. Build targets include executables, static libraries, and shared libraries, each using gfortran, ifort, and a few other compilers, on Windows, OSX, or linux/Unix.
The src and bin directories are not automatically created but Makefiles are managed by Eclipse. Eclipse will automatically create a directory like Debug or Debug_Intel64 and place executables, *.o, and *.mod files within it. If you make your own src directory, object files will be placed under Debug/src but the executables and module files will be under Debug. This directory name changes if you create a Release build configuration (Debug is the default).
By default, Eclipse puts all new source files in the project root but you can create Fortran source folders and organize your project however you want. The downside of automatic build management is that Eclipse is really only set up to build one thing, so if your project contains multiple executables (e.g. separate unit test, utilities, etc.) you need to manually build with make, CMake, etc.
Eclipse takes some getting used to, but integration with Mercurial is pretty good, and the Photran plugin (part of the Parallel Tools Project) helps with source templates, code analysis & Fortran refactoring tools, and a reasonably decent UI for configuring the compiler (at least for ifort on Linux). Plus it's cross-platform and works with a lot of other languages so it's far from the worst choice, especially if you're using Photran.
If you're not sold on Eclipse, I'd look at some less heavyweight options. Eclipse can be a pain some times and you might find a better IDE for what you're doing. OTOH, Eclipse works fine and the Photran plugin has some really nice features.
For fortran beginners the best IDE is codeblocks
download the binaries with Mingw fortran compiler.
As your requirement it puts all your .mod and .o files to obj folder and executables in bin folder and source files are just used from the directory where they are.

What is the meaning of the /dist directory in open source projects?

Since I first saw a dist/ directory in many open source projects, usually on GitHub, I've been wondering what it means.
With dist, vendor, lib, src, and many other folder names that we see quite often, I sometimes wonder how I should name my own folders.
Correct me if I'm wrong!
src: Contains the sources. Sometimes only the pure sources, sometimes with the minified version, depends on the project.
vendor: Contains other dependencies, like other open source projects.
lib: Good question, it's really close to vendor actually, depending on the project we can see one or another or both...
dist: From what I saw, it contains the "production" files, the one we should use if we want to use the library.
Why is open source so confusing? Isn't it possible to do things clearer? At least per language because some languages use specific names.
To answer your question:
/dist means "distributable", the compiled code/library.
Folder structure varies by build system and programming language. Here are some standard conventions:
src/: "source" files to build and develop the project. This is where the original source files are located, before being compiled into fewer files to dist/, public/ or build/.
dist/: "distribution", the compiled code/library, also named public/ or build/. The files meant for production or public use are usually located here.
There may be a slight difference between these three:
build/: is a compiled version of your src/ but not a production-ready.
dist/: is a production-ready compiled version of your code.
public/: usually used as the files runs on the browser. which it may be the server-side JS and also include some HTML and CSS.
assets/: static content like images, video, audio, fonts etc.
lib/: external dependencies (when included directly).
test/: the project's tests scripts, mocks, etc.
node_modules/: includes libraries and dependencies for JS packages, used by Npm.
vendor/: includes libraries and dependencies for PHP packages, used by Composer.
bin/: files that get added to your PATH when installed.
Markdown/Text Files:
README.md: A help file which addresses setup, tutorials, and documents the project. README.txt is also used.
LICENSE.md: any rights given to you regarding the project. LICENSE or LICENSE.txt are variations of the license file name, having the same contents.
CONTRIBUTING.md: how to help out with the project. Sometimes this is addressed in the README.md file.
Specific (these could go on forever):
package.json: defines libraries and dependencies for JS packages, used by Npm.
package-lock.json: specific version lock for dependencies installed from package.json, used by Npm.
composer.json: defines libraries and dependencies for PHP packages, used by Composer.
composer.lock: specific version lock for dependencies installed from composer.json, used by Composer.
gulpfile.js: used to define functions and tasks to be run with Gulp.
.travis.yml: config file for the Travis CI environment.
.gitignore: Specification of the files meant to be ignored by Git.
To answer your original question about the meaning of the dist folder:
The shortform dist stands for distributable and refers to a directory where files will be stored that can be directly used by others without the need to compile or minify the source code that is being reused.
Example: If I want to use the source code of a Java library that someone wrote, then you need to compile the sources first to make use of it. But if a library author puts the already compiled version into the repository, then you can just go ahead. Such an already compiled version is saved into the dist directory.
Something similar applies to JavaScript modules. Usually JavaScript code is minified and obfuscated for use in production. Therefore, if you want to distribute a JavaScript library, it's advisable to put the plain (not minified) source code into an src (source) directory and the minified and obfuscated version into the dist (distributable) directoy, so others can grab the minified version right away without having to minify it themselves.
Note: Some developers use names like target, build or dest (destination) instead of dist. But the purpose of these folders is identical.
Summary of the folders:
bin: binaries
contrib: contributions to the project
dist: -- see 1. and 2.
doc/s: documentation
include: headers (C/C++)
lib: libraries (C/C++)
man: short for man/manual pages (Unix/Linux), c.f. man(1)
src: source
"/dist means "distributable", the compiled code/library." ref.
"The shortform dist stands for distributable and refers to a directory where files will be stored that can be directly used by others without the need to compile or minify the source code that is being reused." ref.
Actually! "dist folder" is the result you get after modifying a source code with "npm run build" or "ng build" or "ng build --prod" for production.
Meanwhile! After getting "dist folder" there might still be few things that you still need to do depending on your project type ✌️

Matlab dependency management

I am looking to apply dependency management to a large-scale Matlab project.
This project imports a large number of java libraries, as well as some compiled C++ code, to the extent that some software best practices are now becoming more essential.
Is anyone aware of something along the lines of Maven/Ivy for use with Matlab?
I'm not very familiar with Matlab, but sounds like your issue is that you're trying to put a large set of binary files under some sort of version control?
If those files are available in Maven Central, you can use my ant2ivy script to generate a starting set of ivy.xml and ivysettings.xml files.
One of the great things about ivy is that it can be run stand-alone as follows:
java -jar ivy.jar -retrieve "lib/[artifact].[ext]" -ivy ivy.xml -settings ivysettings.xml
This will download the jars and place them into a "lib" directory (Or whatever directory Matlab uses).
matlab isn't really made for large-scale projects. You'll have to come up with your own code to check for all necessary dependencies.
I created a simple maven based dependency management for matlab projects using jitpack.io and zip as release format.
Sample project - https://github.com/ragavsathish/mmockito
Simple archetype can be found in https://github.com/ragavsathish/matlab-simple-archetype
Please provide your comments on what can be improved further