How does one automatically make a `COQ_PROJ.opam` install script automatically from a Coq Project/Package? - coq

I have a very long list of coq projects I want to automatically install with opam pin/install. I'd like to install them with opam because I am using this python tool (PyCoq) that uses opam pin/install. How can I automatically create a COQ_PROJ.opam / *.opam file given a coq project that I can install with make?
For example, a format of a coq project/package (proj/pkg) that works for me is this: https://github.com/IBM/pycoq/tree/main/pycoq/test/lf
An idea is that in pip one can easily create a pip requirements file from a python project that is already installed (Automatically create requirements.txt). Thus, one possible solution could be:
install the coq project with make (or it's instructions)
run the equivalent of pip freeze > requirements.txt but for coq.
How does one do that?
sample list of coq projs:
[
{
"project_name": "constructive-geometry",
"train_files": [
"problems.v",
"affinity.v",
"basis.v",
"orthogonality.v",
"part1.v",
"part3.v",
"part2.v"
],
"test_files": [],
"switch": "coq-8.10"
},
{
"project_name": "higman-s",
"train_files": [
"inductive_wqo.v",
"higman_aux.v",
"higman.v",
"list_embeding.v",
"tree.v"
],
"test_files": [],
"switch": "coq-8.10"
},
{
"project_name": "int-map",
"train_files": [
"Mapcanon.v",
"Mapc.v",
"Mapaxioms.v",
"Map.v",
"Adalloc.v",
"Fset.v",
"Mapsubset.v",
"Mapfold.v",
"Maplists.v",
"Lsort.v",
"Mapcard.v",
"Allmaps.v",
"FMapIntMap.v",
"Mapiter.v"
],
...
I do know about opam switch export/import/create but I doubt it works due to this reason:
I think opam switch export assumes that everything I've installed so far was already available in your opam switch already: Save the current switch state to a file.. Thus I think it assumes that you've installed everything so far with opam which is not always true for every coq project afaik and if it were I wouldn't have this issue in the first place -- since I would have used opam pin/install in the first place to install the coq proj/pkg. I think some coq projects/packages use a coq_makefile -f _CoqProject -o CoqMakefile followed by a make command (ref: https://coq.inria.fr/refman/practical-tools/utilities.html#building-a-coq-project, https://stackoverflow.com/questions/72586352/what-is-the-best-practice-for-installing-external-dependencies-in-a-coq-project). Those that use make are my target. I know make can run arbitrary code but I am hoping there is a standard way to install things in coq that will help me connect it with opam...or python...
though I could try running make and then opam switch export though I don't know how opam would know how make compiled and got dependencies if make uses only coqc or something else that doesn't install it to opam...any ideas?
Perhaps useful self contained coq proj/pkg: https://github.com/brando90/pycoq/tree/main/debug_proj though it uses no external dependencies for now :(, help extending it for even the simplest depedency for the sake of example is welcomed!
references:
proverbot's discussion: https://github.com/UCSD-PL/proverbot9001/issues/27
question/discussion in the original coq-gym repo: https://github.com/princeton-vl/CoqGym/discussions/77
coq projects: https://github.com/UCSD-PL/proverbot9001/tree/master/coq-projects
useful reference for install coq projs: What is the best practice for installing external dependencies in a Coq project?
gitissue in my PyCoq fork: https://github.com/brando90/pycoq/issues/4
https://coq.zulipchat.com/#narrow/stream/252087-Machine-learning-and-automation/topic/Installation.20of.20a.20diverse.20set.20of.20Coq.20package.20for.20ML/near/294876190

Related

What is the best practice for installing external dependencies in a Coq project?

I understand what I believe is the essence of the official utilties doc https://coq.inria.fr/refman/practical-tools/utilities.html#building-a-coq-project:
one creates a _CoqProject with arguments to coqc and the file names to compile (hopefully in an order that takes into account dependencies)
then one make an automatic CoqMakefile with coq_makefile -f _CoqProject -o CoqMakefile
Then you use their recommended Makefile to run the automatically generated make file.
But then if we need other dependencies, it doesn't say how to install them (or uninstall) them. What is the standard way to do that?
My guess is that one likely adds a target to your Makefile at the end of it and do some sort of opam install?
e.g.
# KNOWNTARGETS will not be passed along to CoqMakefile
KNOWNTARGETS := CoqMakefile
# KNOWNFILES will not get implicit targets from the final rule, and so
# depending on them won't invoke the submake. TODO: Not sure what this means
# Warning: These files get declared as PHONY, so any targets depending
# on them always get rebuilt -- or better said, any rules which those names will have their cmds be re-ran (which is
# usually rebuilding functions since that is what make files are for)
KNOWNFILES := Makefile _CoqProject
# Runs invoke-coqmakefile rule if you do run make by itself. Sets the default goal to be used if no targets were specified on the command line.
.DEFAULT_GOAL := invoke-coqmakefile
# Depends on two files to run this, itself and our _CoqProject
CoqMakefile: Makefile _CoqProject
$(COQBIN)coq_makefile -f _CoqProject -o CoqMakefile
# Note make knows what is the make file in this case thanks to -f CoqMakefile
invoke-coqmakefile: CoqMakefile install_external_dependencies
$(MAKE) --no-print-directory -f CoqMakefile $(filter-out $(KNOWNTARGETS),$(MAKECMDGOALS))
#
.PHONY: invoke-coqmakefile $(KNOWNFILES)
####################################################################
## Your targets here ##
####################################################################
# This should be the last rule, to handle any targets not declared above
%: invoke-coqmakefile
#true
# I know this is not a good coq dependency example but just to make it concrete wrote some opam command
install_external_dependencies:
opam install coq-serapi
I think I wrote the install_external_dependencies in the right place but I'm not sure. Is that correct? Anyone has a real example?
For all the code see: https://github.com/brando90/ultimate-utils/tree/master/tutorials_for_myself/my_makefile_playground/beginner_coq_project_with_makefiles/debug_proj
related: question on official tutorial for building coq projects https://coq.discourse.group/t/official-place-to-learn-how-to-setup-coq-make-files-for-beginner/1682
Btw,
I don't understand the last like in the makefile yet.
# This should be the last rule, to handle any targets not declared above
%: invoke-coqmakefile
#true
i.e.
%true in the make file template coq gave us.
% in the name of the rule.
What does that line do?
Update
I'm seeking an end-to-end small demo of how to install all dependencies with whatever the recommended approach when using _CoqProject and coq_makefile as shown in the utilities doc https://coq.inria.fr/refman/practical-tools/utilities.html. The ideal answer would contain a single script to install and compile everything in one go -- say in a install_project_name.sh. Including opam switches etc.
Related:
How does one install a new version of Coq when it cannot find the repositories in a new Mac M1 machine?
Installing packages for Coq using OPAM
https://coq.discourse.group/t/official-place-to-learn-how-to-setup-coq-make-files-for-beginner/1682
The simplest setup is to install external dependencies manually with opam.
opam install packages-needed-by-my-project
Then they will be immediately available to build your own project.
The next level of organization is to package up your project. Refer to the following Coq community resources:
Coq community templates
Recommended Project Structure
The main thing immediately relevant to your question is to have a *.opam file at the root of your project which lists its dependencies (possibly with version requirements). Then you can install them using opam install . --deps-only.
The Makefile part of your question is about a bit of overengineering for passing options seamlessly to CoqMakefile. I'm not sure how it works off-hand, but it's not important anyway, especially because Dune is superseding make as the recommended build system for Coq project.

Coq file generated by WP does not compile

I have installed frama-c (18.0) and coqide (8.9) through opam (plus other needed dependencies of course, but that may not be the matter here). Well the point is I simply installed it through opam, not done anything else strange (and I didn't see any particular instruction that I should do otherwise).
Frama-c works as expected when I use Alt-ergo with WP, but if I try to use coq or coqide instead of Alt-ergo, then I get the following error for each goal that Qed does not manage to prove immediately:
[wp] 13 goals scheduled
[wp] [Coq] 'Qed.v' compilation failed.
------------------------------------------------------------
--- Coqc (stderr) :
------------------------------------------------------------
File "/tmp/wp7fe5dc.dir/coqwp/Qed.v", line 27, characters 8-17:
Error:
Cannot find a physical path bound to logical path matching suffix bool.
------------------------------------------------------------
[wp] [Coq] Goal typed_nondet_loop_inv_preserved : Failed
Compilation of 'Qed.v' failed.
As a note, before displaying the error, it manages to compile some other .v files. I have tried to open manually the files in coqide, and I get the same result. For the record, here are the lines coq complains about:
Require bool.Bool.
Require int.Int.
Require int.Abs.
Require int.ComputerDivision.
Require real.Real.
Require real.RealInfix.
Require real.FromInt.
I have also tried to downgrade coq a bit, but not lower than 8.7 (as otherwise opam complains of incompatible base installation package and I don't really want to screw up my installation), and got the same result.
If someone would have an idea of what is causing that and how I can get to set it up that correctly, that'd be nice. Even if for what I'm doing with it for now, Alt-ergo is sufficient, I would have liked to play a bit with coq to see how it can be used.
Regards,
--
Vincent Penelle.
First, indeed you need coq < 8.8 (e.g. 8.7.2) if you want to use it with Frama-C/WP, as newer versions are not supported for the moment.
Second, the order in which you have installed your packages is relevant. In particular, if the appropriate version of coq has been installed after frama-c, WP did not compile and install its coq libraries, which are the ones that are missing here. Thus, you may want to do opam reinstall frama-c to compile the package against a compatible coq version.

where is Coq aac_tactics installed?

I was testing the AAC tactics library for rewrites modulo associativity and commutativity. According to a Coq website, one should:
Depending on your installation, either modify the following two lines,
or add them to your .coqrc files, replacing "." with the path to the
aac_tactics library.
Add Rec LoadPath "." as AAC_tactics.
Add ML Path ".".
Require Import AAC.
Require Instances.
But I don't know how to find the path to the aac_tactics library, and using "." didn't work.
I installed Coq under Ubuntu 16.04 LTS along the lines of:
opam repo add coq-released https://coq.inria.fr/opam/released
opam install coq-mathcomp-ssreflect.1.6
opam install coq-aac-tactics.8.5.1
Does anyone know where to find the library location?
It seems this will work (at least for this tutorial):
(*
Add Rec LoadPath "." as AAC_tactics.
Add ML Path ".".
*)
Require Import AAC_tactics.AAC.
Require Import AAC_tactics.Instances.
Usually, OPAM stores its stuff in ~/.opam. You can look it up with the following command in your terminal:
$ opam config var root
Then you can have multiple configurations called switches (it's mainly for keeping different version of the OCaml compiler). The root for your current switch can be found this way:
$ opam config var prefix
And the libraries for the current switch are kept in the directory which you can look up here:
$ opam config var lib
There you'll find the AAC_tactics subdirectory, which is the prefix we needed to add to our imports above.

Installing Cairo, Helm on Windows

How do I install Helm (https://hackage.haskell.org/package/helm) on Windows 7 (64-bit)?
(Update: I had posted a lot of error messages here, but I've moved them to my answer to not clutter up the question.)
Installation for Windows 64-bit:
I'm including error messages, for if you follow all the steps up to that point and then just try to install directly. This is a conglomeration of a bunch of ad-hoc steps from following many different posts. Any simplification would be appreciated!
Note: Do all work in directories without spaces. I'm doing all work in C:/PF; modify this to your directory.
Download MSYS2-x86_64 from https://msys2.github.io/ and install it. Cabal install cairo (or helm) will give something like:
Configuring cairo-0.13.1.0...
setup.exe: Missing dependencies on foreign libraries:
Missing C libraries: z, cairo, z, gobject-2.0, ffi, pixman-1, fontconfig,
expat, freetype, iconv, expat, freetype, z, bz2, harfbuzz, glib-2.0, intl,
ws2_32, ole32, winmm, shlwapi, intl, png16, z
Download C libraries. In MINGW64 (NOT MSYS2 - I had trouble with MSYS2 at random stages in the process), use the package manager:
pacman -Ss cairo
to search for the Cairo package. You'll find "mingw64/mingw-w64-x86_64-cairo", so install that:
pacman -S mingw64/mingw-w64-x86_64-cairo
*.pc files should have been added to C:\PF\msys64\mingw64\lib\pkgconfig and C:\PF\msys64\usr\lib\pkgconfig. (pkg-config needs to be able to find these files. It looks in PKG_CONFIG_PATH, which by default should have the lib/pkgconfig folder above. Moving the file here is easiest. See Can't install sdl2 via cabal) If you get
The pkg-config package ... version ... cannot be found
errors then check your *.pc files.
Repeat with other required libraries, like atk
pacman -S mingw64/mingw-w64-x86_64-atk
(I don't know the complete list, but error messages later on will let you know what to get.)
Get the development files for these libraries (as suggested by How to install cairo on Windows). Most of them are bundled up at http://ftp.gnome.org/pub/gnome/binaries/win64/gtk+/2.22/. Unzip.
Copy files (.a, .dll.a) in lib to C:\PF\msys64\mingw64\lib. Copy the pkgconfig folder, which contains the .pc files.
Copy files in include to C:\PF\msys64\mingw64\include.
Add C:\PF\gtk+-2.22.1\bin to the path.
(2) and (3) might be redundant. I don't know - I did them both.
At this point you can probably do "cabal install cairo". (Warning: if your end goal is something else, you may not want to "cabal install" intermediate packages, see https://wiki.haskell.org/Cabal/Survival#Issue_.232_--_Not_installing_all_the_packages_in_one_go.)
See (4) for the syntax in specifying extra-include-dirs and extra-lib-dirs (but if you copied the files above this shouldn't be necessary),
Any time you get
Missing (or bad) header file
check to see you copied the *.h files to mingw64\include and/or add the include folder to the PATH. Use cabal install -v3 to get verbose error messages if the problem persists.
If you get something like
cairo-0.13.1.0: include-dirs: /mingw64/include/freetype2 is a relative path
which makes no sense (as there is nothing for it to be relative to). You can
make paths relative to the package database itself by using ${pkgroot}. (use
--force to override)
try --ghc-pkg-options="--force" (as mentioned at https://github.com/gtk2hs/gtk2hs/issues/139).
Get SDL. Otherwise you'll get
configure: error: *** SDL not found! Get SDL from www.libsdl.org.
If you already installed it, check it's in the path. If problem remains,
please send a mail to the address that appears in ./configure --version
indicating your platform, the version of configure script and the problem.
Failed to install SDL-0.6.5.1
Follow the instructions in (2) to get sdl/sdl2 libraries. (See instructions here Installing SDL on Windows for Haskell (GHC).)
The new version helm-0.7.1 requires sdl2, but there are other dependency issues with helm-0.7.1 as of writing. Download SDL from http://sourceforge.net/projects/msys2/files/REPOS/MINGW/x86_64/ (direct download link to newest version as of writing http://sourceforge.net/projects/msys2/files/REPOS/MINGW/x86_64/mingw-w64-x86_64-SDL-1.2.15-7-any.pkg.tar.xz.sig/download), unzip. "cabal install sdl" gives
* Missing (or bad) header file: SDL/SDL.h
* Missing C library: SDL
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
so we specify where the dirs are (change the name depending on where you extracted sdl to)
cabal install sdl --extra-include-dirs=C:/PF/sdl\include --extra-lib-dirs=C:/sdl/lib
If you got SDL2 (http://libsdl.org/download-2.0.php) (for a newer version of Helm): there is a fatal bug that hasn't been fixed in the release version. (If you don't fix it, cabal install -v3 things which depends on it will give error
winapifamily.h: No such file or directory
("winapifamily.h: No such file or directory" when compiling SDL in Code::Blocks) Download https://hg.libsdl.org/SDL/raw-file/e217ed463f25/include/SDL_platform.h, replace the file in the include folder and in C:/PF/msys64/mingw64/include/SDL2.
Download gtk2hs from http://code.haskell.org/gtk2hs and run
the following
cd gtk2hs/tools
cabal install
cd ../glib
cabal install
cd ../gio
cabal install
cd ../pango
cabal install --ghc-pkg-options="--force"
(Maybe you have already installed glib and gio from before? I did this step because normal install of Pango caused an error for me (https://github.com/gtk2hs/gtk2hs/issues/110)
pango-0.13.1.0: include-dirs: /mingw64/include/freetype2 is a relative path
which makes no sense (as there is nothing for it to be relative to). You can
make paths relative to the package database itself by using ${pkgroot}. (use
--force to override)
Once the Helm developers get things updated you should be able to do "cabal install helm" but right now there seem to be dependency issues. For me, cabal automatically tries to install helm-0.4 (probably because 0.4 didn't give upper bounds on dependencies, while newer versions do. You could try "cabal unpack"ing and deleting the upper bounds...). Then
cabal unpack helm-0.4
Installing gives an error because "pure" got moved to Prelude. Open helm-0.4\src\FRP\Helm\Automaton.hs and change line 17:
import Prelude hiding (id, (.), pure)
Now
cabal install
Try to compile and run a program using Helm
(This is 0.4 - look on the website for a newer sample if you tried a newer Helm)
import FRP.Helm
import qualified FRP.Helm.Window as Window
render :: (Int, Int) -> Element
render (w, h) = collage w h [filled red $ rect (fromIntegral w) (fromIntegral h)]
main :: IO ()
main = run $ fmap (fmap render) Window.dimensions
If you get an error about a missing .dll (sdl.dll), find it in a bin/ folder and add the folder to your PATH (or copy it to somewhere on your path).

Coq: manage LoadPath in project with subdirectories

I have a Coq project with its libraries organised into subdirectories, something like:
…/MyProj/Auxiliary/Aux.v
…/MyProj/Main/Main.v (imports Auxiliary/Aux.v)
When I compile the files, I expect to do so from working directory MyProj (via a makefile). But I also want to work on the files using Proof General/Coqtop, in which case the working directory is by default the directory in which the file lives.
But this means that the LoadPath is different between the two contexts, and so the logical path needed for the library import is different. How do I set up the coqc invocation, the LoadPath, and the import declarations so that they work in both contexts?
Each approach I have tried, something goes wrong. For instance, if I compile Aux.v by invoking
coqc -R "." "MyProj" Auxiliary/Aux.v
and import it in Main.v as
Require Import MyProj.Auxiliary.Aux.
then this works when I compile Main.v with
coqc -R "." "MyProj" Main/Main.v
but fails in Coqtop, with Error: Cannot find library MyProj.Auxiliary.Aux in loadpath. On the other hand, if before the Require Import I add
Add LoadPath ".." as MyProj.
then this works in Coqtop, but fails under coqc -R "." "MyProj" Main/Main.v, with
Error: The file […]/MyProj/Auxiliary/Aux.vo contains library
MyProj.Auxiliary.Aux and not library MyProj.MyProj.Auxiliary.Aux
I’m looking for a solution that’s robust for a library that’s shared with collaborators (and hopefully eventually with users), so in particular it can’t use absolute file paths. The best I have found for now is to add emacs local variables to set the LoadPath up when Proof General invokes Coqtop:
((coq-mode . ((coq-prog-args . ("-R" ".." "MyProj" "-emacs")))))
but this (a) seems a little hacky, and (b) only works for Proof General, not in Coqide or plain Coqtop. Is there a better solution?
Allow me to side-step your question by suggesting an alternative process, hinted at by Tiago.
Assuming that your project's tree looks like this:
MyProj/Auxiliary/Aux.v
MyProj/Main/Main.v
In MyProj, write a _CoqProject file listing all your Coq files
-R . ProjectName
Auxiliary/Aux.v
Main/Main.v
When you open one of these Coq files, emacs will look for the _CoqProject and do-the-right-thing (tm).
As shown by Tiago, coq_makefile will also give you a Makefile for free.
I know you explicitly asked for something that works across different platforms, but there's already a Proof-General-specific solution that is less hacky than the one you have. Proof General has a special variable called coq-load-path that you can set with local variables, much like you did for coq-prog-args. The advantage is that you don't have to worry about any other arguments that need to be passed to coqtop (such as -emacs in your example). Thus, your .dir-locals.el file could have a line like this:
((coq-mode . ((coq-load-path . ((".." "MyProj"))))))
Unfortunately, I am not aware of anything that works across platforms, although I'm pretty sure that something specific for CoqIDE must exist. If this is the case, maybe you could set up a script to keep these configuration files updated across different platforms?
If you use coq_makefile you can install the library in your system.
Without OPAM
To initialize your project:
coq_makefile -f _CoqProject -o Makefile
Share your library with other projects:
make install
With OPAM
Assuming you have OPAM installed, you can use coq-shell to help you take care of dependencies.
To setup your project:
coq_shell_url="https://raw.githubusercontent.com/gares/opam-coq-shell/master/src/opam-coq"
curl -s "${coq_shell_url}" | bash /dev/stdin init 8.4 # Install Coq and its dependencies
eval `opam config env --switch=coq-shell-8.4` # Setup the environment
coq_makefile -f _CoqProject -o Makefile # Generates the makefile
opam pin add coq:YOURLIBRARY . # Add your library to OPAM
When you update your library you should do:
opam upgrade coq:YOURLIBRARY
Here is an example of a project that uses the OPAM method:
https://bitbucket.org/cogumbreiro/aniceto-coq/src