Related
There is a programming "style" (or maybe paradigm, i'm not sure what to call it) which is as follows:
First, you write a specification: a formal description of what your (whole, or part of) program is to do. This is done within the programming system; it is not a separate artifact.
Then, you write the program, but - and this is the key distinction between this programming style and others - every step of this writing task is guided in some way by the specification you've written in the previous step. How exactly this guidance happens varies wildly; in Coq you have a metaprogramming language (Ltac) which lets you "refine" the specification while building the actual program behind the scenes, whereas in Agda you compose a program by filling "holes" (i'm not actually sure how it goes in Agda, as i'm mostly used to Coq).
This isn't exactly everyone's favorite style of programming, but i'd like to try practicing it in general-purpose, popular programming languages. At least in Coq i've found it to be fairly addictive!
...but how would i even search for ways to do it outside proof assistants? Which leads us to the question: I'm looking for a name for this programming style, so that i can try looking up tools that let me program like that in other programming languages.
Mind you, of course a more proper question would be directly asking for examples of such tools, but AFAIK questions asking for lists of answers aren't appropriate for Stack Exchange sites.
And to be clear, i'm not all that hopeful i'm really going to find much; these are mostly academic pastimes, and your typical programming language isn't really amenable to this style of programming (for example, the specification language might end up being impossibly complex). But it's worth a shot!
It is called proof-driven development (or type-driven development). However, there is very little information about it.
This process you mention about slowly creating your program by means of ltac (in the case of coq) or holes (in the case of Agda and Idris) is called refinement. So you will also find reference in the literature for this style as proof by refinement or programming by refinement.
Now the most important thing to realize is that this style of programming is intrinsic to more complex type system that will allow you to extract as much information as possible the current environment. So it is natural to find attached with dependent types, although it is not necessarily the case.
As mentioned in another response you're also going to find references to it as Type-Driven Development, there is an idris book about it.
You may be interested in looking into some other projects such as Lean, Isabelle, Idris, Agda, Cedille, and maybe Liquid Haskell, TLA+ and SAW.
As pointed out by the two previous answers, a possible name for the program style you mention certainly is: type-driven development.
From the Coq viewpoint, you might be interested in the following two references:
Certified Programming with Dependent Types (CPDT, by Adam Chlipala): a Coq textbook that teaches advanced techniques to develop dependently-typed Coq theories and automate related proofs.
Experience Report: Type-Driven Development of Certified Tree Algorithms in Coq (by Reynald Affeldt, Jacques Garrigue, Xuanrui Qi, Kazunari Tanaka), published at the Coq Workshop 2019 (slides, extended abstract):
The authors also use the acronym TDD, which interestingly enough, also has another acceptation in the software engineering community: test-driven development (this widely used methodology naturally leads to high-quality test suites).
Actually, both acceptations of TDD share a common idea: one systematically starts by writing the specification (of the considered unit), then only after that, writing some code that fulfills the spec (make the unit tests pass), then we loop and incrementally specify+implement(+refactor) other code units.
Last but not least, there are some extra pointers in this discussion from the Discourse OCaml forum.
I was under the impression that there are folks out there that do write pure applications using Scalaz, but based on this example: [ stacking StateT in scalaz ], it looks like anything real would also be impossibly hairy.
Are there any guidelines or examples of real, modular, loosely-coupled, pure applications in Scala? I'm expecting that this means scalaz.effect.SafeApp and RWST over IO, but I'd like to hear from folks who have done it.
Thanks.
Edit: In the absence of an answer, I've started collecting resources as an answer below. If you have any examples or related links to contribute, please do.
i think you are mixing two different things. one is pure functional programming and second is scala type system. you can do 'pure' programming in any language, even in java. if the language is funvtional than you will have pure functional programming.
does it make your programs work faster? depends on the program - it scales better but for single threaded parts you will rather loose performance.
does it 'save your cognition'? it depends on how good you are in what you are doing. if you work with FP, monads, arrows etc on the daily basis then i assume it may help significantly. if you show the code to the OO developer he probably won't understand anything.
does it save the development time? as previously, i think it may but to be honest it doesn't matter that much. you more often read the code rather than write it
can you do useful stuff in PFP? yes, some companies makes money on haskell
and now, can it be done in scala? for sure. will anyone do it in scala? probably not because it's too easy to break the purity, because type system is too weak and because there are better, 'more pure' tools for it (but currently not on jvm)
I guess I will start collecting resources here, and update as I find more.
Functional Reactive Programming: stefan hoeck's blog, github, examples
Monadic effect worlds for interacting safely with mutable data. (tpolecat)
Mellow database access for Scala (tpolecat)
Dependency Injection without the Gymnastics (tony, rúnar)
Google search for "extends SafeApp"
I see a couple of different research groups, and at least one book, that talk about using Coq for designing certified programs. Is there are consensus on what the definition of certified program is? From what I can tell, all it really means is that the program was proved total and type correct. Now, the program's type may be something really exotic such as a list with a proof that it's nonempty, sorted, with all elements >= 5, etc. However, ultimately, is a certified program just one that Coq shows is total and type safe, where all the interesting questions boil down to what was included in the final type?
Edit 1
Based on wjedynak's answer, I had a look at Xavier Leroy's paper "Formal Verification of a Realistic Compiler", which is linked in the answers below. I think this contains some good information, but I think the more informative information in this sequence of research can be found in the paper Mechanized Semantics for the Clight Subset of the C Language by Sandrine Blazy and Xavier Leroy. This is the language that the "Formal Verification" paper adds optimizations to. In it, Blazy and Leroy present the syntax and semantics of the Clight language and then discuss the validation of these semantics in section 5. In section 5, there's a list of different strategies used for validating the compiler, which in some sense provides an overview of different strategies for creating a certified program. These are:
Manual reviews
Proving properties of the semantics
Verified translations
Testing executable semantics
Equivalence with alternate semantics
In any case, there are probably points that could be added and I'd certainly like to hear about more.
Going back to my original question of what the definition is of a certified program, it's still a little unclear to me. Wjedynak sort of provides an answer, but really the work by Leroy involved creating a compiler in Coq and then, in some sense, certifying it. In theory, it makes it possible to now prove things about the C programs since we can now go C->Coq->proof. In that sense, it seems like there's this work flow where we could
Write a program in X language
Form of a model of the program from step 1 in Coq or some other proof assistant tool. This could involve creating a model of the programming language in Coq or it could involve creating a model of the program directly (i.e. rewriting the program itself in Coq).
Prove some property about the model. Maybe it's a proof about the values. Maybe it's the proof of the equivalence of statements (stuff like 3=1+2 or f(x,y)=f(y,x), whatever.)
Then, based on these proofs, call the original program certified.
Alternatively, we could create a specification of a program in a proof assistant tool and then prove properties about the specification, but not the program itself.
In any case, I'm still interested in hearing alternative definitions if anyone has them.
I agree that the notion seems vague, but in my understanding a certified program is a program equipped/together with the proof of correctness. Now, by using rich and expressive type signatures you can make it so there is no need for a separate proof, but this is often only a matter of convenience. The real issue is what do we mean by correctness: this a matter of specification. You can take a look at e.g. Xavier Leroy. Formal verification of a realistic compiler.
First note that the phrase "certified" has a slightly French bias: elsewhere the expression "verified" or "proven" is often used.
In any case it is important to ask what that actually means. X. Leroy and CompCert is a very good starting point: it is a big project about C compiler verification, and Leroy is always keen to explain to his audience why verification matters. Especially when talking to people from "certification agencies" who usually mean testing, not proving.
Another big verification project is L4.verified which uses Isabelle/HOL. This part of the exposition explains a bit what is actually stated and proven, and what are the consequences. Unfortunately, the actual proof is top secret, so it cannot be checked publicly.
A certified program is a program that is paired with a proof that the program satisfies its specification, i.e., a certificate. The key is that there exists a proof object that can be checked independently of the tool that produced the proof.
A verified program has undergone verification, which in this context may typically mean that its specification has been formalized and proven correct in a system like Coq, but the proof is not necessarily certified by an external tool.
This distinction is well attested in the scientific literature and is not specific to Francophones. Xavier Leroy describes it very clearly in Section 2.2 of A formally verified compiler back-end.
My understanding is that "certified" in this sense is, as Makarius pointed out, an English word chosen by Francophones where native speakers might instead have used "formally verified". Coq was developed in France, and has many Francophone developers and users.
As to what "formal verification" means, Wikipedia notes (license: CC BY-SA 3.0) that it:
is the act of proving ... the correctness of intended algorithms underlying a system with respect to a certain formal specification or property, using formal methods of mathematics.
(I realise you would like a much more precise definition than this. I hope to update this answer in future, if I find one.)
Wikipedia especially notes the difference between verification and validation:
Validation: "Are we trying to make the right thing?", i.e., is the product specified to the user's actual needs?
Verification: "Have we made what we were trying to make?", i.e., does the product conform to the specifications?
The landmark paper seL4: Formal Verification of an OS Kernel (Klein, et al., 2009) corroborates this interpretation:
A cynic might say that an implementation proof only shows that the
implementation has precisely the same bugs that the specification
contains. This is true: the proof does not guarantee that the
specification describes the behaviour the user expects. The
difference [in a verified approach compared to a non-verified one]
is the degree of abstraction and the absence of whole classes of bugs.
Which classes of bugs are those? The Agda tutorial gives some idea:
no runtime errors (inevitable errors like I/O errors are handled; others are excluded by design).
no non-productive infinite loops.
It may means free of runtime error (numeric overflow, invalid references …), which is already good compared to most developed software, while still weak. The other meaning is proved to be correct according to a domain formalization; that is, it does not only have to be formally free of runtime errors, it also has to be proved to do what it's expected to do (which must have been precisely defined).
Soooo...
Semigroups, Monoids, Monads, Functors, Lenses, Catamorphisms, Anamorphisms, Arrows... These all sound good, and after an exercise or two (or ten), you can grasp their essence. And with Scalaz, you get them for free...
However, in terms of real-world programming, I find myself struggling to find usages to these notions. Yes, of course I always find someone on the web using Monads for IO or Lenses in Scala, but... still...
What I am trying to find is something along the "prescriptive" lines of a pattern. Something like: "here, you are trying to solves this, and one good way to solve it is by using lenses this way!"
Suggestions?
Update: Something along these lines, with a book or two, would be great (thanks Paul): Examples of GoF Design Patterns in Java's core libraries
The key to functional programming is abstraction, and composability of abstractions. Monads, Arrows, Lenses, these are all abstractions which have proven themselves useful, mostly because they are composable. You've asked for a "prescriptive" answer, but I'm going to say no. Perhaps you're not convinced that functional programming matters?
I'm sure plenty of people on StackOverflow would be more than happy to try and help you solve a specific problem the FP way. Have a list of stuff and you want to traverse the list and build up some result? Use a fold. Want to parse XML? hxt uses arrows for that. And monads? Well, tons of data types turn out to be Monads, so learn about them and you'll discover a wealth of ways you can manipulate these data types. But its kind of hard to just pull examples out of thin air and say "lenses are the Right Way to do this", "monoids are the best way to do that", etc. How would you explain to a newbie what the use of a for loop is? If you want to [blank], then use a for loop [in this way]. It's so general; there are tons of ways to use a for loop. The same goes for these FP abstractions.
If you have many years of OOP experience, then don't forget you were once a newbie at OOP. It takes time to learn the FP way, and even more time to unlearn some OOP tendencies. Give it time and you will find plenty of uses for a Functional approach.
I gave a talk back in September focused on the practical application of monoids and applicative functors/monads via scalaz.Validation. I gave another version of the same talk at the scala Lift Off, where the emphasis was more on the validation. I would watch the first talk until I start on validations and then skip to the second talk (27 minutes in).
There's also a gist I wrote which shows how you might use Validation in a "practical" application. That is, if you are designing software for nightclub bouncers.
I think you can take the reverse approach and instead when writing a small piece of functionality, ask yourself whether any of those would apply: Semigroups, Monoids, Monads, Functors, Lenses, Catamorphisms, Anamorphisms, Arrows... A lots of those concepts can be used in a local way.
Once you start down that route, you may see usage everywhere. For me, I sort of get Semigroups, Monoids, Monads, Functors. So take the example of answering this question How do I populate a list of objects with new values. It's a real usage for the person asking the question (a self described noob). I am trying to answer in a simple way but I have to refrain myself from scratching the itch "there are monoids in here".
Scratching it now: using foldMap and the fact that Int and List are monoids and that the monoid property is preserved when dealing with tuple, maps and options:
// using scalaz
listVar.sliding(2).toList.foldMap{
case List(prev, i) => Some(Map(i -> (1, Some(List(math.abs(i - prev))))))
case List(i) => Some(Map(i -> (1, None)))
case _ => None
}.map(_.mapValues{ case (count, gaps) => (count, gaps.map(_.min)) })
But I don't come to that result by thinking I will use hard core functional programming. It comes more naturally by thinking this seems simpler if I compose those monoids combined with the fact that scalaz has utility methods like foldMap. Interestingly when looking at the resulting code it's not obvious that I'm totally thinking in terms of monoid.
You might like this talk by Chris Marshall. He covers a couple of Scalaz goodies - namely Monoid and Validation - with many practical examples. Ittay Dror has written a very accessible post on how Functor, Applicative Functor, and Monad can be useful in practice. Eric Torreborre and Debasish Gosh's blogs also have a bunch of posts covering use cases for categorical constructs.
This answer just lists a few links instead of providing some real substance here. (Too lazy to write.) Hope you find it helpful anyway.
I understand your situation, but you will find that to learn functional programming you will need to adjust your point of view to the documentation you find, instead of the other way around. Luckily in Scala you have the possibility of becoming a functional programmer gradually.
To answer your questions and explain the point-of-view difference, I need to distinguish between "type classes" (monoids, functors, arrows), mathematically called "structures", and generic operations or algorithms (catamorphisms or folds, anamorphisms or unfolds, etc.). These two often interact, since many generic operations are defined for specific classes of data types.
You look for prescriptive answers similar to design patterns: when does this concept apply? The truth is that you have surely seen the prescriptive answers, and they are simply the definitions of the different concepts. The problem (for you) is that those answers are intrinsically different from design patterns, but it is so for good reasons.
On the one hand, generic algorithms are not design patterns, which suggest a structure for the code you write; they are abstractions defined in the language which you can directly apply. They are general descriptions for common algorithms which you already implement today, but by hand. For instance, whenever you are computing the maximum element of a list by scanning it, you are hardcoding a fold; when you sum elements, you are doing the same; and so on. When you recognize that, you can declare the essence of the operation you are performing by calling the appropriate fold function. This way, you save code and bugs (no opportunity for off-by-one errors), and you save the reader the effort to read all the needed code.
On the other hand, structures concern not the goal you have in mind but properties of the entities you are modeling. They are more useful for bottom-up software construction, rather than top-down: when defining your data, you can declare that it is a e.g. a monoid. Later, when processing your data, you have the opportunity to use operations on e.g. monoids to implement your processing. In some cases it is useful to strive to express your algorithm in terms of the predefined ones. For instance, very often if you need to reduce a tree to a single value, a fold can do most or all of what you need. Of course, you can also declare that your data type is a monoid when you need a generic algorithm on monoids; but the earlier you notice that, the earlier you can start reusing generic algorithms for monoids.
Last advice is that probably most of the documentation you will find about these concepts concerns Haskell, because this language has been around for much more time and supports them in a quite elegant way. Quite recommended here are Learn you a Haskell for Great Good, a Haskell course for beginners, where among others chapters 11 to 14 focus on some type classes, and Typeclassopedia (which contains links to various articles with specific examples). EDIT: Finally, an example of applications of Monoids, taken from Typeclassopedia, is here: http://apfelmus.nfshost.com/articles/monoid-fingertree.html. I'm not saying there is little documentation for Scala, just that there is more in Haskell, and Haskell is where the application of these concepts to programming was born.
I'm going to be giving a short (30-40 mins) lunch-time talk on Scala to technical staff at my company. I'd like some suggestions for what would be the most appropriate content. Most people attending will have experience in Java and/or C# (plus various other languages).
What are the key things to cover? I'd like to give a brief introduction to the Scala syntax so that people don't feel lost when looking at code examples. I'll also cover some of the history behind the language and its designers. What would help people get the most out of the talk?
People are almost certainly coming to talk to get an answer to the question, "Why should I use Scala?" Anything you can provide to help them answer that will be valuable.
Keep the discussion of the history and the personalities behind Scala to a minimum.
A whirlwind tour of the syntax is useful, but keep it short.
Spend a good chunk of the talk demonstrating examples and comparisons to Java. Show cases where Scala shines. You should literally be running and executing code so that people get a real, hands-on feel for how things work.
Make sure to cover weaknesses, too! Provide an objective and balanced overview.
I gave a similar talk - mostly to those with a Java background. I felt that taking a piece of real Java (about 30 lines) and iteratively adding scala features worked pretty well. The 30 lines of Java eventually ended up as 6 (six!) of scala. The point being (of course) that 6 lines are more readable and maintainable than 30.
I converted the scala to line-by-line Java equivalent and then introduced:
Type inference
Option
Closures
Pattern-matching (on lists)
Type aliases
Tail recursion
I found that this segment took quite a long time because the audience were very interested in the minutiae of scala's syntax (especially around function-expressions). Before undertaking the pattern-matching bit, I had a slide explaining the various things you could use in a match.
Tough. One has to balance the new and the familiar. For instance:
Talk about traits, how they differ from interfaces and multiple inheritance. Note that most methods in all of Scala collections can actually be found on the trait Traversable, which has a single abstract method: foreach.
Speak of functions and partial functions, show map/filter/foreach, and how they make use of functions.
Talk about pattern matching -- show how unapply is used to enable representation independence, while at the same time case classes make the common case easy.
Above all AVOID any topic that might be difficult to understand quickly, or you may waste time on them. For example of great topics I wouldn't talk about: self types, variance, for-comprehensions.
Pick more topics than you have time for. Let the public steer the conversation towards the topcis they are more interested in. If anyone starts to boggle down a topic too much, say you'll be pleased to explain it in more details later, and ask if they would mind if you moved to another topic. On the other hand, if everyone seems to be picking up on one thing in particular, stay with it. Otherwise, it might feel like you want to hide something.
I gave a presentation on re-writing Java classes in Scala. It has lots of examples of Java -> Scala and (hopefully) makes the gains obvious. Feel free to borrow any content you want... presentation took 1hr 10minutes so you might want to cut some stuff out.
Presentation: http://www.colinhowe.co.uk/downloads/rewriting-java-in-scala.ppt
Video: http://skillsmatter.com/podcast/java-jee/re-writing-java-classes-in-scala-and-making-your-code-lovely
You could do worse than running through Jonas Bonér's presentation, Pragmatic Real-World Scala. Perhaps skip some advanced topics in there on different applications of traits and self-type annotations.