What does "existential type" mean in Swift? - swift

Question

I am reading Swift Evolution proposal 244 (Opaque Result Types) and don't understand what the following means:
... existential type ...
One could compose these transformations by
using the existential type Shape instead of generic arguments, but
doing so would imply more dynamism and runtime overhead than may be
desired.

Answer 1

An example of an existential type is given in the evolution proposal itself:
protocol Shape {
func draw(to: Surface)
}
An example of using protocol Shape as an existential type would look like
func collides(with: any Shape) -> Bool
as opposed to using a generic argument Other:
func collides<Other: Shape>(with: Other) -> Bool
Important to note here that the Shape protocol is not an existential type by itself, only using it in "protocols-as-types" context as above "creates" an existential type from it. See this post from the member of Swift Core Team:
Also, protocols currently do double-duty as the spelling for existential types, but this relationship has been a common source of confusion.
Also, citing the Swift Generics Evolution article (I recommend reading the whole thing, which explains this in more details):
The best way to distinguish a protocol type from an existential type is to look at the context. Ask yourself: when I see a reference to a protocol name like Shape, is it appearing at a type level, or at a value level? Revisiting some earlier examples, we see:
func addShape<T: Shape>() -> T
// Here, Shape appears at the type level, and so is referencing the protocol type
var shape: any Shape = Rectangle()
// Here, Shape appears at the value level, and so creates an existential type
Deeper dive
Why is it called an "existential"? I never saw an unambiguous confirmation of this, but I assume that the feature is inspired by languages with more advanced type systems, e.g. consider Haskell's existential types:
class Buffer -- declaration of type class `Buffer` follows here
data Worker x y = forall b. Buffer b => Worker {
buffer :: b,
input :: x,
output :: y
}
which is roughly equivalent to this Swift snippet (if we assume that Swift's protocols more or less represent Haskell's type classes):
protocol Buffer {}
struct Worker<X, Y> {
let buffer: any Buffer
let input: X
let output: Y
}
Note that the Haskell example used forall quantifier here. You could read this as "for all types that conform to the Buffer type class ("protocol" in Swift) values of type Worker would have exactly the same types as long as their X and Y type parameters are the same". Thus, given
extension String: Buffer {}
extension Data: Buffer {}
values Worker(buffer: "", input: 5, output: "five") and Worker(buffer: Data(), input: 5, output: "five") would have exactly the same types.
This is a powerful feature, which allows things such as heterogenous collections, and can be used in a lot more places where you need to "erase" an original type of a value and "hide" it under an existential type. Like all powerful features it can be abused and can make code less type-safe and/or less performant, so should be used with care.
If you want even a deeper dive, check out Protocols with Associated Types (PATs), which currently can't be used as existentials for various reasons. There are also a few Generalized Existentials proposals being pitched more or less regularly, but nothing concrete as of Swift 5.3. In fact, the original Opaque Result Types proposal linked by the OP can solve some of the problems caused by use of PATs and significantly alleviates lack of generalized existentials in Swift.

Answer 2

I'm sure you've already used existentials a lot before without noticing it.
A summarized answer of Max is that for:
var rec: Shape = Rectangle() // Example A
only Shape properties can be accessed. While for:
func addShape<T: Shape>() -> T // Example B
Any property of T can be accessed. Because T adopts Shape then all properties of Shape can also be accessed as well.
The first example is an existential the second is not.
Example of real code:
protocol Shape {
var width: Double { get }
var height: Double { get }
}
struct Rectangle: Shape {
var width: Double
var height: Double
var area: Double
}
let rec1: Shape = Rectangle(width: 1, height: 2, area: 2)
rec1.area // ❌
However:
let rec2 = Rectangle(width: 1, height: 2, area: 2)
func addShape<T: Shape>(_ shape: T) -> T {
print(type(of: shape)) // Rectangle
return shape
}
let rec3 = addShape(rec2)
print(rec3.area) // ✅
I'd argue that for most Swift users, we all understand Abstract class and Concrete class. This extra jargon makes it slightly confusing.
The trickiness is that with the 2nd example, to the compiler, the type you return isn't Shape, it's Rectangle i.e. the function signature transforms to this:
func addShape(_ shape: Rectangle) -> Rectangle {
This is only possible because of (constrained) generics.
Yet for rec: Shape = Whatever() to the compiler the type is Shape regardless of the assigning type. <-- Box Type
Why is it named Existential?
The term "existential" in computer science and programming is borrowed from philosophy, where it refers to the concept of existence and being. In the context of programming, "existential" is used to describe a type that represents the existence of any specific type, without specifying which type it is.
The term is used to reflect the idea that, by wrapping a value in an existential type, you are abstracting away its specific type and only acknowledging its existence.
In other words, an existential type provides a way to handle values of different types in a unified way, while ignoring their specific type information†. This allows you to work with values in a more generic and flexible manner, which can be useful in many situations, such as when creating collections of heterogeneous values, or when working with values of unknown or dynamic types.
The other day I took my kid to an ice-cream shop. She asked what are you having, and I didn't know the flavor I picked, so I didn't say it's strawberry flavored or chocolate, I just said "I'm having an ice-cream".
I just specified that it's an ice-cream without saying its flavor. My daughter could no longer determine if it was Red, or Brown. If it was having a fruit flavor or not. I gave her existential-like information.
Had I told her it's a chocolate, then I would have gave her specific information. Which is then not existential.
†: In Example B, we're not ignoring the specific type information.
Special thanks to a friend who helped me come up with this answer

Answer 3

I feel like it's worth adding something about why the phrase is important in Swift. And in particular, I think almost always, Swift is talking about "existential containers". They talk about "existential types", but only really with reference to "stuff that is stored in an existential container". So what is an "existential container"?
As I see it, the key thing is, if you have a variable that you're passing as a parameter or using locally, etc. and you define the type of the variable as Shape then Swift has to do some things under the hood to make it work and that's what they are (obliquely) referring to.
If you think about defining a function in a library/framework module that you're writing that is publicly available and takes for example the parameters public func myFunction(shape1: Shape, shape2: Shape, shape1Rotation: CGFloat?) -> Shape... imagine it (optionally) rotates shape1, "adds" it to shape2 somehow (I leave the details up to your imagination) then returns the result. Coming from other OO languages, we instinctively think that we understand how this works... the function must be implemented only with members available in the Shape protocol.
But the question is for the compiler, how are the parameters represented in memory? Instinctively, again, we think... it doesn't matter. When someone writes a new program that uses your function at some point in the future, they decide to pass their own shapes in and define them as class Dinosaur: Shape and class CupCake: Shape. As part of defining those new classes, they will have to write implementations of all the methods in protocol Shape, which might be something like func getPointsIterator() -> Iterator<CGPoint>. That works just fine for classes. The calling code defines those classes, instantiates objects from them, passes them into your function. Your function must have something like a vtable (I think Swift calls it a witness table) for the Shape protocol that says "if you give me an instance of a Shape object, I can tell you exactly where to find the address of the getPointsIterator function". The instance pointer will point to a block of memory on the stack, the start of which is a pointer to the class metadata (vtables, witness tables, etc.) So the compiler can reason about how to find any given method implementation.
But what about value types? Structs and enums can have just about any format in memory, from a one byte Bool to a 500 byte complex nested struct. These are usually passed on the stack or registers on function calls for efficiency. (When Swift exactly knows the type, all code can be compiled knowing the data format and passed on the stack or in registers, etc.)
Now you can see the problem. How can Swift compile the function myFunction so it will work with any possible future value type/struct defined in any code? As I understand it, this is where "existential containers" come in.
The simplest approach would be that any function that takes parameters of one of these "existential types" (types defined just by conforming to a Protocol) must insist that the calling code "box" the value type... that it store the value in a special reference counted "box" on the heap and pass a pointer to this (with all the usual ARC retain/release/autorelease/ownership rules) to your function when the function takes a parameter of type Shape.
Then when a new, weird and wonderful, type is written by some code author in the future, compiling the methods of Shape would have to include a way to accept "boxed" versions of the type. Your myFunction would always handle these "existential types" by handling the box and everything works. I would guess that C# and Java do something like this (boxing) if they have the same problem with non class types (Int, etc.)?
The thing is that for a lot of value types, this can be very inefficient. After all, we are compiling mostly for 64 bit architecture, so a couple of registers can handle 8 bytes, enough for many simple structures. So Swift came up with a compromise (again I might be a bit inaccurate on this, I'm giving my idea of the mechanism... feel free to correct). They created "existential containers" that are always 4 pointers in size. 16 bytes on a "normal" 64 bit architecture (most CPUs that run Swift these days).
If you define a struct that conforms to a protocol and it contains 12 bytes or less, then it is stored in the existential container directly. The last 4 byte pointer is a pointer to the type information/witness tables/etc. so that myFunction can find an address for any function in the Shape protocol (just like in the classes case above). If your struct/enum is larger than 12 bytes then the 4 pointer value points to a boxed version of the value type. Obviously this was considered an optimum compromise, and seems reasonable... it will be passed around in 4 registers in most cases or 4 stack slots if "spilled".
I think the reason the Swift team end up mentioning "existential containers" to the wider community is because it then has implications for various ways of using Swift. One obvious implication is performance. There's a sudden performance drop when using functions in this way if the structs are > 12 bytes in size.
Another, more fundamental, implication I think is that protocols can be used as parameters only if they don't have protocol or Self requirements... they are not generic. Otherwise you're into generic function definitions which is different. That's why we sometimes need to change things like: func myFunction(shape: Shape, reflection: Bool) -> Shape into something like func myFunction<S:Shape>(shape: S, reflection: Bool) -> S. They are implemented in very different ways under the hood.