Idiomatic callbacks in Rust - callback

Question

In C/C++ I'd normally do callbacks with a plain function pointer, maybe passing a void* userdata parameter too. Something like this:
typedef void (*Callback)();
class Processor
{
public:
void setCallback(Callback c)
{
mCallback = c;
}
void processEvents()
{
//...
mCallback();
}
private:
Callback mCallback;
};
What is the idiomatic way of doing this in Rust? Specifically, what types should my setCallback() function take, and what type should mCallback be? Should it take an Fn? Maybe FnMut? Do I save it Boxed? An example would be amazing.

Answer 1

Short answer: For maximum flexibility, you can store the callback as a boxed FnMut object, with the callback setter generic on callback type. The code for this is shown in the last example in the answer. For a more detailed explanation, read on.
"Function pointers": callbacks as fn
The closest equivalent of the C++ code in the question would be declaring callback as a fn type. fn encapsulates functions defined by the fn keyword, much like C++'s function pointers:
type Callback = fn();
struct Processor {
callback: Callback,
}
impl Processor {
fn set_callback(&mut self, c: Callback) {
self.callback = c;
}
fn process_events(&self) {
(self.callback)();
}
}
fn simple_callback() {
println!("hello world!");
}
fn main() {
let p = Processor {
callback: simple_callback,
};
p.process_events(); // hello world!
}
This code could be extended to include an Option<Box<Any>> to hold the "user data" associated with the function. Even so, it would not be idiomatic Rust. The Rust way to associate data with a function is to capture it in an anonymous closure, just like in modern C++. Since closures are not fn, set_callback will need to accept other kinds of function objects.
Callbacks as generic function objects
In both Rust and C++ closures with the same call signature come in different sizes to accommodate the different values they might capture. Additionally, each closure definition generates a unique anonymous type for the closure's value. Due to these constraints, the struct cannot name the type of its callback field, nor can it use an alias.
One way to embed a closure in the struct field without referring to a concrete type is by making the struct generic. The struct will automatically adapt its size and the type of callback for the concrete function or closure you pass to it:
struct Processor<CB>
where
CB: FnMut(),
{
callback: CB,
}
impl<CB> Processor<CB>
where
CB: FnMut(),
{
fn set_callback(&mut self, c: CB) {
self.callback = c;
}
fn process_events(&mut self) {
(self.callback)();
}
}
fn main() {
let s = "world!".to_string();
let callback = || println!("hello {}", s);
let mut p = Processor { callback };
p.process_events();
}
As before, set_callback() will accept functions defined with fn, but this one will also accept closures as || println!("hello world!"), as well as closures that capture values, such as || println!("{}", somevar). Because of this the processor doesn't need userdata to accompany the callback; the closure provided by the caller of set_callback will automatically capture the data it needs from its environment and have it available when invoked.
But what's the deal with the FnMut, why not just Fn? Since closures hold captured values, Rust's usual mutation rules must apply when calling the closure. Depending on what the closures do with the values they hold, they are grouped in three families, each marked with a trait:
Fn are closures that only read data, and may be safely called multiple times, possibly from multiple threads. Both above closures are Fn.
FnMut are closures that modify data, e.g. by writing to a captured mut variable. They may also be called multiple times, but not in parallel. (Calling a FnMut closure from multiple threads would lead to a data race, so it can only be done with the protection of a mutex.) The closure object must be declared mutable by the caller.
FnOnce are closures that consume some of the data they capture, e.g. by passing a captured value to a function that takes it by value. As the name implies, these may be called only once, and the caller must own them.
Somewhat counter-intuitively, when specifying a trait bound for the type of an object that accepts a closure, FnOnce is actually the most permissive one. Declaring that a generic callback type must satisfy the FnOnce trait means that it will accept literally any closure. But that comes with a price: it means the holder is only allowed to call it once. Since process_events() may opt to invoke the callback multiple times, and as the method itself may be called more than once, the next most permissive bound is FnMut. Note that we had to mark process_events as mutating self.
Non-generic callbacks: function trait objects
Even though the generic implementation of the callback is extremely efficient, it has serious interface limitations. It requires each Processor instance to be parameterized with a concrete callback type, which means that a single Processor can only deal with a single callback type. Given that each closure has a distinct type, the generic Processor cannot handle proc.set_callback(|| println!("hello")) followed by proc.set_callback(|| println!("world")). Extending the struct to support two callbacks fields would require the whole struct to be parameterized to two types, which would quickly become unwieldy as the number of callbacks grows. Adding more type parameters wouldn't work if the number of callbacks needed to be dynamic, e.g. to implement an add_callback function that maintains a vector of different callbacks.
To remove the type parameter, we can take advantage of trait objects, the feature of Rust that allows automatic creation of dynamic interfaces based on traits. This is sometimes referred to as type erasure and is a popular technique in C++[1][2], not to be confused with Java and FP languages' somewhat different use of the term. Readers familiar with C++ will recognize the distinction between a closure that implements Fn and an Fn trait object as equivalent to the distinction between general function objects and std::function values in C++.
A trait object is created by borrowing an object with the & operator and casting or coercing it to a reference to the specific trait. In this case, since Processor needs to own the callback object, we cannot use borrowing, but must store the callback in a heap-allocated Box<dyn Trait> (the Rust equivalent of std::unique_ptr), which is functionally equivalent to a trait object.
If Processor stores Box<dyn FnMut()>, it no longer needs to be generic, but the set_callback method now accepts a generic c via an impl Trait argument. As such, it can accept any kind of callable, including closures with state, and properly box it before storing it in the Processor. The generic argument to set_callback doesn't limit what kind of callback the processor accepts, as the type of the accepted callback is decoupled from the type stored in the Processor struct.
struct Processor {
callback: Box<dyn FnMut()>,
}
impl Processor {
fn set_callback(&mut self, c: impl FnMut() + 'static) {
self.callback = Box::new(c);
}
fn process_events(&mut self) {
(self.callback)();
}
}
fn simple_callback() {
println!("hello");
}
fn main() {
let mut p = Processor {
callback: Box::new(simple_callback),
};
p.process_events();
let s = "world!".to_string();
let callback2 = move || println!("hello {}", s);
p.set_callback(callback2);
p.process_events();
}
Lifetime of references inside boxed closures
The 'static lifetime bound on the type of the c argument accepted by set_callback is a simple way to convince the compiler that references contained in c, which might be a closure that refers to its environment, only refer to global values and will therefore remain valid throughout the use of the callback. But the static bound is also very heavy-handed: while it accepts closures that own objects just fine (which we've ensured above by making the closure move), it rejects closures that refer to local environment, even when they only refer to values that outlive the processor and would in fact be safe.
As we only need the callbacks alive as long as the processor is alive, we should try to tie their lifetime to that of the processor, which is a less strict bound than 'static. But if we just remove the 'static lifetime bound from set_callback, it no longer compiles. This is because set_callback creates a new box and assigns it to the callback field defined as Box<dyn FnMut()>. Since the definition doesn't specify a lifetime for the boxed trait object, 'static is implied, and the assignment would effectively widen the lifetime (from an unnamed arbitrary lifetime of the callback to 'static), which is disallowed. The fix is to provide an explicit lifetime for the processor and tie that lifetime to both the references in the box and the references in the callback received by set_callback:
struct Processor<'a> {
callback: Box<dyn FnMut() + 'a>,
}
impl<'a> Processor<'a> {
fn set_callback(&mut self, c: impl FnMut() + 'a) {
self.callback = Box::new(c);
}
// ...
}
With these lifetimes being made explicit, it is no longer necessary to use 'static. The closure can now refer to the local s object, i.e. no longer has to be move, provided that the definition of s is placed before the definition of p to ensure that the string outlives the processor.

Answer 2

If you are willing to deal with lifetime and cannot afford the heap allocation, then here is an implementation that uses reference to implement the callback:
use core::ffi::c_void;
use core::mem::transmute;
use core::ptr::null_mut;
use core::marker::PhantomData;
/// ErasedFnPointer can either points to a free function or associated one that
/// `&mut self`
struct ErasedFnPointer<'a, T, Ret> {
struct_pointer: *mut c_void,
fp: *const (),
// The `phantom_*` field is used so that the compiler won't complain about
// unused generic parameter.
phantom_sp: PhantomData<&'a ()>,
phantom_fp: PhantomData<fn(T) -> Ret>,
}
impl<'a, T, Ret> Copy for ErasedFnPointer<'a, T, Ret> {}
impl<'a, T, Ret> Clone for ErasedFnPointer<'a, T, Ret> {
fn clone(&self) -> Self {
*self
}
}
impl<'a, T, Ret> ErasedFnPointer<'a, T, Ret> {
pub fn from_associated<S>(struct_pointer: &'a mut S, fp: fn(&mut S, T) -> Ret)
-> ErasedFnPointer<'a, T, Ret>
{
ErasedFnPointer {
struct_pointer: struct_pointer as *mut _ as *mut c_void,
fp: fp as *const (),
phantom_sp: PhantomData,
phantom_fp: PhantomData,
}
}
pub fn from_free(fp: fn(T) -> Ret) -> ErasedFnPointer<'static, T, Ret> {
ErasedFnPointer {
struct_pointer: null_mut(),
fp: fp as *const (),
phantom_sp: PhantomData,
phantom_fp: PhantomData,
}
}
pub fn call(&self, param: T) -> Ret {
if self.struct_pointer.is_null() {
let fp = unsafe { transmute::<_, fn(T) -> Ret>(self.fp) };
fp(param)
} else {
let fp = unsafe { transmute::<_, fn(*mut c_void, T) -> Ret>(self.fp) };
fp(self.struct_pointer, param)
}
}
}
fn main() {
let erased_ptr = ErasedFnPointer::from_free(|x| {
println!("Hello, {}", x);
x
});
erased_ptr.call(2333);
println!("size_of_val(erased_ptr) = {}", core::mem::size_of_val(&erased_ptr));
ErasedFnPointer::from_associated(
&mut Test { x: 1},
Test::f
).call(1);
let mut x = None;
ErasedFnPointer::from_associated(&mut x, |x, param| {
*x = Some(param);
println!("{:#?}", x);
}).call(1);
}
struct Test {
x: i32
}
impl Test {
fn f(&mut self, y: i32) -> i32 {
let z = self.x + y;
println!("Hello from Test, {}", z);
z
}
}

Answer 3

For the type of scenario where a callback is used, you should consider the Promise alternative.
It is much easier to use than callbacks because it avoids nesting (callback hell).
Consider this:
fn main() {
let fut = do_async(&Calculation{ value: 12 });
let resp = fut().unwrap(); // call fut() to wait for the respbnse
println!("{}", resp);
}
For any calculation:
you define an struct whose fields are its inputs (name is not important).
you implement the Runner trait:
you choose what to return
you write the code of run(), which will be exected by a separate thread
struct Calculation { // <---- choose: name
value: i32 // <----- choose: inputs for your async work
}
impl Runner for Calculation {
type ReturnType = i32; // <--- choose: calculation return type
fn run(&self) -> Option<Self::ReturnType> { // <-- implement: code executed by a thread
println!("async calculation starts");
thread::sleep(Duration::from_millis(3000));
return Some(self.value * 2);
}
}
Finaly, this is the "magic":
trait Runner: Send + Sync {
type ReturnType: Send; // associated type
fn run(&self) -> Option<Self::ReturnType>;
}
fn do_async<TIn: Runner>(f: &'static TIn) -> impl FnOnce()-> Option<TIn::ReturnType> {
let (sender, receiver) = channel::<Option<TIn::ReturnType>>();
let hand = thread::spawn(move || {
sender.send(f.run()).unwrap();
});
let f = move || -> Option<TIn::ReturnType> {
let res = receiver.recv().unwrap();
hand.join().unwrap();
return res;
};
return f;
}

Answer 4

A simpler version of https://stackoverflow.com/a/70943671/286335 just for closures.
fn main() {
let n = 2;
let fut = do_async(move || {
thread::sleep(Duration::from_millis(3000));
return n * 1234;
});
let resp = fut(); // call fut() to wait for the response
println!("{}", resp);
} // ()
where do_async is
fn do_async<TOut, TFun>(foo: TFun) -> (impl FnOnce() -> TOut)
where
TOut: Send + Sync + 'static,
TFun: FnOnce() -> TOut + Send + Sync + 'static,
{
let (sender, receiver) = channel::<TOut>();
let hand = thread::spawn(move || {
sender.send(foo()).unwrap();
});
let f = move || -> TOut {
let res = receiver.recv().unwrap();
hand.join().unwrap();
return res;
};
return f;
} // ()