I am writing a toy software library in Rust that needs to be able to load images of almost any type into an internal data structure for the image. It is early days for the Rust ecosystem, and there is no one library/set of bindings that I would trust for this task.
I would ideally like:
Support multiple redundant external libraries that may or may not be available at runtime
Support multiple redundant external libraries that may or may not be available at compile-time.
Include at least one fallback implementation that ships with my code.
Fully encapsulate all of the file loading stuff behind a function that does path -> InternalImage loading.
Is there a best practice way to implement optional dependencies like this in Rust? Some of the libraries will be Rust, and some of them will probably be C libraries with Rust bindings.
Cargo, the Rust package manager, can help with that. It lets you declare optional compile-time dependencies. See the [features] section of Cargo's documentation.
For runtime dependencies I'm not sure. I think std::dynamic_lib could be helpful. See an example of using DynamicLibrary in a previous SO question.
Related
In JavaScript we have estree which is the AST definition evolving from Mozilla's implementation. But nowadays if you build an AST transformer in JS of a JS AST, you probably use this structure. Do we have anything like this for Swift, of the Swift AST?
I see we have a grammar, but what about an AST? I guess I can make one from that, but still.
If nothing standard, do we have any AST examples?
The Swift project offers SwiftSyntax as a package for working with Swift source code, in Swift. Under the hood, it's powered by the compiler's own libSyntax, written in C++.
Note that this currently isn't the representation that the Swift compiler proper uses for actually compiling Swift code — libSyntax focuses on source code itself for rewriting, formatting, transformations, etc., but is largely void of the semantic information that you may find in a compiler AST necessary for transforming the source into machine code. If you're just looking to operate on the AST without those semantics, this may be sufficient for your use case.
The repo README should have some info to get you started, an example, and some real-world use cases showing concrete usage of the library.
I want to outsource some code for a plugin system. Inside my project, I have a trait called Provider which is the code for my plugin system. If you activate the feature "consumer" you can use plugins; if you don't, you are an author of plugins.
I want authors of plugins to get their code into my program by compiling to a shared library. Is a shared library a good design decision? The limitation of the plugins is using Rust anyway.
Does the plugin host have to go the C way for loading the shared library: loading an unmangled function?
I just want authors to use the trait Provider for implementing their plugins and that's it.
After taking a look at sharedlib and libloading, it seems impossible to load plugins in a idiomatic Rust way.
I'd just like to load trait objects into my ProviderLoader:
// lib.rs
pub struct Sample { ... }
pub trait Provider {
fn get_sample(&self) -> Sample;
}
pub struct ProviderLoader {
plugins: Vec<Box<Provider>>
}
When the program is shipped, the file tree would look like:
.
├── fancy_program.exe
└── providers
├── fp_awesomedude.dll
└── fp_niceplugin.dll
Is that possible if plugins are compiled to shared libs? This would also affect the decision of the plugins' crate-type.
Do you have other ideas? Maybe I'm on the wrong path so that shared libs aren't the holy grail.
I first posted this on the Rust forum. A friend advised me to give it a try on Stack Overflow.
UPDATE 3/27/2018:
After using plugins this way for some time, I have to caution that in my experience things do get out of sync, and it can be very frustrating to debug (strange segfaults, weird OS errors). Even in cases where my team independently verified the dependencies were in sync, passing non-primitive structs between the dynamic library binaries tended to fail on OS X for some reason. I'd like to revisit this, find what cases it happens in, and perhaps open an issue with Rust, but I'm going to advise caution with this going forward.
LLDB and valgrind are near-essential to debug these issues.
Intro
I've been investigating things along these lines myself, and I've found there's little official documentation for this, so I decided to play around!
First let me note, as there is little official word on these properties please do not rely on any code here if you're trying to keep planes in the air or nuclear missiles from errantly launching, at least not without doing far more comprehensive testing than I've done. I'm not responsible if the code here deletes your OS and emails an erroneous tearful confession of committing the Zodiac killings to your local police; we're on the fringes of Rust here and things could change from one release or toolchain to another.
I have personally tested this on Rust 1.20 stable in both debug and release configurations on Windows 10 (stable-x86_64-pc-windows-msvc) and Cent OS 7 (stable-x86_64-unknown-linux-gnu).
Approach
The approach I took was a shared common crate both crates listed as a dependency defining common struct and trait definitions. At first, I was also going to test having a struct with the same structure, or trait with the same definitions, defined independently in both libraries, but I opted against it because it's too fragile and you wouldn't want to do it in a real design. That said, if anybody wants to test this, feel free to do a PR on the repository above and I will update this answer.
In addition, the Rust plugin was declared dylib. I'm not sure how compiling as cdylib would interact, since I think it would mean that upon loading the plugin there are two versions of the Rust standard library hanging around (since I believe cdylib statically links the Rust stdlib into the shared object).
Tests
General Notes
The structs I tested were not declared #repr(C). This could provide an extra layer of safety by guaranteeing a layout, but I was most curious about writing "pure" Rust plugins with as little "treating Rust like C" fiddling as possible. We already know you can use Rust via FFI by wrapping things in opaque pointers, manually dropping, and such, so it's not very enlightening to test this.
The function signature I used was pub fn foo(args) -> output with the #[no_mangle] directive, it turns out that rustfmt automatically changes extern "Rust" fn to simply fn. I'm not sure I agree with this in this case since they are most certainly "extern" functions here, but I will choose to abide by rustfmt.
Remember that even though this is Rust, this has elements of unsafety because libloading (or the unstable DynamicLib functionality) will not type check the symbols for you. At first I thought my Vec test was proving you couldn't pass Vecs between host and plugin until I realized on one end I had Vec<i32> and on the other I had Vec<usize>
Interestingly, there were a few times I pointed an optimized test build to an unoptimized plugin and vice versa and it still worked. However, I still can't in good faith recommending building plugins and host applications with different toolchains, and even if you do, I can't promise that for some reason rustc/llvm won't decide to do certain optimizations on one version of a struct and not another. In addition, I'm not sure if this means that passing types through FFI prevents certain optimizations such as Null Pointer Optimizations from occurring.
You're still limited to calling bare functions, no Foo::bar because of the lack of name mangling. In addition, due to the fact that functions with trait bounds are monomorphized, generic functions and structs are also out. The compiler can't know you're going to call foo<i32> so no foo<i32> is going to be generated. Any functions over the plugin boundary must take only concrete types and return only concrete types.
Similarly, you have to be careful with lifetimes for similar reasons, since there's no static lifetime checking Rust is forced to believe you when you say a function returns &'a when it's really &'b.
Native Rust
The first tests I performed were on no custom structures; just pure, native Rust types. This would give a baseline for if this is even possible. I chose three baseline types: &mut i32, &mut Vec, and Option<i32> -> Option<i32>. These were all chosen for very specific reasons: the &mut i32 because it tests a reference, the &mut Vec because it tests growing the heap from memory allocated in the host application, and the Option as a dual purpose of testing passing by move and matching a simple enum.
All three work as expected. Mutating the reference mutates the value, pushing to a Vec works properly, and the Option works properly whether Some or None.
Shared Struct Definition
This was meant to test if you could pass a non-builtin struct with a common definition on both sides between plugin and host. This works as expected, but as mentioned in the "General Notes" section, can't promise you Rust won't fail to optimize and/or optimize a structure definition on one side and not another. Always test your specific use case and use CI in case it changes.
Boxed Trait Object
This test uses a struct whose definition is only defined on the plugin side, but implements a trait defined in a common crate, and returns a Box<Trait>. This works as expected. Calling trait_obj.fun() works properly.
At first I actually anticipated there would be issues with dropping without making the trait explicitly have Drop as a bound, but it turns out Drop is properly called as well (this was verified by setting the value of a variable declared on the test stack via raw pointer from the struct's drop function). (Naturally I'm aware drop is always called even with trait objects in Rust, but I wasn't sure if dynamic libraries would complicate it).
NOTE:
I did not test what would happen if you load a plugin, create a trait object, then drop the plugin (which would likely close it). I can only assume this is potentially catastrophic. I recommend keeping the plugin open as long as the trait object persists.
Remarks
Plugins work exactly as you'd expect just linking a crate naturally, albeit with some restrictions and pitfalls. As long as you test, I think this is a very natural way to go. It makes symbol loading more bearable, for instance, if you only need to load a new function and then receive a trait object implementing an interface. It also avoids nasty C memory leaks because you couldn't or forgot to load a drop/free function. That said, be careful, and always test!
There is no official plugin system, and you cannot do plugins loaded at runtime in pure Rust. I saw some discussions about doing a native plugin system, but nothing is decided for now, and maybe there will never be any such thing. You can use one of these solutions:
You can extend your code with native dynamic libraries using FFI. To use the C ABI, you have to use repr(C), no_mangle attribute, extern etc. You will find more information by searching Rust FFI on the internets. With this solution, you must use raw pointers: they come with no safety guarantee (i.e. you must use unsafe code).
Of course, you can write your dynamic library in Rust, but to load it and call the functions, you must go through the C ABI. This means that the safety guarantees of Rust do not apply there. Furthermore, you cannot use the highest level Rust's functionalities as trait, enum, etc. between the library and the binary.
If you do not want this complexity, you can use a language adapted to expand Rust: with which you can dynamically add functions to your code and execute them with same guarantees as in Rust. This is, in my opinion, the easier way to go: if you have the choice, and if the execution speed is not critical, use this to avoid tricky C/Rust interfaces.
Here is a (not exhaustive) list of languages that can easily extend Rust:
Gluon, a functional language like Haskell
Dyon, a small but powerful scripting language intended for video games
Lua with rlua or hlua
You can also use Python or Javascript, or see the list in awesome-rust.
Does purescript have something like Haskell's System.Plugins?
I need to create some 'generic interface' (sorry for this, I've been programming in object oriented languages for almost 15 years) that other developers will be able to use just by putting a module file in a plugins directory.
I wonder if it is possible since as far as I know Purescript does not have any metadata carried with types at runtime.
From a cursory glance, Haskell's plugins package is about dynamic loading of Haskell code. The similar concept in JavaScript is eval or adding a script element to the DOM.
You can make any type assumption for eval'd code using a foreign import or unsafeCoerce. However, you must take care to ensure that the assumption is correct.
I am not aware of a purescript package oriented around these sorts of plugins. In my estimation there would be too much variability in what a plugin could be to really have a sole package for it.
I am using the MPJ-api for my current project. The two implementations I am using are MPJ-express and Fast-MPJ. However, since they both implement the same API, namely the MPJ-API, I cannot simultaneously support both implementations due to name-space collisions.
Is there any way to wrap two different libraries with the same package and class-names such that both can be supported at the same time in Java or Scala?
So far, the only way I can think of is to move the module into separate projects, but I am not sure this would be the way to go.
If your code use only a subset of MPI functions (like most of the MPI code I've reviewed), you can write an abstraction layer (traits or even Cake-Pattern) which defines the ops your are actually using. You can then implement a concrete adapter for each implementation.
This approach will also work with non-MPI communication layers (think Akka, JGroups, etc.)
As a bonus point, you could use the SLF4J approach: the correct implementation is chosen at runtime according to what's actually in the classpath.
I'm still at the beginning in learning scala in addition to java and i didn't get it how is one supposed to do DI there? can or should i use an existing DI library, should it be done manually or is there another way?
Standard Java DI frameworks will usually work with Scala, but you can also use language constructs to achieve the same effect without external dependencies.
A new dependency injection library specifically for Scala is Dick Wall's SubCut.
Whereas the Jonas Bonér article referenced in Dan Story's answer emphasizes compile-time bound instances and static injection (via mix-ins), SubCut is based on runtime initialization of immutable modules, and dynamic injection by querying the bound modules by type, string names, or scala.Symbol names.
You can read more about the comparison with the Cake pattern in the GettingStarted document.
Dependency Injection itself can be done without any tool, framework or container support. You only need to remove news from your code and move them to constructors. The one tedious part that remains is wiring the objects at "the end of the world", where containers help a lot.
Though with Scala's 2.10 macros, you can generate the wiring code at compile-time and have auto-wiring and type-safety.
See the Dependency Injection in Scala Guide
A recent project illustrates a DI based purely on constructor injection: zalando/grafter
What's wrong with constructor injection again?
There are many libraries or approaches for doing dependency injection in Scala. Grafter goes back to the fundamentals of dependency injection by just using constructor injection: no reflection, no xml, no annotations, no inheritance or self-types.
Then, Grafter add to constructor injection just the necessary support to:
instantiate a component-based application from a configuration
fine-tune the wiring (create singletons)
test the application by replacing components
start / stop the application
Grafter is targeting every possible application because it focuses on associating just 3 ideas:
case classes and interfaces for components
Reader instances and shapeless for the configuration
tree rewriting and kiama for everything else!
I haven't done so myself, but most DI frameworks work at the bytecode level (AFAIK), so it should be possible to use them with any JVM language.
Previous posts covered the techniques. I wanted to add a link to Martin Odersky's May 2014 talk on the Scala language objectives. He identifies languages that "require" a DI container to inject dependencies as poorly implemented. I agree with this personally, but it is only an opinion. It does seem to indicate that including a DI dependency in your Scala project is non-idiomatic, but again this is opinion. Practically speaking, even with a language designed to inject dependencies natively, there is a certain amount of consistency gained by using a container. It is worth considering both points of view for your purposes.
https://youtu.be/ecekSCX3B4Q?t=1154
I would suggest you to try distage (disclaimer: I'm the author).
It allows you to do much more than a typical DI does and has many unique traits:
distage supports multiple configurations (e.g. you may run your app
with different sets of component implementations),
distage allows you to correctly share dependencies across your tests
and easily run same tests for different implementations of your
components,
distage supports roles so you may run multiple services within the same process sharing dependencies between them,
distage does not depend on scala-reflect
(but supports all the necessary features of Scala typesystem, like
higher-kinded types).
You may also watch our talk at Functional Scala 2019 where we've discussed and demonstrated some important capabiliteis of distage.
I have shown how I created a very simple functional DI container in scala using 2.10 here.
In addition to the answer of Dan Story, I blogged about a DI variant that also uses language constructs only but is not mentioned in Jonas's post: Value Injection on Traits (linking to web.archive.org now).
This pattern is working very well for me.