I would like to have a DML device with interfaces and register banks as the TOP-level of my device but offload processing to Python. Is there a lightweight method of calling into Python from DML?
This post How can I unit test a specific DML method? addresses calling from Python into DML, but I am interested in the reverse.
I think I can create a bunch of custom interfaces to do this, but I'm interested to know if there's a better way.
Implementing parts of a device in Python can make sense, in particular for code that seldom is invoked (user interaction comes to mind), and when faced with tasks like string manipulation where the shortcomings of a C-like language is particularly painful, or when code sharing with CLI commands is desired.
You can use the SIM_call_python_function API to call out to Python from DML. The function uses the equivalent of eval on the passed string, so you can find a module-local function by __import__:
local attr_value_t a = SIM_make_attr_list(0);
local attr_value_t result = SIM_call_python_function(
"__import__('simics').SIM_version", &a);
log info: "%s", SIM_attr_string(result);
SIM_attr_free(&a);
SIM_attr_free(&result);
This API is admittedly not very pretty. Parts of the standard lib for DML 1.2 is in fact written in Python and uses the internal function VT_call_python_module_function for this task instead. That API is nicer, but we cannot recommend use of VT_ functions outside the core Simics team.
This question is really more about the Simics simulator framework than about DML. Given how Simics works, Python code lives in a separate context. There is no obvious light-weight way. Rather, you need to put the Python code in a separate Python class and a separate runtime object and use a Simics simulator interface to orchestrate the calling.
The Python code would in general not be able to access the state of the object anyway, so it would have to be a strict "compute this and return all computed values". Which is not very convenient for device behavior that is really all about mutating the state of the object.
Even with a bunch of interfaces, there is still the matter of state handling. I guess you need an interface for that as well.
Related
I know we can call Rust from Flutter/Dart via FFI. But Flutter only allows the C ABI when doing FFI. Therefore, I have to manually write down boilerplate code. Especially, Rust unsafe code - since I have to deal with lots of raw pointers :(
Therefore, is there any approaches to do it in a safe way? We know Rust itself is very safe (since its unique memory management approach), and Dart/Flutter itself is also very safe (since GC). But I do not want the ffi call be the Achilles heel and destroy the safety of my app!
There are several ways to do it.
a. JSON/Protobuf-based Approach
The first way that I have used in the production environment for a year is that, you can use JSON or Protobuf to pass all the data between Rust and Dart/Flutter. By doing this, you do not need to write down tons of boilerplate code to allocate/free a String, a List of bytes, a struct/class, etc. All you need to do is to write down one single function that accepts a byte array payload and outputs a byte array result. By saying "one" function, I mean, you can have an action field in your JSON/Protobuf, so calls to indeed different Rust functions can be interleaved into this one thin interface.
Despite its convenience (only a bit of unsafe boilerplate), the drawback is also evident. The serialization and deserialization does not come for free. You will have to pay the CPU time and memory for it, which can be quite large sometimes. Moreover, you cannot easily pass around big objects. For example, if you have an image (you know, at least megabytes of size), serializing it to Protobuf, then deserialize it from Protobuf can be quite a waste of both CPU and memory - useless copies! Even worse, since Flutter/Dart FFI does not support a convenient way of async FFI, you have to make it running in a separate worker isolate - one more memory copy. You can see more here: https://github.com/dart-lang/language/issues/1862 (this is an issue that I opened).
b. Code generator
The second way that I use recently is to write down a code generator. Indeed the code follows several common patterns, such as "allocate - fill data - call FFI - free", etc. So it is not that hard to write a generator to automatically do such kind of things. The idea is to mimic what human beings will do when they write down boilerplate code manually.
I did hope that there already exist some code generator such that I could directly use, but it seemed that none exists... So, go and write it by yourself.
c. Use existing open-source code generator
After I write down the code generator, I guess people may have the same problem as me, so I open-sourced it: https://github.com/fzyzcjy/flutter_rust_bridge
Indeed, my code generator not only solves the problem above, but also have rich type support, allows zero-copy, allows async programming and direct call from main isolate, etc, which can be implemented via code generator but will require lots of boilerplate code if you do it by hand.
Disclaimer: This is a Q&A-style answer to show my thoughts and what I have done on this problem that is critical to my own app in production environment. Indeed I have used the JSON approach since last year, and later refactor into the code generator approach. Hope it also helps other people who faces the same situation!
I want to outsource some code for a plugin system. Inside my project, I have a trait called Provider which is the code for my plugin system. If you activate the feature "consumer" you can use plugins; if you don't, you are an author of plugins.
I want authors of plugins to get their code into my program by compiling to a shared library. Is a shared library a good design decision? The limitation of the plugins is using Rust anyway.
Does the plugin host have to go the C way for loading the shared library: loading an unmangled function?
I just want authors to use the trait Provider for implementing their plugins and that's it.
After taking a look at sharedlib and libloading, it seems impossible to load plugins in a idiomatic Rust way.
I'd just like to load trait objects into my ProviderLoader:
// lib.rs
pub struct Sample { ... }
pub trait Provider {
fn get_sample(&self) -> Sample;
}
pub struct ProviderLoader {
plugins: Vec<Box<Provider>>
}
When the program is shipped, the file tree would look like:
.
├── fancy_program.exe
└── providers
├── fp_awesomedude.dll
└── fp_niceplugin.dll
Is that possible if plugins are compiled to shared libs? This would also affect the decision of the plugins' crate-type.
Do you have other ideas? Maybe I'm on the wrong path so that shared libs aren't the holy grail.
I first posted this on the Rust forum. A friend advised me to give it a try on Stack Overflow.
UPDATE 3/27/2018:
After using plugins this way for some time, I have to caution that in my experience things do get out of sync, and it can be very frustrating to debug (strange segfaults, weird OS errors). Even in cases where my team independently verified the dependencies were in sync, passing non-primitive structs between the dynamic library binaries tended to fail on OS X for some reason. I'd like to revisit this, find what cases it happens in, and perhaps open an issue with Rust, but I'm going to advise caution with this going forward.
LLDB and valgrind are near-essential to debug these issues.
Intro
I've been investigating things along these lines myself, and I've found there's little official documentation for this, so I decided to play around!
First let me note, as there is little official word on these properties please do not rely on any code here if you're trying to keep planes in the air or nuclear missiles from errantly launching, at least not without doing far more comprehensive testing than I've done. I'm not responsible if the code here deletes your OS and emails an erroneous tearful confession of committing the Zodiac killings to your local police; we're on the fringes of Rust here and things could change from one release or toolchain to another.
I have personally tested this on Rust 1.20 stable in both debug and release configurations on Windows 10 (stable-x86_64-pc-windows-msvc) and Cent OS 7 (stable-x86_64-unknown-linux-gnu).
Approach
The approach I took was a shared common crate both crates listed as a dependency defining common struct and trait definitions. At first, I was also going to test having a struct with the same structure, or trait with the same definitions, defined independently in both libraries, but I opted against it because it's too fragile and you wouldn't want to do it in a real design. That said, if anybody wants to test this, feel free to do a PR on the repository above and I will update this answer.
In addition, the Rust plugin was declared dylib. I'm not sure how compiling as cdylib would interact, since I think it would mean that upon loading the plugin there are two versions of the Rust standard library hanging around (since I believe cdylib statically links the Rust stdlib into the shared object).
Tests
General Notes
The structs I tested were not declared #repr(C). This could provide an extra layer of safety by guaranteeing a layout, but I was most curious about writing "pure" Rust plugins with as little "treating Rust like C" fiddling as possible. We already know you can use Rust via FFI by wrapping things in opaque pointers, manually dropping, and such, so it's not very enlightening to test this.
The function signature I used was pub fn foo(args) -> output with the #[no_mangle] directive, it turns out that rustfmt automatically changes extern "Rust" fn to simply fn. I'm not sure I agree with this in this case since they are most certainly "extern" functions here, but I will choose to abide by rustfmt.
Remember that even though this is Rust, this has elements of unsafety because libloading (or the unstable DynamicLib functionality) will not type check the symbols for you. At first I thought my Vec test was proving you couldn't pass Vecs between host and plugin until I realized on one end I had Vec<i32> and on the other I had Vec<usize>
Interestingly, there were a few times I pointed an optimized test build to an unoptimized plugin and vice versa and it still worked. However, I still can't in good faith recommending building plugins and host applications with different toolchains, and even if you do, I can't promise that for some reason rustc/llvm won't decide to do certain optimizations on one version of a struct and not another. In addition, I'm not sure if this means that passing types through FFI prevents certain optimizations such as Null Pointer Optimizations from occurring.
You're still limited to calling bare functions, no Foo::bar because of the lack of name mangling. In addition, due to the fact that functions with trait bounds are monomorphized, generic functions and structs are also out. The compiler can't know you're going to call foo<i32> so no foo<i32> is going to be generated. Any functions over the plugin boundary must take only concrete types and return only concrete types.
Similarly, you have to be careful with lifetimes for similar reasons, since there's no static lifetime checking Rust is forced to believe you when you say a function returns &'a when it's really &'b.
Native Rust
The first tests I performed were on no custom structures; just pure, native Rust types. This would give a baseline for if this is even possible. I chose three baseline types: &mut i32, &mut Vec, and Option<i32> -> Option<i32>. These were all chosen for very specific reasons: the &mut i32 because it tests a reference, the &mut Vec because it tests growing the heap from memory allocated in the host application, and the Option as a dual purpose of testing passing by move and matching a simple enum.
All three work as expected. Mutating the reference mutates the value, pushing to a Vec works properly, and the Option works properly whether Some or None.
Shared Struct Definition
This was meant to test if you could pass a non-builtin struct with a common definition on both sides between plugin and host. This works as expected, but as mentioned in the "General Notes" section, can't promise you Rust won't fail to optimize and/or optimize a structure definition on one side and not another. Always test your specific use case and use CI in case it changes.
Boxed Trait Object
This test uses a struct whose definition is only defined on the plugin side, but implements a trait defined in a common crate, and returns a Box<Trait>. This works as expected. Calling trait_obj.fun() works properly.
At first I actually anticipated there would be issues with dropping without making the trait explicitly have Drop as a bound, but it turns out Drop is properly called as well (this was verified by setting the value of a variable declared on the test stack via raw pointer from the struct's drop function). (Naturally I'm aware drop is always called even with trait objects in Rust, but I wasn't sure if dynamic libraries would complicate it).
NOTE:
I did not test what would happen if you load a plugin, create a trait object, then drop the plugin (which would likely close it). I can only assume this is potentially catastrophic. I recommend keeping the plugin open as long as the trait object persists.
Remarks
Plugins work exactly as you'd expect just linking a crate naturally, albeit with some restrictions and pitfalls. As long as you test, I think this is a very natural way to go. It makes symbol loading more bearable, for instance, if you only need to load a new function and then receive a trait object implementing an interface. It also avoids nasty C memory leaks because you couldn't or forgot to load a drop/free function. That said, be careful, and always test!
There is no official plugin system, and you cannot do plugins loaded at runtime in pure Rust. I saw some discussions about doing a native plugin system, but nothing is decided for now, and maybe there will never be any such thing. You can use one of these solutions:
You can extend your code with native dynamic libraries using FFI. To use the C ABI, you have to use repr(C), no_mangle attribute, extern etc. You will find more information by searching Rust FFI on the internets. With this solution, you must use raw pointers: they come with no safety guarantee (i.e. you must use unsafe code).
Of course, you can write your dynamic library in Rust, but to load it and call the functions, you must go through the C ABI. This means that the safety guarantees of Rust do not apply there. Furthermore, you cannot use the highest level Rust's functionalities as trait, enum, etc. between the library and the binary.
If you do not want this complexity, you can use a language adapted to expand Rust: with which you can dynamically add functions to your code and execute them with same guarantees as in Rust. This is, in my opinion, the easier way to go: if you have the choice, and if the execution speed is not critical, use this to avoid tricky C/Rust interfaces.
Here is a (not exhaustive) list of languages that can easily extend Rust:
Gluon, a functional language like Haskell
Dyon, a small but powerful scripting language intended for video games
Lua with rlua or hlua
You can also use Python or Javascript, or see the list in awesome-rust.
I'm trying to get to grips with how Lua works, and how it can be integrated into a project, like an Obj-C based iPhone game. I've got Lua's source integrated fine, and have managed to use a simple manager class to translate function calls written in a .lua file, to Obj-C methods in the project.
I'm now a little stuck; I'm not sure how to go about what I'd like to use Lua for, and having Googled a lot on it I haven't found an article which fills the little gaps in my understanding. What I'd like to do is use Lua to script simple game logic for levels in my game; for example, I might want to check that the player has moved to a certain part of the level, then once he does that I check if he picks up an object, then once he throws that object I check if it hits another object, etc, etc.
In the script, I'd like to have something like this:
if (objectsAreTouching(level.objects["objIndex1"], level.objects["objIndex2"]))
{
//Move to next win-condition.
}
So I know how to access a function, for example the one named above, objectsAreTouching, but I can't see how I would, from within the .lua file, access a Dictionary or Array from within one of the game's main classes. I assume this must be possible, am I right? If so, how do you do this? If you can, can you then have a method in your game engine which returns a BOOL pass that return value to the script, so that the if statement above would then execute the code within it?
I may well have asked this question badly, and so if I need to clarify anything let me know. Also, I'm aware of the issues surrounding Apple's ban of interpreted code, and it doesn't concern me for this little project.
Thanks for any help!
You have (at least) three options:
Create the Dictionary (using a Lua table) and Array (also using a Lua table) on the Lua side rather than on the C++ side; you can do this in C++ using Lua's C API, or in a Lua script that you load to construct the game environment.
Provide access functions to Lua to read the C++ based Dictionary and Array. These access functions are just like the objectsAreTouching function you described, but at the level of data accessors. Typically the results returned from the accessors would be lightuserdata.
Use Lua userdata objects to share the state of the Dictionary and Array between Lua and C++. In this case they would be C++ objects requiring accessors, just like (2), but would be managed by Lua and could have methods (a metatable) and an environment associated with them.
How do I create an event handler in my Perl code to intercept all File/Directory/system-based calls, so that I can untaint input in a just-in-time fashion.
I have lots of IO access in my script, and I find adding manual code for untainting cumbersome.
Can this be done without need to install a third-party CPAN module?
You could try taking an aspect-oriented approach but it does require installing a CPAN module, Aspect.
To capture calls to a particular method / function, you define a pointcut (taken from the Aspect POD):
$pointcut = call qr/^Person::[gs]et_/; # defines a collection of events
Then define the code to take before the call:
$before = before {
print "g/set will soon be called";
} $pointcut;
Although I'm not sure if the Aspect module allows you to trap calls to the CORE::* namespace.
How do expect to untaint general data? If you're just going to blindly accept everything despite its source, there's no point in using taint-checking.
You might want to read the "Secure Programming Techniques" chapter in Mastering Perl. I give quite a bit of advice for dealing with this sort of stuff. However, any good advice is going to be targeted at specific situations, not generalizations.
I am doing a compilers discipline at college and we must generate code for our invented language to any platform we want to. I think the simplest case is generating code for the Java JVM or .NET CLR. Any suggestion which one to choose, and which APIs out there can help me on this task? I already have all the semantic analysis done, just need to generate code for a given program.
Thank you
From what I know, on higher level, two VMs are actually quite similar: both are classic stack-based machines, with largely high-level operations (e.g. virtual method dispatch is an opcode). That said, CLR lets you get down to the metal if you want, as it has raw data pointers with arithmetic, raw function pointers, unions etc. It also has proper tailcalls. So, if the implementation of language needs any of the above (e.g. Scheme spec mandates tailcalls), or if it is significantly advantaged by having those features, then you would probably want to go the CLR way.
The other advantage there is that you get a stock API to emit bytecode there - System.Reflection.Emit - even though it is somewhat limited for full-fledged compiler scenarios, it is still generally enough for a simple compiler.
With JVM, two main advantages you get are better portability, and the fact that bytecode itself is arguably simpler (because of less features).
Another option that i came across what a library called run sharp that can generate the MSIL code in runtime using emit. But in a nicer more user friendly way that is more like c#. The latest version of the library can be found here.
http://code.google.com/p/runsharp/
In .NET you can use the Reflection.Emit Namespace to generate MSIL code.
See the msdn link: http://msdn.microsoft.com/en-us/library/3y322t50.aspx