I’m using AsyncMachine to model multiple objects interacting asynchronously and in principle, all works as expected, really a cool extension ;-)
However, when I use logging, the millisecond delays which are reported in log entries between processing multiple, asynchronous events are higher than I would expect, so I’m wondering whether this is due to logging output created by a blocking call to e.g. logger.info(). So the timings I’m trying to obtain by looking at log entry timestamps could be distorted by the creation of exactly those logs.
Using https://pypi.org/project/aiologger/ seems like a reasonable way forward, given that aiologger implements non-blocking logging specifically for asyncio.
After a quick look at pytransitions source code I’m wondering however what would happen if pytransitions itself still uses the logging module from the standard library while my code would use the logger offered by aiologger. My guess is that in this case, only the logs created by my code would be non-blocking, so how could I make pytransitions use aiologger as well?
Thanks in advance for any help,
Eradian
so how could I make pytransitions use aiologger as well?
As of September 2022 there is no way of making transitions work with aiologger. Due to backward compatibility reasons, transitions uses the %-formatting style for debug messages:
logging.warn("Object %s is initialized with value %d", object.name, object.value)
Theoretically you could override the internal logger instance used by AsyncMachine:
from aiologger import Logger
import transitions.extensions.asyncio as taio
taio._LOGGER = Logger.with_default_handlers(name='transitions.async')
# ...
m = taio.AsyncMachine(states=['A', 'B', 'C'], initial='A')
But this will throw errors during the first logging attempt because of the aforementioned reason:
Invalid LogRecord args type: <class 'str'>. Expected Mapping
Related
I am trying out this MCU / SoC emulator, Renode.
I loaded their existing model template under platforms/cpus/stm32l072.repl, which just includes the repl file for stm32l071 and adds one little thing.
When I then load & run a program binary built with STM32CubeIDE and ST's LL library, and the code hits the initial function of SystemClock_Config(), where the Flash:ACR register is being probed in a loop, to observe an expected change in value, it gets stuck there, as the Renode Monitor window is outputting:
[WARNING] sysbus: Read from an unimplemented register Flash:ACR (0x40022000), returning a value from SVD: 0x0
This seems to be expected, not all existing templates model nearly everything out of the box. I also found that the stm32L071 model is missing some of the USARTs and NVIC channels. I saw how, probably, the latter might be added, but there seems to be not a single among the default models defining that Flash:ACR register that I could use as example.
How would one add such a missing register for this particular MCU model?
Note1: For this test, I'm using a STM32 firmware binary which works as intended on actual hardware, e.g. a devboard for this MCU.
Note2:
The stated advantage of Renode over QEMU, which does apparently not emulate peripherals, is also allowing to stick together a more complex system, out of mocked external e.g. I2C and other devices (apparently C# modules, not yet looked into it).
They say "use the same binary as on the real system".
Which is my reason for trying this out - sounds like a lot of potential for implementing systems where the hardware is not yet fully available, and also automatted testing.
So the obvious thing, commenting out a lot of parts in init code, to only test some hardware-independent code while sidestepping such issues, would defeat the purpose here.
If you want to just provide the ACR register for the flash to pass your init, use a tag.
You can either provide it via REPL (recommended, like here https://github.com/renode/renode/blob/master/platforms/cpus/stm32l071.repl#L175) or via RESC.
Assuming that your software would like to read value 0xDEADBEEF. In the repl you'd use:
sysbus:
init:
Tag <0x40022000, 0x40022003> "ACR" 0xDEADBEEF
In the resc or in the Monitor it would be just:
sysbus Tag <0x40022000, 0x40022003> "ACR" 0xDEADBEEF
If you want more complex logic, you can use a Python peripheral, as described in the docs (https://renode.readthedocs.io/en/latest/basic/using-python.html#python-peripherals-in-a-platform-description):
flash: Python.PythonPeripheral # sysbus 0x40022000
size: 0x1000
initable: false
filename: "script_with_complex_python_logic.py"
```
If you really need advanced implementation, then you need to create a complete C# model.
As you correctly mentioned, we do not want you to modify your binary. But we're ok with mocking some parts we're not interested in for a particular use case if the software passes with these mocks.
Disclaimer: I'm one of the Renode developers.
When my connection is open, the application won't exit, this causes some nasty problems for me (highly concurrent and nested using a shared sesssion, don't know when each part is finished) - is there a way to make sure that the cluster doesn't "hang" the application?
For example here:
object ZombieTest extends App {
val session= Cluster.builder().addContactPoint("localhost").build().connect()
// app doesn't exit unless doing:
session.getCluster.close() // won't exit unless this is called
}
In a slightly biased answer, you could look at https://github.com/outworkers/phantom instead of using the standard java driver.
You get scala.concurrent.Future, monix.eval.Task or even com.twitter.util.Future from a query automatically. You can choose between all three.
DB connection pools are better isolated inside ContactPoint and Database abstraction layers, which have shutdown methods you can easily wire in to your app lifecycle.
It's far faster than the Java driver, as the serialization an de-serialisation of types is wired in compile time via more advanced macro mechanisms.
The short answer is that you need to have a lifecycle way of calling session.close or session.closeAsync when you shut down everything else, it's how it's designed to work.
For example, if I implement some simple object caching, which method is faster?
1. return isset($cache[$cls]) ? $cache[$cls] : $cache[$cls] = new $cls;
2. return #$cache[$cls] ?: $cache[$cls] = new $cls;
I read somewhere # takes significant time to execute (and I wonder why), especially when warnings/notices are actually being issued and suppressed. isset() on the other hand means an extra hash lookup. So which is better and why?
I do want to keep E_NOTICE on globally, both on dev and production servers.
I wouldn't worry about which method is FASTER. That is a micro-optimization. I would worry more about which is more readable code and better coding practice.
I would certainly prefer your first option over the second, as your intent is much clearer. Also, best to keep away edge condition problems by always explicitly testing variables to make sure you are getting what you are expecting to get. For example, what if the class stored in $cache[$cls] is not of type $cls?
Personally, if I typically would not expect the index on $cache to be unset, then I would also put error handling in there rather than using ternary operations. If I could reasonably expect that that index would be unset on a regular basis, then I would make class $cls behave as a singleton and have your code be something like
return $cls::get_instance();
The isset() approach is better. It is code that explicitly states the index may be undefined. Suppressing the error is sloppy coding.
According to this article 10 Performance Tips to Speed Up PHP, warnings take additional execution time and also claims the # operator is "expensive."
Cleaning up warnings and errors beforehand can also keep you from
using # error suppression, which is expensive.
Additionally, the # will not suppress the errors with respect to custom error handlers:
http://www.php.net/manual/en/language.operators.errorcontrol.php
If you have set a custom error handler function with
set_error_handler() then it will still get called, but this custom
error handler can (and should) call error_reporting() which will
return 0 when the call that triggered the error was preceded by an #.
If the track_errors feature is enabled, any error message generated by
the expression will be saved in the variable $php_errormsg. This
variable will be overwritten on each error, so check early if you want
to use it.
# temporarily changes the error_reporting state, that's why it is said to take time.
If you expect a certain value, the first thing to do to validate it, is to check that it is defined. If you have notices, it's probably because you're missing something. Using isset() is, in my opinion, a good practice.
I ran timing tests for both cases, using hash keys of various lengths, also using various hit/miss ratios for the hash table, plus with and without E_NOTICE.
The results were: with error_reporting(E_ALL) the isset() variant was faster than the # by some 20-30%. Platform used: command line PHP 5.4.7 on OS X 10.8.
However, with error_reporting(E_ALL & ~E_NOTICE) the difference was within 1-2% for short hash keys, and up 10% for longer ones (16 chars).
Note that the first variant executes 2 hash table lookups, whereas the variant with # does only one lookup.
Thus, # is inferior in all scenarios and I wonder if there are any plans to optimize it.
I think you have your priorities a little mixed up here.
First of all, if you want to get a real world test of which is faster - load test them. As stated though suppressing will probably be slower.
The problem here is if you have performance issues with regular code, you should be upgrading your hardware, or optimize the grand logic of your code rather than preventing proper execution and error checking.
Suppressing errors to steal the tiniest fraction of a speed gain won't do you any favours in the long run. Especially if you think that this error may keep happening time and time again, and cause your app to run more slowly than if the error was caught and fixed.
Do the risks caused by bypassing Perl safe signals for example like shown in the second timeout example in the DBI documentation concern only the code that uses such bypassing?
The code in that example works hard to localize the change to just that section of code, or any code called from it.
There is not 100% guarantee that no code will be effected outside the code that bypasses safe signals, because signals are no longer safe. In the example the call being timed out is a DBI->connect. For most DBD's this will be implemented mostly in C, unless the C code can handle being aborted and tried again you might find that some data structures internal to the DBD, or the libraries it uses, are left in a inconstant state.
The chances of the example code going wrong is probably incredibly tiny. My personal anecdote on the issues is that I had used the traditional Perl signal handling for years before safe signals were introduced and for a long time I had never had a problem. I hadn't even been very cautious about what I did in my signal handlers. Then we managed to hit a data set that actually did trigger memory corruptions in about 1 out of ever 100 runs. Just modifying the signal handlers to use better practices, similar to those in the example, eliminated our issues.
What does that even mean? By using unsafe signals, you can corrupt Perl's internals and Perl variables. It can also cause problem if a non-reentrant C library call is interrupted.
This can lead to SEGFAULTs and other problems, and those may only manifest themselves outside the block where the timeout is in effect.
I'm using logging vai Log::Log4perl in my perl script; I'm wondering if making multiple calls to write to the same log object will cause incorrect / erroneous behavior.
I'm using a Appender::File to write out the log in the following manner:
$log->info("Launching commands...");
foreach my $param (#params) {
push #thread_handles, async {
system("$param");
$log->info("$param COMPLETE");
logstatus($?);
};
}
$_->join() foreach #thread_handles;
$log->info("Commands completed...");
The Log::Log4perl with the default file based appender will work, but some overlapping may occur in a multi-threaded or multi-processed environment using the same log file.
One solution is to use Log::Log4perl::Appender::Synchronized as an appender. See How can I run Log::Log4perl under mod_perl? in the FAQ for more info.
Using the Synchronized appender makes a lot of sense.
My question is:
Will the logger variable be passed around to the threads? According to perlthrtut it will not. I have yet to verify this. Unless somebody's already done it?
You may be tempted to use threads::shared to share the logger variable but the threads::shared documentation specifies that you can only share scalars, arrays, or hashes. I've tried that anyway with perl 5.8.8 and, as expected, it will not work.
The other approach would be to create a separate logger for each subroutine that will be invoked in a thread. The log4perl appenders can be configured to avoid locking and interleaving. But I am seriously concerned about the performance impact incurred with generating a separate logger instance for each active thread.
Updated:
It turns out one doesn't have to get complicated. If you initialize log4perl as if you are writing a single thread script and you call the logger objects methods without any special tricks, everything works as advertised. The logger object doesn't have to be passed to thread entry point. The sub invoked in the thread entry point gets access to the logger methods the same way as if it were called the regular way. The Synchronized appender keeps everything lines up.
The Log4perl is working in threaded enviroment, however you have to choose the appenders carefully.
Except the synced appenders the logs messages with buffering won't appear to be in order, but this is not a problem.
Either use different files for each thread or add the pid to the logmessage.
The synced appenders may cause a lot of overhead in your threaded application, use it carefully.
I opt for a single logfile with pid or some kind of thread identifier. I am using this kind of logging without a problem.
regards,