Pytransitions: AsyncMachine Sequential Resolution of Asynchronous Dependent Callbacks - pytransitions

Question

Note:
This question is related to Python's FSM library pytransitions
I'm looking for a way to resolve method callbacks sequentially when they've been mentioned as a list in prepare or/and before or/and after. I'm using AsyncMachine module from transitions.extensions.asyncio
Expected Result:
1Done_2Done_3Done
Getting:
None_3Done
A sample code to replicate current situation:
import asyncio
from transitions.extensions.asyncio import AsyncMachine
class Model:
STATES = ['A', 'B']
TRANSITIONS = [
{'trigger': 'next', 'source': 'A', 'dest': 'B',
'prepare': ['initialize1', 'initialize2', 'initialize3'], 'before': [], 'after': ['show_attributes']}
]
def __init__(self, name, state='initial'):
self.name = name
self.state = state
self.attribute_1 = None
self.attribute_2 = None
self.attribute_3 = None
async def initialize1(self):
await asyncio.sleep(1) # This is expensive operation and will take some time.
self.attribute_1 = '1Done'
print(f'{self.name} {self.state} -> Initialized1: ', self.attribute_1)
async def initialize2(self):
await asyncio.sleep(0.5) # This is expensive operation and will take some time.
self.attribute_2 = f'{self.attribute_1}_2Done'
print(f'{self.name} {self.state} -> Initialized2: ', self.attribute_2)
async def initialize3(self):
self.attribute_3 = f'{self.attribute_2}_3Done'
print(f'{self.name} {self.state} -> Initialized3: ', self.attribute_3)
async def show_attributes(self):
print(f'{self.name} {self.state} -> Showing all: {self.attribute_3}')
machine = AsyncMachine(
model=None,
states=Model.STATES,
transitions=Model.TRANSITIONS,
initial=None,
queued='model'
# queued=True
)
async def main():
model1 = Model(name='Model1', state='A')
machine.add_model(model1, initial=model1.state)
await machine.dispatch('next')
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
As it's been shown in the code 'prepare': ['initialize1', 'initialize2', 'initialize3'] I'm looking for a way to call initialize2 once initialize1 is resolved and initialize3 once both initialize1 and initialize2 methods are resolved. Currently, they've been called parallelly which is a good feature but it would be awesome if there is a way to make them execute/resolve in sequence.
Of course, I can add one more method like initialize_all and then call all the above methods inside it. But think about how many new methods I've to keep adding to deal with real-world problems. I want to make my functions reusable and smaller just for a specific task.

Answer 1

I went through pytransitions source code and I found two ways to achieve the feature I was looking for.
I think it would be nice if I mention how I achieved the feature I was looking for.
Since I was looking for a way to have an asynchronous resolution of callback events (which is by default) and sequential resolution as per the requirement, I had to override the callbacks method of AsyncMachine.
Method 1:
import asyncio
from functools import partial
from transitions.extensions.asyncio import AsyncMachine
class EnhancedMachine(AsyncMachine):
async def callbacks(self, funcs, event_data):
""" Overriding callbacks method:
Get `parallel_callback` keyword argument to decide whether
callback events should be resolved in parallel or in sequence.
"""
parallel_callback = event_data.kwargs.get('parallel_callback', None)
resolved_funcs = [partial(event_data.machine.callback, func, event_data) for func in funcs]
if parallel_callback is False:
for func in resolved_funcs:
await func()
else:
await self.await_all(resolved_funcs)
class Model:
STATES = ['A', 'B']
TRANSITIONS = [
{'trigger': 'next', 'source': 'A', 'dest': 'B',
'prepare': ['initialize1', 'initialize2', 'initialize3'], 'before': [], 'after': ['show_attributes']}
]
def __init__(self, name, state='initial'):
self.name = name
self.state = state
self.sequential_transition = True
self.attribute_1 = None
self.attribute_2 = None
self.attribute_3 = None
async def initialize1(self, ed):
await asyncio.sleep(1) # This is expensive operation and will take some time.
self.attribute_1 = '1Done'
print(f'{self.name} {self.state} -> Initialized1: ', self.attribute_1)
async def initialize2(self, ed):
await asyncio.sleep(0.5) # This is expensive operation and will take some time.
self.attribute_2 = f'{self.attribute_1}_2Done'
print(f'{self.name} {self.state} -> Initialized2: ', self.attribute_2)
async def initialize3(self, ed):
self.attribute_3 = f'{self.attribute_2}_3Done'
print(f'{self.name} {self.state} -> Initialized3: ', self.attribute_3)
async def show_attributes(self, ed):
print(f'{self.name} {self.state} -> Showing all: {self.attribute_3}')
machine = EnhancedMachine(
model=None,
states=Model.STATES,
transitions=Model.TRANSITIONS,
initial=None,
send_event=True, # this will pass EventData instance for each method.
queued='model'
# queued=True
)
async def main():
model1 = Model(name='Model1', state='A')
machine.add_model(model1, initial=model1.state)
# Passing `parallel_callback` as False for synchronous events
await machine.dispatch('next', parallel_callback=False)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
Drawbacks:
send_event=True is added and all the method definitions have been added with extra argument ed (event_data) to handle the parallel_callback keyword argument.
Transition callback requires passing parallel_callback=False and has to change all possible places in the code.
If the next transition has to be decided from the definition of the transition itself, then the keyword argument parallel_callback can not be passed (at least I'm not sure how to do this):
TRANSITIONS = [
{'trigger': 'next', 'source': 'A', 'dest': 'B',
'prepare': [], 'before': [], 'after': ['next2']},
{'trigger': 'next2', 'source': 'B', 'dest': 'C',
'prepare': ['initialize1', 'initialize2', 'initialize3'], 'before': [], 'after': ['show_attributes']}
]
Method 2 (I personally prefer this way):
In the definition of transitions, grouping the callbacks together which are dependent on each other and should be resolved sequentially.
Using this method, final transitions would look something like this:
TRANSITIONS = [
{'trigger': 'next', 'source': 'A', 'dest': 'B',
'prepare': [('initialize1', 'initialize2', 'initialize3')], 'before': [],
'after': ['show_attributes']}
]
Explanation:
'prepare': [('callback1', 'callback2'), 'callback3']
Here group1 (callback1 and callback2), group2 (callback3) will be resolved asynchronously (parallelly). But callback1 and callback2 in group1 will be resolved synchronously (sequentially).
Overriden callbacks method will look slightly different now along with a new static method await_sequential:
class EnhancedMachine(AsyncMachine):
async def callbacks(self, func_groups, event_data):
""" Triggers a list of callbacks """
resolved_func_groups = []
for funcs in func_groups:
if isinstance(funcs, (list, tuple)):
resolved_funcs = [partial(event_data.machine.callback, func, event_data) for func in funcs]
else:
resolved_funcs = [partial(event_data.machine.callback, funcs, event_data)]
resolved_func_groups.append(resolved_funcs)
# await asyncio.gather(*[self.await_sequential(funcs) for funcs in resolved_func_groups])
await self.await_all([partial(self.await_sequential, funcs) for funcs in resolved_func_groups])
#staticmethod
async def await_sequential(funcs):
return [await func() for func in funcs]
Cons:
Nothing changed in the definition of the methods and the method calls.
Changed one place and it fixed all the places.
Drawbacks:
You should be aware of what your methods are doing. Sometimes unwanted grouping will cause unnecessary delays in the resolution of the events.
Using both ways, I got the same desired output:
Model1 A -> Initialized1: 1Done
Model1 A -> Initialized2: 1Done_2Done
Model1 A -> Initialized3: 1Done_2Done_3Done
Model1 B -> Showing all: 1Done_2Done_3Done
I'm sticking with the 2nd approach though I'd be happy to know other efficient ways to implement such a feature :)

Answer 2

I think your 'method 2' looks alright. If you know that all callbacks should be executed sequentially and you do not need parallel execution at all, you could also just override await_all with:
class EnhancedMachine(AsyncMachine):
#staticmethod
async def await_all(callables):
return [await func() for func in callables]
If you switch the meaning of tuples/lists, you could shorten the code a bit to something like this:
class EnhancedMachine(AsyncMachine):
async def callbacks(self, func_groups, event_data):
results = []
for funcs in func_groups:
if isinstance(funcs, (list, tuple)):
results.extend(await self.await_all(
[partial(event_data.machine.callback, func, event_data)
for func in funcs]
))
else:
results.append(await self.callback(funcs, event_data))
return results
This enables callback annotation like [stage_1, (stage_2a, stage_2b, stage_2c), stage_3] where each stage is sequentially executed but sub-stages are called in parallel.