Build and add children in performLayout - flutter

Question

I'm making a skill tree package and I'm wanting to have a robust api for users to alter the appearance of edges between nodes. For that, I want the signature of an edgeBuilder to contain some layout information of the from and to nodes it connects. For example, I could pass through to the builder the angle an imaginary straight line from one node to another makes. This angle could then be used to rotate the widgets I have at the ends of edges to properly look like they're pointing at the right things (in the image below, I have a triangle clipper that is just always pointed down).
I have this structure for laying out the children (both edges and nodes) inside my multi-child custom RenderObject.
#override
void performLayout() {
// ...
final skillNodeLayout = delegate.layoutNodes(
loosenedConstraints,
graph,
nodeChildren,
);
size = skillNodeLayout.size;
// ...
delegate.layoutEdges(
loosenedConstraints,
skillNodeLayout,
graph,
edgeChildren,
nodeChildren,
);
}
But now I'm curious to know whether Flutter advises against building and adding widgets in the layout phase. The ContainerRenderObjectMixin I'm mixing in specifically has methods to add to the list of children such as adoptChild(child) or addAll.
Can I avoid passing through the edge children to the RenderObject constructor and instead pass them through as a plain list of builder functions I call after the node layout and then add to the RenderObject? Something roughly like this:
#override
void performLayout() {
// ...
final skillNodeLayout = delegate.layoutNodes(
loosenedConstraints,
graph,
nodeChildren,
);
size = skillNodeLayout.size;
final edgeChildren = edgeChildrenBuilders.map((builder) {
return builder(skillNodeLayout);
}).toList();
// Found in the `ContainerRenderObjectMixin`
addAll(edgeChildren);
delegate.layoutEdges(
loosenedConstraints,
skillNodeLayout,
graph,
edgeChildren,
nodeChildren,
);
}
My worry is that it keeps adding children to the RenderObject every time a performLayout is requested or that this was never the intended way to work with renderObjects.