dynamically switch scroll physics form never scroll to scroll - flutter

Question

The title might not be totally accurate but forgive me I don't really know what to write. I am trying to create an effect where a noScroll phyicas Listview that is nested with other elements in a SingleChildScrollView, would need to change to a scrollable physics when it reaches the top of the SingleChildScrollView (thus occupying the whole view) in order to utilize the lazy loading of items.
I must say I ma trying to avoid rebuilding the nested Listview because of that cost and especially the cost of building it with the NeverScrolling physics since that would stall the main UI thread until all items are built.
The necessity of using the NeverScrolling physics in the first place it to allow the SingleChildScrollView to properly scroll (if the list view is set to normal scrolling, the SingleChildScrollView will not be able to scroll if the gesture is coming from the nested Listview)
My code looks like this :
SingleChildScrollView(
scrollDirection: Axis.vertical,
child: Material(
child: Container(
child: Column(
children: <Widget>[
ItemListDevider(),
Container(
height: 120,
child: Container(height:120)
),
ItemListDevider(),
Flexible(
fit: FlexFit.loose,
child: THeWidgetHoldingTheListView(album),
)
],
),
),
),
)
// the build method of the THeWidgetHoldingTheListView widget would return this :
Container(
child: Row(
mainAxisSize: MainAxisSize.max,
children: <Widget>[
Expanded(
child: Column(
children: <Widget>[
Flexible(
child: ListView.builder(
controller: controller,
shrinkWrap: true,
itemExtent: FixedItemExtent,
physics: AlwaysScrollableScrollPhysics(),
itemCount: FixedItemCount,
itemBuilder: (context, index) {
return MyCard()
},
),
),
],
),
),
MyScrollbar(
controller: controller,
),
],
),
)
*Note: * I had the idea of creating a custom ScrollPhysics that starts as a NeverScrollPhysics and the switches on a certain scroll offset but that wouldn't change the fact that I have to load the entire list as a Neverscroll and thus stalling the main UI thread.
Is there a way to get around this change of scroll physics, if not how can I implement it without compromising on performance like mentioned above.