synapse spark notebook strange errors reading from ADLS Gen 2 and writing to another ADLS Gen 2 both using private endpoints and TokenLibrary - pyspark

Question

The premise is simple two ADLS Gen 2 accounts. Accessing both as abfss://
Source account with directory format yyyy/mm/dd records stored as jsonl format.
I need to read this account recursively starting at any directory.
Transform the data.
Write the data to the target account in parquet with format Year=yyyy/Month=mm/Day=dd.
The source account is an ADLS Gen 2 account with private endpoints that is not a part of Synapse Analytics.
The target account is the default ADLS Gen 2 for Azure Synapse Analytics.
Using Spark notebook within Synapse Analytics with managed virtual network.
Source storage account has private endpoints.
Code written in pyspark.
Using linked services for both ADLS Gen 2 accounts setup with private endpoints.
from pyspark.sql.functions import col, substring
spark.conf.set("spark.storage.synapse.linkedServiceName", psourceLinkedServiceName)
spark.conf.set("fs.azure.account.auth.type", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedSASProvider")
#read the data into a data frame
df = spark.read.option("recursiveFileLookup","true").schema(inputSchema).json(sourceFile)
#perform the transformations to flatten this structure
#create the partition columns for delta lake
dfDelta = df.withColumn('ProductSold_OrganizationId',col('ProductSold.OrganizationId'))\
.withColumn('ProductSold_ProductCategory',col('ProductSold.ProductCategory'))\
.withColumn('ProductSold_ProductId',col('ProductSold.ProductId'))\
.withColumn('ProductSold_ProductLocale',col('ProductSold.ProductLocale'))\
.withColumn('ProductSold_ProductName',col('ProductSold.ProductName'))\
.withColumn('ProductSold_ProductType',col('ProductSold.ProductType'))\
.withColumn('Year',substring(col('CreateDate'),1,4))\
.withColumn('Month',substring(col('CreateDate'),6,2))\
.withColumn('Day',substring(col('CreateDate'),9,2))\
.drop('ProductSold')
#dfDelta.show()
spark.conf.set("spark.storage.synapse.linkedServiceName", psinkLinkedServiceName)
spark.conf.set("fs.azure.account.auth.type", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type", "com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedSASProvider")
dfDelta.write.partitionBy("Year","Month","Day").mode('append').format("parquet").save(targetFile) ;
when trying to access a single file you get the error message
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (vm-19231616 executor 2): java.nio.file.AccessDeniedException: Operation failed: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.", 403, HEAD,
The error indicates that it cannot authenticate to the source file. However
Here is where it gets strange.
I uncomment the line
#dfDelta.show()
And it works.
However, if you put the comment back and run it again it continues to work. The only way to see the failure again is to totally stop the spark session and then restart it.
Ok more strangeness
change the sourcefile path so that its something like 2022/06 which should read multiple files and regardless of if the dfDelta.show() statement is uncommented you get the same error.
The only method I have found to get this to work is to process one file at a time within the spark notebook. option("recursiveFileLookup","true") only works when there is one file that is processed.
Finally what have I tried.
I have tried creating another spark session.
Spark session 1 reads the data and puts into a view.
Spark session 2 reads the data in the view and and attempts to write it.
The configuration for spark session two uses the token library for the sink file.
Results in the same error message.
My best guess - this has something to do with the spark cores processing multiple files and when I change the config they get confused as to how to read from the source file.
I had this working perfectly before I changed the Synapse Analytics account to use a managed virtual network. But in that case I accessed the source storage account using a managed identity linked service and had not issues writing to the default Synapse Analytics ADLS Gen 2 account.
Also tried option("forwardSparkAzureStorageCredentials", "true") on both the read and write dataframes
Any suggestions on how to get this to work with multiple files would be appreciated.