Here is my pyspark code used in Notebook
data_lake_container = 'abfss://abc.dfs.core.windows.net'
stage_folder = 'abc'
delta_lake_folder = 'abc'
source_folder = 'abc'
source_wildcard = 'abc.parquet'
key_column = 'Id'
key_column1 = 'LastModifiedDate'
source_path = data_lake_container + '/' + stage_folder + '/' + source_folder + '/' + source_wildcard
delta_table_path = data_lake_container + '/' + delta_lake_folder + '/' + source_folder
sdf = spark.read.format('parquet').option("recursiveFileLookup", "true").load(source_path)
if (DeltaTable.isDeltaTable(spark, delta_table_path)):
delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.alias("existing").merge(
source=sdf.alias("updates"),
condition=("existing." + key_column + " = updates." + key_column + " and existing." + key_column1 + " = updates." + key_column1) # We look for matches on the name column
).whenMatchedUpdateAll(
).whenNotMatchedInsertAll(
).execute()
else:
sdf.write.format('delta').save(delta_table_path)
while executing above code I'm getting below error
Py4JJavaError: An error occurred while calling o771.save.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231)
at org.apache.spark.sql.delta.files.TransactionalWrite.$anonfun$writeFiles$1(TransactionalWrite.scala:216)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:107)
Kindly help me in resolving error
Py4JJavaError: An error occurred while calling o771.save.
: org.apache.spark.SparkException: Job aborted.
The above error generally occurred because of non-compatible versions of spark connector and spark.
Refer - org.apache.spark.SparkException: Job aborted due to stage failure: Task from application
If the above solution does not work for you, please share a full stack trace of error. It is difficult to identify issues with shared information.
#AbhishekKhandave, when I looked into full error, there was date column with data range less than '1900-01-01'. That was the issue. Finally, I was able to run script. Thank you for your response.
I use jupyter notebooks in vscode all the time. I noticed about a day or two ago that I could no longer run code in cells. It always displayed
Failed to start the Kernel. OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"c:'. View Jupyter log for further details.
The logs also show
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"c:'... View Jupyter [log](command:jupyter.viewOutput) for further details.
at ChildProcess.<anonymous> (c:\Users\XY_User\.vscode\extensions\ms-toolsai.jupyter-2022.6.1201981810\out\extension.node.js:24:230120)
at ChildProcess.emit (node:events:402:35)
at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)] {
category: 'kerneldied',
kernelConnectionMetadata: {
kind: 'startUsingPythonInterpreter',
kernelSpec: {
specFile: 'c:\\Users\\XY_User\\.vscode\\extensions\\ms-toolsai.jupyter-2022.7.1001951036\\temp\\jupyter\\kernels\\python383jvsc74a57bd0ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186\\kernel.json',
interpreterPath: 'c:\\ProgramData\\Anaconda3\\python.exe',
isRegisteredByVSC: 'registeredByNewVersionOfExt',
name: 'python383jvsc74a57bd0ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186',
argv: [Array],
language: 'python',
executable: 'python',
display_name: "Python 3.8.3 ('base')",
metadata: [Object],
env: {}
},
interpreter: {
id: 'C:\\PROGRAMDATA\\ANACONDA3\\PYTHON.EXE',
sysPrefix: 'C:\\ProgramData\\Anaconda3',
envType: 'Conda',
envName: 'base',
envPath: [w],
architecture: 3,
sysVersion: '3.8.3 (default, Jul 2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]',
version: [Object],
companyDisplayName: 'ContinuumAnalytics',
displayName: "Python 3.8.3 ('base')",
detailedDisplayName: "Python 3.8.3 ('base': conda)",
uri: [w]
},
id: '.jvsc74a57bd0ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186.c:\\ProgramData\\Anaconda3\\python.exe.c:\\ProgramData\\Anaconda3\\python.exe.-m#ipykernel_launcher'
},
exitCode: 1,
stdErr: 'Traceback (most recent call last):\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel\\kernelapp.py", line 248, in init_connection_file\r\n' +
" self.connection_file = filefind(self.connection_file, ['.', self.connection_dir])\r\n" +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipython_genutils\\path.py", line 71, in filefind\r\n' +
' raise IOError("File %r does not exist in any of the search paths: %r" %\r\n' +
"OSError: File 'c:\\\\Users\\\\XY_User\\\\AppData\\\\Roaming\\\\jupyter\\\\runtime\\\\kernel-v2-7868dLFyv3ry3NTk.json' does not exist in any of the search paths: ['.', 'C:\\\\Users\\\\XY_User\\\\AppData\\\\Roaming\\\\jupyter\\\\runtime']\r\n" +
'\r\n' +
'During handling of the above exception, another exception occurred:\r\n' +
'\r\n' +
'Traceback (most recent call last):\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\runpy.py", line 194, in _run_module_as_main\r\n' +
' return _run_code(code, main_globals, None,\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\runpy.py", line 87, in _run_code\r\n' +
' exec(code, run_globals)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py", line 16, in <module>\r\n' +
' app.launch_new_instance()\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\traitlets\\config\\application.py", line 663, in launch_instance\r\n' +
' app.initialize(argv)\r\n' +
' File "<decorator-gen-125>", line 2, in initialize\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\traitlets\\config\\application.py", line 87, in catch_config_error\r\n' +
' return method(app, *args, **kwargs)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel\\kernelapp.py", line 565, in initialize\r\n' +
' self.init_connection_file()\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel\\kernelapp.py", line 252, in init_connection_file\r\n' +
' ensure_dir_exists(os.path.dirname(self.abs_connection_file), 0o700)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipython_genutils\\path.py", line 167, in ensure_dir_exists\r\n' +
' os.makedirs(path, mode=mode)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\os.py", line 213, in makedirs\r\n' +
' makedirs(head, exist_ok=exist_ok)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\os.py", line 213, in makedirs\r\n' +
' makedirs(head, exist_ok=exist_ok)\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\os.py", line 213, in makedirs\r\n' +
' makedirs(head, exist_ok=exist_ok)\r\n' +
' [Previous line repeated 3 more times]\r\n' +
' File "c:\\ProgramData\\Anaconda3\\lib\\os.py", line 223, in makedirs\r\n' +
' mkdir(name, mode)\r\n' +
`OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"c:'\r\n`,
vslsStack: [ CallSite {}, CallSite {}, CallSite {} ]
}
info 2:51:27.573: Process Execution: > c:\ProgramData\Anaconda3\python.exe -c "import ipykernel"
> c:\ProgramData\Anaconda3\python.exe -c "import ipykernel"
This topic seems to recur a lot so I found and tried these but no dice.
python -m ipykernel install --user
checking the kernel.json file in anaconda3/share/jupyter/kernels/python3/ directory.
switching between pre-release and release versions of the jupyter extension
Any help would be appreciated.
Turns out this can be caused by a mismatch between the version of ipykernel installed by anaconda and the base requirements of the jupyter plugin for vscode.
My case was solved with the following steps.
Launch an instance of anaconda prompt
Type pip install -U ipykernel --user and hit enter
Switch to vscode and push Ctrl + Shift + P and enter Developer: Reload Window
Try executing a cell in your .ipynb file
Hope this helps someone out there.
I am using the Powershell script to load data from about 5 Excel sheets (as a sample) from column A to AL. In reality, I have 100 Excel Sheets with 25 Tabs in each file.
I have used the link How to use powershell to copy several excel worksheets and make a new one?
And it has been of great help. However, data from the first sheet is copied and entered into the new sheet but from the 2nd File onwards it starts giving the errors as below.
The 2nd loop should go to Last Used Row of the New Excel Sheet (row 151) and paste data from the 2nd Sheet. However I run into the error "Unable to get the Select property of the Range class". Then the data from the 2nd Sheet never gets pasted.
So out of the 5 sheets data from 2 sheets gets copied while the remaining data from the other 3 sheets never gets pasted onto the new Excel Sheet mainly due to the error "Unable to get the Select property of the Range class"
Error As Received
2nd Loop
151
Unable to get the Select property of the Range class
At C:\Users\Desktop\ExcelCopy_ETC 2.ps1:84 char:51
+ ... t.Range("A$(($objDestExcel.ActiveSheet.UsedRange.Rows|Select -Last 1) ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
2nd Loop
151
2nd Loop
Destination UsedRange Row: - 311
Unable to get the Select property of the Range class
At C:\Users\Desktop\ExcelCopy_ETC 2.ps1:84 char:51
+ ... t.Range("A$(($objDestExcel.ActiveSheet.UsedRange.Rows|Select -Last 1) ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
2nd Loop
Destination UsedRange Row: - 311
Unable to get the Select property of the Range class
At C:\Users\Desktop\ExcelCopy_ETC 2.ps1:84 char:51
+ ... t.Range("A$(($objDestExcel.ActiveSheet.UsedRange.Rows|Select -Last 1) ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
Invalid index. (Exception from HRESULT: 0x8002000B (DISP_E_BADINDEX))
At C:\Users\Desktop\ExcelCopy_ETC 2.ps1:70 char:5
+ $SourceWorksheet = $SourceWorkBook.WorkSheets.item("3. Automation ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], COMException
+ FullyQualifiedErrorId : System.Runtime.InteropServices.COMException
2nd Loop
Destination UsedRange Row: - 311
0
I'm searching where can I read this log.
03-29 02:47:31.249 774 795 I ActivityManager: Displayed com.android.email/.activity.setup.AccountSetupFinal: +120ms
03-29 02:47:34.014 774 1200 I ActivityManager: START u0 {flg=0x10804000 cmp=com.android.systemui/.recents.RecentsActivity} from uid 10023 on display 0
03-29 02:47:35.488 774 785 I ActivityManager: Killing 2225:com.android.email/u0a39 (adj 900): remove task
This log shows in ActivityManager but I can't find the source in ActivityManager.java in AOSP full source.
So would you tell me where is the log source file?
This logs are printed by the files under "frameworks/base/services/core/java/com/android/server/am/" .
For eg , the last line in your logs is printed by ProcessRecord.java
Slog.i(TAG, "Killing " + toShortString() + " (adj " + setAdj + "): " + reason);
where TAG is defined as :
private static final String TAG = TAG_WITH_CLASS_NAME ? "ProcessRecord" : TAG_AM;
This question already has answers here:
$string.Substring Index/Length exception
(3 answers)
Closed 7 years ago.
What am I missing?
This doesn't work:
"20151129".Length
"20151129".Length - 1
"20151129".Substring(6,7)
"20151129".Substring(6,"20151129".Length -1)
But this does work:
"20151129".Substring(0,3)
Even though I'm not exceeding ("20151129".length - 1), I'm still obtaining an error:
Exception calling "Substring" with "2" argument(s): "Index and length
must refer to a location within the string. Parameter name: length" At
C:\Projects\Support\Microsoft\Powershell\DateRegex.ps1:23 char:1
+ "20151129".Substring(6,7)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : ArgumentOutOfRangeException
Never mind:
"20151129".Substring(6,2)