This is my Pig script
$QueryString = "A = load 'wasb://$containername#$StorageAccount.blob.core.windows.net/table1' using PigStorage(',') as (col1 chararray,col2 chararray,col3 chararray,col4 chararray,col5 chararray,col6 chararray,col7 int,col8 int);" +
"user_list = foreach A GENERATE $0;" +
"unique_user = DISTINCT user_list;" +
"unique_users_group = GROUP unique_user ALL;" +
"uu_count = FOREACH unique_users_group GENERATE COUNT(unique_user);" +
"DUMP uu_count;"
i get this error when i execute above pig script
'2015-04-14 23:17:55,177 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. <line 1, column 166> mismatched input 'chararray' expecting RIGHT_PAREN
Failed to parse: <line 1, column 166> mismatched input 'chararray' expecting RIGHT_PAREN
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:769)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:509)
at org.apache.pig.Main.main(Main.java:156)
2015-04-14 23:17:55,177 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 1, column 166> mismatched input 'chararray' expecting RIGHT_PAREN
I edited the LOAD statement like this and the rest of the script is same
$QueryString = "A = load 'wasb://$containername#$StorageAccount.blob.core.windows.net/table1';" +
the error i get now is
2015-04-14 23:23:00,117 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. <line 1, column 162> Syntax error, unexpected symbol at or near ';'
Failed to parse: <line 1, column 162> Syntax error, unexpected symbol at or near ';'
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:241)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:179)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:769)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:509)
at org.apache.pig.Main.main(Main.java:156)
2015-04-14 23:23:00,132 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 1, column 162> Syntax error, unexpected symbol at or near ';'
Details at logfile: C:\apps\dist\hadoop-2.4.0.2.1.9.0-2196\logs\pig_1429053777602.log
I don't understand what the error is. Can you someone help me with executing this query on windows powershell(i am using windows powershell ISE, so i can edit the queries)
The issue is at this statement user_list = foreach A GENERATE $0;. PowerShell is interpreting $0 as a parameter, and since it is not defined PowerShell is substituting an empty string. You can define a parameter in the script like $0 = '$0'; or just escape the $ like:
user_list = foreach A GENERATE `$0;
PowerShell uses the ` (backtick, next to the '1' key) as an escape character for double-quoted strings.
so the script can look like:
$0 = '$0';
$QueryString = "A = load 'wasb://$containerName#$storageAccountName.blob.core.windows.net/table1' using PigStorage(',') as (col1,col2,col3,col4,col5,col6,col7,col8) ;"+
"user_list = foreach A GENERATE $0;" +
"unique_user = DISTINCT user_list;" +
"unique_users_group = GROUP unique_user ALL;" +
"uu_count = FOREACH unique_users_group GENERATE COUNT(unique_user);" +
"DUMP uu_count;"
or
$QueryString = "A = load 'wasb://$containerName#$storageAccountName.blob.core.windows.net/table1' using PigStorage(',') as (col1,col2,col3,col4,col5,col6,col7,col8) ;"+
"user_list = foreach A GENERATE `$0;" +
"unique_user = DISTINCT user_list;" +
"unique_users_group = GROUP unique_user ALL;" +
"uu_count = FOREACH unique_users_group GENERATE COUNT(unique_user);" +
"DUMP uu_count;"
Related
We periodically runs jobs and we need to save the output into a PDS and then parse the output to extract parts of it to save into another member. It needs to be done by issuing a REXX command using the percent sign and the REXX member name as an SDSF command line. I've attempted to code a REXX to do this, but it is getting an error when trying to invoke an ISPF service, saying the ISPF environment has not been established. But, this is SDSF running under ISPF.
My code has this in it (copied from several sources and modified):
parse arg PSDSFPARMS "(" PUSERPARMS
parse var PSDSFPARMS PCURRPNL PPRIMPNL PROWTOKEN PPRIMCMD .
PRIMCMD=x2c(PPRIMCMD)
RC = isfquery()
if RC <> 0 then
do
Say "** SDSF environment does not exist, exec ending."
exit 20
end
RC = isfcalls("ON")
Address SDSF "ISFGET" PPRIMPNL "TOKEN('"PROWTOKEN"')" ,
" (" VERBOSE ")"
LRC = RC
if LRC > 0 then
call msgrtn "ISFGET"
if LRC <> 0 then
Exit 20
JOBNAME = value(JNAME.1)
JOBNBR = value(JOBID.1)
SMPDSN = "SMPE.*.OUTPUT.LISTINGS"
LISTC. = ''
SMPODSNS. = ''
SMPODSNS.0 = 0
$ = outtrap('LISTC.')
MSGVAL = msg('ON')
address TSO "LISTC LVL('"SMPDSN"') ALL"
MSGVAL = msg(MSGVAL)
$ = outtrap('OFF')
do LISTCi = 1 to LISTC.0
if word(LISTC.LISTCi,1) = 'NONVSAM' then
do
parse var LISTC.LISTCi . . DSN
SMPODSNS.0 = SMPODSNS.0 + 1
i = SMPODSNS.0
SMPODSNS.i = DSN
end
IX = pos('ENTRY',LISTC.LISTCi)
if IX <> 0 then
do
IX = pos('NOT FOUND',LISTC.LISTCi,IX + 8)
if IX <> 0 then
do
address ISPEXEC "SETMSG MSG(IPLL403E)"
EXITRC = 16
leave
end
end
end
LISTC. = ''
if EXITRC = 16 then
exit 0
address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
I execute this code by typing %SMPSAVE next to the spool output line on the "H" SDSF panel and it runs fine until it gets to this point in the REXX:
114 *-* address ISPEXEC "TBCREATE SMPDSNS NOWRITE" ,
"NAMES(TSEL TSMPDSN)"
>>> "TBCREATE SMPDSNS NOWRITE NAMES(TSEL TSMPDSN)"
ISPS118S SERVICE NOT INVOKED. A VALID ISPF ENVIRONMENT DOES NOT EXIST.
+++ RC(20) +++
Does anyone know why it says I don't have a valid ISPF environment and how I can get around this?
I've done quite a bit in the past with REXX, including writing REXX code to handle line commands, but this is the first time I've tried to use ISPEXEC commands within this code.
Thank you,
Alan
Here is my pyspark code used in Notebook
data_lake_container = 'abfss://abc.dfs.core.windows.net'
stage_folder = 'abc'
delta_lake_folder = 'abc'
source_folder = 'abc'
source_wildcard = 'abc.parquet'
key_column = 'Id'
key_column1 = 'LastModifiedDate'
source_path = data_lake_container + '/' + stage_folder + '/' + source_folder + '/' + source_wildcard
delta_table_path = data_lake_container + '/' + delta_lake_folder + '/' + source_folder
sdf = spark.read.format('parquet').option("recursiveFileLookup", "true").load(source_path)
if (DeltaTable.isDeltaTable(spark, delta_table_path)):
delta_table = DeltaTable.forPath(spark, delta_table_path)
delta_table.alias("existing").merge(
source=sdf.alias("updates"),
condition=("existing." + key_column + " = updates." + key_column + " and existing." + key_column1 + " = updates." + key_column1) # We look for matches on the name column
).whenMatchedUpdateAll(
).whenNotMatchedInsertAll(
).execute()
else:
sdf.write.format('delta').save(delta_table_path)
while executing above code I'm getting below error
Py4JJavaError: An error occurred while calling o771.save.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231)
at org.apache.spark.sql.delta.files.TransactionalWrite.$anonfun$writeFiles$1(TransactionalWrite.scala:216)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:107)
Kindly help me in resolving error
Py4JJavaError: An error occurred while calling o771.save.
: org.apache.spark.SparkException: Job aborted.
The above error generally occurred because of non-compatible versions of spark connector and spark.
Refer - org.apache.spark.SparkException: Job aborted due to stage failure: Task from application
If the above solution does not work for you, please share a full stack trace of error. It is difficult to identify issues with shared information.
#AbhishekKhandave, when I looked into full error, there was date column with data range less than '1900-01-01'. That was the issue. Finally, I was able to run script. Thank you for your response.
I've had the pleasure to get the assignment of posting emojis in Powershell, the only problem is they have to be on the same line, and there are three. This is, my first assignment, and we have no prior teaching in this subject so after googling and searching YouTube, my best shot was this below, however, it came with some error saying something about either too high value, or too low value.
Full error text: Exception calling "ToInt32" with "2" argument (s): "The value was either too large or too small to a UInt32. "
At C: \ Users \ EG \ Downloads \ Herningsholm \ Powershell H1 \ Hardware Information.ps1: 3 char: 5
$ UnicodeInt = [System.Convert] :: toInt32 ($ StrippedUnicode, 16)
CategoryInfo: NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId: OverflowException
$FullUnicode = ('U+1F60E') + ('U+1F436') + ('U+1F642')
$StrippedUnicode = $FullUnicode -replace 'U\+',''
$UnicodeInt = [System.Convert]::toInt32($StrippedUnicode,16)
[System.Char]::ConvertFromUtf32($UnicodeInt)
Try this out:
Full emoji list > here
# saves unicode for each emoji https://unicode.org/emoji/charts/full-emoji-list.html
$FullUnicode0 = 'U+1F606'
$FullUnicode1 = 'U+1F605'
$FullUnicode2 = 'U+1F605'
# removes the U+ bit
$StrippedUnicode0 = $FullUnicode0 -replace 'U\+',''
$StrippedUnicode1 = $FullUnicode1 -replace 'U\+',''
$StrippedUnicode2 = $FullUnicode2 -replace 'U\+',''
# Converts the value of the specified object to a 32-bit signed integer
$UnicodeInt0 = [System.Convert]::toInt32($StrippedUnicode0,16)
$UnicodeInt1 = [System.Convert]::toInt32($StrippedUnicode1,16)
$UnicodeInt2 = [System.Convert]::toInt32($StrippedUnicode2,16)
# Converts the specified Unicode code point into a UTF-16 encoded string so that you have an emoji
$Emoji0 = [System.Char]::ConvertFromUtf32($UnicodeInt0)
$Emoji1 = [System.Char]::ConvertFromUtf32($UnicodeInt1)
$Emoji2 = [System.Char]::ConvertFromUtf32($UnicodeInt2)
write-host "$($Emoji0), $($Emoji1), $($Emoji2)"
I create a passthru query SELECTINg rows of a Postgres v.11 database table.
Running currentDb.execute generates ODBC error 3146 and "invalid argument" mentioned in DBEngine.errors.
Opening same query in the database explorer generates ODBC error 3146 and a message box with "permission denied ..." which actually reflects the source of error.
My questions is how can I programmatically get hold of latter more informative error message ?
I think the following will provide what you are looking for:
Public Function DbEngineErrors() As String
Dim intErr As Integer
Dim strRet As String
Dim strErr As String
If DBEngine.Errors.Count > 0 Then
strRet = "DbEngineErrors:"
For intErr = 0 To DBEngine.Errors.Count - 1
strErr = DBEngine.Errors(intErr).Number & " / " & DBEngine.Errors(intErr).Description & " / " & DBEngine.Errors(intErr).Source
strRet = strRet & vbCrLf & strErr
Next
End If
DbEngineErrors = strRet
End Function
I am trying to create a program in Qbasic wherein a person can enter their name and label themselves as admin or unwanted user. How do I save these preferences in my program?
If you have inputed the username with something like,
INPUT "Type your username: ", uName$
To save it to a file, simply use these commands:
OPEN "User.dat" FOR OUTPUT AS #1
PRINT #1, uName$
CLOSE #1
Here's a complete program:
DEFINT A-Z
'Error handler for the first time we run the program. The data file won't exist, so we create it.
ON ERROR GOTO FileNotExist
'Create a type and an Array of users that would include Username and the Status (adminstrator vs. Unwanted user)
TYPE user
Uname AS STRING * 16
Status AS STRING * 1
END TYPE
DIM Users(1 TO 100) AS user
'Gets all the users stored in the file. i is a variable which represents the number of users before adding a new user
i = 0
OPEN "User.txt" FOR INPUT AS #1
WHILE NOT EOF(1)
i = i + 1
INPUT #1, Users(i).Uname
INPUT #1, Users(i).Status
WEND
CLOSE #1
TryAgain:
'Gets info for the new user
CLS
INPUT "User name: ", Users(i + 1).Uname
PRINT "Admin (a), Unwanted user (u), or Regular user (r) ?"
Users(i + 1).Status = LCASE$(INPUT$(1))
'Ensure there are no blank lines in the file
IF Users(i + 1).Uname = "" OR Users(i + 1).Status = "" THEN GOTO TryAgain
'Outputs user data to the file "User.txt"
OPEN "User.txt" FOR OUTPUT AS #1
FOR j = 1 TO i + 1
PRINT #1, Users(j).Uname
PRINT #1, Users(j).Status
NEXT j
CLOSE #1
'Just for a closer: Prints all the current users.
CLS
FOR j = 1 TO i + 1
PRINT Users(j).Uname,
IF Users(j).Status = "a" THEN PRINT "Amdinistrator" ELSE IF Users(j).Status = "u" THEN PRINT "Unwanted User" ELSE IF Users(j).Status = "r" THEN PRINT "Regular user" ELSE PRINT Users(j).Status
NEXT j
END
'*** ERROR HANDLER: ***
FileNotExist:
OPEN "User.txt" FOR OUTPUT AS #1
CLOSE
RESUME
To save a name into a file, you will need to use the WRITE statement.
Eg:
OPEN "Name.txt" FOR OUTPUT AS #1
INPUT"Enter a name";a$
WRITE #1,a$
CLOSE #1
END