In a cloud datafusion wrangler I can perform multiple transformation using directives. The scenario that i am looking into is there are a set of directives which might change with each execution.
for e.g.
some-directives
.
.
set-column :concat_col exp:{concat(col1,col2)}
hash concat_col SHA-256 true
.
.
some-other-directives
I want to parameterize the part of the directive. I used the below steps
some-directives
.
.
${custom_directive}
.
.
some-other-directives
This works if i pass only 1 directive as the parameter value i.e. set-column :concat_col exp:{concat(col1,col2)}
However when i try to pass more than 1 directive the pipeline fails.
I have tried the below formats for the parameter value
set-column :concat_col exp:{concat(col1,col2)};hash concat_col SHA-256 true
throws an error expected column name but got an identifier
set-column :concat_col exp:{concat(col1,col2)}\r\nhash concat_col SHA-256 true
pipeline fails with the error expecting ; but got \r
set-column :concat_col exp:{concat(col1,col2)} \r\nhash concat_col SHA-256 true
succeeds but does not perform the hash
"set-column :concat_col exp:{concat(col1,col2)}\r\nhash concat_col SHA-256 true"
pipeline succeeds but doesn't perform both the steps
set-column :concat_col exp:{concat(col1,col2)},hash concat_col SHA-256 true
pipeline fails with the error expected ; but got ,
Is there a way i can pass multiple lines of directives to a single parameter?
Supplying multiple directives to a single parameter in the wrangler works if we send the parameters via HTTP request. each line of directive needs to be separated by a \r\n character.
Strangely it does not not work when we manually enter the values in the runtime args
Related
I have a Rundeck job where I'm loading the env Option values from remote file.
File has values as shown below
domain-dev
In NodeFilter CommandLine Argument, I'm loading values as environment: ${option.env}
I want to do substring of the values of env till -. Eg: I want to give domain to NodeFilter
is it possible to do something like below
NodeFilter = environment: ${option.env.substring(0, option.env.indexOf("-"))}
Is not possible in that way, but you can use regular expressions. To get values from a specific string. For domain-dev you can use this regex name: ^([^-]*).*$ to get only the string before the - character, of course, this example uses the tags attribute to apply the filter.
You can use regex101 or freeformater to test your regex patterns.
I have an Azure Devops Pipeline Task for .NET Core, specifying the 'test' command.
One of the parameters is 'Path to project(s)'. You can specify multiple globbing patterns, each on its own line.
Now I want to make this value settable at queueing time by using a variable $(UnitTestPatterns). But a variable cannot have a multiline value. How can I specify 2 or more globbing patterns in a way that all are evaluated?
I have tried a pipe '|', a comma ',' and a semicolon ';' as separators, none have worked. The log of the task then shows ##[warning]Project file(s) matching the specified pattern were not found.
Example multiline value:
**/Project.*.Tests/*.csproj
!**/Project.Module2.Tests/*.csproj
I want a variable like this (with probably some secret separator, the ';' does not work):
$(UnitTestPatterns) = **/Project.*.Tests/*.csproj; !**/Project.Module2.Tests/*.csproj
I am using the UI to set the variable, not YAML.
It looks that this is not supported. Please check this topic on Developer Community. You may also check this topic where you will find workaround.
I've got a client-side script I'm making that communicates with GNU-FTP. I want to be able to send it a custom argument on the command line, so I've created an argument --ftp-args
This is what it looks like
GetOptions(
.. redacted stuff..
"ftp-args=s%" => \$FTP_ARGS
) or die("Error in command line arguments\n");
However, whenever I try to call it I get an error,
$ ./script/dm-ftp360 --ftp-args="-E"
Option ftp-args, key "-E", requires a value
Error in command line arguments
Is it possible to get around this, and make this possible?
You've specified s% - defining an option that specifies hash entries. That implies a form key=value for each argument to the option. But you've only specified -E. The error message is about the missing =value part, not the leading -.
Perhaps use s# instead to ingest a set of simple options? Or give an empty value using "-E=" if you need to separate the keys and values before passing them to ftp.
I'm passing a URL parameter to a form via a get request. I need to URL encode the parameter when the parameter contains a '#' . Otherwise the request fails. Why is this required ? Why do I need to URL encode the '#' parameter but not other text ?
'#' is used in URLs to indicate where a fragment identifier
(bookmarks/anchors in HTML) begins.
The part following the # is never seen by the server. It is generally used for navigation at the client-end.
The following characters need to be encoded in order to be used literally.
When using GET, anything after # (and the # itself) will not be seen by the server.
This is a weird one. Normally when I execute an external command from powershell I use the & operator like this:
& somecommand.exe -p somearguments
However, today I came across the . operator used like this:
.$env:systemdrive\chocolatey\chocolateyinstall\chocolatey.cmd install notepadplusplus
What purpose does the period serve in this scenario? I don't get it.
The "." dot sourcing operator will send AND receive variables from other scripts you have called. The "&" call operator will ONLY send variables.
For instance, considering the following:
Script 1 (call-operator.ps1):
clear
$funny = "laughing"
$scriptpath = split-path -parent $MyInvocation.MyCommand.Definition
$filename = "laughing.ps1"
"Example 1:" # Call another script. Variables are passed only forward.
& $scriptpath\$filename
"Example 2:" # Call another script. Variables are passed backwards and forwards.
. $scriptpath\$filename
$variableDefinedInOtherScript
Script 2 (laughing.ps1):
# This is to test the passing of variables from call-operator.ps1
"I am $funny so hard. Passing variables is so hilarious."
$variableDefinedInOtherScript = "Hello World!"
Create both scripts and ONLY run the first one. You'll see that the "." dot sourcing operator sends and receives variables.
Both have their uses, so be creative. For instance, the "&" call operator would be useful if you wanted to modify the value(s) of variables in another script while preserving the original value(s) in the your current script. Kinda a safeguard. ;)
The Short:
It is a Special Operator used to achieve what regular operators cannot achieve. This particular operator . actually has two distinctively different Special Operator use cases.
The Long:
As with any other language, scripting or otherwise, PowerShell script also supports many different types of Operators to help manipulate values. These regular operators include:
Arithmetic
Assignment
Comparison
Logical
Redirection
List item
Split and Join
Type
Unary
However, PowerShell also supports whats known as Special Operators which are used to perform tasks that cannot be performed by the other types of operators.
These Special Operators Include:
#() Array subexpression operator
& Call operator
[ ] Cast operator
, Comma operator
. Dot sourcing operator
-f Format operator
[ ] Index operator
| Pipeline operator
. Property dereference operator
.. Range operator
:: Static member operator
$( ) Subexpression operator
. Dot sourcing operator: is used in this context to allow a script to run in the current scope essentially allowing any functions, aliases, and variables which has been created by the script to be added to the current script.
Example:
. c:\scripts.sample.ps1
NoteThat this application of the . Special Operator is followed by a space to distinguish it from the (.) symbol that represents the current directory
Example:
. .\sample.ps1
. Property dereference operator: Allows access to the properties and methods of of an object which follows the . by indicating that the expression on the left side of the . character is an object and the expression on the right side of the is an object member (a property or method).
Example:
$myProcess.peakWorkingSet
(get-process PowerShell).kill()
Disclaimer & Sources:
I had the same question while looking at a PowerShell script that I was trying to expand on its feature sets and landed here when doing my research for the answer. However I managed to find my answer using this magnificent write up on the Microsoft Development Network supplemented with this further expansion of the same ideas from IT Pro.
Cheers.
The dot is a call operator:
$a = "Get-ChildItem"
. $a # (executes Get-ChildItem in the current scope)
In your case, however, I don't see what it does.
.Period or .full stop for an objects properties; like
$CompSys.TotalPhysicalMemory
See here: http://www.computerperformance.co.uk/powershell/powershell_syntax.htm#Operators_
This answer is to expand slightly upon those already provided by David Brabant and his commenters. While those remarks are all true and pertinent, there is something that has been missed.
The OPs use of & when invoking external commands is unnecessary. Omitting the & would have no effect (on the example of his usage). The purpose of & is to allow the invocation of commands whose names are the values of a (string) expression. By using the & above, powershell then (essentially) treats the subsequent arguments as strings, the first of which is the command name that & duly invokes. If the & were omitted, powershell would take the first item on the line as the command to execute.
However, the . in the second example is necessary (although, as noted by others, & would work just as well in this case). Without it, the command line would begin with a variable access ($env:systemdrive) and so powershell would be expecting an expression of some form. However, immediately following the variable reference is a bare file path which is not a valid expression and will generate an error. By using the . (or &) at the beginning of the line, it is now treated as a command (because the beginning doesn't look like a valid expression) and the arguments are processed as expandable strings (" "). Thus, the command line is treated as
. "$env:systemdrive\chocolatey\chocolateyinstall\chocolatey.cmd" "install" "notepadplusplus"
The first argument has $env:systemdrive substituted into it and then . invokes the program thus named.
Note: the full description of how powershell processes command line arguments is way more complicated than that given here. This version was cut down to just the essential bits needed to answer the question. Take a look at about_Parsing for a comprehensive description. It is not complete but should cover most normal usage. There are other posts on stackoverflow and github (where powershell now resides) that cover some of the seemingly quirky behaviour not listed in the official documentation. Another useful resource is about_Operators though again this isn't quite complete. An example being the equivalence of . and & when invoking something other than a powershell script/cmdlet getting no mention at all.