Trying to run Spark Submit, Hadoop and other command line commands - powershell

I'm using PowerShell to set up a test instance that is running on Windows. When the instance is up and running it will then run a few commands to get Hadoop set up and will then run a Spark job.
This all works fine when done manually from within the instance itself. I'm now trying to translate those commands into powershell.
These two for example are failing with the message that it is not a recognised cmdlet or function etc:
& $env:HADOOP_HOME + "\bin\winutils.exe" chmod 777 /tmp/hive
& $env:HADOOP_HOME + "\bin" hadoop namenode -format -force
The error I receive is:
& : The term 'c:\hadoop\bin' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:3
I've then tried various combinations of wrapping in quotes, using iex, assigning it to a variable, but all fail.

PowerShell interprets the expanded value of $env:HADOOP_HOME as the command to execute, which fails, because a folder isn't a cmdlet, function, script file, or operable program. Do the concatenation in a subexpression:
& ($env:HADOOP_HOME + "\winutils.exe") chmod 777 /tmp/hive
or put the environment variable directly in the command string:
& "$env:HADOOP_HOME\winutils.exe" chmod 777 /tmp/hive

Related

Error 'mongodump' is not recognized as an internal or external command, operable program or batch file

I have read the documentation and followed the steps to run mongodump on windows machine, but still getting this error.
I tried to set the path using this command set path="C:\Program Files\MongoDB\Server\4.4\bin" and added the path in environmental variables but still getting the error. What should I do ?

How to create Greengrass group(aws iot)

I need to create Greengrass group and core in aws iot(windows).
I have referred with the document https://docs.aws.amazon.com/cli/latest/reference/greengrass/create-group.html
I have tried with powershell script aws greengrass create-group \ --name ggawsgreen
Gets error when executing above powershell script . error => aws : The term 'aws' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again
How to create Greengrass group and core(aws iot) in windows powershell
the error appears because you haven't installed the AWS Command Line Interface or aws cli for short.
Or ´aws cli´ is not in your path.
download here

Anaconda Powershell Prompt not exporting environment to .yml file

I'm trying to create a new environment from file, when i run conda env export > environment.yml the file is created but when running vim environment.yml i get this error
vim : The term 'vim' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is correct and try again.
when trying to run conda env create --prefix .\env -f ..\environment.yml on my new project directory i get An unexpected error has occurred.

Run output of command as powershell incantation

I have a bash script that generates a program incantation (docker run ... with a bunch of arguments) and prints it to standard output. I want to run the output from powershell, with the optional possibility to tack on other arguments afterwards.
I've tried
D:\> & $(wsl ./my-bash-script.sh) some more args
but I get the error
The term 'docker run --rm -ti [<redacted>]' is not recognized as the name of cmdlet, function, script file, or operable program. Check the spelling of the name, or ifa path was included, verify that the path is correct and try again.
Apparently, & interprets the entire output as a single command, when in fact it's only the first word of the output that is a command; the rest is arguments.
Is there a way to do what I want?
Yes, & only supports a command name or path as its first argument - you cannot include arguments.
What you need is Invoke-Expression:
Invoke-Expression "$(wsl ./my-bash-script.sh) some more args"
Note: You'd only need & inside the string if the .sh script output a command with a quoted executable path.
Also note that PowerShell will not always interpret a given command line the same way as a POSIX-like shell (such as Bash), given that it recognizes additional metacharacters at the start of arguments; notably, arguments prefixed with # or { will not be taken as literals by PowerShell.
As an aside: While this is a legitimate use case for Invoke-Expression - assuming you trust the output from your .sh script - Invoke-Expression should generally be avoided.

Trying to update SVN using Powershell in a SQL Server Job as the Service account

I have a powershell script that I developed and can run fine manually. However when I put the script into a sql server job and attempt to run the following line...
SET-LOCATION "$NewTargetPath\"
TortoiseProc /command:update /path:$SVNRepository /closeonend:3
I get the following error:
Executed as user: myserviceaccount. A job step received an error at line 146 in a PowerShell script. The corresponding line is 'TortoiseProc /command:update /path:$SVNRepository /closeonend:3'. Correct the script and reschedule the job. The error information returned by PowerShell is: 'The term 'TortoiseProc' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. '. Process Exit Code -1. The step failed.
I've checked the paths they all look good and I can even run the command from a powershell command line on the server. Any ideas on where to look for why this error may be occurring?