Running curl.exe in PowerShell - powershell

I am trying to recreate a working curl command in PowerShell 5.1, but when I run it from the script, it errors out. If I pipe out the constructed command to a text file and run it as-is in a command shell it works. I resorted to attempting curl in PowerShell because I have been unable to get the multipart/form-data to work with the Invoke-RestMethod function. Here is the code I'm using, and the error message is shown below. This is on Windows 10 machine. In a nutshell, this is to upload a zip file to a remote server.
$accessToken = '<Bearer token value from prior API call>'
$inputFile = 'C:\MyFolder1\MyFolder2\MyFile.zip'
$curlCmd = 'C:\Curl\bin\curl.exe'
$uriImport = 'https://api.somecompany.com/import'
$curlArgs = '-X', 'POST',
'--header', '"Content-Type: multipart/form-data"',
'--header', '"Accept: application/json"',
'--header', -join('"Authorization: Bearer ', $accessToken, '"'),
'--form', -join('"files=#', $inputFile, ';type=application/zip"'),
-join('"', $uriImport, '"'), '-s'
Write-Host "$curlCmd $cURLargs"
"$curlCmd $curlArgs" | Out-File 'C:\MyFolder\MyFolder2\Curl_Output.txt'
& $curlCmd $curlArgs
The error PoSH returns:
curl.exe : % Total % Received % Xferd Average Speed Time Time Time Current
At C:\CurlTesting.ps1:65 char:9
+ & $curCmd $cURLargs
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ( % Total % ... Time Current:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
curl: (6) Could not resolve host: multipart
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (6) Could not resolve host: application
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (6) Could not resolve host: Bearer
curl: (3) URL using bad/illegal format or missing URL
I have tried constructing the quotes with both single and double, but it still errors out; the Windows command prompt does require double-quotes. When I run this same constructed script from a command window (cmd.exe), it runs and I don't get all of that output nor the error - just the expected value from the API.
What am I doing wrong?
Here is the modified, slimmed-down version of the code I got working:
$accessToken = '<Bearer token value from prior API call>'
$inputFile = 'C:\MyFolder1\MyFolder2\MyFile.zip'
$uriImport = 'https://api.somecompany.com/import'
curl.exe -X POST `
--header "Content-Type: multipart/form-data" `
--header "Accept: application/json" `
--header "Authorization: Bearer $accessToken" `
--form "filename=#$inputFile;type=application/zip" `
-s

This is what process monitor says is running. It seems ok to me. I don't have that curl but windows 10 (since 1803) comes with curl.exe now. Beware the curl alias in ps 5.
"C:\windows\system32\curl.exe" -X POST --header "Content-Type: multipart/form-data" --header "Accept: application/json" --header "Authorization: Bearer <Bearer token value from prior API call>" --form "files=#C:\MyFolder1\MyFolder2\MyFile.zip;type=application/zip" "https://api.somecompany.com/import" -s
I would run it this way. It's possible that powershell could lose doublequotes to an external command, that you would need to backslash, but this is as much as I can reproduce. It doesn't look like you're running it in silent mode, "-s".
C:\Curl\bin\curl.exe -X POST --header "Content-Type: multipart/form-data" --header "Accept: application/json" --header "Authorization: Bearer $accessToken" --form "files=#$inputFile;type=application/zip" $uriImport -s

Related

cURL fails in GitHub Actions

I'm running a Raspberry Pi 4 as a server for a .NET Core side-project of mine. Nothing too fancy or heavy. After trying to get going with a webhook and uploading files with scp to the Pi and failed (still don't know why at this point; scp problem might be the same as cURL problem), I decided to make myself a small API which accepts a file and deploys it to the specified path. The API is working both from inside and outside the Pi as I've tested it using cURL and Postman with a 20MB zip file, but when I run this command from inside a GitHub Action, I get a long waiting time and then a fail message.
Command:
curl --request POST --url https://example.com/ --header 'cache-control: no-cache' --form path=DEPLOY_PATH --form archive=#FILE_PATH --form token=TOKEN
Output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 8776k 0 0 0 65536 0 42750 0:03:30 0:00:01 0:03:29 42722
0 8776k 0 0 0 65536 0 25862 0:05:47 0:00:02 0:05:45 25852
0 8776k 0 0 0 65536 0 18533 0:08:04 0:00:03 0:08:01 18528
0 8776k 0 0 0 65536 0 14444 0:10:22 0:00:04 0:10:18 14441
0 8776k 0 0 0 65536 0 11833 0:12:39 0:00:05 0:12:34 13091
0 8776k 0 0 0 65536 0 10022 0:14:56 0:00:06 0:14:50 0
0 8776k 0 0 0 65536 0 8691 0:17:14 0:00:07 0:17:07 0
...
0 8776k 0 0 0 65536 0 63 39:37:37 0:17:11 39:20:26 0
0 8776k 0 0 0 65536 0 63 39:37:37 0:17:11 39:20:26 0
curl: (55) SSL_write() returned SYSCALL, errno = 110
##[error]Process completed with exit code 55.
With both scp and cURL commands there seems to be a common problem. If I try to send a simple text file or a tar.gz containing a text.file, it works. If I try to do the same with a .dll file or a tar.gz containing a .dll file, it does not. I don't really know if the problem is because of the files or their size. To be noted that the API accepts files as big as 100MB at the moment and I'm only trying to deploy a small package of ~10MB.
Output with -v arg:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying IP...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to URL (IP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [112 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2861 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=URL
* start date: Sep 18 16:51:41 2020 GMT
* expire date: Dec 17 16:51:41 2020 GMT
* subjectAltName: host "URL" matched cert's "URL"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
} [5 bytes data]
> POST /api/Deployment/ HTTP/1.1
> Host: URL
> User-Agent: curl/7.58.0
> Accept: */*
> cache-control: no-cache
> Content-Length: 3333
> Content-Type: multipart/form-data; boundary=------------------------ca1748c91973ca89
> Expect: 100-continue
>
{ [5 bytes data]
< HTTP/1.1 100 Continue
} [5 bytes data]
100 3333 0 0 100 3333 0 2154 0:00:01 0:00:01 --:--:-- 2153
100 3333 0 0 100 3333 0 1307 0:00:02 0:00:02 --:--:-- 1307
100 3333 0 0 100 3333 0 938 0:00:03 0:00:03 --:--:-- 938
100 3333 0 0 100 3333 0 732 0:00:04 0:00:04 --:--:-- 732
100 3333 0 0 100 3333 0 600 0:00:05 0:00:05 --:--:-- 614
100 3333 0 0 100 3333 0 508 0:00:06 0:00:06 --:--:-- 0
100 3333 0 0 100 3333 0 441 0:00:07 0:00:07 --:--:-- 0
...
100 3333 0 0 100 3333 0 57 0:00:58 0:00:57 0:00:01 0
100 3333 0 0 100 3333 0 56 0:00:59 0:00:58 0:00:01 0
100 3333 0 0 100 3333 0 55 0:01:00 0:00:59 0:00:01 0
100 3333 0 0 100 3333 0 54 0:01:01 0:01:00 0:00:01 0* Empty reply from server
100 3333 0 0 100 3333 0 54 0:01:01 0:01:00 0:00:01 0
* Connection #0 to host URL left intact
curl: (52) Empty reply from server
##[error]Process completed with exit code 52.
EDIT: Switching to a Windows runner instead of Ubuntu solved the cURL problem, but I'm still open to suggestions regarding this question, as this is merely a workaround rather than a solution.

Check value of PF_NO_SETAFFINITY

Is it possible to tell whether a process/thread has the PF_NO_SETAFFINITY flag set? I'm running taskset on a series of process ids and some are throwing errors of the following form:
taskset: failed to set pid 30's affinity: Invalid argument
I believe this is because some processes have PF_NO_SETAFFINITY set (see Answer).
Thank you!
Yes - look at /proc/PID/stat's 'flag' field
<linux/sched.h
#define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */
Look here for details on using /proc:
http://man7.org/linux/man-pages/man5/proc.5.html
https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk65143
Example:
ps -eaf
www-data 30084 19962 0 07:09 ? 00:00:00 /usr/sbin/apache2 -k start
...
cat /proc/30084/stat
30084 (apache2) S 19962 19962 19962 0 -1 4194624 554 0 3 0 0 0 0 0 20 0 1 0 298837672 509616128 5510 18446744073709551615 1 1 0 0 0 0 0 16781312 201346799 0 0 0 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0
The flags are 4194624
Q: Do you mind specifying how you'd write a simple script that outputs
true/false based on whether you're allowed to set affinity?
A: I don't feel comfortable providing this without the opportunity to test, but you can try something like this...
flags=$(cut -f 9 -d ' ' /proc/30084/stat)
echo $(($flags & 0x40000000))

How can I execute a shell script in Swift?

I am creating a macOS app with Swift that needs to download and run a shell script. I have been able to download the script with curl, but I can't run it. I am using a function from this answer to run other commands. When I use the App Sandbox in Xcode, I get the error /bin/bash: ./file.sh: Permission denied. When I try changing file permissions, I get the error chmod: Unable to change file mode on file.sh: Operation not permitted.
Here is my code:
func shell(_ command: String) -> String {
let task = Process()
task.launchPath = "/bin/bash"
task.arguments = ["-c", command]
let pipe = Pipe()
task.standardOutput = pipe
task.launch()
let data = pipe.fileHandleForReading.readDataToEndOfFile()
let output: String = NSString(data: data, encoding: String.Encoding.utf8.rawValue)! as String
return output
}
shell("curl https://www.example.com/file.sh -o file.sh")
shell("./file.sh")
Output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
1 16.0M 1 224k 0 0 231k 0 0:01:11 --:--:-- 0:01:11 231k
47 16.0M 47 7808k 0 0 4064k 0 0:00:04 0:00:01 0:00:03 4062k
100 16.0M 100 16.0M 0 0 6037k 0 0:00:02 0:00:02 --:--:-- 6037k
/bin/bash: ./file.sh: Permission denied

Error for REST API in nifi

when i start Executestream prosessor with src.bat file it shows me execution code 0 but it doesn't stop processor and insede outputstream flowfile it shows errors like this:
execution.error
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 299 0 0 100 299 0 1472 --:--:-- --:--:-- --:--:-- 1472
I use commands like this:
curl -X PUT --data-binary "#C:\Users\user\Desktop\nifi-1.3.0\conf\stop.json" "http://localhost:8080/nifi-api/processors/ea5db028-015d-1000-5ad5-80fd006dda92"
and json file like this, what should i do?:
{
"status": {
"runStatus": "STOPPED"
},
"component": {
"state": "STOPPED",
"id": "f511a6a1-015d-1000-970e-969eac1e6fc5"
},
"id": "f511a6a1-015d-1000-970e-969eac1e6fc5",
"revision": {
"version": 30,
"clientId": "0343f0b9-015e-1000-7cd8-570f8953ec11"
}
}
I suspect that if curl is reporting progress then somehow the output/process is reporting strangely to NiFi. Perhaps try the "silent" option for your curl command (see the documentation for how to enable that, often it's the -s flag)

Listing the users who have actually submitted jobs in LSF

I am use the following command to list all users and there pending jobs in LSF:
busers -w all
Example output:
USER/GROUP JL/P MAX NJOBS PEND RUN SSUSP USUSP RSV MPEND
aaaaa - - 0 0 0 0 0 0 200
bbbbb - - 100 50 50 0 0 0 200
ccccc - - 1 0 1 0 0 0 200
I'm looking for a command that will only display the users whose NJOBS value is greater than 0, i.e. those who have actually submitted at least one job.
In the example, that would mean that only the lines for users 'bbbbb' and 'ccccc' should appear.
To check all users jobs the simpler method would be:
bjobs -u all
To check suspended ones just use:
bjobs -u all | grep SUSP