when i start Executestream prosessor with src.bat file it shows me execution code 0 but it doesn't stop processor and insede outputstream flowfile it shows errors like this:
execution.error
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 299 0 0 100 299 0 1472 --:--:-- --:--:-- --:--:-- 1472
I use commands like this:
curl -X PUT --data-binary "#C:\Users\user\Desktop\nifi-1.3.0\conf\stop.json" "http://localhost:8080/nifi-api/processors/ea5db028-015d-1000-5ad5-80fd006dda92"
and json file like this, what should i do?:
{
"status": {
"runStatus": "STOPPED"
},
"component": {
"state": "STOPPED",
"id": "f511a6a1-015d-1000-970e-969eac1e6fc5"
},
"id": "f511a6a1-015d-1000-970e-969eac1e6fc5",
"revision": {
"version": 30,
"clientId": "0343f0b9-015e-1000-7cd8-570f8953ec11"
}
}
I suspect that if curl is reporting progress then somehow the output/process is reporting strangely to NiFi. Perhaps try the "silent" option for your curl command (see the documentation for how to enable that, often it's the -s flag)
Related
I'm running a Raspberry Pi 4 as a server for a .NET Core side-project of mine. Nothing too fancy or heavy. After trying to get going with a webhook and uploading files with scp to the Pi and failed (still don't know why at this point; scp problem might be the same as cURL problem), I decided to make myself a small API which accepts a file and deploys it to the specified path. The API is working both from inside and outside the Pi as I've tested it using cURL and Postman with a 20MB zip file, but when I run this command from inside a GitHub Action, I get a long waiting time and then a fail message.
Command:
curl --request POST --url https://example.com/ --header 'cache-control: no-cache' --form path=DEPLOY_PATH --form archive=#FILE_PATH --form token=TOKEN
Output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 8776k 0 0 0 65536 0 42750 0:03:30 0:00:01 0:03:29 42722
0 8776k 0 0 0 65536 0 25862 0:05:47 0:00:02 0:05:45 25852
0 8776k 0 0 0 65536 0 18533 0:08:04 0:00:03 0:08:01 18528
0 8776k 0 0 0 65536 0 14444 0:10:22 0:00:04 0:10:18 14441
0 8776k 0 0 0 65536 0 11833 0:12:39 0:00:05 0:12:34 13091
0 8776k 0 0 0 65536 0 10022 0:14:56 0:00:06 0:14:50 0
0 8776k 0 0 0 65536 0 8691 0:17:14 0:00:07 0:17:07 0
...
0 8776k 0 0 0 65536 0 63 39:37:37 0:17:11 39:20:26 0
0 8776k 0 0 0 65536 0 63 39:37:37 0:17:11 39:20:26 0
curl: (55) SSL_write() returned SYSCALL, errno = 110
##[error]Process completed with exit code 55.
With both scp and cURL commands there seems to be a common problem. If I try to send a simple text file or a tar.gz containing a text.file, it works. If I try to do the same with a .dll file or a tar.gz containing a .dll file, it does not. I don't really know if the problem is because of the files or their size. To be noted that the API accepts files as big as 100MB at the moment and I'm only trying to deploy a small package of ~10MB.
Output with -v arg:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying IP...
* TCP_NODELAY set
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to URL (IP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [112 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2861 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=URL
* start date: Sep 18 16:51:41 2020 GMT
* expire date: Dec 17 16:51:41 2020 GMT
* subjectAltName: host "URL" matched cert's "URL"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
} [5 bytes data]
> POST /api/Deployment/ HTTP/1.1
> Host: URL
> User-Agent: curl/7.58.0
> Accept: */*
> cache-control: no-cache
> Content-Length: 3333
> Content-Type: multipart/form-data; boundary=------------------------ca1748c91973ca89
> Expect: 100-continue
>
{ [5 bytes data]
< HTTP/1.1 100 Continue
} [5 bytes data]
100 3333 0 0 100 3333 0 2154 0:00:01 0:00:01 --:--:-- 2153
100 3333 0 0 100 3333 0 1307 0:00:02 0:00:02 --:--:-- 1307
100 3333 0 0 100 3333 0 938 0:00:03 0:00:03 --:--:-- 938
100 3333 0 0 100 3333 0 732 0:00:04 0:00:04 --:--:-- 732
100 3333 0 0 100 3333 0 600 0:00:05 0:00:05 --:--:-- 614
100 3333 0 0 100 3333 0 508 0:00:06 0:00:06 --:--:-- 0
100 3333 0 0 100 3333 0 441 0:00:07 0:00:07 --:--:-- 0
...
100 3333 0 0 100 3333 0 57 0:00:58 0:00:57 0:00:01 0
100 3333 0 0 100 3333 0 56 0:00:59 0:00:58 0:00:01 0
100 3333 0 0 100 3333 0 55 0:01:00 0:00:59 0:00:01 0
100 3333 0 0 100 3333 0 54 0:01:01 0:01:00 0:00:01 0* Empty reply from server
100 3333 0 0 100 3333 0 54 0:01:01 0:01:00 0:00:01 0
* Connection #0 to host URL left intact
curl: (52) Empty reply from server
##[error]Process completed with exit code 52.
EDIT: Switching to a Windows runner instead of Ubuntu solved the cURL problem, but I'm still open to suggestions regarding this question, as this is merely a workaround rather than a solution.
I am trying to recreate a working curl command in PowerShell 5.1, but when I run it from the script, it errors out. If I pipe out the constructed command to a text file and run it as-is in a command shell it works. I resorted to attempting curl in PowerShell because I have been unable to get the multipart/form-data to work with the Invoke-RestMethod function. Here is the code I'm using, and the error message is shown below. This is on Windows 10 machine. In a nutshell, this is to upload a zip file to a remote server.
$accessToken = '<Bearer token value from prior API call>'
$inputFile = 'C:\MyFolder1\MyFolder2\MyFile.zip'
$curlCmd = 'C:\Curl\bin\curl.exe'
$uriImport = 'https://api.somecompany.com/import'
$curlArgs = '-X', 'POST',
'--header', '"Content-Type: multipart/form-data"',
'--header', '"Accept: application/json"',
'--header', -join('"Authorization: Bearer ', $accessToken, '"'),
'--form', -join('"files=#', $inputFile, ';type=application/zip"'),
-join('"', $uriImport, '"'), '-s'
Write-Host "$curlCmd $cURLargs"
"$curlCmd $curlArgs" | Out-File 'C:\MyFolder\MyFolder2\Curl_Output.txt'
& $curlCmd $curlArgs
The error PoSH returns:
curl.exe : % Total % Received % Xferd Average Speed Time Time Time Current
At C:\CurlTesting.ps1:65 char:9
+ & $curCmd $cURLargs
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ( % Total % ... Time Current:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
curl: (6) Could not resolve host: multipart
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (6) Could not resolve host: application
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (6) Could not resolve host: Bearer
curl: (3) URL using bad/illegal format or missing URL
I have tried constructing the quotes with both single and double, but it still errors out; the Windows command prompt does require double-quotes. When I run this same constructed script from a command window (cmd.exe), it runs and I don't get all of that output nor the error - just the expected value from the API.
What am I doing wrong?
Here is the modified, slimmed-down version of the code I got working:
$accessToken = '<Bearer token value from prior API call>'
$inputFile = 'C:\MyFolder1\MyFolder2\MyFile.zip'
$uriImport = 'https://api.somecompany.com/import'
curl.exe -X POST `
--header "Content-Type: multipart/form-data" `
--header "Accept: application/json" `
--header "Authorization: Bearer $accessToken" `
--form "filename=#$inputFile;type=application/zip" `
-s
This is what process monitor says is running. It seems ok to me. I don't have that curl but windows 10 (since 1803) comes with curl.exe now. Beware the curl alias in ps 5.
"C:\windows\system32\curl.exe" -X POST --header "Content-Type: multipart/form-data" --header "Accept: application/json" --header "Authorization: Bearer <Bearer token value from prior API call>" --form "files=#C:\MyFolder1\MyFolder2\MyFile.zip;type=application/zip" "https://api.somecompany.com/import" -s
I would run it this way. It's possible that powershell could lose doublequotes to an external command, that you would need to backslash, but this is as much as I can reproduce. It doesn't look like you're running it in silent mode, "-s".
C:\Curl\bin\curl.exe -X POST --header "Content-Type: multipart/form-data" --header "Accept: application/json" --header "Authorization: Bearer $accessToken" --form "files=#$inputFile;type=application/zip" $uriImport -s
I am creating a macOS app with Swift that needs to download and run a shell script. I have been able to download the script with curl, but I can't run it. I am using a function from this answer to run other commands. When I use the App Sandbox in Xcode, I get the error /bin/bash: ./file.sh: Permission denied. When I try changing file permissions, I get the error chmod: Unable to change file mode on file.sh: Operation not permitted.
Here is my code:
func shell(_ command: String) -> String {
let task = Process()
task.launchPath = "/bin/bash"
task.arguments = ["-c", command]
let pipe = Pipe()
task.standardOutput = pipe
task.launch()
let data = pipe.fileHandleForReading.readDataToEndOfFile()
let output: String = NSString(data: data, encoding: String.Encoding.utf8.rawValue)! as String
return output
}
shell("curl https://www.example.com/file.sh -o file.sh")
shell("./file.sh")
Output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
1 16.0M 1 224k 0 0 231k 0 0:01:11 --:--:-- 0:01:11 231k
47 16.0M 47 7808k 0 0 4064k 0 0:00:04 0:00:01 0:00:03 4062k
100 16.0M 100 16.0M 0 0 6037k 0 0:00:02 0:00:02 --:--:-- 6037k
/bin/bash: ./file.sh: Permission denied
I'm trying to make a simple api application using beego. During the stress test, there was an unexpected problem. Before ~16400 requests everything executes at fantastic speed. After 16400 queries almost all stops, runs 1-2 requests per second. I have a feeling that beego can not allocate a connection to the database. I tried to change maxIdle, maxConn parameters but no effect.
UPD. the same problem with other databases
MainController:
package controllers
import (
models "github.com/Hepri/taxi/models"
"github.com/astaxie/beego"
"github.com/astaxie/beego/orm"
)
type MainController struct {
beego.Controller
}
func (c *MainController) Get() {
o := orm.NewOrm()
app := models.ApiApp{}
err := o.Read(&app)
if err == orm.ErrMissPK {
// do nothing
}
c.ServeJson()
}
Model:
package models
const (
CompanyAccessTypeAll = 1
CompanyAccessTypeSpecific = 2
)
type ApiApp struct {
Id int `orm:"auto"`
Token string `orm:"size(100)"`
}
func (a *ApiApp) TableName() string {
return "api_apps"
}
main.go:
package main
import (
models "github.com/Hepri/taxi/models"
_ "github.com/Hepri/taxi/routers"
"github.com/astaxie/beego"
"github.com/astaxie/beego/orm"
_ "github.com/lib/pq"
)
func main() {
orm.RegisterDriver("postgres", orm.DR_Postgres)
orm.RegisterDataBase("default", "postgres", "user=test password=123456 dbname=test sslmode=disable")
orm.RegisterModel(new(models.ApiApp))
beego.EnableAdmin = true
orm.RunCommand()
beego.Run()
}
before reach ~16400:
Benchmarking localhost (be patient)
^C
Server Software: beegoServer:1.4.2
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 4 bytes
Concurrency Level: 10
Time taken for tests: 3.844 seconds
Complete requests: 16396
Failed requests: 0
Write errors: 0
Total transferred: 2492192 bytes
HTML transferred: 65584 bytes
Requests per second: 4264.91 [#/sec] (mean)
Time per request: 2.345 [ms] (mean)
Time per request: 0.234 [ms] (mean, across all concurrent requests)
Transfer rate: 633.07 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 2.2 0 275
Processing: 0 2 10.9 1 370
Waiting: 0 1 8.6 1 370
Total: 0 2 11.1 2 370
Percentage of the requests served within a certain time (ms)
50% 2
66% 2
75% 2
80% 2
90% 2
95% 3
98% 3
99% 4
100% 370 (longest request)
after reach ~16400:
Benchmarking localhost (be patient)
^C
Server Software: beegoServer:1.4.2
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 4 bytes
Concurrency Level: 10
Time taken for tests: 15.534 seconds
Complete requests: 16392
Failed requests: 0
Write errors: 0
Total transferred: 2491584 bytes
HTML transferred: 65568 bytes
Requests per second: 1055.22 [#/sec] (mean)
Time per request: 9.477 [ms] (mean)
Time per request: 0.948 [ms] (mean, across all concurrent requests)
Transfer rate: 156.63 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 11
Processing: 0 2 16.7 1 614
Waiting: 0 1 15.7 1 614
Total: 0 2 16.7 1 614
Percentage of the requests served within a certain time (ms)
50% 1
66% 1
75% 2
80% 2
90% 2
95% 2
98% 3
99% 3
100% 614 (longest request)
same picture even after 30 seconds
Benchmarking localhost (be patient)
^C
Server Software: beegoServer:1.4.2
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 4 bytes
Concurrency Level: 10
Time taken for tests: 25.585 seconds
Complete requests: 16391
Failed requests: 0
Write errors: 0
Total transferred: 2491432 bytes
HTML transferred: 65564 bytes
Requests per second: 640.65 [#/sec] (mean)
Time per request: 15.609 [ms] (mean)
Time per request: 1.561 [ms] (mean, across all concurrent requests)
Transfer rate: 95.10 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 10.1 0 617
Processing: 0 2 16.2 1 598
Waiting: 0 1 11.1 1 597
Total: 0 2 19.1 1 618
Percentage of the requests served within a certain time (ms)
50% 1
66% 2
75% 2
80% 2
90% 2
95% 2
98% 3
99% 3
100% 618 (longest request)
I have a network program written in C using TCP sockets. Sometimes the client program hangs forever expecting input from server. Specifically, the client hangs on select() call set on an fd intended to read characters sent by server.
I am using strace to know where the process got stuck. However, sometimes when I attach the hung client process to strace, it immediately resumes it's execution and properly exits. Not all hung processes exhibit this behavior, some processes stuck in the select() even if I attach them to strace. But most of the processes resume their execution when attached to strace.
I am curious what causing the processes resume when attached to strace. It might give me clues to know why client processes are getting hung.
Any ideas? what causes a hung process to resume it's execution when attached to strace?
Update:
Here's the output of strace on hung processes.
> sudo strace -p 25645
Process 25645 attached - interrupt to quit
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
[ Process PID=25645 runs in 32 bit mode. ]
select(6, [3 5], NULL, NULL, NULL) = 2 (in [3 5])
read(5, "\0", 8192) = 1
write(2, "", 0) = 0
read(3, "====Setup set_oldtempbehaio"..., 8192) = 555
write(1, "====Setup set_oldtempbehaio"..., 555) = 555
select(6, [3 5], NULL, NULL, NULL) = 2 (in [3 5])
read(5, "", 8192) = 0
read(3, "", 8192) = 0
close(5) = 0
kill(25652, SIGKILL) = 0
exit_group(0) = ?
Process 25645 detached
_
> sudo strace -p 14462
Process 14462 attached - interrupt to quit
[ Process PID=14462 runs in 32 bit mode. ]
read(0, 0xff85fdbc, 8192) = -1 EIO (Input/output error)
shutdown(3, 1 /* send */) = 0
exit_group(0) = ?
_
> sudo strace -p 7517
Process 7517 attached - interrupt to quit
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
--- SIGSTOP (Stopped (signal)) # 0 (0) ---
[ Process PID=7517 runs in 32 bit mode. ]
connect(3, {sa_family=AF_INET, sin_port=htons(300), sin_addr=inet_addr("100.64.220.98")}, 16) = -1 ETIMEDOUT (Connection timed out)
close(3) = 0
dup(2) = 3
fcntl64(3, F_GETFL) = 0x1 (flags O_WRONLY)
close(3) = 0
write(2, "dsd13: Connection timed out\n", 30) = 30
write(2, "Error code : 110\n", 17) = 17
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(1) = ?
Process 7517 detached
Not just select(), but the processes(of same program) are stuck in various system calls before I attach them to strace. They suddenly resume after attaching to strace. If I don't attach them to strace, they just hang there forever.
Update 2:
I learned that strace could start a process which was previously stopped (process in T sate). Now I am trying to understand why did these processes go to 'T' state, what's the cause. Here's the /proc//status information:
> cat /proc/12554/status
Name: someone
State: T (stopped)
SleepAVG: 88%
Tgid: 12554
Pid: 12554
PPid: 9754
TracerPid: 0
Uid: 5000 5000 5000 5000
Gid: 48986 48986 48986 48986
FDSize: 256
Groups: 9149 48986
VmPeak: 1992 kB
VmSize: 1964 kB
VmLck: 0 kB
VmHWM: 608 kB
VmRSS: 608 kB
VmData: 156 kB
VmStk: 20 kB
VmExe: 16 kB
VmLib: 1744 kB
VmPTE: 20 kB
Threads: 1
SigQ: 54/73728
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000006
SigCgt: 0000000000004000
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
Cpus_allowed: 00000000,00000000,00000000,0000000f
Mems_allowed: 00000000,00000001
strace uses ptrace. The ptrace man page has this:
Since attaching sends SIGSTOP and the tracer usually suppresses it,
this may cause a stray EINTR return from the currently executing system
call in the tracee, as described in the "Signal injection and
suppression" section.
Are you seeing select return EINTR?