I'm using scala-ssh 0.7.0 to download a file from a remote server.
The download itself seems to be successful (I can find the downloaded file). Here is the last part of the log output:
16:14:43.824 [main] TRACE
net.schmizz.sshj.xfer.LoggingTransferListener -- transferred 100% of
log-file.txt 16:14:43.824 [main] INFO
net.schmizz.sshj.common.StreamCopier -- 44.82421875 KiB transferred in
1.572 seconds (28.514134064885496 KiB/s) 16:14:43.825 [main] DEBUG net.schmizz.sshj.xfer.scp.SCPEngine -- Remote agrees transfer done
16:14:43.826 [main] DEBUG net.schmizz.sshj.xfer.scp.SCPEngine --
Signalling: Transfer done 16:14:43.826 [main] TRACE
net.schmizz.sshj.transport.Encoder -- Encoding packet #10: 5e 00 00 00
00 00 00 00 01 00 16:14:43.826 [main] DEBUG
net.schmizz.sshj.connection.channel.Window$Remote -- Consuming by 1
down to 2097148 16:14:44.220 [reader] TRACE
net.schmizz.sshj.transport.Decoder -- Received packet #14: 62 00 00 00
00 00 00 00 0b 65 78 69 74 2d 73 74 61 74 75 73 00 00 00 00 00
16:14:44.221 [reader] TRACE net.schmizz.sshj.transport.TransportImpl
-- Received packet CHANNEL_REQUEST 16:14:44.221 [reader] DEBUG net.schmizz.sshj.connection.channel.direct.SessionChannel -- Got chan
request for exit-status 16:14:44.222 [reader] TRACE
net.schmizz.sshj.transport.Decoder -- Received packet #15: 60 00 00 00
00 16:14:44.222 [reader] TRACE
net.schmizz.sshj.transport.TransportImpl -- Received packet
CHANNEL_EOF 16:14:44.222 [reader] DEBUG
net.schmizz.sshj.connection.channel.direct.SessionChannel -- Got EOF
16:14:44.223 [reader] TRACE net.schmizz.sshj.transport.Decoder --
Received packet #16: 61 00 00 00 00 16:14:44.223 [reader] TRACE
net.schmizz.sshj.transport.TransportImpl -- Received packet
CHANNEL_CLOSE 16:14:44.223 [main] DEBUG
net.schmizz.sshj.connection.channel.direct.SessionChannel -- Sending
close 16:14:44.223 [reader] DEBUG
net.schmizz.sshj.connection.channel.direct.SessionChannel -- Got close
16:14:44.224 [main] TRACE net.schmizz.sshj.transport.Encoder --
Encoding packet #11: 61 00 00 00 00 16:14:44.225 [main] DEBUG
net.schmizz.concurrent.Promise -- Awaiting <>
16:14:44.225 [reader] DEBUG net.schmizz.sshj.connection.ConnectionImpl
-- Forgetting session channel (#0)
16:14:44.225 [reader] DEBUG net.schmizz.concurrent.Promise -- Setting <> to SOME
My code is very simple:
import com.decodified.scalassh._
val sshClient = new SshClient(new HostConfig(login = PasswordLogin(userName, SimplePasswordProducer(password)), hostName))
sshClient.download(filePath, targetDirectory)
Now, although the download finishes successfully, and the application execution continues after the download command, the application itself doesn't exit.
Any ideas?
The implicit conversion to RichSshClient seems to be behind this.
I avoided the implicit convertion to RichSshClient by creating a new class -- 'SshClientEnhanced' which extends SshClient with ScpTransferable. Invoking download method on an instance of SshClientEnhanced works fine.
Related
I'm using Akka Streams to create a TCP server using akka.stream.scaladsl.TLS with client certificate authentication. I'm working on creating an echo server as a first proof of concept.
In the meantime, I'm new to Scala/Akka/Akka Streams and so I created a similar server and TCP client in Python to provide tooling in testing my work in Scala. The Python server/client are functional using client cert authentication. When connecting to the server, the client takes the following steps:
Creates and configures an SSLContext
Creates a socket using socket.create_connection()
Wraps the socket with the SSLContext using SSLContext.wrap_socket(). This creates the peer connection
Once connected, prints the server certificate
Infinite loop asking for input and sending each input to the server
I believe I have the server completed using Akka Streams and akka.stream.scaladsl.TLS, but when I attempt to connect using my Python client the client never gets past connecting to the peer using context.wrap_socket(sock, server_hostname=host). The server successfully binds the TCP connection and creates the corresponding IncomingConnection object. The client/server also never timeout (the client just sits awaiting the handshake?).
My biggest problem is that I see no information from my TLS BidiFlow, akka.stream.scaladsl.TLS. I have no idea what step in the handshake I'm stuck at, which makes troubleshooting very difficult.
Is there any way to output some information throughout the TLS handshake process? It seems as though all of the functionality is encapsulated and I don't know if there's any way to troubleshoot.
Otherwise, I'm attempting to troubleshoot with openssl and get the following:
bash$ openssl s_client -connect myserver.com:443 -state -debug
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x7fd914100080 [0x7fd915001000] (318 bytes => 318 (0x13E))
0000 - 16 03 01 01 39 01 00 01-35 03 03 e3 ff 5d fb 26 ....9...5....].&
0010 - 15 e3 32 89 37 e2 cb 95-f5 00 bd df 13 3d ae a6 ..2.7........=..
0020 - d7 37 db 4e 80 19 63 ad-d6 6c f1 00 00 98 cc 14 .7.N..c..l......
0030 - cc 13 cc 15 c0 30 c0 2c-c0 28 c0 24 c0 14 c0 0a .....0.,.(.$....
0040 - 00 a3 00 9f 00 6b 00 6a-00 39 00 38 ff 85 00 c4 .....k.j.9.8....
0050 - 00 c3 00 88 00 87 00 81-c0 32 c0 2e c0 2a c0 26 .........2...*.&
0060 - c0 0f c0 05 00 9d 00 3d-00 35 00 c0 00 84 c0 2f .......=.5...../
0070 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67 .+.'.#.........g
0080 - 00 40 00 33 00 32 00 be-00 bd 00 45 00 44 c0 31 .#.3.2.....E.D.1
0090 - c0 2d c0 29 c0 25 c0 0e-c0 04 00 9c 00 3c 00 2f .-.).%.......<./
00a0 - 00 ba 00 41 c0 11 c0 07-c0 0c c0 02 00 05 00 04 ...A............
00b0 - c0 12 c0 08 00 16 00 13-c0 0d c0 03 00 0a 00 15 ................
00c0 - 00 12 00 09 00 ff 01 00-00 74 00 0b 00 04 03 00 .........t......
00d0 - 01 02 00 0a 00 3a 00 38-00 0e 00 0d 00 19 00 1c .....:.8........
00e0 - 00 0b 00 0c 00 1b 00 18-00 09 00 0a 00 1a 00 16 ................
00f0 - 00 17 00 08 00 06 00 07-00 14 00 15 00 04 00 05 ................
0100 - 00 12 00 13 00 01 00 02-00 03 00 0f 00 10 00 11 ................
0110 - 00 23 00 00 00 0d 00 26-00 24 06 01 06 02 06 03 .#.....&.$......
0120 - ef ef 05 01 05 02 05 03-04 01 04 02 04 03 ee ee ................
0130 - ed ed 03 01 03 02 03 03-02 01 02 02 02 03 ..............
SSL_connect:unknown state
At which point openssl just hangs.
The Akka TLS support uses the built in Java TLS support behind the scenes, so to get debug output for TLS you'll have to enable debugging for that. It can be done through passing a system property to the JVM when starting it like so -Djavax.net.debug=all
Ultimately I found that the ssl-config logging is very sparse and wasn't helpful to resolving my issue. It does provide some debugging information but not much. Much better for debugging the TLS handshake is to use the -Djavax.net.debug=all flag when running the JVM. However, even this provides mixed results. For example, the resulting error I received is that the server couldn't find a matching cipher suite. Eventually I resolved my issue by realizing that when creating the input streams for my keystore/truststore I was specifying my path incorrectly.
Note for anyone coming across this: if you specify your keystore and truststore incorrectly the resulting input streams will be null and SSLContext.init will happily use these and provide an error that is unrelated to the keystore/truststore! This was very difficult to troubleshoot due to the incorrect error handling in SSLContext.
i try to investigate a hung with windbg and want to know if my assumptions are right. If I call the command
!mlocks i got the following:
0:000> !mlocks
Examining SyncBlocks...
Scanning for ReaderWriterLock instances...
Scanning for holders of ReaderWriterLock locks...
Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
ClrThread DbgThread OsThread LockType Lock LockLevel
--------------------------------------------------------------------------
0x640064 -1 0xffffffff RWLock 000000000339a338 Writer
0x6 7 0x1ea8 thinlock 000000000343ddd8 (recursion:0)
When executeing rwlocks i got the following:
0:000> !rwlock 000000000339a338
WriterThread: 0x640064 (DEAD) WriterLevel: 115 WaitingWriterCount: 0
WriterEvent: 0
WaitingWriterThreadIds: None
ReaderCount: 116
CurrentReaderThreadIds:
WaitingReaderCount: 576
ReaderEvent: 80400002
WaitingReaderThreadIds:
*This lock has 116 orphaned reader locks.
0:007> !rwlock
Address ReaderCount WaitingReaderCount WriterThread WaitingWriterCount
...
000000000339a338 116 576 0x640064 0
...
00000000053f0688 568 499 -- 6
...
i got this.
When I call
0:000> !dlk Examining SyncBlocks... Scanning for ReaderWriterLock instances... Scanning for holders of ReaderWriterLock locks... Scanning for ReaderWriterLockSlim instances...
Scanning for holders of ReaderWriterLockSlim locks...
Examining CriticalSections...
Scanning for threads waiting on SyncBlocks...
Scanning for threads waiting on ReaderWriterLock locks...
Scanning for threads waiting on ReaderWriterLocksSlim locks...
Scanning for threads waiting on CriticalSections...
No deadlocks detected.
then no deadlock will be detected.
I found this on Tess's blog
My question is if this threads have to do with my hunging application,
and what the scenario could be.
What does it mean that the thread is DEAD. Can a lock happen on a dead thread? Or do I have to find somewhere else root cause of my hunging application?
Please help me to interpret this output.
Regards,
Bernhard
Here is some addional info:
0:000> dc 000000000339a338
00000000`0339a338 f2073268 000007fe 00000000 80000000 h2..............
00000000`0339a348 f2066960 000007fe 00000010 00650047 `i..........G.e.
00000000`0339a358 00480074 0073006f 00410074 00640064 t.H.o.s.t.A.d.d.
00000000`0339a368 00650072 00730073 00730065 00000000 r.e.s.s.e.s.....
00000000`0339a378 00000000 00000000 00000000 00000000 ................
00000000`0339a388 f2066960 000007fe 0000001c 00650066 `i..........f.e.
00000000`0339a398 00300038 003a003a 00380034 00310038 8.0.:.:.4.8.8.1.
00000000`0339a3a8 0034003a 00660039 003a0063 00300039 :.4.9.f.c.:.9.0.
0:000> db 000000000339a338
00000000`0339a338 68 32 07 f2 fe 07 00 00-00 00 00 00 00 00 00 80 h2..............
00000000`0339a348 60 69 06 f2 fe 07 00 00-10 00 00 00 47 00 65 00 `i..........G.e.
00000000`0339a358 74 00 48 00 6f 00 73 00-74 00 41 00 64 00 64 00 t.H.o.s.t.A.d.d.
00000000`0339a368 72 00 65 00 73 00 73 00-65 00 73 00 00 00 00 00 r.e.s.s.e.s.....
00000000`0339a378 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00000000`0339a388 60 69 06 f2 fe 07 00 00-1c 00 00 00 66 00 65 00 `i..........f.e.
00000000`0339a398 38 00 30 00 3a 00 3a 00-34 00 38 00 38 00 31 00 8.0.:.:.4.8.8.1.
00000000`0339a3a8 3a 00 34 00 39 00 66 00-63 00 3a 00 39 00 30 00 :.4.9.f.c.:.9.0.
0:000> !mdt 000000000339a338
Can't get name for module 000007ff0068c3c0. Error = 0x80070057.
Can't get name for module 000007ff00791908. Error = 0x80070057.
Can't get name for module 000007ff0068c3c0. Error = 0x80070057.
Can't get name for module 000007ff00791908. Error = 0x80070057.
000000000339a338 (System.Threading.ReaderWriterLock)
_hWriterEvent:8000000000000000 (System.IntPtr)
_hReaderEvent:000007fef2066960 (System.IntPtr)
_hObjectHandle:0065004700000010 (System.IntPtr)
_dwState:0x480074 (System.Int32)
_dwULockID:0x73006f (System.Int32)
_dwLLockID:0x410074 (System.Int32)
_dwWriterID:0x640064 (System.Int32)
_dwWriterSeqNum:0x650072 (System.Int32)
_wWriterLevel:0x0073 (System.Int16)
0:000> !mdt 000000000343ddd8
Can't get name for module 000007ff0068c3c0. Error = 0x80070057.
Can't get name for module 000007ff00791908. Error = 0x80070057.
000000000343ddd8 (System.Collections.Generic.LinkedList`1[[TAU.GuiAccess.PopupHandler.ClientInfo, TAU.GuiAccess.PopupHandler]])
head:000000000823e148 (System.Collections.Generic.LinkedListNode`1[[TAU.GuiAccess.PopupHandler.ClientInfo, TAU.GuiAccess.PopupHandler]])
count:0x1 (System.Int32)
version:0x3b (System.Int32)
_syncRoot:NULL (System.Object)
siInfo:NULL (System.Runtime.Serialization.SerializationInfo)
0:000> !do 000000000343ddd8
Name: System.Collections.Generic.LinkedList`1[[TAU.GuiAccess.PopupHandler.ClientInfo, TAU.GuiAccess.PopupHandler]]
MethodTable: 000007ff009ce218
EEClass: 000007ff009dda20
Size: 48(0x30) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007ff009ced28 4000586 8 ...ss.PopupHandler]] 0 instance 000000000823e148 head
000007fef206c848 4000587 20 System.Int32 1 instance 1 count
000007fef206c848 4000588 24 System.Int32 1 instance 59 version
000007fef2065ab8 4000589 10 System.Object 0 instance 0000000000000000 _syncRoot
000007fef208a1b8 400058a 18 ...SerializationInfo 0 instance 0000000000000000 siInfo
ThinLock owner 6 (0000000000000000), Recursive 0
The "DEAD" indication means that there is no corresponding OS thread. When a managed thread is created, a managed thread object is created and assigned a thread ID. On Windows, there is currently always a 1:1 mapping between managed and native threads, so there is also an OS thread ID assigned. When a thread terminates, the association between the managed thread object and the native thread is obviously broken. However, it takes a period of time before the managed thread object gets cleaned up. A thread in this state is listed by sosex as "DEAD" and is listed as XXXX in the !sos.threads output.
In this case, the CLR thread ID (00640064) looks suspicious. It looks curiously like Unicode text 'dd'. There may be corruption around the managed lock address. Look around with the debugger's 'dc' or 'db' commands to see if some text has overwritten the thread ID field.
Out of curiosity, how much data is actually sent when establishing a connection to a port (using Java sockets). Is it the size of a Socket object? SocketConnection object?
Your understanding of TCP network connections seems to conflate them with electrical circuits. (Understandable, given your background.)
From a physical standpoint, there's no such thing as a connection, only data packets. Through the TCP protocol, two devices agree to establish a logical (that is, software) connection. A connection is established by a client first sending data to the remote host (SYN), the server sending data back to the client (SYN-ACK), and the client sending a final acknowledgement (ACK). All of this negotation necessarily consumes bandwidth, and when you terminate a connection, you must negotiate a completely new connection to begin sending data again.
For example, I'll connect from my machine to a local web server, 192.168.1.2:80.
First, my machine sends a TCP SYN. This sends 66 bytes over the wire: (headers deliniated with |)
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00|45 00 .$..L... hf y..E.
0010 00 34 53 98 40 00 80 06 00 00 c0 a8 01 0b c0 a8 .4S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a7 00 00 00 00 80 02 ..6..P.. :.......
0030 20 00 50 c8 00 00 02 04 05 b4 01 03 03 02 01 01 .P..... ........
0040 04 02 ..
The first 14 bytes are the Ethernet frame, specifying that this packet's destination MAC address. This will typically be an upstream router, but in this case, the server happens to be on the same switch, so it's the machine's MAC address, 00:24:8c:a9:4c:b4. The source (my) MAC follows, along with the payload type (IP, 0x0800). The next 20 bytes are the IPv4 headers, followed by 32 bytes of TCP headers.
The server responds with a 62-byte SYN-ACK:
0000 00 1e 68 66 20 79 00 24 8c a9 4c b4 08 00|45 00 ..hf y.$ ..L...E.
0010 00 30 69 b9 40 00 80 06 0d b1 c0 a8 01 02 c0 a8 .0i.#... ........
0020 01 0b|00 50 36 0a d3 ae 9a 73 09 ef 3a a8 70 12 ...P6... .s..:.p.
0030 20 00 f6 9d 00 00 02 04 05 b4 01 01 04 02 ....... ......
Again, 14 bytes of Ethernet headers, 20 bytes of IP headers, and 28 bytes of TCP headers. I send an ACK:
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00|45 00 .$..L... hf y..E.
0010 00 28 53 9a 40 00 80 06 00 00 c0 a8 01 0b c0 a8 .(S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a8 d3 ae 9a 74 50 10 ..6..P.. :....tP.
0030 fa f0 83 78 00 00 ...x..
14 + 20 + 20 = 54 bytes over the wire (this is the smallest possible TCP packet size, by the way – the SYN and SYN-ACK packets were larger because they included options).
This adds up to 182 bytes over the wire to establish a connection; now I can begin sending actual data to the server:
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00 45|00 .$..L... hf y..E.
0010 01 9d 53 9d 40 00 80 06 00 00 c0 a8 01 0b c0 a8 ..S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a8 d3 ae 9a 74 50 18 ..6..P.. :....tP.
0030 fa f0 84 ed 00 00|47 45 54 20 2f 20 48 54 54 50 ......GE T / HTTP
0040 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 66 73 0d 0a /1.1..Ho st: fs..
...
14 Ethernet + 20 IP + 20 TCP + data, in this case HTTP.
So we can see that it costs ~182 bytes to establish a TCP connection, and an additional 162-216 bytes to terminate a TCP connection (depending on whether a 4-way FIN ACK FIN ACK or more common 3-way FIN FIN-ACK ACK termination handshake is used), adding up to nearly 400 bytes to "pulse" a connection by disconnecting and reconnecting.
Compared to the 55 bytes you'd use to send one byte of data over an already established connection, this is obviously wasteful.
What you want to do is establish one connection and then send data as-needed. If you're really bandwidth constrained, you could use UDP (which requires no handshaking at all and has an overhead of only 14 Ethernet + 20 IP + 8 UDP bytes per packet), but then you face the problem of using an unreliable transport, and having to handle lost packets on your own.
The minimum size of a TCP packet is 40 bytes. It takes three exchanged packets to create a connection, two from the client and one from the server, and four more to close it, two in each direction. The last packet in a connect exchange can also contain data, as can the first in the close exchange in each direction, which can amortize it a bit, as can combining the outgoing FIN and ACK as per #josh3736's comment below.
Upgraded Webbit to 0.4.6 to use the new SSL support but immediately realized that all wss:// handshakes are failing silently and I don't have any errors to show for it. Chrome only reports a "success" for a response without a HTTP code or any other headers. I check server logs and it doesn't even register an "open" event.
The catch here is that any ws:// connection works great. So what could be possible problems and how can I get an error out of it? Could it be something wrong with the java keystore and SSL handshake?
Edit
I was able to find an openSSL command for a test handshake. Here's the output:
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
Edit 2
I realized I could debug this further
CONNECTED(0000016C)
SSL_connect:before/connect initialization
write to 0x1f57750 [0x1f6a730] (210 bytes => 210 (0xD2))
0000 - 16 03 01 00 cd 01 00 00-c9 03 01 4f 6b 8d 68 63 ...........Ok.hc
0010 - 99 06 08 30 93 2a 42 88-f8 f1 c4 c5 dc 89 71 0b ...0.*B.......q.
0020 - b6 04 42 4e 11 79 b4 76-6c f7 66 00 00 5c c0 14 ..BN.y.vl.f..\..
0030 - c0 0a 00 39 00 38 00 88-00 87 c0 0f c0 05 00 35 ...9.8.........5
0040 - 00 84 c0 12 c0 08 00 16-00 13 c0 0d c0 03 00 0a ................
0050 - c0 13 c0 09 00 33 00 32-00 9a 00 99 00 45 00 44 .....3.2.....E.D
0060 - c0 0e c0 04 00 2f 00 96-00 41 00 07 c0 11 c0 07 ...../...A......
0070 - c0 0c c0 02 00 05 00 04-00 15 00 12 00 09 00 14 ................
0080 - 00 11 00 08 00 06 00 03-00 ff 01 00 00 44 00 0b .............D..
0090 - 00 04 03 00 01 02 00 0a-00 34 00 32 00 01 00 02 .........4.2....
00a0 - 00 03 00 04 00 05 00 06-00 07 00 08 00 09 00 0a ................
00b0 - 00 0b 00 0c 00 0d 00 0e-00 0f 00 10 00 11 00 12 ................
00c0 - 00 13 00 14 00 15 00 16-00 17 00 18 00 19 00 23 ...............#
00d2 - <SPACES/NULS>
SSL_connect:SSLv2/v3 write client hello A
read from 0x1f57750 [0x1f6fc90] (7 bytes => 0 (0x0))
12488:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:.\ssl\s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 210 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
Edit 3
Ok I've nailed the problem to Webbit initialization, but it doesn't throw any errors so I could use some input to getting getResourceAsStream functioning properly. Here's how the server is initialized:
def startWebSocketServer(webSocketHandler:PartialFunction[WebSocketEvent, Unit]) {
val webServer = WebServers.createWebServer(port)
try {
webServer.setupSsl(getClass.getResourceAsStream("/keystore"), "webbit")
webServer.add("/", new WebSocketEventAdapter(webSocketHandler))
webServer.start
} catch {
case e => e.printStackTrace()
}
}
Unfortunately setupSsl won't output any information, and I've tried both what I thought would be the path and inserting a fake path. In either case, I can't get an error. How on earth would I properly locate the path? Thanks!
The OMFG Answer
In a hysterical twist of fate, I found the problem. This particular issue took up 48 hours of my time, but the cause was not even code related and a funny miscommunication.
So as it turns out, another developer had copied our websocket code into a new file he was working on for development. All this time we were trying to debug code in a file that wasn't even executing at run-time. So upon further investigation we scrolled to the bottom of a very long and different file, and found the webbit init code and excuted it perfectly.
Moral of the story: don't commit an incomplete file to the master branch and point everyone there for debugging ;)
ACR122U Get Data
Get UID
< FF CA 00 00 00
ACR122U APDU Command
90 00
UID: 13 29 DF 97
Get ATS
< FF CA 01 00 00
6A 81
Function not supported