Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 1 year ago.
Improve this question
As TCP contains a checksum, and the TCP/IP stack will detect broken packets, is it redundant to add an extra checksum or CRC in a TCP payload (I mean the user data which could be read out by the socket api on the remote end), to make it possible for the receiver verify whether the data is same with the sent one?
Some thought about this question:
It's very common seen that there is a SHA256 value to verify the consistency when downloading files from internet.
The checksum contained in TCP packets already could detect broken packets in most cases.
The Modbus protocol for TCP dropped the CRC, which is used by the Modbus protocol for serial because there is already a checksum in TCP packets.
So, I am really confused now. Could somebody shed some light on this question?
I have googled, it's really not a new question, but the answer is still not clear, there are two opposite voices about this question.
For details, see these:
CRC checking done automatically on Tcp/Ip?
Is it possible for packets sent using TCP to ever arrive with different data?
ADDED:
The two questions aforementioned have been there for more than ten years! But there are still two opposite voices.
TCP has a checksum, which provides only some protection, simply because the checksum is only 16bit. For how much robustness the TCP checksum actually provides, or how much it is lacking, see Can a TCP checksum fail to detect an error? If yes, how is this dealt with?.
If you need more protection, you need to have additional and longer protection, because the more bits are used in protection, the better protection can be provided. If you need protection against active tampering with the traffic (ie not just accidental errors), you also need a cryptographic protection, such as offered by TLS.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I always hear about the layers of internet and i vaguely understand this. But, what confuse me most is that transport layer (including TCP protocol) lies on top of the internet layer(including IP protocol)..
What does this mean? For one who has a foggy understanding of the internet mechanism (I'm not a CS student or something I am just a hobby programmer)
The picture I have about the internet is that the network card sends/receives signals (packets) from the internet through wired connection / wifi then the OS using socket API sends/receives these packets acting as a layer between the hardware and the application which in turn uses some high-level protocol such as HTTP to interpret the data transferred - these protocol usually provided by languages e.g. python or java
.. I guess then that IP and TCP protocol are used at the level of the socket API? but I need more details ? I hope the explanation be in terms of coding/programming/implementation because abstractions used in this area confuse me.
Thank you , and sorry for my bad English
This is part of a layered solution to solve networking. Each layer has its own functionality:
IP (Internet Protocol) is in charge of delivering a packet (or datagram) from one interface, in one machine, with an IP address assigned to another interface in the same or other machine (node). Both nodes can be in the same LAN or different LAN connected through different paths (LAN's and routers). Basically it will make the packet get from source IP to destination IP. It provides a best-effort services, it doesn't assure the IP packet is going to arrive, it can be lost in the middle.
Above layer 3 or IP in the so-called TCP/IP stack, there is the transport layer. Its main functionality is to multiplex the lower layer (IP) service (take a packet from src to dst) among different applications. This is why in all transport layer protocols there is the concept of port or more generically Transport Service Access Point (TSAP). UDP, TCP, SCTP do that. UDP provides an unreliable service to the application. TCP provides a connected, reliable transport service to the application. This layer will make a message sent from application A in node Y reach application A1 in node Z, either reliably or unreliably (while IP only takes care of carrying the packet from node Y to node Z).
You will need to read a little about the OSI layered model and the TCP/IP layered model.
If you need to get more info I can address you to a training I have about IPv6 with a good introduction to networking: http://www.slideshare.net/rodolk/networking-tcpip-stack-introduction-ipv6
TCP is a protocol, known as "Transmission Control Protocol" - by specification it has features in place which makes sure that transmitted data is checked. On the other hand, there are things such as UDP, aka "User Datagram Protocol" which also works on top of IP - by specification this method does not check any transmitted data, so it's less useful where files must be fully intact (more utilised for video streaming, where some lost frames is acceptable, as opposed to binary file transfers where incorrect data means corruption and the whole file would be useless).
On to IP, IP is an addressing protocol, allowing a network to address and communicate with any machine that lives within it. IP stands for Internet Protocol, and it defines the fundamental way that two machines communicate over the "internet". It does not define how communications are handled, in ways such as being checked for data integrity, etc.
So, to summarise, the TCP and UDP are just extensions of IP. It is entirely possible, however, to have a socket based TCP or UDP connection, and I expect it's also possible to have some sort of MAC address protocol (as opposed to an IP address protocol). I don't know of any protocols which are similar to IP, but I imagine they do exist. In reality, using TCP over something other than IP is entirely unlikely. If you're going to the effort to create a custom protocol, chances are you'll want it fully custom and won't want to stick to design specifications designed for another protocol layer.
Note that calling it a "TCP/IP" connection is probably only ever used for legacy reasons. A lot of terms like this still exist because before the technology "bubble" growth, there were competing alternatives to IP. Even today, there is IPv6 which is technically an alternative to IPv4. It's also possible that we might one day outgrow IPv6, and at that point in time, there could be something other than IP to worry about.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I have some problem.
I have client and server. Client connect with server over TCP.
Then, client send some data (separated by chunks), I don't know what is the length of data (TLS handshake). But I know that client send some data with fixed length, and then stop, until not received some response, then he send data with fixed length again.
I need read all chunks, until client stopped send (because so many chunks). How to do that ?
I have only one idea, it's timeout. Read data in loop and set timeout between iterate. If timeout is ended, then data complete collected.
Perhaps there is a more elegant solution?
Based on the information in your comments, you're doing this wrong. The correct way to write an HTTPS proxy is to read the CONNECT line, make the upstream connection, send the appropriate response back o the client, and then if successful start copying bytes in both directions simultaneously. You're not in the least concerned with packets or read sizes, and you should certainly not make any attempt to 'collect' packets before retransmission, as that will just add latency to the system.
You can accomplish this either by starting two threads per connection, one in each direction, or via non-blocking sockets and select()/poll()/epoll(), or whatever that looks like in Go.
BUT I have no idea why you're doing this at all. There are plenty of open-source HTTP proxies already in existence, and as you're dealing with HTTPS there is no value you can possibly add to them. Your claim about 'business logic' is meaningless, or at least unimplementable.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I am planning on using TCP/IP and UDP protocols together for my game server. I have alread setup the server, in java, and the client, in c++, and had them both connected with a nice gui using the TCP/IP protocol. I need their to be UDP interactions since the UDP protocol allows the loss of packets which is essential for an online game that has position and other updates being transferred that wouldn't be beneficial to resend if an error occurs. My problem occurred when I tried converting my client and server to the UDP protocol. In java I have noticed that it will be harder to create a UDP connection because it isn't as easy as binding to a port and awaiting connections IMHO so:
Is it possible via the UDP protocol, or any other protocol, to have my server "broadcast" data, such as entity location, and allow the client to receive the data without having to pragmatically establish a permanent connection to that server.
If the above is a no, is it possible to use the UDP and TCP/IP protocols together without having to establish two separate connection.
Is it possible via the UDP protocol, or any other protocol, to have my server "broadcast" data, such as entity location, and allow the client to receive the data without having to pragmatically establish a permanent connection to that server.
That's all you can do, because UDP has no notion of a permanent connection. All it supports is datagrams.
If the above is a no, is it possible to use the UDP and TCP/IP protocols together without having to establish two separate connection.
UDP is connectionless, there wouldn't be a "separate connection". Probably the best way to think about is that the connection is the TCP thing, but data can also be sent using UDP while the connection exists.
Since UDP doesn't always work over the Internet, I would strongly suggest separating the transport layer from other program logic. Your transport layer can make the TCP connection and, using the TCP connection, agree to try UDP. If UDP works, your layer can route data that's best sent over UDP over the UDP connection. If UDP fails, you can include that data in the TCP connection. You'll need to design your own encapsulation/messaging protocol to be used over the TCP connection, including the ability to negotiate and test a "side" UDP connection.
You should keep in mind that TCP does a lot more than just retransmissions, and if you need many of the other things, you're probably better off using TCP than UDP. This is especially true because the TCP layer is developed by network experts and built into the operating system and it's unlikely you can do a better job, even if you don't have to worry about retransmissions. Some of those things are:
Session setup, tear-down, and tracking
Last ACK handling and Byzantine failure
Slow start
Checksumming and verification
Exponential backoff
Reordering
Duplicate detection
Path MTU detection
Path bandwidth detection
Acknowledgements with piggybacking
Dead connection detection
"Short" packet avoidance
And more.
UDP is connectionless, it works by just sending datagrams to the receiver and hope that it will reach the destination (no confirmation or anything like TCP)
Take a look at this example:
Java UDP server and client
And maybe read up on the documentation:
Writing a datagram client and server
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I am learning about sockets and client-server applications. It seems as if the recommended approach is that the server should have a fixed IP address so that the client can find it. Is this correct? It seems as if there should be some type of startup technique where on initialization a program on the server (or the client) could generally broadcast their IP addresses to the other computers on the network, so that in case the server IP address changes the clients can still find it. Is this possible? I believe that "multicast" may be helpful, but it seems as if that is not always supported and/or the routers/switches must be configured to accept this or the technique would fail. Is this correct? I see that there are ways to perhaps use UDP to broadcast to any computer on the network, but it's not clear to me how the recipient computers can access this in a clear manner without a socket already set up - wouldn't there be confusion if other computers are already broadcasting on the same port? Is it just a protocol issue for the program to sort out to recognize the received packets? Do I just need to learn more about sockets? Any and all suggestions/pointers for where I could look would be greatly appreciated. Thanks so much in advance!
I am learning about sockets and client-server applications. It seems
as if the recommended approach is that the server should have a fixed
IP address so that the client can find it. Is this correct?
Yes, it is.
It seems as if there should be some type of startup technique where on
initialization a program on the server (or the client) could generally
broadcast their IP addresses to the other computers on the network, so
that in case the server IP address changes the clients can still find
it. Is this possible?
Yes, it is.
I believe that "multicast" may be helpful, but it seems as if that is
not always supported and/or the routers/switches must be configured to
accept this or the technique would fail. Is this correct?
No, multicast is not really helpful here, you mentioned why, and you should know multicast address in advance.
I see that there are ways to perhaps use UDP to broadcast to any
computer on the network, but it's not clear to me how the recipient
computers can access this in a clear manner without a socket already
set up - wouldn't there be confusion if other computers are already
broadcasting on the same port?
You just advertise your clients once in a while, for example every 5 secs or every 1 min, whatever interval you find ok. If someone sends on this port too - then you should use some unique structure, magic numbers and so on to distinguish one packets from another.
Is it just a protocol issue for the program to sort out to recognize
the received packets?
Yes, it is. On UDP.
Do I just need to learn more about sockets?
Absolutly yes. I suggest this book: Unix Network programming, it is good, even if you running non-UNIX environment.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 years ago.
Improve this question
I am trying to make a realtime notification just like facebook.After learning and searching alot i m very confuse please explain me what is right and what is wrong..
Please make sure that the site may would have same number of users as Facebook
We can make Realtime notification with long polling or not? IF yes what is the advantages, disadvantages and limitations.
we can make Realtime notifiaction with websockets or not?Please mind the number of users can be same as facebook .If yes what is the advantages, disadvantages and limitations.
If there is any other method please explain.
Confusion
How Far I learn in web and found that Websocket is good but there is a limitation(Max 5K) in number of open connection which means that at a time the max number of user is just 5K,this is very less than facebook's number of users.. if I m wrong please explain.
You're wrong, a websocket based solution is not limited to 5K concurrent connections.
According to the Facebook Newsroom they have about 727 million daily active users on average in September 2013 or about 504k unique users that hit the Facebook page every minute. Given an average visit time of 18 minutes (researched by statisticbrain.com) their notification infrastructure must be able to serve about 9 million (18*504k) concurrent TCP connections 24/7. Although this number is a very far approximation it gives a far idea of what they are dealing with and what you have to deal with if you are going to build such a system.
You can use long polling as well as websockets to build your realtime notification system. In both cases you face similar problems which are related to your OS (Explanations are for a Unix based system):
limitation of ports, one tcp listener socket can only accept 2^16 connections on the same IP/Port it is listening, so you'll need to listen on multiple ports and/or multiple IP adresses.
memory, every open connection uses at least one file descriptor
Read more about the limitations in What is the theoretical maximum number of open TCP connections that a modern Linux box can have
Long-polling vs. Websockets:
Every poll in your long-poll solution requires a new HTTP request, which requires more bandwidth than what is needed to keep a websocket connection alive. Moreover the notification is returned as a HTTP response resulting in a new poll request. Although the websocket solution can be more efficient in terms of bandwidth and consumption of system resources, it has a major drawback: lack of browser support.
Depending on the stats at hand, a websocket-only solution ignores about 20-40% of your visitors (stats from statscounter.com). For this reason different server libraries were developed that abstract the concept of a connection away from the 'physical' underlying transport model. As a result more modern browsers create the connection using websockets and older browsers fall back to an alternative transport such as e.g. HTTP long polling, jsonp polling, or flash. Prominent examples of such libraries are Sock.js and Socket.io.