I met an error when running codes at the bottom. It's like a simple ftp.
I use python2.6.6 and CentOS release 6.8
In most linux server, it gets right results like this:(I'm very sorry that I have just sign up and couldn't )
Clinet:
[root#Test ftp]# python client.py
path:put|/home/aaa.txt
Server:
[root#Test ftp]# python server.py
connected...
pre_data:put|aaa.txt|4
cmd: put
file_name: aaa.txt
file_size: 4
upload successed.
But I get errors in some server(such as my own VM in my PC). I have done lots of tests(python2.6/python2.7, Centos6.5/Centos6.7) and found this error is not because them. Here is the error imformation:
[root#Lewis-VM ftp]# python server.py
connected...
pre_data:put|aaa.txt|7sdfsdf ###Here gets the wrong result, "sdfsdf" is the content of /home/aaa.txt and it shouldn't be sent here to 'file_size' and so it cause the "ValueError" below
cmd: put
file_name: aaa.txt
file_size: 7sdfsdf
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 10699)
Traceback (most recent call last):
File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__
self.handle()
File "server.py", line 30, in handle
if int(file_size)>recv_size:
ValueError: invalid literal for int() with base 10: '7sdfsdf\n'
What's more, I found that if I insert a time.sleep(1) between sk.send(cmd+"|"+file_name+'|'+str(file_size)) and sk.send(data) in client.py, the error will disappear. I have said that I did tests in different system and python versions and the error is not because them. So I guess that is it because of some system configs? I have check about socket.send() and socket.recv() in python.org but fount nothing helpful. So could somebody help me to explain why this happend?
The code are here:
#!/usr/bin/env python
#coding:utf-8
################
#This is server#
################
import SocketServer
import os
class MyServer(SocketServer.BaseRequestHandler):
def handle(self):
base_path = '/home/ftp/file'
conn = self.request
print 'connected...'
while True:
#####receive pre_data: we should get data like 'put|/home/aaa|7'
pre_data = conn.recv(1024)
print 'pre_data:' + pre_data
cmd,file_name,file_size = pre_data.split('|')
print 'cmd: ' + cmd
print 'file_name: '+ file_name
print 'file_size: '+ file_size
recv_size = 0
file_dir = os.path.join(base_path,file_name)
f = file(file_dir,'wb')
Flag = True
####receive 1024bytes each time
while Flag:
if int(file_size)>recv_size:
data = conn.recv(1024)
recv_size+=len(data)
else:
recv_size = 0
Flag = False
continue
f.write(data)
print 'upload successed.'
f.close()
instance = SocketServer.ThreadingTCPServer(('127.0.0.1',9999),MyServer)
instance.serve_forever()
#!/usr/bin/env python
#coding:utf-8
################
#This is client#
################
import socket
import sys
import os
ip_port = ('127.0.0.1',9999)
sk = socket.socket()
sk.connect(ip_port)
while True:
input = raw_input('path:')
#####we should input like 'put|/home/aaa.txt'
cmd,path = input.split('|')
file_name = os.path.basename(path)
file_size=os.stat(path).st_size
sk.send(cmd+"|"+file_name+'|'+str(file_size))
send_size = 0
f= file(path,'rb')
Flag = True
#####read 1024 bytes and send it to server each time
while Flag:
if send_size + 1024 >file_size:
data = f.read(file_size-send_size)
Flag = False
else:
data = f.read(1024)
send_size+=1024
sk.send(data)
f.close()
sk.close()
The TCP is a stream of data. That is the problem. TCP do not need to keep message boundaries. So when a client calls something like
connection.send("0123456789")
connection.send("ABCDEFGHIJ")
then a naive server like
while True;
data = conn.recv(1024)
print data + "_"
may print any of:
0123456789_ABCDEFGHIJ_
0123456789ABCDEFGHIJ_
0_1_2_3_4_5_6_7_8_9_A_B_C_D_E_F_G_H_I_J_
The server has no chance to recognize how many sends client called because the TCP stack at client side just inserted data to a stream and the server must be able to process the data received in different number of buffers than the client used.
Your server must contain a logic to separate the header and the data. All of application protocols based on TCP use a mechanism to identify application level boundaries. For example HTTP separates headers and body by an empty line and it informs about the body length in a separate header.
Your program works correctly when server receives a header with the command, name and size in a separate buffer it it fails when client is fast enough and push the data into stream quickly and the server reads header and data in one chunk.
Related
I need to set timeout on python's socket recv method. How to do it?
The typical approach is to use select() to wait until data is available or until the timeout occurs. Only call recv() when data is actually available. To be safe, we also set the socket to non-blocking mode to guarantee that recv() will never block indefinitely. select() can also be used to wait on more than one socket at a time.
import select
mysocket.setblocking(0)
ready = select.select([mysocket], [], [], timeout_in_seconds)
if ready[0]:
data = mysocket.recv(4096)
If you have a lot of open file descriptors, poll() is a more efficient alternative to select().
Another option is to set a timeout for all operations on the socket using socket.settimeout(), but I see that you've explicitly rejected that solution in another answer.
there's socket.settimeout()
As mentioned both select.select() and socket.settimeout() will work.
Note you might need to call settimeout twice for your needs, e.g.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(("",0))
sock.listen(1)
# accept can throw socket.timeout
sock.settimeout(5.0)
conn, addr = sock.accept()
# recv can throw socket.timeout
conn.settimeout(5.0)
conn.recv(1024)
You could set timeout before receiving the response and after having received the response set it back to None:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(5.0)
data = sock.recv(1024)
sock.settimeout(None)
The timeout that you are looking for is the connection socket's timeout not the primary socket's, if you implement the server side. In other words, there is another timeout for the connection socket object, which is the output of socket.accept() method. Therefore:
sock.listen(1)
connection, client_address = sock.accept()
connection.settimeout(5) # This is the one that affects recv() method.
connection.gettimeout() # This should result 5
sock.gettimeout() # This outputs None when not set previously, if I remember correctly.
If you implement the client side, it would be simple.
sock.connect(server_address)
sock.settimeout(3)
Got a bit confused from the top answers so I've wrote a small gist with examples for better understanding.
Option #1 - socket.settimeout()
Will raise an exception in case the sock.recv() waits for more than the defined timeout.
import socket
sock = socket.create_connection(('neverssl.com', 80))
timeout_seconds = 2
sock.settimeout(timeout_seconds)
sock.send(b'GET / HTTP/1.1\r\nHost: neverssl.com\r\n\r\n')
data = sock.recv(4096)
data = sock.recv(4096) # <- will raise a socket.timeout exception here
Option #2 - select.select()
Waits until data is sent until the timeout is reached. I've tweaked Daniel's answer so it will raise an exception
import select
import socket
def recv_timeout(sock, bytes_to_read, timeout_seconds):
sock.setblocking(0)
ready = select.select([sock], [], [], timeout_seconds)
if ready[0]:
return sock.recv(bytes_to_read)
raise socket.timeout()
sock = socket.create_connection(('neverssl.com', 80))
timeout_seconds = 2
sock.send(b'GET / HTTP/1.1\r\nHost: neverssl.com\r\n\r\n')
data = recv_timeout(sock, 4096, timeout_seconds)
data = recv_timeout(sock, 4096, timeout_seconds) # <- will raise a socket.timeout exception here
You can use socket.settimeout() which accepts a integer argument representing number of seconds. For example, socket.settimeout(1) will set the timeout to 1 second
try this it uses the underlying C.
timeval = struct.pack('ll', 2, 100)
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, timeval)
As mentioned in previous replies, you can use something like: .settimeout()
For example:
import socket
s = socket.socket()
s.settimeout(1) # Sets the socket to timeout after 1 second of no activity
host, port = "somehost", 4444
s.connect((host, port))
s.send("Hello World!\r\n")
try:
rec = s.recv(100) # try to receive 100 bytes
except socket.timeout: # fail after 1 second of no activity
print("Didn't receive data! [Timeout]")
finally:
s.close()
I hope this helps!!
#! /usr/bin/python3.6
# -*- coding: utf-8 -*-
import socket
import time
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.settimeout(5)
PORT = 10801
s.bind(('', PORT))
print('Listening for broadcast at ', s.getsockname())
BUFFER_SIZE = 4096
while True:
try:
data, address = s.recvfrom(BUFFER_SIZE)
except socket.timeout:
print("Didn't receive data! [Timeout 5s]")
continue
Shout out to: https://boltons.readthedocs.io/en/latest/socketutils.html
It provides a buffered socket, this provides a lot of very useful functionality such as:
.recv_until() #recv until occurrence of bytes
.recv_closed() #recv until close
.peek() #peek at buffer but don't pop values
.settimeout() #configure timeout (including recv timeout)
i've written the following script on PyCharm IDE:
import socket
import time
sock = socket(socket.AF_INET, socket.SOCK_DGRAM)
hexpacket "00 FF"
ip = raw_input('Target IP: ')
port = input('Port: ')
duration = ('Number of seconds to send packets: ')
timeout = time.time() + duration
sent = 0
while True:
if time.time() > timeout:
break
else:
pass
sock.sendto(hexpacket,(ip,port))
sent = sent +1
print "Send %s packet to %s through ports %s"(send, ip, port)
I get an output from the console of:
TypeError: 'module' object is not callable
I have tried to change the "import socket" statement to either "from socket import socket" and "from socket import *" but both did not help.
I also tried to use "pip install socket" but I get "could not find any downloads that satisfy the requirement socket".
Any idea how do I solve this simple issue? I thought socket is a basic module that comes with every python download.
Thanks for the answers..!
sock = socket(socket.AF_INET, socket.SOCK_DGRAM)
you are using socket object directly it's wrong it's throwing an error
TypeError: 'module' object is not callable
Try this::
client_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
I 've tried to use tornado.platform.twisted to integrate txyam memcached client, but when I try to check it for functioning, next error is thrown:
Traceback (most recent call last):
File "swcomet/tx_memcache_helper.py", line 32, in <module>
mem_helper = MemcacheHelper()
File "swcomet/tx_memcache_helper.py", line 19, in __init__
self.add(4)
File "/home/rustem/work/sw.services.swcomet.python/venv/local/lib/python2.7/site-packages/tornado/gen.py", line 117, in wrapper
gen = func(*args, **kwargs)
File "swcomet/tx_memcache_helper.py", line 25, in add
self.mem.getPickled(user_id, decompress=True)
File "/home/rustem/work/sw.services.swcomet.python/venv/lib/python2.7/site-packages/txyam/client.py", line 133, in getPickled
return self.get(key, **kwargs).addCallback(handleResult, uncompress)
File "/home/rustem/work/sw.services.swcomet.python/venv/lib/python2.7/site-packages/txyam/client.py", line 27, in wrapper
func = getattr(self.getClient(key), cmd)
File "/home/rustem/work/sw.services.swcomet.python/venv/lib/python2.7/site-packages/txyam/client.py", line 48, in getClient
raise NoServerError, "No connected servers remaining."
txyam.client.NoServerError: No connected servers remaining.
The source code which dumps that error:
import tornado.ioloop
import tornado.gen
from txyam.client import YamClient
from swtools.date import _ts
import tornado.platform.twisted
MEMHOSTS = ['127.0.0.1111']
USER_EXPIRATION_TIME = 61
class MemcacheHelper(object):
def __init__(self, *a, **kw):
try:
self.mem = YamClient(["127.0.0.1"])
except Exception, e:
print "ERror", e
self.clients = set()
self.add(4)
#tornado.gen.engine
def add(self, user_id, expire=None):
self.clients.add(user_id)
expire = expire or USER_EXPIRATION_TIME
self.mem.getPickled(user_id, decompress=True)
print "hmmm"
if __name__ == '__main__':
print "trying to start on top of IOLOOP"
ioloop = tornado.ioloop.IOLoop.instance()
#reactor = TornadoReactor(ioloop)
mem_helper = MemcacheHelper()
#mem_helper.add(4)
ioloop.start()
Please, help me to solve this problem!
txyam appears not to let you perform any memcache operations until after at least one connection has been established:
def getActiveConnections(self):
return [factory.client for factory in self.factories if not factory.client is None]
def getClient(self, key):
hosts = self.getActiveConnections()
log.msg("Using %i active hosts" % len(hosts))
if len(hosts) == 0:
raise NoServerError, "No connected servers remaining."
return hosts[ketama(key) % len(hosts)]
It attempts to set up these connections right away:
def __init__(self, hosts):
"""
#param hosts: A C{list} of C{tuple}s containing hosts and ports.
"""
self.connect(hosts)
But connection setup is asynchronous, and it doesn't expose an event to indicate when at least one connection has been established.
So your code fails because you call add right away, before any connections exist. A good long-term fix would be to file a bug report against txyam, because this isn't a very nice interface. YamClient could have a whenReady method that returns a Deferred that fires when you are actually allowed to use the YamClient instance. Or there could be an alternate constructor that returns a Deferred that fires with the YamClient instance, but only after it can be used.
So I am working on an iPhone app that requires a socket to handle multiple clients for online gaming. I have tried Twisted, and with much effort, I have failed to get a bunch of info to be sent at once, which is why I am now going to attempt socket.
My question is, using the code below, how would you be able to have multiple clients connected? I've tried lists, but I just can't figure out the format for that. How can this be accomplished where multiple clients are connected at once and I am able to send a message to a specific client?
Thank you!
#!/usr/bin/python # This is server.py file
import socket # Import socket module
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 50000 # Reserve a port for your service.
print 'Server started!'
print 'Waiting for clients...'
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
c, addr = s.accept() # Establish connection with client.
print 'Got connection from', addr
while True:
msg = c.recv(1024)
print addr, ' >> ', msg
msg = raw_input('SERVER >> ')
c.send(msg);
#c.close() # Close the connection
Based on your question:
My question is, using the code below, how would you be able to have multiple clients connected? I've tried lists, but I just can't figure out the format for that. How can this be accomplished where multiple clients are connected at once and I am able to send a message to a specific client?
Using the code you gave, you can do this:
#!/usr/bin/python # This is server.py file
import socket # Import socket module
import thread
def on_new_client(clientsocket,addr):
while True:
msg = clientsocket.recv(1024)
#do some checks and if msg == someWeirdSignal: break:
print addr, ' >> ', msg
msg = raw_input('SERVER >> ')
#Maybe some code to compute the last digit of PI, play game or anything else can go here and when you are done.
clientsocket.send(msg)
clientsocket.close()
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 50000 # Reserve a port for your service.
print 'Server started!'
print 'Waiting for clients...'
s.bind((host, port)) # Bind to the port
s.listen(5) # Now wait for client connection.
print 'Got connection from', addr
while True:
c, addr = s.accept() # Establish connection with client.
thread.start_new_thread(on_new_client,(c,addr))
#Note it's (addr,) not (addr) because second parameter is a tuple
#Edit: (c,addr)
#that's how you pass arguments to functions when creating new threads using thread module.
s.close()
As Eli Bendersky mentioned, you can use processes instead of threads, you can also check python threading module or other async sockets framework. Note: checks are left for you to implement how you want and this is just a basic framework.
accept can continuously provide new client connections. However, note that it, and other socket calls are usually blocking. Therefore you have a few options at this point:
Open new threads to handle clients, while the main thread goes back to accepting new clients
As above but with processes, instead of threads
Use asynchronous socket frameworks like Twisted, or a plethora of others
Here is the example from the SocketServer documentation which would make an excellent starting point
import SocketServer
class MyTCPHandler(SocketServer.BaseRequestHandler):
"""
The RequestHandler class for our server.
It is instantiated once per connection to the server, and must
override the handle() method to implement communication to the
client.
"""
def handle(self):
# self.request is the TCP socket connected to the client
self.data = self.request.recv(1024).strip()
print "{} wrote:".format(self.client_address[0])
print self.data
# just send back the same data, but upper-cased
self.request.sendall(self.data.upper())
if __name__ == "__main__":
HOST, PORT = "localhost", 9999
# Create the server, binding to localhost on port 9999
server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
# Activate the server; this will keep running until you
# interrupt the program with Ctrl-C
server.serve_forever()
Try it from a terminal like this
$ telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Hello
HELLOConnection closed by foreign host.
$ telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Sausage
SAUSAGEConnection closed by foreign host.
You'll probably need to use A Forking or Threading Mixin too
This program will open 26 sockets where you would be able to connect a lot of TCP clients to it.
#!usr/bin/python
from thread import *
import socket
import sys
def clientthread(conn):
buffer=""
while True:
data = conn.recv(8192)
buffer+=data
print buffer
#conn.sendall(reply)
conn.close()
def main():
try:
host = '192.168.1.3'
port = 6666
tot_socket = 26
list_sock = []
for i in range(tot_socket):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind((host, port+i))
s.listen(10)
list_sock.append(s)
print "[*] Server listening on %s %d" %(host, (port+i))
while 1:
for j in range(len(list_sock)):
conn, addr = list_sock[j].accept()
print '[*] Connected with ' + addr[0] + ':' + str(addr[1])
start_new_thread(clientthread ,(conn,))
s.close()
except KeyboardInterrupt as msg:
sys.exit(0)
if __name__ == "__main__":
main()
def get_clients():
first_run = True
startMainMenu = False
while True:
if first_run:
global done
done = False
Thread(target=animate, args=("Waiting For Connection",)).start()
Client, address = objSocket.accept()
global menuIsOn
if menuIsOn:
menuIsOn = False # will stop main menu
startMainMenu = True
done = True
# Get Current Directory in Client Machine
current_client_directory = Client.recv(1024).decode("utf-8", errors="ignore")
# beep on connection
beep()
print(f"{bcolors.OKBLUE}\n***** Incoming Connection *****{bcolors.OKGREEN}")
print('* Connected to: ' + address[0] + ':' + str(address[1]))
try:
get_client_info(Client, first_run)
except Exception as e:
print("Error data received is not a json!")
print(e)
now = datetime.now()
current_time = now.strftime("%D %H:%M:%S")
print("* Current Time =", current_time)
print("* Current Folder in Client: " + current_client_directory + bcolors.WARNING)
connections.append(Client)
addresses.append(address)
if first_run:
Thread(target=threaded_main_menu, daemon=True).start()
first_run = False
else:
print(f"{bcolors.OKBLUE}* Hit Enter To Continue.{bcolors.WARNING}\n#>", end="")
if startMainMenu == True:
Thread(target=threaded_main_menu, daemon=True).start()
startMainMenu = False
#!/usr/bin/python
import sys
import os
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
port = 50000
try:
s.bind((socket.gethostname() , port))
except socket.error as msg:
print(str(msg))
s.listen(10)
conn, addr = s.accept()
print 'Got connection from'+addr[0]+':'+str(addr[1]))
while 1:
msg = s.recv(1024)
print +addr[0]+, ' >> ', msg
msg = raw_input('SERVER >>'),host
s.send(msg)
s.close()
I am trying to send the contents of a file from the server to the client , I am reading the file line by line using fgets and writing to the socket descriptor line by line , on the client side , i am in an a while loop , reading the sent contents.I am not being able to terminate the server sending sequence , i.e the client keeps reading the buffer and the next program line is not executed , I think thers something wrong with my way of sending or recieving . here is the code :
server :
filefd = fopen("clients.txt","a+");
while(fgets(filcont,300,filefd) != NULL)
{// write whole file contents to client
n=write(newsockfd,filcont,strlen(filcont));
if(n==0) break;
memset(filcont,'\0',300);
}
fclose(filefd);
client side :
while(n>0){
n = read(sockfd,buffer,sizeof(buffer)-1);
if(n==0) break;
printf("%s\nbytes read :%d \n",buffer,n);
memset(buffer,'\0',256);
}
printf("Enter peer name ( except yours ) to send connection request : \n");
the above line ( printf , peer name doesnot get executed until i terminate the server)
I was able to figure it out , I sent the file contents from the server using fread instead of fgets ( line by line ) and used a single read() at the client . this was the quick fix.
But I also figured out another technique when in case you have to compulsorily use fgets , where the while loop at the client side makes the socket nonblocking for read and then blocking again , the code is pasted below.
flags = fcntl(sockfd, F_GETFL, 0);
fcntl(sockfd, F_SETFL, flags | O_NONBLOCK);
while(n>0){
n = read(sockfd,buffer,sizeof(buffer)-1);
if(n==0) break;
if(n==-1) printf("\nNon blocking read failed congrats");
printf("%s\n",buffer);
memset(buffer,'\0',256);
}
printf("\nbytes read :%d \n",n);
val = fcntl(sockfd, F_GETFL, 0);
flags = O_NONBLOCK;
val &= ~flags; // makes it blocking again
fcntl(sockfd,F_SETFL,val);
The code from stackoverflow was refered to make the socket blocking