I am getting a 500 Internal Server Error on a certain page of my site after a few hours of being up. I restart uWSGI instance with uwsgi --ini /home/metheuser/webapps/ers_portal/ers_portal_uwsgi.ini and it works again for a few hours.
The rest of the site seems to be working. When I navigate to my_table, I am directed to the login page. But, I get the 500 error on my table page on login. I followed the instructions here to set up my nginx and uwsgi configs.
That is, I have ers_portal_nginx.conf located i my app folder that is symlinked to /etc/nginx/conf.d/. I start my uWSGI "instance" (not sure what exactly to call it) in a Screen instance as mentioned above, with the .ini file located in my app folder
My ers_portal_nginx.conf:
server {
listen 80;
server_name www.mydomain.com;
location / { try_files $uri #app; }
location #app {
include uwsgi_params;
uwsgi_pass unix:/home/metheuser/webapps/ers_portal/run_web_uwsgi.sock;
}
}
My ers_portal_uwsgi.ini:
[uwsgi]
#user info
uid = metheuser
gid = ers_group
#application's base folder
base = /home/metheuser/webapps/ers_portal
#python module to import
app = run_web
module = %(app)
home = %(base)/ers_portal_venv
pythonpath = %(base)
#socket file's location
socket = /home/metheuser/webapps/ers_portal/%n.sock
#permissions for the socket file
chmod-socket = 666
#uwsgi varible only, does not relate to your flask application
callable = app
#location of log files
logto = /home/metheuser/webapps/ers_portal/logs/%n.log
Relevant parts of my views.py
data_modification_time = None
data = None
def reload_data():
global data_modification_time, data, sites, column_names
filename = '/home/metheuser/webapps/ers_portal/app/static/' + ec.dd_filename
mtime = os.stat(filename).st_mtime
if data_modification_time != mtime:
data_modification_time = mtime
with open(filename) as f:
data = pickle.load(f)
return data
#a bunch of authentication stuff...
#app.route('/')
#app.route('/index')
def index():
return render_template("index.html",
title = 'Main',)
#app.route('/login', methods = ['GET', 'POST'])
def login():
login stuff...
#app.route('/my_table')
#login_required
def my_table():
print 'trying to access data table...'
data = reload_data()
return render_template("my_table.html",
title = "Rundata Viewer",
sts = sites,
cn = column_names,
data = data) # dictionary of data
I installed nginx via yum as described here (yesterday)
I am using uWSGI installed in my venv via pip
I am on CentOS 6
My uwsgi log shows:
Wed Jun 11 17:20:01 2014 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 287] during GET /whm-server-status (127.0.0.1)
IOError: write error
[pid: 9586|app: 0|req: 135/135] 127.0.0.1 () {24 vars in 292 bytes} [Wed Jun 11 17:20:01 2014] GET /whm-server-status => generated 0 bytes in 3 msecs (HTTP/1.0 404) 2 headers in 0 bytes (0 switches on core 0)
When its working, the print statement in the views "my_table" route prints into the log file. But not once it stops working.
Any ideas?
Related
I am writing a REST plugin for Foswiki using Perl and I am facing an reliability issue when using File::Find. I have tried my best to write a minimal reproducible example. The plugin uses File::Find to traverse directories and print the filenames in the HTTP response. The REST request is working properly 4 times, but stop to work the 5th time. The HTTP status remain “HTTP/1.1 200 OK” but no file is reported by File::Find anymore.
The webserver is nginx and is configured to use FastCGI. It appear to run 4 working threads managed by foswiki-fcgi-pm:
> ps aux
www-data 16957 0.0 7.7 83412 78332 ? Ss 16:52 0:00 foswiki-fcgi-pm
www-data 16960 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
www-data 16961 0.0 7.6 84004 76828 ? S 16:52 0:00 foswiki-fcgi
www-data 16962 0.0 7.6 83956 76844 ? S 16:52 0:00 foswiki-fcgi
www-data 16963 0.0 7.5 83960 76740 ? S 16:52 0:00 foswiki-fcgi
Firstly, the plugin initialization simply register the REST handler:
sub initPlugin {
my ( $topic, $web, $user, $installWeb ) = #_;
# check for Plugins.pm versions
if ( $Foswiki::Plugins::VERSION < 2.3 ) {
Foswiki::Func::writeWarning( 'Version mismatch between ',
__PACKAGE__, ' and Plugins.pm' );
return 0;
}
Foswiki::Func::registerRESTHandler(
'restbug', \&RestBug,
authenticate => 0, # Set to 0 if handler should be useable by WikiGuest
validate => 0, # Set to 0 to disable StrikeOne CSRF protection
http_allow => 'GET,POST', # Set to 'GET,POST' to allow use HTTP GET and POST
description => 'Debug'
);
# Plugin correctly initialized
return 1;
}
Secondly, the REST handler is implemented as follow, printing all the files it can possibly find:
sub RestBug {
my ($session, $subject, $verb, $response) = #_;
my #Directories = ("/var/www/foswiki/tools");
sub findfilestest
{
$response->print("FILE $_\n");
}
find({ wanted => \&findfilestest }, #Directories );
}
When I test the REST service with a HTTP request, the first 4 times I get the following HTTP response, which seems quite satisfying:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:23:10 GMT
Content-Length: 541
Connection: keep-alive
Set-Cookie: SFOSWIKISID=385db599c5d66bb19591e1eef7f1a854; path=/; secure; HttpOnly
FILE .
FILE foswiki.freebsd.init-script
FILE bulk_copy.pl
FILE dependencies
FILE mod_perl_startup.pl
FILE geturl.pl
FILE extender.pl
FILE extension_installer
FILE configure
FILE lighttpd.pl
FILE foswiki.freebsd.etc-defaults
FILE save-pending-checkins
FILE babelify
FILE upgrade_emails.pl
FILE tick_foswiki.pl
FILE foswiki.defaults
FILE rewriteshebang.pl
FILE fix_file_permissions.sh
FILE foswiki.init-script
FILE convertTopicSettings.pl
FILE mailnotify
FILE html2tml.pl
FILE tml2html.pl
FILE systemd
FILE foswiki.service
The following attempts give this unexpected response:
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Tue, 22 Nov 2022 09:24:56 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: SFOSWIKISID=724b2c4b1ddfbebd25d0dc2a0f182142; path=/; secure; HttpOnly
Note that if I restart Foswiki with the command systemctl restart foswiki, the REST service work again 4 more times.
How to make this REST service work more than 4 times in a row?
Selenium WebDriver (run from Eclipse Oxygen.3a Release (4.7.3a)) seems to be unable to handle any HTTPS website.
By that I mean this: upon loading the page (regardless of whether it redirects or not, tried with gitlab and AWS logon sites)
Logon page loads properly (no SSL certificate issues, any redirect happens just fine), yet Selenium doesn't do anything (debug mode shows it's not even trying to execute the logon code)
It times out (browserWaitTimeout=15) and closes/reopens the logon page
This happens until the 4th time, upon which Selenium finally reaches my logon code (a breakpoint I set there is finally hit), but this happens between browser page closing and reopening, thus it never detects the DOM and fails.
And the most frustrating thing is, console shows NO ERRORS.
Here: Chrome:
Jul 03, 2019 8:40:44 AM com.fincad.vcr.qa.support.WebDriverFactory createWebDriver
INFO: Web driver is created successfully
Jul 03, 2019 8:40:55 AM com.fincad.vcr.qa.support.WebDriverFactory quitWebDriver
INFO: Web driver quits successfully
That's it, as in literally it (notice the 15 second gap? That's browserWaitTimeout).
a) I tried FF, where I saw errors from Marionette:
1562172157538 Marionette INFO Listening on port 56792
1562172157572 addons.xpi-utils DEBUG Successfully read XPI database
1562172157603 addons.manager DEBUG Registering upgrade listener for
formautofill#mozilla.org
Jul 03, 2019 9:42:37 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Jul 03, 2019 9:42:38 AM com.fincad.vcr.qa.support.WebDriverFactory createWebDriver
INFO: Web driver is created successfully
1562172170219 Marionette INFO Stopped listening on port 56792
1562172170243 addons.xpi DEBUG Calling bootstrap method shutdown on webcompat#mozilla.org version 4.0.0
1562172170249 addons.xpi DEBUG Calling bootstrap method shutdown on screenshots#mozilla.org version 37.1.0
1562172170253 addons.xpi DEBUG Calling bootstrap method shutdown on fxmonitor#mozilla.org version 3.0
1562172170254 addons.xpi DEBUG Calling bootstrap method shutdown on formautofill#mozilla.org version 1.0
[Parent 12824, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 21788, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 21788, Chrome_Chi[Parent 12824, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 6644, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 6644, Chrome_ChildThread] WAR[Parent 12824, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 17844, Chrome_ChildThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Child 17844, Chrome_ChildThread] WARNING: pipe er[Parent 12824, Gecko_IOThread] WARNING: pipe error: 109: file z:/task_1560988628/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 341
[Chi###!!! [Parent][MessageChannel] Error: (msgtype=0x1F0099,name=PBrowser::Msg_UpdateNativeWindowHandle) Closed channel: cannot send/recv
I googled and people mentioned it's due to outdated FF driver, so I upgraded, but it didn't do any good (same errors appear). And in IE it's like Chrome: no error or warning msg at all.
My drivers are these:
Selenium WebDriver: 3.14.0 (32-bit)
geckodriver 0.24.0 (32-bit)
chromedriver 74.0.3729.6 (32-bit)
IEDriverServer 3.8.0 (32-bit)
And my browsers:
Chrome browser: 75.0.3770.100
FF browser: 67.0.4
IE browser: 11.557.17763
This is how I create my drivers:
private static void createFirefoxDriver() {
setCapability("firefox");
GeckoDriverService.Builder builder = new GeckoDriverService.Builder();
GeckoDriverService service = builder.build();
FirefoxOptions options = new FirefoxOptions(capabilities);
// Read the default firefox profile
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myProfile = profile.getProfile("default");
// Disable the web page from asking if really want to leave
myProfile.setPreference("dom.disable_beforeunload", true);
options.setProfile(myProfile);
driver = new FirefoxDriver(service, options);
}
private static void createChromeDriver() {
setCapability("chrome");
ChromeDriverService.Builder builder = new ChromeDriverService.Builder();
ChromeDriverService service = builder.build();
ChromeOptions options = new ChromeOptions();
options.merge(capabilities);
driver = new ChromeDriver(service, options);
}
This is how I load the URL (local webdriver):
private static void createLocalWebDriver() {
String browser = ConfigParser.getBrowser();
LOGGER.info("Target Browser: " + browser);
switch (browser) {
case "firefox":
System.setProperty("webdriver.gecko.driver", ConfigParser.getGeckoDriver());
System.out.println("GeckoDriver on: " + ConfigParser.getGeckoDriver());
createFirefoxDriver();
break;
case "chrome":
System.setProperty("webdriver.chrome.driver", ConfigParser.getChromeDriver());
System.out.println("ChromeDriver on: " + ConfigParser.getChromeDriver());
createChromeDriver();
break;
case "ie":
System.setProperty("webdriver.ie.driver", ConfigParser.getIEDriverServer());
System.out.println("IEDriver on: " + ConfigParser.getIEDriverServer());
createIEDriver();
break;
default:
LOGGER.warning("Unsupported Browser: " + browser);
break;
}
}
private static void createFirefoxDriver() {`enter code here`
setCapability("firefox");
GeckoDriverService.Builder builder = new GeckoDriverService.Builder();
GeckoDriverService service = builder.build();
FirefoxOptions options = new FirefoxOptions(capabilities);
ProfilesIni profile = new ProfilesIni();
FirefoxProfile myProfile = profile.getProfile("default");
myProfile.setPreference("dom.disable_beforeunload", true);
options.setProfile(myProfile);
driver = new FirefoxDriver(service, options);
}
private static void createChromeDriver() {
setCapability("chrome");
ChromeDriverService.Builder builder = new ChromeDriverService.Builder();
ChromeDriverService service = builder.build();
ChromeOptions options = new ChromeOptions();
options.merge(capabilities);
driver = new ChromeDriver(service, options);
}
private static void createIEDriver() {
setCapability("ie");
InternetExplorerDriverService.Builder builder = new InternetExplorerDriverService.Builder();
InternetExplorerDriverService service = builder.build();
InternetExplorerOptions options = new InternetExplorerOptions(capabilities);
driver = new InternetExplorerDriver(service, options);
CommonJS.executeScript(driver, "window.localStorage.clear();");
CommonJS.executeScript(driver, "window.sessionStorage.clear();");
}
private static void setCapability(String browser) {
capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.ForSeleniumServer.ENSURING_CLEAN_SESSION, true);
capabilities.setCapability(CapabilityType.ELEMENT_SCROLL_BEHAVIOR, true);
if (browser.equalsIgnoreCase("ie")) {
capabilities.setCapability(InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS, true);
capabilities.setCapability(InternetExplorerDriver.IGNORE_ZOOM_SETTING, true);
capabilities.setCapability(InternetExplorerDriver.REQUIRE_WINDOW_FOCUS, true);
capabilities.setCapability(InternetExplorerDriver.INITIAL_BROWSER_URL, ConfigParser.getAppUrl());
} else if (browser.equalsIgnoreCase("chrome")) {
ChromeOptions options = new ChromeOptions();
options.addArguments("chrome.switches","--disable-extensions");
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
}
}
Sorry for all the code, but I'm sure I'll be asked about those details, hence my putting there.
Long story short:
This only happens with HTTPS URL's
Non HTTPS URL's work just fine
It's not an SSL certificate issue otherwise I would have seen it on the screen "This connection is untrusted".
It was a problem at page load code. Found a section that was explicitly expecting HTTP only, thus refusing to load any HTTPS URL.
i have docker image containing python files which should download satellite imageries from scihub website. The docker image is working fine. Now when i want to create the deployment thorugh kubectl so that i can expose it as a service, its's container keeps on crashing. That's what the pod description says when seen through kubectl describe pod.
this is how i am trying to deploy sudo kubectl run back --image=back:latest --port=8080 --image-pull-policy Never. i also tried changing the port but it did not work. Here are the files within docker image.
Docker File
FROM python:3.7-stretch
COPY . /code
WORKDIR /code
RUN pip install -r requirements.txt
ENTRYPOINT ["python", "ingestion.py"]
** ingestion **
import os
import shutil
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(name)s - %(message)s')
logger = logging.getLogger("ingestion")
import requests
import datahub
scihub_username = os.environ["scihub_username"]
scihub_password = os.environ["scihub_password"]
result_url = "http://" + os.environ["CDINRW_BASE_URL"] + "/jobs/" + os.environ["CDINRW_JOB_ID"] + "/results"
logger.info("Searching the Copernicus Open Access Hub")
scenes = datahub.search(username=scihub_username,
password=scihub_password,
producttype=os.getenv("producttype"),
platformname=os.getenv("platformname"),
days_back=os.getenv("days_back", 2),
footprint=os.getenv("footprint"),
max_cloud_cover_percentage=os.getenv("max_cloud_cover_percentage"),
start_date = os.getenv("start_date"),
end_date = os.getenv("end_date"))
logger.info("Found {} relevant scenes".format(len(scenes)))
job_results = []
for scene in scenes:
# do not donwload a scene that has already been ingested
if os.path.exists(os.path.join("/out_data", scene["title"]+".SAFE")):
logger.info("The scene {} already exists in /out_data and will not be downloaded again.".format(scene["title"]))
filename = scene["title"]+".SAFE"
else:
logger.info("Starting the download of scene {}".format(scene["title"]))
filename = datahub.download(scene, "/tmp", scihub_username, scihub_password, unpack=True)
logger.info("The download was successful.")
shutil.move(filename, "/out_data")
result_message = {"description": "test",
"type": "Raster",
"format": "SAFE",
"filename": os.path.basename(filename)}
job_results.append(result_message)
res = requests.put(result_url, json=job_results, timeout=60)
res.raise_for_status()
*datahub
import logging
import os
import urllib.parse
import zipfile
import requests
# constructing URLs for querying the data hub
_BASE_URL = "https://scihub.copernicus.eu/dhus/"
SITE = {}
SITE["SEARCH"] = _BASE_URL + "search?format=xml&sortedby=beginposition&order=desc&rows=100&start={offset}&q="
_PRODUCT_URL = _BASE_URL + "odata/v1/Products('{uuid}')/"
SITE["CHECKSUM"] = _PRODUCT_URL + "Checksum/Value/$value"
SITE["SAFEZIP"] = _PRODUCT_URL + "$value"
logger = logging.getLogger(__name__)
def _build_search_url(producttype=None, platformname=None, days_back=2, footprint=None, max_cloud_cover_percentage=None, start_date=None, end_date=None):
search_terms = []
if producttype:
search_terms.append("producttype:{}".format(producttype))
if platformname:
search_terms.append("platformname:{}".format(platformname))
if start_date and end_date:
search_terms.append(
"beginPosition:[{}+TO+{}]".format(start_date, end_date))
elif days_back:
search_terms.append(
"beginPosition:[NOW-{}DAYS+TO+NOW]".format(days_back))
if footprint:
search_terms.append("footprint:%22Intersects({})%22".format(
footprint.replace(" ", "+")))
if max_cloud_cover_percentage:
search_terms.append("cloudcoverpercentage:[0+TO+{}]".format(max_cloud_cover_percentage))
url = SITE["SEARCH"] + "+AND+".join(search_terms)
return url
def _unpack(zip_file, directory, remove_after=False):
with zipfile.ZipFile(zip_file) as zf:
# This assumes that the zipfile only contains the .SAFE directory at root level
safe_path = zf.namelist()[0]
zf.extractall(path=directory)
if remove_after:
os.remove(zip_file)
return os.path.normpath(os.path.join(directory, safe_path))
def search(username, password, producttype=None, platformname=None ,days_back=2, footprint=None, max_cloud_cover_percentage=None, start_date=None, end_date=None):
""" Search the Copernicus SciHub
Parameters
----------
username : str
user name for the Copernicus SciHub
password : str
password for the Copernicus SciHub
producttype : str, optional
product type to filter for in the query (see https://scihub.copernicus.eu/userguide/FullTextSearch#Search_Keywords for allowed values)
platformname : str, optional
plattform name to filter for in the query (see https://scihub.copernicus.eu/userguide/FullTextSearch#Search_Keywords for allowed values)
days_back : int, optional
number of days before today that will be searched. Default are the last 2 days. If start and end date are set the days_back parameter is ignored
footprint : str, optional
well-known-text representation of the footprint
max_cloud_cover_percentage: str, optional
percentage of cloud cover per scene. Can only be used in combination with Sentinel-2 imagery.
(see https://scihub.copernicus.eu/userguide/FullTextSearch#Search_Keywords for allowed values)
start_date: str, optional
start point of the search extent has to be used in combination with end_date
end_date: str, optional
end_point of the search extent has to be used in combination with start_date
Returns
-------
list
a list of scenes that match the search parameters
"""
import xml.etree.cElementTree as ET
scenes = []
search_url = _build_search_url(producttype, platformname, days_back, footprint, max_cloud_cover_percentage, start_date, end_date)
logger.info("Search URL: {}".format(search_url))
offset = 0
rowsBreak = 5000
name_space = {"atom": "http://www.w3.org/2005/Atom",
"opensearch": "http://a9.com/-/spec/opensearch/1.1/"}
while offset < rowsBreak: # Next pagination page:
response = requests.get(search_url.format(offset=offset), auth=(username, password))
root = ET.fromstring(response.content)
if offset == 0:
rowsBreak = int(
root.find("opensearch:totalResults", name_space).text)
for e in root.iterfind("atom:entry", name_space):
uuid = e.find("atom:id", name_space).text
title = e.find("atom:title", name_space).text
begin_position = e.find(
"atom:date[#name='beginposition']", name_space).text
end_position = e.find(
"atom:date[#name='endposition']", name_space).text
footprint = e.find("atom:str[#name='footprint']", name_space).text
scenes.append({
"id": uuid,
"title": title,
"begin_position": begin_position,
"end_position": end_position,
"footprint": footprint})
# Ultimate DHuS pagination page size limit (rows per page).
offset += 100
return scenes
def download(scene, directory, username, password, unpack=True):
""" Download a Sentinel scene based on its uuid
Parameters
----------
scene : dict
the scene to be downloaded
path : str
the path where the file will be downloaded to
username : str
username for the Copernicus SciHub
password : str
password for the Copernicus SciHub
unpack: boolean, optional
flag that defines whether the downloaded product should be unpacked after download. defaults to true
Raises
------
ValueError
if the size of the downloaded file does not match the Content-Length header
ValueError
if the checksum of the downloaded file does not match the checksum provided by the Copernicus SciHub
Returns
-------
str
path to the downloaded file
"""
import hashlib
md5hash = hashlib.md5()
md5sum = requests.get(SITE["CHECKSUM"].format(
uuid=scene["id"]), auth=(username, password)).text
download_path = os.path.join(directory, scene["title"] + ".zip")
# overwrite if path already exists
if os.path.exists(download_path):
os.remove(download_path)
url = SITE["SAFEZIP"].format(uuid=scene["id"])
rsp = requests.get(url, auth=(username, password), stream=True)
cl = rsp.headers.get("Content-Length")
size = int(cl) if cl else -1
# Actually fetch now:
with open(download_path, "wb") as f: # Do not read as a whole into memory:
written = 0
for block in rsp.iter_content(8192):
f.write(block)
written += len(block)
md5hash.update(block)
written = os.path.getsize(download_path)
if size > -1 and written != size:
raise ValueError("{}: size mismatch, {} bytes written but expected {} bytes to write!".format(
download_path, written, size))
elif md5sum:
calculated = md5hash.hexdigest()
expected = md5sum.lower()
POD events
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 2m39s (x18636 over 2d19h) kubelet, minikube Back-off restarting failed container
the system which wants to use this service already has another main front end service running(which just runs the application ) on 8081 so maybe i need to expose this on the same port. How can i make the deployments running?
I met an error when running codes at the bottom. It's like a simple ftp.
I use python2.6.6 and CentOS release 6.8
In most linux server, it gets right results like this:(I'm very sorry that I have just sign up and couldn't )
Clinet:
[root#Test ftp]# python client.py
path:put|/home/aaa.txt
Server:
[root#Test ftp]# python server.py
connected...
pre_data:put|aaa.txt|4
cmd: put
file_name: aaa.txt
file_size: 4
upload successed.
But I get errors in some server(such as my own VM in my PC). I have done lots of tests(python2.6/python2.7, Centos6.5/Centos6.7) and found this error is not because them. Here is the error imformation:
[root#Lewis-VM ftp]# python server.py
connected...
pre_data:put|aaa.txt|7sdfsdf ###Here gets the wrong result, "sdfsdf" is the content of /home/aaa.txt and it shouldn't be sent here to 'file_size' and so it cause the "ValueError" below
cmd: put
file_name: aaa.txt
file_size: 7sdfsdf
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 10699)
Traceback (most recent call last):
File "/usr/lib64/python2.6/SocketServer.py", line 570, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib64/python2.6/SocketServer.py", line 332, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib64/python2.6/SocketServer.py", line 627, in __init__
self.handle()
File "server.py", line 30, in handle
if int(file_size)>recv_size:
ValueError: invalid literal for int() with base 10: '7sdfsdf\n'
What's more, I found that if I insert a time.sleep(1) between sk.send(cmd+"|"+file_name+'|'+str(file_size)) and sk.send(data) in client.py, the error will disappear. I have said that I did tests in different system and python versions and the error is not because them. So I guess that is it because of some system configs? I have check about socket.send() and socket.recv() in python.org but fount nothing helpful. So could somebody help me to explain why this happend?
The code are here:
#!/usr/bin/env python
#coding:utf-8
################
#This is server#
################
import SocketServer
import os
class MyServer(SocketServer.BaseRequestHandler):
def handle(self):
base_path = '/home/ftp/file'
conn = self.request
print 'connected...'
while True:
#####receive pre_data: we should get data like 'put|/home/aaa|7'
pre_data = conn.recv(1024)
print 'pre_data:' + pre_data
cmd,file_name,file_size = pre_data.split('|')
print 'cmd: ' + cmd
print 'file_name: '+ file_name
print 'file_size: '+ file_size
recv_size = 0
file_dir = os.path.join(base_path,file_name)
f = file(file_dir,'wb')
Flag = True
####receive 1024bytes each time
while Flag:
if int(file_size)>recv_size:
data = conn.recv(1024)
recv_size+=len(data)
else:
recv_size = 0
Flag = False
continue
f.write(data)
print 'upload successed.'
f.close()
instance = SocketServer.ThreadingTCPServer(('127.0.0.1',9999),MyServer)
instance.serve_forever()
#!/usr/bin/env python
#coding:utf-8
################
#This is client#
################
import socket
import sys
import os
ip_port = ('127.0.0.1',9999)
sk = socket.socket()
sk.connect(ip_port)
while True:
input = raw_input('path:')
#####we should input like 'put|/home/aaa.txt'
cmd,path = input.split('|')
file_name = os.path.basename(path)
file_size=os.stat(path).st_size
sk.send(cmd+"|"+file_name+'|'+str(file_size))
send_size = 0
f= file(path,'rb')
Flag = True
#####read 1024 bytes and send it to server each time
while Flag:
if send_size + 1024 >file_size:
data = f.read(file_size-send_size)
Flag = False
else:
data = f.read(1024)
send_size+=1024
sk.send(data)
f.close()
sk.close()
The TCP is a stream of data. That is the problem. TCP do not need to keep message boundaries. So when a client calls something like
connection.send("0123456789")
connection.send("ABCDEFGHIJ")
then a naive server like
while True;
data = conn.recv(1024)
print data + "_"
may print any of:
0123456789_ABCDEFGHIJ_
0123456789ABCDEFGHIJ_
0_1_2_3_4_5_6_7_8_9_A_B_C_D_E_F_G_H_I_J_
The server has no chance to recognize how many sends client called because the TCP stack at client side just inserted data to a stream and the server must be able to process the data received in different number of buffers than the client used.
Your server must contain a logic to separate the header and the data. All of application protocols based on TCP use a mechanism to identify application level boundaries. For example HTTP separates headers and body by an empty line and it informs about the body length in a separate header.
Your program works correctly when server receives a header with the command, name and size in a separate buffer it it fails when client is fast enough and push the data into stream quickly and the server reads header and data in one chunk.
I'm trying to make a local docker-dovecot machine to archive my e-mails. I would like to query them with Apple Mail. I have a simple ubuntu docker machine (on an VM with parallels, because I'm on a Mac).
I have this local.conf:
# A comma separated list of IPs or hosts where to listen in for connections.
# "*" listens in all IPv4 interfaces, "::" listens in all IPv6 interfaces.
# If you want to specify non-default ports or anything more complex,
# edit conf.d/master.conf.
listen = *,::
# Protocols we want to be serving.
protocols = imap
# Static passdb.
# This can be used for situations where Dovecot doesn't need to verify the
# username or the password, or if there is a single password for all users:
passdb {
driver = static
args = password=dovecot
}
# Location for users' mailboxes. The default is empty, which means that Dovecot
# tries to find the mailboxes automatically. This won't work if the user
# doesn't yet have any mail, so you should explicitly tell Dovecot the full
# location.
#
# If you're using mbox, giving a path to the INBOX file (eg. /var/mail/%u)
# isn't enough. You'll also need to tell Dovecot where the other mailboxes are
# kept. This is called the "root mail directory", and it must be the first
# path given in the mail_location setting.
#
# There are a few special variables you can use, eg.:
#
# %u - username
# %n - user part in user#domain, same as %u if there's no domain
# %d - domain part in user#domain, empty if there's no domain
# %h - home directory
#
# See doc/wiki/Variables.txt for full list. Some examples:
#
# mail_location = maildir:~/Maildir
# mail_location = mbox:~/mail:INBOX=/var/mail/%u
# mail_location = mbox:/var/mail/%d/%1n/%n:INDEX=/var/indexes/%d/%1n/%n
#
# <doc/wiki/MailLocation.txt>
#
mail_location = maildir:/var/mail/%n
# System user and group used to access mails. If you use multiple, userdb
# can override these by returning uid or gid fields. You can use either numbers
# or names. <doc/wiki/UserIds.txt>
# mail_uid = CHANGE_THIS_to_your_short_user_name_or_uid
# mail_gid = admin
# SSL/TLS support: yes, no, required. <doc/wiki/SSL.txt>
ssl = no
# Login user is internally used by login processes. This is the most untrusted
# user in Dovecot system. It shouldn't have access to anything at all.
# default_login_user = _dovenull
# Internal user is used by unprivileged processes. It should be separate from
# login user, so that login processes can't disturb other processes.
# default_internal_user = _dovecot
# Setting limits.
default_process_limit = 10
default_client_limit = 50
and I'm getting this from Apple Mail
May 23 07:15:58 Mail[87524] <Debug>: <0x7fe16f021cd0:[Non-authenticated]> Wrote: 1.11 ID ("name" "Mac OS X Mail" "version" "9.3 (3124)" "os" "Mac OS X" "os-version" "10.11.5 (15F34)" "vendor" "Apple Inc.")
May 23 07:15:58 Mail[87524] <Debug>: <0x7fe16f021cd0:[Non-authenticated]> Read: * ID {
name = Dovecot;
}
May 23 07:15:58 Mail[87524] <Debug>: <0x7fe16f021cd0:[Non-authenticated]> Read: 1.11 OK
May 23 07:15:58 Mail[87524] <Debug>: <0x7fe16f021cd0:[Non-authenticated]> Wrote: 3.11 LOGOUT
May 23 07:16:00 Mail[87524] <Debug>: <0x7fe16aa14590:[Disconnected]> Read: * OK [CAPABILITY (
IMAP4REV1,
"LITERAL+",
"SASL-IR",
"LOGIN-REFERRALS",
ID,
ENABLE,
IDLE,
"AUTH=PLAIN",
"AUTH=LOGIN"
)]
May 23 07:16:00 Mail[87524] <Debug>: <0x7fe16aa14590:[Non-authenticated]> Wrote: 1.23 ID ("name" "Mac OS X Mail" "version" "9.3 (3124)" "os" "Mac OS X" "os-version" "10.11.5 (15F34)" "vendor" "Apple Inc.")
May 23 07:16:00 Mail[87524] <Debug>: <0x7fe16aa14590:[Non-authenticated]> Read: * ID {
name = Dovecot;
}
May 23 07:16:00 Mail[87524] <Debug>: <0x7fe16aa14590:[Non-authenticated]> Read: 1.23 OK
May 23 07:16:00 Mail[87524] <Debug>: <0x7fe16aa14590:[Non-authenticated]> Wrote: 3.23 LOGOUT
and this from dovecot (mail.log):
May 23 05:07:22 f8ab3e20742f dovecot: master: Dovecot v2.2.9 starting up (core dumps disabled)
May 23 05:07:22 f8ab3e20742f dovecot: ssl-params: Generating SSL parameters
May 23 05:07:29 f8ab3e20742f dovecot: ssl-params: SSL parameters regeneration completed
May 23 05:07:52 f8ab3e20742f dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=10.211.55.2, lip=172.17.0.2, session=<IJwtbHszbgAK0zcC>
May 23 05:07:54 f8ab3e20742f dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=10.211.55.2, lip=172.17.0.2, session=<qsRNbHszdgAK0zcC>
The output of doveconf -n is (so "disable_plaintext_auth = no" is active):
# 2.2.9: /etc/dovecot/dovecot.conf
# OS: Linux 4.4.8-boot2docker x86_64 Ubuntu 14.04.4 LTS aufs
auth_mechanisms = plain login
default_client_limit = 50
default_process_limit = 10
disable_plaintext_auth = no
listen = *,::
mail_location = maildir:/var/mail/%n
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
args = password=dovecot
driver = static
}
protocols = imap
ssl = no
Any suggestions why this login isn't working?
Thanks!
The solution is to fix and configure the following line correctly (from local.conf):
# mail_uid = CHANGE_THIS_to_your_short_user_name_or_uid
How did I find out? Thanks to #Kondybas for the pointer to try another client. I used Thunderbird and it produced dovecot log entries (why didn't Apple Mail produce these lines? No clue), saying that it couldn't switch to mail_uid user context. I extended dovecot Dockerfile and switched the user appropriately. Afterwards it worked with Thunderbird and then with Apple Mail.