I have got follow code:
void main() async {
connection = PostgreSQLConnection('localhost', 5432, 'db', username: 'postgres', password: '12345');
await connection.open();
//...
// some http handler that have next code inside:
List<List<dynamic>> result = await connection.query(body['sql']).timeout(Duration(seconds: 90));
If I run a lot of queries to http handler I am getting error:
TimeoutException after 0:00:30.000000: Future not completed
It's very strange, because on query I have timeout 90 seconds. So it's happens on connection. But I am opening connection only once in main.
Could anybody explain how it's happens? And how to fix it?
PostgreSQLConnection has parameters: timeoutInSeconds and queryTimeoutInSeconds and they both default to 30. I suggest you use them instead of timeout of Future.
Related
I have a server running on a Raspberry Pi which is accessible from a browser.
http://192.168.1.67:55XXX/?email=a#b.com
yields:-
{Order=[{no=0, day_0=1, price=0, name=PPAC SUPERTHERM 20K, display_colour=blue, notice=1, special_order=false}, {no=1, day_0=1, price=0, name=SLACK 50KG , display_colour=blue, notice=1, special_order=false}, {no=0, day_0=1, price=0, name=PPAC SUPERTHERM 20K, display_colour=blue, day_5=1, notice=1, special_order=false, day_3=1}, {no=1, day_0=1, price=0, name=SLACK 50KG , display_colour=blue, day_5=1, notice=1, special_order=false, day_3=1}], Details={address=xx Farriers Lea, phone=0xxxx 606635, name=Fred Bloggs, mobile=, details=end shed on drive, email=a#b.com}}
I am using VSCODE to debug dart code.
my dart code:-
String remote_ip = '192.168.1.67'; //212.159.118.177';
var remote_port = 55XXX;
Socket socket;
String _dataToBeSent = "http://?email=a#b.com\n";
var reply;
// connect
main(List<String> arguments) async {
await _remoteServerConnect();
}
// REMOTE SERVER CONNECT
Future _remoteServerConnect() async {
// await Socket.connect(remote_ip, remote_port).then((Socket sock){
await Socket.connect(remote_ip, remote_port).then((Socket sock) {
socket = sock;
print('Got connected ${socket.remoteAddress}');
socket.listen(dataHandler,
onError: errorHandler, onDone: doneHandler, cancelOnError: false);
}).catchError((AsyncError e) {
print("Unable to connect: $e");
exit(1);
});
}
void dataHandler(data) async {
await print('"'+String.fromCharCodes(data).trim()+'"');
if (String.fromCharCodes(data).trim().endsWith('html')) {
print("Send Data = $_dataToBeSent");
socket.add(utf8.encode(_dataToBeSent));
// socket.writeln(_dataToBeSent);
socket.flush();
await Future.delayed(Duration(seconds: 5));
}
}
void errorHandler(error, StackTrace trace) {
print(error);
}
void doneHandler() {
// socket.destroy();
exit(0);
}
The dart debug consol yields:-
Connecting to VM Service at ws://127.0.0.1:54799/kE9Xa1JQclk=/ws
Got connected InternetAddress('192.168.1.67', IPv4)
"HTTP/1.1 200 OK" <sent by server
"ContentType: text/html" <sent by server
Send Data = http://?email=a#b.com
Exited
The server consol yields:-
Server is ready
WEB Client connected: /192.168.1.66
05.08/10:34:21.17 - Waiting for command..
05.08/10:34:31.80 - Socket Timeout
05.08/10:34:31.82 - Done -------------
05.08/10:34:31.82 - Waiting for command..
Connection has been closed
Server is ready
It would seem the
socket.add(utf8.encode(_dataToBeSent));
// socket.writeln(_dataToBeSent);
socket.flush();
did not send data to the server?? Why?? any ideas gratefully received!
Disabling my dev machines firewall did NOT improve answer!
Steve
The server used
final DataInputStream in = new DataInputStream(clientSocket.getInputStream());
while (estimatedTime < 60000) {
report("Waiting for command..");
try {
// in.readNBytes(command.data, 0, command.length);
in.readFully(command.data);
to receive the data command.data is 100bytes long so was waiting for the complete input. I padded to 100 bytes worked fine - will sort a better solution later.
So I've been trying for while to establish a websocket connection between my flutter app and FastAPI.
I believe the problem lies in Flutter.
So far i've tried the flutter packages socket_io_client, web_socket_channel and websocket_manager to no awail.
I suspect it might have to do with the app architecture maybe... bit at a loss atm.
Here is the flutter errors:
I/onListen(26110): arguments: null
I/EventStreamHandler(26110): 🔴 event sink
I/onListen(26110): arguments: null
I/EventStreamHandler(26110): 🔴 event sink
W/System.err(26110): java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
W/System.err(26110): at okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess$okhttp(RealWebSocket.kt:185)
W/System.err(26110): at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:156)
W/System.err(26110): at okhttp3.RealCall$AsyncCall.run(RealCall.kt:140)
W/System.err(26110): at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/System.err(26110): at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/System.err(26110): at java.lang.Thread.run(Thread.java:923)
I/EventStreamHandler(26110): ✅ sink is not null
I/flutter (26110): websocket closed
Im aware it says the 403 forbidden came from my API, though i know websocket connection is possible, as i've tested it with javascript.
here is the log from the API:
DEBUG | websockets.protocol:__init__:244 - server - state = CONNECTING
DEBUG | websockets.protocol:connection_made:1340 - server - event = connection_made(<_SelectorSocketTransport fd=484 read=polling write=<idle, bufsize=0>>)
DEBUG | websockets.protocol:data_received:1412 - server - event = data_received(<422 bytes>)
DEBUG | websockets.server:read_http_request:237 - server < GET /11 HTTP/1.1
DEBUG | websockets.server:read_http_request:238 - server < Headers([('authorization', 'Bearer *JWTTOKEN*'), ('upgrade', 'websocket'), ('connection', 'Upgrade'), ('sec-websocket-key', 'zytoCsWVlcmsKghL5XFEdA=='), ('sec-websocket-version', '13'), ('host', '10.0.2.2:8000'), ('accept-encoding', 'gzip'), ('user-agent', 'okhttp/4.3.1')])
INFO | uvicorn.protocols.websockets.websockets_impl:asgi_send:198 - ('127.0.0.1', 50772) - "WebSocket /11" 403
DEBUG | websockets.server:write_http_response:256 - server > HTTP/1.1 403 Forbidden
DEBUG | websockets.server:write_http_response:257 - server > Headers([('Date', 'Fri, 09 Apr 2021 11:10:11 GMT'), ('Server', 'Python/3.7 websockets/8.1'), ('Content-Length', '0'), ('Content-Type', 'text/plain'), ('Connection', 'close')])
DEBUG | websockets.server:write_http_response:267 - server > body (0 bytes)
DEBUG | websockets.protocol:fail_connection:1261 - server ! failing CONNECTING WebSocket connection with code 1006
DEBUG | websockets.protocol:connection_lost:1354 - server - event = connection_lost(None)
DEBUG | websockets.protocol:connection_lost:1356 - server - state = CLOSED
DEBUG | websockets.protocol:connection_lost:1365 - server x code = 1006, reason = [no reason]
I have all the WebSocket code in a Class that is beeing 'provided', I.E WebSocketState:
return runApp(
MultiProvider(
providers: [
Provider<AuthenticationState>(
create: (_) => new AuthenticationState(),
),
Provider<WebSocketState>(
create: (_) => new WebSocketState(),
),
],
child: MyApp(),
),
);
WebSocketState:
class WebSocketState {
final _socketMessage = StreamController<Message>();
Sink<Message> get getMessageSink => _socketMessage.sink;
Stream<Message> get getMessageStream => _socketMessage.stream;
WebsocketManager socket;
bool isConnected() => true;
void connectAndListen(int userId) async {
var token = await secureStorage.read(key: 'token');
socket = WebsocketManager(
'ws://10.0.2.2:8000/$userId', {'Authorization': 'Bearer $token'});
socket.onClose((dynamic message) {
print('websocket closed');
});
// Listen to server messages
socket.onMessage((dynamic message) {
print("Message = " + message.toString());
});
// Connect to server
socket.connect();
}
void dispose() {
_socketMessage.close();
socket.close();
}
}
the connectAndListen method is called in the first/main page after user has authenticated, then in other Pages the websocket is beeing used.
#override
void didChangeDependencies() {
super.didChangeDependencies();
Provider.of<WebSocketState>(context, listen: false).connectAndListen(
Provider.of<AuthenticationState>(context, listen: false).id);
}
API websocket 'class':
websocket_notifier.py
from enum import Enum
import json
from typing import List
class SocketClient:
def __init__(self, user_id: int, websocket: WebSocket):
self.user_id = user_id
self.websocket = websocket
class WSObjects(Enum):
Message = 0
class Notifier:
def __init__(self):
self.connections: List[SocketClient] = []
self.generator = self.get_notification_generator()
async def get_notification_generator(self):
while True:
message = yield
await self._notify(message)
async def push(self, msg: str):
await self.generator.asend(msg)
async def connect(self, user_id: int, websocket: WebSocket):
await websocket.accept()
self.connections.append(SocketClient(user_id, websocket))
def remove(self, websocket: WebSocket):
client: SocketClient
for x in self.connections:
if x.websocket == websocket:
client = x
self.connections.remove(client)
async def _notify(self, message: str):
living_connections = []
while len(self.connections) > 0:
# Looping like this is necessary in case a disconnection is handled
# during await websocket.send_text(message)
client = self.connections.pop()
await client.websocket.send_text(message)
living_connections.append(client)
self.connections = living_connections
async def send(self, user_id: int, info: WSObjects, json_object: dict):
print("WS send running")
msg = {
"info": info,
"data": json_object
}
print("connections count: " + str(len(self.connections)))
for client in self.connections:
if client.user_id == user_id:
print("WS sending msg to ${client.user_id}")
await client.websocket.send_text(json.dumps(msg))
break
notifier = Notifier()
API main:
from fastapi import FastAPI
from websocket_notifier import notifier
from starlette.websockets import WebSocket, WebSocketDisconnect
app = FastAPI()
#app.get("/")
async def root():
return {"message": "Root"}
#app.websocket("/ws/{user_id}")
async def websocket_endpoint(user_id: int, websocket: WebSocket):
await notifier.connect(user_id, websocket)
try:
while True:
data = await websocket.receive_text()
await websocket.send_text(f"Message text was: {data}")
except WebSocketDisconnect:
notifier.remove(websocket)
#app.on_event("startup")
async def startup():
# Prime the push notification generator
await notifier.generator.asend(None)
Any ideas what Im doing wrong? (the other flutter websocket packages I've used virutally In the same way as the one I showed)
through lots of testing i finally found a way to get websockets to work with my flutter app and fastapi.
https://github.com/tiangolo/fastapi/issues/129
Had to try a bit of different things from that issue thread. But endend up with using python-socketio. I had to use a lower version of python-socketio to be compatible with the newest flutter socket_io_client package.
For those who have the same problem, please also check #2639. Prefix of the APIRouter does not work in websocket decorator.
I'm trying to learn Flutter/Dart and I'm having to much problems. Now I'm trying to obtain some values from an API. My code is:
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'package:myapp01_apirequest/src/models/uplink_models.dart';
class UplinksProvider{
String _url = 'xxx.yyy.com';
Future<List<Uplink>> getEnCines() async{
try{
final url = Uri.https(_url, ':14442/api/external/login', {
'username': 'Joe689',
'password': '15.Job_1825zz'
});
final resp = await http.get(url);
final decodedData = json.decode(resp.body);
print('Patata');
print(decodedData);
return [];
}catch (error){
print('++++++++++///////++++++++++++++++');
print(error);
print('++++++++++*******++++++++++++++++');
}
}
}
Reading the Uri constructor documentation I understood that I have to split in 3 my url.
First one is authority. In my case I think is xxx.yyy.com.
Second is the unencodedPath. In my case I think is :14442/api/external/login.
Finally a map with params in my case the username and pass (the only thing I'm pretty sure is correct in my code).
If I do this, any problem appears, but the print('Patata'); and print(decodeData); don't appear. In addition, a file called io_client.dart opens and marks the next line.
var ioRequest = (await _inner.openUrl(request.method, request.url))
The console shows nothing (I think):
Launching lib\main.dart on LG M700 in debug mode...
lib\main.dart:1
Formato de par�metros incorrecto:
√ Built build\app\outputs\flutter-apk\app-debug.apk.
Connecting to VM Service at ws://127.0.0.1:55883/KBAtv2-Jljc=/ws
Why no errors appears but I can't obtain my desired data?
EDIT:
According for what #Preet Shah's said I press the "VS run button" and appears the following exception three times:
I/flutter ( 1932): ++++++++++///////++++++++++++++++
I/flutter ( 1932): HandshakeException: Handshake error in client (OS Error:
I/flutter ( 1932): CERTIFICATE_VERIFY_FAILED: Hostname mismatch(handshake.cc:354))
I/flutter ( 1932): ++++++++++*******++++++++++++++++
And finally appears:
I/Choreographer( 1932): Skipped 148531 frames! The application may be doing too much work on its main thread.
D/vndksupport( 1932): Loading /vendor/lib/hw/android.hardware.graphics.mapper#2.0-impl.so from current namespace instead of sphal namespace.
D/vndksupport( 1932): Loading /vendor/lib/hw/gralloc.msm8937.so from current namespace instead of sphal namespace.
I/Choreographer( 1932): Skipped 35 frames! The application may be doing too much work on its main thread.
Then the problem is a certification problem, CERTIFICATE_VERIFY_FAILED. As I feel more comfortable with python, I have done some tests to understand the problem. I have been able to verify that this API requires having all the certification verifications in false, otherwise it never leaves the loop. Here is my Python code (just to show what I'm saying).
import requests
import json
log_params = {'username': 'Joe689', 'password': '15.Job_1825zz'}
headers = {'Content-type': 'application/json'}
url = 'https://xxx.yyy.com:14442/api/external/login'
response = requests.post(url, data=json.dumps(params), headers=self.headers, verify=False)
finalRes = json.loads(response.text)
As I said, this code is just for me to understand the problem because I am a newbie to Dart. Here I found this answer and it seems has the solution but I don't know how to implement it, using my Uri.https estructure (maybe it's not possible).
I tried this, but isn't working:
Map<String, String> requestHeaders = {
'Content-type': 'application/json'
};
final resp = await http.get(url, headers:requestHeaders);
Thank you very much.
Try this:
HttpClient client = new HttpClient();
client.badCertificateCallback = ((X509Certificate cert, String host, int port) => true);
String url ='xxx.yyy.com:14442/api/external/login';
Map map = {
"email" : "Joe689" ,
"password" : "15.Job_1825zz"
};
HttpClientRequest request = await client.getUrl(Uri.parse(url));
request.headers.set('content-type', 'application/json');
request.add(utf8.encode(json.encode(map)));
HttpClientResponse response = await request.close();
String reply = await response.transform(utf8.decoder).join();
print(reply);
Now, check what reply contains. And accordingly, return the data.
When we set idle_in_transaction_session_timeout, the database will terminate connections that are idle for some time.
This works as expected, but I wonder how we should deal with this situation in the aplication code.
We are using pg-promise 10.3.1.
Details of the test:
we set the connection pool size to 1, so that we only have a single session
we set the for the idle-transaction-session-timeout to 2.5 sec:
SET idle_in_transaction_session_timeout TO 2500
now the active transaction will be in state idle in transaction:
see What can cause “idle in transaction” for “BEGIN” statements
now we start a transaction and sleep for 5 seconds
after 2.5sec the database will terminate the session and send an error to the client:
pgp-error error: terminating connection due to idle-in-transaction timeout
after another 2.5sec the transactional code tries to send a query (via the already terminated session), and this fails as expected:
dbIdle failed Error: Client has encountered a connection error and is not queryable
then pg-promise will try to rollback the transaction which will also fail (also expected, I guess)
But now we start a new query and also this query fails with
dbCall failed Client has encountered a connection error and is not queryable
is this expected? I was hoping that pg-promise can somehow remove the "broken" connection from the pool and that we could get a new one
obvously this is not the case, so how should we deal with this situation: i.e. how to recover, so that we can send new queries to the database?
Code example:
import pgPromise, { IMain } from "pg-promise";
import * as dbConfig from "./db-config.json";
import { IConnectionParameters } from "pg-promise/typescript/pg-subset";
const cll = "pg";
console.time(cll);
const pgp: IMain = pgPromise({
query(e) {
console.timeLog(cll,`> ${e.query}`);
},
error(e, ctx) {
console.timeLog(cll,"pgp-error", e);
}
});
const connectParams: IConnectionParameters = {
...dbConfig,
application_name: "pg-test",
max: 1
};
const db = pgp(connectParams);
/**
* #param timeoutMs 0 is no timeout
*/
async function setDbIdleInTxTimeout(timeoutMs: number = 0) {
await db.any("SET idle_in_transaction_session_timeout TO $1;", timeoutMs);
}
async function dbIdle(sleepTimeSec: number) {
console.timeLog(cll, `starting db idle ${sleepTimeSec}`);
const result = await db.tx(async t => {
await new Promise(resolve => setTimeout(resolve, sleepTimeSec * 1000));
return t.one("Select $1 as sleep_sec", sleepTimeSec);
});
console.timeLog(cll, result);
}
async function main() {
await setDbIdleInTxTimeout(2500);
try {
await dbIdle(5);
} catch (e) {
console.timeLog(cll, "dbIdle failed", e);
}
try {
await db.one("Select 1+1 as res");
} catch (e) {
console.timeLog(cll, "dbCall failed", e);
}
}
main().finally(() => {
pgp.end();
});
Console output (removed some useless lines):
"C:\Program Files\nodejs\node.exe" D:\dev_no_backup\pg-promise-tx\dist\index.js
pg: 23.959ms > SET idle_in_transaction_session_timeout TO 2500;
pg: 28.696ms starting db idle 5
pg: 29.705ms > begin
pg: 2531.247ms pgp-error error: terminating connection due to idle-in-transaction timeout
at TCP.onStreamRead (internal/stream_base_commons.js:182:23) {
name: 'error',
severity: 'FATAL',
code: '25P03',
}
pg: 2533.569ms pgp-error Error: Connection terminated unexpectedly
pg: 5031.091ms > Select 5 as sleep_sec
pg: 5031.323ms pgp-error Error: Client has encountered a connection error and is not queryable
pg: 5031.489ms > rollback
pg: 5031.570ms pgp-error Error: Client has encountered a connection error and is not queryable
pg: 5031.953ms dbIdle failed Error: Client has encountered a connection error and is not queryable
pg: 5032.094ms > Select 1+1 as res
pg: 5032.164ms pgp-error Error: Client has encountered a connection error and is not queryable
pg: 5032.303ms dbCall failed Error: Client has encountered a connection error and is not queryable
Process finished with exit code 0
This issue #680 has been fixed in pg-promise 10.3.5
Does Mongoid has any method like ActiveRecord::Base.connected??
I want to check if the connection that's accessible.
We wanted to implement a health check for our running Mongoid client that tells us whether the established connection is still alive. This is what we came up with:
Mongoid.default_client.database_names.present?
Basically it takes your current client and tries to query the databases on its connected server. If this server is down, you will run into a timeout, which you can catch.
My solution:
def check_mongoid_connection
mongoid_config = File.read("#{Rails.root}/config/mongoid.yml")
config = YAML.load(mongoid_config)[Rails.env].symbolize_keys
host, db_name, user_name, password = config[:host], config[:database], config[:username], config[:password]
port = config[:port] || Mongo::Connection::DEFAULT_PORT
db_connection = Mongo::Connection.new(host, port).db(db_name)
db_connection.authenticate(user_name, password) unless (user_name.nil? || password.nil?)
db_connection.collection_names
return { status: :ok }
rescue Exception => e
return { status: :error, data: { message: e.to_s } }
end
snrlx's answer is great.
I use following in my puma config file, FYI:
before_fork do
begin
# load configuration
Mongoid.load!(File.expand_path('../../mongoid.yml', __dir__), :development)
fail('Default client db check failed, is db connective?') unless Mongoid.default_client.database_names.present?
rescue => exception
# raise runtime error
fail("connect to database failed: #{exception.message}")
end
end
One thing to remind is the default server_selection_timeout is 30 seconds, which is too long for db status check at least in development, you can modify this in your mongoid.yml.