Too many tasks running at the same time on Airflow, some tasks get heartbeat exception and they get stuck. Sometimes even if the tasks are completed, the airflow can't get the completed log.
our setup:
Airflow v2.3.4 in Docker
PostgreSQL 11.6 on x86_64-pc-linux-gnu
example log:
[2022-10-19, 09:01:15 +03] {base_job.py:229} ERROR - LocalTaskJob heartbeat got an exception
....
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "xxx" (xxx), port xxx failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
....
return self.dbapi.connect(*cargs, **cparams)
File "/home/airflow/.local/lib/python3.7/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "xxx" (xxx), port xxx failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
(Background on this error at: https://sqlalche.me/e/14/e3q8)
Related
connecting to: mongodb://ac-impfze5-shard-00-01.wygb8zb.mongodb.net:27017,ac-impfze5-shard-00-02.wygb8zb.mongodb.net:27017,ac-impfze5-shard-00-00.wygb8zb.mongodb.net:27017/myFirstDatabase?authSource=admin&compressors=disabled&gssapiServiceName=mongodb&replicaSet=atlas-x9mb3l-shard-0&ssl=true
{"t":{"$date":"2023-02-10T20:59:39.312Z"},"s":"I", "c":"NETWORK", "id":5490002, "ctx":"thread1","msg":"Started a new thread for the timer service"}
*** You have failed to connect to a MongoDB Atlas cluster. Please ensure that your IP whitelist allows connections from your network.
Error: can't connect to new replica set master [ac-impfze5-shard-00-01.wygb8zb.mongodb.net:27017], err: AuthenticationFailed: bad auth : authentication failed :
connect#src/mongo/shell/mongo.js:374:17
#(connect):2:6
exception: connect failed
exiting with code 1
I made a script for telegram that changes the avatar every minute when it starts, the following error appears
Attempt 1 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Attempt 2 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Attempt 3 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Attempt 4 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Attempt 5 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Attempt 6 at connecting failed: ConnectionRefusedError: [Errno 111] Connect call failed ('149.154.167.51', 443)
Traceback (most recent call last):
File "/home/ainurfast/clocks/main.py", line 13, in
client.start()
File "/home/ainurfast/.local/lib/python3.9/site-packages/telethon/client/auth.py", line 133, in start
else self.loop.run_until_complete(coro)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/ainurfast/.local/lib/python3.9/site-packages/telethon/client/auth.py", line 140, in _start
await self.connect()
File "/home/ainurfast/.local/lib/python3.9/site-packages/telethon/client/telegrambaseclient.py", line 525, in connect
if not await self._sender.connect(self._connection(
File "/home/ainurfast/.local/lib/python3.9/site-packages/telethon/network/mtprotosender.py", line 127, in connect
await self._connect()
File "/home/ainurfast/.local/lib/python3.9/site-packages/telethon/network/mtprotosender.py", line 253, in _connect
raise ConnectionError('Connection to Telegram failed {} time(s)'.format(self._retries))
ConnectionError: Connection to Telegram failed 5 time(s)
The mtproto protocol for telegram does not work from a free account on PythonAnywhere. You can use the HTTP protocol or upgrade your account.
Created an app with flask and connected with postgresql via psycopg2 as connector. Created Dockerfile and while building it shows error of
'''Traceback (most recent call last):
File "init_db.py", line 5, in <module>
conn = psycopg2.connect(
File "/usr/local/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: connection to server at "192.168.0.167", port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?'''
have mentioned host, port, database, username and password in it. Still there an issue. Have also tried changing ports but still the error persists.
MongoDB shell version v4.4.6
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: NetworkTimeout: Error connecting to 127.0.0.1:27017 :: caused by :: Socket operation timed out :
connect#src/mongo/shell/mongo.js:374:17
#(connect):2:6
exception: connect failed
exiting with code 1
Everytime when I run mongo command , it says error connecting to 127.0.0.1:27017 caused by socket opeartion timed out.
Tried following many instructions given by peers , but it didn't work.
Hope I get rid of this by your instructions.
Thanks!
Reason: mongodb service is not running
Run mongodb service method:
Open the terminal and execute mongod -dbpath "D:\Program Files\MongoDB\data\db"
("D:\Program Files\MongoDB\data\db" is the directory you created to store data)
Open another terminal and execute mongo
I am upgrading a mongo sharded cluster, and am in the first step of replacing the mongos process. Can I just kill this process or is there a safer way to shut it down before replacing it?
It is the answer. But may be you misunderstood. You cannot shutdown mongos from command line, but you have to logon to the shell. Here is the result I am trying on my own:
$ mongo --port 27077 (please replace with your own port on mongos instance)
**mongos**> db.shutdownServer({timeoutSecs:30})
shutdown command only works with the admin database; try 'use admin'
mongos> use admin
switched to db admin
mongos> db.shutdownServer({timeoutSecs:30})
2015-02-27T12:53:54.408+0800 DBClientCursor::init call() failed
**server should be down...**
2015-02-27T12:53:54.410+0800 trying reconnect to 127.0.0.1:27077 (127.0.0.1) failed
2015-02-27T12:53:54.410+0800 warning: Failed to connect to 127.0.0.1:27077, reason: errno:111 Connection refused
2015-02-27T12:53:54.410+0800 reconnect 127.0.0.1:27077 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27077 (127.0.0.1), connection attempt failed
2015-02-27T12:53:54.413+0800 trying reconnect to 127.0.0.1:27077 (127.0.0.1) failed
2015-02-27T12:53:54.413+0800 warning: Failed to connect to 127.0.0.1:27077, reason: errno:111 Connection refused
2015-02-27T12:53:54.413+0800 reconnect 127.0.0.1:27077 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27077 (127.0.0.1), connection attempt failed
>
Try to run "top" or "htop" to display all processes running on your computer. You shall see the "mongos" process has gone. Read the log file. Here is the last two lines of my log:
2015-02-27T12:53:54.406+0800 [conn1] terminating, shutdown command received
2015-02-27T12:53:54.406+0800 [conn1] dbexit: shutdown called rc:0 shutdown called
And here is the link of my testing script:
https://github.com/babycaseny/QuickStart/blob/master/StartShard.sh
Note that you have to replace the "localhost" in the mongod/mongos command with the hostname of your computer, or you will not be able to config your shards.
See also this one:
https://groups.google.com/forum/#!topic/mongodb-user/TQLlRI6HG1M
In case you need a command line to do the work, here is one:
mongo admin --port portnumber --eval "db.shutdownServer()"
Notice that you have to run the command in localhost.