How make a correct query from pymongo - mongodb

I am using pymongo to subset a collection in mongodb, I am having problems with the syntax of the query, this is my query:
import pymongo
client=pymongo.MongoClient("localhost",27017)
db=client("publications")
collec=db("zikaVirus2")
pipeline=db.collec.aggregate([
{"$match":{"fullText":{"$exists": "true"}}},
{"$out": "pub_fulltext"}
])
db.pub_fulltext.aggregate(pipeline)
I am getting this error:
Traceback (most recent call last):
File "genColFulltext.py", line 3, in <module>
db=client("publications")
TypeError: 'MongoClient' object is not callable
UPDATE 1
If I do
import pymongo
client=pymongo.MongoClient("localhost",27017)
db=client.publications
collec=db.zikaVirus2
pipeline=db.collec.aggregate([
{"$match":{"fullText":{"$exists": "true"}}},
{"$out": "pub_fulltext"}
])
db.pub_fulltext.aggregate(pipeline)
I get
Traceback (most recent call last):
File "genColFulltext.py", line 11, in <module>
db.pub_fulltext.aggregate(pipeline)
File "/home/danielo/.local/lib/python3.5/site-packages/pymongo /collection.py", line 2397, in aggregate
**kwargs)
File "/home/danielo/.local/lib/python3.5/site-packages/pymongo/collection.py", line 2242, in _aggregate
common.validate_list('pipeline', pipeline)
File "/home/danielo/.local/lib/python3.5/site-packages/pymongo/common.py", line 428, in validate_list
raise TypeError("%s must be a list" % (option,))
TypeError: pipeline must be a list
UPDATE 2
This code runs without problems, but it make nothing of his purpose, the collection pub_fulltext have anything, when it should have data.
import pymongo
client=pymongo.MongoClient("localhost",27017)
db=client.publications
collec=db.zikaVirus2
pipeline=db.collec.aggregate([
{"$match":{"fullText":{"$exists": "true"}}},
{"$out": "pub_fulltext"}
])
for item in pipeline:
db.pub_fulltext.aggregate(item)
Then I try this:
import pymongo
client=pymongo.MongoClient("localhost",27017)
db=client.publications
collec=db.zikaVirus2
pipeline=[db.collec.aggregate([
{"$match":{"fullText":{"$exists": "true"}}},
{"$out": "pub_fulltext"}
])]
for item in db.collection.aggregate(pipeline):
db.collection.aggregate(item)
Getting this error:
Traceback (most recent call last):
File "genColFulltext.py", line 10, in <module>
for item in collection.aggregate(pipeline):
Traceback (most recent call last): File "genColFulltext.py", line 10, in <module>
for item in db.collection.aggregate(pipeline): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/collection.py", line 2397, in aggregate
**kwargs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/collection.py", line 2304, in _aggregate
client=self.__database.client) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/pool.py", line 584, in command
self._raise_connection_failure(error) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/pool.py", line 745, in _raise_connection_failure
raise error File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/network.py", line 114, in command
codec_options, ctx=compression_ctx) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymongo/message.py", line 679, in _op_msg
flags, command, identifier, docs, check_keys, opts) bson.errors.InvalidDocument: Cannot encode object: <pymongo.command_cursor.CommandCursor object at 0x1008e4198>

Related

CWL: Creating input yaml - issue with ruamel comments.py

I am using cwltool version 3.0.20201203173111 that came with pip install cwl-runner. This might be issue with ruamel or with my workflow but here is run command and traceback:
cwltool --make-template --debug tumor_only_exome.cwl > hiv_input.yml
traceback:
Traceback (most recent call last):
File "/home/coyote/anaconda3/envs/cwl/lib/python3.6/site-packages/cwltool/main.py", line 991, in main
make_template(tool)
File "/home/coyote/anaconda3/envs/cwl/lib/python3.6/site-packages/cwltool/main.py", line 744, in make_template
generate_input_template(tool),
File "/home/coyote/anaconda3/envs/cwl/lib/python3.6/site-packages/cwltool/main.py", line 301, in generate_input_template
template.insert(0, name, value, comment)
File "/home/coyote/anaconda3/envs/cwl/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 722, in insert
self.yaml_add_eol_comment(comment, key=key)
File "/home/coyote/anaconda3/envs/cwl/lib/python3.6/site-packages/ruamel/yaml/comments.py", line 295, in yaml_add_eol_comment
if comment[0] != '#':
IndexError: string index out of range

Error while executing basic code in Locust

from locust import Locust, TaskSet
def login(l):
print("I am logged In")
def logout(m):
print("I am logged Out")
class UserBehaviour(TaskSet):
task=[login,logout]
class User(Locust):
task_set = UserBehaviour
Error Message---
(venv) C:\pythnprojects\LearnLocustProject\venv\locust_test>locust -f firstlocust.py
[2020-03-11 00:38:57,259] DELLXPS/INFO/locust.main: Starting web monitor at *:8089
[2020-03-11 00:38:57,259] DELLXPS/INFO/locust.main: Starting Locust 0.11.0
[2020-03-11 00:39:05,581] DELLXPS/INFO/locust.runners: Hatching and swarming 1 clients at the rate 1 clients/s...
[2020-03-11 00:39:05,585] DELLXPS/ERROR/stderr: Traceback (most recent call last):
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 358, in run
self.schedule_task(self.get_next_task())
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 419, in get_next_task
return random.choice(self.tasks)
File "C:\DOWNLOADS\lib\random.py", line 290, in choice
raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence
[2020-03-11 00:39:06,582] DELLXPS/INFO/locust.runners: All locusts hatched: User: 1
[2020-03-11 00:39:06,591] DELLXPS/ERROR/stderr: Traceback (most recent call last):
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 358, in run
self.schedule_task(self.get_next_task())
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 419, in get_next_task
return random.choice(self.tasks)
File "C:\DOWNLOADS\lib\random.py", line 290, in choice
raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence
[2020-03-11 00:39:07,597] DELLXPS/ERROR/stderr: Traceback (most recent call last):
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 358, in run
self.schedule_task(self.get_next_task())
File "c:\pythnprojects\learnlocustproject\venv\lib\site-packages\locust\core.py", line 419, in get_next_task
return random.choice(self.tasks)
File "C:\DOWNLOADS\lib\random.py", line 290, in choice
raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence
It looks like you've misspelled tasks (it currently seems to say task).

Python errors with telnetlib

I have multiple errors with python's library telnetlib.
When I use it from the repl, here is the code I enter:
import telnetlib
telnetlib.Telnet("<an_ip>", <a_port>) as tn:
write("SYST:BEEP")
It retrurns the following error:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "/usr/local/lib/python3.6/telnetlib.py", line 287, in write
if IAC in buffer:
TypeError: 'in <string>' requires string as left operand, not bytes
Now, something which is strange is when I execute the following script:
import telnetlib
def beep(tn):
"""Beep the pulse generator."""
tn.write("SYST:BEEP\n")
# Main for test purpose
if __name__ == "__main__":
with telnetlib.Telnet("<ip>", <port>) as tn:
beep(tn)
I have the following error and I have it each time I import telnetlib...
raceback (most recent call last):
File "test.py", line 1, in <module>
import telnetlib
File "/usr/lib/python3.7/telnetlib.py", line 37, in <module>
import socket
File "/usr/lib/python3.7/socket.py", line 53, in <module>
from enum import IntEnum, IntFlag
File "/usr/lib/python3.7/enum.py", line 2, in <module>
from types import MappingProxyType, DynamicClassAttribute
File "/home/vpecatte/dev/soft/soft/types.py", line 1, in <module>
from typing import *
File "/usr/lib/python3.7/typing.py", line 28, in <module>
import re as stdlib_re # Avoid confusion with the re we export.
File "/usr/lib/python3.7/re.py", line 143, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
if not enabled():
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
import re
File "/usr/lib/python3.7/re.py", line 122, in <module>
import enum
File "/usr/lib/python3.7/enum.py", line 2, in <module>
from types import MappingProxyType, DynamicClassAttribute
File "/home/vpecatte/dev/soft/soft/types.py", line 1, in <module>
from typing import *
File "/usr/lib/python3.7/typing.py", line 31, in <module>
from types import WrapperDescriptorType, MethodWrapperType, MethodDescriptorType
ImportError: cannot import name 'WrapperDescriptorType' from 'types' (/home/vpecatte/dev/soft/soft/types.py)
Original exception was:
Traceback (most recent call last):
File "pulse_generator.py", line 1, in <module>
import telnetlib
File "/usr/lib/python3.7/telnetlib.py", line 37, in <module>
import socket
File "/usr/lib/python3.7/socket.py", line 53, in <module>
from enum import IntEnum, IntFlag
File "/usr/lib/python3.7/enum.py", line 2, in <module>
from types import MappingProxyType, DynamicClassAttribute
File "/home/vpecatte/dev/soft/soft/types.py", line 1, in <module>
from typing import *
File "/usr/lib/python3.7/typing.py", line 28, in <module>
import re as stdlib_re # Avoid confusion with the re we export.
File "/usr/lib/python3.7/re.py", line 143, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
I use python v3.7.5. I replaced correct informations by , .
Thank you for your help.
Looks like you have installed your own types module in /home/vpecatte/dev/soft/soft/types.py which conflicts with the Python built-in types module. The best solution is probably to rename that types module.

KeyError: 'found' on Elastic2DocManager when syncing data from MongoDB

From MongoDB to Elastic Search(5.6.5), I sync the database with Mongo-Connector using Elastic2DocManager:
mongo-connector -m localhost:27017 -t localhost:9200 -d elastic2_doc_manager
After seeing some update on docs.deleted of mongodb_meta on Elastic Search:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open mongodb_meta 3wd6OjTT6tD3f6ZGezZw 5 1 1337173 8372 192.9mb 192.9mb
mongo-connector stops working with below error:
2018-07-11 07:16:41,977 [WARNING] elasticsearch:97 - POST http://localhost:9200/_bulk [status:N/A request:10.003s]
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 387, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 383, in _make_request
httplib_response = conn.getresponse()
File "c:\programdata\anaconda3\lib\http\client.py", line 1331, in getresponse
response.begin()
File "c:\programdata\anaconda3\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "c:\programdata\anaconda3\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "c:\programdata\anaconda3\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
...
Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 150, in run
self._docman.send_buffered_operations()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 482, in send_buffered_operations
action_buffer = self.BulkBuffer.get_buffer()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 696, in get_buffer
self.update_sources()
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\util.py", line 35, in wrapped
return f(*args, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\mongo_connector\doc_managers\elastic2_doc_manager.py", line 628, in update_sources
if ES_doc['found']:
KeyError: 'found'
What is the reason for this error?
Make sure the versions are compatible.
The required python packages are installed, this is my requirements.txt look like.
astroid==1.6.5
autopep8==1.3.5
certifi==2018.8.24
elasticsearch==6.3.1
elasticsearch-dsl==6.2.1
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
pycodestyle==2.4.0
pylint==1.9.2
pymongo==3.7.1
rope==0.11.0
six==1.11.0
urllib3==1.23
wrapt==1.10.11

django pipeline Error: The file build/fonts/glyphicons-halflings-regular.eot' could not be found with <pipeline.storage.PipelineCachedStorage object

I am using django-pipeline, On running "sudo python manage.py collectstatic"
Getting this error:
Traceback (most recent call last):
File "manage.py", line 10, in <module> execute_from_command_line(sys.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line utility.execute()
File "/Users/office/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 390, in run_from_argv self.execute(*args, **cmd_options)
File "/Users/office/lib/python2.7/site-packages/django/core/management/base.py", line 441, in execute output = self.handle(*args, **options)
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle collected = self.collect()
File "/Users/office/lib/python2.7/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 120, in collect raise processed ValueError: The file 'bower_components/eonasdan-bootstrap-datetimepicker/build/fonts/glyphicons-halflings-regular.eot' could not be found with <pipeline.storage.PipelineCachedStorage object at 0x10d274e10>.
Any idea how to fix this ?
Using django-pipeline-forgiving(https://pypi.python.org/pypi/django-pipeline-forgiving) solved the issue.
pip install django-pipeline-forgiving
Set in your settings.py: STATICFILES_STORAGE = 'django_pipeline_forgiving.storages.PipelineForgivingStorage'