bulbs rexster system error - bulbs

I am using Rexster 2.4.0 and Bulbs 0.3.14
With Rexster running on localhost, I am trying to get familiar with Bulbs, yet when trying:
>>>from bulbs.rexster import Graph
>>>g = Graph()
Traceback (most recent call last):
File "", line 1, in
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/graph.py", line 54, in init
super(Graph, self).init(config)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/base/graph.py", line 58, in init
self.vertices = self.build_proxy(Vertex)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/base/graph.py", line 124, in build_proxy
return self.factory.build_element_proxy(element_class, index_class)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/factory.py", line 19, in build_element_proxy
primary_index = self.get_index(element_class,index_class,index_name)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/factory.py", line 27, in get_index
index = index_proxy.get_or_create(index_name)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/index.py", line 80, in get_or_create
resp = self.client.get_or_create_vertex_index(index_name, index_params)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/client.py", line 660, in get_or_create_vertex_index
resp = self.gremlin(script, params)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/client.py", line 354, in gremlin
return self.request.post(gremlin_path,params)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rest.py", line 128, in post
return self.request(POST, path, params)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rest.py", line 183, in request
return self.response_class(http_resp, self.config)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/client.py", line 198, in init
self.handle_response(response)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rexster/client.py", line 222, in handle_response
response_handler(http_resp)
File "/Users/lolmac/anaconda/lib/python2.7/site-packages/bulbs/rest.py", line 50, in server_error
raise SystemError(http_resp)
SystemError: ({'status': '500', 'transfer-encoding': 'chunked', 'server': 'grizzly/2.2.16', 'connection': 'close', 'date': 'Mon, 14 Oct 2013 19:43:45 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'}, '{"message":"","error":"javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of method: groovy.lang.MissingMethodException.stopTransaction() is applicable for argument types: () values: []","api":{"description":"evaluate an ad-hoc Gremlin script for a graph.","parameters":{"rexster.returnKeys":"an array of element property keys to return (default is to return all element properties)","rexster.showTypes":"displays the properties of the elements with their native data type (default is false)","load":"a list of \'stored procedures\' to execute prior to the \'script\' (if \'script\' is not specified then the last script in this argument will return the values","rexster.offset.end":"end index for a paged set of data to be returned","rexster.offset.start":"start index for a paged set of data to be returned","params":"a map of parameters to bind to the script engine","language":"the gremlin language flavor to use (default to groovy)","script":"the Gremlin script to be evaluated"}},"success":false}')
this is an old post: https://groups.google.com/forum/#!msg/gremlin-users/s7Ag1tjbxLs/kaBOSyed_9kJ, but it seems other folks encountered the same problem. Still, i was not able to find any documentation that indicates what is wrong or what to change in the default configuration.
grateful to any docs/discussions or ideas that can provide a hint.

You are getting this error because Bulbs 0.3.14 hadn't been updated to TinkerPop 2.4, but that's fixed now -- I just updated Bulbs-Rexster to TinkerPop 2.5.0-SNAPSHOT and pushed the Bulbs 0.3.15 to both GitHub and PyPi. All tests pass. Please let me know if that fixes it for you.

Related

Odoo 16: Playing with translations and Boom ! No longer can access my custom model data in my server

After successfully adding a pot file to my new i18n folder in my local machine, as well as setting "translate=True" in a couple of fields in my carddecks module, and verifying that in localhost I could acess my model data, I decided to update my server.
But when I try to access my model data I get the following error:
LINE 1: ..."write_date", COALESCE("carddecks_card"."cardText"->>'pt_PT'...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
Anyone might know what may be causing this?
source code for the module can be found at https://github.com/diogocsc/carddecks
Full error:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1579, in _serve_db
return service_model.retrying(self._serve_ir_http, self.env)
File "/usr/lib/python3/dist-packages/odoo/service/model.py", line 134, in retrying
result = func()
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1608, in _serve_ir_http
response = self.dispatcher.dispatch(rule.endpoint, args)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 1805, in dispatch
result = self.request.registry['ir.http']._dispatch(endpoint)
File "/usr/lib/python3/dist-packages/odoo/addons/website/models/ir_http.py", line 235, in _dispatch
response = super()._dispatch(endpoint)
File "/usr/lib/python3/dist-packages/odoo/addons/base/models/ir_http.py", line 144, in _dispatch
result = endpoint(**request.params)
File "/usr/lib/python3/dist-packages/odoo/http.py", line 698, in route_wrapper
result = endpoint(self, *args, **params_ok)
File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/dataset.py", line 42, in call_kw
return self._call_kw(model, method, args, kwargs)
File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/dataset.py", line 33, in _call_kw
return call_kw(request.env[model], method, args, kwargs)
File "/usr/lib/python3/dist-packages/odoo/api.py", line 457, in call_kw
result = _call_kw_model(method, model, args, kwargs)
File "/usr/lib/python3/dist-packages/odoo/api.py", line 430, in _call_kw_model
result = method(recs, *args, **kwargs)
File "/usr/lib/python3/dist-packages/odoo/addons/web/models/models.py", line 62, in web_search_read
records = self.search_read(domain, fields, offset=offset, limit=limit, order=order)
File "/usr/lib/python3/dist-packages/odoo/models.py", line 4968, in search_read
result = records.read(fields, **read_kwargs)
File "/usr/lib/python3/dist-packages/odoo/models.py", line 2992, in read
self._read(stored_fields)
File "/usr/lib/python3/dist-packages/odoo/models.py", line 3235, in _read
cr.execute(query_str, params + [sub_ids])
File "/usr/lib/python3/dist-packages/odoo/sql_db.py", line 315, in execute
res = self._obj.execute(query, params)
psycopg2.errors.UndefinedFunction: operator does not exist: character varying ->> unknown
LINE 1: ..."write_date", COALESCE("carddecks_card"."cardText"->>'pt_PT'...
^
HINT: No operator matches the given name and argument types. You might need to add explicit type casts.
The above server error caused the following client error:
OwlError: The following error occurred in onWillStart: "Odoo Server Error"
at wrapError (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1445:77)
at onWillStart (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1451:117)
at useModel (https://www.relationalgames.com/web/assets/505-11285c6/web.assets_backend.min.js:4709:1)
at ListController.setup (https://www.relationalgames.com/web/assets/505-11285c6/web.assets_backend.min.js:4430:645)
at new ComponentNode (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1407:136)
at https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1929:6
at View.slot1 (eval at compile (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1892:370), <anonymous>:15:36)
at callSlot (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1508:25)
at WithSearch.template (eval at compile (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1892:370), <anonymous>:8:12)
at Fiber._render (https://www.relationalgames.com/web/assets/504-41b52e3/web.assets_common.min.js:1336:96)
Caused by: RPC_ERROR: Odoo Server Error
at makeErrorFromResponse (https://www.relationalgames.com/web/assets/505-11285c6/web.assets_backend.min.js:967:163)
at XMLHttpRequest.<anonymous> (https://www.relationalgames.com/web/assets/505-11285c6/web.assets_backend.min.js:974:13)
Found the answer for this. In my localhost I was updating my module. In my server i was just running docker-compose up. Not updating the module in the db.
The solution was going to settings -> apps search by cardecks and upgrade it.

Long URL (including a key) causes unicode idna codec decoding error whilst using RedisChannelLayer in Django Channels

I've encountered a problem deploying Django Channels on Heroku, whilst using a RedisChannelLayer.
I get a UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long) during connection (full traceback below).
That seems to be a problem related to one of the labels in the host address being too long as shown in this python issue.
I printed some information out of my consumer, and also wrapped python's socket.getaddrinfo module to display host and connection information.
This related post has has the same problem connecting to shopify, not a redis instance, where they got around it by placing credentials into the request header. But I don't have control over channels_redis or asyncio.
Any clues?
Properties of the Django Channels Consumer:
.groups []
.channel_layer RedisChannelLayer(hosts=[{'address': ('h:alongkeycomprisingof65charsintotalxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx#ec2-18-202-152-61.eu-west-1.compute.amazonaws.com', '9759')}])
.channel_name specific.OSfTzyqY!pdvgHnaCxWiv
.room_name 8e3d3083-8bb1-4d85-89d3-4496d9b9e946
.room_group_name twined_8e3d3083-8bb1-4d85-89d3-4496d9b9e946
Full Traceback:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.6/site-packages/channels/sessions.py", line 183, in __call__
return await self.inner(receive, self.send)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/middleware.py", line 41, in coroutine_call
await inner_instance(receive, send)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 59, in __call__
[receive, self.channel_receive], self.dispatch
File "/app/.heroku/python/lib/python3.6/site-packages/channels/utils.py", line 51, in await_many_dispatch
await dispatch(result)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/consumer.py", line 73, in dispatch
await handler(message)
File "/app/.heroku/python/lib/python3.6/site-packages/channels/generic/websocket.py", line 175, in websocket_connect
await self.connect()
File "/app/backend/pink/consumers.py", line 24, in connect
await self.channel_layer.group_add(self.room_group_name, self.channel_name)
File "/app/.heroku/python/lib/python3.6/site-packages/channels_redis/core.py", line 589, in group_add
async with self.connection(self.consistent_hash(group)) as connection:
File "/app/.heroku/python/lib/python3.6/site-packages/channels_redis/core.py", line 835, in __aenter__
self.conn = await self.pool.pop()
File "/app/.heroku/python/lib/python3.6/site-packages/channels_redis/core.py", line 73, in pop
conns.append(await aioredis.create_redis(**self.host, loop=loop))
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/commands/__init__.py", line 175, in create_redis
loop=loop)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/connection.py", line 113, in create_connection
timeout)
File "/app/.heroku/python/lib/python3.6/asyncio/tasks.py", line 339, in wait_for
return (yield from fut)
File "/app/.heroku/python/lib/python3.6/site-packages/aioredis/stream.py", line 24, in open_connection
lambda: protocol, host, port, **kwds)
File "/app/.heroku/python/lib/python3.6/asyncio/base_events.py", line 750, in create_connection
infos = f1.result()
File "/app/.heroku/python/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "./backend/amy/asgi.py", line 69, in mygetaddrinfo
for res in socket._socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
I've been able to work around by establishing my RedisChannelLayer using a dict of arguments to create_connection (as mentioned here, instead of providing a really long host name.
By manually parsing the password out of the REDIS_URL variable that heroku provides me, and rebuilding a host uri without it, I can add that password as a separate field to the create_connection dict, keeping the host string length below 64 characters.
I do this in my settings.py file which now looks like:
def parse_redis_url(url):
""" parses a redis url into component parts, stripping password from the host.
Long keys in the url result in parsing errors, since labels within a hostname cannot exceed 64 characters under
idna rules.
In that event, we remove the key/password so that it can be passed separately to the RedisChannelLayer.
Heroku REDIS_URL does not include the DB number, so we allow for a default value of '0'
"""
parsed = urlparse(url)
parts = parsed.netloc.split(':')
host = ':'.join(parts[0:-1])
port = parts[-1]
path = parsed.path.split('/')[1:]
db = int(path[0]) if len(path) >= 1 else 0
user, password = (None, None)
if '#' in host:
creds, host = host.split('#')
user, password = creds.split(':')
host = f'{user}#{host}'
return host, port, user, password, db
REDIS_URL = env('REDIS_URL', default='redis://localhost:6379')
REDIS_HOST, REDIS_PORT, REDIS_USER, REDIS_PASSWORD, REDIS_DB = parse_redis_url(REDIS_URL)
# DJANGO CHANNELS
CHANNEL_LAYERS = {
'default': {
'BACKEND': 'channels_redis.core.RedisChannelLayer',
'CONFIG': {
"hosts": [{
'address': f'redis://{REDIS_HOST}:{REDIS_PORT}',
'db': REDIS_DB,
'password': REDIS_PASSWORD,
}],
},
},
}

Unable to set credentials with Google texttoaudio API

I was able to set to set the credentials for Google's translation but with texttospeech I'm having a lot of trouble. First, I couldn't get the credentials as a json file, but got it as just a string. I haven't been able to find anyone else whose credentials are strings every one else has a json file. I converted the string to a json file but I don't think that is helping because it seems that they json object has to be a dictionary. In any case when I try this:
from google.oauth2 import service_account
key1 = 'key.json'
credentials = service_account.Credentials.from_service_account_file(key1)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-13-c7d030662a35>", line 1, in <module>
credentials = service_account.Credentials.from_service_account_file(key1)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/oauth2/service_account.py", line 209, in from_service_account_file
filename, require=['client_email', 'token_uri'])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_service_account_info.py", line 73, in from_filename
return data, from_dict(data, require=require)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_service_account_info.py", line 46, in from_dict
missing = keys_needed.difference(six.iterkeys(data))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/six.py", line 575, in iterkeys
return iter(d.keys(**kw))
AttributeError: 'str' object has no attribute 'keys'
When I try this code the following happens:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'key.json'
client = texttospeech.TextToSpeechClient()
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-14-d30e5cd41087>", line 2, in <module>
client = texttospeech.TextToSpeechClient()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/texttospeech_v1/gapic/text_to_speech_client.py", line 159, in __init__
address=api_endpoint, channel=channel, credentials=credentials
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/texttospeech_v1/gapic/transports/text_to_speech_grpc_transport.py", line 61, in __init__
channel = self.create_channel(address=address, credentials=credentials)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/cloud/texttospeech_v1/gapic/transports/text_to_speech_grpc_transport.py", line 91, in create_channel
address, credentials=credentials, scopes=cls._OAUTH_SCOPES, **kwargs
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 177, in create_channel
credentials, _ = google.auth.default(scopes=scopes)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_default.py", line 305, in default
credentials, project_id = checker()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_default.py", line 165, in _get_explicit_environ_credentials
os.environ[environment_vars.CREDENTIALS])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/auth/_default.py", line 102, in _load_credentials_from_file
credential_type = info.get('type')
AttributeError: 'str' object has no attribute 'get'
I think this is because my json object is not a dict but a string. But the key that Google gave me was a string and not a json file, so I really don't know what to do here. Plus their documentation is too hard to understand.
When you say you have a string, are you referring to the Key ID? You will still need the json file associated with that key.
To create a new json file, go to Google Cloud Console -> IAM & Admin -> Service Accounts. Select one of your service accounts and click "Create Key" which will download the key as a json file. It can only be downloaded once.

download file from mongo gridfs with python

I have uploaded files to the mongo.But when I want to download from mongo by httpresonse on the web browser,that did not work.
Here is the views.py:
if filename is not None:
file_ = db.fs.files.find_one({
'filename':filename
})
file_id = file_['_id']
wrapper = fs.get(file_id).read()
response = StreamingHttpResponse(FileWrapper(wrapper),content_type=file_['contentType'])
response['Content-Disposition'] = 'attachment; filename=%s' % str(filename)
response['Content-Length'] = file_['length']
return response
I got this error:
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 126, in finish_response
for data in self.result:
File "/usr/local/lib/python2.7/dist-packages/django/utils/six.py", line 473, in next
return type(self).__next__(self)
File "/usr/local/lib/python2.7/dist-packages/django/http/response.py", line 292, in __next__
return self.make_bytes(next(self._iterator))
File "/usr/lib/python2.7/wsgiref/util.py", line 30, in next
data = self.filelike.read(self.blksize)
AttributeError: 'str' object has no attribute 'read'
But when I change the StreamingHttpResponse to HttpResponse,the error is as follow:
[30/Jul/2014 17:29:43] "GET /download/cs101/ HTTP/1.1" 200 664
/usr/lib/python2.7/wsgiref/handlers.py:126: DeprecationWarning:
Creating streaming responses with `HttpResponse` is deprecated.
Use `StreamingHttpResponse`instead if you need the streaming behavior.
for data in self.result:
Traceback (most recent call last):
File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/lib/python2.7/wsgiref/handlers.py", line 126, in finish_response
for data in self.result:
File "/usr/local/lib/python2.7/dist-packages/django/utils/six.py", line 473, in next
return type(self).__next__(self)
File "/usr/local/lib/python2.7/dist-packages/django/http/response.py", line 292, in __next__
return self.make_bytes(next(self._iterator))
File "/usr/lib/python2.7/wsgiref/util.py", line 30, in next
data = self.filelike.read(self.blksize)
AttributeError: 'str' object has no attribute 'read'
Thanks in advance!
You're calling the read method in:
wrapper = fs.get(file_id).read()
So you're getting a str (assuming Python 2, if 3, you're getting bytes). FileWrapper needs file like object, which of course str is not one.
Try to use:
wrapper = fs.get(file_id)
This will return file like object.
OTOH, pymongo's .get() returns a GridOut instance, which already supports iteration, so why not try something like:
wrapper = fs.get(file_id)
response = StreamingHttpResponse(wrapper, content_type=file_['contentType'])

Issue with multiline TokenInputTransformer in IPython extension

I am trying to use a TokenInputTransformer within an IPython extension module, but it seems that there is something wrong with the standard implementation of token transformers with multiline input. Consider the following minimal extension:
from IPython.core.inputtransformer import TokenInputTransformer
#TokenInputTransformer.wrap
def test_transformer(tokens):
return tokens
def load_ipython_extension(ip):
for s in (ip.input_splitter, ip.input_transformer_manager):
s.python_line_transforms.extend([test_transformer()])
print "Test activated"
When I load the extension in IPython 1.1.0 I get a non-handled exception with multiline input:
In [1]: %load_ext test
Test activated
In [2]: abs(
...: 2
...: )
Traceback (most recent call last):
File "/Applications/anaconda/bin/ipython", line 6, in <module>
sys.exit(start_ipython())
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 118, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 545, in launch_instance
app.start()
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 362, in start
self.shell.mainloop()
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 436, in mainloop
self.interact(display_banner=display_banner)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 548, in interact
self.input_splitter.push(line)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputsplitter.py", line 620, in push
out = self.push_line(line)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputsplitter.py", line 655, in push_line
line = transformer.push(line)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputtransformer.py", line 152, in push
return self.output(tokens)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputtransformer.py", line 157, in output
return untokenize(self.func(tokens)).rstrip('\n')
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 276, in untokenize
return ut.untokenize(iterable)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 214, in untokenize
self.add_whitespace(start)
File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 199, in add_whitespace
assert row >= self.prev_row
AssertionError
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev#scipy.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True
Am I doing something wrong or is it really an IPython bug?
It is really an IPython bug, I think. Specifically, the way we handle tokenize fails when an expression involving brackets (()[]{}) is spread over more than one line. I'm trying to work out what we can do about this.
Kinda late answer but, I was trying to use it in my own extension and just had the same problem. I've solved it by simply removing NL from the list (it's not the same as NEWLINE token which end statement), NL token only appears inside [], (), {} so it should be safely removable.
from tokenize import NL
#TokenInputTransformer.wrap
def mat_transformer(tokens):
tokens = list(filter(lambda t: t.type != NL, tokens))
return tokens
If you are looking for full example, I've posted my goofy code there: https://github.com/Quinzel/pymat