Load RDF file to OrientDB using Apache tinkerpop gremlin - orientdb

I'm using OrientDB with gremlin server (https://s3.us-east-2.amazonaws.com/orientdb3/releases/3.1.1/orientdb-tp3-3.1.1.zip) and my requirement is to load a RDF file to orientDB. I'm using rdflib python library to act as a middleware to parse and load the file but it fails. Please see the error logs below:
Invalid OpProcessor requested [null]
org.apache.tinkerpop.gremlin.server.op.OpProcessorException: Invalid OpProcessor requested [null]
at org.apache.tinkerpop.gremlin.server.handler.OpSelectorHandler.decode(OpSelectorHandler.java:85)
at org.apache.tinkerpop.gremlin.server.handler.OpSelectorHandler.decode(OpSelectorHandler.java:48)
Also, when the server starts i get a few warnings about some classes not able to load. i suspect they might be responsible for the failure.
020-07-25 15:47:35:541 WARNI The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0 serialization class is deprecated. [AbstractChannelizer]
2020-07-25 15:47:35:604 WARNI The org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0 serialization class is deprecated. [AbstractChannelizer]
2020-07-25 15:47:35:626 INFO Configured application/vnd.gremlin-v3.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0 [AbstractChannelizer]
2020-07-25 15:47:35:626 WARNI The org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0 serialization class is deprecated. [AbstractChannelizer]
2020-07-25 15:47:35:629 INFO Configured application/vnd.gremlin-v2.0+json with org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0 [AbstractChannelizer]
2020-07-25 15:47:35:629 INFO application/json already has org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0 configured - it will not be replaced by org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, change order of serialization configuration if this is not desired. [AbstractChannelizer]
The python code using RDFlib is as follows
import os
from rdf2g import setup_graph, load_rdf2g
from rdflib import Graph
from pathlib import Path
import pprint
# connect to the graph server configured to OrientDB
DEFAULT_LOCAL_CONNECTION_STRING = "ws:localhost:8182/gremlin"
g = setup_graph(DEFAULT_LOCAL_CONNECTION_STRING)
# get the working directory
current_dir = os.getcwd()
# temp holder for rdf filename
rdf_file = 'C:\\Users\\nnn\\Desktop\\rdf_config\\Education_simple.rdf'
# define the path to the rdf
rdf_path = os.path.join(current_dir, r'rdf_config', rdf_file)
# read the rdf file to a graph
rdf_graph = Graph()
rdf_graph.parse(rdf_file)
# load the RDF graph into a property graph
load_rdf2g(g, rdf_graph)
and below are the logs after running this python script:
File "c:/Users/nnn/Desktop/rdf2gremlin0721.py", line 36, in <module>
load_rdf2g(g, rdf_graph)
File "C:\Program Files\Python37\lib\site-packages\rdf2g\update.py", line 33, in load_rdf2g
link_nodes(g, subj_node, obj_node, p, rdf_graph)
File "C:\Program Files\Python37\lib\site-packages\rdf2g\update.py", line 122, in link_nodes
print(g.V(Bindings.of('id', source_node)).addE(property_label).to(target_node).iterate())
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\process\traversal.py", line 66, in iterate
try: self.nextTraverser()
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\process\traversal.py", line 71, in nextTraverser
self.traversal_strategies.apply_strategies(self)
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\process\traversal.py", line 573, in apply_strategies
traversal_strategy.apply(traversal)
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\driver\remote_connection.py", line 149, in apply
remote_traversal = self.remote_connection.submit(traversal.bytecode)
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\driver\driver_remote_connection.py", line 56, in submit
results = result_set.all().result()
File "C:\Program Files\Python37\lib\concurrent\futures\_base.py", line 435, in result
return self.__get_result()
File "C:\Program Files\Python37\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\driver\resultset.py", line 90, in cb
f.result()
File "C:\Program Files\Python37\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Program Files\Python37\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Program Files\Python37\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\driver\connection.py", line 83, in _receive
status_code = self._protocol.data_received(data, self._results)
File "C:\Program Files\Python37\lib\site-packages\gremlin_python\driver\protocol.py", line 111, in data_received
del results_dict[request_id]
KeyError: None
Please let me know if you know any details about it.

Related

Apache Beam ReadFromSpanner decoding issue

I'm trying to run the following script in a GCP Dataflow pipeline.
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from typing import NamedTuple, Optional
from apache_beam.io.gcp.spanner import *
from past.builtins import unicode
import logging
class ItemRow(NamedTuple):
item_id: unicode
class LogResults(beam.DoFn):
"""Just log the results"""
def process(self, element):
logging.info("row: %s", element)
yield element
class SpannerToSpannerAndBigQueryPipelineOptions(PipelineOptions):
"""
Runtime Parameters given during template execution
path parameter is necessary for execution of pipeline
"""
#classmethod
def _add_argparse_args(cls, parser):
parser.add_argument(
'--SOURCE_SPANNER_PROJECT_ID', type=str, help='Source Spanner project ID',
default='project_id')
parser.add_argument(
'--SOURCE_SPANNER_DATASET_ID', type=str, help='Source Spanner dataset ID',
default='dataset_id')
parser.add_argument(
'--SOURCE_SPANNER_INSTANCE_ID', type=str, help='Source Spanner instance ID',
default='instance_id')
parser.add_argument(
'--SOURCE_QUERY', type=str, help='SQL to run in Source Spanner Instance',
required=True)
# Setup pipeline
def run():
beam.coders.registry.register_coder(ItemRow, beam.coders.RowCoder)
pipeline_options = PipelineOptions()
p = beam.Pipeline(options=pipeline_options)
importer_options = pipeline_options.view_as(
SpannerToSpannerAndBigQueryPipelineOptions)
rows = (
p
| "Read from source Spanner" >> ReadFromSpanner(
project_id=importer_options.SOURCE_SPANNER_PROJECT_ID,
instance_id=importer_options.SOURCE_SPANNER_INSTANCE_ID,
database_id=importer_options.SOURCE_SPANNER_DATASET_ID,
row_type=ItemRow,
sql='Select item_id from Items WHERE created_ts BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 5 SECOND) AND CURRENT_TIMESTAMP()',
timestamp_bound_mode=TimestampBoundMode.MAX_STALENESS,
staleness=3,
time_unit=TimeUnit.HOURS,
).with_output_types(ItemRow)
)
rows | 'Log results' >> beam.ParDo(LogResults())
result = p.run()
result.wait_until_finish()
if __name__ == "__main__":
run()
However, I've been running into issues for decoding the results obtained from Spanner. These are the output logs from my Dataflow job:
"An exception was raised when trying to execute the workitem 6665479626992209510 : Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/batchworker.py", line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/executor.py", line 179, in execute
op.start()
File "dataflow_worker/native_operations.py", line 38, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 39, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 44, in dataflow_worker.native_operations.NativeReadOperation.start
File "dataflow_worker/native_operations.py", line 48, in dataflow_worker.native_operations.NativeReadOperation.start
File "/usr/local/lib/python3.7/site-packages/dataflow_worker/inmemory.py", line 108, in __iter__
yield self._source.coder.decode(value)
File "/usr/local/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 468, in decode
return self.get_impl().decode(encoded)
File "apache_beam/coders/coder_impl.py", line 226, in apache_beam.coders.coder_impl.StreamCoderImpl.decode
File "apache_beam/coders/coder_impl.py", line 228, in apache_beam.coders.coder_impl.StreamCoderImpl.decode
File "apache_beam/coders/coder_impl.py", line 123, in apache_beam.coders.coder_impl.CoderImpl.decode_from_stream
File "/usr/local/lib/python3.7/site-packages/apache_beam/coders/row_coder.py", line 215, in decode_from_stream
is_null in zip(self.components, nulls)))
File "/usr/local/lib/python3.7/site-packages/apache_beam/coders/row_coder.py", line 215, in <genexpr>
is_null in zip(self.components, nulls)))
File "apache_beam/coders/coder_impl.py", line 259, in apache_beam.coders.coder_impl.CallbackCoderImpl.decode_from_stream
File "apache_beam/coders/coder_impl.py", line 261, in apache_beam.coders.coder_impl.CallbackCoderImpl.decode_from_stream
File "/usr/local/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 414, in decode
return value.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 9: invalid start byte
"
I'm unsure as to how to solve this problem. I'm using this https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.io.gcp.spanner.html?highlight=spanner#module-apache_beam.io.gcp.spanner example as a starting point. The issue appears to be in decoding the results obtained from Spanner. There is little to no documentation on how to specify the schema for the Spanner table/tables I'm trying to query.
There is also an experimental IO module for Spanner which does not use the Java expansion module. Is it recommended to switch to the experimental version?
Thanks
I could not run a pipeline using the apache_beam.io.gcp.spanner module so I ended up using the apache_beam.io.gcp.experimental.spannerio module instead.

AttributeError: module 'os' has no attribute 'getcwd'?

i am trying to run a python program which takes input from the users like,directory name,file name and its content and then i try to create the directory in the current path with the file and then write the data into it but i get an error that "os has no attribute mkdir".
//Code
import os
print(os.getcwd())
class createfile:
__name=''
__file_name=''
__input_text=''
def read_and_create(self):
name = input("Enter the directory name, you want to create: ")
os.mkdir(self.__name)
#sdos.mkdir(self.__name)
self.__file_name=input("Enter the file name, you want to create: ")
f=open(self.___file_name,'w')
self.__input_text=input("Enter the text : ")
f.write(self.__input_text)
f.close()
def read_file_content(self):
x=open(self.__file_name,'r')
print(x.read())
o=createfile()
o.read_and_create()
o.read_file_content()
this is the error i got using Pycharm.
> "C:\Users\Aryan
> Kumar\AppData\Local\Programs\Python\Python37-32\python.exe"
> "C:/Users/Aryan Kumar/Desktop/0/os.py"
> Fatal Python error: initsite: Failed to import the site module
> Traceback (most recent call last):
> File "C:\Users\Aryan Kumar\AppData\Local\Programs\Python\Python37-32\lib\site.py", line 73,
> in <module>
> import os
> File "C:\Users\Aryan Kumar\Desktop\0\os.py", line 2, in <module>
> print(os.getcwd())
> AttributeError: module 'os' has no attribute 'getcwd'
>
> Process finished with exit code 1
this is the error i got using Spyder ide in anaconda environment
runfile('C:/Users/Aryan Kumar/Desktop/0/untitled0.py', wdir='C:/Users/Aryan Kumar/Desktop/0')
C:\Users\Aryan Kumar\Desktop\0
Enter the directory name, you want to create: aac
Traceback (most recent call last):
File "<ipython-input-4-fb1c179833e1>", line 1, in <module>
runfile('C:/Users/Aryan Kumar/Desktop/0/untitled0.py', wdir='C:/Users/Aryan Kumar/Desktop/0')
File "C:\Users\Aryan Kumar\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 709, in runfile
execfile(filename, namespace)
File "C:\Users\Aryan Kumar\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 108, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/Aryan Kumar/Desktop/0/untitled0.py", line 32, in <module>
o.read_and_create()
File "C:/Users/Aryan Kumar/Desktop/0/untitled0.py", line 18, in read_and_create
os.mkdir(self.__name)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ''
This is an old question but someone might have the same problem. I also received the following error: AttributeError: module 'os' has no attribute 'getcwd'. This is because the file name is os.py. Problem solved after changing the file name.
Also found that a similar answer also provided in a different question:
https://stackoverflow.com/a/43565408/298160
Use os.curdir() in Python3 instead of os.cwd()

Issue with Geodjango OGR failure - GDAL Exception

I have made it to which the model in django is built, but once adding data and clicking save my app crash
i get GDAL exception - OGR Failure, and it highlights in the crash page " {{ field.field }}" as the reason for the error.
here's screen shot Admin view for adding data to the model
i get this right away after clicking save
Anyone been through this? any help?
Thanks
Update :
I'm getting this error :
[12/Apr/2019 14:38:46] "GET /static/admin/img/gis/move_vertex_off.svg HTTP/1.1"
200 1129
[12/Apr/2019 14:39:14] "GET /static/admin/img/gis/move_vertex_on.svg HTTP/1.1" 2
00 1129
GDAL_ERROR 4: b'Unable to open EPSG support file gcs.csv.\nTry setting the GDAL_
DATA environment variable to point to the\ndirectory containing EPSG csv files.'
Internal Server Error: /admin/trial2/shop/add/
Traceback (most recent call last):
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\core\
handlers\exception.py", line 34, in inner
response = get_response(request)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\core\
handlers\base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\core\
handlers\base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\admin\options.py", line 606, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\utils
\decorators.py", line 142, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\views
\decorators\cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\admin\sites.py", line 223, in inner
return view(request, *args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\admin\options.py", line 1634, in add_view
return self.changeform_view(request, None, form_url, extra_context)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\utils
\decorators.py", line 45, in _wrapper
return bound_method(*args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\utils
\decorators.py", line 142, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\admin\options.py", line 1522, in changeform_view
return self._changeform_view(request, object_id, form_url, extra_context)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\admin\options.py", line 1554, in _changeform_view
form_validated = form.is_valid()
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\forms
\forms.py", line 185, in is_valid
return self.is_bound and not self.errors
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\forms
\forms.py", line 180, in errors
self.full_clean()
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\forms
\forms.py", line 381, in full_clean
self._clean_fields()
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\forms
\forms.py", line 399, in _clean_fields
value = field.clean(value)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\forms\fields.py", line 79, in clean
geom.transform(self.srid)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\geos\geometry.py", line 471, in transform
g = gdal.OGRGeometry(self._ogr_ptr(), srid)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\geometries.py", line 115, in __init__
self.srs = srs
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\geometries.py", line 284, in _set_srs
sr = SpatialReference(srs)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\srs.py", line 92, in __init__
self.import_epsg(srs_input)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\srs.py", line 277, in import_epsg
capi.from_epsg(self.ptr, epsg)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\prototypes\errcheck.py", line 118, in check_errcode
check_err(result, cpl=cpl)
File "C:\Users\lgcge\Documents\copypostgis\venv\lib\site-packages\django\contr
ib\gis\gdal\error.py", line 59, in check_err
raise e(msg)
django.contrib.gis.gdal.error.GDALException: OGR failure.
[12/Apr/2019 14:39:45] "POST /admin/trial2/shop/add/ HTTP/1.1" 500 160518
admin.py :
from django import forms
from django.contrib.gis import admin
from django.contrib.gis.db import models
from django.contrib.gis.admin import OSMGeoAdmin
from .models import Shop
#admin.register(Shop)
class ShopAdmin(OSMGeoAdmin):
list_display = ('name', 'location')
model.py
from __future__ import unicode_literals
from django.contrib.gis.db import models
from django.contrib.gis.geos import Point
class Shop(models.Model):
name = models.CharField(max_length=100)
location = models.PointField()
address = models.CharField(max_length=100)
city = models.CharField(max_length=50)
Apologies for copy/pasting and only slightly modifying #Kiwi's answer. I tried adding a comment to the original answer but I could not add formatted code.
It seems now that you cannot install the 64 bit version of OSGEO4W any more. I installed the only version available here and modified #Kiwi's answer to reflect the lack of 64 bit. I'm on Windows 10. All works fine with the below.
import platform
import environ
WINDOWS = platform.system() == "Windows"
if WINDOWS:
# the below needs to change for linux
GDAL_LIBRARY_PATH = r'C:\OSGeo4W\bin\gdal303.dll'
GEOS_LIBRARY_PATH = r'C:\OSGeo4W\bin\geos_c.dll'
OSGEO4W = r"C:\OSGeo4W"
os.environ['OSGEO4W_ROOT'] = OSGEO4W
os.environ['GDAL_DATA'] = "C:\Program Files\GDAL\gdal-data" # OSGEO4W + r"\share\gdal"
os.environ['PROJ_LIB'] = OSGEO4W + r"\share\proj"
os.environ['PATH'] = OSGEO4W + r"\bin;" + os.environ['PATH']
if os.name == 'nt':
import platform
OSGEO4W = r"C:\OSGeo4W"
if '64' in platform.architecture()[0]:
OSGEO4W += "64"
assert os.path.isdir(OSGEO4W), "Directory does not exist: " + OSGEO4W
os.environ['OSGEO4W_ROOT'] = OSGEO4W
os.environ['GDAL_DATA'] = "C:\Program Files\GDAL\gdal-data" #OSGEO4W + r"\share\gdal"
os.environ['PROJ_LIB'] = OSGEO4W + r"\share\proj"
os.environ['PATH'] = OSGEO4W + r"\bin;" + os.environ['PATH']
the issue was that this code enforces the paths defined in it over geodjango, so even if you set system variables, still this code would dominate over the system variables.
If anyone has issue with OGR don't hesitate to comment
I had the same problem and this is how I was able to solve it.
first check your GDAL version and GEOS
by running following scripts
gdal-config --version
and
geos-config --version
If the versions of GDAL and GEOS do not match, you will need to either install a compatible version of GEOS or update GDAL to a version that is compatible with the version of GEOS you have installed.
For example if you have GEOS version 3.8.0 or above you will need to upgrade GDAL >= 3.1.2

Django-pipeline invalid syntax

I am trying to setup django-pipeline in my test project, however, I am running into a bug which doesn't have an obvious solution. Here is the pipeline configure in the settings.py
PIPELINE = {
'PIPELINE_ENABLED': True,
'COMPILERS': ('pipeline_browserify.compiler.BrowserifyCompiler', ),
'CSS_COMPRESSOR': 'pipeline.compressors.NoopCompressor',
'JS_COMPRESSOR': 'pipeline.compressors.uglifyjs.UglifyJSCompressor',
'STYLESHEETS': {
# 'animelist_css': {
# 'source_filenames': ( 'animelist/css/style.css', ),
# 'output_filename': 'animelist/css/animelist_css.css',
# }
},
'JAVASCRIPT': {
# 'animelist_js': {
# 'source_filenames': (
# 'animelist/js/bower_components/jquery/dist/jquery.min.js',
# 'animelist/js/bower_components/react/JSXTransformer.js',
# 'animelist/js/bower_components/react/react-with-addons.js',
# 'animelist/js/app.browserify.js',
# ),
# 'output_filename': 'animelist/js/animelist_js.js',
# }
}
}
As you can see from the configuration, this setting will allow collectstatic to run without any problem. However, collectstatic will fail if I were to uncomment those lines. The error is shown below
You have requested to collect static files at the destination
location as specified in your settings:
/var/www/anime/static
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/opt/virtualenvs/django_project/lib64/python3.4/site- packages/django/core/management/__init__.py", line 338, in execute_from_command_line
utility.execute()
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/core/management/__init__.py", line 330, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/core/management/base.py", line 441, in execute
output = self.handle(*args, **options)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 168, in handle
collected = self.collect()
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
for original_path, processed_path, processed in processor:
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/pipeline/storage.py", line 33, in post_process
packager.pack_javascripts(package)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/pipeline/packager.py", line 112, in pack_javascripts
return self.pack(package, self.compressor.compress_js, js_compressed, templates=package.templates, **kwargs)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/pipeline/packager.py", line 105, in pack
paths = self.compile(package.paths, force=True)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/pipeline/packager.py", line 99, in compile
return self.compiler.compile(paths, force=force)
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/pipeline/compilers/__init__.py", line 49, in compile
from concurrent import futures
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/opt/virtualenvs/django_project/lib64/python3.4/site-packages/concurrent/futures/_base.py", line 355
raise type(self._exception), self._exception, self._traceback
^
SyntaxError: invalid syntax
Help is greatly appreciated.
Apparently, the error is the result of installing futures backport in pip3. Simply uninstall futures packages in your pip3.

Google Cloud Storage Client

I make first steps with GCS
First, I created service account in my project and got keyfile in json
Next, I tried to write script like this sample
But...
from gcloud import storage
client = storage.Client.from_service_account_json('/path/to/keyfile.json')
bucket = client.get_bucket('enggeo')
# Then do other things...
blob = bucket.get_blob('/ETicket.pdf')
print blob.download_as_string()
blob2 = bucket.blob('/wtt.txt')
blob2.upload_from_filename(filename='/home/test2/.www/test')
And getting error
File "tt.py", line 3, in <module>
client = storage.Client.from_service_account_json('/path/to/keyfile.json') # TODO: rel paths
File "/home/test2/lib/python2.7/site-packages/gcloud/client.py", line 64, in from_service_account_json
return cls(*args, **kwargs)
File "/home/test2/lib/python2.7/site-packages/gcloud/storage/client.py", line 53, in __init__
http=http)
File "/home/test2/lib/python2.7/site-packages/gcloud/client.py", line 181, in __init__
_ClientProjectMixin.__init__(self, project=project)
File "/home/test2/lib/python2.7/site-packages/gcloud/client.py", line 146, in __init__
raise ValueError('Project was not passed and could not be '
Project was not passed and could not be determined from the environment.
What is wrong?
client = storage.Client.from_service_account_json('/path/to/keyfile.json', 'project')
Need to specified args for JSONClient
https://googlecloudplatform.github.io/gcloud-python/stable/gcloud-api.html#gcloud.client.JSONClient