tf2onnx fails to convert TextVectorizer - tensorflow2

Reproduction Steps
Setup environment
python3.10 -m venv venv
source activate venv/bin/activate
pip install -U pip
pip install tensorflow==2.8.0 tf2onnx==1.13.0 packaging==23.0
Create a model with TextVectorizer
import tensorflow as tf
text_dataset = tf.data.Dataset.from_tensor_slices(["foo", "bar", "baz"])
vectorize_layer = tf.keras.layers.TextVectorization(max_tokens=100, output_mode='int', output_sequence_length=4)
vectorize_layer.adapt(text_dataset)
model = tf.keras.models.Sequential([
tf.keras.Input(shape=(1,), dtype=tf.string),
vectorize_layer
])
model.save("text_vectorizer")
Try to convert the model to ONNX
python -m tf2onnx.convert --saved-model text_vectorizer --output text_vectorizer.onnx
Result:
<...>
/tf2onnx/lib/python3.10/site-packages/tf2onnx/tf_utils.py:58: FutureWarning: In the future `np.str` will be defined as the corresponding NumPy scalar.
np_data = np_data.astype(np.str).astype(object)
Traceback (most recent call last):
File "/tf2onnx/lib/python3.10/site-packages/tf2onnx/tf_utils.py", line 58, in tf_to_onnx_tensor
np_data = np_data.astype(np.str).astype(object)
File "/tf2onnx/lib/python3.10/site-packages/numpy/__init__.py", line 305, in __getattr__
raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'str'.
`np.str` was a deprecated alias for the builtin `str`. To avoid this error in existing code, use `str` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.str_` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'std'?
During handling of the above exception, another exception occurred:
<...>
File "/tf2onnx/lib/python3.10/site-packages/tf2onnx/tf_utils.py", line 63, in tf_to_onnx_tensor
raise RuntimeError("Not support type: {}".format(type(np_data.flat[0])))
RuntimeError: Not support type: <class 'bytes'>
How to fix the problem? Is there a way to make this conversion?

Related

ERROR: gcloud failed to load: No module named 'googlecloudsdk.third_party.apis.binaryauthorization.v1alpha2.binaryauthorization_v1alpha2_messages'

I am trying to install gcloud CLI on macOS 64-bit(x86_64) by following these instructions: https://cloud.google.com/sdk/docs/install
When I run ./google-cloud-sdk/install.sh I get this error:
Welcome to the Google Cloud CLI!
Traceback (most recent call last):
File "/Users/benkula/./google-cloud-sdk/bin/bootstrapping/install.py", line 30, in <module>
from googlecloudsdk import gcloud_main
File "/Users/benkula/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 37, in <module>
from googlecloudsdk.command_lib.util.apis import yaml_command_translator
File "/Users/benkula/google-cloud-sdk/lib/googlecloudsdk/command_lib/util/apis/yaml_command_translator.py", line 39, in <module>
from googlecloudsdk.command_lib.iam import iam_util
File "/Users/benkula/google-cloud-sdk/lib/googlecloudsdk/command_lib/iam/iam_util.py", line 85, in <module>
binaryauthorization_message_v1alpha2 = core_apis.GetMessagesModule(
File "/Users/benkula/google-cloud-sdk/lib/googlecloudsdk/api_lib/util/apis.py", line 339, in GetMessagesModule
return __import__(api_def.apitools.messages_full_modulepath,
ModuleNotFoundError: No module named 'googlecloudsdk.third_party.apis.binaryauthorization.v1alpha2.binaryauthorization_v1alpha2_messages'
./google-cloud-sdk/bin/gcloud init shows a similar error:
ERROR: gcloud failed to load: No module named 'googlecloudsdk.third_party.apis.binaryauthorization.v1alpha2.binaryauthorization_v1alpha2_messages'
gcloud_main = _import_gcloud_main()
import googlecloudsdk.gcloud_main
from googlecloudsdk.command_lib.util.apis import yaml_command_translator
from googlecloudsdk.command_lib.iam import iam_util
binaryauthorization_message_v1alpha2 = core_apis.GetMessagesModule(
return __import__(api_def.apitools.messages_full_modulepath,
This usually indicates corruption in your gcloud installation or problems with your Python interpreter.
Please verify that the following is the path to a working Python 2.7 or 3.5+ executable:
/Applications/Xcode.app/Contents/Developer/usr/bin/python3
If it is not, please set the CLOUDSDK_PYTHON environment variable to point to a working Python 2.7 or 3.5+ executable.
If you are still experiencing problems, please reinstall the Cloud SDK using the instructions here:
https://cloud.google.com/sdk/
Python executable seems to be where it should be, so I don't think that's the problem. I can see it when I run ls /Applications/Xcode.app/Contents/Developer/usr/bin/.
python3 -V shows Python 3.8.9.
What can I do to install gcloud CLI successfully?
I used this command instead, and it worked for me:
curl https://sdk.cloud.google.com | bash
Found here: https://cloud.google.com/sdk/docs/downloads-interactive.

Issue when converting ONNX model to Caffe2

I converted a TF model to ONNX and then ONNX model to Caffe2. The conversion happened successfully. However, I am getting a RunTime Error when trying to load and infer from the obtained model.
This is the error that I am receiving. How do I add the attribute 'is_true' to SpatialBN node?
I went through the pytorch repo and saw this issue, however, it is unresolved.
In the code base of ONNX here, it adds is_test attribute for opset >=7 and I am using 8. However, it is still giving the error.
[W common_gpu.cc:35] Insufficient cuda driver. Cannot use cuda.
[W init.h:137] Caffe2 GlobalInit should be run before any other API calls.
[W init.h:137] Caffe2 GlobalInit should be run before any other API calls.
[W predictor_config.cc:90] Caffe2 is compiled without optimization passes.
[E operator_schema.cc:101] Argument 'is_test' is required for Operator 'SpatialBN'.
Traceback (most recent call last):
File "main.py", line 91, in <module>
test_caffe("mod-caffe-net.pb", "mod-caffe-init-net.pb", "../data/mouth")
File "main.py", line 70, in test_caffe
predictor = workspace.Predictor(param_values, model_net)
File "/home/ubuntu/.local/lib/python3.6/site-packages/caffe2/python/workspace.py", line 187, in Predictor
return C.Predictor(StringifyProto(init_net), StringifyProto(predict_net))
RuntimeError: [enforce fail at operator.cc:199] schema->Verify(operator_def). Operator def did not pass schema checking: input: "conv1/Relu:0" input: "batchNorm1/gamma/read/_1__cf__1:0" input: "batchNorm1/beta/read/_0__cf__0:0" input: "batchNorm2/moving_mean/read/_6__cf__6:0" input: "batchNorm1/moving_variance/read/_3__cf__3:0" output: "batchNorm1/FusedBatchNorm:0" name: "batchNorm1/FusedBatchNorm" type: "SpatialBN" arg { name: "epsilon" f: 0.001 } device_option { device_type: 0 device_id: 0 }
The issue is resolved. I was using the command-line utility suggested on their README. However, it points to their tutorial in deprecated version of the code.
The command-line utility (installed using the pip install onnx-caffe2) still has the _known_opset_version = 3. This was causing the error. After I used the conversion utility through Python APIs in PyTorch library by importing,
from caffe2.python.onnx.backend import Caffe2Backend as c2
I was successfully able to run inference on the converted model.

Unable to use jsonpath_rw_ext with pyspark

These are the steps I am following:
mkdir spark_lib; cd spark_lib
pip install jsonpath_rw_ext==1.1.3 -t .
zip -r9 ../spark_lib.zip *
Initialize spark context variable sc
sc.addPyFile('spark_lib.zip')
def f(x):
import jsonpath_rw_ext
return jsonpath_rw_ext.match('$.a.b', x)
sc.parallelize([{"a":{"b":10}}]).map(f).collect()
This gives me a pbr versioning error
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
vs = list(itertools.islice(iterator, batch))
File "<ipython-input-11-2619fc011e24>", line 6, in f
File "./spark_lib.zip/jsonpath_rw_ext/__init__.py", line 18, in <module>
File "./spark_lib.zip/pbr/version.py", line 467, in version_string
return self.semantic_version().brief_string()
File "./spark_lib.zip/pbr/version.py", line 462, in semantic_version
self._semantic = self._get_version_from_pkg_resources()
File "./spark_lib.zip/pbr/version.py", line 449, in _get_version_from_pkg_resources
result_string = packaging.get_version(self.package)
File "./spark_lib.zip/pbr/packaging.py", line 824, in get_version
name=package_name))
Exception: Versioning for this project requires either an sdist tarball,
or access to an upstream git repository. It's also possible that
there is a mismatch between the package name in setup.cfg
and the argument given to pbr.version.VersionInfo.
Project name jsonpath_rw_ext was given, but was not able to be found.
After reading through another similar bug https://bugs.launchpad.net/python-swiftclient/+bug/1379579, found out that pbr needs updated setuptools.
I included setuptools in my pip installation command and everything worked fine.
pip install jsonpath_rw_ext setuptools -t .

iptest3 with IPython

I'm brand new to Python programming and trying to get myself a functional base from which I can run things like the IPython Notebook which looks pretty exciting.
Thus far I have both Python 2.7 and 3.3 from python.org installed in OS X 10.6 (Snow Leopard) as well as ActiveTcl 8.5.13. Almost everything that I've tried thus far works as expected. I'm focused on learning 3.3, but want to have the option of using 2.7 too. I read up in several documents that I need to start gaining access to PyPI packages using a Python package manager and that distribute is the one I should use for 3k. So I installed that according to the documentation I found and it seemed to work fine.
I also installed pip as directed, and a number of others.
At this point, I have:
$ pip freeze
distribute==0.6.34
ipython==0.13.1
nose==1.2.1 (installed after IPython)
pexpect==2.4 (installed after IPython)
pyflakes3k==0.4.3
readline==6.2.4.1 (installed after IPython)
At this point, I'm doing this from ipython.org guidance
And when I did $ easy_install pexpect, I got a bunch of errors:
$ easy_install pexpect
Searching for pexpect
Reading http://pypi.python.org/simple/pexpect/
Reading http://pexpect.sourceforge.net/
Reading http://sourceforge.net/project/showfiles.php?group_id=59762
Best match: pexpect 2.4
Downloading http://pypi.python.org/packages/source/p/pexpect/pexpect-2.4.tar.gz#md5=fe82d69be19ec96d3a6650af947d5665
Processing pexpect-2.4.tar.gz
Writing /var/folders/td/td0Sh8EfGFuMCnKex1v+q++++TI/-Tmp-/easy_install-s4dtyy/pexpect-2.4/setup.cfg
Running pexpect-2.4/setup.py -q bdist_egg --dist-dir /var/folders/td/td0Sh8EfGFuMCnKex1v+q++++TI/-Tmp-/easy_install-s4dtyy/pexpect-2.4/egg-dist-tmp-5h5cg4
File "build/bdist.macosx-10.6-intel/egg/fdpexpect.py", line 36
raise ExceptionPexpect, 'The fd argument is not a valid file descriptor.'
^
SyntaxError: invalid syntax
File "build/bdist.macosx-10.6-intel/egg/FSM.py", line 77
return `self.value`
^
SyntaxError: invalid syntax
File "build/bdist.macosx-10.6-intel/egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/fdpexpect.py", line 36
raise ExceptionPexpect, 'The fd argument is not a valid file descriptor.'
^
SyntaxError: invalid syntax
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/FSM.py", line 77
return `self.value`
^
SyntaxError: invalid syntax
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
Adding pexpect 2.4 to easy-install.pth file
Installed /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg
Processing dependencies for pexpect
Finished processing dependencies for pexpect
That looks bad to me (although I don't yet have the expertise to really interpret it), and so I'm not sure if I have a complete install of pexpect.
After installing nose (before pexpect as per URL above), I tried running iptest and iptest3 from the command line, and both failed to find the command, but after I did easy_install ipython again (after nose), I noticed that this install added iptest3 (as well as ipcluster3 and a few other scripts) to my path, and now my bash shell can find iptest3, but when I run it, I get some more bad-looking output:
$ iptest3
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.3/bin/iptest3", line 9, in <module>
load_entry_point('ipython==0.13.1', 'console_scripts', 'iptest3')()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 343, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 2308, in load_entry_point
return ep.load()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/distribute-0.6.34-py3.3.egg/pkg_resources.py", line 2014, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/__init__.py", line 43, in <module>
from .config.loader import Config
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/__init__.py", line 16, in <module>
from .application import *
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/application.py", line 31, in <module>
from IPython.config.configurable import SingletonConfigurable
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/configurable.py", line 26, in <module>
from .loader import Config
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/config/loader.py", line 27, in <module>
from IPython.utils.path import filefind, get_ipython_dir
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/path.py", line 25, in <module>
from IPython.utils.process import system
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/process.py", line 27, in <module>
from ._process_posix import _find_cmd, system, getoutput, arg_split
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/utils/_process_posix.py", line 22, in <module>
from IPython.external import pexpect
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/IPython/external/pexpect/__init__.py", line 2, in <module>
import pexpect
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pexpect-2.4-py3.3.egg/pexpect.py", line 82
except ImportError, e:
^
SyntaxError: invalid syntax
After doing all that, I noted that my /Library/Frameworks/Python.framework/Versions/Current had been pointing to 2.7, and I guessed that that might be related to my problems and changed the symbolic link to point to 3.3, but iptest3 still fails with the error above.
Any other thoughts on what to do to fix this? It's clear that iptest is pretty important to doing anything else (like IPython Notebook) I want to do.
There is a py3-compatible fork of pexpect called pexpect-u (the u is for unicode, the main difference between the two). You need this to run the pexpect-based parts of IPython on Python 3.
Should be a simple
pip install pexpect-u
side note: pexpect-u is by IPython developer Thomas Kluyver, who did most of the heavy lifting bringing py3 compatibility to IPython.

Eclipse using multiple Python interpreters with execnet

I'm using the execnet package to allow communication between Python scripts interpreted by different Python interpreters.
The following code (test_execnet.py):
import execnet
for python_version in ('python', 'python3'):
try:
gw = execnet.makegateway("popen//python="+python_version)
ch = gw.remote_exec('channel.send(1/3)')
res = ch.receive()
print(python_version, ': ', res, sep ="")
except:
print('problems with ', python_version)
Runs perfectly in the command-line Terminal, showing the following output:
$ python3 test_execnet.py
python: 0
python3: 0.333333333333
However, if I try to run the same code from within the Eclipse IDE, I get the following error:
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 4, in <module>
File "<string>", line 2, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/site-packages/execnet/gateway_base.py", line 8, in <module>
import sys, os, weakref
File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/os.py", line 380, in <module>
from _abcoll import MutableMapping # Can't use collections (bootstrap)
File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/_abcoll.py", line 54
class Hashable(metaclass=ABCMeta):
^
SyntaxError: invalid syntax
problems with python
problems with python3
NOTE:
Eclipse Version: 3.6.0
PyDev Interpreter configured for the project: python3
"Preferences/Interpreter - Python"'s Python Interpreters:
python (/usr/bin/python)
python3 (/Library/Frameworks/Python.Framework/Versions/3.1/Resources/Python.app/Contents/MacOS/Python
EDIT:
I write a code to show the os.environ like this:
for python_version in ('python', 'python3'):
try:
import os
for item in os.environ:
print(item, '= ', os.environ[item])
except:
print('problems with ', python_version)
I got the following outputs:
eclipse_output.txt
terminal_output.txt
A FileMerge comparison of the files can be found at eclipse_output.txt vs. terminal_output.pdf.
Any hints?
Thanks
seems like pydev does site-customizations and particularly modifies things for interactive/console usage (judging from a very quick skim of http://github.com/aptana/Pydev/blob/master/plugins/org.python.pydev/pysrc/pydev_sitecustomize/sitecustomize.py ). This is not useful or fitting for execnet-mediated processes.
You could try to "del os.environ['PYTHONPATH']" before you invoke execnet.makegateway, or, to be more careful, just delete the sitecustomize part of it.
hth,
holger
'import site' failed; use -v for traceback
I have seen that when python was unable to find its landmark. Which that indicates there is a PYTHONHOME problem.
Check out http://docs.python.org/using/cmdline.html#envvar-PYTHONHOME maybe eclipse is screwing your environment up.
Edit:
Looked at your env dumps, looks like eclipse is definitely messing with PYTHONPATH, which will cause your child python processes to not work correctly.
Basically what you have going on here is eclipse starts a python v2 instance with a PYTHONPATH pointing to the python v2 directories. Then you spawn a python v3 process which tries to load its landmark from the python v2 directories...
You need to find a way to have eclipse not mess with the PYTHONPATH. I am not sure what eclipse is trying to do by doing that, but it is certainly no friend when you want to spawn new python processes.