Trying to migrate data from Firebird DB to MS Sql Server using fdb(2.0.1) and pyodbc. Since there are blobs in the Firebird database which are over 64K, they are being returned as BlobReader objects. Since i would like not to deal with the bytes myself and write them using pyodbc. The docs say that you can turn off the 64K threshold by passing -1 to the cursor.set_stream_blob_threshold. However that doesn't seem to work, since fdb.fbcore.ProgrammingError is thrown...
https://fdb.readthedocs.io/en/v2.0/reference.html#fdb.Cursor.set_stream_blob_treshold
Here is how i call the function:
import fdb
class Firebird:
def __init__(self, db_name: str):
self.__fb_conn = fdb.connect(database=db_name, user='someuser', password='somepass', charset='ISO8859_1')
self.__fb_cursor = self.__fb_conn.cursor()
#change the blob safety threshold to unlimited for troubleshooting
self.__fb_cursor.set_stream_blob_treshold(-1) #doesn't work :(
Here is a stack trace for the error:
(.venv) >python3.8.exe -i
Python 3.8.5 (tags/v3.8.5:580fbb0, Jul 20 2020, 15:57:54) [MSC v.1924 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from commonlibs import Firebird
>>>
>>> fb = Firebird('somedb.fdb')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\user1\dev\commonlibs\Firebird.py", line 13, in __init__
self.__fb_cursor.set_stream_blob_treshold(int(-1)) #doesn't work :(
File "C:\Users\user1\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\fdb\fbcore.py", line 3930, in set_stream_blob_treshold
raise ProgrammingError
fdb.fbcore.ProgrammingError
Per Mark's comment:
I don't know much about the data source and what sort of blobs. It was one of those situations where the other teams guy said: "Hey, here is some data from this partner, let's see what inside"
However when trying to pass the obj.read() value to the pyodbc for BlobReader objects, it did insert some of the blobs. However with a lot of them pyodbc would report this error:
pyodbc.Error: ('HY000', '[HY000] [Microsoft][ODBC SQL Server Driver]Warning: Partial insert/update. The insert/update of a text or image column(s) did not succeed. (0) (SQLPutData); [HY000] [Microsoft][ODBC SQL Server Driver][SQL Server]The text, ntext, or image pointer value conflicts with the column name specified. (7125)')
I was kind hoping i could avoid all this pyodbc and .read() stuff by setting that threshold, but i wonder if the pyodbc error would show up regardless...
Related
I'm connecting to DB2-LUW database using Python 3.7 and some queries get error : "SQL0332N Character conversion from the source code page "1252" to the target code page "874" is not supported.***".
First I try to test the connection of Python to the database on DB2 by recreating a new table.
I insert 1 record and read it back. When I read the inserted row, I get the error.
Results in interactive python:
import ibm_db_dbi as dbi
print(dbi.__version__)
3.0.2
conn = dbi.connect("DATABASE=<db>;HOSTNAME=<hostname>;PORT=<port>;PROTOCOL=TCPIP;UID=<user>;PWD=<pwd>;", "", "")
c = conn.cursor()
c.execute('create table ibm_db_tst (col1 int)')
Out[5]: True
c.execute('insert into ibm_db_tst values(2)')
Out[6]: True
c.execute('select col1 from ibm_db_tst')
Out[7]: True
print(c.fetchone())
Traceback (most recent call last):
File "C:\Users\2400566\Anaconda3\lib\site-packages\ibm_db_dbi.py",
line 1449, in _fetch_helper
row = ibm_db.fetch_tuple(self.stmt_handler)
SQLCODE=-332lumn information cannot be retrieved: [IBM][CLI
Driver][DB2/NT64] SQL0332N Character conversion from the source code
page "1252" to the target code page "874" is not supported.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
print(c.fetchone())
File "C:\Users\2400566\Anaconda3\lib\site-packages\ibm_db_dbi.py", line 1475, in fetchone
row_list = self._fetch_helper(1)
File "C:\Users\2400566\Anaconda3\lib\site-packages\ibm_db_dbi.py", line 1456, in _fetch_helper
raise self.messages[len(self.messages) - 1]
SQLCODE=-332_dbi::Error: [IBM][CLI Driver][DB2/NT64] SQL0332N Character conversion from the source code page "1252" to the target code page "874" is not supported.
I'm not sure what's wrong? Need advice.
my Python version is Python 3.7.7 running on Window 10 PC x64
DB2 is on Windows server 2012 x64 .
DB2 version is DB2 v11.1.0.1527.
Database territory : GB
Database code page : 1252
Database code set : 1252
Database country/region code : 44
Thanks in advance.
In case that your python script is using db2dsdriver to connect to database on server, try to set the DisableUnicode keyword to 0 to enforce unicode code page (i.e. 1208) on Windows.
https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.swg.im.dbclient.config.doc/doc/r0054636.html
I'm learning Python 3.7, and Sublime Text3. Most programs run fine using control + b, but when I try to import a module from the standard library like random, I get the following error(Please note that I am not running the program from OneDrive, but from my own computer!)
Traceback (most recent call last):
File "C:\Users\chris\OneDrive\Documents\Python Programs\guess.py", line 1, in <module>
import random
File "C:\Users\chris\OneDrive\Documents\Python Programs\random.py", line 1
>>> import os
^
SyntaxError: invalid syntax
Obviously the OneDrive thing is a problem ,but what can I do?
Many thanks.
The part of the error message that you want to focus on here is the last part of the stack trace, which is showing you the line that has the problem and where the problem is, i.e.:
>>> import os
^
SyntaxError: invalid syntax
Based on this, your input file has the characters >>> in front of the import statement which, as the message indicates, is not valid Python syntax. Removing those characters should solve the problem for you. The line immediately above the error is showing you the name of the file that has the problem (in this case random.py).
The >>> is the prompt that the interactive Python interpreter gives you when you run it, as it's prompting you for what text to enter. That's not something that should appear in your code at all.
Check the random.py file, as the error message states, it's a syntax error due to the the >>> repl symbol in there.
I'm running ipython with the following:
c:\python27\scripts\ipython
I'm trying to restore old Python prompt behaviour (">>>") for some reasons.
I've tried to search internet extensively for that purpose, to no avail.
Then I reached for IPython documentation, which turned out to be confusing and unhelpful.
According to http://ipython.readthedocs.io/en/stable/config/details.html
To set the new prompt, assign it to the prompts attribute of the IPython shell:
In [2]: ip = get_ipython()
...: ip.prompts = MyPrompt(ip)
/home/bob >>> # it works
I get the exception that get_ipython is undefined:
[TerminalIPythonApp] ERROR | Exception while loading config file C:\Users\xxx\.ipython\profile_default\ipython_config.py
Traceback (most recent call last):
File "c:\python27\lib\site-packages\traitlets\config\application.py", line 562, in _load_config_files
config = loader.load_config()
File "c:\python27\lib\site-packages\traitlets\config\loader.py", line 457, in load_config
self._read_file_as_dict()
File "c:\python27\lib\site-packages\traitlets\config\loader.py", line 489, in _read_file_as_dict
py3compat.execfile(conf_filename, namespace)
File "c:\python27\lib\site-packages\ipython_genutils\py3compat.py", line 278, in execfile
exec(compiler(scripttext, filename, 'exec'), glob, loc)
File "C:\Users\rgomulk\.ipython\profile_default\ipython_config.py", line 9, in <module>
ip = get_ipython()
NameError: name 'get_ipython' is not defined
Python 2.7.8 (default, Jun 30 2014, 16:03:49) [MSC v.1500 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
(With the following complete code in my ipython_config.py:
from IPython.terminal.prompts import Prompts, Token
class MyPrompt(Prompts):
def in_prompt_tokens(self, cli=None):
return [(Token.Prompt, ' >>>')]
ip = get_ipython()
ip.prompts = MyPrompt(ip)
After next round of googling I've added the following line to the config:
from IPython import get_ipython
This time the result was different:
[TerminalIPythonApp] ERROR | Exception while loading config file C:\Users\xxx\.ipython\profile_default\ipython_config.py
Traceback (most recent call last):
File "c:\python27\lib\site-packages\traitlets\config\application.py", line 562, in _load_config_files
config = loader.load_config()
File "c:\python27\lib\site-packages\traitlets\config\loader.py", line 457, in load_config
self._read_file_as_dict()
File "c:\python27\lib\site-packages\traitlets\config\loader.py", line 489, in _read_file_as_dict
py3compat.execfile(conf_filename, namespace)
File "c:\python27\lib\site-packages\ipython_genutils\py3compat.py", line 278, in execfile
exec(compiler(scripttext, filename, 'exec'), glob, loc)
File "C:\Users\rgomulk\.ipython\profile_default\ipython_config.py", line 11, in <module>
ip.prompts = MyPrompt(ip)
AttributeError: 'NoneType' object has no attribute 'prompts'
So the question is twofold:
1. How can I actually set the prompt/restore old prompt behaviour?
2. Why doesn't the code from IPython documentation work? Is that a bug in implementation or documentation?
IPython version and other versions already given in IPython output.
Regards,
Robert
After extensive search (it was really tedious and others are confused as well, especially from distinguishing between startup and configuration scripts) I've found this page: Jupyter prompts
Which led to (working) solution:
from IPython.terminal.prompts import Prompts
from pygments.token import Token
class MyPrompt(Prompts):
def in_prompt_tokens(self, cli=None):
return [(Token.Prompt, '>>> ')]
c.TerminalInteractiveShell.prompts_class = MyPrompt
(Please notice the lack of what seems to be required by official documentation Official IPython docs:
The files typically start by getting the root config object:
c = get_config()
Regards,
Robert
Another possibility - less generic - is to use predefined Prompt class:
from IPython.terminal.prompts import ClassicPrompts
c = get_config()
c.TerminalInteractiveShell.prompts_class = ClassicPrompts
PsychoPy looks like just what I need. But I want to use my own development environment (a straightforward IPython prompt combined with the editor of my choice) instead of the provided IDE.
The trouble is that you seem to have to quit Python and relaunch after every PsychoPy run. If for example I cd to the ...../demos/coder/stimuli directory and type run gabor.py it runs fine, but if I then type run gabor.py again I get this exception from pyglet:
C:\snap\PsychoPy2\lib\site-packages\pyglet\window\win32\__init__.pyc in _create(self)
259 if not self._wgl_context:
260 self.canvas = Win32Canvas(self.display, self._view_hwnd, self._dc)
--> 261 self.context.attach(self.canvas)
262 self._wgl_context = self.context._context
263
C:\snap\PsychoPy2\lib\site-packages\pyglet\gl\win32.pyc in attach(self, canvas)
261 self._context = wglext_arb.wglCreateContextAttribsARB(canvas.hdc,
262 share, attribs)
--> 263 super(Win32ARBContext, self).attach(canvas)
C:\snap\PsychoPy2\lib\site-packages\pyglet\gl\win32.pyc in attach(self, canvas)
206 raise RuntimeError('Share context has no canvas.')
207 if not wgl.wglShareLists(share._context, self._context):
--> 208 raise gl.ContextException('Unable to share contexts')
209
210 def set_current(self):
ContextException: Unable to share contexts
Is there some sort of pyglet.cleanup() I can call (analogous to pygame.quit()) to allow PsychoPy scripts to run more than once in the same session? Or other way of avoiding this problem?
I'm using the Standalone PsychoPy distro version 1.81.02, untouched. The problem is not specific to IPython---it also can also be demonstrated from the plain Python prompt if you disable sys.exit and type execfile('gabor.py') twice:
C:\snap\PsychoPy2\Lib\site-packages\PsychoPy-1.81.02-py2.7.egg\psychopy\demos\coder\stimuli>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import sys; sys.exit = lambda x:x
>>> execfile('gabor.py')
0.6560 WARNING Movie2 stim could not be imported and won't be available
1.6719 WARNING Monitor specification not found. Creating a temporary one...
>>>
>>> execfile('gabor.py')
Traceback (most recent call last):
[snip]
File "C:\snap\PsychoPy2\lib\site-packages\pyglet\gl\win32.py", line 208, in attach
raise gl.ContextException('Unable to share contexts')
pyglet.gl.ContextException: Unable to share contexts
I don't know how to undo all the pyglet/psychopy initialisation - neither are really designed for you to do this, so there would be some work here. But I'm not sure it's a good idea anyway to run scripts the way you are doing.
The PsychoPy app itself gets around the by launching each script in a new process. It means that you know the namespace is clean on each run. Running your script on top of the previous one can lead to some really hard-to-find bugs because you don't know in what state the previous script left the memory, graphics card and namespace.
cheers
Jon
The following gives an error
drop function testing();
CREATE FUNCTION testing()
RETURNS text
AS $$
import ctypes
try:
ctypes.windll.LoadLibrary("D:\\jcc.dll")
except:
import traceback
plpy.error(traceback.format_exc())
return ''
$$ LANGUAGE plpythonu;
select testing();
Error message:
ERROR: ('Traceback (most recent call last):\n File "<string>", line 5, in __plpython_procedure_testing_1517640\n File "D:\\Python26\\Lib\\ctypes\\__init__.py", line 431, in LoadLibrary\n return self._dlltype(name)\n File "D:\\Python26\\Lib\\ctypes\\__init__.py", line 353, in __init__\n self._handle = _dlopen(self._name, mode)\nWindowsError: [Error 126] The specified module could not be found\n',)
It works fine in a python interpretor.
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> ctypes.windll.LoadLibrary("D:\\jcc.dll")
<WinDLL 'D:\jcc.dll', handle 410000 at 1d9cb10>
>>>
"The specified module could not be found" is one of those helpful error messages Windows emits that doesn't always mean what you think it means.
Windows will produce that message if the DLL you tried to load or any dll it depends on could not be found.
Since PostgreSQL runs in its own user account it has a different PATH to that which your interpreter runs in when you're testing. If jcc.dll depends on (say) c:\jccsupportfiles\aaa.dll and c:\jccsupportfiles is on your PATH but not the Pg server's PATH, that would explain your problem.
Try using Dependency Walker (depends.exe) to determine which DLLs your DLL requires and where they are. See if it's a PATH issue.
Rather than messing with the Pg server's PATH, consider just putting all the DLLs required by jcc.dll in the same directory as jcc.dll. IIRC Windows will always look in the same directory as the module it's loading first when it tries to load a module it depends on.