In My SSIS script task I need to refer to ressources (eg. database tables) that contain special european characters. How do I force my script file (C# code) to be stored using another encoding such as UTF-8.
At the moment I have to escaping the characters using \uXXX which is near unreadable.
While I'm waiting on a repro, how I think you would accomplish this is the same manner you would use to save any file out with encoding in VS/SSMS
I added some arabic looking text in there, assuming that would qualify as UTF-8
Closing and re-opening the script task retains the script.
Related
I am new to using Pentaho Spoon. I have about 100 text files in a folder, none of which have file extensions. I have found that if I create a job and move a file, one at a time, that I can simply rename that file, adding a .txt extension to the end. What I'd like to do is create a job that goes through and renames each file and adds the .txt extension. I've tried using the regex, but can't seem to get it to work because there's no file extension.
Any help would be greatly appreciated.
It's a pretty straightforward solution but you need to use a Transformation, as Job steps won't do it, ok?
You need the following steps:
Get File Names: just add your folder and the RegExp ".*" (without the double quotes), so everything is listed. Check if it's ok with "Show filename(s)..." button.
Modified Java Script Value: declare a new_filename var concatenating the desired extension. Remember to click "Get Variables" after adding the script to output the new field.
var new_filename = filename + '.txt';
Process Files: select Operation = Move and filename/new_filename as your source/target filenames.
That's it!
Renaming a group of files is one thing I wouldn't use Kettle for. Why not let the shell do what the shell does best?
rem example for Windows CMD shell
ren absolute-path-to-folder\*. *.txt
This can be done using a Shell job entry, if you find reason to do it in Kettle at all.
I've seen "just use a shell script" answers for this before. Works great if you can guarantee you're Kettle server is on the same OS as the developer workstation. I'm in an environment where the Dev/Spoon instance is Windows, but the Prod/Kettle environment is Linux, so you can't write one script file to rule them all.
As for "Why on earth would you do this?", my scenario is an integration scenario. We're using Pentaho for Data Integration, but a different tool for Enterprise Integration. I want a Pentaho Job to produce an output file, and I want my Enterprise Integration tool to pick up the file and do something with it, but not before Pentaho is done writing the file. Renaming helps avoid a race condition when the Enterprise Integration solution recognizes the file is there, but Pentaho isn't done writing it yet.
If I could rename a set of files, for example change from test..csv.processing to test..csv, then Pentaho would create the file initially with the .processing extension, and then remove the extension once it's done. The Enterprise Integration solution that's looking for test.*.csv won't start processing the file until Pentaho renames it. Bingo, no race condition.
Is there a way to change the encoding for the deploy script generated by Sql Server Data Tools from UTF-8 to ANSI?
The problem that I have is that I have a string in a function that contains the caracter "è" and in the generated deploy script a box appears instead of this character.
EDIT: I'm using SSDT with Visual studio 2013 and 2015.
You should change encoding of all files in your project to UTF-8 and deploy script will be correct.
I'm attempting to into import oracle 11gR2 using the command line tool for SqlDeveloper 4.0. The ultimate reason is we are attempting to import a lot of freetext fields that need to preserve the exact formatting. CR LF, etc for legal reasons. End users need to edit these in Excel.
SQLLoader baulks at the CR LF's, You can achieve this in SqlDeveloper by switching the formatting to UTF-8 for import / export. We are now trying to build up some scripts after discovering how to do this in the command line runtime sdcli64... BUT there doesn't appear to be an option to import from a flat file or .xlsx in that utility??
Any pointers or are we missing an obvious parameter?
(we are using the latest version of SqlDeveloper we can find, 4.03)
Cheers,
Chris
New version of Oracle developer 4.1 was released as an Early adopter today. You can run the sdcli or sdcli64 command line version with the new parameters. This will import excel files as possible in the sqlDeveloper GUI and it will preserve the formatting using the new [-utility] switch.
You can then use the scripting tool/method of choice to build scripts to do all files in a directory, etc.
With new SQL Developer 4.1 you may import XLSX file (and other formats too) via command line:
Use sdcli utility import.
You will need a config XML file. To create one, start the import in the UI, configure the columns, etc, and at the last step click the button 'Save State'. It will create an XML file you may re-use in command line.
I am encoding every py script in my project to utf-8, as we are definitely migrating our application from Jython 2.2.1 to Jython 2.5.2. For that reason, I have added a 'magic comment' at the first line of every py file (#encoding=utf-8) and I have started testing whether everything is OK by debugging the application in Eclipse.
The problem appears in a script that contains the string straße, because it is automatically converted to straße.
My doubt is if this change is caused by Pydev or it happens because utf-8 doesn't cover this kind of characters.
What can I do to automatically avoid this issue with other 'strange' strings I haven't detected yet?
Are you sure your .py files use UTF-8 encoding? Try to open it with WebBrowser (as text) and check various encodings. While you see straße if seems that ß is encoded by two bytes (most probably UTF-8) but ensure it is really UTF-8.
Also check in Eclipse settings on Project/Properties. There is Resource panel with "Text file encoding" setting (I use Eclipse only for Java projects and do not know if Pydev uses this setting).
Try such code with PyDev and check if result file contains UTF-8 text:
# -*- coding: utf8 -*-
import codecs
f = codecs.open('strasse.txt', 'wb', 'UTF-8')
f.write('straße'.decode('UTF-8'))
f.close()
My guess is that you had a different encoding at that file (say cp1252, which is the default windows encoding) and when you put utf-8 it became garbled (so, it wasn't really PyDev who garbled it, but the fact that it was previously in another encoding).
While you're at it, also make sure you also set the default encoding for Eclipse to utf-8 (which is usually the default platform encoding) -- you can do this at preferences > general > workspace.
As a note, I believe the most common way of putting that comment is #coding: utf-8, followed by #-*- coding: utf-8 -*- (i.e.: not #encoding:utf-8) -- although all those formats work (see pep: https://www.python.org/dev/peps/pep-0263/)
We have a HSQL .script file in source control. Some of our developers use Linux, some use Windows. Each time there is a commit we have to deal with conflicts (each line in file has one) due to platform specific newline characters in script.
Is there a way to specify newline format for the HSQL script file.
You cannot specify the end of line (eol) format for the HSQLDB script. HSQLDB can read the .script file regardless of the eol format used when the file was saved.
Source control system usually allow to specify the eol format to use for text files. For example, Subversion has a svn:eol-style property which can be set to "native" for all or individual files.