I am currently working in SQR process, and had encountered an issue during file processing as it results to error due to special character, slanted apostrophe (’). It is a flat file saved as UTF-8. I tried changing it as ANSI and worked well, but how do I make my code work without changing the encoding?
On your open statement, try adding:
ENCODING=UTF-8
Related
I've got a little batch file and it looks like this:
.\batchisp.exe –device at32uc3b1512 –hardware usb –operation erase f memory flash blankcheck loadbuffer G3Pro_USB.hex program verify start reset 0
The whole line is fine and works correctly if I run it straight in PowerShell. However, if I run the batch file, it runs this:
.\batchisp.exe ΓÇôdevice at32uc3b1512 ΓÇôhardware usb ΓÇôoperation erase f memory flash blankcheck loadbuffer G3Pro_USB.hex program verify start reset 0
Which does not work, because as you can see, the -'s have changed into ΓÇô's... Can anybody tell me why this is and how to fix it?
This is because the – marks are not - characters. They are actually endashes. These usually are caused by Word's automatic en/emdashing.
Powershell is smart enough to convert the endashes to dashes as "arguments", but cmd is not.
To fix this issue, replace – with -. A regex search/replace that catches all the alternative dash types that works in notepad++ is: [–—‒] to -.
I am trying to import a large csv file (~4.5gb) into Postgres but it keeps throwing the following error:
ERROR: unquoted carriage return found in data
HINT: Use quoted CSV field to represent carriage return.
CONTEXT: COPY abc_complete_file_261115, line 9041959
I opened my csv in SublimeText2 and jumped to line 9041959, found the URN for record I needed, loaded the file in Vim and went to that line. I have hidden characters enabled in Vim (by using :set list) so I would expect to see a carriage return ^M somewhere on the line within the data but the only one I could find is at the end of the line as expected.
After an entire day of research and having gotten no further with this issue I ended up deleting the record on line 9041959 - this didn't fix the issue.
Then I figured well maybe it's something strange going on between records - so I ended up deleting about 5 records on either side of the line that threw the error - but it gave the the same error again. (I'll worry about preserving the data later on, right now I'm just trying to import the file so that I can have a look in Postgres). I made sure that I had saved the changes to the csv file before rerunning my query but it just gave the same error.
I feel like I am missing something really really obvious - does anyone have any ideas what might be causing the issue?
I'm using a Mac running El Capitan.
Many thanks
Update 27/11/15
Hi #JakubKania. Sorry for not putting up the query - the reason I didn't was because I am 99.9% sure that the issue is to do with the csv file rather than the query. A generalised version is:
CREATE TABLE large_file_test(
urn VARCHAR,
forename CHAR(32),
surname CHAR(32));
COPY large_file_test FROM '/Users/Shared/largefile1.csv' (FORMAT CSV, DELIMITER ',', HEADER, ENCODING LATIN1);
COPY large_file_test FROM '/Users/Shared/largefile2.csv' (FORMAT CSV, DELIMITER ',', HEADER, ENCODING LATIN1);
COPY large_file_test FROM '/Users/Shared/largefile3.csv' (FORMAT CSV, DELIMITER ',', HEADER, ENCODING LATIN1);
ALTER TABLE large_file_test
ADD CONSTRAINT large_urn
PRIMARY KEY (large_urn);
ANALYZE large_file_test;
So I am actually trying to load 3 separate files into the Table that I created. The issue is that there seems to be hidden characters in part 1 that are preventing it from importing into Postgres. I haven't tried anything with part 2 or 3 yet.
The easiest way I found to solve this in MAC -El Capitan is:
1) Open the file with Sublime Text
2) in menu Reopen the file with encoding UTF8
3) in menu Save the file with encoding UTF8
Sublime "normalize" all end of line EOF.
This likely is caused by Windows line endings. Try installing the utility dos2unix and running dos2unix <filename> before executing the COPY command.
In my case, I noticed that the csv file had an extra blank at the end. After removing it, the file imported properly.
I created a separate folder and gave read/write permissions to "everybody" and that solved all this problem as well as the problem of access being denied when trying to import the file through pgAdmin4 as well. Seems to have been the "cure all".
Now, just to find out which user I need to give these permissions to instead of "everybody".
Using PostgreSQL v 9.6 on Windows 10.
I have an XML file from a third party and it is broken up into non-static length lines delimited with Line Feeds.
There is also a strange first character that needs to be stripped from the file.
I am fairly certain that the two items above are causing issues loading the XML file with SSIS. If I remove them by hand (from a smaller sample), the XML file loads fine thru SSIS.
At this point I've tried using both SSIS and Powershell to attempt to fix the file.
SSIS resulted in strangely adding delimiters back in.
My attempts using Powershell were a little more fruitful on a small subset of the data, but my actual file will regularly be in excess of 2GB and I keep getting System.OutOfMemory exceptions. I have increased the "maxmemorypershellmb" for Powershell (and restarted winrm) but it seems to have had no effect preventing the OutOfMemory error.
Current Powershell attempts that work with a small sample file:
-join ((Get-Content "C:\Code\input.xml") -replace '^\w$', '') | Set-Content -Path "C:\Code\output-nolinefeed.xml"
-join ((Get-Content "C:\Code\output-nolinefeed.xml").substring(1) | Set-Content -Path "C:\Code\output-nolinefeed-removefirstchar.xml"
If both of the above commands run on the sample file, it will then import into SSIS (XML Source -> Derived Column -> SQL Server Destination).
Looking at it in Notepad++ (below), see how it appears to have some type of black smudge on the "less than" character. The other "less than" characters all appear normal. Also, the neon green is some type of space I can select as well.
When I open the file, it is UTF-8. Choosing Encoding -> Covert to ASCII results in the space and "less than" character being converted to a single question mark.
Reasons for weird first character and line feeds aside (assuming both cannot be handled with SSIS), at this point I need some command line editor that will gracefully handle large files.
I would use the Foxe editor for monster XML files:
http://www.firstobject.com/dn_editor.htm
It is astoundingly quick and seems reliable as far as I've used it.
Apparently it has a scripting language based on C++, but I havent tried that.
I made Localizable.string files and compiled my project (my source code uses NSLocalizedString macro function) but my project doesn't compile because of the Localizable.string file. If I comment all the lines in the Localizable.string file, my project compiles successfully.
As result, the problem is related with the Localizable.string files. I searched about it on Google, I found that UTF-8 files (Localizable.string) has changed to UTF-16. And though I tried this... this way didn't work, too.
===============================================================
My Localizable.string file contains:
"LOCAL_APP_GRADE" = "Basic"
"LOCAL_APP_LAST_UPDATED_DATE" =
"2011/04/20"
"LOCAL_MAIN_MENU_TITLE" = "Main Menu"
In my source code:
NSLocalizedString( #"LOCAL_MAIN_MENU_TITLE", #"" );
Error message:
Copy .strings file Error Validation failed: The data couldn't be read because it has been corrupted.
I'm assuming Xcode 4 here. Double check what it shows for the encoding on each of the Localization.string files in the file inspector. When I was having that error it was due to one of the files being read as Mac Roman instead of UTF-16. Once I changed the encoding the warning went away. What was driving me nuts at first was that the warning was only happening in Xcode 4. Xcode 3 did not give it.
You also have an issue with the formatting of your .string file. All of the lines should end in a semicolon.
"LOCAL_APP_GRADE" = "Basic";
"LOCAL_APP_LAST_UPDATED_DATE" = "2011/04/20";
"LOCAL_MAIN_MENU_TITLE" = "Main Menu";
I don't think this is the cause of the warning though. At least I've never seen a warning for it. It usually only manifests itself at runtime when LOCAL_MAIN_MENU_TITLE shows up in app instead of Main Menu. It would be nice if the build process did check for semicolons though. It's easy to miss adding one when editing the files.
Per Apple:
If you run into problems during testing and find that the functions
and macros for retrieving strings are always returning the same key
(as opposed to the translated value), run the /usr/bin/plutil tool on
your strings file. A strings file is essentially a property-list file
formatted in a special way. Running plutil with the -lint option can
uncover hidden characters or other errors that are preventing strings
from being retrieved correctly.
Fire up a console window, go into your project folder, and do:
/usr/bin/plutil -lint ja.lproj/Localizable.strings
(obviously replace the correct language folder name). This will tell you exactly where the problem is!
All of the lines in .strings file should end with a semicolon. It worked for me.
I had the same issue today after importing the localisations from Apple Notes and the cause was really subtle. The standard double quotes had been swapped with slanting double quotes.
Slanting double quote: ”
Standard double quote: "
I've been struggling with this same error, and ended up having a couple similar issues, but with different details. First off, even though it appears that Xcode's internal "builtin-copyStrings" tool should be able to handle either little-endian or big-endian UTF-16 files, it really only handles big endian. My theory is it's running some extra validation step (perhaps using the plutil command line utility) that didn't used to happen in Xcode 3, and that tool barfs on anything but big-endian UTF-16. Not entirely sure though.
The second trick is that you need to make sure your strings files are saved with no BOM (Byte Order Marker). I did some editing of my .strings files in BBEdit, which ended up saving a BOM to the file, and that also appears to make Xcode 4 have a conniption fit. Xcode itself doesn't appear to have any way to remove the BOM from the file, so this has to be done in a text editor such as BBEdit or TextWrangler which can do that for you.
With Xcode 10.1, one missing semicolon stops compilation with this error:
Localizable.strings: read failed: Couldn't parse property list because
the input data was in an invalid format
BTW, you can find out if the error is general to your file (an encoding issue) or specific to one or more lines by temporarily removing most of the file content. You can then locate the problem by incrementally adding content back in until the error returns.
Perhaps you have something like:
"Bla"="bla";;
Note the duplicate ; symbol. If you have that, it will compile properly but will fail in run time.
Related to this - take care when manually merging strings into Localizable.strings - I managed to copy/paste BOTH strings from a NSLocalizedString() macro, so that the Localizable.strings entry was in this form:
"KEY" = "STRING", #"COMMENT STRING COPIED ACROSS ALSO, IN ERROR";
The bit ,#"xxx" on building caused me to get the error:
Read failed: The data couldn't be read because it isn't in the correct
format.
In this case doing a quick search on #" helped identify the places I'd done this.
I had this problem. My fix? Add a newline at the top of the file. Bizarre but it got it working for me.
So instead of at the top of my file having this:
/* comment */
"LOCAL_APP_GRADE" = "Basic"
I had to do:
[newline]
/* comment */
"LOCAL_APP_GRADE" = "Basic"
(Can't get the formatting right - don't type 'newline', just hit return!)
Looks like this is an standard message for error reading the strings file.
In my case it was a (json force of habit) colon instead of equal sign:
"key1" = "String1";
"key2" : "String2";
"key3" = "String3";
Changed it to = and everything worked fine.
My problem was, that I've forgotten ; in one of the lines
Make sure that you have declared string in following format:
"YOUR_STRING_LABEL" = "Your message";
Note: Don't forget Quotation Marks ("), Equals Sign (=) and Semicolon (;) at end.
This may be because the translation file format is wrong. You can download a mac software called Localizable.
This is the download link: https://apps.apple.com/cn/app/localizable-翻译文件工具/id1268616588?mt=12
You only need to drag Localizable.strings file to the software. and it willtell you which line in the file may have a problem.
It is useful .It saved me a lot of time. Now I share it with you, I hope it will be helpful to you.
My Localizable.strings file has somehow been corrupted and I don't know how to restore it.
If I open it as a Plain Text File it starts with weird characters that I can't copy here.
If I leave the file be the app builds. If I make any changes either the values aren't interpreted properly or I get an error at compile time.
Localizable.strings: Conversion of string failed. The string is empty.
Command /Developer/Library/Xcode/Plug-ins/CoreBuildTasks.xcplugin/Contents/Resources/copystrings failed with exit code 1
I suspect this is an encoding problem but I don't know how it happened (maybe SVN is to blame?) nor how to solve it. Any tips will be much more appreciated.
I have issues with the same file that sound very similar to your own. What happens for me is that Xcode doesn't know the correct file formating. I often get this when rearranging the project and I remove and re-add this file to the Xcode project. When I re-add the file, its encoding gets set to something like Western Roman which can't seem to render anything other than ASCII.
Here's what I do to fix the problem:
In Xcode select the Localizable.stings file in the Groups&Files panel.
Do a Get Info on that file.
On the info panel select the General tab.
In that tab go to the File Encoding and change its value.
The last step is where the trick lies as you now have to guess the right encoding. I find that for most European languages that "Unicode (UTF-8)" works. And for Asian languages I find that "Unicode (UTF-16/32)" are the ones to try.
I just had that error because I forgot a semicolon. Took me a while to figure it out. Seems like a really ambiguous compiler error but the fix was simple.
Make sure in File-Get Info, that UTF-16 is selected. If it's set to none or UTF-8 as encoding then you need to change it. If your characters have spaces between them then you choose to "re-interpret" the file as UTF-16. If there are weird characters in the file, then you need to remove them.
Execpt the UTF-8 problem, sometimes you still have to check the content in case if there are some syntax problems.
Use the following Regular Expression to verify your text line by line, if there's any line not matched, there must be a problem.
"(.+?)"="(.+?)";
You can use the plutil command line tool. Without options or with the -lint option, it checks the syntax of the file given as argument. It will tell you more precisely where the error is.
This happens to me when there is a missing quote or something not right with the file. MOst commonly, since my language files are done by another team member, he tends to forget a quote or something. Usually XCode shows an error on that line, sometimes it does'nt and just throws "Corrupted data" error.
Double check if all your strings are properly closed in quotes
Open the file in Xcode.
Right click it in Project Navigator.
Select Open as -> ASCII Property List