I run into this one while trying to parse a text file with Perl. The original file looks like this in vim:
When I tried to print the 2nd column (87 here), somehow, the ^M showed up in vim:
I'm just curious what this "^M" is? Does anyone know? Thanks!
^M is ASCII character 13, known as a carriage return. MS-DOS uses a carriage return followed by a line feed (ASCII 10) to mark the end of a line. Unix systems use a line feed only. Usually you will "see" a carriage return when using an editor that thinks your file is using Unix style line endings but actually has MS-DOS style line endings.
Related
So I have over 100 text files, all of which are over the size required to be opened in a normal text editor (eg; notepad, notepad++). Meaning I cannot use those mentioned.
All text files contain the same format, they contain:
abc0001:00000009a
abc0054:000000809a
abc00888:054450000009a
and so on..
I was wondering, how do I replace the ":" in each of those text files to then be "\n" (regex for new line)
So then it would be:
abc0001
00000009a
abc0054
000000809a
abc00888
054450000009a
How would I do this to all of the 100 text files, without doing this manually and individually. (if there's any way?)
Any help is appreciated.
You can use sed. The following does something similar to what you want. The question concerns Unix, but a lot of Unix utilities have been ported to MS Windows (even sed): http://gnuwin32.sourceforge.net/packages/sed.htm
UNIX: Replace Newline w/ Colon, Preserving Newline Before EOF
Something like (where you provide your text file as input, and the output becomes your new text file):
sed 's/:/\n/g'
In BBEdit (v11.6), when I search for the "\r" character in a txt file previoulsy saved as "Unix (LF)" from the "Save as..." dialog, the result is the end of each individual line of the file.
Why?
The BBEdit hex dump correctly shows that no CR (OD) chars are present in the file.
From the 11.6 release notes:
BBEdit now uses the line feed (ASCII decimal 10) as line breaks in its internal representation for text in open documents, instead of the carriage return (ASCII decimal 13) that was the standard Mac format for many years. This (theoretically) reduces the time required to open documents, since in the normal case, no conversion is necessary; it also eliminates conversion logic when copying and pasting text, since LF-delimited text is also the standard interchange format on the Clipboard.
As before, you may use \n and \r interchangeably in search strings and Grep patterns. (The latter usage is for compatibility with old versions of BBEdit.)
I am exporting some Access tables to txt files and there are a lot of problems with the txt file. One of those problems being line breaks not visible in the txt file itself. If I copy a line with a line break into Notepad++ from Notepad, it'll break into 2 lines.
So I believe this may be a code format problem, but I can't find the proper one to resolve this. I'm currently exporting to the default Western European, but should I export tot UTF, Unicode, ASCII or something else?
When exporting from MS Access (or VB/VBA in general), make sure you're using vbCrLf constant (Carriage Return plus Line Feed) for line breaks. That constant corresponds to HEX values 0D 0A.
In Windows, it is a convention to use the above 2 characters together as line breaks, while in many other platforms, such as Unix/Linux/MacOS/etc. typically just 0A is used.
That brings up an issue: Notepad, the standard Windows text file viewer, cannot deal with 0A alone and does not treat such symbols as line breaks. More advanced editors, such as Notepad++ or UltraEdit, display such files correctly, though.
The CSV export function in Microsoft Office applications (Excel, Access) terminate a data row with CR+LF and write for a line break within a data value (multi-line string) just LF into the file. (I think just CR was written into the CSV file for a line break in older versions of Office before Office 2007.)
Most text editors detect those LF without CR (respectively CR without LF) and convert them to CR+LF on loading the CSV file which results on viewing of the CSV file in text editor in supposed wrong CSV lines as number of data values is not correct on data rows with data values containing a line break.
However, newline characters within a double quoted value in a CSV file are correct according to CSV specification as described in Wikipedia article about Comma-separated values.
But most applications with support on import from CSV file do not support CSV files with newline characters within a double quoted value and therefore some data values are imported wrong. Also regular expression replaces can't be done on a CSV file with newline characters within a data value because the number of separator character is not constant on all lines.
UltraEdit has for editing such CSV files with only LF (or CR) for a line break within a data value a special configuration setting. At Advanced - Configuration - File Handling - DOS/Unix/Mac Handling the option Never prompt to convert files to DOS format or Prompt to convert if file is not DOS format with clicking on button No if this prompt is displayed must be selected and additionally Only recognize DOS terminated lines (CR/LF) as new lines for editing must be enabled.
The CSV file with CR+LF for end of data row and only LF (or CR) for a line-break within a data value is loaded with those settings in UltraEdit with number of lines equal the number of data rows. And the line-feeds without carriage return (respectively the carriage returns without line-feed) in the CSV file are displayed as character in the lines with a small rectangle as no font has a glyph for a carriage return or line-feed defined because they are whitespace characters with no width. A Perl regular expression find searching for \r(?!\n)|\n(?<!\r) can be used now to find those line breaks within data values and replace them with something different like a space character or remove them.
Which character encoding (ASCII, ANSI, Unicode (UTF-16), UTF-8) to use on export depends on which characters can exist in string values. A Unicode encoding is necessary if string values can have also characters not included in local code page.
I know that file line separaters are very different under certain operating systems, for windows it's CRLF, under linux is LF, and under MacOS is CR. But who on earth named those ascii characters? Are those (LF and CR etc.) abbreviation or something else? And dose every ascii character have a name like this?
CR stands for Carriage Return, LF stands for Line Feed. These names come from the age of typewriters. In order to start writing on the next line, you would push your carriage (the moving part of the typewriter) all the way back to the left, then engage the feed lever to pull the paper one line up.
And yes, other "control characters" have names like those too. See here: http://en.wikipedia.org/wiki/ASCII#ASCII_control_characters
CR stands for "carriage return", which means the returning of the typewriters head to the start of the line. LF is for "line feed" which advances the sheet of paper in the typewriter to the next line.
In most typewriters, CR and LF could be triggered by a single mechanism, but sometimes you also had an additional line feed key to quickly advance to the next line without moving the head (useful for formulars). And you could also omit the LF action on CR in order to write to a given line more than once.
I used http://icalvalid.cloudapp.net/Default.aspx link to validate the link of my calendar. And i got the warning "Line endings must be a Carriage Return/Line Feed (CRLF) pair.". What does this mean? What do I need to this to remove this warning. Can anyone please help me on this issues.
Your help will be appreciated. Thanks.
It means the file has to be in "Windows" format, rather than UNIX format. If you're creating the file from code, it means that you need to write "\r\n" at the end of the lines, rather than just "\n" -- assuming you're using some language where that makes sense, of course. If you're creating the file by hand, save it in "DOS mode" from your editor.
It seems you are running a Linux or Mac OS X box, which uses just the LF (line feed) ASCII value for line endings.
Windows uses CRLF (carriage return; line feed) for line endings.
You must convert this, for example with Notepad++ (though dedicated application exist).
tr -d '\r' < userlist.txt > newlist.txt