Adding data to a text file with SED not changing the file size - sed

I have some text files where I need to add 1 character to the beginning of every line of the file.
In windows, I found that a quick way to do this was by installing Cygwin and using the following command, which prepends the letter N to every line of the file:
$ sed 's/^/N/' inputFile.txt > outputFile.txt
What I found strange, was that after I added a new character to the front of each line, the file size was almost completely unchanged. I tested this further, to see if I could recreate the problem with the following steps:
Created a text file called "Test.txt", which had 10,000 lines with the word "TEST" on each line.
Created a text file called "TestWithNPrefix.txt" which had 10,000 lines with the word "NTEST" on each line.
Executed the following command to create another file which had 10,000 lines of "NTEST"
$ sed 's/^/N/' Test.txt > "SEDTest.txt"
Results
"Test" and "SEDTest" were almost the exact same size, while "TestWithNPrefix" was 10KB larger.
Test = 59,998 Bytes; SEDTest = 59,999 Bytes; TestWithNPrefix = 69,998 Bytes
When I ran the "fc" command in Command Prompt, it returned that there were no differences between "SEDTest" and "TestWithNPrefix". "FC" between "SEDTest" and "Test" returned "Resync Filed. Files are too different".
Can someone please help me understand what is causing these file size discrepancies?
EDIT: I created the files "Test.txt" and "TestWithNPrefix.txt" in UltraEdit. I just typed out the word "TEST"/"NTEST", then copied and pasted it 10,000 times.

Not an answer, but a comment with formatting:
You seem to be running into some odd situation with DOS versus Unix line endings. I have to ask: How are you creating the files? I would expect 10,000 lines of "TEST\r\n" would be exactly 60,000 bytes in size, not 59,999
On Linux (I don't have access to a cygwin environment at the moment):'
$ yes $'TEST\r' | head -n 10000 > Test
$ ll Test
-rw-r--r-- 1 jackman jackman 60000 Jan 8 13:06 Test
$ sed 's/^/N/' Test > SEDTest
$ ll *Test
-rw-r--r-- 1 jackman jackman 70000 Jan 8 13:06 SEDTest
-rw-r--r-- 1 jackman jackman 60000 Jan 8 13:06 Test

Related

rsync tries to copy although file is copied

I'm trying to understand the following, maybe one of you guys can help me out by explaining on what exactly happens here:
My goal is to write a script, which copies files from the find-command, parallels rsync-commands and backup those, by re-creating the folder-structure from the source on the destination as well.
Somehow my initial script does not work like that, and i don't really have an idea on how to fix it to behave that way.
Initially, i've copied the files with "ls -1 /foldername" as sourceFolder, which is not best-practice. So i've tried to change it to using "find" as described beyond.
find "$sourceFolder" -maxdepth 2 -mindepth 2 -print0 | xargs --verbose -0 -I {} -P $maxThreads -n 1 rsync -valhP {}/ "$destFolder"/ --checksum --human-readable --stats --dry-run >> "$logDir"/result.log
--
If i run this script, it literaly would recopy the file(s), although those exist in the destination folder i would expect it to be.
sending incremental file list
.d..tp..... ./
>f+++++++++ macs.txt
Number of files: 2 (reg: 1, dir: 1)
Number of created files: 1 (reg: 1)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 242 bytes
Total transferred file size: 242 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.484 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 83
Total bytes received: 22
So as it turned out, sourceFolder and destFolder have to match, and i can't run anything like sourceFolder/folder1/folder1a destFolder/.
Can anyone help me out on what's wrong and why it behaves like that?
Thanks a Lot,
M.

Read .mat files from Unix shell

Is there a way to read .mat files from the UNIX command line shell?
Such as cat mymat.mat ?
I am aware of the possibilities to load it in MATLAB or python, but these are not available for me atm.
GNU Octave may be an option as it can be freely installed without cost.
Say you ran a session something like this and created two arrays, A and B:
octave:1> A = [ 1:3; 4:6; 7:9 ];
octave:2> B = [ 11:13; 14:16; 17:19 ];
octave:3> save -7 myfile.mat A B
Then, in the shell, outside of Octave, you could do this to see the names of the variables in the file:
$ octave-cli <<< "who -file myfile.mat"
Sample Output
Variables in the file myfile.mat:
A B
And then this to dump the variables:
$ octave-cli <<< "load myfile.mat;A"
Sample Output
A =
1 2 3
4 5 6
7 8 9
And:
$ octave-cli <<< "load myfile.mat;B"
Sample Output
B =
11 12 13
14 15 16
17 18 19
No. .mat-files are saved in a binary format. You will not be able to interpret their content from the UNIX shell.

Perl created files with future timestamps on FreeBSD 9.3

I just encountered some strange behavior with Perl 5.16.3 on FreeBSD 9.3-RELEASE-p3. We've got a cron job which runs every five minutes and generates some text status files. I just happened to list the contents of the output directory and saw that the timestamps for some of the files were in the future! The files are created like this:
if (open(OUT, "> $status_file_path")) {
print OUT "$status_info\n";
close OUT;
}
Now, the file handle OUT is used in several places, however it is opened and closed within the same block as shown above. And like I said, out of ten files, only a few had future dates when displayed using ls.
For example, files with the current date had timestamps like 04/02/2015 20:29:46, files with future timestamps were out in November, e.g. 11/10/2015 09:38:41.
What might be going on here?
EDIT
I've got two tests running:
1) a perl script running a loop of 1000 iterations, sleeping a random time up to 10 seconds between iterations, using the open/print/close logic to create an output file and abort the script if the file's modification time is in the future.
2) a cron entry to touch a test file every minute, e.g. touch /home/test/test_file_date_with_cron.txt
TEST RESULTS
Neither of the tests generated output files with a timestamp in the future.
This is scary.
EDIT 2
Here is the filesystem info, the files are written in the /usr directory.
# df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/gprootfs 2G 133M 1.7G 7% /
devfs 1.0k 1.0k 0B 100% /dev
/dev/gpt/gpusrfs 431G 3.8G 392G 1% /usr
procfs 4.0k 4.0k 0B 100% /proc
EDIT 3
Running the script outside of cron for several hundred iterations didn't duplicate the problem. HOWEVER, I just found some other files, which are created by a CGI script which have the future dates:
-rw-r--r-- 1 test test 5783 Nov 10 2015 Config.xml_20150210_104151
-rw-r--r-- 1 test test 34548 Nov 10 2015 Config2.xml_20150210_104151
-rw-r--r-- 1 test test 6105 Nov 10 2015 Config.xml_20151109_232210
-rw-r--r-- 1 test test 34554 Nov 10 2015 Config2.xml_20151109_232210
-rw-rw-r-- 1 root test 2075 Nov 9 2015 Config.xml_20151109_231055
-rw-rw-r-- 1 root test 1232 Nov 9 2015 Config2.xml_20151109_231055
These are archive files, which get moved and renamed with the file's mtime timestamp. Note that BOTH ls and Perl's stat() function report the future date -- stat() is used to generate the file's timestamp portion of the name.
Looking at the first entry, ls reports "Nov 10 2015", whereas when the CGI script processed it, Perl's stat() reported "20150210_104151", i.e. "Feb 02 2015" which is most likely correct.
Further down, we see ls showing "Nov 10 2015" and stat() reported "20151109_232210", i.e. "Nov 09 2015".
Finding those additional archived config files helped me track down the cause, which was as others have suggested, that the system date and timezone changed.
From: 1447147328 and America/Adak
To: 1426637771 and America/New_York
What was throwing me off, was that I thought the cron script wrote ALL of the output files each time it executes, but that's not the case. The files have different "refresh intervals".

Wierd behavior of db2ls when run from directory containing spaces

I am facing a strange issue with getting the db2 version using db2ls.
Below are 2 instances of db2ls execution
[root#dummy 6]# cd /tmp
[root#dummy tmp]# db2ls
Install Path Level Fix Pack Special Install Number Install Date Installer UID
---------------------------------------------------------------------------- -----------------------------------------
/opt/ibm/db2/V10.5 10.5.0.3 3 Tue Mar 10 01:38:53 2015 PDT 0
[root#dummy tmp]# mkdir test\ dir
[root#dummy tmp]# cd test\ dir/
[root#dummy test dir]# db2ls
/usr/local/bin/db2ls: line 43: cd: /tmp/test: No such file or directory
Install Path Level Fix Pack Special Install Number Install Date Installer UID
---------------------------------------------------------------------------------------------------------------------
/opt/ibm/db2/V10.5 10.5.0.3 3 Tue Mar 10 01:38:53 2015 PDT 0
It looks like db2ls is having issues when executed from a directory having spaces. Is this a known issue? I could not find any documentation for this. I am trying to circumvent this problem by using db2ls 2>/dev/null.
If there is a more efficient way please let me know.
http://www-01.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.qb.server.doc/doc/t0023683.html
for DB2 installation paths it says
DB2 installation paths have the following rules:
Can include lowercase letters (a-z), uppercase letters (A-Z), and the underscore character ( _ )
Cannot exceed 128 characters
Cannot contain spaces
Cannot contain non-English characters
Reading statements like this are always a warning sign not to use spaces in paths in general. BTW, shell scripts don't play well with spaces in paths which is for me a good reason to avoid spaces in general.

In Solaris server with Putty, unable to delete copied file with some irregular special character

I use putty to log in to a solaris server. while i was performing a copy operation I pressed left arrow key to edit the file name but it kept adding this character ^[[D desperate I pressed return key and the copy operation got complete
cp temp.jar temp.jar^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D
I was planning to rename is as temp.jar.test, I used 'ls' command to check what has happened and to my surprise two files came up with same name!
root[dev1]# ls -lt temp*
-rw-r--r-- 1 root other 488554 Apr 11 02:25 temp.jar
-rw-r--r-- 1 root other 488554 Apr 11 02:22 temp.jar
-rw-r--r-- 1 root other 488554 Apr 11 02:22 temp.jar.041114
-rw-r--r-- 1 root other 488487 Sep 30 2013 temp.jar.032514
and I used 'rm' command to delete, the original file got deleted but the file copied with ^[[D character is not getting deleted. And I'm getting a msg like 'eisvr.jar.: No such file or directory'
Help me delete the file. I tried issuing 'rm temp.jar^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D'. It resulted in more errors.
The simplest way would be to run this command:
rm -i temp.jar?????????*
and answer yes when prompted to remove the bogus one.