Emacs: import a CSV into org-mode - emacs

Emacs has a very nice extension by the name of org-mode.
I would like to be able to easily load CSV files into org-mode without significant grief. All I've been able to find is table-import or table-capture, which, simply put, don't work even approximately well.
Note that part of my issue is text strings with a comma within them. 1,2,3,4 is different than 1,2,"3,4".
Is there a function out there or a perl script that one could run to transform a csv file into org-mode format?
Thanks!

From the org-mode manual:
C-c | Convert the active region to
table. If every line contains at least
one TAB character, the function
assumes that the material is tab
separated. If every line contains a
comma, comma-separated values (CSV)
are assumed. If not, lines are split
at whitespace into fields. You can use
a prefix argument to force a specific
separator: C-u forces CSV, C-u C-u
forces TAB, and a numeric argument N
indicates that at least N consecutive
spaces, or alternatively a TAB will be
the separator. If there is no active
region, this command creates an empty
Org table.
So just paste the data into an org file, select it, and do C-u C-c | .

Have a look at:
C-h f org-table-convert-region
I'm always converting csv so I added this to my .emacs.
(defun org-convert-csv-table (beg end)
(interactive (list (mark) (point)))
(org-table-convert-region beg end ",")
)
(add-hook 'org-mode-hook
(lambda ()
(define-key org-mode-map (kbd "<f6>") 'org-convert-csv-table)))
update
Here is another function that considers the quoted commas as well :
a,"12,12",b --> a | 12,12 |b
feel free to improve it :-).
(defun org-convert-csv-table (beg end)
; convert csv to org-table considering "12,12"
(interactive (list (point) (mark)))
(replace-regexp "\\(^\\)\\|\\(\".*?\"\\)\\|," (quote (replace-eval-replacement
replace-quote (cond ((equal "^" (match-string 1)) "|")
((equal "," (match-string 0)) "|")
((match-string 2))) )) nil beg end)

I'm assuming you want to convert your CSV specifically into org-mode tables. If that's not the case, you may want to be more explicit about output format in your question.
Something like this should do it, or at least get you a starting point you can hack on:
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
my $csv = Text::CSV->new();
while ( my $line = <DATA> ) {
if ( $csv->parse( $line )) {
my $str = join '|' , $csv->fields();
print "|$str|\n";
}
}
__DATA__
1,2,3,4
1,2,"3,4"

Try this:
;; Insert a file and convert it to an org table
(defun aleblanc/insert-file-as-org-table (filename)
"Insert a file into the current buffer at point, and convert it to an org table."
(interactive (list (ido-read-file-name "csv file: ")))
(let* ((start (point))
(end (+ start (nth 1 (insert-file-contents filename)))))
(org-table-convert-region start end)
))

Here is a bit of a hack-job, but it works.
when you export the CSV file, force quotes around each entry, then replace all "," as a vertical bar.
Finally, I make a macro that does something like this:
C-a ; Beginning-of-line
| ; Self-insert-char
C-e ; end-of-line
| ; Self-insert-char
<down> ; Down one line
(I am not 100% sure if | is a self-insert-char or not, so its best to record your own macro)
Hit tab somewhere in the middle of the table, and org-mode formats it properly. I find this easier to do in a narrowed region.
Caveat: If you have a vertical bar in your input.. it probably won't work quite right. Situations like that should be easy to spot, and relatively easy to fix.
Generally, when importing something text-like into org-mode, I have found a combination of some smart macro writing, and search-replace to be the easiest way
I hope that helps.

Related

Creating org-tables from the results of a code block

I am trying to create an org-table based on the content of a CSV file that uses ";" as a delimiter.
I thought it would be easy to have a source code block to "cat" the content of the file and then pass the result to a function that creates the table, but I got stuck: I can't find a way to use the "results" of the first source code block. The function I know (org-table-convert-region) expects a region to work on and I don't know how to pass the "cat"-ed text as a region.
#+NAME: csvraw
#+BEGIN_SRC sh :results raw
cat afile.csv
#+END_SRC
I'd appreciate your help to generate a code block that produces an org-table out of my csv file, which contains lines like the following:
ID;Region;SubRegion;Area;No
1234;Asia;India;45;2
24251;Europe;Romania;456;67
There is org-table-convert-region (bound to C-c |) which can do the transformation fairly simply. The only trick is to specify ; as the separator. You can do that by invoking it with the proper prefix argument - the doc string says:
(org-table-convert-region BEG0 END0 &optional SEPARATOR)
Convert region to a table.
The region goes from BEG0 to END0, but these borders will be moved
slightly, to make sure a beginning of line in the first line is included.
SEPARATOR specifies the field separator in the lines. It can have the
following values:
(4) Use the comma as a field separator
(16) Use a TAB as field separator
(64) Prompt for a regular expression as field separator
integer When a number, use that many spaces, or a TAB, as field separator
regexp When a regular expression, use it to match the separator
nil When nil, the command tries to be smart and figure out the
separator in the following way:
- when each line contains a TAB, assume TAB-separated material
- when each line contains a comma, assume CSV material
- else, assume one or more SPACE characters as separator.
The (64) value is just three C-u in a row, so the process is as follows:
insert the CSV file with C-x i.
C-x C-x to mark the inserted contents as the active region.
C-u C-u C-u C-c | ; RET
What's even cooler, leaving an empty line in the CSV file between the first line and the rest of the lines, will make the first line a header in the table automatically.
And you can wrap it up in a code block as well:
#+begin_src elisp :var file="/tmp/foo.csv" :results raw
(defun csv-to-table (file)
(with-temp-buffer
(erase-buffer)
(insert-file file)
(org-table-convert-region (point-min) (point-max) ";")
(buffer-string)))
(csv-to-table file)
#+end_src
#+RESULTS:
| a | b | c |
|---+---+---|
| d | e | f |
| g | h | i |
(defun jea-convert-csv-to-org-table (fname)
(interactive "fCSV to convert: ")
(let ((result '("|-\n")))
(with-temp-buffer
(save-excursion (insert-file-contents-literally fname))
(while (and (not (eobp)) (re-search-forward "^\\(.+\\)$" nil t nil))
(push (concat "|" (replace-regexp-in-string ";" "|" (match-string 1)) "|\n")
result))
(push '"|-\n" result))
(concat (seq-mapcat #'identity (reverse result)))))
install the elisp code in your ~/.emacs file and restart emacs. Or, better yet, eval it into existence (CTRL+x, CTRL+e or ALT+x eval-last-sexp).
#+NAME: csvraw
#+BEGIN_SRC elisp :results raw
(jea-convert-csv-to-org-table "/Users/jamesanderson/Downloads/test1.csv")
#+END_SRC
note the change to elisp from sh in the above. here is a gif of it in action:
emacs converting csv to org table
The code is not optimized for large files, and, to be frank, quite quickly thrown together. But, after a tiny bit of testing, seems to work.

Emacs: convert items on separate lines to a comma-separated list

I often paste items separated by newlines or line feeds into an Emacs buffer resulting in each item residing on a different line like this:
one
two
three
four
Very often I actually want a list of comma-separated values like this:
"one", "two", "three", "four"
It would be great to be able to do a one-touch conversion from lines to list. I imagine I can convert this using a regex, but it seems like the kind of commonly used operation that might already have a built-in Emacs function. Can anybody suggest one?
M-q would replace linebreaks with spaces (in reasonably short lists of short words) but won't add quotes and commas. Or, maybe M-^ many times, until you have them all on the same line. Other than that - nothing built in comes to mind.
Obviously, a keyboard macro is a good candidate for this.
But a faster way, that doesn't create many undo steps would be something along these lines:
(defun lines-to-cslist (start end &optional arg)
(interactive "r\nP")
(let ((insertion
(mapconcat
(lambda (x) (format "\"%s\"" x))
(split-string (buffer-substring start end)) ", ")))
(delete-region start end)
(insert insertion)
(when arg (forward-char (length insertion)))))
Edit: I do see you're looking for a function... but since the only answer is simply to write your own (i.e. no built-in one exists), I figured I'd chime in with what the regex would be, since others may stumble on this and appreciate an alternative to writing the function and putting it in .emacs.
This is two steps, but only because you wanted your text quoted:
As pasted in Emacs *scratch* buffer (added five six to show it works with multiple words per line, if that's of interest):
one
two
three
four
five six
First, replace individual word with "word":
M-x replace-regexp RET \(.*\) RET "\1" RET produces:
"one"
"two"
"three"
"four"
"five six"
Now, replace each carriage return (in Emacs, C-q C-j) with a ,:
M-x replace-regexp RET C-q C-j RET , RET produces:
"one", "two", "three", "four", "five six"
I wrote a solution for this at work today. Below are functions to convert from lines to csv, and from csv to lines, with a user-specify-able separator. This function operates on the currently-highlighted region.
(defun lines-to-csv (separator)
"Converts the current region lines to a single line, CSV value, separated by the provided separator string."
(interactive "sEnter separator character: ")
(setq current-region-string (buffer-substring-no-properties (region-beginning) (region-end)))
(insert
(mapconcat 'identity
(split-string current-region-string "\n")
separator)))
(defun csv-to-lines (separator)
"Converts the current region line, as a csv string, to a set of independent lines, splitting the string based on the provided separator."
(interactive "sEnter separator character: ")
(setq current-region-string (buffer-substring-no-properties (region-beginning) (region-end)))
(insert
(mapconcat 'identity
(split-string current-region-string separator)
"\n")))
To use this, highlight the region you want to edit, then do M-x and specify the separator to use.
I usually accomplish these kinds of tasks using macros. M-X kmacro-start-macro and M-x kmacro-end-or-call-macro, which you can then do repeatedly then.

(re)number numbered lists in emacs (muse)

suppose I have a text list in emacs like this:
a
b
c
...
d
Is there a way to assign numbers to those items in Emacs, by selecting the region? End results should look like:
1. a
2. b
3. c
j. ...
n. d
Thanks.
The way I do this, which may not be optimal, is to use regex search and replace. This, of course, requires that you be able to define a regex to match the start of the lines you want numbers on. Taking your example, I'd use a search regex like this:
\([a-z]\)
note the capturing brackets, we'll need that first letter soon. And a replace regex like this:
\#. \1
where:
\# is a special form which is replaced, by Emacs, by the right number (though see the warning below);
. writes a stop; and
\1 writes a space and the captured group.
WARNING: Emacs will number your items 0, 1, 2, .... Until someone posts to tell us how to start at 1, I always insert a dummy 0th element before the edit, then delete it.
You can use the Emacs Keyboard Macro Counter.
Put the cursor one line ABOVE your list.
Start a macro: F3
Insert the counter value: C-x C-k C-i. A 0 will appear
Insert the DOT and a space: .
Move the cursor to the next line
Stop the macro: F4
Select your list
M-x apply-macro-to-region-lines
You can delete the 0 you added on the top and enjoy :)
NOTE: This will create a numbered list. It will not use letters.
A much simpler way is to use the CUA library's advanced rectangle editing commands. CUA is included in Emacs (at least 23.1, I think it's in earlier versions as well), so there isn't any new code to get.
You can use cua-set-rectangle-mark (bound to C-Return by default) to start a rectangle, and then use cua-sequence-rectangle to insert increasing values. It also gives you control over the format and starting value, so there is a lot of flexibility.
As an aside, CUA is primarily designed to make Emacs operate more like standard text editors (with C-c for copy, C-v for paste, etc), but it also includes some unrelated niceties, like rectangle editing. Don't ask me why :). If you want to use the rectangle editing without enabling the CUA keybindings (which is what I do), set cua-enable-cua-keys to nil, which can be done via customize.
(defun number-region (start end)
(interactive "r")
(let* ((count 1)
(indent-region-function (lambda (start end)
(save-excursion
(setq end (copy-marker end))
(goto-char start)
(while (< (point) end)
(or (and (bolp) (eolp))
(insert (format "%d. " count))
(setq count (1+ count)))
(forward-line 1))
(move-marker end nil)))))
(indent-region start end)))
Here's some elisp code to do it; would be easy to customize if you like tinkering.
This will number the current region (unless it is already numbered), and also the last line binds to the M-n keys. You could use a function key "[F6]" as needed.
Modified to take a format string to use. The default is 1. but you could do something like %d) to get a bracket instead of a . and so on.
(defun number-region(fmt)
(interactive "sFormat : ")
(if (or (null fmt) (= 0 (length fmt)))
(setf fmt "%d. "))
(save-excursion
(save-restriction
(narrow-to-region (point) (mark))
(goto-char (point-min))
(let ((num 1))
(while (> (point-max) (point))
(if (null (number-at-point))
(insert (format fmt num)))
(incf num)
(forward-line))))))
(global-set-key "\M-n" 'number-region)
Not a direct answer to your question, but if you find yourself manipulating numbered lists frequently, you may want to look into org-mode. In particular, the section on plain lists.

By Emacs, how to join two lines into one?

I am new to Emacs. I have googled this but no good answer there. One of them is
Ctrl-n Ctrl-a Backspace
This works but is stupid. Is there a quick and simple way to join a block of lines into a single line?
Actually, I can use Esc-q to auto-fill a paragraph now, but how could I get it to revert without UNDO?
Place point anywhere on the last line of the group of lines that need joining and call
M-^
repeatedly until all the lines are merged.
Note: It leaves one space between all of the now joined lines.
M-x join-line will join two lines. Just bind it to a convenient keystroke.
Multiple Cursors combined with M-^ will collapse all selected lines into one with all extraneous white-space removed.
For example to select an entire buffer, invoke multiple cursors mode, collapse into one line, and then disable multiple cursors mode:
C-x h
M-x mc/edit-lines
M-^
C-g
The Emacs conventional name for "join" is "fill". Yes, you can join
two lines with M-^ -- and that's handy -- but more generally you'll
want to join n lines. For this, see the fill* commands, such as
fill-region, fill-paragraph, etc.
See this for more info
on selecting things which can then be filled.
Also, you can join multiple lines with M-^ by selecting those lines first. (Note that the universal argument does not work with this.)
Just replace newlines with nothing.
I like the way Sublime text Join line with Command J so I do it this way:
(defun join-lines (arg)
(interactive "p")
(end-of-line)
(delete-char 1)
(delete-horizontal-space)
(insert " "))
You could define a new command for this, temporarily adjusting the fill width before using the the Esc-q command:
;; -- define a new command to join multiple lines together --
(defun join-lines () (interactive)
(setq fill-column 100000)
(fill-paragraph nil)
(setq fill-column 78)
)
Obviously this only works, if your paragraph has less than 100000 characters.
I use the following function and bind it to 'M-J'.
(defun concat-lines ()
(interactive)
(next-line)
(join-line)
(delete-horizontal-space))
If you prefer to keep your cursor position, you can use save-excursion.
The most simplest way ever:
Select paragraph/lines by M-h or C-SPC
Press M-q
Witness the Emagics (Emacs Magic)!!
Because join-line will left one space between two lines, also it only support join two lines. In case of you want to join plenty of lines without one space left, you can use "search-replace" mode to solve, as follows:
C-%
Query: input C-q C-j Enter
Replace: Enter
Run the replacement. Enter
Done.
Two ways come to mind:
Once you think of it, the most obvious (or at least easiest to remember) way is to use M-q format-paragraph with a long line length C-x-f 1000.
There is also a built-in tool M-^ join-line. More usefully, if you select a region then it will combine them all into one line.
"how could I get it to revert without UNDO?":
(defun toggle-fill-paragraph ()
;; Based on http://xahlee.org/emacs/modernization_fill-paragraph.html
"Fill or unfill the current paragraph, depending upon the current line length.
When there is a text selection, act on the region.
See `fill-paragraph' and `fill-region'."
(interactive)
;; We set a property 'currently-filled-p on this command's symbol
;; (i.e. on 'toggle-fill-paragraph), thus avoiding the need to
;; create a variable for remembering the current fill state.
(save-excursion
(let* ((deactivate-mark nil)
(line-length (- (line-end-position) (line-beginning-position)))
(currently-filled (if (eq last-command this-command)
(get this-command 'currently-filled-p)
(< line-length fill-column)))
(fill-column (if currently-filled
most-positive-fixnum
fill-column)))
(if (region-active-p)
(fill-region (region-beginning) (region-end))
(fill-paragraph))
(put this-command 'currently-filled-p (not currently-filled)))))
(global-set-key (kbd "M-q") 'toggle-fill-paragraph)
From EmacsWiki: Unfill Paragraph
;;; Stefan Monnier <foo at acm.org>. It is the opposite of fill-paragraph
(defun unfill-paragraph (&optional region)
"Takes a multi-line paragraph and makes it into a single line of text."
(interactive (progn (barf-if-buffer-read-only) '(t)))
(let ((fill-column (point-max))
;; This would override `fill-column' if it's an integer.
(emacs-lisp-docstring-fill-column t))
(fill-paragraph nil region)))
A basic join of 2 lines:
(delete-indentation)
I like to line below to be joined to the current without moving the cursor:
("C-j" .
(lambda (iPoint)
"Join next line onto current line"
(interactive "d")
(next-line)
(delete-indentation)
(goto-char iPoint)))
This one behaves like in vscode. So it add space only if join line consisted something else than whitespace. And I bind it to alt+shift+j.
Shorter version based on crux-top-join-line:
(global-set-key (kbd "M-J") (lambda () (interactive) (delete-indentation 1)))
Longer version based on https://stackoverflow.com/a/33005183/588759.
;; https://stackoverflow.com/questions/1072662/by-emacs-how-to-join-two-lines-into-one/68685485#68685485
(defun join-lines ()
(interactive)
(next-line)
(join-line)
(delete-horizontal-space)
(unless (looking-at-p "\n") (insert " ")))
(global-set-key (kbd "M-J") 'join-lines)

How to get Emacs to unwrap a block of code?

Say I have a line in an emacs buffer that looks like this:
foo -option1 value1 -option2 value2 -option3 value3 \
-option4 value4 ...
I want it to look like this:
foo -option1 value1 \
-option2 value2 \
-option3 value3 \
-option4 value4 \
...
I want each option/value pair on a separate line. I also want those subsequent lines indented appropriately according to mode rather than to add a fixed amount of whitespace. I would prefer that the code work on the current block, stopping at the first non-blank line or line that does not contain an option/value pair though I could settle for it working on a selected region.
Anybody know of an elisp function to do this?
Nobody had what I was looking for so I decided to dust off my elisp manual and do it myself. This seems to work well enough, though the output isn't precisely what I asked for. In this version the first option goes on a line by itself instead of staying on the first line like in my original question.
(defun tcl-multiline-options ()
"spread option/value pairs across multiple lines with continuation characters"
(interactive)
(save-excursion
(tcl-join-continuations)
(beginning-of-line)
(while (re-search-forward " -[^ ]+ +" (line-end-position) t)
(goto-char (match-beginning 0))
(insert " \\\n")
(goto-char (+(match-end 0) 3))
(indent-according-to-mode)
(forward-sexp))))
(defun tcl-join-continuations ()
"join multiple continuation lines into a single physical line"
(interactive)
(while (progn (end-of-line) (char-equal (char-before) ?\\))
(forward-line 1))
(while (save-excursion (end-of-line 0) (char-equal (char-before) ?\\))
(end-of-line 0)
(delete-char -1)
(delete-char 1)
(fixup-whitespace)))
In this case I would use a macro. You can start recording a macro with C-x (, and stop recording it with C-x ). When you want to replay the macro type C-x e.
In this case, I would type, C-a C-x ( C-s v a l u e C-f C-f \ RET SPC SPC SPC SPC C-x )
That would record a macro that searches for "value", moves forward 2, inserts a slash and newline, and finally spaces the new line over to line up. Then you could repeat this macro a few times.
EDIT: I just realized, your literal text may not be as easy to search as "value1". You could also search for spaces and cycle through the hits. For example, hitting, C-s a few times after the first match to skip over some of the matches.
Note: Since your example is "ad-hoc" this solution will be too. Often you use macros when you need an ad-hoc solution. One way to make the macro apply more consistently is to put the original statement all on one line (can also be done by a macro or manually).
EDIT: Thanks for the comment about ( versus C-(, you were right my mistake!
Personally, I do stuff like this all the time.
But I don't write a function to do it unless I'll be doing it
every day for a year.
You can easily do it with query-replace, like this:
m-x (query-replace " -option" "^Q^J -option")
I say ^Q^J as that is what you'll type to quote a newline and put it in
the string.
Then just press 'y' for the strings to replace, and 'n' to skip the wierd
corner cases you'd find.
Another workhorse function is query-replace-regexp that can do
replacements of regular expressions.
and also grep-query-replace, which will perform query-replace by parsing
the output of a grep command. This is useful because you can search
for "foo" in 100 files, then do the query-replace on each occurrence
skipping from file to file.
Your mode may support this already. In C mode and Makefile mode, at least, M-q (fill-paragraph) will insert line continuations in the fill-column and wrap your lines.
What mode are you editing this in?