How to maintain an emacs-based knowledge base? - emacs

Question

I've been using org-mode for a little while, I've kept it really simple for now, with only two files :
one to act as an inbox, with remember-mode
another where I stick just about anything that's been processed from the inbox
This is great for managing somewhat 'actionable' items, but I keep adding things of a more general nature, that I won't be needing on a day-to-day basis (how-tos, reading notes etc), so it's getting slow and hard to manage.
The material I'm concerning myself with doesn't fit the /projects/tasks/sub-tasks paradigms, they are more like little knowledge nuggets on selected topics, which are inherently more complex to classify and manage.
I've been wondering what kind of structure could be used to handle that kind of information (classification and retrieval), and if there are maybe other modes that could help with the job ?
I guess there is no pre-made answer to this question, since everyone may have different needs.
Noufal gave good conceptual tips that I'll keep in mind, but overall, the accepted answer provided more pragmatic views on this, the linked resource was a GREAT read.

Answer 1

I think this excellent document on how to use org-mode to it's fullest potential will be very helpful to you: "Org Mode: Organize Your Life In Plain Text". It is lengthy reading, but trust me, completely worth the effort.
UPDATE:
You can use the remember-mode section mentioned in the document for your use-case. (I use it for the same use-case) Remember-mode is extremely handy to make quick notes. I use it when I have to store random observations or information that won't go anywhere else. I use the following templates for remember:
(setq org-default-notes-file (concat org-directory "/remember-notes.org"))
(setq org-remember-templates
`(("Todo" ?t "* TODO %?\n %i\n" ,(concat org-directory "/remember-notes.org") bottom)
("Misc" ?m "* %?\n %i\n" ,(concat org-directory "/Notes.org") "Misc")
("iNfo" ?n "* %?\n %i\n" ,(concat org-directory "/Notes.org") "Information")
("Idea" ?i "* %?\n %i\n" ,(concat org-directory "/Notes.org") "Ideas")
("Journal" ?j "* %T %?\n\n %i\n" ,(concat org-directory "/journal.org") bottom)
("Blog" ?b "* %T %? :BLOG:\n\n %i\n" ,(concat org-directory "/journal.org") bottom)
))
As you can see, misc notes and other information goes in the notes.org file under the headings Misc and Information. If the note I'm making doesn't fall in any of the categories defined above, it gets filed in the default file (remember-notes.org) and I can always refile it to another location at a convenient time. This makes my note-taking, jotting down random ideas, and such things extremely simple, without taking away the focus from the job I'm currently doing.

Answer 2

[org-mode] is great for managing somewhat 'actionable' items, but I keep adding things of a more general nature, that I won't be needing on a day-to-day basis (how-tos, reading notes etc), so it's getting slow and hard to manage.
I'm a follower of David Allen and his Getting Things Done methodology. I'm using Emacs for three of the lists he recommends:
Next Actions
Project Resources
The Someday/Maybe list
The material I'm concerning myself with doesn't fit the /projects/tasks/sub-tasks paradigms, they are more like little knowledge nuggets on selected topics, which are inherently more complex to classify and manage.
I've been wondering what kind of structure could be used to handle that kind of information (classification and retrieval), and if there are maybe other modes that could help with the job ?
For this kind of information I've migrated away from emacs. Instead I keep a directory ~/etc/howto, and in that directory I put files that contain "little knowledge nuggets on selected topics", where the key criterion is that the information has long-term value.
I could search this directory with Emacs, but my Emacs Lisp is not so hot, so I wrote a howto shell script instead (some error checking omitted for clarity):
case $# in
1) ;;
*) echo "Usage: $0 <topic>" 1>&2; exit 2 ;;
esac
topic="$1"
# Note the ordering: first exact matches, then beginning matches, then any matches
set xxx `find $HOME/etc/howto/. -name "$topic" -not -type d -print` \
`find $HOME/etc/howto/. -name "${topic}?*" -not -type d -not -name '*~' -print` \
`find $HOME/etc/howto/. -name "?*$topic*" -not -type d -not -name '*~' -print`
shift
case $# in
0) echo "No file found matching *$topic*" 1>&2 ; exit 1 ;;
*) for i
do
less "$i"
done
;;
esac
Examples include:
howto football brings up three nuggets, in this order:
Instructions to give to my wife on how to record a football game on the computer
Instructions for me as exactly what to take and how to dress when I have tickets to a football game
Instructions for transcoding a football game so it can be transmitted over the net and viewed away from home
howto filesystem brings up instructions on how to copy a filesystem
howto batteries brings up a list of recommended rechargeable batteries
One reason I don't use Emacs is that my real script is a little more complicated than what you see above: it also handles PDF and djvu files, so for example howto razor brings up a djvu document of the manual which came with my electric razor.
I have over 500 items in the main directory or in subdirectories, and even at this scale the system works quite well for me. I hope you may find it helpful too.

Answer 3

I personally keep a list of project directories with somewhat similar structure. Each has a tasklist.org, a tracking subdirectory (where I do project estimates and time tracking and always maintain a diary which is the main thing for the project - it will have links to other files for the project), a docs subdirectory which usually consist of stuff I'm going to publish (docs for the project, proposals etc.). I get my agenda-files to the tasklist.org in each of the subdirectories so that my agenda works fine.
I think the organisation of the data would change a little in your case (perhaps topics like "functional programming" etc.). I'm sceptical of how much a hierarchical structure would help since that would confine you into one way of looking at things (tags vs. folders again). Here are some things that come to mind.
Keep a 'master' org file that has links to all the 'interesting' top level pieces of the other content (similar to the diary I mentioned above).
Tag all your material properly (you'll settle on a set of useful tags after a while) and then use the Tag search feature to search through the files quickly. This assumes that all the files are in your agenda-files though.
Finally, if your data is too exotic to put into a structure, you can consider using a full text indexer (like xapian) and integrate that into your Emacs. There was some discussion of this over here.
Update : 26/Nov/2019
I recently came across the hyperbole package which claims to be a information organiser. I haven't used it but I'm quite tempted to and will update this when I do.

Answer 4

I have tried several ways to manage a knowledge base in the past. I have a bunch of "knowledge nuggets" (btw, thanks, I like that term a lot) on all sorts of diverse topics ranging from how to set up apache tomcat ssl cert, to checklists for doing a monthly family budget, to keeping a list of weights and reps completed on workouts.
I've tried keeping these in a wordpress blog, on a personal wiki, using pen and paper, etc.
In the end, emacs and org-mode is the clear winner for me. I love to have the ability to start simple, and build more complex functionality as I need it. I've used a lot of the tips described by Sacha Chua.
In my case, I always end up with a bunch of notes (more formal and organized) mixed in with action items (less formal). In general I maintain one master "action item" list and then create a separate file for notes on each topic. So far, grep has worked well for me to quickly find the file containing notes. I'll often create a emacs bookmark C-x r m to quickly navigate to notes files as well.
Simple Blogs, CMS, and wikis (like Drupal and Wordpress) are good at classification and retrieval. Maybe you could export org files to html and publish them to a blog, cms or wiki? It might not be too difficult to hook into the blog/wiki/cms tagging capability.

Answer 5

At work, we use a wiki (actually, several - a global wiki plus a wiki per project) for this. It's ideal for non-hierarchical data, but can also be used for hierarchical data. It's formattable, hypertextual, linkable-to, searchable, shareable but also ownable, it maintains history, and other good stuff.
Personally, i used to also use a wiki for this. But these days, i generally just forget things instead. Far easier.

Answer 6

You can speed up org-mode by keeping your notes in a separate file (or one for each broad topic) which is not included in the usual list of agenda files. Customise org-agenda-files to see the list.
Use org-remember to enter notes quickly without interrupting your flow. Either tag them at the time, or save them somewhere to be refiled later. You can use a tag in the remember template (customise org-remember-templates) to mark notes for refiling, and use a custom agenda search (org-agenda-custom-commands) to list them.
Tag each note with relevant topics, and use the agenda view's search facilities to find them. You can define a custom search which knows to look in the right files, or you can visit the file and restrict the agenda search to just that file.
I keep a file of notes, and your question has just inspired me to go back and start tagging them all. Works a treat!

Answer 7

I hold tips in .rst (reStructuredText) files in single directory. Each file have own topic.
For search I use M-x occur, or M-x lgrep, or M-x ack.
Web hosted example: http://tips.defun.work/frame.html and it is easy to turn that pages into blog like solution.
Original sources with build script: http://hg.defun.work/tips/
Main advantage of reStructuredText format:
TOC support.
include syntax.
Ability to build JavaScript based full text offline search index in HTML site with TOC, index, reference by Sphinx!!
Remember Markdown suck at extensibility, RST have regular syntax for marking data by your tokens and include any plain text format as inline into your document. Thinks about graphs with dot, prog lang syntax highlighting, etc.

Answer 8

The material I'm concerning myself with doesn't fit the /projects/tasks/sub-tasks paradigms, they are more like little knowledge nuggets on selected topics, which are inherently more complex to classify and manage.
Use bookmarks and Bookmark+. You can create bookmarks for sets of files and directories, in addition to individual files, and you can tag bookmarks or files, a la delicious, for organization and search purposes.