Related
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it. I have asked this on the Emacs forum with no success. See:
https://emacs.stackexchange.com/questions/35913/program-rewriting-systems-unexpanded-a-defmacro-given-a-list-of-macros-to-undo
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
Ideally, in certain situations such a tool would be able to work directly off the defmacro to do its pattern find and replace on. However even if I have to come up with specific search and replace patterns manually to add to the transformation system, having such a framework to work in would still be useful
Summary of results so far: Although there have been a few answers that explore interesting possibilities, right now there is nothing definitive. So I think best to leave this open. I'll summarize some of the suggestions. (I've upvoted all the answers that were in fact answers instead of commentary on the difficulty.)
First, many people suggest considered the special form of macros that do expansion only,or as Drew puts it:
macro-expansion (i.e., not expansion followed by Lisp evaluation).
Macro-expansion is another way of saying reduction semantics, or
rewriting.
The current front-runner to my mind is in phils post where he uses a pattern-matching facility that seems specific to Emacs: pcase. I will be exploring this and will post results of my findings. If anyone else has thoughts on this please chime in.
Drew wrote a program called FTOC whose purpose was to convert Franz Lisp to Common Lisp; googling turns up a comp.lang.lisp posting
I found a Common Lisp package called optima with fare-quasiquote. Paulo thinks however this might not be powerful enough since it doesn't handle backtracking out of the box, but might be programmed in by hand. Although the generality of backtracking might be nice, I'm not convinced I need that for the most-used situations.)
Side note: Some seem put off by the specific application causing my initial interest. (But note that in research, it is not uncommon for good solutions to get applied in ways not initially envisioned.)
So in that spirit, here are a couple of suggestions for changing the end application. A good solution for these would probably translate to a solution for Emacs Lisp. (And if if helps you to pretend I'm not interested in Emacs Lisp, that's okay with me). Instead of a decompiler for Emacs Lisp, suppose I want to write a decompiler for clojure or some Common Lisp system. Or as suggested by Sylwester's answer, suppose I would like to automatically refactor my code by taking into account the benefit of using more concise macros that exist or that have gotten improved. Recall that at one time Emacs Lisp didn't have "when" or "unless" macros.
30-some years ago I did something similar, using macrolet.
(Actually, I used defmacro because we had only an early implementation of Common Lisp, which did not yet have macrolet. But macrolet is the right thing to use.)
I didn't translate macro-expanded code to what it was expanded from, but the idea is pretty much the same. You will come across some different difficulties, I expect, since your translation is even farther away from one-to-one.
I wrote a translator from (what was then) Franz Lisp to Common Lisp, to help with porting lots of existing code to a Lisp+Prolog-machine project. Franz Lisp back then was only dynamically scoped, while Common Lisp is (in general) lexically scoped.
And yes, obviously there is no general way to automatically translate Lisp code (in particular), especially considering that it can generate and then evaluate other code - but even ignoring that special case. Many functions are quite similar, but there is the lexical/dynamic difference, as well as significant differences in the semantics of some seemingly similar functions.
All of that has to be understood and taken for granted from the outset, by anyone wanting to make use of the results of translation.
Still, much that is useful can be done. And if the resulting code is self-documenting, telling you what it was derived from etc., then when in the resulting context you can decide just what to do with this or that bit that might be tricky (e.g., rewrite it manually, from scratch or just tweak it). In practice, lots of code was easily converted from Franz to Common - it saved much reprogramming effort.
The translator program was written in Common Lisp. It could be used interactively as well as in batch. When used interactively it provided, in effect, a Franz Lisp interpreter on top of Common Lisp.
The program used only macro-expansion (i.e., not expansion followed by Lisp evaluation). Macro-expansion is another way of saying reduction semantics, or rewriting.
Input Franz-Lisp code was macro-expanded via function-definition mapping macros to produce Common-Lisp code. Code that was problematic for translation was flagged (in code) with a description/analysis that described the situation.
The program was called FTOC. I think you can still find it, or at least references to it, by googling (ftoc lisp). (It was the first Lisp program I wrote, and I still have fond memories of the experience. It was a good way to learn both Lisp dialects and to learn Lisp in general.)
Have fun!
In general, I don't think you can do this. The expansion of an lisp macro is Turing complete, so you have to be able to predict the output of a program which could have arbitrary input.
There are some simple things that you could do. defmacros with backquoted forms in appear fairly similar in the output form and might be detected. This sort of heuristic would probably get you a long way.
What I don't understand is your use case. The macro-expanded version of a piece of code is usually only present in the compiled (or in emacs-lisp byte-compiled) form.
Ok so other people have pointed out the fact that this problem is impossible in general. There are two hard parts to this problem: one is that it could be a lot of work to find a preimage of some code fragment through a macro and it is also impossible to determine whether a macro was called or not—there are examples where one may write code which could have come from a macro without using that macro. Imagine for the sake of illustration an sha macro which expands to the SHA hash of the string literal passed to it. Then if you see some sha hash in your expanded code, it would obviously be silly to try to unexpand it. But it may be that the hash was put into the code as a literal, e.g. referencing a specific point in the history of a git repository so it would also be unhelpful to unexpand the macro.
Tractable subproblems
Let me preface this by saying that whilst these may be a little tractable, I still wouldn’t try to solve this problem.
Let’s ignore all the macros that do weird things (like the example above) and all the macros that are just as likely to not have been used in the original (e.g. cond vs if) and all the macros which generate complex code which seems like it would be difficult to unravel (e.g. loop, do, and backquote. Annoyingly these difficult cases are some of those which you would perhaps most want to unexpand). The type this leaves us with (that I’d like to focus on) are macros which basically just reduce boilerplate, e.g. save-excursion or with-XXXX. These are macros whose implementation consists of possibly making some fresh symbols (via gensym) and then having a big simple backquoted block of code. I still think it would be too hard to automatically go from defmacro to a function for unexpansion but I think you could attack some of these on a case-by-case basis. Do this by looking for the forms generated by the macro that delimit (I.e. begin/end) the expanded code. I can’t really offer much beyond that. This is still a hard problem and I don’t think any existing solutions (to other problems) will get you very far on your way.
A further complication I understand is that you do not start at the macroexpanded code but rather at the bytecode. Without knowing anything about the elisp compiler, I worry that more information would be lost in the compilation step and you would have to undo that as well, e.g. perhaps it is hard to determine which code goes inside a let or even when a let begins, or bytecode starts using goto type features even though elisp doesn’t have them.
You suggest that the reason you would like to unexpand macros is so you can decompile bytecode which sometimes comes up in the Emacs debugger and that this would be useful as even though the source code is available in theory, it isn’t always at your fingertips. I put it to you that if you want to make your life debugging elisp easier it would be more worthwhile to figure out how to have the Emacs debugger always take you to the source code for internal functions. This might involve installing extra debugging related packages or downloading the Emacs source code and setting some variable so Emacs knows where to find it or compiling Emacs yourself from source. I don’t really know about that but I bet getting thrown into bytecode instead of source would have been enough of a problem for Emacs developers over the past thirty years that a solution to that problem does exist.
If however what you really want to do is to try to implement a decompiler for elisp then I suppose that’s what you should do. A final observation is that while Lisp provides facilities which make manipulating Lisp code easy, this doesn’t help much with decompiling as all these facilities can be used in compilation so there are infinitely more patterns one might want to detect than in e.g. a C decompiler. Perhaps scheme style macros would be easier to unexpand, although they would still be hard.
If you’re decompiling because you want to give a better idea of which exact subexpression rather than line is being evaluated (normally Lisp debuggers work on expressions not lines anyway) in the debugger then perhaps it would actually be useful to see the code at the expanded level rather than the unexpanded one. Or perhaps it would be best to see both and maybe in between as well. Keeping track of what’s what through forwards macroexpansion is already difficult and fiddly. Doing it in reverse certainly won’t be easier. Good luck!
Edit: seeing as your not currently using Lisp anyway, I wonder if you might have more success using something like prolog for your unexpanding. You’d still have to manually write rules but I think it would be a large amount of work to try to derive rules from macro definitions.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it.
Macros generate arbitrary expressions, which may contain macros recursively. You have no general way to revert the transformations, because it's not pattern-based.
Even if macros were pattern-based, they could still be infinite.
Even if macros were not infinite, they can certainly contain bugs in expansions of patterns that never matched. Given arbitrary code to try to unwind, it could match an expansion that looks like the code and try to revert to its pattern. Without bugs, you could still abuse this.
Even if you could revert macro expansion, some macros expand to the same code. An approach could be signalling a warning with a restart when all reversions expand equally minus the operator, such that if the restart doesn't handle the signal, it would choose the first expansion; and otherwise signalling an error with a restart, such that if the restart doesn't handle the signal, it errors. Or you could configure it to choose certain macros under certain conditions, such as in which package the code was found.
In practice, there are very few cases where reverting an expansion makes any sense. It could be a useful development tool that suggests macros, but I wouldn't generally rely on it for whole source transformations.
One way you could achieve what you want is through a controlled pattern matching. You could initially create patterns manually, which would already handle cases you care about directly, such as the ones you mention:
(if (not <cond>) <expr>) and (if (not <cond>) (progn <&expr>)) to (unless <cond> <&expr>)
You'd have to decide whether null would be equivalent to not. I personally don't mix the boolean meaning of nil with that of empty list or something else, e.g. no result, nothing found, null object, a designator, etc. But perhaps Lisp code as old as that in Emacs just uses them interchangeably.
(if <cond> <expr>) and (if <cond> (progn <&expr>)) to (when <cond> <&expr>)
If you feel like improving code overall, include cond with a single condition. And be careful with cond clauses with only the condition.
You should have a few dozen more, to see how the pattern matching behaves with more patterns to match in terms of time (CPU) and space (memory).
From the description of fare-quasiquote, optima doesn't support backtracking, which you probably want.
But you can do backtracking with optima by yourself, using recursion on complex inner patterns, and if nothing matches, return a control value to keep searching for matching patterns from the outer input.
Another approach is to treat a pattern as a description of a state machine, and handle each new token to advance the current state machines until one of them reaches the end, discarding the state machines that couldn't advance. This approach may consume more memory, depending on the amount of patterns, the similarity between patterns (if many have the same starting token, many state machines will be generated on a matching token), the length of the patterns and, last but not least, the length of the input (s-expression).
An advantage of this approach is that you can use it interactively to see which patterns have matched the most tokens, and you can give weights to patterns instead of just taking the first that matches.
A disadvantage is that, most probably, you'll have to spend effort to develop it.
EDIT: I just lousily described a kind of trie or radix tree.
Once you got something working, maybe try to obtain patterns automatically. This is really hard, you must probably limit it to simple backquoting and accept the fact you can't generalize for anything that contains more complex code.
I believe the hardest will be code walking, which is hard enough with source code, but much more with macro-expanded code. Perhaps if you could expand the whole picture a bit further to understand the goal, maybe someone could suggest a better approach other than operating on macro-expanded code.
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
There's a huge step from expanding code with defmacro and all that generality. Most Lisp developers will know about hygienic macros, at least in terms of symbols as variables.
But there's still hygienic macros in terms of symbols as operators1, code walking, interaction with a containing macro (usually using macrolet), etc. It's way too complex.
1.
Common Lisp evaluates the operator in a compound form in the lexical environment, and probably everyone makes macros that assume that the global macro or function definition of a symbol will be used.
But it might not be so:
(defmacro my-macro-1 ()
`1)
(defmacro my-macro-2 ()
`(my-function (my-macro-1)))
(defun my-function (n)
(* n 100))
(macrolet ((my-macro-1 ()
`2))
(flet ((my-function (n)
(* n 1000)))
(my-macro-2)))
That last line will expand to (my-function (my-macro-2)), which will be recursively expanded to (my-function 2). When evaluated, it will yield 2000.
For proper operator hygiene, you'd have to do something like this:
(defmacro my-macro-2 ()
;; capture global bindings of my-macro-1 and my-function-1 by name
(flet ((my-macro-1-global (form env)
(funcall (macro-function 'my-macro-1) form env))
(my-function-global (&rest args)
;; hope the compiler can optimize this
(apply 'my-function args)))
;; store them globally in uninterned symbols
;; hopefully, no one will mess with them
(let ((my-macro-1-symbol (gensym (symbol-name 'my-macro-1)))
(my-function-symbol (gensym (symbol-name 'my-function))))
(setf (macro-function my-macro-1-symbol) #'my-macro-1-global)
(setf (symbol-function my-function-symbol) #'my-function-global)
`(,my-function-symbol (,my-macro-1-symbol)))))
With this definition, the example will yield 100.
Common Lisp has some restrictions to avoid this, but it only states the consequences are undefined when (re)defining symbols in the common-lisp package, globally or locally. It doesn't require errors or warnings to be signaled.
I don't think it is possible to do this in general, but you can undo a pattern back into a macro use for every match if you supply code for each unmacroing. Code that mixed cond and if will end up being just if and your code would remove all if into cond making the reverse not the same as the starting point. The more macros you have and the more they expand into each other the more uncertain of the end result will be of the starting point.
You could have rules such that if is not translated into cond unless you used one of the features, like more than one predicate or implicit progn, but you have no idea if the coder actually did use cond everywhere because he liked in consistent regardless. Thus your unmacroing will acyually be more of a simplification.
I don't believe there's a general solution to that, and you certainly
can't guarantee that the structure of the output would match that of
the original code, and I'm not going near the idea of auto-generating
patterns and desired transformations from macro definitions; but you
might achieve a simple version of this with Emacs' own pcase pattern
matching facility.
Here's the simplest example I could think of:
With reference to the definition of when:
(defmacro when (cond &rest body)
(list 'if cond (cons 'progn body)))
We can transform code using a pcase pattern like so:
(let ((form '(if (and foo bar baz) (progn do (all the) things))))
(pcase form
(`(if ,cond (progn . ,body))
`(when ,cond ,#body))
(_ form)))
=> (when (and foo bar baz) do (all the) things)
Obviously if the macro definitions change, then your patterns will
cease to work (but that's a pretty safe kind of failure).
Caveat: This is the first time I've written a pcase form, and I
don't know what I don't know. It seems to work as intended, though.
I am new to Emacs Lisp and the feeling is like it lacks strictness (and namespaces, and more...).
To be more comfortable with it I need a
way to make interpreter/byte compiler complain a lot if I use deprecated or obsolete function or variable (even better - hide them). Why this is not looks so simple and removing corresponding .el packages will not work is obvious - they may be needed by some legacy code.
Also, if it is possible, turning off all aliases would be nice. In my opinion they are there only for backwards compatibility, which I do not need. Because of setting this one globally can ruin something, I hope there is something like use strict in JavaScript, which can be applied to the inner body, so the effect is neatly localized.
Do not get me wrong, I think that global namespace of the Elisp is like a dump and if it could be any cleaner, why not?
To put in one sentence: how to make Elisp global namespace obsoleteless and deprecateless, as slim as possible?
I don't know of an Emacs Lisp linter that is built in to Emacs.
I do two things for my own code to try to ensure some level of cleanliness.
First, I make sure that byte-compiling the code doesn't give any errors or warnings. The byte compiler does a certain amount of checking.
Second, I enable lexical binding. This lets the byte compiler detect a few more possible warnings.
This is about the best you can do with the built-in tools. If you want to go further you could write your own tree walker to perform whatever other tests you like.
Some people use double dash to indicate that the function is subject to change:
What does the double minus (--) convention in function names mean in Emacs Lisp
Does including internal in function names mean similar things?
Two examples
where-is-internal
internal-make-var-non-special
The function where-is-internal has a detailed docstring and is mentioned in the manual as well. Is where-is-internal an exception?
Is there a difference between having -internal as suffix and having internal- as prefix?
Adding to confusion, there are also function names with internal-- (with double dash) as prefix.
The confusion is not just in the naming convention (variability due to history and perhaps sometimes whim). The confusion is in the very notion of "internal" in free software, where the source code is open to everyone to use or modify (even fork) as they please.
To answer your question from (what I think is) the point of view of Emacs Dev, and thus in terms of the underlying intention: "internal" means that someone using such a function is perhaps more likely to encounter future changes in the Emacs-Dev implementation and use of that function than might be the case for a non-"internal" function. IOW, you might not want to count on it remaining as it is now. That's all.
But there's a lot of "perhaps", "more likely", and "might" in there. In practice, some non-"internal" functions change more radically or more quickly than some "internal" functions. It might be the case that for the former there will be a deprecation grace period, during which the pre-change situation is tolerated, i.e., still works. That might not be the case for something "internal". But again, in practice there is some gray between the black of "internal" and the white of non-"internal".
Someone from Emacs Dev (e.g. #Stefan) will perhaps put this differently or correct my interpretation.
My own take: there have sometimes (often) been functions and variables that the author did not expect users to make use of directly, and thus naturally thought of as "internal", which users have nevertheless put to good use, or even "had" to use (modulo rewriting lots of code). Some such have had their "internal" status removed (no, I don't have examples memorized). Or sometimes a new, non-"internal" function has been added to make the behavior available - e.g., a wrapper or function-valued argument has been added (again, I have no offhand examples to give).
IOW, for Emacs Dev too it is not always clear what should be considered "internal". Just take the label as a flag that you might not want to count too much on that function or variable.
Wrt the various notations: My impression is that the -- convention seems recently to be used more (though there is also some old code that uses it); using internal is an older convention, for the most part.
The "internal" and the "--" conventions are similar. Basically "internal" is used when there's no prefix after which to put a double dash (which is usually the case for functions implemented in C).
And yes, as Drew explains, the intention behind the notion of something being "internal" is just to recommend people not use it directly. IOW if they need the corresponding functionality, they should report a bug requesting to promote its status to "non-internal".
Depending on my mood I seem to waffle back and forth between wanting a Lisp-1 and a Lisp-2. Unfortunately beyond the obvious name space differences, this leaves all kinds of amusing function name/etc problems you run into. Case in point, trying to write some code tonight I tried to do (map #'function listvar) which, of course, doesn't work in CL, at all. Took me a bit to remember I wanted mapcar, not map. Of course it doesn't help when slime/emacs shows map IS defined as something, though obviously not the same function at all.
So, pointers on how to minimize this short of picking one or the other and sticking with it?
Map is more general than mapcar, for example you could do the following rather than using mapcar:
(map 'list #'function listvar)
How do I keep scheme and CL separate in my head? I guess when you know both languages well enough you just know what works in one and not the other. Despite the syntactic similarities they are quite different languages in terms of style.
Well, I think that as soon you get enough experience in both languages this becomes a non-issue (just with similar natural languages, like Italian and Spanish). If you usually program in one language and switch to the other only occasionally, then unfortunately you are doomed to write Common Lisp in Scheme or vice versa ;)
One thing that helps is to have a distinct visual environment for both languages, using syntax highlighting in some other colors etc. Then at least you will always know whether you are in Common Lisp or Scheme mode.
I'm definitely aware that there are syntactic differences, though I'm certainly not fluent enough yet to automatically use them, making the code look much more similar currently ;-).
And I had a feeling your answer would be the case, but can always hope for a shortcut <_<.
The easiest way to keep both languages straight is to do your thinking and code writing in Common Lisp. Common Lisp code can be converted into Scheme code with relative ease; however, going from Scheme to Common Lisp can cause a few headaches. I remember once where I was using a letrec in Scheme to store both variables and functions and had to split it up into the separate CL functions for the variable and function namespaces respectively.
In all practicality though I don't make a habit of writing CL code, which makes the times that I do have to all the more painful.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have read a lot that LISP can redefine syntax on the fly, presumably with macros. I am curious how far does this actually go? Can you redefine the language structure so much that it borderline becomes a compiler for another language? For example, could you change the functional nature of LISP into a more object oriented syntax and semantics, maybe say having syntax closer to something like Ruby?
Especially, is it possible to get rid of the parenthesis hell using macros? I have learned enough (Emacs-)LISP to customize Emacs with my own micro-features, but I am very curious how far macros can go in customizing the language.
That's a really good question.
I think it's nuanced but definitely answerable:
Macros are not stuck in s-expressions. See the LOOP macro for a very complex language written using keywords (symbols). So, while you may start and end the loop with parentheses, inside it has its own syntax.
Example:
(loop for x from 0 below 100
when (even x)
collect x)
That being said, most simple macros just use s-expressions. And you'd be "stuck" using them.
But s-expressions, like Sergio has answered, start to feel right. The syntax gets out of the way and you start coding in the syntax tree.
As for reader macros, yes, you could conceivably write something like this:
#R{
ruby.code.goes.here
}
But you'd need to write your own Ruby syntax parser.
You can also mimic some of the Ruby constructs, like blocks, with macros that compile to the existing Lisp constructs.
#B(some lisp (code goes here))
would translate to
(lambda () (some lisp (code goes here)))
See this page for how to do it.
Yes, you can redefine the syntax so that Lisp becomes a compiler. You do this using "Reader Macros," which are different from the normal "Compiler Macros" that you're probably thinking of.
Common Lisp has the built-in facility to define new syntax for the reader and reader macros to process that syntax. This processing is done at read-time (which comes before compile or eval time). To learn more about defining reader macros in Common Lisp, see the Common Lisp Hyperspec -- you'll want to read Ch. 2, "Syntax" and Ch. 23, "Reader". (I believe Scheme has the same facility, but I'm not as familiar with it -- see the Scheme sources for the Arc programming language).
As a simple example, let's suppose you want Lisp to use curly braces rather than parentheses. This requires something like the following reader definitions:
;; { and } become list delimiters, along with ( and ).
(set-syntax-from-char #\{ #\( )
(defun lcurly-brace-reader (stream inchar) ; this was way too easy to do.
(declare (ignore inchar))
(read-delimited-list #\} stream t))
(set-macro-character #\{ #'lcurly-brace-reader)
(set-macro-character #\} (get-macro-character #\) ))
(set-syntax-from-char #\} #\) )
;; un-lisp -- make parens meaningless
(set-syntax-from-char #\) #\] ) ; ( and ) become normal braces
(set-syntax-from-char #\( #\[ )
You're telling Lisp that the { is like a ( and that the } is like a ). Then you create a function (lcurly-brace-reader) that the reader will call whenever it sees a {, and you use set-macro-character to assign that function to the {. Then you tell Lisp that ( and ) are like [ and ] (that is, not meaningful syntax).
Other things you could do include, for example, creating a new string syntax or using [ and ] to enclose in-fix notation and process it into S-expressions.
You can also go far beyond this, redefining the entire syntax with your own macro characters that will trigger actions in the reader, so the sky really is the limit. This is just one of the reasons why Paul Graham and others keep saying that Lisp is a good language in which to write a compiler.
I'm not a Lisp expert, heck I'm not even a Lisp programmer, but after a bit of experimenting with the language I came to the conclusion that after a while the parenthesis start becoming 'invisible' and you start seeing the code as you want it to be. You start paying more attention to the syntactical constructs you create via s-exprs and macros, and less to the lexical form of the text of lists and parenthesis.
This is specially true if you take advantage of a good editor that helps with the indentation and syntax coloring (try setting the parenthesis to a color very similar to the background).
You might not be able to replace the language completely and get 'Ruby' syntax, but you don't need it. Thanks to the language flexibility you could end having a dialect that feels like you are following the 'Ruby style of programming' if you want, whatever that would mean to you.
I know this is just an empirical observation, but I think I had one of those Lisp enlightenment moments when I realized this.
Over and over again, newcomers to Lisp want to "get rid of all the parenthesis." It lasts for a few weeks. No project to build a serious general purpose programming syntax on top of the usual S-expression parser ever gets anywhere, because programmers invariably wind up preferring what you currently perceive as "parenthesis hell." It takes a little getting used to, but not much! Once you do get used to it, and you can really appreciate the plasticity of the default syntax, going back to languages where there's only one way to express any particular programming construct is really grating.
That being said, Lisp is an excellent substrate for building Domain Specific Languages. Just as good as, if not better than, XML.
Good luck!
The best explanation of Lisp macros I have ever seen is at
https://www.youtube.com/watch?v=4NO83wZVT0A
starting at about 55 minutes in. This is a video of a talk given by Peter Seibel, the author of "Practical Common Lisp", which is the best Lisp textbook there is.
The motivation for Lisp macros is usually hard to explain, because they really come into their own in situations that are too lengthy to present in a simple tutorial. Peter comes up with a great example; you can grasp it completely, and it makes good, proper use of Lisp macros.
You asked: "could you change the functional nature of LISP into a more object oriented syntax and semantics". The answer is yes. In fact, Lisp originally didn't have any object-oriented programming at all, not surprising since Lisp has been around since way before object-oriented programming! But when we first learned about OOP in 1978, we were able to add it to Lisp easily, using, among other things, macros. Eventually the Common Lisp Object System (CLOS) was developed, a very powerful object-oriented programming system that fits elegantly into Lisp. The whole thing can be loaded as an extension -- nothing is built-in! It's all done with macros.
Lisp has an entirely different feature, called "reader macros", that can be used to extend the surface syntax of the language. Using reader macros, you can make sublanguages that have C-like or Ruby-like syntax. They transform the text into Lisp, internally. These are not used widely by most real Lisp programmers, mainly because it is hard to extend the interactive development environment to understand the new syntax. For example, Emacs indentation commands would be confused by a new syntax. If you're energetic, though, Emacs is extensible too, and you could teach it about your new lexical syntax.
Regular macros operate on lists of objects. Most commonly, these objects are other lists (thus forming trees) and symbols, but they can be other objects such as strings, hashtables, user-defined objects, etc. These structures are called s-exps.
So, when you load a source file, your Lisp compiler will parse the text and produce s-exps. Macros operate on these. This works great and it's a marvellous way to extend the language within the spirit of s-exps.
Additionally, the aforementioned parsing process can be extended through "reader macros" that let you customize the way your compiler turns text into s-exps. I suggest, however, that you embrace Lisp's syntax instead of bending it into something else.
You sound a bit confused when you mention Lisp's "functional nature" and Ruby's "object-oriented syntax". I'm not sure what "object-oriented syntax" is supposed to be, but Lisp is a multi-paradigm language and it supports object-oriented programming extremelly well.
BTW, when I say Lisp, I mean Common Lisp.
I suggest you put your prejudices away and give Lisp an honest go.
Parenthesis hell? I see no more parenthesis in:
(function toto)
than in:
function(toto);
And in
(if tata (toto)
(titi)
(tutu))
no more than in:
if (tata)
toto();
else
{
titi();
tutu();
}
I see less brackets and ';' though.
What you are asking is somewhat like asking how to become an expert chocolatier so that you can remove all that hellish brown stuff from your favourite chocolate cake.
Yes, you can fundamentally change the syntax, and even escape "the parentheses hell". For that you will need to define a new reader syntax. Look into reader macros.
I do suspect however that to reach the level of Lisp expertise to program such macros you will need to immerse yourself in the language to such an extent that you will no longer consider parenthese "hell". I.e. by the time you know how to avoid them, you will have come to accept them as a good thing.
If you want lisp to look like Ruby use Ruby.
It's possible to use Ruby (and Python) in a very lisp like way which is one of the main reasons they have gained acceptance so quickly.
see this example of how reader macros can extend the lisp reader with complex tasks like XML templating:
http://common-lisp.net/project/cl-quasi-quote/present-class.html
this user library compiles the static parts of the XML into UTF-8 encoded literal byte arrays at compile time that are ready to be write-sequence'd into the network stream. and they are usable in normal lisp macros, they are orthogonal... the placement of the comma character influences which parts are constant and which should be evaluated at runtime.
more details available at: http://common-lisp.net/project/cl-quasi-quote/
another project that for Common Lisp syntax extensions: http://common-lisp.net/project/cl-syntax-sugar/
#sparkes
Sometimes LISP is the clear language choice, namely Emacs extensions. I'm sure I could use Ruby to extend Emacs if I wanted to, but Emacs was designed to be extended with LISP, so it seems to make sense to use it in that situation.
It's a tricky question. Since lisp is already structurally so close to a parse tree the difference between a large number of macros and implementing your own mini-language in a parser generator isn't very clear. But, except for the opening and closing paren, you could very easily end up with something that looks nothing like lisp.
One of the uses of macros that blew my mind was the compile-time verification of SQL requests against DB.
Once you realize you have the full language at hand at compile-time, it opens up interesting new perspectives. Which also means you can shoot yourself in the foot in interesting new ways (like rendering compilation not reproducible, which can very easily turn into a debugging nightmare).