Non destructive subsequence-except in lisp - lisp

I know subseq in lisp return a subsequence through a range. Is there anything which will return the subsequence except that range and should be non-destructive? Any help is appreciated.

You can do it for any sequence with concatenate and subseq:
(defun sequence-except (sequence start end)
(concatenate (sequence-type sequence)
(subseq sequence 0 start)
(subseq sequence end)))
The following should be enough to determine the type of the input sequence for bit-vectors, strings, etc:
(defun sequence-type (sequence)
(etypecase sequence
(list 'list)
(array `(array ,(array-element-type sequence) (*)))))
Tests:
(loop for test in (list
#*10101001
"abcd"
'(0 3 2)
nil
#(1 2 3 4)
(make-array 4
:adjustable t
:fill-pointer T
:initial-contents
'(a b c d)))
collect (concatenate (sequence-type test) test))

can't you just:
(defun seq-drop-subseq (SEQ START &optional END)
(if END
(nconc (seq-take START) (seq-drop END))
(seq-take START)))
edit: had to go look. remove does it.

Related

How to concatenate all the elements of the argument lists into a single list

I am trying to concatenate all elements in the list argument into a single list.
I have this code:
(define (concatenate . lsts)
(let rec ([l lsts]
[acc '()])
(if (empty? l)
acc
(rec (cons (list* l)
acc)))))
An example of output is here:
> (concatenate '(1 2 3) '(hi bye) '(4 5 6))
'(1 2 3 hi bye 4 5 6)
But I keep getting this error:
rec: arity mismatch;
the expected number of arguments does not match the given number
expected: 2
given: 1
Can someone please explain this?
Another answer explains the OP error,
and shows how the code can be fixed using append.
But there could be reasons for append to be disallowed in this assignment
(of course, it could be replaced with, for example, an inner "named let" iteration).
This answer will present an alternative approach and describe how it can be derived.
#lang racket
(require test-engine/racket-tests)
(define (conc . lols) ;; ("List of Lists" -> List)
;; produce (in order) the elements of the list elements of lols as one list
;; example: (conc '(1 2 3) '(hi bye) '(4 5 6)) => '(1 2 3 hi bye 4 5 6)
(cond
[(andmap null? lols) empty ] ;(1) => empty result
[else
(cons (if (null? (car lols)) ;(2) => head of result
(car (apply conc (cdr lols)))
(caar lols))
(apply conc ;(3) => tail of result
(cond
[(null? (car lols))
(list (cdr (apply conc (cdr lols)))) ]
[(null? (cdar lols))
(cdr lols) ]
[else
(cons (cdar lols) (cdr lols)) ]))) ]))
(check-expect (conc '() ) '())
(check-expect (conc '() '() ) '())
(check-expect (conc '(1) ) '(1))
(check-expect (conc '() '(1) ) '(1))
(check-expect (conc '() '(1 2) ) '(1 2))
(check-expect (conc '(1) '() ) '(1))
(check-expect (conc '(1) '(2) ) '(1 2))
(check-expect (conc '(1 2) '(3 4) ) '(1 2 3 4))
(check-expect (conc '(1 2 3) '(hi bye) '(4 5 6)) '(1 2 3 hi bye 4 5 6))
(test)
Welcome to DrRacket, version 8.6 [cs].
Language: racket, with debugging; memory limit: 128 MB.
All 8 tests passed!
>
How was this code derived?
"The observation that program structure follows data structure is a key lesson in
introductory programming" [1]
A systematic program design method can be used to derive function code from the structure
of arguments. For a List argument, a simple template (natural recursion) is often appropriate:
(define (fn lox) ;; (Listof X) -> Y ; *template*
;; produce a Y from lox using natural recursion ;
(cond ;
[(empty? lox) ... ] #|base case|# ;; Y ;
[else (... #|something|# ;; X Y -> Y ;
(first lox) (fn (rest lox))) ])) ;
(Here the ...s are placeholders to be replaced by code to create a particular list-argumented
function; eg with 0 and + the result is (sum list-of-numbers), with empty and cons it's
list-copy; many list functions follow this pattern. Racket's "Student Languages" support
placeholders.)
Gibbons [1] points out that corecursion, a design recipe based on result structure, can also
be helpful, and says:
For a structurally corecursive program towards lists, there are three questions to ask:
When is the output empty?
If the output isn’t empty, what is its head?
And from what data is its tail recursively constructed?
So for simple corecursion producing a List result, a template could be:
(define (fn x) ;; X -> ListOfY
;; produce list of y from x using natural corecursion
(cond
[... empty] ;(1) ... => empty
[else (cons ... ;(2) ... => head
(fn ...)) ])) ;(3) ... => tail data
Examples are useful to work out what should replace the placeholders:
the design recipe for structural recursion calls for examples that cover all possible input variants,
examples for co-programs should cover all possible output variants.
The check-expect examples above can be worked through to derive (1), (2), and (3).
[1] Gibbons 2021 How to design co-programs
Assuming you are allowed to call append, for simplicity. You have
(define (concatenate . lsts)
(let rec ([l lsts]
[acc '()])
(if (empty? l)
acc
(rec (cons (list* l) ; only ONE
acc) ; argument
))))
calling rec with only one argument. I have added a newline there so it becomes more self-evident.
But your definition says it needs two. One way to fix this is
(define (conc . lsts)
(let rec ([ls lsts]
[acc '()])
(if (empty? ls)
acc
(rec (cdr ls) ; first argument
(append acc (car ls)) ; second argument
))))
Now e.g.
(conc (list 1 2) (list 3 4))
; => '(1 2 3 4)
I used append. Calling list* doesn't seem to do anything useful here, to me.
(edit:)
Using append that way was done for simplicity. Repeatedly appending on the right is actually an anti-pattern, because it leads to quadratic code (referring to its time complexity).
Appending on the left with consequent reversing of the final result is the usual remedy applied to that problem, to get the linear behavior back:
(define (conc2 . lsts)
(let rec ([ls lsts]
[acc '()])
(if (empty? ls)
(reverse acc)
(rec (cdr ls)
(append (reverse (car ls))
acc)))))
This assumes that append reuses its second argument and only creates new list structure for the copy of its first.
The repeated reverses pattern is a bit grating. Trying to make it yet more linear, we get this simple recursive code:
(define (conc3 . lols)
(cond
[(null? lols) empty ]
[(null? (car lols))
(apply conc3 (cdr lols)) ]
[else
(cons (caar lols)
(apply conc3
(cons (cdar lols) (cdr lols))))]))
This would be even better if the "tail recursive modulo cons" optimization was applied by a compiler, or if cons were evaluated lazily.
But we can build the result in the top-down manner ourselves, explicitly, set-cdr!-ing the growing list's last cell. This can be seen in this answer.

Lisp nested list iteration

I just started to learn Common Lisp and this is my first functional programming language.
I am trying to learn about iterating through lists. I wrote these two functions:
(defun reverseList (liste)
(defvar reversedList(list))
(loop for i downfrom (-(length liste)1) to 0 do
(setf reversedList (append reversedList (list(nth i liste)))))
reversedList ;return
)
(defun countAppearance(liste element)
(defvar count 0)
(loop for i from 0 to (-(length liste) 1)do
(if (= (nth i liste) element)
(setf count (+ count 1))))
count
)
Both work fine for a regular list(ex: (1 3 5 7 3 9) but I want them to work for nested lists too.
Examples:
countAppearance - Input: (1 (3 5) (3 7 8) 2) 3 -> Expected output:2
reverseList - Input: (1 (2 3)) -> Expected output: ((3 2) 1)
Before I will show you solutions for nested lists, some notes about your code:
There is already function reverse for non-nested lists, so you don't have to reinvent the wheel.
=> (reverse (list 1 2 3 4 5))
(5 4 3 2 1)
If you need some local variables, use let or let*.
Lisp uses kebab-case, not camelCase, so rename reverseList as reverse-list and so on.
For (setf ... (+ ... 1)), use incf.
For iterating over list, use dolist.
Function count-occurrences can be written using recursion:
(defun count-occurrences (lst elem)
(cond ((null lst) 0)
((= (car lst) elem) (+ 1 (count-occurrences (cdr lst) elem)))
(t (count-occurrences (cdr lst) elem))))
CL-USER 3 > (count-occurrences (list 1 2 3 1 2 3) 2)
2
Or it can be written with let, dolist and incf:
(defun count-occurrences2 (lst elem)
(let ((count 0))
(dolist (e lst)
(when (= e elem) (incf count)))
count))
CL-USER 4 > (count-occurrences2 (list 1 2 3 1 2 3) 2)
2
Solutions for nested lists use recursion:
(defun deep-reverse (o)
(if (listp o)
(reverse (mapcar #'deep-reverse o))
o))
CL-USER 11 > (deep-reverse '(1 (2 3)))
((3 2) 1)
(defun deep-count (lst elem)
(cond ((null lst) 0)
((listp (car lst)) (+ (deep-count (car lst) elem)
(deep-count (cdr lst) elem)))
((= (car lst) elem) (+ 1 (deep-count (cdr lst) elem)))
(t (deep-count (cdr lst) elem))))
CL-USER 12 > (deep-count '(1 (3 5) (3 7 8) 2) 3)
2
Welcome to functional programming.
Firstly, there are some problems with the code that you have provided for us. There are some spaces missing from the code. Spaces are important because they separate one thing from another. The code (xy) is not the same as (x y).
Secondly, there is an important difference between local and global variables. So, in both cases, you want a local variable for reversedList and count. This is the tricky point. Common Lisp doesn't have global or local variables, it has dynamic and lexical variables, which aren't quite the same. For these purposes, we can use lexical variables, introduced with let. The keyword let is used for local variables in many functional languages. Also, defvar may not do what you expect, since it is way of writing a value once, which cannot be overwritten - I suspect that defparameter is what you meant.
Thirdly, looking at the reverse function, loop has its own way of gathering results into a list called collect. This would be a cleaner solution.
(defun my-reverse (lst)
(loop for x from (1- (length lst)) downto 0 collect (nth x lst)))
It can also be done in a tail recursive way.
(defun my-reverse-tail (lst &optional (result '()))
(if lst
(my-reverse-tail (rest lst) (cons (first lst) result))
result))
To get it to work with nested lists, before you collect or cons each value, you need to check if it is a list, using listp. If it is not a list, just add it onto the result. If it is a list, add on instead a call to your reverse function on the item.
Loop also has functionality to count items.

Implementing Interesting Encoding Method in Lisp

Preface
I am working on implementing the Genetic Algorithm for the Traveling Salesman problem. I am making some base line assumption such as you can travel to any city from any city. While this is for an assignment, I have extended this to a personal project as the deadline is past and I have chosen to use Lisp which definitely was not required. The point of encoding my data in this way listed below is to easily perform cross-over later in the algorithm.
Problem
Suppose you have a list of cities, given similar to the following
(defvar *data* (list
'(A 20 10)
'(B 5 16)
'(C 12 18)
'(D x y)
'(E x y)
...
I want to encode this data in a way similar to this:
and I for the life of me cannot figure out how to implement this in Lisp. If anyone has some insight, it would be much appreciated. If there is a better way to create my *data* set that would make this easier feel free to include it!
Now I understood it. Here is the solution:
(defparameter *data* (list
'(A 20 10)
'(B 5 16)
'(C 12 18)
'(D x y)
'(E x y)))
For one step you need a function which looks up index position of the city in the city-list (*data*) and also removes its entry in the city-list and returns the updated city-list.
(defun choose-city (city-list city-name)
"Return city-name with its index position
and city-list with the chosen city removed, keeping the order."
(let* ((cities (mapcar #'car city-list))
(pos (position city-name cities)))
(list city-name
pos
(append (subseq city-list 0 pos)
(subseq city-list (+ pos 1) (length city-list))))))
;; improved version by #Kaz - thanks! (lispier)
(defun choose-city (city-list city-name)
(list city-name
(positiion city-name city-list :key #'car :test #'eql)
(remove city-name city-list :key #'car :test #'eql)))
Then, you need a function which applies the previous function
over and over again while collecting the index positions and updates from step to step the city-list by removing the matched current-city in the city-sequence.
A typical pattern occuring in lisp for this is
to define the to-be-mutated variable as a local variable in a let expression and from the body of the let-expression to update the variable value using setf (setf-ing).
(defun choose-cities-subsequently (city-list city-sequence)
"Return sequence of subsequent-index-positions of the cities
given in city-sequence. After choosing a sequence, the city is
removed from the city-list and its index position of the previous
pool taken for record."
(let ((index-positions '()) ; initiate collector variable
(current-city-list city-list)) ; current state of city-list
(loop for current-city in city-sequence
do (progn
;; call `choose-city` and capture its results
(destructuring-bind
(name index new-city-list) ; capturing vars
;; and in the following the function call:
(choose-city current-city-list current-city)
;; update collector variable and
;; current-city-list using the captured values
(setf index-positions (cons index index-positions))
(setf current-city-list new-city-list)))
;; if city-sequence processed in this way,
;; return the collected index-positions.
;; remark: cons-ing during collecting and
;; at the end nreverse-ing the result
;; when/while returning
;; is a very typical lisp idiom
finally (return (nreverse index-positions)))))
;; improved version by #Kaz - thanks!
(defun choose-cities-subsequently (city-list city-sequence)
(let ((index-positions '()) ; initiate collector variable
(current-city-list city-list)) ; current state of city-list
(loop for current-city in city-sequence
collect (destructuring-bind
(name index new-city-list)
(choose-city current-city-list current-city)
(setf current-city-list new-city-list)
index)
into index-positions
finally (return index-positions)))))
Now, if you run
(choose-cities-subsequently *data* '(A D E B C))
it returns correctly:
(0 2 2 0 0)
By defining more let-variables in the last function and
setf-fing to those in the destructuring-bind expression's body, and returning the final value in the final list,
you can collect more informations and make them visibile.
Tried to simplify a little and recursive definition
(defparameter *data* (list
'(A 20 10)
'(B 5 16)
'(C 12 18)
'(D x y)
'(E x y)))
(defun choose-city (city-list city-name)
(list (position city-name city-list :key #'car :test #'eql)
(remove city-name city-list :key #'car :test #'eql)))
;; when city names are strings use `:test #'string=
(defun choose-cities-subsequently (city-list city-sequence)
(let ((current-cities city-list))
(loop for current-city in city-sequence
for (idx updated-cities) = (choose-city current-cities current-city)
collect (progn (setf current-cities updated-cities)
idx)
into index-positions
finally (return index-positions))))
(choose-cities-subsequently *cities* '(A D E B C))
;; (0 2 2 0 0)
;; a tail-call recursive version:
(defun choose-cities-subsequently (cities city-sequence
&key (acc-cities '())
(acc-positions '())
(pos-counter 0)
(test #'eql))
(cond ((or (null city-sequence) (null cities)) (nreverse acc-positions))
((funcall test (car city-sequence) (car cities))
(choose-cities-subsequently (append (nreverse acc-cities) (cdr cities))
(cdr city-sequence)
:acc-cities '()
:acc-positions (cons pos-counter acc-positions)
:pos-counter 0
:test test))
(t (choose-cities-subsequently (cdr cities)
city-sequence
:acc-cities (cons (car cities) acc-cities)
:acc-positions acc-positions
:pos-counter (1+ pos-counter)
:test test))))

Trying to rewrite an ugly macro

I'm new to lisp, and have been trying to learn Common Lisp by diving in and writing some code. I've read plenty of documentation on the subject, but it's taking a while to really sink in.
I have written a couple of macros (? and ??) for performing unit tests, but I'm having some difficulty. The code is at the end of the post, to avoid cluttering the actual question.
Here is an example of usage:
(??
(? "Arithmetic tests"
(? "Addition"
(= (+ 1 2) 3)
(= (+ 1 2 3) 6)
(= (+ -1 -3) -4))))
And an example of output:
[Arithmetic tests]
[Addition]
(PASS) '(= (+ 1 2) 3)'
(PASS) '(= (+ 1 2 3) 6)'
(PASS) '(= (+ -1 -3) -4)'
Results: 3 tests passed, 0 tests failed
Now, the existing code works. Unfortunately, the (? ...) macro is ugly, verbose, resistant to change - and I'm pretty sure also badly structured. For example, do I really have to use a list to store pieces of output code and then emit the contents at the end?
I'd like to modify the macro to permit description strings (or symbols) to optionally follow each test, whereupon it would replace the test literal in the output, thus:
(??
(? "Arithmetic tests"
(? "Addition"
(= (+ 1 2) 3) "Adding 1 and 2 results in 3"
(= (+ 1 2 3) 6)
(= (+ -1 -3) -4))))
Output:
[Arithmetic tests]
[Addition]
(PASS) Adding 1 and 2 results in 3
(PASS) '(= (+ 1 2 3) 6)'
(PASS) '(= (+ -1 -3) -4)'
But unfortunately I can't find a sensible place in the macro to insert this change. Depending on where I put it, I get errors like you're not inside a backquote expression, label is not defined or body-forms is not defined. I know what these errors mean, but I can't find a way to avoid them.
Also, I'll be wanting to handle exceptions in the test, and treat that as a failure. Currently, there is no exception handling code - the test result is merely tested against nil. Again, it is not clear how I should add this functionality.
I'm thinking that maybe this macro is over-complex, due to my inexperience in writing macros; and perhaps if I simplify it, modification will be easier. I don't really want to separate it out into several smaller macros without good reason; but maybe there's a terser way to write it?
Can anyone help me out here, please?
A complete code listing follows:
(defmacro with-gensyms ((&rest names) &body body)
`(let ,(loop for n in names collect `(,n (gensym)))
,#body))
(defmacro while (condition &body body)
`(loop while ,condition do (progn ,#body)))
(defun flatten (L)
"Converts a list to single level."
(if (null L)
nil
(if (atom (first L))
(cons (first L) (flatten (rest L)))
(append (flatten (first L)) (flatten (rest L))))))
(defun starts-with-p (str1 str2)
"Determine whether `str1` starts with `str2`"
(let ((p (search str2 str1)))
(and p (= 0 p))))
(defmacro pop-first-char (string)
`(with-gensyms (c)
(if (> (length ,string) 0)
(progn
(setf c (schar ,string 0))
(if (> (length ,string) 1)
(setf ,string (subseq ,string 1))
(setf ,string ""))))
c))
(defmacro pop-chars (string count)
`(with-gensyms (result)
(setf result ())
(dotimes (index ,count)
(push (pop-first-char ,string) result))
result))
(defun format-ansi-codes (text)
(let ((result ()))
(while (> (length text) 0)
(cond
((starts-with-p text "\\e")
(push (code-char #o33) result)
(pop-chars text 2)
)
((starts-with-p text "\\r")
(push (code-char 13) result)
(pop-chars text 2)
)
(t (push (pop-first-char text) result))
))
(setf result (nreverse result))
(coerce result 'string)))
(defun kv-lookup (values key)
"Like getf, but works with 'keys as well as :keys, in both the list and the supplied key"
(setf key (if (typep key 'cons) (nth 1 key) key))
(while values
(let ((k (pop values)) (v (pop values)))
(setf k (if (typep k 'cons) (nth 1 k) k))
(if (eql (symbol-name key) (symbol-name k))
(return v)))))
(defun make-ansi-escape (ansi-name)
(let ((ansi-codes '( :normal "\\e[00m" :white "\\e[1;37m" :light-grey "\\e[0;37m" :dark-grey "\\e[1;30m"
:red "\\e[0;31m" :light-red "\\e[1;31m" :green "\\e[0;32m" :blue "\\e[1;34m" :dark-blue "\\e[1;34m"
:cyan "\\e[1;36m" :magenta "\\e[1;35m" :yellow "\\e[0;33m"
:bg-dark-grey "\\e[100m"
:bold "\\e[1m" :underline "\\e[4m"
:start-of-line "\\r" :clear-line "\\e[2K" :move-up "\\e[1A")))
(format-ansi-codes (kv-lookup ansi-codes ansi-name))
))
(defun format-ansi-escaped-arg (out-stream arg)
(cond
((typep arg 'symbol) (format out-stream "~a" (make-ansi-escape arg)))
((typep arg 'string) (format out-stream arg))
(t (format out-stream "~a" arg))
))
(defun format-ansi-escaped (out-stream &rest args)
(while args
(let ((arg (pop args)))
(if (typep arg 'list)
(let ((first-arg (eval (first arg))))
(format out-stream first-arg (second arg))
)
(format-ansi-escaped-arg out-stream arg)
))
))
(defmacro while-pop ((var sequence &optional result-form) &rest forms)
(with-gensyms (seq)
`(let (,var)
(progn
(do () ((not ,sequence))
(setf ,var (pop ,sequence))
(progn ,#forms))
,result-form))))
(defun report-start (form)
(format t "( ) '~a'~%" form))
(defun report-result (result form)
(format-ansi-escaped t "(" (if result :green :red) `("~:[FAIL~;PASS~]" ,result) :normal `(") '~a'~%" ,form))
result)
(defmacro ? (name &body body-forms)
"Run any number of test forms, optionally nested within further (?) calls, and print the results of each test"
(with-gensyms (result indent indent-string)
(if (not body-forms)
:empty
(progn
(setf result () indent 0 indent-string " ")
(cond
((typep (first body-forms) 'integer)
(setf indent (pop body-forms))))
`(progn
(format t "~v#{~A~:*~}" ,indent ,indent-string)
(format-ansi-escaped t "[" :white ,name :normal "]~%")
(with-gensyms (test-results)
(setf test-results ())
,(while-pop (body-form body-forms `(progn ,#(nreverse result)))
(cond
( (EQL (first body-form) '?)
(push `(progn
(setf test-results (append test-results (? ',(nth 1 body-form) ,(1+ indent) ,#(nthcdr 2 body-form))))
(format t "~%")
test-results
) result)
)
(t
(push `(progn
(format t "~v#{~A~:*~}" ,(1+ indent) ,indent-string)
(report-start ',body-form)
(with-gensyms (result label)
(setf result ,body-form)
(format-ansi-escaped t :move-up :start-of-line :clear-line)
(format t "~v#{~A~:*~}" ,(1+ indent) ,indent-string)
(push (report-result result ',body-form) test-results)
test-results
)) result))))))))))
(defun ?? (&rest results)
"Run any number of tests, and print a summary afterward"
(setf results (flatten results))
(format-ansi-escaped t "~&" :white "Results: " :green `("~a test~:p passed" ,(count t results)) :normal ", "
(if (find NIL results) :red :normal) `("~a test~:p failed" ,(count NIL results))
:yellow `("~[~:;, ~:*~a test~:p not run~]" ,(count :skip results))
:brown `("~[~:;, ~:*~a empty test group~:p skipped~]" ,(count :empty results))
:normal "~%"))
For my part, the ? macro is rather technical and it's hard to follow the logic behind the formatting functions. So instead of tracking errors I'd like to suggest my own attempt, perhaps it'll be of use.
I think that actually your ?? doesn't want to evaluate anything, but rather to treat its body as individual tests or sections. If the body includes a list starting with ?, this list represents a section; other elements are test forms optionally followed by descriptions. So in my implementation ?? will be a macro, and ? will be just a symbol.
I start with wishful thinking. I suppose I can create individual tests using a function make-test-item and test sections using a function make-test-section (their implementation is unimportant for now), that I can display them using an auxiliary function display-test and compute results using the function results, which returns two values: the total number of tests and the number of passed ones. Then I'd like the code
(??
(? "Arithmetic tests"
(? "Addition"
(= (+ 1 2) 3) "Adding 1 and 2 results in 3"
(= (+ 1 2 3) 6)
(= (+ -1 -3) 4))
(? "Subtraction"
(= (- 1 2) 1)))
(= (sin 0) 0) "Sine of 0 equals 0")
to expand into something like
(let ((tests (list (make-test-section :header "Arithmetic tests"
:items (list (make-test-section :header "Addition"
:items (list (make-test-item :form '(= (+ 1 2) 3)
:description "Adding 1 and 2 results in 3"
:passp (= (+ 1 2) 3))
(make-test-item :form '(= (+ 1 2 3) 6)
:passp (= (+ 1 2 3) 6))
(make-test-item :form '(= (+ -1 -3) 4)
:passp (= (+ -1 -3) 4))))
(make-test-section :header "Subtraction"
:items (list (make-test-item :form '(= (- 1 2) 1)
:passp (= (- 1 2) 1))))))
(make-test-item :form '(= (sin 0) 0)
:passp (= (sin 0) 0)
:description "Sine of 0 equals 0"))))
(loop for test in tests
with total = 0
with passed = 0
do (display-test test 0 t)
do (multiple-value-bind (ttl p) (results test)
(incf total ttl)
(incf passed p))
finally (display-result total passed t)))
Here a list of tests is created; then we traverse it printing each test (0 denotes the zero level of indentation and t is as in format) and keeping track of the results, finally displaying the total results. I don't think explicit eval is needed here.
It may not be the most exquisite piece of code ever, but it seems manageable. I supply missing definitions below, they are rather trivial (and can be improved) and have nothing to do with macros.
Now we pass on to the macros. Consider both pieces of code as data, then we want a list processing function which would turn the first one into the second. A few auxiliary functions would come in handy.
The major task is to parse the body of ?? and generate the list of test to go inside the let.
(defun test-item-form (form description)
`(make-test-item :form ',form :description ,description :passp ,form))
(defun test-section-form (header items)
`(make-test-section :header ,header :items (list ,#items)))
(defun parse-test (forms)
(let (new-forms)
(loop
(when (null forms)
(return (nreverse new-forms)))
(let ((f (pop forms)))
(cond ((and (listp f) (eq (first f) '?))
(push (test-section-form (second f) (parse-test (nthcdr 2 f))) new-forms))
((stringp (first forms))
(push (test-item-form f (pop forms)) new-forms))
(t (push (test-item-form f nil) new-forms)))))))
Here parse-test essentially absorbs the syntax of ??. Each iteration consumes one or two forms and collects corresponding make-... forms. The functions can be easily tested in REPL (and, of course, I did test them while writing).
Now the macro becomes quite simple:
(defmacro ?? (&body body)
`(let ((tests (list ,#(parse-test body))))
(loop for test in tests
with total = 0
with passed = 0
do (display-test test 0 t)
do (multiple-value-bind (ttl p) (results test)
(incf total ttl)
(incf passed p))
finally (display-result total passed t))))
It captures a few symbols, both in the variable name space and in the function one (the expansion may contain make-test-item and make-test-section). A clean solution with gensyms would be cumbersome, so I'd suggest just moving all the definitions in a separate package and exporting only ?? and ?.
For completeness, here is an implementation of the test API. Actually, it's what I started coding with and proceeded until I made sure the big let-form works; then I passed on to the macro part. This implementation is fairly sloppy; in particular, it doesn't support terminal colours and display-test can't even output a section into a string.
(defstruct test-item form description passp)
(defstruct test-section header items)
(defun results (test)
(etypecase test
(test-item (if (test-item-passp test)
(values 1 1)
(values 1 0)))
(test-section (let ((items-count 0)
(passed-count 0))
(dolist (i (test-section-items test) (values items-count passed-count))
(multiple-value-bind (i p) (results i)
(incf items-count i)
(incf passed-count p)))))))
(defparameter *test-indent* 2)
(defun display-test-item (i level stream)
(format stream "~V,0T~:[(FAIL)~;(PASS)~] ~:['~S'~;~:*~A~]~%"
(* level *test-indent*)
(test-item-passp i)
(test-item-description i)
(test-item-form i)))
(defun display-test-section-header (s level stream)
(format stream "~V,0T[~A]~%"
(* level *test-indent*)
(test-section-header s)))
(defun display-test (test level stream)
(etypecase test
(test-item (display-test-item test level stream))
(test-section
(display-test-section-header test level stream)
(dolist (i (test-section-items test))
(display-test i (1+ level) stream)))))
(defun display-result (total passed stream)
(format stream "Results: ~D test~:P passed, ~D test~:P failed.~%" passed (- total passed)))
All the code is licenced under WTFPL.

reduce using cl-loop facility

CL library manual "map over sequences" says "All of these mapping operations can be expressed conveniently in terms of the cl-loop macro" but I don't see how cl-reduce can be expressed in terms of cl-loop
Not sure how "conveniently" expressed it is, but here's my take on it:
(defun loop-reduce (func sequence &rest initial-element)
(loop with result =
(or (car initial-element)
(prog1 (car sequence)
(setf sequence (cdr sequence))))
for x in sequence do (setf result (funcall func result x))
finally (return result)))
(loop-reduce '+ '(1 2 3 4 5))
;; 15
(loop-reduce '+ '(1 2 3 4 5) 10)
;; 25