lisp: chunking a vector - lisp

Question

As mentioned here, I'm trying to teach myself lisp by implementing lodash.
I have basically no experience with lisp, so work that would be trivial in js is foreign to me.
For instance, I'm working on implementation of a _.chunk method, which in js takes an array and a size variable and 'chunks' the array by the size:
_.chunk(['a', 'b', 'c', 'd'], 2);
// => [['a', 'b'], ['c', 'd']]
_.chunk(['a', 'b', 'c', 'd'], 3);
// => [['a', 'b', 'c'], ['d']]
As somebody totally new to common lisp data types, I would assume that the analogous type would be a vector, not an array, is that correct?
Secondly, my way of solving this algorithmically would be to retain a length variable, and a pointer variable, and to grab a subset of the array/vector, [pointer to pointer + size], while pointer + size was < length, and then return [pointer to length] when that was no longer true, and incrementing pointer to pointer + size + 1 otherwise.
No idea how to implement this in lisp, here is my code so far.
(defun _.chunk (vector &optional (size 1 size-p))
(if (or (not size-p) (eq size 1))
vector
((let (
(array_length (array-total-size array))
(pointer)
)
???
))
)
)

Answer 1

For this implementation I would first write an idiomatic Common Lisp version of chunk that can be useful in a CL program (efficient, etc.), and then write a thin lodash layer that only wraps around those functions.
For example, I would first write a helper function to allow sharing storage with the chunked vector. A displaced array refers to another array but with an offset and different size. It may be useful to have chunks be only views of the original vector, so that they all share the same underlying storage array. It is not only a memory optimization: the behaviour is different when mutating either a chunk or the original vector, since any change in one is visible in the other. But as far as I know lodash is (was?) a pure functional language, so it makes sense to share some data if you don't mutate them. Some languages call those kind of indirect arrays "slices".
(defun slice (vector start end)
(make-array (- end start)
:element-type (array-element-type vector)
:displaced-to vector
:displaced-index-offset start))
So I would also make chunk-vector accept :start and :end parameters, as commonly done, along with sharedp which specifies if storage should be shared with the original vector:
(defun chunk-vector (size vector &key start end sharedp)
(check-type size (integer 1))
(loop
with slicer = (if sharedp #'slice #'subseq)
and low = (or start 0)
and high = (or end (length vector))
for s from low below high by size
for e from (+ low size) by size
collect (funcall slicer vector s (min e high))))
Note: I assume nil is a possible value for end that means the end of the vector, to mirror how subseq works. I do the same for start, because for those variables the nil value can be used without ambiguity to mean "default value". I could also have defined defaults in the lambda list, as done in tfb's answer.
Here are some tests:
(chunk-vector 3 #(0 1 2 3 4 5 6 7 8 9) :sharedp t)
(#(0 1 2) #(3 4 5) #(6 7 8) #(9))
(chunk-vector 2 #(0 1 2 3 4 5 6 7 8 9))
(#(0 1) #(2 3) #(4 5) #(6 7) #(8 9))
(chunk-vector 1 #(0 1 2 3 4 5 6 7 8 9))
(#(0) #(1) #(2) #(3) #(4) #(5) #(6) #(7) #(8) #(9))
Likewise, you could also define a chunk-list function and have the lodash chunck function dispatch to each specialized version based on the sequence type.
This can be done with CLOS, but since that is already demonstrated in another answer, I'll just define individual specialized functions.
Here is an implementation of chunk-list that is based on LDIFF.
I tried first mixing all cases in one function, but this becomes needlessly complex.
Here is first an unbounded chunk function:
(defun chunk-list/unbounded (size list)
(loop
for front = list then next
for next = (nthcdr size front)
collect (ldiff front next)
while next))
front is defined as initially list, then the current value of next at each step
next is the next chunk, computed using size; this plays nicely with lists that have not enough elements, since in that case nthcdr just returns the remaining elements.
A bit more complex case is required to handle the end argument, and for that we define the bounded version where there is also an additional upper-limit counter, that decreases by size at each step of iteration. It represents remaining number of elements to add, and is used along with size to compute (min size upper-limit), the size of the next chunk:
(defun chunk-list/bounded (size list upper-limit)
(loop
for front = list then next
for next = (nthcdr (min size upper-limit) front)
collect (ldiff front next)
do (decf upper-limit size)
while (and next (plusp upper-limit))))
Finally, chunk-list dispatches on both versions based on whether end is nil or not; the calls are inlined here (because we can):
(defun chunk-list (size list &key (start 0) end)
(declare (inline check-list/bounded check-list/simple))
(check-type size (integer 1))
(let ((list (nthcdr start list)))
(when list
(if end
(chunk-list/bounded size list (- end start))
(chunk-list/unbounded size list)))))
Some examples:
(chunk-list 3 '(1 2 3 4 5 6 7))
((1 2 3) (4 5 6) (7))
(chunk-list 29 '(1 2))
((1 2))
(chunk-list 2 (alexandria:iota 100 :start 0) :start 10 :end 20)
((10 11) (12 13) (14 15) (16 17) (18 19))

Answer 2

i would propose step-by step slicing iterating over the chunk index (since you can easily find out the total amount of chunks), using dotimes.
this could look something like the following:
(defun chunked (seq size)
(let* ((total (length seq))
(amount (ceiling total size))
(res (make-array amount :fill-pointer 0)))
(dotimes (i amount res)
(vector-push (subseq seq (* i size) (min (* (1+ i) size) total))
res))))
CL-USER> (chunked "abcdefgh" 3)
;; #("abc" "def" "gh")
CL-USER> (chunked #*00101 2)
;; #(#*00 #*10 #*1)
CL-USER> (chunked (list :a :b :c :d :e) 1)
;; #((:A) (:B) (:C) (:D) (:E))
CL-USER> (chunked (list :a :b :c :d :e) 4)
;; #((:A :B :C :D) (:E))

Answer 3

This is an addendum to coredump's answer, as well as referring to a comment by Kaz. Most of this is about style, which is always a matter of opinion and I do not claim my opinion is better than theirs: I just think it is interesting to talk about the choices as Lisp programming is very much about style choice, since the language is so flexible compared to most others. The last section ('extending') might be interesting however.
Argument order
The problem with a signature which is (size vector ...) is that size can't be optional. If you want it to be, it can't be the first argument to the function. Whether that outweighs the easy utility of partial-application libraries I don't know (however, in the 'do the right thing' spirit, if I wrote a partial application library it would allow you to specify which args it was currying, so this would not be a problem).
So if size needs to be optional then the argument order must be (vector size ...).
Further, since coredump's answer uses keyword arguments, I would make size be one as well as you almost never want to mix keyword & optional arguments. So that leads to a signature which would be (vector &key size start end sharedp), and I'd then write the actual function as
(defun chunk-vector (vector &key (size 1) (start 0) (end (length vector))
(sharedp nil))
(check-type size (integer 1))
(let ((slicer (if sharedp #'slice #'subseq)))
(loop for s from start below end by size
for e from (+ start size) by size
collect (funcall slicer thing s (min e end)))))
This slightly improves on coredump's version by defaulting the arguments in the arglist rather than later.
Extending chunk-vector
Pretty obviously you might want to chunk other kinds of things, such as lists, and pretty obviously the algorithm for chunking a list will be very different than that for chunking a vector, because you really do not want to repeatedly call subseq on a list.
Well, this is what CLOS is for. First of all we can define a generic chunk function:
(defgeneric chunk (thing &key)
;; in real life we might want to specify some of the keyword
;; arguments at the GF level, but we won't
)
And now define methods for classes we care about. Firstly the method to chunk vectors, which is pretty much the previous function:
(defmethod chunk ((thing vector) &key
(size 1) (start 0) (end (length thing)) (sharedp nil))
(check-type size (integer 1))
(let ((slicer (if sharedp #'slice #'subseq)))
(loop for s from start below end by size
for e from (+ start size) by size
collect (funcall slicer thing s (min e end)))))
And now, for instance, one to chunk lists. Note this may be buggy, and there may be better ways of doing this.
(defmethod chunk ((thing list) &key
(size 1) (start 0) (end nil endp) (sharedp nil))
;; This does not implemenent SHAREDP: this could only be useful for
;; the last chunk, and since you don't know if you could share a
;; chunk until you have already walked the list it did not seem
;; worth it. It may also be buggy in its handling of END.
(declare (ignorable sharedp))
(flet ((next (lt)
(nthcdr size lt))
(the-chunk (lt p)
(loop for c below (if endp (min size (- end p)) size)
for e in lt
do (print c)
collect e)))
(loop for tail on (nthcdr start thing) by #'next
for pos upfrom start by size
while (or (not endp) (< pos end))
collect (the-chunk tail pos))))
And of course you can now define methods on this function for other appropriate types.

Answer 4

The input could certainly be a vector (a vector is a 1-dimensional array). Lisp has a few more sensible options of how to represent the result: it could be a 2-dimensional array, a vector of vectors, or maybe even a list of vectors.
To get a 2-dimensional array:
(defun reshape-2d (column-count vector &optional padding-element)
(let* ((row-count (ceiling (length vector) column-count))
(array (make-array (list row-count column-count)
:initial-element padding-element)))
(loop :for i :below (length vector)
:do (setf (row-major-aref array i) (aref vector i)))
array))
To get a vector of vectors:
(defun chunkv (size vector)
(let ((vectors (make-array (ceiling (length vector) size))))
(loop :for i :below (length vector) :by size
:for j :below (length vectors)
:do (setf (aref vectors j) (subseq vector
i
(min (1- (length vector))
(+ i size)))))
vectors))
To get a list of vectors:
(defun chunkl (size vector)
(loop :for i :below (length vector) :by size
:collect (subseq vector
i
(min (1- (length vector))
(+ i size)))))
This last version could actually chunk any sequence because it only uses sequence functions.