Penetrating the `set-process-sentinel` hierarchy with let-bound variables - emacs

Question

I have never been able to come up with a method to penetrate the set-process-sentinel hierarchy with let-bound variables defined at the outset of the function -- only buffer-local or global variables can penetrate it. Let-bound variables can reach the first start-process, but that is as far as they can penetrate without being rejected due to being unrecognized -- let-bound variables defined at the outset of the function do not appear to be able to penetrate the section that begins with (lambda (p e) . . .. Can anyone think of a way to do it, including penetrating nested sentinels like in the example below?
(set-process-sentinel
(start-process
"my-process-name-one"
"*OUTPUT-BUFFER*"
"/path/to/executable"
"argument-one"
"argument-two"
"argument-three")
(lambda (p e) (when (= 0 (process-exit-status p))
(set-process-sentinel
(start-process
"my-process-name-two"
nil ;; example of not using an output buffer
"/path/to/executable"
"argument-one"
"argument-two"
"argument-three")
(lambda (p e) (when (= 0 (process-exit-status p))
(set-process-sentinel
(start-process . . . ))))))))

Answer 1

The problem is that Emacs Lisp variable bindings are dynamic by default. That is, when a function is evaluated, bound variables are looked up not in the environment where the function was defined, but in the environment where the function was called.
Emacs 24 or later supports lexical binding (that is, the function sees the variables that were bound around the function definition) natively, but since it alters the semantics of existing code you need to enable it explicitly. Usually this is done by adding a file local variable setting to the first line of the .el file:
;; -*- lexical-binding: t; -*-
Another alternative is to use lexical-let from the cl library. This works in earlier Emacs versions as well. Note that in this way you explicitly specify which variables should have lexical binding, so code such as (lexical-let ((foo foo)) ...) is not uncommon — foo is an existing variable which needs to be "carried over" into the function.

Answer 2

The following is an example using dynamic bindings:
(defun example-dynamic-fn ()
"Doc-string"
(interactive)
(let ((test-variable "Hello-world!"))
(set-process-sentinel
(start-process "process-one" "*one*" "echo" test-variable)
`(lambda (p e) (when (= 0 (process-exit-status p))
(set-process-sentinel
(start-process "process-two" "*two*" "echo" ,test-variable)
'(lambda (p e) (when (= 0 (process-exit-status p))
(start-process "process-three" "*three*" "echo" ,test-variable)
(set-process-sentinel
(start-process "process-four" "*four*" "echo" ,test-variable)
'(lambda (p e) (when (= 0 (process-exit-status p))
(set-process-sentinel
(start-process "process-five" "*five*" "echo" ,test-variable)
'(lambda (p e) (when (= 0 (process-exit-status p))
(message "test-variable: %s" ,test-variable)))))))))))))))

Answer 3

OK, I think I've got this now. The link above provides a good example; here's another in case anyone else has this difficulty:
;;; ensure VAR1 has no binding
(makunbound 'VAR1)
;;;
(defun f1 (&optional VAR1)
(interactive)
(unless VAR1
(set 'VAR1 "variable1"))
(pop-to-buffer "*test*")
; (lexical-let ( (VAR1 VAR1) ) ;;;
(set-process-sentinel
(start-process-shell-command "test"
"*test*"
(concat "echo " VAR1))
(lambda (process event)
(condition-case err
(when (string-match-p "finished" event)
(f2 VAR1))
(error
(princ
(format "Sentinel error: %s" err))))))
; ) ;;;
)
;;;
(defun f2 (&optional VAR2)
(interactive)
(unless VAR2
(set 'VAR2 "VARIABLE2"))
(print VAR2))
We load everything (with the lines in (f1) commented out) and run (f1). The value of VAR1 is passed to (f2) before the error occurs. The error (void-variable VAR1) appears to come from the scoping environment of (set-process sentinel PROCESS SENTINEL); VAR1 is not defined there even though it remains in scope for the SENTINEL ((lambda)) function.
Also using (set ) as above is not best practice when a variable is only meant to have scope local to the function.
If we uncomment the lines marked with ; then everything works as expected. Happily, we can pass the value through to another function, which prevents a long series of (set-process sentinel )s building up. It also allows us to generate processes with additional sub-processes, if required.
One of my mistakes was naming the SENTINEL as a discrete function, rather than keeping it inside the (lexical-let ) function. While the lexical-binding: t; approach is attractive, it will tend to break working code which relies on the standard (let ).