[Cs254f11] Macro confusion

[Lee Spector]

On Oct 11, 2011, at 10:45 PM, Omri Bernstein wrote:

> Thanks Lee! Writing macros does feel quite like an art, and I seem to figure it out more with trial and error than with formal logic.
> 
> Yeah, I already had a look at clojure.contrib.combinatorics. It has a lot of cool stuff, but not anything that completely did what I was trying to do, so I figured I'd make my own. And, as you said, it's a good learning experience.
> 
> I did a macro so that I could pass the "for" a theoretically limitless vector. I could just changed the "defmacro" to a "defn", but then I would just have to evaluate what it returns. If I could come up with an alternative for using "for" as the core function, then I could probably do it as a function. Actually, I was originally trying to use "map", but I wasn't successful in getting past a combins function that could handle any more than two collections:
> 
> (defn combins
>   [f c1 c2]
>     (map (fn [n] (map (partial f n) c2)) c1)
> 
> I did a mental back-flip just getting this far, and I'm struggling with how to even add a third collection to the series, let alone generalize it to [f & colls]. Any thoughts?


Ah -- now I see the temptation of writing a macro here. You wanted to construct a "for" structure and then evaluate it. Ironically, the reason that this would seem to call for a macro is that "for" itself is a macro, and this can cause the kinds of complications to which I was alluding when I warned about macro over-use. You can't just dynamically create the bindings part of a "for" and then apply "for" to it, because "for" is a macro... Of course "for" *is* very handy, and it has to be a macro to do what it does, so I'm not saying it's bad. But it can't be used as flexibly as a function can be used.

In any event, there are a variety of ways in which you might do what you want without writing a macro, and the clearest (to me, right now anyway) would do it without using "for" at all. Here's one way that's particularly simple because it leverages clojure.contrib.combinatorics:

(use 'clojure.contrib.combinatorics)

(defn combins
  [f & colls]
  (map #(apply f %) (apply cartesian-product colls)))


> As for the errors I got, I see why my fix solved the "eager evaluation" problem that was happening. As you said, it was trying to call \t as a function. What I don't understand is what the problem was when I originally tried casting the argument as a vector--which should evaluate to itself and not call \t as a function--which it didn't. However, it doesn't return the correct answer:
> 
> (combins try-args [+ -] (vec (keys arg-types))
> >> (nil nil nil nil nil nil nil nil nil nil nil nil nil nil)
> 
> When I did that fix, it didn't "error", but it did give me an incorrect result. I don't understand why it was giving me incorrect results, and I don't understand why doing `~colls instead of colls changed that.


It's hard for me to reconstruct what definitions you had in place when you got this output. When I try I get an error because you have "test-types" in your "try-args" function where I think you meant to have "arg-types".... Maybe you had something called test-types previously? In any event, when I change this then from your call I get a different (maybe correct?) result:

(nil
 nil
 nil
 (#<core$_PLUS_ clojure.core$_PLUS_ at 27af9da2> :number)
 nil
 nil
 nil
 nil
 nil
 nil
 (#<core$_ clojure.core$_ at 1506291d> :number)
 nil
 nil
 nil)

 -Lee





> 
> -Omri
> 
> 
> On Tue, Oct 11, 2011 at 4:50 PM, Lee Spector <lspector at hampshire.edu> wrote:
> 
> One other thing that may be of interest: clojure.contrib.combinatorics
> 
>  -Lee
> 
> 
> On Oct 11, 2011, at 4:22 PM, Lee Spector wrote:
> 
> >
> > Lots of good stuff in there!
> >
> > A few comments:
> >
> > - Is there really any reason to make this a macro? Why not just a function that takes the same "& colls" argument? As far as I can tell that would work just as well in this case and be less confusing. On the other hand, you got some good practice writing macros :-). But on the third hand you should only use macros when you really need them -- which is usually when you want to control whether code passed in as arguments gets evaluated (or how many times it gets evaluated). One of the downsides of writing macros when functions could do the job is that you can pass a function to another function, map it down a sequence, etc., but you can't do any such thing with a macro.
> >
> > - You may well be able to get the argument/type information that you're computing here from Java "reflection" rather than trying things and catching exceptions. I'm not sure, and again what you're doing is a cool exercise in any event, but you might want to look into this.
> >
> > - Writing macros is an art, and a pretty deep one. Paul Graham's book "On Lisp" is largely about the art of macro writing, and I think it's extremely cool. But it uses Common Lisp rather than Clojure, so some but not all of it will apply to Clojure.
> >
> > - In my macro-writing past, which was mostly in Common Lisp, I sometimes ended up doing the same kind of trick that you did with backquote-tilde, although it would have been backquote-comma in common lisp. Generally this happens when you want the macro's argument to be evaluated, but then you want that value to be quoted in the macro's expansion code. In your case evaluating the argument to the macro call will give (\t [1 2 3 4] + :42 #"ab" #{1 2 3 4} "Text is in here.") and you want that to be quoted in the macro expansion's call to interleave. If it isn't quoted there then when the expansion is evaluated it will try to call \t as a function, which gives the error that you saw.
> >
> > I think that that last point answers your question, but I also think that you should look into the other points, particularly the first.
> >
> > -Lee
> >
> >
> > On Oct 11, 2011, at 3:50 PM, Omri Bernstein wrote:
> >
> >> As a presage to this email, I spend a lot of time explaining, and my question doesn't really come until the very end. Also, by the time I got to the end of writing , I managed to accidentally solve my original question, and now I'm curious whether anybody can tell me why what I did solved the problem.
> >>
> >> I've been trying to work on getting the arities for all the clojure functions, and what types of arguments each arity for each function will accept. I'm on the second part now, and I plan on going about it by trying all the possible combinations of argument types, and returning the types of the ones that don't error. So I decided to make a general-purpose macro for returning a list of all the possible combinations of applying a function f to a series of collections--using each value from the first collection as the first argument to f, and each from the second collection as the second argument to f, etc. If you understand how the "for" loop comprehension works (if not, maybe the examples here would help), this is an example of how the concept would work, using + as the function, [1 2 3] as the first collection, and [10 20] as the second collection:
> >>
> >> (for [elem1 [1 2 3] elem2 [10 20]]
> >>  (+ elem1 elem2))
> >>>> (11 21 12 22 13 23)
> >>
> >> This could be generalized to:
> >>
> >> (defn combins
> >>  [f c1 c2]
> >>  (for [elem1 c1 elem2 c2]
> >>    (f elem1 elem2))
> >>
> >> (combins + [1 2 3] [10 20])
> >>>> (11 21 12 22 13 23)
> >>
> >> The problem, which may be apparent after looking at it, is that something like the "combins" function above is not well suited to a long series of collections (frankly, for what I plan on using it for, this is not a necessary feature--but still). In order to do a combination of + with [0 128], [0 64], [0 32], [0 16], [0 8], [0 4], [0 2], and [0 1] (which should return a list of all the positive integers that are less than 256), I would have to create a function that looked something like:
> >>
> >> (defn combins2
> >>  [f c1 c2 c3 c4 c5 c6 c7 c8]
> >>  (for [elem1 c1 elem2 c2 elem3 c3 elem4 c4 elem5 c5 elem6 c6 elem7 c7 elem8 c8]
> >>    (f elem1 elem2 elem3 elem4 elem5 elem6 elem7 elem8))
> >>
> >> (combins + [0 128] [0 64] [0 32] [0 16] [0 8] [0 4] [0 2] [0 1])
> >>>> (1 2 3 4 5 6 7 8 9 10 ... 250 251 252 253 254 255)
> >>
> >> I figured that creating a macro would be a more elegant way to handle a theoretically limitless series of collections. A quick introduction to macros: macros are functions that generate code as output. When writing a macro, you use symbols like `, ~, and ~@. The ` symbol (called the backquote symbol) is used a lot like the ' symbol (but they are not the same thing), to denote something that should be returned as is, not evaluated. The ~ symbol (called the unquote symbol) is used to denote something that should be evaluated and then that return value should be inserted as is. The ~@ symbol (called the unqoute splice symbol, or something) is pretty cool, it returns the elements inside of what it evaluates to. Both of the unquote symbols have to be used inside a backquoted block of code. Here's an example of how it works:
> >>
> >> (def example-collection [1 2 3])
> >>
> >> ~example-collection
> >>>> #<CompilerException java.lang.IllegalStateException: Var clojure.core/unquote is unbound. (NO_SOURCE_FILE:0)>
> >>
> >> ~@example-collection
> >>>> #<CompilerException java.lang.IllegalStateException: Var clojure.core/unquote-splicing is unbound. (NO_SOURCE_FILE:0)>
> >>
> >> `(list 0 example-collection 4 5 6)
> >>>> (clojure.core/list 0 user.core/example-collection 4 5 6)
> >>
> >> `(list 0 ~example-collection 4 5 6)
> >>>> (clojure.core/list 0 [1 2 3] 4 5 6)
> >>
> >> `(list 0 ~@example-collection 4 5 6)
> >>>> (clojure.core/list 0 1 2 3 4 5 6)
> >>
> >> Note the difference between the above examples versus using ' instead of `, which just returns exactly what comes after it.
> >>
> >> '(list 0 example-collection 4 5 6)
> >>>> (list 0 example-collection 4 5 6)
> >>
> >> '(list 0 ~example-collection 4 5 6)
> >>>> (list 0 ~example-collection 4 5 6)
> >>
> >> '(list 0 ~@example-collection 4 5 6)
> >>>> (list 0 ~@example-collection 4 5 6)
> >>
> >> In my case I want to generate a "for" with a variable number of bindings. Essentially, instead of passing a "hard-coded" vector of variables and bindings to the "for" list comprehension, I'm passing a vector that has been derived. After a lot (a lot) of attempts, I managed to come up with something elegant and concise, using an infinite lazy list of "anonymous" symbols (in clojure, I believe it is idiomatic to use an underscore to denote a variable that is being defined whose name is not important). For people unfamiliar with macros--I'm kind of jumping right into the deep-end here, apologies:
> >>
> >> (def infinite-symbol-list
> >>  (map #(symbol (format "_%d" %)) (iterate inc 1)))
> >>
> >> (take 4 infinite-symbol-list)
> >>>> (_1 _2 _3 _4)
> >>
> >> (defmacro combins
> >>  [f & colls]
> >>  `(for ~(vec (interleave infinite-symbol-list colls))
> >>     (~f ~@(take (count colls) infinite-symbol-list))))
> >>
> >> (combins + [0 128] [0 64] [0 32] [0 16] [0 8] [0 4] [0 2] [0 1])
> >>>> (1 2 3 4 5 6 7 8 9 10 ... 250 251 252 253 254 255)
> >>
> >> I'll try to quickly explain the macro I wrote. When you have "& var-name" at the end of an argument list for a function, it makes a list out of the remaining arguments and stores that in "var-name". For example:
> >>
> >> (defn example-fn
> >>  [a & more]
> >>  (list a more))
> >>
> >> (example-fn 1 2 3 4 5 6 7 8)
> >>>> (1 (2 3 4 5 6 7 8))
> >>
> >> The "interleave" function takes two collections, and then returns a list of (first coll1), (first coll2), (second coll1), (second coll2), etc. until one of the collections runs out of elements. Here, I am interleaving the infinite symbol list (_1 _2 _3 ...) with the "colls"--which should be a collection of the series of collections passed to combins. Let's represent colls as (c1 c2 c3 ... cN) where cN is the last collection in the series. Interleave these will return (_1 c1 _2 c2 _3 c3... _N cN). Calling "vec" on this will return [_1 c1 _2 c2 _3 c3 ... _N cN]. Since ~ will return the result of evaluating what is after it, `(for ~(vec (interleave infinite-symbol-list colls)) ...) will return (clojure.core/for [_1 c1 _2 c2 _3 c3 ... _N cN] ...) which is exactly what I want. After this, I want to call f using _1 as the first argument, _2 as the second argument, etc. I have to do ~f to ensure that I am using "the function inside f" (e.g. "+") as opposed to "f itself" (which doe
> > s not exist). Then, because infinite-symbol-list is infinite (shocker, I know) we have to make sure to take only what we need of it. Doing (take (count colls) infinite-symbol-list) will return (_1 _2 _3 ... _N), where N is the number of collections in the series (i.e. "(count colls)"). Since ~@ will return the elements inside of the return value of the stuff after it, ~@ should return _1 _2 _3 ... _N. So, in the example that f is +, (~f ~@(take (count colls) infinite-symbol-list)) will return (+ _1 _2 _3 ... _N). Great. Still using + as the example f, the macro as a whole will return and then evaluate:
> >>
> >> (clojure.core/for [_1 c1 _2 c2 _3 c3 ... _N cN]
> >>  (+ _1 _2 _3 ... _N))
> >>
> >> I hope all of that made sense. So...my question. Originally, I told you that I was planning on testing each arity of each function, using all the possible combinations of argument types, to discover what argument types each arity would accept. If I were just trying to test a single list of argument types on a single function, I could do this:
> >>
> >> (def arg-types
> >>  {\t :character
> >>   [1 2 3 4] :collection
> >>   + :function
> >>   42 :number
> >>   #"ab" :regex
> >>   #{1 2 3 4} :set
> >>   "Text is in here." :string})
> >>
> >> (defn try-args
> >>  [f & elems]
> >>  (try (do (apply f elems) (cons f (map test-types elems)))
> >>       (catch Exception e)))
> >>
> >> (try-args + \t)
> >>>> nil
> >>
> >> (try-args + 42)
> >>>> (#<core$_PLUS_ clojure.core$_PLUS_ at 507726> :number)
> >>
> >> (try-args + 42 \t)
> >>>> nil
> >>
> >> (try-args + 42 42)
> >>>> (#<core$_PLUS_ clojure.core$_PLUS_ at 507726> :number :number)
> >>
> >> All that #<core$_PLUS...> nonsense is equivalent to +. So it makes sense to me that I could do the following to test what argument types a 1-arity call to + or - accepts:
> >>
> >> (combins try-args [+ -] (keys arg-types))
> >>>> #<ClassCastException java.lang.ClassCastException: java.lang.Character cannot be cast to clojure.lang.IFn>
> >>
> >> But I can't, and it looks like it's trying to evaluate the result of (keys (arg-types)), and I don't understand why, but if I cast the result into a vector that should fix the problem:
> >>
> >> (combins try-args [+ -] (vec (keys arg-types))
> >>>> (nil nil nil nil nil nil nil nil nil nil nil nil nil nil)
> >>
> >> So it "works" now, but it's not returning what it "should" be. Here's the equivalent of what I expect the macro to return and evaluate:
> >>
> >> (for [_1 [+ -] _2 (vec (keys arg-types))]
> >>  (try-args _1 _2))
> >>>> (nil nil nil (#<core$_PLUS_ clojure.core$_PLUS_ at 507726> :number) nil nil nil nil nil nil (#<core$_ clojure.core$_ at ac9cbe> :number) nil nil nil)
> >>
> >> Wow, just right now, at this point in writing this email, I just fixed the problem. If instead of:
> >>
> >> (defmacro combins
> >>  [f & colls]
> >>  `(for ~(vec (interleave infinite-symbol-list colls))
> >>     (~f ~@(take (count colls) infinite-symbol-list))))
> >>
> >> I change colls to `~colls inside the interleave:
> >>
> >> (defmacro combins
> >>  [f & colls]
> >>  `(for ~(vec (interleave infinite-symbol-list `~colls))
> >>     (~f ~@(take (count colls) infinite-symbol-list))))
> >>
> >> (combins try-args [+ -] (keys arg-types))
> >>>> (nil nil nil (#<core$_PLUS_ clojure.core$_PLUS_ at 507726> :number) nil nil nil nil nil nil (#<core$_ clojure.core$_ at ac9cbe> :number) nil nil nil)
> >>
> >> It works! Can anybody explain why that works? I just now tried this thinking it might solve the problem where it seemed to be evaluating the terms in colls. I figured that back-quoting the return value might stop the return value from further evaluation. But I have no idea why this solves the problem of it returning nils where it should not have been--I still don't even know why it was doing that incorrectly in the first place.
> >>
> >> -Omri
> >> _______________________________________________
> >> Cs254f11 mailing list
> >> Cs254f11 at lists.hampshire.edu
> >> https://lists.hampshire.edu/mailman/listinfo/cs254f11
> >
> > --
> > Lee Spector, Professor of Computer Science
> > Cognitive Science, Hampshire College
> > 893 West Street, Amherst, MA 01002-3359
> > lspector at hampshire.edu, http://hampshire.edu/lspector/
> > Phone: 413-559-5352, Fax: 413-559-5438
> >
> > _______________________________________________
> > Cs254f11 mailing list
> > Cs254f11 at lists.hampshire.edu
> > https://lists.hampshire.edu/mailman/listinfo/cs254f11
> 
> --
> Lee Spector, Professor of Computer Science
> Cognitive Science, Hampshire College
> 893 West Street, Amherst, MA 01002-3359
> lspector at hampshire.edu, http://hampshire.edu/lspector/
> Phone: 413-559-5352, Fax: 413-559-5438
> 
> 
> _______________________________________________
> Cs254f11 mailing list
> Cs254f11 at lists.hampshire.edu
> https://lists.hampshire.edu/mailman/listinfo/cs254f11

--
Lee Spector, Professor of Computer Science
Cognitive Science, Hampshire College
893 West Street, Amherst, MA 01002-3359
lspector at hampshire.edu, http://hampshire.edu/lspector/
Phone: 413-559-5352, Fax: 413-559-5438