[Cs254f11] was: subst now: multithreading

[Wm. Josiah Erikson]

Interestingly, if I put the pmap here in sort-by-error:

(defn sort-by-error
   [population]
   (vec (pmap second
             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
                   (map #(vector (error %) %) population)))))

It works, and I see all four CPU's working, though not very hard, I 
guess because the second "map" call is lazy, so clojure is helping me 
out here. Putting pmapall there works too... so I guess that solves the 
problem and actually is fine, but I still don't know why it didn't work 
in the other place. Maybe because lazy calls to pmap don't work? That 
might make some sense, actually...

However, I'm not seeing a lot of CPU utilization on each core, though 
all of them are working. I've only got like ~132% maximum. I guess that 
means that it's not CPU that's slowing me down, it's memory bandwidth? 
Interesting. Would you agree with my assessment, Lee?

     -Josiah



On 11/21/11 10:28 AM, Wm. Josiah Erikson wrote:
> Wow yeah, a load of 48 is pretty much perfect CPU utilization on that 
> box (yes, folks, it has 48 physical processor cores!)
>
> Unfortunately, I haven't been able to test this yet, due to probably 
> bugs in my program or something about function definition scope or 
> something. So I'm just changing this in your definition:
>
> (defn sort-by-error
>   [population]
>   (vec (map second
>             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
>                   (map #(vector (error %) %) population)))))
>
> to:
>
> (defn sort-by-error
>   [population]
>   (vec (map second
>             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
>                   (pmap #(vector (error %) %) population)))))
>
> ...and then it complains that it doesn't know what pmod, or pd, or 
> pquot, or prem (whichever it encounters first) are:
>
> #'critic_evolution.core/sort-by-error
> critic_evolution.core=>
> (evolve 1000)
> Starting evolution...
> #<CompilerException java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
> java.lang.Exception: Unable to resolve symbol: pmod in this context 
> (NO_SOURCE_FILE:0) (NO_SOURCE_FILE:0)>
>
> critic_evolution.core=>
>
> Weird. pmod is just defined like this:
>
> (defn pmod
>   "Protected modulus; returns 0 if the denominator is zero."
>   [num denom]
>   (if (zero? denom)
>     0
>     (mod num denom)))
>
> and the rest of them are similar. I tried using your pmap-all and got 
> a different error:
>
> critic_evolution.core=>
> (evolve 1000)
> Starting evolution...
> #<CompilerException java.lang.RuntimeException: 
> java.lang.ClassCastException: clojure.lang.Symbol cannot be cast to 
> java.lang.Number (NO_SOURCE_FILE:0)>
>
> Well, it'll be fun to figure that out. The problem with pmap I don't 
> get, but Lee, you might have an answer. The problem with pmapall I'm 
> sure is easily solvable - I'm gonna try to figure it out now, but 
> perhaps somebody else can see it right away.
>
> pmapall is defined below in the quoted part of this email. The 
> relevant call to pmapall looks like this:
>
> (pmapall #(vector (error %) %) population)))
>
> The error function looks like this:
>
> ;Using the terminal map in a new, better way! Just replace those 
> terminals with numbers and then eval. Nice.
> (defn error
>   "Takes an individual, replaces its terminals with numbers looked up 
> in terminal-map, then evaluates it,
>    casts that to an int, compares its values to scores, adds the 
> differences up, and returns the total error"
>   [individual]
>    (reduce + (map #(Math/abs (- (int (eval (postwalk-replace % 
> individual))) %2)) terminal-map scores)))
>
> It works, and returns a number (an int, I think, unless it's been 
> automatically promoted). I'm not sure where we are accidentally 
> passing a symbol to something that wants a number. Perhaps somebody 
> else can spot it.
>
> -Josiah
>
> On 11/18/11 7:23 PM, Lee Spector wrote:
>> (defn pmapall
>>    "Like pmap but: 1) coll should be finite, 2) the returned sequence
>>     will not be lazy, 3) calls to f may occur in any order, to maximize
>>     multicore processor utilization, and 4) takes only one coll so far."
>>    [f coll]
>>    (let [agents (map #(agent % :error-handler (fn [agnt except] 
>> (println except))) coll)]
>>      (dorun (map #(send % f) agents))
>>      (apply await agents)
>>      (doall (map deref agents))))
>

-- 
Wm. Josiah Erikson
Network Engineer
Hampshire College
Amherst, MA 01002
(413) 559-6091