[Cs254f11] was: subst now: multithreading

[Wm. Josiah Erikson]

On second thought, after some more benchmarking, I don't think putting a 
pmap there does anything for me at all. You can't have a lazy sort call 
- that makes no sense, and my benchmarking bears that out. I think that 
all I was seeing with all four CPU's working, though not much, was just 
clojure letting the other lazy calls happen concurrently when possible 
elsewhere.
Benchmarking with "time" says I spend 95% of my time in sort-by-error, 
which makes sense to me - that's where everything happens, obviously. I 
do need to make the calls to error concurrent, unless it's actually just 
the sorting itself that takes all the time. Time to figure that out, I 
guess!
     -Josiah



On 11/21/11 11:16 AM, Wm. Josiah Erikson wrote:
> Interestingly, if I put the pmap here in sort-by-error:
>
> (defn sort-by-error
>   [population]
>   (vec (pmap second
>             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
>                   (map #(vector (error %) %) population)))))
>
> It works, and I see all four CPU's working, though not very hard, I 
> guess because the second "map" call is lazy, so clojure is helping me 
> out here. Putting pmapall there works too... so I guess that solves 
> the problem and actually is fine, but I still don't know why it didn't 
> work in the other place. Maybe because lazy calls to pmap don't work? 
> That might make some sense, actually...
>
> However, I'm not seeing a lot of CPU utilization on each core, though 
> all of them are working. I've only got like ~132% maximum. I guess 
> that means that it's not CPU that's slowing me down, it's memory 
> bandwidth? Interesting. Would you agree with my assessment, Lee?
>
>     -Josiah
>
>
>
> On 11/21/11 10:28 AM, Wm. Josiah Erikson wrote:
>> Wow yeah, a load of 48 is pretty much perfect CPU utilization on that 
>> box (yes, folks, it has 48 physical processor cores!)
>>
>> Unfortunately, I haven't been able to test this yet, due to probably 
>> bugs in my program or something about function definition scope or 
>> something. So I'm just changing this in your definition:
>>
>> (defn sort-by-error
>>   [population]
>>   (vec (map second
>>             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
>>                   (map #(vector (error %) %) population)))))
>>
>> to:
>>
>> (defn sort-by-error
>>   [population]
>>   (vec (map second
>>             (sort (fn [[err1 ind1] [err2 ind2]] (< err1 err2))
>>                   (pmap #(vector (error %) %) population)))))
>>
>> ...and then it complains that it doesn't know what pmod, or pd, or 
>> pquot, or prem (whichever it encounters first) are:
>>
>> #'critic_evolution.core/sort-by-error
>> critic_evolution.core=>
>> (evolve 1000)
>> Starting evolution...
>> #<CompilerException java.lang.RuntimeException: 
>> java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
>> java.lang.Exception: Unable to resolve symbol: pmod in this context 
>> (NO_SOURCE_FILE:0) (NO_SOURCE_FILE:0)>
>>
>> critic_evolution.core=>
>>
>> Weird. pmod is just defined like this:
>>
>> (defn pmod
>>   "Protected modulus; returns 0 if the denominator is zero."
>>   [num denom]
>>   (if (zero? denom)
>>     0
>>     (mod num denom)))
>>
>> and the rest of them are similar. I tried using your pmap-all and got 
>> a different error:
>>
>> critic_evolution.core=>
>> (evolve 1000)
>> Starting evolution...
>> #<CompilerException java.lang.RuntimeException: 
>> java.lang.ClassCastException: clojure.lang.Symbol cannot be cast to 
>> java.lang.Number (NO_SOURCE_FILE:0)>
>>
>> Well, it'll be fun to figure that out. The problem with pmap I don't 
>> get, but Lee, you might have an answer. The problem with pmapall I'm 
>> sure is easily solvable - I'm gonna try to figure it out now, but 
>> perhaps somebody else can see it right away.
>>
>> pmapall is defined below in the quoted part of this email. The 
>> relevant call to pmapall looks like this:
>>
>> (pmapall #(vector (error %) %) population)))
>>
>> The error function looks like this:
>>
>> ;Using the terminal map in a new, better way! Just replace those 
>> terminals with numbers and then eval. Nice.
>> (defn error
>>   "Takes an individual, replaces its terminals with numbers looked up 
>> in terminal-map, then evaluates it,
>>    casts that to an int, compares its values to scores, adds the 
>> differences up, and returns the total error"
>>   [individual]
>>    (reduce + (map #(Math/abs (- (int (eval (postwalk-replace % 
>> individual))) %2)) terminal-map scores)))
>>
>> It works, and returns a number (an int, I think, unless it's been 
>> automatically promoted). I'm not sure where we are accidentally 
>> passing a symbol to something that wants a number. Perhaps somebody 
>> else can spot it.
>>
>> -Josiah
>>
>> On 11/18/11 7:23 PM, Lee Spector wrote:
>>> (defn pmapall
>>>    "Like pmap but: 1) coll should be finite, 2) the returned sequence
>>>     will not be lazy, 3) calls to f may occur in any order, to maximize
>>>     multicore processor utilization, and 4) takes only one coll so 
>>> far."
>>>    [f coll]
>>>    (let [agents (map #(agent % :error-handler (fn [agnt except] 
>>> (println except))) coll)]
>>>      (dorun (map #(send % f) agents))
>>>      (apply await agents)
>>>      (doall (map deref agents))))
>>
>

-- 
Wm. Josiah Erikson
Network Engineer
Hampshire College
Amherst, MA 01002
(413) 559-6091