[Clusterusers] request for CPU time

Fri Jun 28 09:10:00 EDT 2013

Yes, you are probably correct. I will look forward to thinking about 
this with others at the meeting today. I could get up and describe the 
mechanisms we have available to us, and list them on the whiteboard, and 
we could brainstorm about the best way to do it right to be most fair to 
all, given the mechanisms that we have, which are many, actually. I will 
list them here. Why not? Thinking out loud here, in preparation for the 
meeting. Consider this a brain-dump and don't look for answers here, 
just thoughts:

1. We can limit the number of concurrent jobs of a particular executable 
that can be launched at once on the same node
2. We can limit the number of concurrent jobs of a particular executable 
that can be launched at once cluster-wide
3. We can craft tags that people can dispatch with that are only 
provided by certain nodes (tom's got one. It's called "tom". We can do 
whatever we want with it. We also have "slow" and "fast" tags that were 
kindof for this purpose. "Slow" was for long, slow jobs and "fast" was 
for quick-turnaround jobs. Perhaps tom could be dispatching with 
"slow,linux", Jaime could be dispatching with "fast,linux", and bassam 
could be dispatching with "blender" and it would do what we want). 
Certain nodes are really good for some things and not so good for 
others. For instance, the high-core-count nodes rock really hard on many 
blender jobs (though not all, unfortunately), and do really poorly for 
Jaime's stuff, and are mediocre for Tom's stuff, even though it looks 
like it's using up all the CPU available (but look at the jobs that 
finished first - none of them are on rack 4). The problem is that Tom's 
stuff does badly on rack 4, but uses all the available CPU to no 
advantage, while Jaime's stuff does badly on rack 4, but leaves slots 
and CPU open for other jobs while doing so... so the way to use the 
cluster to full advantage is clear to me while everybody wants CPU time, 
but we don't want to leave ourselves in a situation where the cluster is 
empty and somebody comes along and wants to render and doesn't get to 
use the whole cluster. I think we can get this right, possibly just with 
the right combination of tags and limits.
4. We can play with priorities. Jobs can be dispatched with different 
priorities that will determine whether they will get a slot 
preferentially over another job when they both want it. This still won't 
fix the fact that once a job gets a slot, it holds on to it until it 
finishes... I don't think we can or want to fix that, exactly. There 
used to be some pre-empt functionality, but I think we don't want to do 
that (certain jobs would have the ability to actually bump others off), 
since when that happens to Lee's and Tom's long-running jobs, they lose 
work and then we're wasting CPU time.
5. There are crews and also time-scheduling, but I'm not sure that they 
will do anything useful for us. I think NIMBYing machines that we don't 
want jobs dispatched to is easy enough, now that it works from the 
tractor monitor.
     -Josiah

On 6/28/13 8:00 AM, Jaime Davila wrote:
> HMMM. Interesting. One consequence of that is that the system ends up 
> penalizing tasks that have been parallelized, since those are going to 
> be seen by tractor as a bunch of small tasks, as opposed to a single 
> big one. I've spent several months turning the breve evaluation of a 
> genetic population into something that spreads the evaluation of 
> separate individuals across the cluster, and I'm planning on writing 
> up how I did it. I would be glad to share whatever I learned from that 
> process with others, which will probably apply even outside of breve, 
> if they want to parallelize.
>
> On 06/27/2013 02:23 PM, Wm. Josiah Erikson wrote:
>> The issue right now is that many of Tom's Digital Multiplier 
>> processes seem to be taking around 9 days to finish, so once they've 
>> grabbed a slot, they hold on to it for a very long time, which 
>> effectively means that nobody else can use the cluster while they are 
>> running, since we don't have any kicking-people-out algorithm, and 
>> everybody else's jobs finish in more like 20 minutes.
>> The solution, of course, is to change the "tom" tag to only use every 
>> other slot or something like that, or only run two processes per 
>> machine or something... but then they'll take even longer to finish.
>> Discussing tomorrow at the meeting seems like a good plan.
>>     -Josiah
>>
>>
>> On 6/27/13 2:19 PM, Lee Spector wrote:
>>> Hi Jaime,
>>>
>>> Fine with me personally but I'll check with my lab group to see what 
>>> everyone's expected needs are. I'm also not sure exactly how to 
>>> implement the idea if people do want to run some other things... 
>>> maybe by having you use a subset of machines that everyone else 
>>> excludes?
>>>
>>>   -Lee
>>>
>>> On Jun 27, 2013, at 2:14 PM, Jaime Davila wrote:
>>>
>>>> Greetings everyone,
>>>>
>>>> I wanted to check to see if it was possible for me to grab some 
>>>> more CPU cycles out of the cluster for a week or so. I just placed 
>>>> a new algorithm on my account, and have it run it, but it's fairly 
>>>> different from what I was doing before, and I would rather detect 
>>>> quickly if I need to tweak or change things, as opposed to having 
>>>> to wait a week to realize I need to make a 10 minute change.
>>>>
>>>> Last time that the system load diminished some, I noticed that my 
>>>> processes run at their top speed if the number of CPUs loaded to 
>>>> their maximum drops to about 75%, as opposed to the 97% where they 
>>>> are now. Maybe things will be that way this time around, maybe not? 
>>>> In either case, my grabbing more cpu cycles right now would be 
>>>> extremely useful.
>>>>
>>>> Thoughts?
>>>>
>>>> Thanks a lot,
>>>>
>>>> Jaime
>>>>
>>>> -- 
>>>> ******************************************************
>>>> Jaime J. Dávila
>>>> Associate Professor of Computer Science
>>>> Hampshire College
>>>> jdavila at hampshire dot edu
>>>> http://helios.hampshire.edu/jdavila
>>>> *******************************************************
>>>>
>>>> _______________________________________________
>>>> Clusterusers mailing list
>>>> Clusterusers at lists.hampshire.edu
>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers
>>> -- 
>>> Lee Spector, Professor of Computer Science
>>> Cognitive Science, Hampshire College
>>> 893 West Street, Amherst, MA 01002-3359
>>> lspector at hampshire.edu, http://hampshire.edu/lspector/
>>> Phone: 413-559-5352, Fax: 413-559-5438
>>>
>>> _______________________________________________
>>> Clusterusers mailing list
>>> Clusterusers at lists.hampshire.edu
>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers
>>
>

-- 
Wm. Josiah Erikson
Assistant Director of IT, Infrastructure Group
System Administrator, School of CS
Hampshire College
Amherst, MA 01002
(413) 559-6091