[Clusterusers] sluggishness/errors on new fly launches

[Wm. Josiah Erikson]

Also, some additional notes:

1. I've set the "bash" limit tag to 8 per node. Tom seems to use sh
instead of bash, so this won't affect him, I think.

2. If you want to do 4 per node instead, just add the limit tag
"4pernode" to a RemoteCmd in the .alf file, like this:

    RemoteCmd "bash -c "env blah blah" -tags "4pernode" -service "tom"

3. There's also a 2pernode tag

    -Josiah



On 7/19/17 9:30 AM, Wm. Josiah Erikson wrote:
>
> Yeah, it ran out of RAM - probably tried to take 16 jobs
> simultaneously. It's got 64GB of RAM, so should be able to do quite a
> few. It's currently running at under 1/3 RAM usage with 3 jobs, so 8
> jobs should be safe. I'll limit "bash" limit tag jobs (what yours
> default to since you didn't set one explicitly and that's the command
> you're running) to 8 per node. I could lower it to 4 if you prefer.
> Let me know.
>
>     -Josiah
>
>
>
> On 7/19/17 8:38 AM, Thomas Helmuth wrote:
>> Moving today, so I'll be brief.
>>
>> This sounds like the memory problems we've had on rack 4. There,
>> there are just too many cures for the amount of memory, so if too
>> many runs start, none get enough memory. That's why rack 4 isn't on
>> the Tom or big go service tags. Probably means we should remove rack
>> 0 from those tags for now until we figure it out.
>>
>> I've done runs on rack 4 by limiting each run to 1GB of memory.
>>
>> Tom
>>
>> On Jul 19, 2017 7:46 AM, "Lee Spector" <lspector at hampshire.edu
>> <mailto:lspector at hampshire.edu>> wrote:
>>
>>
>>     I launched some runs this morning (a couple of hours ago -- I'm
>>     in Europe) that behaved oddly. I don't know if this is related to
>>     recent upgrades/reconfiguration, or what, but I thought I'd share
>>     them here.
>>
>>     First, there were weird delays between launches and the
>>     generation of the initial output. I'm accustomed to delays of a
>>     couple of minutes at this point, which I think we've determined
>>     (or just guessed) are due to the necessity of re-downloading
>>     dependencies on fly nodes, which doesn't occur when running on
>>     other machines. But these took much longer, maybe 30-45 minutes
>>     or longer (not sure of the exact times, but they were on that
>>     order). And weirder still, there were long times during which
>>     many had launched but only one was producing output and
>>     proceeding for a long time, with others really kicking in only
>>     after the early one(s) finished. I'd understand this if the
>>     non-producing ones weren't launched at all, because they were
>>     waiting for free slots.... but that's not what was happening.
>>
>>     And then one crashed, on compute-0-1 (this was launched with the
>>     "tom" service tag, using the run-fly script in Clojush), with
>>     nothing in the .err file (from standard error) but the following
>>     in the .out file (from standard output):
>>
>>     Producing offspring...
>>     Java HotSpot(TM) 64-Bit Server VM warning: INFO:
>>     os::commit_memory(0x00000006f0e00000, 1059061760, 0) failed;
>>     error='Cannot allocate memory' (errno=12)
>>     #
>>     # There is insufficient memory for the Java Runtime Environment
>>     to continue.
>>     # Native memory allocation (malloc) failed to allocate 1059061760
>>     bytes for committing reserved memory.
>>     # An error report file with more information is saved as:
>>     # /home/lspector/runs/july19a-9235/Clojush/hs_err_pid57996.log
>>
>>      -Lee
>>
>>     --
>>     Lee Spector, Professor of Computer Science
>>     Director, Institute for Computational Intelligence
>>     Hampshire College, Amherst, Massachusetts, 01002, USA
>>     lspector at hampshire.edu <mailto:lspector at hampshire.edu>,
>>     http://hampshire.edu/lspector/, 413-559-5352 <tel:413-559-5352>
>>
>>     _______________________________________________
>>     Clusterusers mailing list
>>     Clusterusers at lists.hampshire.edu
>>     <mailto:Clusterusers at lists.hampshire.edu>
>>     https://lists.hampshire.edu/mailman/listinfo/clusterusers
>>     <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>
>>
>>
>>
>> _______________________________________________
>> Clusterusers mailing list
>> Clusterusers at lists.hampshire.edu
>> https://lists.hampshire.edu/mailman/listinfo/clusterusers
>
> -- 
> Wm. Josiah Erikson
> Assistant Director of IT, Infrastructure Group
> System Administrator, School of CS
> Hampshire College
> Amherst, MA 01002
> (413) 559-6091
>
>
> _______________________________________________
> Clusterusers mailing list
> Clusterusers at lists.hampshire.edu
> https://lists.hampshire.edu/mailman/listinfo/clusterusers

-- 
Wm. Josiah Erikson
Assistant Director of IT, Infrastructure Group
System Administrator, School of CS
Hampshire College
Amherst, MA 01002
(413) 559-6091

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.hampshire.edu/pipermail/clusterusers/attachments/20170719/61710a4b/attachment-0001.html>