[Clusterusers] problem running lein/clojure jobs on fly

lspector at hampshire.edu lspector at hampshire.edu
Sat Dec 8 12:02:08 EST 2018 

Mostly for Josiah again, but some others here might have the same problem and/or a solution...

Subsequent to the old thread below Josiah upgraded Java and then everything seemed to be fine.

Today I tried to launch a batch of Clojush runs using its run-fly script, which talks to Tractor, and I got a similar error (see below). The error happens when leiningen tries to pull in the dependencies for the Clojure project from the web.

I believe that the upgrade made it work when launching a single run manually on a fly node, but not when launching a batch of runs via run-fly/Tractor.

"But wait!", you may say, "You successfully launched a whole bunch of batches of runs via run-fly/Tractor after the Java upgrade, and it all worked beautifully!"

Ah, but those were all with the same version of Clojush, with the same dependencies. I was testing things via single launches, and once one of those succeeded it stashed all of the retrieved dependencies in some local hidden directory, as it does, and when I subsequently launched batches of runs via run-fly/Tractor they all worked beautifully but only because it didn't need to fetch any dependencies.

Today, I was running with a newer version of Clojush, which has a few updated dependencies.

Piecing some of this together, I ssh-ed to a fly node and launched a single manual run, which I killed shortly after the dependencies had been fetched. Then I repeated my run-fly/Tractor launch and everything is now running fine.

I *could* jump through this hoop every time I hit this error, but it would be nice to make this work correctly.

Not sure where to look for a fix, but any fixes/pointers would be appreciated!

 -Lee

-- here's today's error:

[lspector at fly output]$ more /home/lspector/runs/20181208111011-9054/output/1.err
Retrieving cider/cider-nrepl/0.18.0/cider-nrepl-0.18.0.pom from releases
Retrieving nrepl/nrepl/0.4.4/nrepl-0.4.4.pom from releases
Retrieving nrepl/nrepl/0.4.4/nrepl-0.4.4.jar from releases
Retrieving cider/cider-nrepl/0.18.0/cider-nrepl-0.18.0.jar from releases
Could not transfer artifact org.clojure:tools.logging:jar:0.4.1 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
Could not transfer artifact org.clojure:tools.logging:jar:0.4.1 from/to clojars (https://repo.clojars.org/ <https://repo.clojars.org/>): Received fatal alert: protocol_version
Could not find artifact org.clojure:tools.logging:jar:0.4.1 in releases (https://clojars.org/repo <https://clojars.org/repo>)
Could not transfer artifact org.clojure:tools.logging:pom:0.4.1 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.




> On Oct 3, 2018, at 10:02 PM, Lee Spector <lspector at hampshire.edu <mailto:lspector at hampshire.edu>> wrote:
> 
> Thanks Josiah,
> 
> Tom and I have been experimenting a bit and everything is working fine for him... and in fact he is using Java 8. Here's how he knows:
> 
> $ /share/apps/bin/lein -v
> Leiningen 2.8.1 on Java 1.8.0_172 Java HotSpot(TM) 64-Bit Server VM
> 
> But I get:
> 
> [lspector at fly ~]$  /share/apps/bin/lein -v
> Leiningen 2.8.1 on Java 1.7.0_51 Java HotSpot(TM) 64-Bit Server VM
> 
> So both versions of Java must be installed, and I don't know why I'm getting the old version, while Tom is getting the newer one.
> 
> Any ideas about that?
> 
>  -Lee
> 
> 
> 
> 
>> On Oct 3, 2018, at 10:15 AM, Wm. Josiah Erikson <wjerikson at hampshire.edu <mailto:wjerikson at hampshire.edu>> wrote:
>> 
>> Google says:
>> 
>> Workarounds are: use Java 8, downgrade to lein 2.7 or use LEIN_JVM_OPTS=-Dhttps.protocols=TLSv1.2
>> 
>> I would try setting that environment variable. If that doesn't work, we can look into upgrading to Java 8
>> 
>>    -Josiah
>> 
>> 
>> 
>> 
>> On 10/3/18 9:15 AM, Lee Spector wrote:
>>> Progress! But not yet quite there yet.
>>> 
>>> I edited my .bash_profile to add /share/apps/bin/ to my path, logged out and in again, ran lein, and it did some stuff, and now I can run lein to get the standard message that that usually provides, and everything looks okay, which is an improvement. But trying to run Clojush either on the head node or a compute node gives:
>>> 
>>> [lspector at fly clojush-launcher]$ lein run clojush.problems.demos.simple-regression
>>> Could not transfer artifact com.lowagie:itext:jar:2.1.7 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> Could not transfer artifact com.lowagie:itext:jar:2.1.7 from/to clojars (https://repo.clojars.org/ <https://repo.clojars.org/>): Received fatal alert: protocol_version
>>> Could not find artifact com.lowagie:itext:jar:2.1.7 in releases (https://clojars.org/repo <https://clojars.org/repo>)
>>> Could not transfer artifact org.apache.xmlgraphics:batik-svggen:jar:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> Could not transfer artifact org.apache.xmlgraphics:batik-svggen:jar:1.8 from/to clojars (https://repo.clojars.org/ <https://repo.clojars.org/>): Received fatal alert: protocol_version
>>> Could not find artifact org.apache.xmlgraphics:batik-svggen:jar:1.8 in releases (https://clojars.org/repo <https://clojars.org/repo>)
>>> Could not transfer artifact org.apache.xmlgraphics:batik-dom:jar:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> Could not transfer artifact org.apache.xmlgraphics:batik-dom:jar:1.8 from/to clojars (https://repo.clojars.org/ <https://repo.clojars.org/>): Received fatal alert: protocol_version
>>> Could not find artifact org.apache.xmlgraphics:batik-dom:jar:1.8 in releases (https://clojars.org/repo <https://clojars.org/repo>)
>>> Could not transfer artifact com.lowagie:itext:pom:2.1.7 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> Could not transfer artifact org.apache.xmlgraphics:batik-svggen:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> Could not transfer artifact org.apache.xmlgraphics:batik-dom:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>> This could be due to a typo in :dependencies, file system permissions, or network issues.
>>> If you are behind a proxy, try setting the 'http_proxy' environment variable.
>>> 
>>>  -Lee
>>> 
>>> 
>>> 
>>>> On Oct 3, 2018, at 8:50 AM, Wm. Josiah Erikson <wjerikson at hampshire.edu <mailto:wjerikson at hampshire.edu>> wrote:
>>>> 
>>>> /share/apps/bin/lein
>>>> 
>>>> I updated it to the latest version, then applied the fix, which is really just adding a -1 to the curl command on line 147
>>>>     -Josiah
>>>> 
>>>> 
>>>> 
>>>> On 10/3/18 8:33 AM, Lee Spector wrote:
>>>>> 
>>>>> Hmm, still getting same error, I think, below.
>>>>> 
>>>>> However, I now realize that I've been running my own lein script, in my ~/bin/
>>>>> 
>>>>> So presumably you updated something else, which may not even have had a problem!
>>>>> 
>>>>> However, disabling (renaming) my script, lein gives me a file not found error.
>>>>> 
>>>>> Where is the system-wide lein?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>>  -Lee
>>>>> 
>>>>> 
>>>>> 
>>>>> [lspector at compute-1-1 clojush-launcher]$ lein run clojush.problems.demos.simple-regression
>>>>> Could not transfer artifact com.lowagie:itext:pom:2.1.7 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>> Could not transfer artifact org.apache.xmlgraphics:batik-svggen:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>> Could not transfer artifact org.apache.xmlgraphics:batik-dom:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>> This could be due to a typo in :dependencies or network issues.
>>>>> If you are behind a proxy, try setting the 'http_proxy' environment variable.
>>>>> [lspector at compute-1-1 clojush-launcher]$ lein upgrade
>>>>> The script at /home/lspector/bin/lein will be upgraded to the latest stable version.
>>>>> Do you want to continue [Y/n]? Y
>>>>> 
>>>>> Upgrading...
>>>>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>>>                                  Dload  Upload   Total   Spent    Left  Speed
>>>>>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>>>> curl: (35) SSL connect error
>>>>> Failed to download https://github.com/technomancy/leiningen/raw/stable/bin/lein <https://github.com/technomancy/leiningen/raw/stable/bin/lein> (exit code )
>>>>> It's possible your HTTP client's certificate store does not have the
>>>>> correct certificate authority needed. This is often caused by an
>>>>> out-of-date version of libssl. It's also possible that you're behind a
>>>>> firewall and haven't set HTTP_PROXY and HTTPS_PROXY.
>>>>> 
>>>>> 
>>>>>> On Oct 3, 2018, at 8:10 AM, Wm. Josiah Erikson <wjerikson at hampshire.edu <mailto:wjerikson at hampshire.edu>> wrote:
>>>>>> 
>>>>>> I think it should work now. I added -1 to the curl command line in the lein script to force TLS.
>>>>>> 
>>>>>> Let me know if it doesn't.
>>>>>>     -Josiah
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 10/2/18 7:30 PM, Thomas Helmuth wrote:
>>>>>>> I don't think I've ever seen the first error before, but I've definitely seen the second, I think on fly. I forget the solution. FWIW, both errors mention HTTP_PROXY.
>>>>>>> 
>>>>>>> Tom
>>>>>>> 
>>>>>>> On Tue, Oct 2, 2018 at 5:58 PM Lee Spector <lspector at hampshire.edu <mailto:lspector at hampshire.edu>> wrote:
>>>>>>> 
>>>>>>> Mostly for Josiah, but sending to the list in case anyone else has insights.
>>>>>>> 
>>>>>>> Trying to run Clojure (via leiningen) on Fly for the first time in months, and getting errors.
>>>>>>> 
>>>>>>> This feels familiar, but I can't seem to find a past thread in which it was raised/addressed.
>>>>>>> 
>>>>>>> If I ssh to a node and cd into a Clojush directory and try to run a demo I get:
>>>>>>> 
>>>>>>> [lspector at compute-1-1 clojush-launcher]$ lein run clojush.problems.demos.simple-regression
>>>>>>> Could not transfer artifact com.lowagie:itext:pom:2.1.7 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>>>> Could not transfer artifact org.apache.xmlgraphics:batik-svggen:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>>>> Could not transfer artifact org.apache.xmlgraphics:batik-dom:pom:1.8 from/to central (https://repo1.maven.org/maven2/ <https://repo1.maven.org/maven2/>): Received fatal alert: protocol_version
>>>>>>> This could be due to a typo in :dependencies or network issues.
>>>>>>> If you are behind a proxy, try setting the 'http_proxy' environment variable.
>>>>>>> 
>>>>>>> Thinking that maybe I needed to upgrade leiningen I tried but got this:
>>>>>>> 
>>>>>>> [lspector at compute-1-1 clojush-launcher]$ lein upgrade
>>>>>>> The script at /home/lspector/bin/lein will be upgraded to the latest stable version.
>>>>>>> Do you want to continue [Y/n]? Y
>>>>>>> 
>>>>>>> Upgrading...
>>>>>>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>>>>>>                                  Dload  Upload   Total   Spent    Left  Speed
>>>>>>>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>>>>>>> curl: (35) SSL connect error
>>>>>>> Failed to download https://github.com/technomancy/leiningen/raw/stable/bin/lein <https://github.com/technomancy/leiningen/raw/stable/bin/lein> (exit code )
>>>>>>> It's possible your HTTP client's certificate store does not have the
>>>>>>> correct certificate authority needed. This is often caused by an
>>>>>>> out-of-date version of libssl. It's also possible that you're behind a
>>>>>>> firewall and haven't set HTTP_PROXY and HTTPS_PROXY.
>>>>>>> 
>>>>>>> Any thoughts or suggestions?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>>  -Lee
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Lee Spector, Professor of Computer Science
>>>>>>> Director, Institute for Computational Intelligence
>>>>>>> Hampshire College, Amherst, Massachusetts, 01002, USA
>>>>>>> lspector at hampshire.edu <mailto:lspector at hampshire.edu>, http://hampshire.edu/lspector/ <http://hampshire.edu/lspector/>, 413-559-5352
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Clusterusers mailing list
>>>>>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>>>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Clusterusers mailing list
>>>>>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>>>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>>>>> 
>>>>>> -- 
>>>>>> -----
>>>>>> Wm. Josiah Erikson
>>>>>> Associate Director of IT for Infrastructure
>>>>>> Network Engineer
>>>>>> System Administrator, School of CS
>>>>>> Hampshire College
>>>>>> Amherst, MA 01002
>>>>>> 
>>>>>> pronouns: he/him/his
>>>>>> _______________________________________________
>>>>>> Clusterusers mailing list
>>>>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>>>> 
>>>>> --
>>>>> Lee Spector, Professor of Computer Science
>>>>> Director, Institute for Computational Intelligence
>>>>> Hampshire College, Amherst, Massachusetts, 01002, USA
>>>>> lspector at hampshire.edu <mailto:lspector at hampshire.edu>, http://hampshire.edu/lspector/ <http://hampshire.edu/lspector/>, 413-559-5352
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Clusterusers mailing list
>>>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>>> 
>>>> -- 
>>>> Wm. Josiah Erikson
>>>> Associate Director of IT, Infrastructure Group
>>>> Network Engineer
>>>> System Administrator, School of CS
>>>> Hampshire College
>>>> Amherst, MA 01002
>>>> (413) 559-6091
>>>> pronouns: he/him/his
>>>> _______________________________________________
>>>> Clusterusers mailing list
>>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>>> 
>>> --
>>> Lee Spector, Professor of Computer Science
>>> Director, Institute for Computational Intelligence
>>> Hampshire College, Amherst, Massachusetts, 01002, USA
>>> lspector at hampshire.edu <mailto:lspector at hampshire.edu>, http://hampshire.edu/lspector/ <http://hampshire.edu/lspector/>, 413-559-5352
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Clusterusers mailing list
>>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
>> 
>> -- 
>> Wm. Josiah Erikson
>> Associate Director of IT, Infrastructure Group
>> Network Engineer
>> System Administrator, School of CS
>> Hampshire College
>> Amherst, MA 01002
>> (413) 559-6091
>> pronouns: he/him/his
>> _______________________________________________
>> Clusterusers mailing list
>> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
>> https://lists.hampshire.edu/mailman/listinfo/clusterusers <https://lists.hampshire.edu/mailman/listinfo/clusterusers>
> 
> --
> Lee Spector, Professor of Computer Science
> Director, Institute for Computational Intelligence
> Hampshire College, Amherst, Massachusetts, 01002, USA
> lspector at hampshire.edu <mailto:lspector at hampshire.edu>, http://hampshire.edu/lspector/ <http://hampshire.edu/lspector/>, 413-559-5352
> 
> _______________________________________________
> Clusterusers mailing list
> Clusterusers at lists.hampshire.edu <mailto:Clusterusers at lists.hampshire.edu>
> https://lists.hampshire.edu/mailman/listinfo/clusterusers

--
Lee Spector, Professor of Computer Science
Director, Institute for Computational Intelligence
Hampshire College, Amherst, Massachusetts, 01002, USA
lspector at hampshire.edu <mailto:lspector at hampshire.edu>, http://hampshire.edu/lspector/ <http://hampshire.edu/lspector/>, 413-559-5352

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.hampshire.edu/pipermail/clusterusers/attachments/20181208/1f7f4170/attachment-0001.html>

More information about the Clusterusers mailing list