Clash Royale CLAN TAG#URR8PPP

Spark-Shell cannot use Netlib-Java

I am trying to link against local BLAS libraries in a simple example to test out the performance difference to a "regular" implementation.

I dutifully followed the instructions posted on the official netlib-java GitHub,
but I still get an error, similar to this thread.

I am running precompiled Spark 2.3 with Hadoop 2.7; I tried building Spark from source with the option to enable netlib-java explicitly (as described on the Spark MLlib page, but to no avail.

The specific error messages that are confusing me are looking like this:

spark-shell --packages com.github.fommil.netlib:all:1.1.2
Ivy Default Cache set to: /home/user/.ivy2/cache
The jars for the packages stored in: /home/user/.ivy2/jars
:: loading settings :: url = jar:file:/home/user/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.github.fommil.netlib#all added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.github.fommil.netlib#all;1.1.2 in central
found net.sourceforge.f2j#arpack_combined_all;0.1 in central
found com.github.fommil.netlib#core;1.1.2 in central
found com.github.fommil.netlib#netlib-native_ref-osx-x86_64;1.1 in central
found com.github.fommil.netlib#native_ref-java;1.1 in central
found com.github.fommil#jniloader;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-win-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-win-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_ref-linux-armhf;1.1 in central
found com.github.fommil.netlib#netlib-native_system-osx-x86_64;1.1 in central
found com.github.fommil.netlib#native_system-java;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-i686;1.1 in central
found com.github.fommil.netlib#netlib-native_system-linux-armhf;1.1 in central
found com.github.fommil.netlib#netlib-native_system-win-x86_64;1.1 in central
found com.github.fommil.netlib#netlib-native_system-win-i686;1.1 in central
:: resolution report :: resolve 1196ms :: artifacts dl 16ms
:: modules in use:
com.github.fommil#jniloader;1.1 from central in [default]
com.github.fommil.netlib#all;1.1.2 from central in [default]
com.github.fommil.netlib#core;1.1.2 from central in [default]
com.github.fommil.netlib#native_ref-java;1.1 from central in [default]
com.github.fommil.netlib#native_system-java;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-armhf;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-linux-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-osx-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-win-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_ref-win-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-armhf;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-linux-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-osx-x86_64;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-win-i686;1.1 from central in [default]
com.github.fommil.netlib#netlib-native_system-win-x86_64;1.1 from central in [default]
net.sourceforge.f2j#arpack_combined_all;0.1 from central in [default]
:: evicted modules:
com.github.fommil.netlib#core;1.1 by [com.github.fommil.netlib#core;1.1.2] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 19 | 0 | 0 | 1 || 17 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 17 already retrieved (0kB/15ms)
2018-06-13 07:53:28 WARN Utils:66 - Your hostname, wdfl30003105a resolves to a loopback address: 127.0.0.1; using 10.18.91.86 instead (on interface enp1s0)
2018-06-13 07:53:28 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2018-06-13 07:53:30 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Warning: Local jar /home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-win-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-win-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-armhf-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-osx-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-i686-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-linux-armhf-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-win-x86_64-1.1.jar does not exist, skipping.
Warning: Local jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_system-win-i686-1.1.jar does not exist, skipping.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/net.sourceforge.f2j_arpack_combined_all-0.1.jar not found
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1807)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1837)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1.jar not found
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1807)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1837)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:457)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:457)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply(SparkILoop.scala:79)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:79)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:78)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:77)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:110)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:76)
at org.apache.spark.repl.Main$.main(Main.scala:56)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-06-13 07:53:44 ERROR SparkContext:91 - Failed to add file:/home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar to Spark environment
java.io.FileNotFoundException: Jar /home/user/.ivy2/jars/com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1.jar not found


It basically says that the libraries could not be found; but interestingly, my ~/.ivy2/jars folder shows that all of the required packages are present, except under a slightly different name:

~/.ivy2/jars

~/.ivy2/jars$ ls
com.github.fommil_jniloader-1.1.jar
com.github.fommil.netlib_core-1.1.2.jar
com.github.fommil.netlib_native_ref-java-1.1.jar
com.github.fommil.netlib_native_system-java-1.1.jar
com.github.fommil.netlib_netlib-native_ref-linux-armhf-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-linux-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-linux-x86_64-1.1.jar
com.github.fommil.netlib_netlib-native_ref-linux-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-osx-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-win-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_ref-win-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-armhf-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-linux-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-osx-x86_64-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-win-i686-1.1-natives.jar
com.github.fommil.netlib_netlib-native_system-win-x86_64-1.1-natives.jar
net.sourceforge.f2j_arpack_combined_all-0.1-javadoc.jar


I don't know enough about packaging and/or the process of how Spark/Scala is handling these packages internally. For fixed packages, there seems to be a workaround in maven to include PomOnly(), but sadly the spark-shell --packages option does not allow me to do something similar.

PomOnly()

--packages

2018-06-13 08:32:40 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS

~/.ivy2

~/.ivy2

~/.m2

1 Answer
1

It is possible to build spark with netlib served as spark native libs.

spark

netlib

This solved the problem for me, look at here: https://github.com/Mega-DatA-Lab/SpectralLDA-Spark/wiki/Compile-Spark-with-Native-BLAS-LAPACK-Support

If some package you have declared in your dependecy needs netlibs try to use spark-submit with --exclude option

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.