Repository navigation
Submitting a 32-bit application to Spark #681
Description
I've asked this question on stackoverflow (https://stackoverflow.com/questions/48323609/submitting-a-32-bit-application-to-spark-via-mobius):
I would like to submit to Spark a 32-bit driver program. The driver program is the .NET Framework console app:
the app is rather simple and features a very simple testing code:
public static void Main(string[] args)
{
var conf = new SparkConf();
var sparkContext = new SparkContext(conf);
var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });
var response = rdd.Map(s => s).Collect();
}
After I properly submit this app to sparkclr (that is, to mobius which then passes it to Spark itself), I get the following exception:
System.Exception: JVM method execution failed: Static method collectAndServe failed for class
org.apache.spark.api.python.PythonRDD when called with 1 parameters ([Index=1,
Type=JvmObjectReference, Value=11], )
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object
classNameOrJvmObjectReference, String methodName, Object[] parameters)
at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String
methodName, Object[] parameters)
at Microsoft.Spark.CSharp.Proxy.Ipc.RDDIpcProxy.CollectAndServe()
at Microsoft.Spark.CSharp.Core.RDD`1.Collect()
The exception goes away when I build the app in x64. Also, the exception is not there when the app is built in x86 but does nothing (i.e. has no
var rdd = sparkContext.Parallelize(new List<string>() { "a", "b", "c", "d" });
var response = rdd.Map(s => s).Collect();
lines).
Is there any workaround to submit 32bit app to Spark?
(In the original question I thought the issue could be with Java, but later found this was not the case).
With kind regards,
Lesia
Activity