How to initialize sparkContext class in C# #674

Activity

1. SatishkumarJha

   changed the title [-]How to configure sparkContext in C#[/-] [+]How to initialize sparkContext class in C#[/+] on Oct 30, 2017
2. jayprajapati857 commented on Oct 30, 2017

   

   Jaykumar Prajapati (jayprajapati857)

   on Oct 30, 2017

   More actions

   How you are trying to execute? From command line or from visual studio?
3. SatishkumarJha commented on Oct 31, 2017

   

   SatishkumarJha

   on Oct 31, 2017

   Last edited by SatishkumarJha

   Author

   More actions

   from visual studio,I have installed Spark & tested sample code on command prompt which is working fine.
   I have to use parallelism in C# with Spark.That's why I have installed Spark from nuget packages.
   Sample code:
   `using Microsoft.Spark.CSharp.Core;
   using System;
   using System.Collections.Generic;
   using System.Linq;
   using System.Text;
   using System.Threading.Tasks;

   namespace ConsoleApp5
   {
   class Program
   {
   public static void Main(string[] args)
   {
   string home = "http://127.0.0.1:4040";// "spark://hostname:port");
   var sparkContext = new SparkContext(new SparkConf().SetAppName("test").SetMaster("Master").SetSparkHome(home));//error throwing here

           try
           {
               const int slices = 3;
               var numberOfItems = (int)Math.Min(100000L * slices, int.MaxValue);
               var values = new List<int>(numberOfItems);
               for (var i = 0; i <= numberOfItems; i++)
               {
                   values.Add(i);
               }
   
               var rdd = sparkContext.Parallelize(values, slices);
   
               CalculatePiUsingAnonymousMethod(numberOfItems, rdd);
   
           }
           catch (Exception ex)
           {
           }
   
           sparkContext.Stop();
   
       }
       private static void CalculatePiUsingAnonymousMethod(int n, RDD<int> rdd)
       {
           var count = rdd
                           .Map(i =>
                           {
                               var random = new Random();
                               var x = random.NextDouble() * 2 - 1;
                               var y = random.NextDouble() * 2 - 1;
   
                               return (x * x + y * y) < 1 ? 1 : 0;
                           })
                           .Reduce((x, y) => x + y);
   
       }
   }
   

   }
   `
   Error:
   System.Net.Sockets.SocketException occurred
   HResult=0x80004005
   Message=No connection could be made because the target machine actively refused it
   Source=System
   StackTrace:
   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
   at Microsoft.Spark.CSharp.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.GetConnection()
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters)
   at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallConstructor(String className, Object[] parameters)
   at Microsoft.Spark.CSharp.Proxy.Ipc.SparkCLRIpcProxy.CreateSparkConf(Boolean loadDefaults)
   at Microsoft.Spark.CSharp.Core.SparkConf..ctor(Boolean loadDefaults)
   at ConsoleApp5.Program.Main(String[] args) in C:\Users\satish.jha\documents\visual studio 2017\Projects\ConsoleApp5\ConsoleApp5\Program.cs:line 15
4. jayprajapati857 commented on Nov 2, 2017

   

   Jaykumar Prajapati (jayprajapati857)

   on Nov 2, 2017

   More actions

   If you are trying to execute it from visual studio then you need to try in debug mode to initialise the spark context have you read this link : https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#running-mobius-examples-in-local-mode

   Just read and set the environment accordingly you will able to execute
5. jayprajapati857 commented on Nov 2, 2017

   

   Jaykumar Prajapati (jayprajapati857)

   on Nov 2, 2017

   More actions

   Read the documentation for debug mode from that link
6. SatishkumarJha commented on Nov 4, 2017

   

   SatishkumarJha

   on Nov 4, 2017

   Author

   More actions

   Hi jayprajapati,

   I tried to run this command 'sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug' on scala/spark command but getting error plz. suggest where I'm doing mistake.
   I have pasted my spark console below..plz. have a look .

   Microsoft Windows [Version 10.0.10240]
   (c) 2015 Microsoft Corporation. All rights reserved.

   C:\Users\satish.jha>cd c:\spark

   c:\Spark>spark-shell
   Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
   Setting default log level to "WARN".
   To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
   17/11/04 12:13:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-rdbms-3.2.9.jar."
   17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-core-3.2.10.jar."
   17/11/04 12:14:03 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-api-jdo-3.2.6.jar."
   17/11/04 12:14:12 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
   Spark context Web UI available at http://192.168.22.103:4040
   Spark context available as 'sc' (master = local[*], app id = local-1509777835771).
   Spark session available as 'spark'.
   Welcome to
   ____ __
   / / ___ / /
   \ / _ / _ `/ __/ '/
   // .__/_,// //_\ version 2.2.0
   //

   Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
   Type in expressions to have them evaluated.
   Type :help for more information.

   scala

   sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug

   :1: error: ';' expected but '.' found.
   sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug

                                        ^
   

   Regards,
   Satish