Skip to content

Can we connect to proxy server using com.springml.spark.salesforce driver #80

@Brooody

Description

@Brooody

I am trying to connect to Salesforce url(login.salesforce.com) using a proxy server in pyspark with com.springml.spark.salesforce driver.

I tried passing/adding the proxy server details in the config file, in the spark submit command and even while using the interactive mode but nothing worked. Wanted to check if the proxies are accepted by the com.springml.spark.salesforce driver. I am using this package: spark-salesforce_2.12-1.1.4.jar

Spark version: 3.0.0 Scala Version: 2.12.10 python: 3.6.8

While Execution:

Spark.sparkContext.setSystemProperty("http.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("http.useProxy", "true")

Config file:

.set("spark.driver.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort") \ 
.set("spark.executor.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort"")

code:

import sys
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import concat, col, lit

spark.sparkContext.setSystemProperty("http.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")

df = spark \
     .read \
     .format("com.springml.spark.salesforce")\
     .option("dbtable","dbtable")\
     .option("username", "username") \
     .option("password","password" )\
     .option("login","https://login.salesforce.com") \
     .option("version","44.0")\
     .option("sfObject","sfObject")\
     .option("soql", query) \
     .load()

Error:

py4j.protocol.Py4JJavaError: An error occurred while calling o119.load.
: java.lang.Exception: com.sforce.ws.ConnectionException: Failed to send request to https://login.salesforce.com/services/Soap/u/35.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions