-
Notifications
You must be signed in to change notification settings - Fork 68
Open
Description
I am trying to connect to Salesforce url(login.salesforce.com) using a proxy server in pyspark with com.springml.spark.salesforce driver.
I tried passing/adding the proxy server details in the config file, in the spark submit command and even while using the interactive mode but nothing worked. Wanted to check if the proxies are accepted by the com.springml.spark.salesforce driver. I am using this package: spark-salesforce_2.12-1.1.4.jar
Spark version: 3.0.0 Scala Version: 2.12.10 python: 3.6.8
While Execution:
Spark.sparkContext.setSystemProperty("http.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxy")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("http.useProxy", "true")
Config file:
.set("spark.driver.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort") \
.set("spark.executor.extraJavaOptions", "-Dhttp.proxyHost=proxy -Dhttp.proxyPort=proxyPort -Dhttps.proxyHost=proxy -Dhttps.proxyPort=proxyPort"")
code:
import sys
from pyspark.sql import SparkSession
from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import concat, col, lit
spark.sparkContext.setSystemProperty("http.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("http.proxyPort", "proxyPort")
spark.sparkContext.setSystemProperty("https.proxyHost", "proxyHost")
spark.sparkContext.setSystemProperty("https.proxyPort", "proxyPort")
df = spark \
.read \
.format("com.springml.spark.salesforce")\
.option("dbtable","dbtable")\
.option("username", "username") \
.option("password","password" )\
.option("login","https://login.salesforce.com") \
.option("version","44.0")\
.option("sfObject","sfObject")\
.option("soql", query) \
.load()
Error:
py4j.protocol.Py4JJavaError: An error occurred while calling o119.load.
: java.lang.Exception: com.sforce.ws.ConnectionException: Failed to send request to https://login.salesforce.com/services/Soap/u/35.0
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels