[QUERY] How to use Linked Services with Cosmos DB Spark connector Throughput control in Synapse Analytics

[golfalot]

Query/Question

What is the equivalent to spark.synapse.linkedService for use with spark.cosmos.throughputControl.globalControl.database

I need to use private endpoints because of DEP, and having credentials stored with Linked Services is very handy.

Why is this not a Bug or a feature Request?

Feels like I'm missing something really obvious, but the docs don't seem to cover what I'm seeking,

Setup (please complete the following information if applicable):

• OS: Spark 3.4 Python 3.10 Scala
• IDE: Synapse Analytics Web based
• Library/Libraries: default Spark Pool config

PySpark

(
    df_floodre
    .withColumn('id', F.col('uprn').cast('string'))
    .write
    .format("cosmos.oltp")
    .mode("Append")
    .option("spark.synapse.linkedService", "CosmosDB_GraphQL_databasets")
    .option("spark.cosmos.container", 'floodRe')
    .option('spark.cosmos.throughputControl.enabled','true')
    .option('spark.cosmos.throughputControl.name','floodRe_Loader_Jobs_Group') 
    .option('spark.cosmos.throughputControl.globalControl.database','CosmosDB_GraphQL_databasets_ThroughputControl') 
    .option('spark.cosmos.throughputControl.globalControl.container','ThroughputControl-floodre')
    .option('spark.cosmos.throughputControl.targetThroughput',7000)
    .save()
)

I've tried a few likely nonsensical permutations and got 403 or 404 errors. Guidance much appreciated.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

[kushagraThapar]

@tvaron3 please take a look at this, thanks!

[kushagraThapar]

@tvaron3 / @TheovanKraay can you please take a look at this issue, thanks!

[TheovanKraay]

@golfalot please review throughput control documentation here. If it is still unclear, please describe more clearly what you are trying to do, and the errors you are getting (with full stack trace please). There should be nothing in principle stopping you from using throughput control in Synapse Spark pools, but there are some nuances. E.g. you need to create the throughput control database (if separate from the database you are working with) and throughput control container for the metadata, etc.

[github-actions]

Hi @golfalot. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[golfalot]

Yes, happy to comment back. L
We have throughput control 100% working.

The outstanding issue I raise here is that it does not support Synapse linked services for securing the credentials