Skip to content

Read data in batches from solr using cursorMark #364

@Ahmed-Salah6011

Description

@Ahmed-Salah6011

When attempting to read documents in batches using cursorMark and rows parameters with the above code sample
val solrDF = spark.read .format("solr") .option("zkHost", zookeeperHosts) .option("collection", collectionName) .option("query", configuredSolrQuery) .option("rows", batchSize) .option("cursorMark", cursorMark) .option("wt" , "json") .option("sort","MSG_REF_UK_ID asc") .load()

the returned solrDF doens't contain the nextCursorMark returned by solr in the response which doesn't allow using this to load the data from solr in batches

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions