Skip to content

Commit 33d43bf

Browse files
gaborgsomogyiHyukjinKwon
authored andcommitted
[SPARK-22484][DOC] Document PySpark DataFrame csv writer behavior whe…
## What changes were proposed in this pull request? In PySpark API Document, DataFrame.write.csv() says that setting the quote parameter to an empty string should turn off quoting. Instead, it uses the [null character](https://en.wikipedia.org/wiki/Null_character) as the quote. This PR fixes the doc. ## How was this patch tested? Manual. ``` cd python/docs make html open _build/html/pyspark.sql.html ``` Author: gaborgsomogyi <[email protected]> Closes #19814 from gaborgsomogyi/SPARK-22484.
1 parent 087879a commit 33d43bf

File tree

2 files changed

+3
-3
lines changed

2 files changed

+3
-3
lines changed

python/pyspark/sql/readwriter.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -828,8 +828,7 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No
828828
set, it uses the default value, ``,``.
829829
:param quote: sets the single character used for escaping quoted values where the
830830
separator can be part of the value. If None is set, it uses the default
831-
value, ``"``. If you would like to turn off quotations, you need to set an
832-
empty string.
831+
value, ``"``. If an empty string is set, it uses ``u0000`` (null character).
833832
:param escape: sets the single character used for escaping quotes inside an already
834833
quoted value. If None is set, it uses the default value, ``\``
835834
:param escapeQuotes: a flag indicating whether values containing quotes should always

sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -592,7 +592,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
592592
* <li>`sep` (default `,`): sets the single character as a separator for each
593593
* field and value.</li>
594594
* <li>`quote` (default `"`): sets the single character used for escaping quoted values where
595-
* the separator can be part of the value.</li>
595+
* the separator can be part of the value. If an empty string is set, it uses `u0000`
596+
* (null character).</li>
596597
* <li>`escape` (default `\`): sets the single character used for escaping quotes inside
597598
* an already quoted value.</li>
598599
* <li>`escapeQuotes` (default `true`): a flag indicating whether values containing

0 commit comments

Comments
 (0)