Ability to set options on split_csv_file?

[birdsarah]

Hi, was using split_csv_file and was hoping to specify the max_lines and buffering.

I ended up using split_file directly with the options for the split_csv_file as follows:

split_file(filename, out_dir, max_lines=5000, buffering=10000, line_reader=csv_reader, split_file_writer=split_file_output_csv)

was hoping to be able to do the following

split_csv_file(filename, out_dir, max_lines=5000, buffering=10000)

[johnwlockwood]

Hi Sarah,
It does do that.

#!/usr/bin/python
# _*_ coding: utf-8 _*_
import argparse
from karld.loadump import split_csv_file


def main(*args):
    parser = argparse.ArgumentParser(*args)
    parser.add_argument("path", help="csv file to split")
    parser.add_argument("--out-dir", default="split_data",
                        help="Data output directory")
    args = parser.parse_args()

    split_csv_file(args.path, args.out_dir, max_lines=5000, buffering=5000)


if __name__ == "__main__":
    main()

Usage

With that code in a file shard_file.py:

$ ./shard_file.py mybigfile.csv --out-dir mybigfile_shards