This repository was archived by the owner on Sep 26, 2023. It is now read-only.
Add support for lzo files in input #104
Description
hadoop-lzo.jar is preinstalled on EMR, and in order to be able to split indexed LZO files when reading them (and have one Spark partition per block size instead of one Spark partition per file), we need to use sc.newAPIHadoopFile() to read them instead of sc.textFile() as it's currently the case.
More info can be found here.