Read empty csv entry as nil

[folknor]

Hi

Thanks for the library, I use it extensively and have done for many years now :-D

Would it be possible to get a configure option for how ftcsv should treat empty entries in the parsed CSV when outputting the lua table?

For example, a CSV that looks like this:

"Testing",,"Testing 2"

Ran through this code:

local csvraw = "C1,C2,C3\n\"Testing\",,\"Testing 2\""
local s = require("serpent")
local ftcsv = require("ftcsv")
local luatbl = ftcsv.parse(csvraw, ",", { loadFromString = true, })
print(s.block(luatbl))

Will output this:

{
  {
    C1 = "Testing",
    C2 = "",
    C3 = "Testing 2"
  } --[[table: 0x58e728040940]]
} --[[table: 0x58e728024e40]]

However what I'd like is for C2 to be nil in this case.

Thoughts?

Thanks again!

[folknor]

It might be an idea to add yet another configure option to treat space characters as empty, so for example Testing, ,Testing would also be treated as C1=Testing, C2=nil, C3=Testing.

Alternatively allow the caller to provide a callback function that is ran per line or per value that can modify it before the final table is returned?

So

local function validateValue(val)
    local s = val:match("^%s*(.-)%s*$")
    if type(tonumber(s)) == "number" and tostring(tonumber(s)):len() == s:len() then return tonumber(s) end
    if s:len() == 0 then return end
    return s
end

local luatbl = ftcsv.parse(csvraw, ",", { loadFromString = true, validate = validateValue})

[FourierTransformer]

Thanks for enjoying the library! I am leaning towards creating a new option, maybe something like encodeEmptyAs and let people specify what they want to encode empty values as. Your second option could just be done with the parseLine iterator mode.

[scossu]

Your second option could just be done with the parseLine iterator mode.

I am doing exactly that to convert all empty cells to nil. It's a trivial condition + assignment: https://git.knowledgetx.com/scossu/pocket_archive/src/branch/master/src/submission.lua#L389-L393

[FourierTransformer]

I've been playing around with this in https://github.com/FourierTransformer/ftcsv/tree/feature/encodeEmptyAs. It currently uses the option name encodeEmptyAs, but I will likely change the option name since it specifically deals with non-existent values as opposed to a literal empty string in the csv:

a,b,c,d
1,,"",0

so the value of column b could be specified, but not c. I'm thinking maybe encodeNilAs perhaps. Also, funnily enough the way it's written in that branch doesn't actually let you set encodeEmptyAs=nil. I'll sort through that in the coming days.