[File] recognize encoding for remote resource

[AcckiyGerman]

Original: datopian/datahub.io#105

File = require('data.js').File
// loading ISO8859 resource:
> file = File.load('https://raw.githubusercontent.com/frictionlessdata/test-data/master/files/csv/encodings/iso8859.csv')
> file.encoding
'utf-8'

Acceptance criteria

• File.load('https://raw.githubusercontent.com/frictionlessdata/test-data/master/files/csv/encodings/iso8859.csv').encoding == 'ISO-8859-1'
• File.load('https://raw.githubusercontent.com/frictionlessdata/test-data/master/files/csv/encodings/western-macos-roman.csv').encoding == <macOS-roman-or-so>

Tasks

• add test
• realize encoding recognize

Analysis

We need to change this method:

class FileRemote extends File {
   ...
   get encoding() {
       return DEFAULT_ENCODING
  }

analysis update

encoding() method should:

• connect to remote resource
• get small portion of raw-data
• try to recognize encoding

I tried to implement this schema, using chardet.detectFileSync() lib but it works only with files - any argument is treated as a file-name.

Possible solutions:

• save a part of remote resource in a local temp file, then use chardet.detectFileSync(temp)
• use some other lib to recognize encoding using remote Stream

[AcckiyGerman]

gist

const guessRemoteEncoding = async ()=>{
      let stream
      try {
        stream = await this.stream()
      } catch (err){
        console.warn('Warning! Cannot reach remote file to guess encoding:\n', this.path)
        return
      }

      // Download one piece of remote file on the disk and use 'chardet' lib to guess encoding:
      stream.on('data', chunk =>{
        stream.pause()  // one chunk is enough
        const tmpFileName = 'tmp' + Math.random()
        fs.writeFile(tmpFileName, chunk, ()=>{
          let encoding = chardet.detectFileSync(tmpFileName)
          fs.unlinkSync(tmpFileName)
          this._descriptor.encoding = encoding   // set FileRemote object encoding
          this._encoding = encoding  // set this._encoding for future responses
        })
      })
    }

    if (!this._encoding) {
      guessRemoteEncoding()
    }

[zelima]

@AcckiyGerman What is the status for this one?

[AcckiyGerman]

I had no time to work with this, but I still want to finish it.

[AcckiyGerman]

Uploaded a bigger western-macos-roman encoded file:
https://github.com/frictionlessdata/test-data/blob/master/files/csv/encodings/big-western-macos-roman.csv

Size ~24kb.

Try to recognize with different sniffer libs:

• chardet (originally used in data.js): FAIL (windows-1252)
• html-encoding-sniffer: FAIL (windows-1252)
• Python version of chardet: FAIL (no result)

Conclusion:

MacOS western roman encoding - not recognized

No libs that I found are recognising the MacOS encoding properly. I know that this recognise is possible, coz my LibreOffice open this file, recognizing it as macos-roman.

iso8859 encoding - OK

• chardet works OK even with small samples.

Issue is frozen, as we have other priorities now