feat: add base64 image encoding

[EscapedGibbon]

close: #476

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.48%. Comparing base (bd5cc06) to head (5bd614b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #481   +/-   ##
=======================================
  Coverage   98.48%   98.48%           
=======================================
  Files         243      244    +1     
  Lines       10016    10027   +11     
  Branches     2144     2145    +1     
=======================================
+ Hits         9864     9875   +11     
  Misses        152      152           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EscapedGibbon]

However, Buffer.from() cannot be used in browser and TextDecoder is unstable and doesn't work for Node18 and Node20.

[targos]

Can you compare with https://www.npmjs.com/package/@jsonjoy.com/base64 which claims to be compatible with node and browsers.

[EscapedGibbon]

Can you compare with https://www.npmjs.com/package/@jsonjoy.com/base64 which claims to be compatible with node and browsers.

https://www.npmjs.com/package/@jsonjoy.com/base64 is much faster than current implementation. I used vitest testing tools and it seems it works in the browser as well. I also compared it with a function that @lpatiny sent me, which uses uint8-base64 library and TextDecoder. Here's the code example for it:

function encodeWithUint8Encode(buffer:Uint8Array){
  const base64 = uint8Encode(buffer);
  const binString = new TextDecoder().decode(base64);
  return `data:image/png;base64,${binString}`;
}

[targos]

Then let's use it!

[stropitek]

also compared it with a function that @lpatiny sent me, which uses uint8-base64 library and TextDecoder

Is that the implementation you used in the benchmark as "TextDecoder"?

[EscapedGibbon]

also compared it with a function that @lpatiny sent me, which uses uint8-base64 library and TextDecoder

Is that the implementation you used in the benchmark as "TextDecoder"?

Yes

[stropitek]

Can you show the benchmark code you used?

[EscapedGibbon]

Can you show the benchmark code you used?

import { toBase64 } from "@jsonjoy.com/base64";
// eslint-disable-next-line import/no-extraneous-dependencies
import { encode as uint8Encode } from 'uint8-base64';
import {bench} from 'vitest';

import {readSync} from '../../load/read.js';
import {encode} from '../encode.js';


function testEncodeWithJoin(buffer:Uint8Array){
  const binaryArray = [];
  const format = 'png';
 for (const el of buffer) {
   binaryArray.push(String.fromCodePoint(el));
 }
 const binaryString = binaryArray.join('');
 const base64String = btoa(binaryString);
 const dataURL = `data:image/${format};base64,${base64String}`;
 return dataURL;
};

function encodeWithUint8Encode(buffer:Uint8Array){
  const base64 = uint8Encode(buffer);
  const binString = new TextDecoder().decode(base64);
  return `data:image/png;base64,${binString}`;
}

const buffer1 = encode(readSync("/Users/maxim/git/zakodium/qrcode-reader/dataset/testing/test.png"));
const buffer2 = encode(readSync("/Users/maxim/git/zakodium/qrcode-reader/dataset/testing/smallQR.png"));
const buffer3 = encode(readSync("/Users/maxim/git/zakodium/qrcode-reader/dataset/testing/doc2.png"));


if(typeof window === 'object'){
  console.log("Browser");
}else{
  console.log("Node");
}

console.assert(testEncodeWithJoin(buffer1) ===  `data:image/png;base64,${toBase64(buffer1)}` && encodeWithUint8Encode(buffer1) ===  testEncodeWithJoin(buffer1));

describe("benchmarking different functions buffer 1",()=>{
 bench("with join",()=>{
   testEncodeWithJoin(buffer1);
 })
  bench("with toBase64",() => {
  return `data:image/png;base64,${toBase64(buffer1)}`;
})
bench("with textDecoder",()=>{
  encodeWithUint8Encode(buffer1);
})

})

describe("benchmarking different functions buffer 2",()=>{
  bench("with join",()=>{
    testEncodeWithJoin(buffer2);
  })
   bench("with toBase64",()=>{
    return `data:image/png;base64,${toBase64(buffer2)}`;
 })
 bench("with textDecoder",()=>{
   encodeWithUint8Encode(buffer2);
 })
 })

 describe("benchmarking different functions buffer 3",()=>{
  bench("with join",()=>{
    testEncodeWithJoin(buffer3);
  })
   bench("with toBase64",() => {
   return `data:image/png;base64,${toBase64(buffer3)}`;
 })
 bench("with textDecoder",()=>{
   encodeWithUint8Encode(buffer3);
 })
 })

[stropitek]

I don't think the benchmark you published is valid because you ran it in nodejs and the library will use Buffer.toString() in most cases when it detects it (won't use that only with very small data).

You should do Buffer = undefined before running the benchmark.

[lpatiny]

You can see the Buffer here: https://github.com/jsonjoy-com/base64/blob/master/src/toBase64.ts#L11

Also take care to remove Buffer without importing the library (annoying because our rules reorder the code ...) like in:

Buffer = undefined;

const { toBase64 } = require('@jsonjoy.com/base64');

const bytes = new Uint8Array(64 * 1024 * 1024).map((_, i) =>
  Math.floor(Math.random() * 256),
);

console.time('base64');
const base64 = toBase64(bytes);
console.timeEnd('base64');

console.log(base64.slice(0, 100));

[targos]

annoying because our rules reorder the code ...

When you use import syntax, it doesn't matter where the code is. The imports are always executed first.

[lpatiny]

annoying because our rules reorder the code ...

When you use import syntax, it doesn't matter where the code is. The imports are always executed first.

Ok didn't know this. Thanks !

So the only way would be to 'await import' after removing Buffer ?

[targos]

Or import './someSetupFile.js' or node --import setup.js script.js or other alternatives

[EscapedGibbon]

Vitest has Buffer in jsdom environment because, apparently, there are dependencies that rely on it.
https://github.com/vitest-dev/vitest/blob/63e89702c1e4f61cf94227ba2795d513919c6b8f/packages/vitest/src/integrations/env/jsdom.ts#L73-L77

If I put the Buffer = undefined the tests will simply crash. Since we basically want to test this line here:
https://github.com/jsonjoy-com/base64/blob/6bd3d6f063477680692c9c96fa7e0ebe407d3e6d/src/toBase64.ts#L7

Can I modify the checking in node modules' function JUST for this benchmarking, so that the encodeSmall gets called? I know for a fact that the imported package works in the browser, I just don't know how to remove Buffer from testing. Should I look for a different approach?

[stropitek]

I don't have the context of why you use vitest for the benchmark.

Initially you used a library for benchmarking in Node.js which does not depend on jsdom (forgot the name). Can you use that? I think it could work.

[EscapedGibbon]

Seems to be working. So, in a browser it seems like the fastest one is Luc's version of encoding.

I added it the script in vite app and ran it in index.html, like discussed.
Here 's the code:

import  {toBase64}  from "@jsonjoy.com/base64";
import { encode as uint8encode } from "uint8-base64";
import assert from 'assert';
import {summary,bench,run} from 'mitata';
import { readSync,encode } from "image-js";

console.log(typeof Buffer);
function getDecodedData(id){
      // @ts-ignore
      const binaryString = atob(document.getElementById(id)?.src?.slice(22));
       const binaryData = new Uint8Array(binaryString.length);
       for(let element = 0; element < binaryString.length; element++){
        binaryData[element] = (binaryString.charCodeAt(element));
       }
       return binaryData;
     }

// Method 1: Using btoa + join
function encodeWithBtoaJoin(buffer){
 
  const binaryArray = new Array();
  for(let el of buffer){
    binaryArray.push( String.fromCharCode(el));
  }
   const binaryString = binaryArray.join('');
  const base64String = btoa(binaryString);
  const dataURL = `data:image/png;base64,${base64String}`;
  return dataURL;
}

// Method 2: Using uint8-base64 library
function encodeWithUint8Base64(buffer) {
  const base64 = uint8encode(buffer);
  const binString = new TextDecoder().decode(base64);
  return `data:image/png;base64,${binString}`;
}

// Method 3: Using @jsonjoy.com/base64 library
function encodeWithJsonJoyBase64(buffer) {
  return `data:image/png;base64,${toBase64(buffer)}`;
}

const data = getDecodedData('image2');

summary(() => {
  bench('btoa+join ', function* (ctx) {
    yield () => encodeWithBtoaJoin(data);
  });
 
  bench('uint8-base64', function* (ctx) {
    yield () => encodeWithUint8Base64(data);
  })
  
  bench('jsonjoy/base64 )', function* (ctx) {
    yield () => encodeWithJsonJoyBase64(data);
  })
 
});

await run();

[targos]

What is the size of your image?

[EscapedGibbon]

370x370 . Should i add a bigger image?

[lpatiny]

What is the size of your image?

It seems very small. I would use 10Mb for the tests. Based on my little benchmark (I added it on the README, 10Mb should take 10ms.

[EscapedGibbon]

Ok, will do

[EscapedGibbon]

here are the results for 10mb array.