BatchedText: billboard material

[upisfree]

Hello again :)

I'm trying to combine BatchedText and your createBillboardMaterial, but I'm not good at shaders so I don't know why it doesn't work. Maybe I need to do some magic matrix stuff, but I can't figure it out which matrix to use :)

Could you please suggest what I'm doing wrong? Thanks again :D

createBillboardMaterial source (it works with usual Text class just fine):
#101 (comment)
https://codesandbox.io/s/createbillboardmaterial-xl6mt?file=/src/createBillboardMaterial.js:0-580

BatchedText + createBillboardMaterial — all texts are fixed in camera view

BatchedText + material without modifications — all texts are in right places, but they don't look at the camera

P. S. It might make sense to add the createBillboardMaterial implementation to the repository, since Text looking always at the camera is a very common request among developers?

[lojjic]

Hmm, that's a tricky one. I'll try to see if I can make the batched matrix transforms work with other custom shader transforms. But you may be right about this being a common option that would belong in the base shader.

[Methuselah96]

I am looking to use BatchedText with billboarding as well, text rendering is the most expensive part of our current three.js scene and would love to use BatchedText to speed things up.

[upisfree]

For anyone needed this, I wrote typescript version of this shader:

import { Camera, Vector3 } from 'three'
import { BatchedText } from 'troika-three-text'

const _position = new Vector3()
const _scale = new Vector3()
const _cameraPosition = new Vector3()

// usage: add Text with `billboardBatchedText.addText(text)`,
//        and then add it to three's parent with `group.add(text)`
class BillboardBatchedText extends BatchedText {
  constructor(public scaleFactor: number = 1) {
    super()
  }

  update(camera: Camera) {
    const texts = this._members.keys()

    camera.getWorldPosition(_cameraPosition)

    texts.forEach((text) => {
      const parent = text.parent

      parent.updateMatrix()
      parent.updateWorldMatrix(false, false)
      parent.getWorldPosition(_position)

      text.position.copy(_position)

      camera.getWorldQuaternion(text.quaternion)

      // size attenuation
      const scale =
        _scale.subVectors(text.position, _cameraPosition).length() * this.scaleFactor
      text.scale.set(scale, scale, scale)
    })
  }
}

export default BillboardBatchedText

[crystalfp]

Thanks @upisfree for your code! Unfortunately in the loop on the texts inside the BatchText, the const parent = text.parent is always null so nothing happens.

Sorry for my ignorance, but I cannot decipher the entire content of the loop.

Anyway, leaving only the camera.getWorldQuaternion(text.quaternion); line inside the loop gives a perfectly acceptable billboard text.

Thanks!
mario

[upisfree]

@crystalfp

the const parent = text.parent is always null so nothing happens.

I assumed that it would take one instance of BillboardBatchedText and one instance of three.js' Object3D to use for proper positioning in the world.
Here's how you can use it:

const group = new Group();
// position, scale or rotate the group somehow. it will be used to position our text as a parent
const batchedText = new BillboardBatchedText();

const text = new Text();
batchedText.addText(text);
group.add(text);

[crystalfp]

Thanks @upisfree !
I was not aware that you have to add the Text to a group (and then add the group to the scene, I suppose). What I'm doing is:

const batchedText = new BillboardBatchedText();

const text = new Text();
batchedText.addText(text);
scene.add(batchedText);

What I cannot evaluate is if this setup will speedup label rendering.

Hopefully the documentation for batchedText will be ready someday...
Thanks for clarifying!
mario

[lojjic]

The Text instances should not be added to the scene outside the BatchedText container -- doing so would result in them being rendered twice.

[crystalfp]

@lojjic This means my way to use BatchedText (Adding it to the scene) is the correct one. Correct?

[lojjic]

@crystalfp Yes, it makes sense to me. The billboarding should be achievable by just setting the Text's quaternion, it doesn't need to be added to any external group to do so.

[upisfree]

The reason I did put Text into other group is because I needed BatchedText AND use a different coordinate system (Z+ or something like that). I didn‘t want to calculate positions and quaternions by myself, I wanted to change coordinate system of a group and just put children in it. I don‘t remember why I didn’t just put BatchedText into the group, I guess something didn‘t work there.

[Bangybug]

This achieves billboard effect for BatchedText, it is invariant to the coordinate system you set your texts in.

However if you look in the Y-down direction, it's going to warp.

@lojjic maybe you will have some ideas

/**
 * Creates billboard material for Troika's BatchedText. It is invariant to the coordinate system texts are in, either z-up or y-up.
 * - Troika shaders do not use USE_BATCHING nor USE_INSTANCING defines, don't spell these to avoid confusion as the code branches are likely never to be tested by you.
 * - Use ShaderDebugger tool to get into compiled shader for study.
 * - Don't rely on suffixed variables during customization, e.g. troika_position_5 - these are obtained from the global index that depends on shader compilation order at runtime.
 * - A text instance is set by a number of vertices that get sampled from data-texture, see how 'transformed' is obtained in the final shader code.
 * - createDerivedMaterial is useful only if chained=true.  
 * - The vertexMainOutro can be used but there is no access to the sampled positions from the texture and there is no access to the 'transformed' variable there.
 * - The vertexTransform block is not usable if this material was derived from the derived BatchedText material,
 * because transforms are executed from last to first in the derivation chain. 
 * - If you instead derive BatchedText material from the one that you derive yourself first, your 'vertexTransform' block will get 'position' sampled from a texture matrix by BatchedText block.
 * - Don't be confused as the 'position' in 'vertexTransform' is not a direct item from the 'position' attribute. It's actually computed by previous transforms in the chain.
 * - Sampled 'transformed' points (the ones that you refer as 'position' in your custom 'vertexTransform') likely to lie in a local space of troika defined by Text.orientation, by default its +x+y.
 */
export function createBillboardMaterial<BaseMaterial extends Material>(
  baseMaterial: BaseMaterial
): BaseMaterial {
  return createDerivedMaterial(baseMaterial, {
    chained: true,

      vertexDefs: `
// Helper functions for billboarding
mat3 getBillboardMatrix(vec3 position, vec3 cameraPos, vec3 upAxis) {
  // View vector from object to camera
  vec3 look = normalize(cameraPos - position);
  
  // Set desired up axis 
  vec3 up = upAxis;
  
  // Check for collinearity between look and up
  if (abs(dot(look, up)) > 0.999) {
    // Fallback to something else if almost parallel to the up
    up = vec3(1.0, 0.0, 0.0);
  }
  
  // Compute orthonormal basis
  vec3 right = normalize(cross(up, look));
  up = normalize(cross(look, right));
  
  return mat3(right, up, look);
}
    `,
    
    vertexTransform: `
// 'position' here is already a point sampled from BatchedText's data-texture
// This point is in the text's local space (+x+y orientation)

// Transform local position to world space
vec4 worldPosition = modelMatrix * vec4(position, 1.0);

// Get billboard matrix for this specific text instance
mat3 billboardMat = getBillboardMatrix(
  worldPosition.xyz, 
  cameraPosition, 
  vec3(0.0, 1.0, 0.0)
);

// Apply billboard transformation:
// 1. Return to text's local space
// 2. Apply billboard rotation
// 3. Return to world space

// Calculate local offset from the text quad's center
// Assuming position is already in the text's local space

// Get vertex local coordinates relative to the instance center
// For this we need to know the instance center position
// BatchedText stores this in the data-texture, but we don't have direct access

// Alternative approach: use the fact that all vertices of the text quad
// are already transformed relative to the instance center
// Apply billboard rotation only to local coordinates

vec3 localPos = position;
vec3 billboardLocalPos = billboardMat * localPos;

position.xyz = billboardLocalPos.xyz;
`
  })
}

[Bangybug]

This one is free of warping, but I had to do customRewriter, vertexDefs and vertexTransform cannot be used for this task at the same time.

/**
 * Creates billboard material for BatchedText of troika. It is invariant to the coordinate system texts are in, either z-up or y-up.
 * - Troika shaders do not use USE_BATCHING nor USE_INSTANCING defines, don't spell these to avoid confusion as the code branches are likely never to be tested by you.
 * - Use ShaderDebugger tool to get into compiled shader for study.
 * - Don't rely on suffixed variables during customization, e.g. troika_position_5 - these are obtained from the global index that depends on shader compilation order at runtime.
 * - A text instance is set by a number of vertices that get sampled from data-texture, see how 'transformed' is obtained in the final shader code.
 * - createDerivedMaterial is useful only if chained=true.  
 * - The vertexMainOutro can be used but there is no access to the sampled positions from the texture and there is no access to the 'transformed' variable there.
 * - The vertexTransform block is not usable if this material was derived from the derived BatchedText material,
 * because transforms are executed from last to first in the derivation chain. 
 * - If you instead derive BatchedText material from the one that you derive yourself first, your 'vertexTransform' block will get 'position' sampled from a texture matrix by BatchedText block.
 * - The opposite logic is applied to usability of customRewriter in relation to derivation order, i.e. they both cannot be used at the same time if your goal is to inherit from troika shader.
 * - Don't be confused as the 'position' in 'vertexTransform' is not a direct item from the 'position' attribute. It's actually computed by previous transforms in the chain.
 * - Sampled 'transformed' points (the ones that you refer as 'position' in your custom 'vertexTransform') likely to lie in a space of troika defined by Text.orientation, by default +x+y,  
 * the coordinates also relative to the text world position thus it's not quite local space.
 * - A batchlet is essentially a quad.
 * - Billboard effect is achieved in eye-space by getting pivotal coordinates of every text transformed by modelViewMatrix, adjusted by local offsets in +x+y space per vertex.
 * - It's best done by rewriting shader of the BatchedText material.
 */
export function createBillboardMaterial<BaseMaterial extends Material>(
  baseMaterial: BaseMaterial
): BaseMaterial {
  return createDerivedMaterial(baseMaterial, {
    chained: true,

    /*
    About data structure in the troika. Mere observations.
    
    A batchlet is given by 3 items across various attibutes.
    
    Geometry attributes per batchlet:
    - aTroikaGlyphBounds: 3 per batchlet, 4 components per item
    - aTroikaGlyphIndex: 3 per batchlet, 1 component 
    - aTroikaTextBatchMemberIndex: 3 per batchlet, 1 component
    
    General geometry attributes:
    - normal: always 4 items
    - position: always 4 items to define a unit length rectangle in a specified orientation plane
    - uv: always 4 items
    - index: always 6 items to draw two triangles for a rectangle
    */
    customRewriter: (p) => {
      // 1) place definition of vec3 text_center at the beginning
      const defIndex = p.vertexShader.indexOf('vec3 troika_position')
      p.vertexShader = [p.vertexShader.substring(0, defIndex), 'vec3 text_center;', p.vertexShader.substring(defIndex)].join('\n')

      // 2) modify troika method that returns text points in world-space
      let [before, after] = p.vertexShader.split('troikaBatchTexel(3.0)')
      before = before.trimEnd()
      after = after.substring(after.indexOf(');') + 2).trimStart()

      // 2.1) we need text_center per each quad
      const append = `text_center = (matrix * vec4(0.0, 0.0, 0.0, 1.0)).xyz;`

      // 2.2) instead of world-space we return local space coordinates per each quad vertex
      const pos = after.substring(0, after.indexOf('.')).trim()
      after = after.substring(after.indexOf('\n') + 1)
      // original code pattern: troika_position_6.xyz = (matrix * vec4(troika_position_6, 1.0)).xyz;
      const line = `${pos}.xyz = (matrix * vec4(${pos}, 1.0)).xyz - text_center;`

      p.vertexShader = [before, 'troikaBatchTexel(3.0));', append, line, after].join('\n');

      // 2) modify threejs block to achieve camera-space billboard effect
      p.vertexShader = p.vertexShader.replace('mvPosition = modelViewMatrix * mvPosition;', `
mvPosition = modelViewMatrix * vec4(text_center.xyz, 1.0);
mvPosition.xyz += transformed;
`)

      return p
    },
  })

The derivation order is important, I overload createDerivedMaterial of BatchedText like this:

 createDerivedMaterial<BaseMaterial extends Material>(material: BaseMaterial) {

    // the order of derivation is important, see createBillboardZupMaterial

    // case 1: derive so that vertexTransform is useful
    // return super.createDerivedMaterial(createBillboardMaterial(material))

    // case 2: derive so that customRewriter is useful
    return createBillboardMaterial(super.createDerivedMaterial(material))
  }