Hit Testing Meta Issue

[schenney-chromium]

There are a few concerns about hit testing and APIs that use geometric information (like intersection observer).

The first is that the HTML content that is live in the canvas via placeElement should report geometry information that matches it's location within the canvas within the page. Presumably this also means things like offsetTop should report the real offset, and not the offset relative to the canvas or something else. The various uses should be reviewed as the spec is developed.

A more challenging problem is correctly hit testing when canvas content has been drawn on top of the placed element. How do we know whether the canvas of the element should receive the event? Canvas does not support methods for tracking hit test regions, but maybe it needs to now. Do we need to maintain coverage masks?

[khushalsagar]

There's a few issues brought up above and in other issues/offline conversations. I'll try to summarize them here with some initial thoughts, please add if I missed anything:

• The modifications on the element when drawing it within the canvas can conceptually be modeled as a CSS transform and clip-path. This means any APIs which deal with layout, like offsetTop, wouldn't be affected by how the element is rendered inside the canvas. Similar to how it doesn't change based on the CSS transform value.

  Canvas rendering of the element would also affect stacking, i.e., the element's z-order with respect to other canvas children. This can be treated similar to how z-index affects ordering? Basically instead of the element's z-index property the stacking order will be determined based on whether the element is placed and how it's placed relative to other elements.

• With respect to other content drawing on top of the canvas, for 2D we can track the bounding box of each canvas command. I don't think this is spec'd right now but seems feasible to do? At least in Blink I see there's already some tracking like this for internal optimizations here. Assume we can have a bounding box (maybe with a path based mask), we can use it to detect if there's canvas content drawing on top of the element. We might want to make this configurable, i.e., should a command consume events or not like pointer-events: none.

• A more fundamental question is how to deal with cases where rendering the element inside the canvas is not a simple transformations of the box. For 3D (and also drawElement mode in 2D), the element's texture can have modifications like breaking it up into multiple chunks or rendering it more than once. It's very difficult (or infeasible) to model this in how engine's track boxes for hit testing since a lot of those rendering patterns aren't possible today (-webkit-box-reflect comes closest to rendering the content more than once). So it's not clear how APIs like getBoundingClientRect and IntersectionObserver should behave. The only similar concept I can think of is fragmentation.

  We likely need a combo of 2 APIs here:

  1. A script callback to go from a point in canvas buffer space to the element hit along with the point mapped to the element's texture. And vice versa, an API which maps a point in the element's texture to its location in the canvas buffer. This would be used for precise hit testing for event dispatch/:hover.
  2. A declarative API which provides a transform, clip-path and stacking order of all elements rendered inside the canvas. This would be used for any platform APIs like getBoundingClientRect where we need a single bounding box for the rendered element.
     We could also consider regions for where each subset of the element is drawn.

[khushalsagar]

Copying over a related issue from whatwg/html#10650

And also things like IME popups - since the element's layout depends on the canvas itself, it is unclear how IME related popups can position themself correctly, if the visual representation on the canvas is in totally different place.

Since the browser anchors IME popups to the where the element is in viewport space, positioning IME popups is related to other APIs where we need to map a point in the rendered element to its corresponding position in the canvas buffer.

[chrishtr]

• like offsetTop,

What about getBoundingClientRect? That one is affected by transforms, and is often used to find the screen-space rect for an element.

A similar-feeling analysis was done as part of standardizing the zoom CSS property which may be useful in figuring out what to do here.

[khushalsagar]

What about getBoundingClientRect? That one is affected by transforms, and is often used to find the screen-space rect for an element.

That should be affected by where the element is rendered in the canvas, similar to how it's affected by CSS transform.

[foolip]

I discussed hit testing a bit with @Kangz, who noticed that CanvasHitTestRect is a rect.

To support basic rotations in 3D, should we be working with DOMQuad, effectively asking the developer to provide the 4 corners of the element?

For the problem of occlusion, wouldn't it suffice to add more hit test regions for the things that occlude, which are presumably also elements?

[schenney-chromium]

I was thinking of also providing a mask-based hit testing approach, in order to support partially obscured HTML content. I haven't completely thought through the details, but it seems you could maintain a black/white bitmap in an offscreen canvas and keep it updated as content was drawn in the primary canvas, then extract the bitmap for hit testing. The DOM element would get the event if the hit point passed the mask test.