Add other markdown files in the webkit source

Author: stwrt

Sources/WebKit/WebKit.docc/InDepth/DisplayTree.md

@@ -0,0 +1,127 @@
+# LFC Display Tree
+
+Layout Formatting Context display tree and related functionality (aka "LFC Display").
+
+## Basic concepts
+
+The **display tree** is intended to be a fully-resolved, isolated tree that is suitable for painting and hit-testing.
+
+*Fully-resolved* means that all style values have been resolved to the values use for painting. For example, colors in the display tree are those which result from the application of visited link rules, and from the application of `-apple-color-filter`.
+
+*Isolated* means that a display tree is a stand-alone data structure that can be painted without reference to data structures share with layout. For example, the display tree does not use `RenderStyle` once built, but constructs its own display styles at tree-building time.
+
+All the geometry in the display tree is already in painting coordinates: pixel snapping happens at tree building time.
+
+In general the tree has been designed to push as much of the complexity to tree-building time as possible, so that painting is fast.
+
+The tree objects themselves are intentionally lightweight, mostly **data-only** objects. Complex tree-walking code lives in external objects that know how to do certain things, like paint the box tree.
+
+Boxes in the display tree are intended to be fairly generic, and not have properties which are very CSS- or SVG-specific. It's hoped that SVG and CSS rendering share many of the same box classes.
+
+There isn't necessarily a 1:1 relationship between `Layout::Box` and `Display::Box`. The display tree may elide boxes that paint nothing, or may create extra boxes for functionality like scrolling.
+
+## Relevant classes
+
+### Display::View
+
+Represents a presentation of a display tree. Is the entry point for hit-testing.
+
+Currently `FrameView` owns a `Display::View`.
+
+### Display::Tree
+
+Owns the hierarchy of `Display::Box` objects via `Display::StackingItem` objects, and associated tree-related data structures. It should be possible for a `Display::Tree` to exist independently of a `Display::View`.
+
+### Display::TreeBuilder
+
+`Display::TreeBuilder` is a short-lived class that knows how to build a `Display::StackingItem`/`Display::Box` tree from a `Layout::Box` tree, `LayoutState` and style. It maintains various bits of state in a stack as it's building the display tree in order to track containing blocks etc. `Display::TreeBuilder` knows about hierarchy, not geometry.
+
+### Display::BoxFactory
+
+`Display::BoxFactory` actually creates the `Display::Box` objects and their related objects like `Display::BoxGeometry`. It exists to keep the style and geometry code out of `Display::TreeBuilder`. It only ever deals with a single box at a time.
+
+### Display::Box
+
+`Display::Box` is the basic building block for the display tree; it represents some rectangular region, often with painted content. There are subclasses for text, images etc. `Display::Box` has sibling pointers and a parent pointer. `Display::ContainerBox` is used for boxes with children, so has a pointer to its first child.
+
+The hierarchy of `Display::Box` objects forms a tree, but it's not a complete tree, because each `Display::StackingItem` owns a part of the tree, and the box objects in those subtrees are not connected.
+
+### Display::StackingItem
+
+This class exists to represent parts of the box tree that are painted atomically, as specified in [CSS 2.2 Appendix E](https://www.w3.org/TR/CSS22/zindex.html). There is always a `Display::StackingItem` at the root (via which the `Display::Tree` owns the actual box tree), and CSS positioning and other styles which cause CSS stacking context trigger the creation of a `Display::StackingItem`. `Display::StackingItem` objects that represent CSS stacking contexts have *positive* and *negative z-order lists* which contain owning references to the descendant `Display::StackingItem` objects. These lists are built and sorted at tree-building time.
+
+### Display::Style
+
+`Display::Style` is the display tree's equivalent of `RenderStyle` but with some important differences. Every `Display::Box` has a `Display::Style`, so it should be relatively small. It must only contain data that is independent of box geometry; i.e. it must not contain any `Length` values (which can hide things like `calc()` values that must have been already resolved). It should be *shareable* between boxes with the same appearance, but possibly different sizes. All colors should be their final used values, i.e. after resolving `visitedDependentColor()` and applying `-apple-color-filter`.
+
+### Display::BoxRareGeometry / Display::BoxDecorationData
+
+These classes store geometry-dependent data specific to a single `Display::Box`, and are only allocated for boxes that have the relevant styles (like visible borders, or border-radius).
+
+`Display::BoxDecorationData` stores information about backgrounds and borders.
+
+`Display::BoxRareGeometry` stores border-radius (here because a box with overflow:hidden clips to its border-radius even without any border), and transforms.
+
+### Display::CSSPainter
+
+`Display::CSSPainter` knows how to walk the trees of `Display::StackingItem` and their `Display::Box` objects in order to implement CSS painting. It knows how to apply clipping, opacity and filters.
+
+For painting individual boxes, `Display::CSSPainter` makes use of `Display::BoxDecorationPainter` which knows how to paint CSS borders and backgrounds.
+
+`Display::CSSPainter` objects exist on the stack only during painting; they are not long-lived.
+
+It's intended that SVG would be painted by a `Display::SVGPainter` class at some point.
+
+Hit-testing is very similar to painting, so that code lives in `Display::CSSPainter` too. Maybe it needs a different name.
+
+## Painting
+
+The entry point to painting some subset of the display tree is a `GraphicsLayerClient`'s `paintContents()` paint callback. Eventually the display tree will be presented via a hierarchy of GraphicsLayers; for now, there is a single GraphicsLayer at the root.
+
+To paint the tree, the static `CSSPainter::paintTree()` function is called, and `Display::CSSPainter` traverses the `Display::StackingItem` hierarchy and paints their `Display::Box` objects, making use of `Display::BoxDecorationPainter` on boxes with box decorations.
+
+At CSS/SVG boundaries a new painter object will be created on the stack to paint the inner content.
+
+The destination `GraphicsContext` is passed around as an argument to painting functions, wrapped up in a `PaintingContext` struct that also has the `deviceScaleFactor` and will probably include a dirty rect. 
+
+## Geometry
+
+    Note: this is subject to change. The code needs to make it hard to use the absolute rects in the wrong way.
+
+Display boxes store their position and size as an absolute (i.e. document-relative) `FloatRect` (already pixel-snapped). CSS box-model boxes, which have things like borders and border-radius, also store rects for their padding and content boxes. Thus no paint offsets have to be tracked during painting.
+
+Some boxes act as *coordinate boundaries*; namely those with transforms, and those which are affected by scrolling. The "absolute" rects are computed before accounting for scrolling and transforms, so code that needs to map a box's rect into view coordinate will need to take those into account.
+
+## Clipping
+
+Clipping is a source of complexity in the display tree because CSS clipping does not trigger CSS stacking context; the clipping tree follows the containing block tree, which is different from paint order. For painting normal in-flow content we can just apply clipping at paint time, but when painting out-of-flow content, we have to be able to compute the clip from ancestors. Such out-of-flow content is always represented by the root box of a `Display::StackingItem`.
+
+At tree-building time `Display::TreeBuilder` keeps track of containing blocks, so for boxes which initiate out-of-order painting (i.e. those which are the root box of a `Display::StackingItem`) `Display::BoxFactory` can ask the containing block for the clip that should be applied.
+
+Clips are store in `Display::BoxClip` which tracks an intersection rect, and, if necessary, a set of rounded rects needed to apply border-radius clips from ancestors. `Display::BoxClip` objects are shared between parent/child `Display::BoxModelObject` objects that share the same clip (i.e. have no overflow of their own).
+
+
+## Scrolling
+
+    Note: this is preliminary
+
+A `Layout::Box` with `overflow:auto` or `overflow:scroll` is represented with a hierarchy of three display boxes:
+
+```
+container box
+    scrolling container box
+        scrolled contents box
+
+```
+(and possibly additional boxes for scrollbars and the scroll corner).
+
+The container box exists to paint box decorations on the scrolling element. The "scrolling container" box has geometry of the container's *content box*, and exists to represent the scroll offset. The "scrolled contents" box paints the scrolled content, and has the size of the layout overflow.
+
+It is a goal of the display tree that scrolling should not invalidate any geometry on the subtree (thus avoiding post-scroll tree walks to fix up geometry, which are known to be a source of performance issues in rendering code).
+
+    Note: explain how scrolling interacts with positioned elements.
+
+## Compositing
+
+    Note: write me
+

Sources/WebKit/WebKit.docc/InDepth/MediaSourceExtensions.md

@@ -0,0 +1,77 @@
+# Media Source Extensions
+
+## Basic Concepts
+
+The Media Source Extensions specification defines a set of classes which allows clients to implement their own loading, buffering, and variant switching behavior, as opposed to requiring the UA to handle same.
+
+Clients `fetch()` media initialization segments and media segments, typically subsets of a single [fragmented MP4 file](https://www.w3.org/TR/mse-byte-stream-format-isobmff/) or [WebM file](https://www.w3.org/TR/mse-byte-stream-format-webm/), and append those segments into a SourceBuffer object, which is associated with a HTMLMediaElement through a [MediaSource](#mediasource) object.
+
+## Relevant Classes
+
+### MediaSource ###
+
+_([.idl](MediaSource.idl), [.h](MediaSource.h), [.cpp](MediaSource.cpp))_
+
+MediaSource serves two purposes:
+* Creating SourceBuffer objects.
+* Associating those SourceBuffer objects with a HTMLMediaElement.
+
+Once created, clients can create query for container and codec support via `isTypeSupported(type)`, [SourceBuffer](#sourcebuffer) objects via `addSourceBuffer(type)`, explicitly set the MediaSource's `duration`, and signal an end of the stream via `endOfStream(error)`.
+
+Before creating any [SourceBuffer](#sourcebuffer) objects, the MediaSource must be associated with a HTMLMediaElement. The MediaSource can be set directly as the HTMLMediaElement's `srcObject`. Alternatively, an [extension](DOMURL+MediaSource.idl) to DOMURL allows an ObjectURL to be created from a MediaSource object, and that ObjectURL can be set as the HTMLMediaElement's `src`.
+
+A MediaSource object will fire a `"sourceopen"` event when successfully associated with a HTMLMediaElement, and a `"sourceclose"` event when disassociated. The state of the MediaSource object can be queried via its `readyState` property.
+
+### SourceBuffer ###
+
+_([.idl](SourceBuffer.idl), [.h](SourceBuffer.h), [.cpp](SourceBuffer.cpp))_
+
+SourceBuffer accepts buffers of initialization segments and media segments, which are then parsed into media tracks and media samples. Those samples are cached within the SourceBuffer (inside its [SourceBufferPrivate](../platform/graphics/SourceBufferPrivate.h) object) and enqueued into platform-specific decoders on demand. The primary storage mechanism for these samples is a [SampleMap](#samplemap), which orders those samples both in terms of each sample's DecodeTime and PresentationTime. These two times can differ for codecs that support frame reordering, typically MPEG video codecs such as h.264 and HEVC.
+
+Clients append these segments via `appendBuffer()`, which sets an internal `updating` flag, fires the `"updatestart"` event, and subsequently fires the `"updateend"` event and clears the `updating` flag once parsing is complete. The results of the append are visible by querying the `buffered` property, or by querying the `audioTracks`, `videoTracks`, and `textTracks` TrackList objects.
+
+### MediaSourcePrivate ###
+
+_([.h](../platform/graphics/MediaSourcePrivate.h), [.cpp](../platform/graphics/MediaSourcePrivate.cpp))_
+
+MediaSourcePrivate is an abstract base class which allows [MediaSource](#mediasource) to communicate through the platform boundary to a platform-specific implementation of MediaSource.
+
+When the GPU Process is enabled, the MediaSourcePrivate in the WebContent process is typically a [MediaSourcePrivateRemote](../../WebKit/WebProcess/GPU/media/MediaSourcePrivateRemote.h), which will pass commands and properties across the WebContent/GPU process boundary.
+
+For Apple ports, the MediaSourcePrivate is typically a [MediaSourcePrivateAVFObjC](../platform/graphics/avfoundation/objc/MediaSourcePrivateAVFObjC.h).
+
+For GStreamer-based ports, the MediaSourcePrivate is typically a [MediaSourcePrivateGStreamer](../platform/graphics/gstreamer/mse/MediaSourcePrivateGStreamer.h).
+
+When running in DumpRenderTree/WebKitTestRunner, a "mock" MediaSourcePrivate can be enabled, and a [MockMediaSourcePrivate](../platform/mock/mediasource/MockMediaSourcePrivate.h) can be created. This is useful for writing platform-independent tests which exercise the platform-independent [MediaSource](#mediasource) and [SourceBuffer](#sourcebuffer) objects directly.
+
+### SourceBufferPrivate ###
+
+_([.h](../platform/graphics/SourceBufferPrivate.h), [.cpp](../platform/graphics/SourceBufferPrivate.cpp))_
+
+SourceBufferPrivate is a semi-abstract base class which accepts initialization segment and media segment buffers, parse those buffers with platform-specific parsers, and enqueue the resulting samples into platform-specific decoders. SourceBufferPrivate is also responsible for caching parsed samples in a [SampleMap](#samplemap).
+
+### MediaTime ### 
+
+_([.h](../../WTF/wtf/MediaTime.h), [.cpp](../../WTF/wtf/MediaTime.cpp))_
+
+MediaTime is a rational-number time class for manipulating time values commonly found in media files. The unit of MediaTime is seconds.
+
+Media containers such as mp4 and WebM represent time values as a ratio between a "time base" and a "time value". These values cannot necessarily be accurately represented as floating-point values without incurring cumulative rounding errors.  For example, a common frame rate in video formats is 29.97fps, however that value is an approximation of 30000/1001. So a media file containing a video track with a 29.97fps content will declare a "time base" scalar of 30000, and each frame will have a "time value" duration of 1001.
+
+Media Source Extension algorithms are very sensitive to small gaps between samples, and due to its rational-number behavior, MediaTime guarantees samples are contiguous by avoiding floating-point rounding errors.
+
+MediaTime offers convenience methods to convert from (`createTimeWithDouble()`) and to (`toDouble()`) floating-point values.
+
+### MediaSample ###
+
+_([.h](../platform/MediaSample.h), [.cpp](../platform/MediaSample.cpp))_
+
+MediaSample is an abstract base class representing a sample parsed from a media segment. MediaSamples have `presentationTime()`, `decodeTime()`, and `duration()`, each of which are [MediaTime](#mediatime) values, which are used to order these samples relative to one another in a [SampleMap](#samplemap). For codecs which support frame reordering, `presentationTime()` and `decodeTime()` for each sample may differ.
+
+### SampleMap ###
+
+_([.h](SampleMap.h), [.cpp](SampleMap.cpp))_
+
+SampleMap is a high-performance, binary-tree, storage structure for holding MediaSamples.
+
+Because decoders typically require frames to be enqueued in decode-time order, but many of the Media Source Extension algorithms work in presentation-time order, SampleMap contains two binary-tree structures: a `decodeOrder()` map, and a `presentationOrder()` map.

Sources/WebKit/WebKit.docc/WebKit.md

@@ -55,6 +55,8 @@ For details on submitting your code to the project, read [Contributing Code](htt
 
 ### In Depth Articles
 
+- <doc:DisplayTree>
+- <doc:MediaSourceExtensions>
 - <doc:MemoryManagement>
 - <doc:Libpas>
 - <doc:UnderstandingTheDOM>