Learning Aardvark.Media #1
This page is a central hub linking to all information necessary to program within the Aardvark platform, e.g. language resources, tutorials, and installation guides.
- What is aardvark media?
- What does it do?
- What it doesn't do?
Aardvark is mainly programmed in F#, a functional programming language. Hence, it is often said one has to 'unlearn' imperative programming styles and object oriented thinking. F# for Fun and Profit gives a great and complete overview of F# and also elaborates on why to use it. This is not about learning a new language, this is about evolving a new way to think, a new way to approach problems, and only ultimately expressing this in a new programming language.
Good starting points are the following two talks functional design principles and a functional approach to domain driven design. After getting an idea about the functional paradigms you can dive more into the syntax and language functions covered in the written series.
For programming web-based GUI elements within the Aardvark platform, it is important to know your way around web technologies, such as, HTML, CSS, SVG, and JavaScript. There is no really useful complete tutorial resource for our needs, so we will explain these parts by example in later sections. For most of our visible UI parts we found Semantic UI very useful, but you could use any other UI library together with Aardvark.Media.
The execution model of every App in Aardvark.Media follows the same pattern of unidirectional data flow, known as the ELM architecture (see ELM). It is most natural to start with the Model, which shall hold our data. The View describes how this data is visualized. In Aardvark.Media this is either a DomNode (part of an HTML page) or a SceneGraph node for 3D real-time rendering. The user can interact with what the view produces, e.g. clicking on a button or picking a 3D object, which in turn can generate actions. These actions strive to change our data, e.g. the color of an object the user selects in a colorpicker. The Update logic specifies how these actions change our data and generate a new model (immutable data). What we call an App is simply a combination of Model-Update-View and its Actions.
Apps exist at different granularities. There is an app for a numeric data field while a whole project is also just and app. Larger, more complex apps are simply composed together from smaller apps. It is quite natural that complex model types consist of more primitive model types and ultimately primitive datatypes on the lowest level. In later sections we will see how views, actions and update functions can easily be composed together to build powerful and maintainable apps.
The following example shall lead us through all topics relevant to building powerful apps in Aardvark.Media combining 3D graphics and a web-based GUI in a declarative and functional way. We will render a couple of 3D objects and transform them with controls we write from scratch. We demonstrate how the Model-Update-View paradigma works for generating GUI and 3D views and how to build powerful apps by composition. In the later chapters we we will demonstrate how our mod system and adaptive data structures handle change and are essential to tame complexity in larger apps.
First, we will build a numeric control, which lets the users increment and decrement an float value. We will approach this problem in the sequence of Model-Update-View, starting with the model.
We simply build a record type holding our float value. For future use, we also attribute it with 'DomainType'.
[<DomainType>]
type Model = { value : float }Specifying update is a two-fold effort of first defining actions of what does our app do to the model ...
type Action =
| Increment
| Decrement... and second of defining how do these actions affect our model.
let update (m : Model) (a : Action) =
match a with
| Increment -> { m with value = m.value + 0.1 }
| Decrement -> { m with value = m.value - 0.1 } The typical signature of update is Model->Action->Model, taking the current model and an action and returning an updated version of our model. Although, actions can in principle come from anywhere our actions for now will originate from our app's UI, which we specify in the view function. Consequently, the view function defines how our model is visualized and how the user can manipulate the model via UI elements.
First, we create two buttons , one for incrementing and one for decrementing our model's value.
button [clazz "ui button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui button"; onClick (fun _ -> Decrement)] [text "-"]Besides the architecture of ELM we were also inspired by ELMs way of constructing DOM (Document Object Model) elements, such as divs, buttons, and unordered lists. This allows us to emit HTML code for our views directly from F# code. The code above roughly equivalents to:
<button class="ui button" onClick=Increment()>+</button>
<button class="ui button" onClick=Decrement()>-</button>You may notice that we write the 'class' attribute as 'clazz'. This is not because we are the cool kids, but naturally 'class' is a reserved keyword in F#. Most of the html and Aardvark.UI tags are 'elems' consisting of an attribute list and a child list, while 'voidelems' such as 'br' only have attributes. A list of all available tags can be found here. The finished view function, also containing a visualization for our model, looks like the following:
let view (m : MModel) =
require Html.semui (
body [] (
[
div [] [
button [clazz "ui button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui button"; onClick (fun _ -> Decrement)] [text "-"]
br []
text "my value:"
br []
Incremental.text (m.value |> Mod.map(fun x -> sprintf "%.1f" x))
]
]
)
)'require Html.semui' enables the emitted html code to refer to web ressources such as css style sheets or javascript libraries. In this case we emit a script reference for the UI styling library Semantic UI. For now we ignore MModel, Incremental.text, and Mod.map which will be discussed in later sections.
The final lines of every app is putting the individual parts together...
let app =
{
unpersist = Unpersist.instance //ignore for now
threads = fun _ -> ThreadPool.empty //ignore for now
initial = { value = 0.0 }
update = update
view = view
}
let start() = App.start app...resulting in the following output:
Our vector control shall allow us to manipulate XYZ of a 3D vector. Naturally, we want to reuse our previous example and just combine three numeric controls via composition. First, we create a composed VectorModel by using a record type consisting of individual NumericModels for X,Y and Z.
[<DomainType>]
type VectorModel = {
x : NumericModel
y : NumericModel
z : NumericModel
}Our composed app now has a composed model and of course needs a composed action describing what our VectorControl can do. If we come accross an action in this control either X,Y or Z is incremented or decremented. However, increment and decrement are already captured in the NumericControl.Action, so we don't want to specify that again. Since an action is either an update on x or on y or on z our composed action is a union type.
type Action =
| UpdateX of NumericControl.Action
| UpdateY of NumericControl.Action
| UpdateZ of NumericControl.ActionLooking at Model and Action, our VectorControl is quite limited, it can't do anything by itself really. Therefore, our update funciton just simply calls update functions from the NumericControl.
let update (m : VectorModel) (a : Action) =
match a with
| UpdateX a -> { m with x = NumericControl.update m.x a }
| UpdateY a -> { m with y = NumericControl.update m.y a }
| UpdateZ a -> { m with z = NumericControl.update m.z a }The results are handed to the relevant part of our composed model, e.g. the X coordinate. The match on UpdateX unpacks the composed action a and we can directly pass a NumericControl.Action to update. Update returns a new NumericModel which we then use to update the VectorModel.
Such pure composition apps are quite frequent, for instance when making a property control consisting of numeric, bool, and string values, or when making a sidebar app consisting of multiple property apps. But more often composing apps add their own actions. In this case we could add Normalize and Reset to our VectorControl.
type Action =
| UpdateX of NumericControl.Action
| UpdateY of NumericControl.Action
| UpdateZ of NumericControl.Action
| Normalize
| Reset
let update (m : VectorModel) (a : Action) =
match a with
| UpdateX a -> { m with x = NumericControl.update m.x a }
| UpdateY a -> { m with y = NumericControl.update m.y a }
| UpdateZ a -> { m with z = NumericControl.update m.z a }
| Normalize ->
let v = V3d(m.x.value,m.y.value,m.z.value)
v.Normalized |> toVectorModel
| Reset -> VectorModel.initialBefore starting with the view, we will adapt the viewing code of the NumericControl a bit. In the way we programmed it above it is not very suitable to be stacked together into a VectorControl. That being said, a composing app is responsible for how the composed parts should be viewed together.
let view' (m : MNumericModel) =
require Html.semui (
table [][
tr[] [
td[] [a [clazz "ui label circular Big"] [Incremental.text (m.value |> Mod.map(fun x -> " " + x.ToString()))]]
td[] [
button [clazz "ui icon button"; onClick (fun _ -> Increment)] [text "+"]
button [clazz "ui icon button"; onClick (fun _ -> Decrement)] [text "-"]
]
]
]
)This is our updated view function view', which uses an html table to align things more neatly. The result looks like this:
Our VectorControl's view function should now call the individual NumericControl view functions and put them in a nice table layout.
let view (m : MVectorModel) =
require Html.semui (
div[][
table [] [
tr[][
td[][a [clazz "ui label circular Big"][text "X:"]]
td[][NumericControl.view' m.x |> UI.map UpdateX]
]
tr[][
td[][a [clazz "ui label circular Big"][text "Y:"]]
td[][NumericControl.view' m.y |> UI.map UpdateY]
]
tr[][
td[][a [clazz "ui label circular Big"][text "Z:"]]
td[][NumericControl.view' m.z |> UI.map UpdateZ]
]
tr[][
td[attribute "colspan" "2"][
div[clazz "ui buttons small"][
button [clazz "ui button"; onClick (fun _ -> Normalize)] [text "Norm"]
button [clazz "ui button"; onClick (fun _ -> Reset)] [text "Reset"]
]
]
]
]
]
)Everything looks quite familiar, except for UI.map. The view function of the VectorControl is supposed to return DomNode, while NumericControl.view' returns a DomNode<NumericControl.Action> resulting in a type mismatch. UI.map resolves this problem by lifting the action type of the subapp to the action type of the composing app by simply mapping it to the respective composing action.
The deep nesting of domnodes results from the quite cumbersome way to specify tables in html. Further, the vector control is probably not the most elegant control you have ever seen. However, we will stick to the this level of verbosity for explanatory reasons. Using the UI primitives of Aardvark.Media results in something more like this:
Html.SemUi.accordion "Scale" "Expand" true [
div [] [Vector3d.view model.scale |> UI.map V3dMessage]
]
In this chapter we covered how the meaning of model-update-view and what apps look like in Aardvark.Media. We built a small app from scratch and composed a more powerful app from that. We further illustrated how to use domnodes which ultimately emit html and javascript code for our GUI.
Before expanding our example with a 3D view we will revisit the ELM architecture and cover incremental evaluation.
When naively implementing the ELM architecture illustrated above we execute the whole view function on every update of the model. This does not scale well with larger DOM trees, larger SceneGraphs, nor high frequency updates, such as camera movement. We actually want to only update those parts of the scene which have actually changed - so we want incremental updates.
Aardvark.Rendering is a powerful rendering engine which supports incremental updates of 3D scenes, while Aardvark.Media does the same thing for DomNodes. We will not go into details here (refer to publications) but at the heart of all this is the Mod System. Instead of specifying on how to react to change explicitly a dependency is created between a data field in the model and a visual representation, e.g. a color : C4b and the rendering color of an Sg.Cube.
Within our nice unidirectional dataflow we don't want to use the mod system directly. Therefore, a compile step generates an MModel from our Model, translating every data field to a mod field which is aware of being changed. Most Sg nodes take mod version and for GUI elements there are incremental tags, such as Incremental.Text taking an IMod.
Here the incremental loop is illustrated with actual code. Our model contains a 3D vector (V3d) specifying a scale. Our compiler, the diff generator, generates an MModel wrapping the V3d into a mod. Our 3D view shows a cube which is transformed and further triggers a reset action on double click. Sg.trafo takes Mod. To generate a Mod we use Mod.map and speficy a mapping Mod -> Mod. The Sg view and the domnode view can both trigger a reset action which changes our model. This change is reflected in an update of MModel.value which only triggers a reevaluation of Sg.trafo.
-
mods
-
adaptive
-
MModels for 3D view
-
MModels for GUI views (AttributeMaps)
-
adaptive pendants of datastructures (alist, aset, amap, ...)
In the previous sections only dealt with very limited scene modifications, namely the change of a single value or property e.g. the scale of a box. In this section we will focus on how to handle structural changes to our scene. Our leading example will be the BoxSelectionDemo, also availabe in Aardvark.Media. Our application can add and remove boxes and further offers interaction to select and hover boxes in 3D as well as in a GUI list view. Adaptive datastructures are crucial for dealing with structural change, but we will also cover other relevant topics along the way, such as integrating a camera controler,...
When recalling the previous chapter, we got to know two types that 'react to change', IMod<'a> for wrapping a value into a Mod and the mutable type (MType) provided by the diffgenerator. Since functional programming is all about data structures we also need adaptive versions of collections such as lists, sequences, or maps.
This tables shows the compilation scheme of the diffgenerator, telling us which type is mapped to which adaptive type, for instance, a plist in our domain type Abc will be an alist in our mutable type MAbc. More details on how the diff generator converts one type to another can be found here.
Back to our BoxSelectionApp, we start again with the model
[<DomainType>]
type BoxSelectionDemoModel = {
boxes : plist<Box3d>
}containing a plist of Box3ds. In MBoxSelectionDemoModel this will be accessible for our view function as alist. We want our 3D view and our GUI, a list of boxes, to react to changes of this list, e.g. add a new 3D cube if a box is added. Alist not only reacts to structural changes, but also to changes to the properties of each element efficiently propagated via the dependency graph.
[<DomainType>]
type VisibleBox = {
geometry : Box3d
color : C4b
[<TreatAsValue>]
id : string
}
[<DomainType>]
type BoxSelectionDemoModel = {
camera : CameraControllerState
boxes : plist<VisibleBox>
boxHovered : option<string>
}To make our naked box geometries a bit more interesting we wrap them into VisibleBox which also holds a color and an id. The attribute TreatAsValue tells the diff generator to not compile this field to a Mod. Of course, we want our ids constant, otherwise it will be difficult to identify our boxes.
As you can see, we also included a CameraControllerState into our model. To have an interactive 3D view, we simply compose a CameraControllerApp into our app starting with just adding its model, which should rather be called CameraControllerModel if we were a 100% strict with the naming. Next, we wrap the camera controller actions into our BoxSelection actions and handle the action in the update function, as with any other composition.
type Action =
| CameraMessage of CameraControllerMessage
...
let update (model : BoxSelectionDemoModel) (act : Action) =
match act with
| CameraMessage m ->
{ model with camera = CameraController.update model.camera m }Finally, we embed the CameraController into our view function. The CameraController's view is exposed as an incremental control. On second thought, this is just another view function taking a model, an action, and some additonal attributes. Most notable here are the AttributeMap and the child nodes as last argument.
let view (model : MBoxSelectionDemoModel) =
let frustum = Mod.constant (Frustum.perspective 60.0 0.1 100.0 1.0)
div [clazz "ui"; style "background: #1B1C1E"] [
CameraController.controlledControl model.camera CameraMessage frustum
(AttributeMap.ofList [
attribute "style" "width:65%; height: 100%; float: left;"
])
(
Sg.box (Mod.constant C4b.Gray) (Mod.constant Box3d.Unit)
|> Sg.shader {
do! DefaultSurfaces.trafo
do! DefaultSurfaces.vertexColor
do! DefaultSurfaces.simpleLighting
}
|> Sg.requirePicking
|> Sg.noEvents
)
]AttributeMaps are a special way of defining attributes for a DomNode, which may change. In the above case we us static attributes for an incremental DomNode, so we can just use static syntax and convert it to an AttributeMap via AttributeMap.ofList. The child node of the renderControl is of the type ISg<'msg>, in our case a static grey box not capable of triggering any actions. Most interestingly the output of the RenderControl is a DomNode of Action, since the 3D rendering frames are written into an image. Therefore our RenderControl composes with any other DomNodes, just as a div or a text would. Our interactive intermediate result looks like this:
Before dealing with multiple boxes we think of the actions we want to support and implement them for a single box.
type Action =
| CameraMessage of CameraControllerMessage
| Select of string
| ClearSelection
| HoverIn of string
| HoverOut
| AddBox
| RemoveBox We want to click on a box and Select it resulting in a persistent highlighting (color red) or a peek highlighting on HoverIn (color blue) which stops on HoverOut. Similar to attaching actions to DomNodes (onclick button) we can specify which actions are triggered when clicking on a Sg.Box
Sg.box color box.geometry
|> Sg.shader {
do! DefaultSurfaces.trafo
do! DefaultSurfaces.vertexColor
do! DefaultSurfaces.simpleLighting
}
|> Sg.requirePicking
|> Sg.noEvents
|> Sg.withEvents [
Sg.onClick (fun _ -> Select (box.id |> Mod.force))
Sg.onEnter (fun _ -> Enter (box.id |> Mod.force))
Sg.onLeave (fun () -> Exit)
]Since we need to know which box has been selected/hovered later on we send the box.id with the action into the model update. With Mod.force we unpack box.id, which is a Mod, to a string. This is the only valid situation where we can just unpack a mod. Further, our id is not modifyable anyway in this case. In any other case we will most likely use adaptive behavior where we still need it. For simplicity we will we will only go through the hovering interaction here.
let update (model : BoxSelectionDemoModel) (act : Action) =
match act with
| CameraMessage m ->
{ model with camera = CameraController.update model.camera m }
| HoverIn id -> { model with boxHovered = Some id }
| HoverOut -> { model with boxHovered = None }
| _ -> model
We simple map our hovering interaction to the option of boxid. Either, there is an id or no hovered has taken place. Now we want to react on this changing hovering state by changing the cubes color. The diffgenerator gives us an IMod<Option> in the viewfunction, while we need an IMod. Once again, we want to just map things from IMod<'a> to Imod<'b> using Mod.map.
let color =
model.boxHovered |> Mod.map (
function x -> match x with
| Some k -> if k = "box" then C4b.Blue else C4b.Gray
| None -> C4b.Gray)- Incremental Domnodes and Attribute maps
- performance implications
- BoxSelection Demo
- Hands On with Aardvark
- Development Environment
- Development Tools
- Git
- Paket
- build
- build -install